Information Science and Statistics
Series Editors:
M. Jordan
J. Kleinberg
B. Schölkopf

Information Science and Statistics
Akaike and Kitagawa: The Practice of Time Series Analysis.
Bishop: Pattern Recognition and Machine Learning.
Cowell, Dawid, Lauritzen, and Spiegelhalter: Probabilistic Networks andExpert Systems.
Doucet, de Freitas, and Gordon: Sequential Monte Carlo Methods in Practice.
Fine: Feedforward Neural Network Methodology.
Hawkins and Olwell: Cumulative Sum Charts and Charting for Quality Improvement.
Jensen: Bayesian Networks and Decision Graphs.
Marchette: Computer Intrusion Detection and Network Monitoring:
A Statistical Viewpoint.
Rubinstein and Kroese: The Cross-Entropy Method: A Unified Approach to
Combinatorial Optimization, Monte Carlo Simulation, and Machine Learning.
Studený: Probabilistic Conditional Independence Structures.
Vapnik: The Nature of Statistical Learning Theory, Second Edition.
Wallace: Statistical and Inductive Inference by Minimum Massage Length.

Christopher M. Bishop

Pattern Recognition andMachine Learning

Christopher M. Bishop F.R.Eng.
Assistant Director
Microsoft Research Ltd
Cambridge CB3 0FB, U.K.
cmbishop@microsoft.com
http://research.microsoft.com/⬃cmbishop
Series Editors
Michael Jordan
Department of Computer
Science and Departmentof Statistics
University of California,
Berkeley
Berkeley, CA 94720
USA

Professor Jon Kleinberg
Department of Computer
Science
Cornell University
Ithaca, NY 14853
USA

Bernhard Schölkopf
Max Planck Institute for
Biological Cybernetics
Spemannstrasse 38
72076 Tübingen
Germany

Library of Congress Control Number: 2006922522
ISBN-10: 0-387-31073-8
ISBN-13: 978-0387-31073-2
Printed on acid-free paper.
© 2006 Springer Science+Business Media, LLC
All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher(Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection
with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation,
computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such,is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
Printed in Singapore.
9 8 7 6 5 4 3 2 1
springer.com(KYO)

This book is dedicated to my family:
Jenna, Mark, and HughTotal eclipse of the sun, Antalya, Turkey, 29 March 2006.

Preface
Pattern recognition has its origins in engineering, whereas machine learning grewout of computer science. However, these activities can be viewed as two facets ofthe same ﬁeld, and together they have undergone substantial development over the
past ten years. In particular, Bayesian methods have grown from a specialist niche tobecome mainstream, while graphical models have emerged as a general frameworkfor describing and applying probabilistic models. Also, the practical applicability ofBayesian methods has been greatly enhanced through the development of a range ofapproximate inference algorithms such as variational Bayes and expectation propagation. Similarly, new models based on kernels have had signiﬁcant impact on bothalgorithms and applications.
This new textbook reﬂects these recent developments while providing a comprehensive introduction to the ﬁelds of pattern recognition and machine learning. It isaimed at advanced undergraduates or ﬁrst year PhD students, as well as researchersand practitioners, and assumes no previous knowledge of pattern recognition or machine learning concepts. Knowledge of multivariate calculus and basic linear algebra
is required, and some familiarity with probabilities would be helpful though not essential as the book includes a self-contained introduction to basic probability theory.
Because this book has broad scope, it is impossible to provide a complete list ofreferences, and in particular no attempt has been made to provide accurate historicalattribution of ideas. Instead, the aim has been to give references that offer greater
detail than is possible here and that hopefully provide entry points into what, in somecases, is a very extensive literature. For this reason, the references are often to more
recent textbooks and review articles rather than to original sources.
The book is supported by a great deal of additional material, including lectureslides as well as the complete set of ﬁgures used in the book, and the reader is
encouraged to visit the book web site for the latest information:

http://research.microsoft.com/∼cmbishop/PRML

vii

viii

PREFACE

Exercises
The exercises that appear at the end of every chapter form an important component of the book. Each exercise has been carefully chosen to reinforce concepts
explained in the text or to develop and generalize them in signiﬁcant ways, and eachis graded according to difﬁculty ranging from (), which denotes a simple exercise
taking a few minutes to complete, through to (  ), which denotes a signiﬁcantly
more complex exercise.
It has been difﬁcult to know to what extent these solutions should be madewidely available. Those engaged in self study will ﬁnd worked solutions very beneﬁcial, whereas many course tutors request that solutions be available only via the
publisher so that the exercises may be used in class. In order to try to meet these
conﬂicting requirements, those exercises that help amplify key points in the text, orthat ﬁll in important details, have solutions that are available as a PDF ﬁle from thebook web site. Such exercises are denoted by www . Solutions for the remaining
exercises are available to course tutors by contacting the publisher (contact details
are given on the book web site). Readers are strongly encouraged to work throughthe exercises unaided, and to turn to the solutions only as required.
Although this book focuses on concepts and principles, in a taught course thestudents should ideally have the opportunity to experiment with some of the key
algorithms using appropriate data sets. A companion volume (Bishop and Nabney,2008) will deal with practical aspects of pattern recognition and machine learning,
and will be accompanied by Matlab software implementing most of the algorithms
discussed in this book.

Acknowledgements
First of all I would like to express my sincere thanks to Markus Svensén who
has provided immense help with preparation of ﬁgures and with the typesetting of
the book in LATEX. His assistance has been invaluable.
I am very grateful to Microsoft Research for providing a highly stimulating research environment and for giving me the freedom to write this book (the views andopinions expressed in this book, however, are my own and are therefore not necessarily the same as those of Microsoft or its afﬁliates).
Springer has provided excellent support throughout the ﬁnal stages of preparation of this book, and I would like to thank my commissioning editor John Kimmel
for his support and professionalism, as well as Joseph Piliero for his help in designing the cover and the text format and MaryAnn Brickner for her numerous contributions during the production phase. The inspiration for the cover design came from a
discussion with Antonio Criminisi.
I also wish to thank Oxford University Press for permission to reproduce excerpts from an earlier textbook, Neural Networks for Pattern Recognition (Bishop,
1995a). The images of the Mark 1 perceptron and of Frank Rosenblatt are reproduced with the permission of Arvin Calspan Advanced Technology Center. I would
also like to thank Asela Gunawardana for plotting the spectrogram in Figure 13.1,
and Bernhard Schölkopf for permission to use his kernel PCA code to plot Figure 12.17.

PREFACE

ix

Many people have helped by proofreading draft material and providing comments and suggestions, including Shivani Agarwal, Cédric Archambeau, Arik Azran,
Andrew Blake, Hakan Cevikalp, Michael Fourman, Brendan Frey, Zoubin Ghahramani, Thore Graepel, Katherine Heller, Ralf Herbrich, Geoffrey Hinton, Adam Johansen, Matthew Johnson, Michael Jordan, Eva Kalyvianaki, Anitha Kannan, Julia
Lasserre, David Liu, Tom Minka, Ian Nabney, Tonatiuh Pena, Yuan Qi, Sam Roweis,
Balaji Sanjiya, Toby Sharp, Ana Costa e Silva, David Spiegelhalter, Jay Stokes, Tara
Symeonides, Martin Szummer, Marshall Tappen, Ilkay Ulusoy, Chris Williams, John
Winn, and Andrew Zisserman.
Finally, I would like to thank my wife Jenna who has been hugely supportive
throughout the several years it has taken to write this book.
Chris Bishop
Cambridge
February 2006

Mathematical notation
I have tried to keep the mathematical content of the book to the minimum necessary to achieve a proper understanding of the ﬁeld. However, this minimum level isnonzero, and it should be emphasized that a good grasp of calculus, linear algebra,
and probability theory is essential for a clear understanding of modern pattern recognition and machine learning techniques. Nevertheless, the emphasis in this book is
on conveying the underlying concepts rather than on mathematical rigour.
I have tried to use a consistent notation throughout the book, although at times
this means departing from some of the conventions used in the corresponding research literature. Vectors are denoted by lower case bold Roman letters such asx, and all vectors are assumed to be column vectors. A superscript T denotes thetranspose of a matrix or vector, so that xT will be a row vector. Uppercase boldroman letters, such as M, denote matrices. The notation (w1 , . . . , wM ) denotes a
row vector with M elements, while the corresponding column vector is written as
w = (w1 , . . . , wM )T .
The notation [a, b] is used to denote the closed interval from a to b, that is theinterval including the values a and b themselves, while (a, b) denotes the corresponding open interval, that is the interval excluding a and b. Similarly, [a, b) denotes aninterval that includes a but excludes b. For the most part, however, there will belittle need to dwell on such reﬁnements as whether the end points of an interval are
included or not.
The M × M identity matrix (also known as the unit matrix) is denoted IM ,which will be abbreviated to I where there is no ambiguity about it dimensionality.
It has elements Iij that equal 1 if i = j and 0 if i = j.
A functional is denoted f [y] where y(x) is some function. The concept of afunctional is discussed in Appendix D.
The notation g(x) = O(f (x)) denotes that |f (x)/g(x)| is bounded as x → ∞.
For instance if g(x) = 3x2 + 2, then g(x) = O(x2 ).
The expectation of a function f (x, y) with respect to a random variable x is denoted by Ex [f (x, y)]. In situations where there is no ambiguity as to which variable
is being averaged over, this will be simpliﬁed by omitting the sufﬁx, for instance
xi

xii

MATHEMATICAL NOTATION
E[x]. If the distribution of x is conditioned on another variable z, then the corresponding conditional expectation will be written Ex [f (x)|z]. Similarly, the varianceis denoted var[f (x)], and for vector variables the covariance is written cov[x, y]. We
shall also use cov[x] as a shorthand notation for cov[x, x]. The concepts of expectations and covariances are introduced in Section 1.2.2.
If we have N values x1 , . . . , xN of a D-dimensional vector x = (x1 , . . . , xD )T ,we can combine the observations into a data matrix X in which the nth row of X
corresponds to the row vector xT
n . Thus the n, i element of X corresponds to theith element of the nth observation xn . For the case of one-dimensional variables weshall denote such a matrix by x, which is a column vector whose nth element is xn .
Note that x (which has dimensionality N ) uses a different typeface to distinguish itfrom x (which has dimensionality D).

Contents
Prefacevii

Mathematical notation

xi

Contents

xiii

1

1
4
12
17
19
21
24
28
30
32
33
38
39
41
42
42
46
48
55
58

Introduction
1.1 Example: Polynomial Curve Fitting . . . . . . . . . . . . . . . . .
1.2 Probability Theory . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.1 Probability densities . . . . . . . . . . . . . . . . . . . . .
1.2.2 Expectations and covariances . . . . . . . . . . . . . . . .
1.2.3 Bayesian probabilities . . . . . . . . . . . . . . . . . . . .
1.2.4 The Gaussian distribution . . . . . . . . . . . . . . . . . .
1.2.5 Curve ﬁtting re-visited . . . . . . . . . . . . . . . . . . . .
1.2.6 Bayesian curve ﬁtting . . . . . . . . . . . . . . . . . . . .
1.3 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 The Curse of Dimensionality . . . . . . . . . . . . . . . . . . . . .
1.5 Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5.1 Minimizing the misclassiﬁcation rate . . . . . . . . . . . .
1.5.2 Minimizing the expected loss . . . . . . . . . . . . . . . .
1.5.3 The reject option . . . . . . . . . . . . . . . . . . . . . . .
1.5.4 Inference and decision . . . . . . . . . . . . . . . . . . . .
1.5.5 Loss functions for regression . . . . . . . . . . . . . . . . .
1.6 Information Theory . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6.1 Relative entropy and mutual information . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xiii

xiv

CONTENTS
2

Probability Distributions
2.1 Binary Variables . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 The beta distribution . . . . . . . . . . . . . . . . . . . . .
2.2 Multinomial Variables . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 The Dirichlet distribution . . . . . . . . . . . . . . . . . . .
2.3 The Gaussian Distribution . . . . . . . . . . . . . . . . . . . . . .
2.3.1 Conditional Gaussian distributions . . . . . . . . . . . . . .
2.3.2 Marginal Gaussian distributions . . . . . . . . . . . . . . .
2.3.3 Bayes’ theorem for Gaussian variables . . . . . . . . . . . .
2.3.4 Maximum likelihood for the Gaussian . . . . . . . . . . . .
2.3.5 Sequential estimation . . . . . . . . . . . . . . . . . . . . .
2.3.6 Bayesian inference for the Gaussian . . . . . . . . . . . . .
2.3.7 Student’s t-distribution . . . . . . . . . . . . . . . . . . . .
2.3.8 Periodic variables . . . . . . . . . . . . . . . . . . . . . . .
2.3.9 Mixtures of Gaussians . . . . . . . . . . . . . . . . . . . .
2.4 The Exponential Family . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 Maximum likelihood and sufﬁcient statistics . . . . . . . .
2.4.2 Conjugate priors . . . . . . . . . . . . . . . . . . . . . . .
2.4.3 Noninformative priors . . . . . . . . . . . . . . . . . . . .
2.5 Nonparametric Methods . . . . . . . . . . . . . . . . . . . . . . .
2.5.1 Kernel density estimators . . . . . . . . . . . . . . . . . . .
2.5.2 Nearest-neighbour methods . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

67
68
71
74
76
78
85
88
90
93
94
97
102
105
110
113
116
117
117
120
122
124
127

3

Linear Models for Regression
3.1 Linear Basis Function Models . . . . . . . . . . . . . . . . . . . .
3.1.1 Maximum likelihood and least squares . . . . . . . . . . . .
3.1.2 Geometry of least squares . . . . . . . . . . . . . . . . . .
3.1.3 Sequential learning . . . . . . . . . . . . . . . . . . . . . .
3.1.4 Regularized least squares . . . . . . . . . . . . . . . . . . .
3.1.5 Multiple outputs . . . . . . . . . . . . . . . . . . . . . . .
3.2 The Bias-Variance Decomposition . . . . . . . . . . . . . . . . . .
3.3 Bayesian Linear Regression . . . . . . . . . . . . . . . . . . . . .
3.3.1 Parameter distribution . . . . . . . . . . . . . . . . . . . .
3.3.2 Predictive distribution . . . . . . . . . . . . . . . . . . . .
3.3.3 Equivalent kernel . . . . . . . . . . . . . . . . . . . . . . .
3.4 Bayesian Model Comparison . . . . . . . . . . . . . . . . . . . . .
3.5 The Evidence Approximation . . . . . . . . . . . . . . . . . . . .
3.5.1 Evaluation of the evidence function . . . . . . . . . . . . .
3.5.2 Maximizing the evidence function . . . . . . . . . . . . . .
3.5.3 Effective number of parameters . . . . . . . . . . . . . . .
3.6 Limitations of Fixed Basis Functions . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

137
138
140
143
143
144
146
147
152
152
156
159
161
165
166
168
170
172
173

CONTENTS

xv

4

Linear Models for Classiﬁcation
4.1 Discriminant Functions . . . . . . . . . . . . . . . . . . . . . . . .
4.1.1 Two classes . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1.2 Multiple classes . . . . . . . . . . . . . . . . . . . . . . . .
4.1.3 Least squares for classiﬁcation . . . . . . . . . . . . . . . .
4.1.4 Fisher’s linear discriminant . . . . . . . . . . . . . . . . . .
4.1.5 Relation to least squares . . . . . . . . . . . . . . . . . . .
4.1.6 Fisher’s discriminant for multiple classes . . . . . . . . . .
4.1.7 The perceptron algorithm . . . . . . . . . . . . . . . . . . .
4.2 Probabilistic Generative Models . . . . . . . . . . . . . . . . . . .
4.2.1 Continuous inputs . . . . . . . . . . . . . . . . . . . . . .
4.2.2 Maximum likelihood solution . . . . . . . . . . . . . . . .
4.2.3 Discrete features . . . . . . . . . . . . . . . . . . . . . . .
4.2.4 Exponential family . . . . . . . . . . . . . . . . . . . . . .
4.3 Probabilistic Discriminative Models . . . . . . . . . . . . . . . . .
4.3.1 Fixed basis functions . . . . . . . . . . . . . . . . . . . . .
4.3.2 Logistic regression . . . . . . . . . . . . . . . . . . . . . .
4.3.3 Iterative reweighted least squares . . . . . . . . . . . . . .
4.3.4 Multiclass logistic regression . . . . . . . . . . . . . . . . .
4.3.5 Probit regression . . . . . . . . . . . . . . . . . . . . . . .
4.3.6 Canonical link functions . . . . . . . . . . . . . . . . . . .
4.4 The Laplace Approximation . . . . . . . . . . . . . . . . . . . . .
4.4.1 Model comparison and BIC . . . . . . . . . . . . . . . . .
4.5 Bayesian Logistic Regression . . . . . . . . . . . . . . . . . . . .
4.5.1 Laplace approximation . . . . . . . . . . . . . . . . . . . .
4.5.2 Predictive distribution . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

179
181
181
182
184
186
189
191
192
196
198
200
202
202
203
204
205
207
209
210
212
213
216
217
217
218
220

5

Neural Networks
5.1 Feed-forward Network Functions . . . . . . . . . . . . . . . . . .
5.1.1 Weight-space symmetries . . . . . . . . . . . . . . . . . .
5.2 Network Training . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1 Parameter optimization . . . . . . . . . . . . . . . . . . . .
5.2.2 Local quadratic approximation . . . . . . . . . . . . . . . .
5.2.3 Use of gradient information . . . . . . . . . . . . . . . . .
5.2.4 Gradient descent optimization . . . . . . . . . . . . . . . .
5.3 Error Backpropagation . . . . . . . . . . . . . . . . . . . . . . . .
5.3.1 Evaluation of error-function derivatives . . . . . . . . . . .
5.3.2 A simple example . . . . . . . . . . . . . . . . . . . . . .
5.3.3 Efﬁciency of backpropagation . . . . . . . . . . . . . . . .
5.3.4 The Jacobian matrix . . . . . . . . . . . . . . . . . . . . .
5.4 The Hessian Matrix . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.1 Diagonal approximation . . . . . . . . . . . . . . . . . . .
5.4.2 Outer product approximation . . . . . . . . . . . . . . . . .
5.4.3 Inverse Hessian . . . . . . . . . . . . . . . . . . . . . . . .

225
227
231
232
236
237
239
240
241
242
245
246
247
249
250
251
252

xvi

CONTENTS
5.4.4 Finite differences . . . . . . . . . . . . . . . . . . . . . . .
5.4.5 Exact evaluation of the Hessian . . . . . . . . . . . . . . .
5.4.6 Fast multiplication by the Hessian . . . . . . . . . . . . . .
5.5 Regularization in Neural Networks . . . . . . . . . . . . . . . . .
5.5.1 Consistent Gaussian priors . . . . . . . . . . . . . . . . . .
5.5.2 Early stopping . . . . . . . . . . . . . . . . . . . . . . . .
5.5.3 Invariances . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5.4 Tangent propagation . . . . . . . . . . . . . . . . . . . . .
5.5.5 Training with transformed data . . . . . . . . . . . . . . . .
5.5.6 Convolutional networks . . . . . . . . . . . . . . . . . . .
5.5.7 Soft weight sharing . . . . . . . . . . . . . . . . . . . . . .
5.6 Mixture Density Networks . . . . . . . . . . . . . . . . . . . . . .
5.7 Bayesian Neural Networks . . . . . . . . . . . . . . . . . . . . . .
5.7.1 Posterior parameter distribution . . . . . . . . . . . . . . .
5.7.2 Hyperparameter optimization . . . . . . . . . . . . . . . .
5.7.3 Bayesian neural networks for classiﬁcation . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

252
253
254
256
257
259
261
263
265
267
269
272
277
278
280
281
284

6

Kernel Methods
6.1 Dual Representations . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Constructing Kernels . . . . . . . . . . . . . . . . . . . . . . . . .
6.3 Radial Basis Function Networks . . . . . . . . . . . . . . . . . . .
6.3.1 Nadaraya-Watson model . . . . . . . . . . . . . . . . . . .
6.4 Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4.1 Linear regression revisited . . . . . . . . . . . . . . . . . .
6.4.2 Gaussian processes for regression . . . . . . . . . . . . . .
6.4.3 Learning the hyperparameters . . . . . . . . . . . . . . . .
6.4.4 Automatic relevance determination . . . . . . . . . . . . .
6.4.5 Gaussian processes for classiﬁcation . . . . . . . . . . . . .
6.4.6 Laplace approximation . . . . . . . . . . . . . . . . . . . .
6.4.7 Connection to neural networks . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

291
293
294
299
301
303
304
306
311
312
313
315
319
320

7

Sparse Kernel Machines
7.1 Maximum Margin Classiﬁers . . . . . . . . . . . . . . . . . . . .
7.1.1 Overlapping class distributions . . . . . . . . . . . . . . . .
7.1.2 Relation to logistic regression . . . . . . . . . . . . . . . .
7.1.3 Multiclass SVMs . . . . . . . . . . . . . . . . . . . . . . .
7.1.4 SVMs for regression . . . . . . . . . . . . . . . . . . . . .
7.1.5 Computational learning theory . . . . . . . . . . . . . . . .
7.2 Relevance Vector Machines . . . . . . . . . . . . . . . . . . . . .
7.2.1 RVM for regression . . . . . . . . . . . . . . . . . . . . . .
7.2.2 Analysis of sparsity . . . . . . . . . . . . . . . . . . . . . .
7.2.3 RVM for classiﬁcation . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

325
326
331
336
338
339
344
345
345
349
353
357

CONTENTS

xvii

8

Graphical Models
8.1 Bayesian Networks . . . . . . . . . . . . . . . . . . . . . . . . . .
8.1.1 Example: Polynomial regression . . . . . . . . . . . . . . .
8.1.2 Generative models . . . . . . . . . . . . . . . . . . . . . .
8.1.3 Discrete variables . . . . . . . . . . . . . . . . . . . . . . .
8.1.4 Linear-Gaussian models . . . . . . . . . . . . . . . . . . .
8.2 Conditional Independence . . . . . . . . . . . . . . . . . . . . . .
8.2.1 Three example graphs . . . . . . . . . . . . . . . . . . . .
8.2.2 D-separation . . . . . . . . . . . . . . . . . . . . . . . . .
8.3 Markov Random Fields . . . . . . . . . . . . . . . . . . . . . . .
8.3.1 Conditional independence properties . . . . . . . . . . . . .
8.3.2 Factorization properties . . . . . . . . . . . . . . . . . . .
8.3.3 Illustration: Image de-noising . . . . . . . . . . . . . . . .
8.3.4 Relation to directed graphs . . . . . . . . . . . . . . . . . .
8.4 Inference in Graphical Models . . . . . . . . . . . . . . . . . . . .
8.4.1 Inference on a chain . . . . . . . . . . . . . . . . . . . . .
8.4.2 Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.4.3 Factor graphs . . . . . . . . . . . . . . . . . . . . . . . . .
8.4.4 The sum-product algorithm . . . . . . . . . . . . . . . . . .
8.4.5 The max-sum algorithm . . . . . . . . . . . . . . . . . . .
8.4.6 Exact inference in general graphs . . . . . . . . . . . . . .
8.4.7 Loopy belief propagation . . . . . . . . . . . . . . . . . . .
8.4.8 Learning the graph structure . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

359
360
362
365
366
370
372
373
378
383
383
384
387
390
393
394
398
399
402
411
416
417
418
418

9

Mixture Models and EM
9.1 K-means Clustering . . . . . . . . . . . . . . . . . . . . . . . . .
9.1.1 Image segmentation and compression . . . . . . . . . . . .
9.2 Mixtures of Gaussians . . . . . . . . . . . . . . . . . . . . . . . .
9.2.1 Maximum likelihood . . . . . . . . . . . . . . . . . . . . .
9.2.2 EM for Gaussian mixtures . . . . . . . . . . . . . . . . . .
9.3 An Alternative View of EM . . . . . . . . . . . . . . . . . . . . .
9.3.1 Gaussian mixtures revisited . . . . . . . . . . . . . . . . .
9.3.2 Relation to K-means . . . . . . . . . . . . . . . . . . . . .
9.3.3 Mixtures of Bernoulli distributions . . . . . . . . . . . . . .
9.3.4 EM for Bayesian linear regression . . . . . . . . . . . . . .
9.4 The EM Algorithm in General . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

423
424
428
430
432
435
439
441
443
444
448
450
455

10 Approximate Inference
10.1 Variational Inference . . . . . . . . . . . . . . . . . . . . . . . . .
10.1.1 Factorized distributions . . . . . . . . . . . . . . . . . . . .
10.1.2 Properties of factorized approximations . . . . . . . . . . .
10.1.3 Example: The univariate Gaussian . . . . . . . . . . . . . .
10.1.4 Model comparison . . . . . . . . . . . . . . . . . . . . . .
10.2 Illustration: Variational Mixture of Gaussians . . . . . . . . . . . .

461
462
464
466
470
473
474

xviii

CONTENTS
10.2.1 Variational distribution . . . . . . . . . . . . . . . . . . . .
10.2.2 Variational lower bound . . . . . . . . . . . . . . . . . . .
10.2.3 Predictive density . . . . . . . . . . . . . . . . . . . . . . .
10.2.4 Determining the number of components . . . . . . . . . . .
10.2.5 Induced factorizations . . . . . . . . . . . . . . . . . . . .
10.3 Variational Linear Regression . . . . . . . . . . . . . . . . . . . .
10.3.1 Variational distribution . . . . . . . . . . . . . . . . . . . .
10.3.2 Predictive distribution . . . . . . . . . . . . . . . . . . . .
10.3.3 Lower bound . . . . . . . . . . . . . . . . . . . . . . . . .
10.4 Exponential Family Distributions . . . . . . . . . . . . . . . . . .
10.4.1 Variational message passing . . . . . . . . . . . . . . . . .
10.5 Local Variational Methods . . . . . . . . . . . . . . . . . . . . . .
10.6 Variational Logistic Regression . . . . . . . . . . . . . . . . . . .
10.6.1 Variational posterior distribution . . . . . . . . . . . . . . .
10.6.2 Optimizing the variational parameters . . . . . . . . . . . .
10.6.3 Inference of hyperparameters . . . . . . . . . . . . . . . .
10.7 Expectation Propagation . . . . . . . . . . . . . . . . . . . . . . .
10.7.1 Example: The clutter problem . . . . . . . . . . . . . . . .
10.7.2 Expectation propagation on graphs . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

475
481
482
483
485
486
486
488
489
490
491
493
498
498
500
502
505
511
513
517

11 Sampling Methods
11.1 Basic Sampling Algorithms . . . . . . . . . . . . . . . . . . . . .
11.1.1 Standard distributions . . . . . . . . . . . . . . . . . . . .
11.1.2 Rejection sampling . . . . . . . . . . . . . . . . . . . . . .
11.1.3 Adaptive rejection sampling . . . . . . . . . . . . . . . . .
11.1.4 Importance sampling . . . . . . . . . . . . . . . . . . . . .
11.1.5 Sampling-importance-resampling . . . . . . . . . . . . . .
11.1.6 Sampling and the EM algorithm . . . . . . . . . . . . . . .
11.2 Markov Chain Monte Carlo . . . . . . . . . . . . . . . . . . . . .
11.2.1 Markov chains . . . . . . . . . . . . . . . . . . . . . . . .
11.2.2 The Metropolis-Hastings algorithm . . . . . . . . . . . . .
11.3 Gibbs Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.4 Slice Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11.5 The Hybrid Monte Carlo Algorithm . . . . . . . . . . . . . . . . .
11.5.1 Dynamical systems . . . . . . . . . . . . . . . . . . . . . .
11.5.2 Hybrid Monte Carlo . . . . . . . . . . . . . . . . . . . . .
11.6 Estimating the Partition Function . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

523
526
526
528
530
532
534
536
537
539
541
542
546
548
548
552
554
556

12 Continuous Latent Variables
12.1 Principal Component Analysis . . . . . . . . . . . . . . . . . . . .
12.1.1 Maximum variance formulation . . . . . . . . . . . . . . .
12.1.2 Minimum-error formulation . . . . . . . . . . . . . . . . .
12.1.3 Applications of PCA . . . . . . . . . . . . . . . . . . . . .
12.1.4 PCA for high-dimensional data . . . . . . . . . . . . . . .

559
561
561
563
565
569

CONTENTS

xix

12.2 Probabilistic PCA . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2.1 Maximum likelihood PCA . . . . . . . . . . . . . . . . . .
12.2.2 EM algorithm for PCA . . . . . . . . . . . . . . . . . . . .
12.2.3 Bayesian PCA . . . . . . . . . . . . . . . . . . . . . . . .
12.2.4 Factor analysis . . . . . . . . . . . . . . . . . . . . . . . .
12.3 Kernel PCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.4 Nonlinear Latent Variable Models . . . . . . . . . . . . . . . . . .
12.4.1 Independent component analysis . . . . . . . . . . . . . . .
12.4.2 Autoassociative neural networks . . . . . . . . . . . . . . .
12.4.3 Modelling nonlinear manifolds . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

570
574
577
580
583
586
591
591
592
595
599

13 Sequential Data
13.1 Markov Models . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.2 Hidden Markov Models . . . . . . . . . . . . . . . . . . . . . . .
13.2.1 Maximum likelihood for the HMM . . . . . . . . . . . . .
13.2.2 The forward-backward algorithm . . . . . . . . . . . . . .
13.2.3 The sum-product algorithm for the HMM . . . . . . . . . .
13.2.4 Scaling factors . . . . . . . . . . . . . . . . . . . . . . . .
13.2.5 The Viterbi algorithm . . . . . . . . . . . . . . . . . . . . .
13.2.6 Extensions of the hidden Markov model . . . . . . . . . . .
13.3 Linear Dynamical Systems . . . . . . . . . . . . . . . . . . . . . .
13.3.1 Inference in LDS . . . . . . . . . . . . . . . . . . . . . . .
13.3.2 Learning in LDS . . . . . . . . . . . . . . . . . . . . . . .
13.3.3 Extensions of LDS . . . . . . . . . . . . . . . . . . . . . .
13.3.4 Particle ﬁlters . . . . . . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

605
607
610
615
618
625
627
629
631
635
638
642
644
645
646

14 Combining Models
14.1 Bayesian Model Averaging . . . . . . . . . . . . . . . . . . . . . .
14.2 Committees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.3 Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.3.1 Minimizing exponential error . . . . . . . . . . . . . . . .
14.3.2 Error functions for boosting . . . . . . . . . . . . . . . . .
14.4 Tree-based Models . . . . . . . . . . . . . . . . . . . . . . . . . .
14.5 Conditional Mixture Models . . . . . . . . . . . . . . . . . . . . .
14.5.1 Mixtures of linear regression models . . . . . . . . . . . . .
14.5.2 Mixtures of logistic models . . . . . . . . . . . . . . . . .
14.5.3 Mixtures of experts . . . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

653
654
655
657
659
661
663
666
667
670
672
674

Appendix A

Data Sets

677

Appendix B

Probability Distributions

685

Appendix C

Properties of Matrices

695

xx

CONTENTS
Appendix D

Calculus of Variations

703

Appendix E

Lagrange Multipliers

707

References

711

Index

729

1
Introduction

The problem of searching for patterns in data is a fundamental one and has a long andsuccessful history. For instance, the extensive astronomical observations of Tycho
Brahe in the 16th century allowed Johannes Kepler to discover the empirical laws ofplanetary motion, which in turn provided a springboard for the development of classical mechanics. Similarly, the discovery of regularities in atomic spectra played a
key role in the development and veriﬁcation of quantum physics in the early twentieth century. The ﬁeld of pattern recognition is concerned with the automatic discovery of regularities in data through the use of computer algorithms and with the use of
these regularities to take actions such as classifying the data into different categories.
Consider the example of recognizing handwritten digits, illustrated in Figure 1.1.
Each digit corresponds to a 28×28 pixel image and so can be represented by a vector
x comprising 784 real numbers. The goal is to build a machine that will take such avector x as input and that will produce the identity of the digit 0, . . . , 9 as the output.
This is a nontrivial problem due to the wide variability of handwriting. It could be12

1. INTRODUCTION
Figure 1.1

Examples of hand-written digits taken from US zip codes.

tackled using handcrafted rules or heuristics for distinguishing the digits based on
the shapes of the strokes, but in practice such an approach leads to a proliferation ofrules and of exceptions to the rules and so on, and invariably gives poor results.
Far better results can be obtained by adopting a machine learning approach inwhich a large set of N digits {x1 , . . . , xN } called a training set is used to tune theparameters of an adaptive model. The categories of the digits in the training set
are known in advance, typically by inspecting them individually and hand-labellingthem. We can express the category of a digit using target vector t, which represents
the identity of the corresponding digit. Suitable techniques for representing categories in terms of vectors will be discussed later. Note that there is one such target
vector t for each digit image x.
The result of running the machine learning algorithm can be expressed as afunction y(x) which takes a new digit image x as input and that generates an output
vector y, encoded in the same way as the target vectors. The precise form of thefunction y(x) is determined during the training phase, also known as the learning
phase, on the basis of the training data. Once the model is trained it can then determine the identity of new digit images, which are said to comprise a test set. The
ability to categorize correctly new examples that differ from those used for training is known as generalization. In practical applications, the variability of the input
vectors will be such that the training data can comprise only a tiny fraction of all
possible input vectors, and so generalization is a central goal in pattern recognition.
For most practical applications, the original input variables are typically preprocessed to transform them into some new space of variables where, it is hoped, thepattern recognition problem will be easier to solve. For instance, in the digit recognition problem, the images of the digits are typically translated and scaled so that each
digit is contained within a box of a ﬁxed size. This greatly reduces the variability
within each digit class, because the location and scale of all the digits are now thesame, which makes it much easier for a subsequent pattern recognition algorithm
to distinguish between the different classes. This pre-processing stage is sometimes
also called feature extraction. Note that new test data must be pre-processed using
the same steps as the training data.
Pre-processing might also be performed in order to speed up computation. For
example, if the goal is real-time face detection in a high-resolution video stream,
the computer must handle huge numbers of pixels per second, and presenting these
directly to a complex pattern recognition algorithm may be computationally infeasible. Instead, the aim is to ﬁnd useful features that are fast to compute, and yet that

1. INTRODUCTION

3

also preserve useful discriminatory information enabling faces to be distinguishedfrom non-faces. These features are then used as the inputs to the pattern recognition
algorithm. For instance, the average value of the image intensity over a rectangularsubregion can be evaluated extremely efﬁciently (Viola and Jones, 2004), and a set ofsuch features can prove very effective in fast face detection. Because the number ofsuch features is smaller than the number of pixels, this kind of pre-processing represents a form of dimensionality reduction. Care must be taken during pre-processing
because often information is discarded, and if this information is important to thesolution of the problem then the overall accuracy of the system can suffer.
Applications in which the training data comprises examples of the input vectors
along with their corresponding target vectors are known as supervised learning problems. Cases such as the digit recognition example, in which the aim is to assign each
input vector to one of a ﬁnite number of discrete categories, are called classiﬁcation
problems. If the desired output consists of one or more continuous variables, then
the task is called regression. An example of a regression problem would be the prediction of the yield in a chemical manufacturing process in which the inputs consistof the concentrations of reactants, the temperature, and the pressure.
In other pattern recognition problems, the training data consists of a set of input
vectors x without any corresponding target values. The goal in such unsupervised
learning problems may be to discover groups of similar examples within the data,
where it is called clustering, or to determine the distribution of data within the input
space, known as density estimation, or to project the data from a high-dimensional
space down to two or three dimensions for the purpose of visualization.
Finally, the technique of reinforcement learning (Sutton and Barto, 1998) is concerned with the problem of ﬁnding suitable actions to take in a given situation inorder to maximize a reward. Here the learning algorithm is not given examples of
optimal outputs, in contrast to supervised learning, but must instead discover themby a process of trial and error. Typically there is a sequence of states and actions inwhich the learning algorithm is interacting with its environment. In many cases, thecurrent action not only affects the immediate reward but also has an impact on the reward at all subsequent time steps. For example, by using appropriate reinforcement
learning techniques a neural network can learn to play the game of backgammon to ahigh standard (Tesauro, 1994). Here the network must learn to take a board position
as input, along with the result of a dice throw, and produce a strong move as theoutput. This is done by having the network play against a copy of itself for perhaps amillion games. A major challenge is that a game of backgammon can involve dozens
of moves, and yet it is only at the end of the game that the reward, in the form ofvictory, is achieved. The reward must then be attributed appropriately to all of the
moves that led to it, even though some moves will have been good ones and others
less so. This is an example of a credit assignment problem. A general feature of reinforcement learning is the trade-off between exploration, in which the system tries
out new kinds of actions to see how effective they are, and exploitation, in which
the system makes use of actions that are known to yield a high reward. Too strong
a focus on either exploration or exploitation will yield poor results. Reinforcement
learning continues to be an active area of machine learning research. However, a4

1. INTRODUCTION
Figure 1.2

Plot of a training data set of N =10 points, shown as blue circles,
each comprising an observationof the input variable x along withthe corresponding target variablet. The green curve shows thefunction sin(2πx) used to generate the data. Our goal is to predict the value of t for some newvalue of x, without knowledge ofthe green curve.

1
t
0

−1

0

x

1

detailed treatment lies beyond the scope of this book.
Although each of these tasks needs its own tools and techniques, many of the
key ideas that underpin them are common to all such problems. One of the main
goals of this chapter is to introduce, in a relatively informal way, several of the mostimportant of these concepts and to illustrate them using simple examples. Later inthe book we shall see these same ideas re-emerge in the context of more sophisticated models that are applicable to real-world pattern recognition applications. This
chapter also provides a self-contained introduction to three important tools that will
be used throughout the book, namely probability theory, decision theory, and information theory. Although these might sound like daunting topics, they are in fact
straightforward, and a clear understanding of them is essential if machine learningtechniques are to be used to best effect in practical applications.

1.1. Example: Polynomial Curve Fitting
We begin by introducing a simple regression problem, which we shall use as a running example throughout this chapter to motivate a number of key concepts. Suppose we observe a real-valued input variable x and we wish to use this observation topredict the value of a real-valued target variable t. For the present purposes, it is instructive to consider an artiﬁcial example using synthetically generated data because
we then know the precise process that generated the data for comparison against any
learned model. The data for this example is generated from the function sin(2πx)
with random noise included in the target values, as described in detail in Appendix A.
Now suppose that we are given a training set comprising N observations of x,
written x ≡ (x1 , . . . , xN )T , together with corresponding observations of the valuesof t, denoted t ≡ (t1 , . . . , tN )T . Figure 1.2 shows a plot of a training set comprising
N = 10 data points. The input data set x in Figure 1.2 was generated by choosing values of xn , for n = 1, . . . , N , spaced uniformly in range [0, 1], and the target
data set t was obtained by ﬁrst computing the corresponding values of the function

1.1. Example: Polynomial Curve Fitting

5

sin(2πx) and then adding a small level of random noise having a Gaussian distribution (the Gaussian distribution is discussed in Section 1.2.4) to each such point inorder to obtain the corresponding value tn . By generating data in this way, we arecapturing a property of many real data sets, namely that they possess an underlying
regularity, which we wish to learn, but that individual observations are corrupted byrandom noise. This noise might arise from intrinsically stochastic (i.e. random) processes such as radioactive decay but more typically is due to there being sources ofvariability that are themselves unobserved.
Our goal is to exploit this training set in order to make predictions of the value
t of the target variable for some new value 
x of the input variable. As we shall see
later, this involves implicitly trying to discover the underlying function sin(2πx).
This is intrinsically a difﬁcult problem as we have to generalize from a ﬁnite data
set. Furthermore the observed data are corrupted with noise, and so for a given xthere is uncertainty as to the appropriate value for t. Probability theory, discussed
in Section 1.2, provides a framework for expressing such uncertainty in a precise
and quantitative manner, and decision theory, discussed in Section 1.5, allows us to
exploit this probabilistic representation in order to make predictions that are optimal
according to appropriate criteria.
For the moment, however, we shall proceed rather informally and consider asimple approach based on curve ﬁtting. In particular, we shall ﬁt the data using apolynomial function of the formMy(x, w) = w0 + w1 x + w2 x + . . . + wM x
2

=

M


wj xj

(1.1)

j =0

where M is the order of the polynomial, and xj denotes x raised to the power of j.
The polynomial coefﬁcients w0 , . . . , wM are collectively denoted by the vector w.
Note that, although the polynomial function y(x, w) is a nonlinear function of x, it
is a linear function of the coefﬁcients w. Functions, such as the polynomial, which
are linear in the unknown parameters have important properties and are called linear
models and will be discussed extensively in Chapters 3 and 4.
The values of the coefﬁcients will be determined by ﬁtting the polynomial to thetraining data. This can be done by minimizing an error function that measures the
misﬁt between the function y(x, w), for any given value of w, and the training set
data points. One simple choice of error function, which is widely used, is given by
the sum of the squares of the errors between the predictions y(xn , w) for each data
point xn and the corresponding target values tn , so that we minimize12{y(xn , w) − tn }2
NE(w) =(1.2)

n=1

where the factor of 1/2 is included for later convenience. We shall discuss the motivation for this choice of error function later in this chapter. For the moment we
simply note that it is a nonnegative quantity that would be zero if, and only if, the6

1. INTRODUCTION
Figure 1.3

The error function (1.2) corresponds to (one half of) the sum of t
the squares of the displacements(shown by the vertical green bars)of each data point from the functiony(x, w).

tn

y(xn , w)xnxfunction y(x, w) were to pass exactly through each training data point. The geometrical interpretation of the sum-of-squares error function is illustrated in Figure 1.3.

Exercise 1.1

We can solve the curve ﬁtting problem by choosing the value of w for whichE(w) is as small as possible. Because the error function is a quadratic function of
the coefﬁcients w, its derivatives with respect to the coefﬁcients will be linear in theelements of w, and so the minimization of the error function has a unique solution,
denoted by w , which can be found in closed form. The resulting polynomial isgiven by the function y(x, w ).
There remains the problem of choosing the order M of the polynomial, and as
we shall see this will turn out to be an example of an important concept called modelcomparison or model selection. In Figure 1.4, we show four examples of the resultsof ﬁtting polynomials having orders M = 0, 1, 3, and 9 to the data set shown in
Figure 1.2.
We notice that the constant (M = 0) and ﬁrst order (M = 1) polynomials
give rather poor ﬁts to the data and consequently rather poor representations of thefunction sin(2πx). The third order (M = 3) polynomial seems to give the best ﬁt
to the function sin(2πx) of the examples shown in Figure 1.4. When we go to amuch higher order polynomial (M = 9), we obtain an excellent ﬁt to the training
data. In fact, the polynomial passes exactly through each data point and E(w ) = 0.
However, the ﬁtted curve oscillates wildly and gives a very poor representation of
the function sin(2πx). This latter behaviour is known as over-ﬁtting.
As we have noted earlier, the goal is to achieve good generalization by making
accurate predictions for new data. We can obtain some quantitative insight into thedependence of the generalization performance on M by considering a separate test
set comprising 100 data points generated using exactly the same procedure used
to generate the training set points but with new choices for the random noise values
included in the target values. For each choice of M , we can then evaluate the residual
value of E(w ) given by (1.2) for the training data, and we can also evaluate E(w )for the test data set. It is sometimes more convenient to use the root-mean-square7

1.1. Example: Polynomial Curve Fitting

M =0

1

M =1

1

t

t
0

0

−1

−1

0

x

1

0

M =3

1

x

M =9

1

t

1

t
0

0

−1

−1

0

x

1

0

x

1

Figure 1.4 Plots of polynomials having various orders M , shown as red curves, ﬁtted to the data set shown in
Figure 1.2.

(RMS) error deﬁned by
ERMS =


2E(w )/N

(1.3)

in which the division by N allows us to compare different sizes of data sets onan equal footing, and the square root ensures that ERMS is measured on the same
scale (and in the same units) as the target variable t. Graphs of the training andtest set RMS errors are shown, for various values of M , in Figure 1.5. The test
set error is a measure of how well we are doing in predicting the values of t for
new data observations of x. We note from Figure 1.5 that small values of M give
relatively large values of the test set error, and this can be attributed to the fact thatthe corresponding polynomials are rather inﬂexible and are incapable of capturing
the oscillations in the function sin(2πx). Values of M in the range 3  M  8give small values for the test set error, and these also give reasonable representationsof the generating function sin(2πx), as can be seen, for the case of M = 3, from
Figure 1.4.

1. INTRODUCTION
Figure 1.5

Graphs of the root-mean-squareerror, deﬁned by (1.3), evaluatedon the training set and on an independent test set for various valuesof M .

1
Training
Test
ERMS

8

0.5

0

0

3

M

6

9

For M = 9, the training set error goes to zero, as we might expect because
this polynomial contains 10 degrees of freedom corresponding to the 10 coefﬁcients
w0 , . . . , w9 , and so can be tuned exactly to the 10 data points in the training set.
However, the test set error has become very large and, as we saw in Figure 1.4, thecorresponding function y(x, w ) exhibits wild oscillations.
This may seem paradoxical because a polynomial of given order contains all
lower order polynomials as special cases. The M = 9 polynomial is therefore capable of generating results at least as good as the M = 3 polynomial. Furthermore, wemight suppose that the best predictor of new data would be the function sin(2πx)
from which the data was generated (and we shall see later that this is indeed thecase). We know that a power series expansion of the function sin(2πx) contains
terms of all orders, so we might expect that results should improve monotonically as
we increase M .
We can gain some insight into the problem by examining the values of the coefﬁcients w obtained from polynomials of various order, as shown in Table 1.1.
We see that, as M increases, the magnitude of the coefﬁcients typically gets larger.
In particular for the M = 9 polynomial, the coefﬁcients have become ﬁnely tuned
to the data by developing large positive and negative values so that the correspondTable 1.1

Table of the coefﬁcients w forpolynomials of various order.
Observe how the typical magnitude of the coefﬁcients increases dramatically as the order of the polynomial increases.

w0
w1
w2
w3
w4
w5
w6
w7
w8
w9

M =0
0.19

M =1
0.82
-1.27

M =6
0.31
7.99
-25.43
17.37

M =9
0.35
232.37-5321.83
48568.31-231639.30
640042.26-1061800.52
1042400.18-557682.99
125201.43

9

1.1. Example: Polynomial Curve Fitting

N = 151

N = 100

1

t

t
0

0

−1

−1

0

x

1

0

x

1

Figure 1.6 Plots of the solutions obtained by minimizing the sum-of-squares error function using the M = 9
polynomial for N = 15 data points (left plot) and N = 100 data points (right plot). We see that increasing thesize of the data set reduces the over-ﬁtting problem.

Section 3.4

ing polynomial function matches each of the data points exactly, but between data
points (particularly near the ends of the range) the function exhibits the large oscillations observed in Figure 1.4. Intuitively, what is happening is that the more ﬂexible
polynomials with larger values of M are becoming increasingly tuned to the random
noise on the target values.
It is also interesting to examine the behaviour of a given model as the size of thedata set is varied, as shown in Figure 1.6. We see that, for a given model complexity,
the over-ﬁtting problem become less severe as the size of the data set increases.
Another way to say this is that the larger the data set, the more complex (in otherwords more ﬂexible) the model that we can afford to ﬁt to the data. One roughheuristic that is sometimes advocated is that the number of data points should be
no less than some multiple (say 5 or 10) of the number of adaptive parameters in
the model. However, as we shall see in Chapter 3, the number of parameters is notnecessarily the most appropriate measure of model complexity.
Also, there is something rather unsatisfying about having to limit the number ofparameters in a model according to the size of the available training set. It would
seem more reasonable to choose the complexity of the model according to the complexity of the problem being solved. We shall see that the least squares approach
to ﬁnding the model parameters represents a speciﬁc case of maximum likelihood(discussed in Section 1.2.5), and that the over-ﬁtting problem can be understood asa general property of maximum likelihood. By adopting a Bayesian approach, theover-ﬁtting problem can be avoided. We shall see that there is no difﬁculty froma Bayesian perspective in employing models for which the number of parameters
greatly exceeds the number of data points. Indeed, in a Bayesian model the effective
number of parameters adapts automatically to the size of the data set.
For the moment, however, it is instructive to continue with the current approachand to consider how in practice we can apply it to data sets of limited size where we10

1. INTRODUCTION

ln λ = −181ln λ = 0

1

t

t
0

0

−1

−1

0

x

1

0

x

1

Figure 1.7 Plots of M = 9 polynomials ﬁtted to the data set shown in Figure 1.2 using the regularized error
function (1.4) for two values of the regularization parameter λ corresponding to ln λ = −18 and ln λ = 0. The
case of no regularizer, i.e., λ = 0, corresponding to ln λ = −∞, is shown at the bottom right of Figure 1.4.

may wish to use relatively complex and ﬂexible models. One technique that is often
used to control the over-ﬁtting phenomenon in such cases is that of regularization,
which involves adding a penalty term to the error function (1.2) in order to discourage
the coefﬁcients from reaching large values. The simplest such penalty term takes theform of a sum of squares of all of the coefﬁcients, leading to a modiﬁed error functionof the form
N
1
λ
2

E(w)
=
{y(xn , w) − tn } + w2
(1.4)
2
2
n=1

where w ≡ w
ative importance of the regularization term compared with the sum-of-squares error
term. Note that often the coefﬁcient w0 is omitted from the regularizer because its
inclusion causes the results to depend on the choice of origin for the target variable(Hastie et al., 2001), or it may be included but with its own regularization coefﬁcient(we shall discuss this topic in more detail in Section 5.5.1). Again, the error functionin (1.4) can be minimized exactly in closed form. Techniques such as this are known
in the statistics literature as shrinkage methods because they reduce the value of thecoefﬁcients. The particular case of a quadratic regularizer is called ridge regression (Hoerl and Kennard, 1970). In the context of neural networks, this approach is
known as weight decay.
Figure 1.7 shows the results of ﬁtting the polynomial of order M = 9 to thesame data set as before but now using the regularized error function given by (1.4).
We see that, for a value of ln λ = −18, the over-ﬁtting has been suppressed and we
now obtain a much closer representation of the underlying function sin(2πx). If,
however, we use too large a value for λ then we again obtain a poor ﬁt, as shown in
Figure 1.7 for ln λ = 0. The corresponding coefﬁcients from the ﬁtted polynomials
are given in Table 1.2, showing that regularization has the desired effect of reducing
2

Exercise 1.2

T2
w = w02 + w12 + . . . + wM
, and the coefﬁcient λ governs the rel-11

1.1. Example: Polynomial Curve Fitting

Section 1.3

Figure 1.8

Table of the coefﬁcients w for M =9 polynomials with various values forthe regularization parameter λ. Notethat ln λ = −∞ corresponds to amodel with no regularization, i.e., to
the graph at the bottom right in Figure 1.4. We see that, as the value of
λ increases, the typical magnitude of
the coefﬁcients gets smaller.

ln λ = −∞0.35
232.37-5321.83
48568.31-231639.30
640042.26-1061800.52
1042400.18-557682.99
125201.43

w0
w1
w2
w3
w4
w5
w6
w7
w8
w9

ln λ = −18
0.35
4.74
-0.77
-31.97
-3.89
55.28
41.32
-45.95-91.5372.68

ln λ = 0
0.13
-0.05
-0.06
-0.05
-0.03
-0.02
-0.01
-0.00
0.00
0.01

the magnitude of the coefﬁcients.
The impact of the regularization term on the generalization error can be seen byplotting the value of the RMS error (1.3) for both training and test sets against ln λ,as shown in Figure 1.8. We see that in effect λ now controls the effective complexityof the model and hence determines the degree of over-ﬁtting.
The issue of model complexity is an important one and will be discussed at
length in Section 1.3. Here we simply note that, if we were trying to solve a practicalapplication using this approach of minimizing an error function, we would have toﬁnd a way to determine a suitable value for the model complexity. The results above
suggest a simple way of achieving this, namely by taking the available data andpartitioning it into a training set, used to determine the coefﬁcients w, and a separate
validation set, also called a hold-out set, used to optimize the model complexity(either M or λ). In many cases, however, this will prove to be too wasteful ofvaluable training data, and we have to seek more sophisticated approaches.
So far our discussion of polynomial curve ﬁtting has appealed largely to intuition. We now seek a more principled approach to solving problems in patternrecognition by turning to a discussion of probability theory. As well as providing thefoundation for nearly all of the subsequent developments in this book, it will also
Graph of the root-mean-square error (1.3) versus ln λ for the M = 9
polynomial.

1
Training
Test
ERMS

Table 1.2

0.5

0

−35

−30

ln λ −25−2012

1. INTRODUCTION
give us some important insights into the concepts we have introduced in the context of polynomial curve ﬁtting and will allow us to extend these to more complex
situations.

1.2. Probability Theory
A key concept in the ﬁeld of pattern recognition is that of uncertainty. It arises boththrough noise on measurements, as well as through the ﬁnite size of data sets. Probability theory provides a consistent framework for the quantiﬁcation and manipulation of uncertainty and forms one of the central foundations for pattern recognition.
When combined with decision theory, discussed in Section 1.5, it allows us to make
optimal predictions given all the information available to us, even though that information may be incomplete or ambiguous.
We will introduce the basic concepts of probability theory by considering a simple example. Imagine we have two boxes, one red and one blue, and in the red box
we have 2 apples and 6 oranges, and in the blue box we have 3 apples and 1 orange.
This is illustrated in Figure 1.9. Now suppose we randomly pick one of the boxes
and from that box we randomly select an item of fruit, and having observed whichsort of fruit it is we replace it in the box from which it came. We could imagine
repeating this process many times. Let us suppose that in so doing we pick the redbox 40% of the time and we pick the blue box 60% of the time, and that when weremove an item of fruit from a box we are equally likely to select any of the piecesof fruit in the box.
In this example, the identity of the box that will be chosen is a random variable,
which we shall denote by B. This random variable can take one of two possible
values, namely r (corresponding to the red box) or b (corresponding to the bluebox). Similarly, the identity of the fruit is also a random variable and will be denoted
by F . It can take either of the values a (for apple) or o (for orange).
To begin with, we shall deﬁne the probability of an event to be the fractionof times that event occurs out of the total number of trials, in the limit that the total
number of trials goes to inﬁnity. Thus the probability of selecting the red box is 4/10
Figure 1.9

We use a simple example of twocoloured boxes each containing fruit(apples shown in green and oranges shown in orange) to introduce the basic ideas of probability.

1.2. Probability Theory

13

ci

}

Figure 1.10 We can derive the sum and product rules of probability byconsidering two random variables, X, which takes the values {xi } wherei = 1, . . . , M , and Y , which takes the values {yj } where j = 1, . . . , L.
In this illustration we have M = 5 and L = 3. If we consider a total
number N of instances of these variables, then we denote the numberof instances where X = xi and Y = yj by nij , which is the number of yj
points in the corresponding cell of the array. The number of points incolumn i, corresponding to X = xi , is denoted by ci , and the number ofpoints in row j, corresponding to Y = yj , is denoted by rj .

nij

}r

j

xi

and the probability of selecting the blue box is 6/10. We write these probabilities
as p(B = r) = 4/10 and p(B = b) = 6/10. Note that, by deﬁnition, probabilities
must lie in the interval [0, 1]. Also, if the events are mutually exclusive and if they
include all possible outcomes (for instance, in this example the box must be either
red or blue), then we see that the probabilities for those events must sum to one.
We can now ask questions such as: “what is the overall probability that the selection procedure will pick an apple?”, or “given that we have chosen an orange,
what is the probability that the box we chose was the blue one?”. We can answer
questions such as these, and indeed much more complex questions associated withproblems in pattern recognition, once we have equipped ourselves with the two elementary rules of probability, known as the sum rule and the product rule. Having
obtained these rules, we shall then return to our boxes of fruit example.
In order to derive the rules of probability, consider the slightly more general example shown in Figure 1.10 involving two random variables X and Y (which could
for instance be the Box and Fruit variables considered above). We shall suppose that
X can take any of the values xi where i = 1, . . . , M , and Y can take the values yj
where j = 1, . . . , L. Consider a total of N trials in which we sample both of thevariables X and Y , and let the number of such trials in which X = xi and Y = yj
be nij . Also, let the number of trials in which X takes the value xi (irrespectiveof the value that Y takes) be denoted by ci , and similarly let the number of trials inwhich Y takes the value yj be denoted by rj .
The probability that X will take the value xi and Y will take the value yj is
written p(X = xi , Y = yj ) and is called the joint probability of X = xi and
Y = yj . It is given by the number of points falling in the cell i,j as a fraction of thetotal number of points, and hencep(X = xi , Y = yj ) =

nij
.
N

(1.5)

Here we are implicitly considering the limit N → ∞. Similarly, the probability that
X takes the value xi irrespective of the value of Y is written as p(X = xi ) and isgiven by the fraction of the total number of points that fall in column i, so thatp(X = xi ) =

ci
.
N

(1.6)

Because the number of instances in column i in Figure 1.10 
is just the sum of thenumber of instances in each cell of that column, we have ci = j nij and therefore,

14

1. INTRODUCTION
from (1.5) and (1.6), we havep(X = xi ) =Lp(X = xi , Y = yj )

(1.7)

j =1

which is the sum rule of probability. Note that p(X = xi ) is sometimes called themarginal probability, because it is obtained by marginalizing, or summing out, theother variables (in this case Y ).
If we consider only those instances for which X = xi , then the fraction ofsuch instances for which Y = yj is written p(Y = yj |X = xi ) and is called theconditional probability of Y = yj given X = xi . It is obtained by ﬁnding thefraction of those points in column i that fall in cell i,j and hence is given byp(Y = yj |X = xi ) =

nij
.
ci

(1.8)

From (1.5), (1.6), and (1.8), we can then derive the following relationshipnij
nij ci·=Nci N= p(Y = yj |X = xi )p(X = xi )p(X = xi , Y = yj ) =(1.9)

which is the product rule of probability.
So far we have been quite careful to make a distinction between a random variable, such as the box B in the fruit example, and the values that the random variable
can take, for example r if the box were the red one. Thus the probability that B takes
the value r is denoted p(B = r). Although this helps to avoid ambiguity, it leads
to a rather cumbersome notation, and in many cases there will be no need for suchpedantry. Instead, we may simply write p(B) to denote a distribution over the random variable B, or p(r) to denote the distribution evaluated for the particular valuer, provided that the interpretation is clear from the context.
With this more compact notation, we can write the two fundamental rules ofprobability theory in the following form.
The Rules of Probability
sum rulep(X) =

p(X, Y )(1.10)

Y

product rulep(X, Y ) = p(Y |X)p(X).

(1.11)

Here p(X, Y ) is a joint probability and is verbalized as “the probability of X and
Y ”. Similarly, the quantity p(Y |X) is a conditional probability and is verbalized as“the probability of Y given X”, whereas the quantity p(X) is a marginal probability

1.2. Probability Theory

15

and is simply “the probability of X”. These two simple rules form the basis for allof the probabilistic machinery that we use throughout this book.
From the product rule, together with the symmetry property p(X, Y ) = p(Y, X),
we immediately obtain the following relationship between conditional probabilitiesp(Y |X) =p(X|Y )p(Y )p(X)(1.12)

which is called Bayes’ theorem and which plays a central role in pattern recognition
and machine learning. Using the sum rule, the denominator in Bayes’ theorem canbe expressed in terms of the quantities appearing in the numeratorp(X) =
p(X|Y )p(Y ).
(1.13)
Y

We can view the denominator in Bayes’ theorem as being the normalization constantrequired to ensure that the sum of the conditional probability on the left-hand side of(1.12) over all values of Y equals one.
In Figure 1.11, we show a simple example involving a joint distribution over two
variables to illustrate the concept of marginal and conditional distributions. Herea ﬁnite sample of N = 60 data points has been drawn from the joint distribution
and is shown in the top left. In the top right is a histogram of the fractions of data
points having each of the two values of Y . From the deﬁnition of probability, thesefractions would equal the corresponding probabilities p(Y ) in the limit N → ∞. We
can view the histogram as a simple way to model a probability distribution given only
a ﬁnite number of points drawn from that distribution. Modelling distributions fromdata lies at the heart of statistical pattern recognition and will be explored in great
detail in this book. The remaining two plots in Figure 1.11 show the corresponding
histogram estimates of p(X) and p(X|Y = 1).
Let us now return to our example involving boxes of fruit. For the moment, weshall once again be explicit about distinguishing between the random variables andtheir instantiations. We have seen that the probabilities of selecting either the red or
the blue boxes are given byp(B = r) = 4/10
p(B = b) = 6/10

(1.14)(1.15)

respectively. Note that these satisfy p(B = r) + p(B = b) = 1.
Now suppose that we pick a box at random, and it turns out to be the blue box.
Then the probability of selecting an apple is just the fraction of apples in the bluebox which is 3/4, and so p(F = a|B = b) = 3/4. In fact, we can write out all four
conditional probabilities for the type of fruit, given the selected box
p(F = a|B = r)p(F = o|B = r)p(F = a|B = b)p(F = o|B = b)

=
=
=
=

1/4
3/4
3/4
1/4.

(1.16)(1.17)(1.18)(1.19)

16

1. INTRODUCTIONp(Y )p(X, Y )

Y =2

Y =1

X
p(X)

p(X|Y = 1)

X

X

Figure 1.11 An illustration of a distribution over two variables, X, which takes 9 possible values, and Y , whichtakes two possible values. The top left ﬁgure shows a sample of 60 points drawn from a joint probability distribution over these variables. The remaining ﬁgures show histogram estimates of the marginal distributions p(X)and p(Y ), as well as the conditional distribution p(X|Y = 1) corresponding to the bottom row in the top left
ﬁgure.

Again, note that these probabilities are normalized so thatp(F = a|B = r) + p(F = o|B = r) = 1(1.20)

p(F = a|B = b) + p(F = o|B = b) = 1.

(1.21)and similarly
We can now use the sum and product rules of probability to evaluate the overall
probability of choosing an applep(F = a) = p(F = a|B = r)p(B = r) + p(F = a|B = b)p(B = b)
4
3
6
11
1
×
+ ×
=
(1.22)=
4 10 4 10
20
from which it follows, using the sum rule, that p(F = o) = 1 − 11/20 = 9/20.

1.2. Probability Theory

17

Suppose instead we are told that a piece of fruit has been selected and it is anorange, and we would like to know which box it came from. This requires that
we evaluate the probability distribution over boxes conditioned on the identity of
the fruit, whereas the probabilities in (1.16)–(1.19) give the probability distribution
over the fruit conditioned on the identity of the box. We can solve the problem ofreversing the conditional probability by using Bayes’ theorem to givep(B = r|F = o) =p(F = o|B = r)p(B = r)
3
4
20
2
= ×
×
= .
p(F = o)
4 10
93

(1.23)

From the sum rule, it then follows that p(B = b|F = o) = 1 − 2/3 = 1/3.
We can provide an important interpretation of Bayes’ theorem as follows. If
we had been asked which box had been chosen before being told the identity ofthe selected item of fruit, then the most complete information we have available isprovided by the probability p(B). We call this the prior probability because it is theprobability available before we observe the identity of the fruit. Once we are told thatthe fruit is an orange, we can then use Bayes’ theorem to compute the probabilityp(B|F ), which we shall call the posterior probability because it is the probability
obtained after we have observed F . Note that in this example, the prior probabilityof selecting the red box was 4/10, so that we were more likely to select the blue box
than the red one. However, once we have observed that the piece of selected fruit isan orange, we ﬁnd that the posterior probability of the red box is now 2/3, so that
it is now more likely that the box we selected was in fact the red one. This result
accords with our intuition, as the proportion of oranges is much higher in the red box
than it is in the blue box, and so the observation that the fruit was an orange provides
signiﬁcant evidence favouring the red box. In fact, the evidence is sufﬁciently strongthat it outweighs the prior and makes it more likely that the red box was chosenrather than the blue one.
Finally, we note that if the joint distribution of two variables factorizes into the
product of the marginals, so that p(X, Y ) = p(X)p(Y ), then X and Y are said tobe independent. From the product rule, we see that p(Y |X) = p(Y ), and so theconditional distribution of Y given X is indeed independent of the value of X. For
instance, in our boxes of fruit example, if each box contained the same fraction ofapples and oranges, then p(F |B) = P (F ), so that the probability of selecting, say,
an apple is independent of which box is chosen.

1.2.1 Probability densities
As well as considering probabilities deﬁned over discrete sets of events, we
also wish to consider probabilities with respect to continuous variables. We shalllimit ourselves to a relatively informal discussion. If the probability of a real-valued
variable x falling in the interval (x, x + δx) is given by p(x)δx for δx → 0, thenp(x) is called the probability density over x. This is illustrated in Figure 1.12. The
probability that x will lie in an interval (a, b) is then given by bp(x) dx.
(1.24)p(x ∈ (a, b)) =a

18

1. INTRODUCTION
Figure 1.12

The concept of probability fordiscrete variables can be extended to that of a probability
density p(x) over a continuous
variable x and is such that theprobability of x lying in the interval (x, x + δx) is given by p(x)δx
for δx → 0. The probability
density can be expressed as thederivative of a cumulative distribution function P (x).

p(x)

δx

P (x)

x

Because probabilities are nonnegative, and because the value of x must lie somewhere on the real axis, the probability density p(x) must satisfy the two conditions

 ∞p(x)  0(1.25)

p(x) dx = 1.

(1.26)−∞

Under a nonlinear change of variable, a probability density transforms differently
from a simple function, due to the Jacobian factor. For instance, if we considera change of variables x = g(y), then a function f (x) becomes f(y) = f (g(y)).
Now consider a probability density px (x) that corresponds to a density py (y) withrespect to the new variable y, where the sufﬁces denote the fact that px (x) and py (y)are different densities. Observations falling in the range (x, x + δx) will, for smallvalues of δx, be transformed into the range (y, y + δy) where px (x)δx  py (y)δy,
and hence 
 dx py (y) = px (x) dy = px (g(y)) |g  (y)| .
(1.27)
Exercise 1.4

One consequence of this property is that the concept of the maximum of a probability
density is dependent on the choice of variable.
The probability that x lies in the interval (−∞, z) is given by the cumulative
distribution function deﬁned by zp(x) dx(1.28)
P (z) =−∞

which satisﬁes P  (x) = p(x), as shown in Figure 1.12.
If we have several continuous variables x1 , . . . , xD , denoted collectively by thevector x, then we can deﬁne a joint probability density p(x) = p(x1 , . . . , xD ) such

1.2. Probability Theory

19

that the probability of x falling in an inﬁnitesimal volume δx containing the point x
is given by p(x)δx. This multivariate probability density must satisfy

p(x)  0(1.29)

p(x) dx = 1(1.30)

in which the integral is taken over the whole of x space. We can also consider joint
probability distributions over a combination of discrete and continuous variables.
Note that if x is a discrete variable, then p(x) is sometimes called a probability
mass function because it can be regarded as a set of ‘probability masses’ concentrated
at the allowed values of x.
The sum and product rules of probability, as well as Bayes’ theorem, apply
equally to the case of probability densities, or to combinations of discrete and continuous variables. For instance, if x and y are two real variables, then the sum andproduct rules take the formp(x, y) dy(1.31)
p(x) =
p(x, y) = p(y|x)p(x).

(1.32)A formal justiﬁcation of the sum and product rules for continuous variables (Feller,1966) requires a branch of mathematics called measure theory and lies outside thescope of this book. Its validity can be seen informally, however, by dividing eachreal variable into intervals of width ∆ and considering the discrete probability distribution over these intervals. Taking the limit ∆ → 0 then turns sums into integrals
and gives the desired result.

1.2.2 Expectations and covariances
One of the most important operations involving probabilities is that of ﬁnding
weighted averages of functions. The average value of some function f (x) under aprobability distribution p(x) is called the expectation of f (x) and will be denoted by
E[f ]. For a discrete distribution, it is given by
p(x)f (x)(1.33)
E[f ] =xso that the average is weighted by the relative probabilities of the different valuesof x. In the case of continuous variables, expectations are expressed in terms of an
integration with respect to the corresponding probability densityE[f ] = p(x)f (x) dx.
(1.34)
In either case, if we are given a ﬁnite number N of points drawn from the probability
distribution or probability density, then the expectation can be approximated as a20

1. INTRODUCTION
ﬁnite sum over these points
1 
f (xn ).
E[f ] NN

(1.35)

n=1

We shall make extensive use of this result when we discuss sampling methods in
Chapter 11. The approximation in (1.35) becomes exact in the limit N → ∞.
Sometimes we will be considering expectations of functions of several variables,in which case we can use a subscript to indicate which variable is being averaged
over, so that for instanceEx [f (x, y)](1.36)
denotes the average of the function f (x, y) with respect to the distribution of x. Notethat Ex [f (x, y)] will be a function of y.
We can also consider a conditional expectation with respect to a conditional
distribution, so that
Ex [f |y] =p(x|y)f (x)(1.37)
xwith an analogous deﬁnition for continuous variables.
The variance of f (x) is deﬁned by2
var[f ] = E (f (x) − E[f (x)])

Exercise 1.5(1.38)and provides a measure of how much variability there is in f (x) around its meanvalue E[f (x)]. Expanding out the square, we see that the variance can also be written
in terms of the expectations of f (x) and f (x)2
var[f ] = E[f (x)2 ] − E[f (x)]2 .

(1.39)

In particular, we can consider the variance of the variable x itself, which is given by
var[x] = E[x2 ] − E[x]2 .

(1.40)For two random variables x and y, the covariance is deﬁned bycov[x, y] = Ex,y [{x − E[x]} {y − E[y]}]= Ex,y [xy] − E[x]E[y]Exercise 1.6(1.41)

which expresses the extent to which x and y vary together. If x and y are independent, then their covariance vanishes.
In the case of two vectors of random variables x and y, the covariance is a matrix
cov[x, y] = Ex,y {x − E[x]}{yT − E[yT ]}= Ex,y [xyT ] − E[x]E[yT ].

(1.42)

If we consider the covariance of the components of a vector x with each other, thenwe use a slightly simpler notation cov[x] ≡ cov[x, x].

1.2. Probability Theory21

1.2.3 Bayesian probabilities
So far in this chapter, we have viewed probabilities in terms of the frequenciesof random, repeatable events. We shall refer to this as the classical or frequentist
interpretation of probability. Now we turn to the more general Bayesian view, inwhich probabilities provide a quantiﬁcation of uncertainty.
Consider an uncertain event, for example whether the moon was once in its own
orbit around the sun, or whether the Arctic ice cap will have disappeared by the endof the century. These are not events that can be repeated numerous times in orderto deﬁne a notion of probability as we did earlier in the context of boxes of fruit.
Nevertheless, we will generally have some idea, for example, of how quickly wethink the polar ice is melting. If we now obtain fresh evidence, for instance from a
new Earth observation satellite gathering novel forms of diagnostic information, we
may revise our opinion on the rate of ice loss. Our assessment of such matters willaffect the actions we take, for instance the extent to which we endeavour to reduce
the emission of greenhouse gasses. In such circumstances, we would like to be able
to quantify our expression of uncertainty and make precise revisions of uncertainty inthe light of new evidence, as well as subsequently to be able to take optimal actions
or decisions as a consequence. This can all be achieved through the elegant, and very
general, Bayesian interpretation of probability.
The use of probability to represent uncertainty, however, is not an ad-hoc choice,but is inevitable if we are to respect common sense while making rational coherent
inferences. For instance, Cox (1946) showed that if numerical values are used torepresent degrees of belief, then a simple set of axioms encoding common sense
properties of such beliefs leads uniquely to a set of rules for manipulating degrees ofbelief that are equivalent to the sum and product rules of probability. This provided
the ﬁrst rigorous proof that probability theory could be regarded as an extension ofBoolean logic to situations involving uncertainty (Jaynes, 2003). Numerous otherauthors have proposed different sets of properties or axioms that such measures of
uncertainty should satisfy (Ramsey, 1931; Good, 1950; Savage, 1961; deFinetti,
1970; Lindley, 1982). In each case, the resulting numerical quantities behave precisely according to the rules of probability. It is therefore natural to refer to thesequantities as (Bayesian) probabilities.
In the ﬁeld of pattern recognition, too, it is helpful to have a more general no-

Thomas Bayes
1701–1761
Thomas Bayes was born in Tunbridge Wells and was a clergyman
as well as an amateur scientist anda mathematician. He studied logic
and theology at Edinburgh University and was elected Fellow of theRoyal Society in 1742. During the 18th century, issues regarding probability arose in connection withgambling and with the new concept of insurance. One
particularly important problem concerned so-called inverse probability. A solution was proposed by Thomas
Bayes in his paper ‘Essay towards solving a problem
in the doctrine of chances’, which was published in
1764, some three years after his death, in the Philosophical Transactions of the Royal Society. In fact,
Bayes only formulated his theory for the case of a uniform prior, and it was Pierre-Simon Laplace who independently rediscovered the theory in general form andwho demonstrated its broad applicability.

22

1. INTRODUCTION
tion of probability. Consider the example of polynomial curve ﬁtting discussed in
Section 1.1. It seems reasonable to apply the frequentist notion of probability to therandom values of the observed variables tn . However, we would like to address andquantify the uncertainty that surrounds the appropriate choice for the model parameters w. We shall see that, from a Bayesian perspective, we can use the machineryof probability theory to describe the uncertainty in model parameters such as w, or
indeed in the choice of model itself.
Bayes’ theorem now acquires a new signiﬁcance. Recall that in the boxes of fruit
example, the observation of the identity of the fruit provided relevant informationthat altered the probability that the chosen box was the red one. In that example,
Bayes’ theorem was used to convert a prior probability into a posterior probability
by incorporating the evidence provided by the observed data. As we shall see indetail later, we can adopt a similar approach when making inferences about quantities
such as the parameters w in the polynomial curve ﬁtting example. We capture ourassumptions about w, before observing the data, in the form of a prior probability
distribution p(w). The effect of the observed data D = {t1 , . . . , tN } is expressed
through the conditional probability p(D|w), and we shall see later, in Section 1.2.5,
how this can be represented explicitly. Bayes’ theorem, which takes the formp(w|D) =p(D|w)p(w)p(D)(1.43)

then allows us to evaluate the uncertainty in w after we have observed D in the formof the posterior probability p(w|D).
The quantity p(D|w) on the right-hand side of Bayes’ theorem is evaluated forthe observed data set D and can be viewed as a function of the parameter vector
w, in which case it is called the likelihood function. It expresses how probable theobserved data set is for different settings of the parameter vector w. Note that thelikelihood is not a probability distribution over w, and its integral with respect to w
does not (necessarily) equal one.
Given this deﬁnition of likelihood, we can state Bayes’ theorem in words
posterior ∝ likelihood × prior(1.44)

where all of these quantities are viewed as functions of w. The denominator in(1.43) is the normalization constant, which ensures that the posterior distribution
on the left-hand side is a valid probability density and integrates to one. Indeed,
integrating both sides of (1.43) with respect to w, we can express the denominatorin Bayes’ theorem in terms of the prior distribution and the likelihood function(1.45)
p(D) = p(D|w)p(w) dw.
In both the Bayesian and frequentist paradigms, the likelihood function p(D|w)
plays a central role. However, the manner in which it is used is fundamentally different in the two approaches. In a frequentist setting, w is considered to be a ﬁxed
parameter, whose value is determined by some form of ‘estimator’, and error bars

1.2. Probability Theory

Section 2.1

Section 2.4.3

Section 1.3

23

on this estimate are obtained by considering the distribution of possible data sets D.
By contrast, from the Bayesian viewpoint there is only a single data set D (namely
the one that is actually observed), and the uncertainty in the parameters is expressed
through a probability distribution over w.
A widely used frequentist estimator is maximum likelihood, in which w is set
to the value that maximizes the likelihood function p(D|w). This corresponds tochoosing the value of w for which the probability of the observed data set is maximized. In the machine learning literature, the negative log of the likelihood function
is called an error function. Because the negative logarithm is a monotonically decreasing function, maximizing the likelihood is equivalent to minimizing the error.
One approach to determining frequentist error bars is the bootstrap (Efron, 1979;
Hastie et al., 2001), in which multiple data sets are created as follows. Suppose ouroriginal data set consists of N data points X = {x1 , . . . , xN }. We can create a newdata set XB by drawing N points at random from X, with replacement, so that somepoints in X may be replicated in XB , whereas other points in X may be absent from
XB . This process can be repeated L times to generate L data sets each of size N andeach obtained by sampling from the original data set X. The statistical accuracy ofparameter estimates can then be evaluated by looking at the variability of predictions
between the different bootstrap data sets.
One advantage of the Bayesian viewpoint is that the inclusion of prior knowledge arises naturally. Suppose, for instance, that a fair-looking coin is tossed threetimes and lands heads each time. A classical maximum likelihood estimate of theprobability of landing heads would give 1, implying that all future tosses will landheads! By contrast, a Bayesian approach with any reasonable prior will lead to amuch less extreme conclusion.
There has been much controversy and debate associated with the relative merits of the frequentist and Bayesian paradigms, which have not been helped by thefact that there is no unique frequentist, or even Bayesian, viewpoint. For instance,
one common criticism of the Bayesian approach is that the prior distribution is often selected on the basis of mathematical convenience rather than as a reﬂection of
any prior beliefs. Even the subjective nature of the conclusions through their dependence on the choice of prior is seen by some as a source of difﬁculty. Reducing
the dependence on the prior is one motivation for so-called noninformative priors.
However, these lead to difﬁculties when comparing different models, and indeed
Bayesian methods based on poor choices of prior can give poor results with highconﬁdence. Frequentist evaluation methods offer some protection from such problems, and techniques such as cross-validation remain useful in areas such as modelcomparison.
This book places a strong emphasis on the Bayesian viewpoint, reﬂecting thehuge growth in the practical importance of Bayesian methods in the past few years,
while also discussing useful frequentist concepts as required.
Although the Bayesian framework has its origins in the 18th century, the practical application of Bayesian methods was for a long time severely limited by the
difﬁculties in carrying through the full Bayesian procedure, particularly the need tomarginalize (sum or integrate) over the whole of parameter space, which, as we shall24

1. INTRODUCTION
see, is required in order to make predictions or to compare different models. The
development of sampling methods, such as Markov chain Monte Carlo (discussed in
Chapter 11) along with dramatic improvements in the speed and memory capacityof computers, opened the door to the practical use of Bayesian techniques in an impressive range of problem domains. Monte Carlo methods are very ﬂexible and canbe applied to a wide range of models. However, they are computationally intensive
and have mainly been used for small-scale problems.
More recently, highly efﬁcient deterministic approximation schemes such as
variational Bayes and expectation propagation (discussed in Chapter 10) have beendeveloped. These offer a complementary alternative to sampling methods and haveallowed Bayesian techniques to be used in large-scale applications (Blei et al., 2003).

1.2.4 The Gaussian distribution
We shall devote the whole of Chapter 2 to a study of various probability distributions and their key properties. It is convenient, however, to introduce here oneof the most important probability distributions for continuous variables, called thenormal or Gaussian distribution. We shall make extensive use of this distribution in
the remainder of this chapter and indeed throughout much of the book.
For the case of a single real-valued variable x, the Gaussian distribution is deﬁned by1
1
exp − 2 (x − µ)2(1.46)
N x|µ, σ 2 =
2
1
/
2
2σ
(2πσ )
which is governed by two parameters: µ, called the mean, and σ 2 , called the variance. The square root of the variance, given by σ, is called the standard deviation,and the reciprocal of the variance, written as β = 1/σ 2 , is called the precision. We
shall see the motivation for these terms shortly. Figure 1.13 shows a plot of theGaussian distribution.
From the form of (1.46) we see that the Gaussian distribution satisﬁes
N (x|µ, σ 2 ) > 0.
Exercise 1.7(1.47)

Also it is straightforward to show that the Gaussian is normalized, so that

Pierre-Simon Laplace
1749–1827
It is said that Laplace was seriously lacking in modesty and at one
point declared himself to be thebest mathematician in France at thetime, a claim that was arguably true.
As well as being proliﬁc in mathematics, he also made numerous contributions to astronomy, including the nebular hypothesis by which theearth is thought to have formed from the condensation and cooling of a large rotating disk of gas anddust. In 1812 he published the ﬁrst edition of Théorie
Analytique des Probabilités, in which Laplace statesthat “probability theory is nothing but common sense
reduced to calculation”. This work included a discussion of the inverse probability calculation (later termed
Bayes’ theorem by Poincaré), which he used to solve
problems in life expectancy, jurisprudence, planetary
masses, triangulation, and error estimation.

1.2. Probability Theory
Figure 1.13

Plot of the univariate Gaussian
showing the mean µ and thestandard deviation σ.

25

N (x|µ, σ 2 )

2σ

µ

 ∞
−∞

Exercise 1.8

N x|µ, σ 2 dx = 1.

x(1.48)

Thus (1.46) satisﬁes the two requirements for a valid probability density.
We can readily ﬁnd expectations of functions of x under the Gaussian distribution. In particular, the average value of x is given by ∞N x|µ, σ 2 x dx = µ.
(1.49)
E[x] =−∞

Because the parameter µ represents the average value of x under the distribution, it
is referred to as the mean. Similarly, for the second order moment ∞2
N x|µ, σ 2 x2 dx = µ2 + σ 2 .
(1.50)
E[x ] =−∞

From (1.49) and (1.50), it follows that the variance of x is given by
var[x] = E[x2 ] − E[x]2 = σ 2
Exercise 1.9(1.51)and hence σ 2 is referred to as the variance parameter. The maximum of a distribution
is known as its mode. For a Gaussian, the mode coincides with the mean.
We are also interested in the Gaussian distribution deﬁned over a D-dimensional
vector x of continuous variables, which is given by
N (x|µ, Σ) =1
1
1
exp − (x − µ)T Σ−1 (x − µ)
D/
2
1
/
2
2
(2π)
|Σ|

(1.52)

where the D-dimensional vector µ is called the mean, the D × D matrix Σ is called
the covariance, and |Σ| denotes the determinant of Σ. We shall make use of themultivariate Gaussian distribution brieﬂy in this chapter, although its properties will
be studied in detail in Section 2.3.

26

1. INTRODUCTION
Figure 1.14

Illustration of the likelihood function fora Gaussian distribution, shown by the
red curve. Here the black points de- p(x)note a data set of values {xn }, andthe likelihood function given by (1.53)
corresponds to the product of the blue
values. Maximizing the likelihood involves adjusting the mean and variance of the Gaussian so as to maximize this product.

N (xn |µ, σ 2 )xnx

Now suppose that we have a data set of observations x = (x1 , . . . , xN )T , representing N observations of the scalar variable x. Note that we are using the typeface x to distinguish this from a single observation of the vector-valued variable(x1 , . . . , xD )T , which we denote by x. We shall suppose that the observations are
drawn independently from a Gaussian distribution whose mean µ and variance σ 2
are unknown, and we would like to determine these parameters from the data set.
Data points that are drawn independently from the same distribution are said to be
independent and identically distributed, which is often abbreviated to i.i.d. We haveseen that the joint probability of two independent events is given by the product ofthe marginal probabilities for each event separately. Because our data set x is i.i.d.,
we can therefore write the probability of the data set, given µ and σ 2 , in the form2p(x|µ, σ ) =NN xn |µ, σ 2 .

(1.53)

n=1

Section 1.2.5

When viewed as a function of µ and σ 2 , this is the likelihood function for the Gaussian and is interpreted diagrammatically in Figure 1.14.
One common criterion for determining the parameters in a probability distribution using an observed data set is to ﬁnd the parameter values that maximize thelikelihood function. This might seem like a strange criterion because, from our foregoing discussion of probability theory, it would seem more natural to maximize theprobability of the parameters given the data, not the probability of the data given theparameters. In fact, these two criteria are related, as we shall discuss in the context
of curve ﬁtting.
For the moment, however, we shall determine values for the unknown parameters µ and σ 2 in the Gaussian by maximizing the likelihood function (1.53). In practice, it is more convenient to maximize the log of the likelihood function. Because
the logarithm is a monotonically increasing function of its argument, maximizationof the log of a function is equivalent to maximization of the function itself. Taking
the log not only simpliﬁes the subsequent mathematical analysis, but it also helps
numerically because the product of a large number of small probabilities can easily
underﬂow the numerical precision of the computer, and this is resolved by computing
instead the sum of the log probabilities. From (1.46) and (1.53), the log likelihood

1.2. Probability Theory

27

function can be written in the form
1 NNln σ 2 −ln(2π).
(xn − µ)2 −
2
2σ
2
2
N

ln p x|µ, σ 2 = −(1.54)

n=1

Exercise 1.11

Maximizing (1.54) with respect to µ, we obtain the maximum likelihood solution
given by
N
1 
µML =
xn
(1.55)
N
n=1

which is the sample mean, i.e., the mean of the observed values {xn }. Similarly,
maximizing (1.54) with respect to σ 2 , we obtain the maximum likelihood solution
for the variance in the form1 (xn − µML )2
N
N

2
=
σML

(1.56)

n=1

Section 1.1

Exercise 1.12

which is the sample variance measured with respect to the sample mean µML . Notethat we are performing a joint maximization of (1.54) with respect to µ and σ 2 , but
in the case of the Gaussian distribution the solution for µ decouples from that for σ 2
so that we can ﬁrst evaluate (1.55) and then subsequently use this result to evaluate(1.56).
Later in this chapter, and also in subsequent chapters, we shall highlight the signiﬁcant limitations of the maximum likelihood approach. Here we give an indicationof the problem in the context of our solutions for the maximum likelihood parameter settings for the univariate Gaussian distribution. In particular, we shall showthat the maximum likelihood approach systematically underestimates the varianceof the distribution. This is an example of a phenomenon called bias and is related
to the problem of over-ﬁtting encountered in the context of polynomial curve ﬁtting.
2
We ﬁrst note that the maximum likelihood solutions µML and σML
are functions of
the data set values x1 , . . . , xN . Consider the expectations of these quantities withrespect to the data set values, which themselves come from a Gaussian distribution
with parameters µ and σ 2 . It is straightforward to show that
E[µML ] = µ


N −1
2
E[σML ] =σ2N

(1.57)(1.58)

so that on average the maximum likelihood estimate will obtain the correct mean but
will underestimate the true variance by a factor (N − 1)/N . The intuition behind
this result is given by Figure 1.15.
From (1.58) it follows that the following estimate for the variance parameter isunbiased
N
N
1 
2
σ
2 =
=
(xn − µML )2 .
(1.59)
σML
N −1
N −1
n=1

28

1. INTRODUCTION
Figure 1.15

Illustration of how bias arises in using maximum likelihood to determine the varianceof a Gaussian. The green curve shows
the true Gaussian distribution from whichdata is generated, and the three red curves
show the Gaussian distributions obtainedby ﬁtting to three data sets, each consisting of two data points shown in blue, using the maximum likelihood results (1.55)
and (1.56). Averaged across the three data
sets, the mean is correct, but the varianceis systematically under-estimated because
it is measured relative to the sample mean
and not relative to the true mean.

(a)

(b)

(c)

In Section 10.1.3, we shall see how this result arises automatically when we adopt a
Bayesian approach.
Note that the bias of the maximum likelihood solution becomes less signiﬁcantas the number N of data points increases, and in the limit N → ∞ the maximum
likelihood solution for the variance equals the true variance of the distribution thatgenerated the data. In practice, for anything other than small N , this bias will notprove to be a serious problem. However, throughout this book we shall be interested
in more complex models with many parameters, for which the bias problems associated with maximum likelihood will be much more severe. In fact, as we shall see,the issue of bias in maximum likelihood lies at the root of the over-ﬁtting problem
that we encountered earlier in the context of polynomial curve ﬁtting.

1.2.5 Curve ﬁtting re-visited
Section 1.1

We have seen how the problem of polynomial curve ﬁtting can be expressed interms of error minimization. Here we return to the curve ﬁtting example and view itfrom a probabilistic perspective, thereby gaining some insights into error functions
and regularization, as well as taking us towards a full Bayesian treatment.
The goal in the curve ﬁtting problem is to be able to make predictions for thetarget variable t given some new value of the input variable x on the basis of a set oftraining data comprising N input values x = (x1 , . . . , xN )T and their corresponding
target values t = (t1 , . . . , tN )T . We can express our uncertainty over the value ofthe target variable using a probability distribution. For this purpose, we shall assume
that, given the value of x, the corresponding value of t has a Gaussian distribution
with a mean equal to the value y(x, w) of the polynomial curve given by (1.1). Thus
we have(1.60)
p(t|x, w, β) = N t|y(x, w), β −1where, for consistency with the notation in later chapters, we have deﬁned a precision parameter β corresponding to the inverse variance of the distribution. This is
illustrated schematically in Figure 1.16.

29

1.2. Probability Theory
Figure 1.16 Schematic illustration of a Gaussian conditional distribution for t given x given by(1.60), in which the mean is given by the polynomial function y(x, w), and the precision is givenby the parameter β, which is related to the variance by β −1 = σ 2 .

ty(x, w)y(x0 , w)2σp(t|x0 , w, β)x0x

We now use the training data {x, t} to determine the values of the unknown
parameters w and β by maximum likelihood. If the data are assumed to be drawn
independently from the distribution (1.60), then the likelihood function is given byp(t|x, w, β) =NN tn |y(xn , w), β −1 .

(1.61)

n=1

As we did in the case of the simple Gaussian distribution earlier, it is convenient tomaximize the logarithm of the likelihood function. Substituting for the form of theGaussian distribution, given by (1.46), we obtain the log likelihood function in the
form
N
β
N
2
ln β −ln(2π).
{y(xn , w) − tn } +222N

ln p(t|x, w, β) = −(1.62)

n=1

Consider ﬁrst the determination of the maximum likelihood solution for the polynomial coefﬁcients, which will be denoted by wML . These are determined by maximizing (1.62) with respect to w. For this purpose, we can omit the last two termson the right-hand side of (1.62) because they do not depend on w. Also, we notethat scaling the log likelihood by a positive constant coefﬁcient does not alter thelocation of the maximum with respect to w, and so we can replace the coefﬁcient
β/2 with 1/2. Finally, instead of maximizing the log likelihood, we can equivalently
minimize the negative log likelihood. We therefore see that maximizing likelihood isequivalent, so far as determining w is concerned, to minimizing the sum-of-squares
error function deﬁned by (1.2). Thus the sum-of-squares error function has arisen asa consequence of maximizing likelihood under the assumption of a Gaussian noise
distribution.
We can also use maximum likelihood to determine the precision parameter β of
the Gaussian conditional distribution. Maximizing (1.62) with respect to β gives
1
βML

1 
2
{y(xn , wML ) − tn } .
N
N

=

n=1

(1.63)

30

1. INTRODUCTION

Section 1.2.4

Again we can ﬁrst determine the parameter vector wML governing the mean and subsequently use this to ﬁnd the precision βML as was the case for the simple Gaussian
distribution.
Having determined the parameters w and β, we can now make predictions for
new values of x. Because we now have a probabilistic model, these are expressedin terms of the predictive distribution that gives the probability distribution over t,rather than simply a point estimate, and is obtained by substituting the maximum
likelihood parameters into (1.60) to give−1.
p(t|x, wML , βML ) = N t|y(x, wML ), βML(1.64)

Now let us take a step towards a more Bayesian approach and introduce a priordistribution over the polynomial coefﬁcients w. For simplicity, let us consider aGaussian distribution of the form
 α (M +1)/2
 α
exp − wT w
(1.65)
p(w|α) = N (w|0, α−1 I) =2π
2
where α is the precision of the distribution, and M +1 is the total number of elementsin the vector w for an M th order polynomial. Variables such as α, which control
the distribution of model parameters, are called hyperparameters. Using Bayes’
theorem, the posterior distribution for w is proportional to the product of the priordistribution and the likelihood functionp(w|x, t, α, β) ∝ p(t|x, w, β)p(w|α).

(1.66)

We can now determine w by ﬁnding the most probable value of w given the data,
in other words by maximizing the posterior distribution. This technique is called
maximum posterior, or simply MAP. Taking the negative logarithm of (1.66) andcombining with (1.62) and (1.65), we ﬁnd that the maximum of the posterior isgiven by the minimum ofαβ{y(xn , w) − tn }2 + wT w.
2
2
N(1.67)

n=1

Thus we see that maximizing the posterior distribution is equivalent to minimizing
the regularized sum-of-squares error function encountered earlier in the form (1.4),
with a regularization parameter given by λ = α/β.

1.2.6 Bayesian curve ﬁtting
Although we have included a prior distribution p(w|α), we are so far still making a point estimate of w and so this does not yet amount to a Bayesian treatment. Ina fully Bayesian approach, we should consistently apply the sum and product rulesof probability, which requires, as we shall see shortly, that we integrate over all values of w. Such marginalizations lie at the heart of Bayesian methods for patternrecognition.

1.2. Probability Theory

31

In the curve ﬁtting problem, we are given the training data x and t, along witha new test point x, and our goal is to predict the value of t. We therefore wish
to evaluate the predictive distribution p(t|x, x, t). Here we shall assume that theparameters α and β are ﬁxed and known in advance (in later chapters we shall discuss
how such parameters can be inferred from data in a Bayesian setting).
A Bayesian treatment simply corresponds to a consistent application of the sum
and product rules of probability, which allow the predictive distribution to be written
in the formp(t|x, x, t) = p(t|x, w)p(w|x, t) dw.
(1.68)
Here p(t|x, w) is given by (1.60), and we have omitted the dependence on α andβ to simplify the notation. Here p(w|x, t) is the posterior distribution over parameters, and can be found by normalizing the right-hand side of (1.66). We shall see
in Section 3.3 that, for problems such as the curve-ﬁtting example, this posterior
distribution is a Gaussian and can be evaluated analytically. Similarly, the integration in (1.68) can also be performed analytically with the result that the predictive
distribution is given by a Gaussian of the formp(t|x, x, t) = N t|m(x), s2 (x)(1.69)

where the mean and variance are given bym(x) = βφ(x) SN

φ(xn )tn(1.70)

s2 (x) = β −1 + φ(x) Sφ(x).

(1.71)

T

n=1
T

Here the matrix S is given by
S−1 = αI + βNφ(xn )φ(x)T(1.72)

n=1

where I is the unit matrix, and we have deﬁned the vector φ(x) with elements
φi (x) = xi for i = 0, . . . , M .
We see that the variance, as well as the mean, of the predictive distribution in(1.69) is dependent on x. The ﬁrst term in (1.71) represents the uncertainty in thepredicted value of t due to the noise on the target variables and was expressed already−1
in the maximum likelihood predictive distribution (1.64) through βML. However, thesecond term arises from the uncertainty in the parameters w and is a consequenceof the Bayesian treatment. The predictive distribution for the synthetic sinusoidal
regression problem is illustrated in Figure 1.17.

32

1. INTRODUCTION
Figure 1.17

The predictive distribution resulting from a Bayesian treatment ofpolynomial curve ﬁtting using an
M = 9 polynomial, with the ﬁxed
parameters α = 5 × 10−3 and β =11.1 (corresponding to the known
noise variance), in which the red
curve denotes the mean of thepredictive distribution and the redregion corresponds to ±1 standard deviation around the mean.

1
t
0

−1

0

x

1

1.3. Model Selection
In our example of polynomial curve ﬁtting using least squares, we saw that there wasan optimal order of polynomial that gave the best generalization. The order of thepolynomial controls the number of free parameters in the model and thereby governs
the model complexity. With regularized least squares, the regularization coefﬁcientλ also controls the effective complexity of the model, whereas for more complex
models, such as mixture distributions or neural networks there may be multiple parameters governing complexity. In a practical application, we need to determine
the values of such parameters, and the principal objective in doing so is usually toachieve the best predictive performance on new data. Furthermore, as well as ﬁnding the appropriate values for complexity parameters within a given model, we may
wish to consider a range of different types of model in order to ﬁnd the best one forour particular application.
We have already seen that, in the maximum likelihood approach, the performance on the training set is not a good indicator of predictive performance on unseen data due to the problem of over-ﬁtting. If data is plentiful, then one approach issimply to use some of the available data to train a range of models, or a given modelwith a range of values for its complexity parameters, and then to compare them on
independent data, sometimes called a validation set, and select the one having thebest predictive performance. If the model design is iterated many times using a limited size data set, then some over-ﬁtting to the validation data can occur and so it may
be necessary to keep aside a third test set on which the performance of the selectedmodel is ﬁnally evaluated.
In many applications, however, the supply of data for training and testing will belimited, and in order to build good models, we wish to use as much of the available
data as possible for training. However, if the validation set is small, it will give a
relatively noisy estimate of predictive performance. One solution to this dilemma is
to use cross-validation, which is illustrated in Figure 1.18. This allows a proportion(S − 1)/S of the available data to be used for training while making use of all of the1.4. The Curse of Dimensionality
Figure 1.18

The technique of S-fold cross-validation, illustrated here for the case of S = 4, involves taking the available data and partitioning it into S
groups (in the simplest case these are of equalsize). Then S − 1 of the groups are used to train
a set of models that are then evaluated on the remaining group. This procedure is then repeated
for all S possible choices for the held-out group,
indicated here by the red blocks, and the performance scores from the S runs are then averaged.

33
run 1
run 2
run 3
run 4

data to assess performance. When data is particularly scarce, it may be appropriate
to consider the case S = N , where N is the total number of data points, which gives
the leave-one-out technique.
One major drawback of cross-validation is that the number of training runs thatmust be performed is increased by a factor of S, and this can prove problematic formodels in which the training is itself computationally expensive. A further problem
with techniques such as cross-validation that use separate data to assess performanceis that we might have multiple complexity parameters for a single model (for instance, there might be several regularization parameters). Exploring combinationsof settings for such parameters could, in the worst case, require a number of training
runs that is exponential in the number of parameters. Clearly, we need a better approach. Ideally, this should rely only on the training data and should allow multiple
hyperparameters and model types to be compared in a single training run. We therefore need to ﬁnd a measure of performance which depends only on the training data
and which does not suffer from bias due to over-ﬁtting.
Historically various ‘information criteria’ have been proposed that attempt tocorrect for the bias of maximum likelihood by the addition of a penalty term tocompensate for the over-ﬁtting of more complex models. For example, the Akaike
information criterion, or AIC (Akaike, 1974), chooses the model for which the quantity(1.73)
ln p(D|wML ) − M
is largest. Here p(D|wML ) is the best-ﬁt log likelihood, and M is the number ofadjustable parameters in the model. A variant of this quantity, called the Bayesian
information criterion, or BIC, will be discussed in Section 4.4.1. Such criteria do
not take account of the uncertainty in the model parameters, however, and in practicethey tend to favour overly simple models. We therefore turn in Section 3.4 to a fully
Bayesian approach where we shall see how complexity penalties arise in a naturaland principled way.

1.4. The Curse of Dimensionality
In the polynomial curve ﬁtting example we had just one input variable x. For practical applications of pattern recognition, however, we will have to deal with spaces

34

1. INTRODUCTION
Figure 1.19

Scatter plot of the oil ﬂow datafor input variables x6 and x7 , inwhich red denotes the ‘homogenous’ class, green denotes the‘annular’ class, and blue denotesthe ‘laminar’ class. Our goal is
to classify the new test point denoted by ‘×’.

2

1.5

x7

1

0.5

0

0

0.25

0.5
x6

0.75

1

of high dimensionality comprising many input variables. As we now discuss, thisposes some serious challenges and is an important factor inﬂuencing the design ofpattern recognition techniques.
In order to illustrate the problem we consider a synthetically generated data set
representing measurements taken from a pipeline containing a mixture of oil, water, and gas (Bishop and James, 1993). These three materials can be present in oneof three different geometrical conﬁgurations known as ‘homogenous’, ‘annular’, and‘laminar’, and the fractions of the three materials can also vary. Each data point comprises a 12-dimensional input vector consisting of measurements taken with gamma
ray densitometers that measure the attenuation of gamma rays passing along narrow beams through the pipe. This data set is described in detail in Appendix A.
Figure 1.19 shows 100 points from this data set on a plot showing two of the measurements x6 and x7 (the remaining ten input values are ignored for the purposes ofthis illustration). Each data point is labelled according to which of the three geometrical classes it belongs to, and our goal is to use this data as a training set in order tobe able to classify a new observation (x6 , x7 ), such as the one denoted by the crossin Figure 1.19. We observe that the cross is surrounded by numerous red points, andso we might suppose that it belongs to the red class. However, there are also plentyof green points nearby, so we might think that it could instead belong to the green
class. It seems unlikely that it belongs to the blue class. The intuition here is that theidentity of the cross should be determined more strongly by nearby points from thetraining set and less strongly by more distant points. In fact, this intuition turns out
to be reasonable and will be discussed more fully in later chapters.
How can we turn this intuition into a learning algorithm? One very simple approach would be to divide the input space into regular cells, as indicated in Figure 1.20. When we are given a test point and we wish to predict its class, we ﬁrst
decide which cell it belongs to, and we then ﬁnd all of the training data points that

1.4. The Curse of Dimensionality
Figure 1.20

Illustration of a simple approachto the solution of a classiﬁcation
problem in which the input spaceis divided into cells and any new
test point is assigned to the classthat has a majority number of representatives in the same cell as
the test point. As we shall see
shortly, this simplistic approach
has some severe shortcomings.

35

2

1.5

x7

1

0.5

0

Section 1.1

0

0.25

0.5
x6

0.75

1

fall in the same cell. The identity of the test point is predicted as being the sameas the class having the largest number of training points in the same cell as the test
point (with ties being broken at random).
There are numerous problems with this naive approach, but one of the most severe becomes apparent when we consider its extension to problems having larger
numbers of input variables, corresponding to input spaces of higher dimensionality.
The origin of the problem is illustrated in Figure 1.21, which shows that, if we dividea region of a space into regular cells, then the number of such cells grows exponentially with the dimensionality of the space. The problem with an exponentially large
number of cells is that we would need an exponentially large quantity of training datain order to ensure that the cells are not empty. Clearly, we have no hope of applying
such a technique in a space of more than a few variables, and so we need to ﬁnd amore sophisticated approach.
We can gain further insight into the problems of high-dimensional spaces by
returning to the example of polynomial curve ﬁtting and considering how we would
x2

Figure 1.21 Illustration of thecurse of dimensionality, showing
how the number of regions of aregular grid grows exponentiallywith the dimensionality D of thespace. For clarity, only a subset ofthe cubical regions are shown for
D = 3.

x2

x1
x1
D=1

x1
D=2

x3
D=3

36

1. INTRODUCTION
extend this approach to deal with input spaces having several variables. If we have
D input variables, then a general polynomial with coefﬁcients up to order 3 would
take the formy(x, w) = w0 +

D

i=1

Exercise 1.16

wi xi +D D

wij xi xj +i=1 j =1

D 
D 
D

i=1 j =1 k=1

As D increases, so the number of independent coefﬁcients (not all of the coefﬁcientsare independent due to interchange symmetries amongst the x variables) grows proportionally to D3 . In practice, to capture complex dependencies in the data, we mayneed to use a higher-order polynomial. For a polynomial of order M , the growth in
the number of coefﬁcients is like DM . Although this is now a power law growth,rather than an exponential growth, it still points to the method becoming rapidly
unwieldy and of limited practical utility.
Our geometrical intuitions, formed through a life spent in a space of three dimensions, can fail badly when we consider spaces of higher dimensionality. As asimple example, consider a sphere of radius r = 1 in a space of D dimensions, andask what is the fraction of the volume of the sphere that lies between radius r = 1−
and r = 1. We can evaluate this fraction by noting that the volume of a sphere ofradius r in D dimensions must scale as rD , and so we write
VD (r) = KD rD

Exercise 1.18

(1.75)

where the constant KD depends only on D. Thus the required fraction is given by
VD (1) − VD (1 − )= 1 − (1 − )DVD (1)

Exercise 1.20

wijk xi xj xk . (1.74)(1.76)

which is plotted as a function of for various values of D in Figure 1.22. We see
that, for large D, this fraction tends to 1 even for small values of . Thus, in spacesof high dimensionality, most of the volume of a sphere is concentrated in a thin shellnear the surface!
As a further example, of direct relevance to pattern recognition, consider the
behaviour of a Gaussian distribution in a high-dimensional space. If we transformfrom Cartesian to polar coordinates, and then integrate out the directional variables,
we obtain an expression for the density p(r) as a function of radius r from the origin.
Thus p(r)δr is the probability mass inside a thin shell of thickness δr located atradius r. This distribution is plotted, for various values of D, in Figure 1.23, and we
see that for large D the probability mass of the Gaussian is concentrated in a thinshell.
The severe difﬁculty that can arise in spaces of many dimensions is sometimes
called the curse of dimensionality (Bellman, 1961). In this book, we shall make extensive use of illustrative examples involving input spaces of one or two dimensions,because this makes it particularly easy to illustrate the techniques graphically. The
reader should be warned, however, that not all intuitions developed in spaces of low
dimensionality will generalize to spaces of many dimensions.

37

1.4. The Curse of Dimensionality
Plot of the fraction of the volume ofa sphere lying in the range r = 1−to r = 1 for various values of thedimensionality D.

1
D = 20

0.8
volume fraction

Figure 1.22

D =5
D =2

0.6
D =1

0.4

0.2

0

0

0.2

0.4

0.6

0.8

1

Although the curse of dimensionality certainly raises important issues for pattern recognition applications, it does not prevent us from ﬁnding effective techniquesapplicable to high-dimensional spaces. The reasons for this are twofold. First, real
data will often be conﬁned to a region of the space having lower effective dimensionality, and in particular the directions over which important variations in the target
variables occur may be so conﬁned. Second, real data will typically exhibit somesmoothness properties (at least locally) so that for the most part small changes in the
input variables will produce small changes in the target variables, and so we can exploit local interpolation-like techniques to allow us to make predictions of the target
variables for new values of the input variables. Successful pattern recognition techniques exploit one or both of these properties. Consider, for example, an applicationin manufacturing in which images are captured of identical planar objects on a conveyor belt, in which the goal is to determine their orientation. Each image is a point
Plot of the probability density withrespect to radius r of a Gaussian distribution for various valuesof the dimensionality D.
In ahigh-dimensional space, most of theprobability mass of a Gaussian is located within a thin shell at a speciﬁc
radius.

2
D=1
p(r)

Figure 1.23

D=2
1

0

D = 20

0

2
r

4

38

1. INTRODUCTION
in a high-dimensional space whose dimensionality is determined by the number ofpixels. Because the objects can occur at different positions within the image andin different orientations, there are three degrees of freedom of variability between
images, and a set of images will live on a three dimensional manifold embedded
within the high-dimensional space. Due to the complex relationships between theobject position or orientation and the pixel intensities, this manifold will be highly
nonlinear. If the goal is to learn a model that can take an input image and output theorientation of the object irrespective of its position, then there is only one degree of
freedom of variability within the manifold that is signiﬁcant.

1.5. Decision Theory
We have seen in Section 1.2 how probability theory provides us with a consistentmathematical framework for quantifying and manipulating uncertainty. Here weturn to a discussion of decision theory that, when combined with probability theory,
allows us to make optimal decisions in situations involving uncertainty such as thoseencountered in pattern recognition.
Suppose we have an input vector x together with a corresponding vector t oftarget variables, and our goal is to predict t given a new value for x. For regression
problems, t will comprise continuous variables, whereas for classiﬁcation problems
t will represent class labels. The joint probability distribution p(x, t) provides acomplete summary of the uncertainty associated with these variables. Determinationof p(x, t) from a set of training data is an example of inference and is typically avery difﬁcult problem whose solution forms the subject of much of this book. In
a practical application, however, we must often make a speciﬁc prediction for thevalue of t, or more generally take a speciﬁc action based on our understanding of thevalues t is likely to take, and this aspect is the subject of decision theory.
Consider, for example, a medical diagnosis problem in which we have taken an
X-ray image of a patient, and we wish to determine whether the patient has cancer
or not. In this case, the input vector x is the set of pixel intensities in the image,and output variable t will represent the presence of cancer, which we denote by theclass C1 , or the absence of cancer, which we denote by the class C2 . We might, forinstance, choose t to be a binary variable such that t = 0 corresponds to class C1 andt = 1 corresponds to class C2 . We shall see later that this choice of label values isparticularly convenient for probabilistic models. The general inference problem theninvolves determining the joint distribution p(x, Ck ), or equivalently p(x, t), which
gives us the most complete probabilistic description of the situation. Although thiscan be a very useful and informative quantity, in the end we must decide either togive treatment to the patient or not, and we would like this choice to be optimal
in some appropriate sense (Duda and Hart, 1973). This is the decision step, and
it is the subject of decision theory to tell us how to make optimal decisions given
the appropriate probabilities. We shall see that the decision stage is generally very
simple, even trivial, once we have solved the inference problem.
Here we give an introduction to the key ideas of decision theory as required for1.5. Decision Theory39

the rest of the book. Further background, as well as more detailed accounts, can befound in Berger (1985) and Bather (2000).
Before giving a more detailed analysis, let us ﬁrst consider informally how wemight expect probabilities to play a role in making decisions. When we obtain the
X-ray image x for a new patient, our goal is to decide which of the two classes toassign to the image. We are interested in the probabilities of the two classes given
the image, which are given by p(Ck |x). Using Bayes’ theorem, these probabilities
can be expressed in the formp(Ck |x) =p(x|Ck )p(Ck ).
p(x)(1.77)

Note that any of the quantities appearing in Bayes’ theorem can be obtained fromthe joint distribution p(x, Ck ) by either marginalizing or conditioning with respect tothe appropriate variables. We can now interpret p(Ck ) as the prior probability for theclass Ck , and p(Ck |x) as the corresponding posterior probability. Thus p(C1 ) represents the probability that a person has cancer, before we take the X-ray measurement.
Similarly, p(C1 |x) is the corresponding probability, revised using Bayes’ theorem inlight of the information contained in the X-ray. If our aim is to minimize the chanceof assigning x to the wrong class, then intuitively we would choose the class having
the higher posterior probability. We now show that this intuition is correct, and we
also discuss more general criteria for making decisions.

1.5.1 Minimizing the misclassiﬁcation rate
Suppose that our goal is simply to make as few misclassiﬁcations as possible.
We need a rule that assigns each value of x to one of the available classes. Such arule will divide the input space into regions Rk called decision regions, one for eachclass, such that all points in Rk are assigned to class Ck . The boundaries between
decision regions are called decision boundaries or decision surfaces. Note that eachdecision region need not be contiguous but could comprise some number of disjoint
regions. We shall encounter examples of decision boundaries and decision regions inlater chapters. In order to ﬁnd the optimal decision rule, consider ﬁrst of all the caseof two classes, as in the cancer problem for instance. A mistake occurs when an input
vector belonging to class C1 is assigned to class C2 or vice versa. The probability ofthis occurring is given byp(mistake) = p(x ∈ R1 , C2 ) + p(x ∈ R2 , C1 )=p(x, C2 ) dx +p(x, C1 ) dx.
R1(1.78)

R2

We are free to choose the decision rule that assigns each point x to one of the twoclasses. Clearly to minimize p(mistake) we should arrange that each x is assigned towhichever class has the smaller value of the integrand in (1.78). Thus, if p(x, C1 ) >p(x, C2 ) for a given value of x, then we should assign that x to class C1 . From theproduct rule of probability we have p(x, Ck ) = p(Ck |x)p(x). Because the factor
p(x) is common to both terms, we can restate this result as saying that the minimum

40

1. INTRODUCTIONxx0p(x, C1 )p(x, C2 )x
R1Figure 1.24

R2

Schematic illustration of the joint probabilities p(x, Ck ) for each of two classes plottedagainst x, together with the decision boundary x = x
b. Values of x  xb are classiﬁed asb are classiﬁed
class C2 and hence belong to decision region R2 , whereas points x < x
as C1 and belong to R1 . Errors arise from the blue, green, and red regions, so that forx<xb the errors are due to points from class C2 being misclassiﬁed as C1 (represented by
the sum of the red and green regions), and conversely for points in the region x  xb theerrors are due to points from class C1 being misclassiﬁed as C2 (represented by the blue
region). As we vary the location xb of the decision boundary, the combined areas of theblue and green regions remains constant, whereas the size of the red region varies. The
optimal choice for x
b is where the curves for p(x, C1 ) and p(x, C2 ) cross, corresponding tox
b = x0 , because in this case the red region disappears. This is equivalent to the minimum
misclassiﬁcation rate decision rule, which assigns each value of x to the class having thehigher posterior probability p(Ck |x).

probability of making a mistake is obtained if each value of x is assigned to the classfor which the posterior probability p(Ck |x) is largest. This result is illustrated fortwo classes, and a single input variable x, in Figure 1.24.
For the more general case of K classes, it is slightly easier to maximize theprobability of being correct, which is given byp(correct) =Kp(x ∈ Rk , Ck )

k=1

=

K 

k=1

Rk

p(x, Ck ) dx

(1.79)

which is maximized when the regions Rk are chosen such that each x is assigned
to the class for which p(x, Ck ) is largest. Again, using the product rule p(x, Ck ) =p(Ck |x)p(x), and noting that the factor of p(x) is common to all terms, we seethat each x should be assigned to the class having the largest posterior probabilityp(Ck |x).

1.5. Decision Theory
Figure 1.25 An example of a loss matrix with elements Lkj for the cancer treatment problem. The rows
correspond to the true class, whereas the columns correspond to the assignment of class made by our decision criterion.

cancernormal41

 cancer normal 
0
1000
1
0

1.5.2 Minimizing the expected loss
For many applications, our objective will be more complex than simply minimizing the number of misclassiﬁcations. Let us consider again the medical diagnosis
problem. We note that, if a patient who does not have cancer is incorrectly diagnosed
as having cancer, the consequences may be some patient distress plus the need forfurther investigations. Conversely, if a patient with cancer is diagnosed as healthy,
the result may be premature death due to lack of treatment. Thus the consequencesof these two types of mistake can be dramatically different. It would clearly be better
to make fewer mistakes of the second kind, even if this was at the expense of making
more mistakes of the ﬁrst kind.
We can formalize such issues through the introduction of a loss function, also
called a cost function, which is a single, overall measure of loss incurred in taking
any of the available decisions or actions. Our goal is then to minimize the total loss
incurred. Note that some authors consider instead a utility function, whose value
they aim to maximize. These are equivalent concepts if we take the utility to be
simply the negative of the loss, and throughout this text we shall use the loss function
convention. Suppose that, for a new value of x, the true class is Ck and that we assign
x to class Cj (where j may or may not be equal to k). In so doing, we incur somelevel of loss that we denote by Lkj , which we can view as the k, j element of a loss
matrix. For instance, in our cancer example, we might have a loss matrix of the formshown in Figure 1.25. This particular loss matrix says that there is no loss incurred
if the correct decision is made, there is a loss of 1 if a healthy patient is diagnosed ashaving cancer, whereas there is a loss of 1000 if a patient having cancer is diagnosed
as healthy.
The optimal solution is the one which minimizes the loss function. However,the loss function depends on the true class, which is unknown. For a given input
vector x, our uncertainty in the true class is expressed through the joint probability
distribution p(x, Ck ) and so we seek instead to minimize the average loss, where theaverage is computed with respect to this distribution, which is given by
E[L] =
Lkj p(x, Ck ) dx.
(1.80)kj

Rj

Each x can be assigned independently to one of the decision regions Rj . Our goal
the expected loss (1.80), whichis to choose the regions Rj in order to minimize
implies that for each x we should minimize k Lkj p(x, Ck ). As before, we can use
the product rule p(x, Ck ) = p(Ck |x)p(x) to eliminate the common factor of p(x).
Thus the decision rule that minimizes the expected loss is the one that assigns each42

1. INTRODUCTION
Figure 1.26

Illustration of the reject option. Inputs
x such that the larger of the two poste- 1.0rior probabilities is less than or equal toθsome threshold θ will be rejected.

p(C1 |x)p(C2 |x)0.0
reject region

new x to the class j for which the quantity
Lkj p(Ck |x)x(1.81)kis a minimum. This is clearly trivial to do, once we know the posterior class probabilities p(Ck |x).

1.5.3 The reject option

Exercise 1.24

We have seen that classiﬁcation errors arise from the regions of input space
where the largest of the posterior probabilities p(Ck |x) is signiﬁcantly less than unity,or equivalently where the joint distributions p(x, Ck ) have comparable values. These
are the regions where we are relatively uncertain about class membership. In some
applications, it will be appropriate to avoid making decisions on the difﬁcult casesin anticipation of a lower error rate on those examples for which a classiﬁcation decision is made. This is known as the reject option. For example, in our hypothetical
medical illustration, it may be appropriate to use an automatic system to classify
those X-ray images for which there is little doubt as to the correct class, while leaving a human expert to classify the more ambiguous cases. We can achieve this byintroducing a threshold θ and rejecting those inputs x for which the largest of theposterior probabilities p(Ck |x) is less than or equal to θ. This is illustrated for thecase of two classes, and a single continuous input variable x, in Figure 1.26. Notethat setting θ = 1 will ensure that all examples are rejected, whereas if there are Kclasses then setting θ < 1/K will ensure that no examples are rejected. Thus thefraction of examples that get rejected is controlled by the value of θ.
We can easily extend the reject criterion to minimize the expected loss, whena loss matrix is given, taking account of the loss incurred when a reject decision is
made.

1.5.4 Inference and decision
We have broken the classiﬁcation problem down into two separate stages, theinference stage in which we use training data to learn a model for p(Ck |x), and the1.5. Decision Theory

43

subsequent decision stage in which we use these posterior probabilities to make optimal class assignments. An alternative possibility would be to solve both problems
together and simply learn a function that maps inputs x directly into decisions. Sucha function is called a discriminant function.
In fact, we can identify three distinct approaches to solving decision problems,
all of which have been used in practical applications. These are given, in decreasingorder of complexity, by:(a) First solve the inference problem of determining the class-conditional densities
p(x|Ck ) for each class Ck individually. Also separately infer the prior classprobabilities p(Ck ). Then use Bayes’ theorem in the formp(Ck |x) =p(x|Ck )p(Ck )p(x)(1.82)

to ﬁnd the posterior class probabilities p(Ck |x). As usual, the denominatorin Bayes’ theorem can be found in terms of the quantities appearing in the
numerator, becausep(x|Ck )p(Ck ).
(1.83)
p(x) =kEquivalently, we can model the joint distribution p(x, Ck ) directly and thennormalize to obtain the posterior probabilities. Having found the posterior
probabilities, we use decision theory to determine class membership for each
new input x. Approaches that explicitly or implicitly model the distribution of
inputs as well as outputs are known as generative models, because by samplingfrom them it is possible to generate synthetic data points in the input space.
(b) First solve the inference problem of determining the posterior class probabilitiesp(Ck |x), and then subsequently use decision theory to assign each new x toone of the classes. Approaches that model the posterior probabilities directly
are called discriminative models.
(c) Find a function f (x), called a discriminant function, which maps each input x
directly onto a class label. For instance, in the case of two-class problems,f (·) might be binary valued and such that f = 0 represents class C1 and f = 1
represents class C2 . In this case, probabilities play no role.
Let us consider the relative merits of these three alternatives. Approach (a) is themost demanding because it involves ﬁnding the joint distribution over both x and
Ck . For many applications, x will have high dimensionality, and consequently we
may need a large training set in order to be able to determine the class-conditional
densities to reasonable accuracy. Note that the class priors p(Ck ) can often be estimated simply from the fractions of the training set data points in each of the classes.
One advantage of approach (a), however, is that it also allows the marginal densityof data p(x) to be determined from (1.83). This can be useful for detecting new data
points that have low probability under the model and for which the predictions may

44

1. INTRODUCTION5

1.2
p(C1 |x)p(x|C2 )1

4class densitiesp(C2 |x)

0.8
3
0.6
2
0.4

p(x|C1 )
1

0

0.2

0

0.2

0.4

0.6
x

0.8

1

0

0

0.2

0.4

0.6

0.8

1

x

Figure 1.27 Example of the class-conditional densities for two classes having a single input variable x (leftplot) together with the corresponding posterior probabilities (right plot). Note that the left-hand mode of theclass-conditional density p(x|C1 ), shown in blue on the left plot, has no effect on the posterior probabilities. The
vertical green line in the right plot shows the decision boundary in x that gives the minimum misclassiﬁcation
rate.

be of low accuracy, which is known as outlier detection or novelty detection (Bishop,
1994; Tarassenko, 1995).
However, if we only wish to make classiﬁcation decisions, then it can be wasteful of computational resources, and excessively demanding of data, to ﬁnd the joint
distribution p(x, Ck ) when in fact we only really need the posterior probabilitiesp(Ck |x), which can be obtained directly through approach (b). Indeed, the classconditional densities may contain a lot of structure that has little effect on the posterior probabilities, as illustrated in Figure 1.27. There has been much interest in
exploring the relative merits of generative and discriminative approaches to machinelearning, and in ﬁnding ways to combine them (Jebara, 2004; Lasserre et al., 2006).
An even simpler approach is (c) in which we use the training data to ﬁnd a
discriminant function f (x) that maps each x directly onto a class label, thereby
combining the inference and decision stages into a single learning problem. In theexample of Figure 1.27, this would correspond to ﬁnding the value of x shown by
the vertical green line, because this is the decision boundary giving the minimum
probability of misclassiﬁcation.
With option (c), however, we no longer have access to the posterior probabilitiesp(Ck |x). There are many powerful reasons for wanting to compute the posterior
probabilities, even if we subsequently use them to make decisions. These include:
Minimizing risk. Consider a problem in which the elements of the loss matrix are
subjected to revision from time to time (such as might occur in a ﬁnancial

1.5. Decision Theory

45

application). If we know the posterior probabilities, we can trivially revise theminimum risk decision criterion by modifying (1.81) appropriately. If we have
only a discriminant function, then any change to the loss matrix would requirethat we return to the training data and solve the classiﬁcation problem afresh.
Reject option. Posterior probabilities allow us to determine a rejection criterion thatwill minimize the misclassiﬁcation rate, or more generally the expected loss,
for a given fraction of rejected data points.
Compensating for class priors. Consider our medical X-ray problem again, andsuppose that we have collected a large number of X-ray images from the general population for use as training data in order to build an automated screening
system. Because cancer is rare amongst the general population, we might ﬁndthat, say, only 1 in every 1,000 examples corresponds to the presence of cancer. If we used such a data set to train an adaptive model, we could run into
severe difﬁculties due to the small proportion of the cancer class. For instance,a classiﬁer that assigned every point to the normal class would already achieve
99.9% accuracy and it would be difﬁcult to avoid this trivial solution. Also,
even a large data set will contain very few examples of X-ray images corresponding to cancer, and so the learning algorithm will not be exposed to abroad range of examples of such images and hence is not likely to generalize
well. A balanced data set in which we have selected equal numbers of examples from each of the classes would allow us to ﬁnd a more accurate model.
However, we then have to compensate for the effects of our modiﬁcations to
the training data. Suppose we have used such a modiﬁed data set and foundmodels for the posterior probabilities. From Bayes’ theorem (1.82), we see thatthe posterior probabilities are proportional to the prior probabilities, which we
can interpret as the fractions of points in each class. We can therefore simply
take the posterior probabilities obtained from our artiﬁcially balanced data set
and ﬁrst divide by the class fractions in that data set and then multiply by theclass fractions in the population to which we wish to apply the model. Finally,
we need to normalize to ensure that the new posterior probabilities sum to one.
Note that this procedure cannot be applied if we have learned a discriminantfunction directly instead of determining posterior probabilities.
Combining models. For complex applications, we may wish to break the problem
into a number of smaller subproblems each of which can be tackled by a separate module. For example, in our hypothetical medical diagnosis problem,
we may have information available from, say, blood tests as well as X-ray images. Rather than combine all of this heterogeneous information into one huge
input space, it may be more effective to build one system to interpret the Xray images and a different one to interpret the blood data. As long as each of
the two models gives posterior probabilities for the classes, we can combine
the outputs systematically using the rules of probability. One simple way to
do this is to assume that, for each class separately, the distributions of inputs
for the X-ray images, denoted by xI , and the blood data, denoted by xB , are46

1. INTRODUCTION
independent, so thatp(xI , xB |Ck ) = p(xI |Ck )p(xB |Ck ).

Section 8.2

This is an example of conditional independence property, because the independence holds when the distribution is conditioned on the class Ck . The posterior
probability, given both the X-ray and blood data, is then given byp(Ck |xI , xB ) ∝ p(xI , xB |Ck )p(Ck )∝ p(xI |Ck )p(xB |Ck )p(Ck )p(Ck |xI )p(Ck |xB )∝p(Ck )

Section 8.2.2(1.84)(1.85)

Thus we need the class prior probabilities p(Ck ), which we can easily estimate
from the fractions of data points in each class, and then we need to normalize
the resulting posterior probabilities so they sum to one. The particular conditional independence assumption (1.84) is an example of the naive Bayes model.
Note that the joint marginal distribution p(xI , xB ) will typically not factorize
under this model. We shall see in later chapters how to construct models for
combining data that do not require the conditional independence assumption(1.84).

1.5.5 Loss functions for regression

Section 1.1

So far, we have discussed decision theory in the context of classiﬁcation problems. We now turn to the case of regression problems, such as the curve ﬁtting
example discussed earlier. The decision stage consists of choosing a speciﬁc estimate y(x) of the value of t for each input x. Suppose that in doing so, we incur aloss L(t, y(x)). The average, or expected, loss is then given byL(t, y(x))p(x, t) dx dt.
(1.86)
E[L] =
A common choice of loss function in regression problems is the squared loss givenby L(t, y(x)) = {y(x) − t}2 . In this case, the expected loss can be written{y(x) − t}2 p(x, t) dx dt.
(1.87)
E[L] =

Appendix D

Our goal is to choose y(x) so as to minimize E[L]. If we assume a completely
ﬂexible function y(x), we can do this formally using the calculus of variations to
give

δE[L]
(1.88)= 2 {y(x) − t}p(x, t) dt = 0.
δy(x)Solving for y(x), and using the sum and product rules of probability, we obtain
tp(x, t) dt 
= tp(t|x) dt = Et [t|x]y(x) =(1.89)p(x)471.5. Decision Theory
Figure 1.28

The regression function y(x),which minimizes the expectedsquared loss, is given by themean of the conditional distribution p(t|x).

ty(x)y(x0 )
p(t|x0 )x0Exercise 1.25

xwhich is the conditional average of t conditioned on x and is known as the regressionfunction. This result is illustrated in Figure 1.28. It can readily be extended to multiple target variables represented by the vector t, in which case the optimal solution
is the conditional average y(x) = Et [t|x].
We can also derive this result in a slightly different way, which will also shedlight on the nature of the regression problem. Armed with the knowledge that the
optimal solution is the conditional expectation, we can expand the square term as
follows{y(x) − t}2 = {y(x) − E[t|x] + E[t|x] − t}2= {y(x) − E[t|x]}2 + 2{y(x) − E[t|x]}{E[t|x] − t} + {E[t|x] − t}2where, to keep the notation uncluttered, we use E[t|x] to denote Et [t|x]. Substituting
into the loss function and performing the integral over t, we see that the cross-term
vanishes and we obtain an expression for the loss function in the form
2
(1.90)
E[L] = {y(x) − E[t|x]} p(x) dx + {E[t|x] − t}2 p(x) dx.
The function y(x) we seek to determine enters only in the ﬁrst term, which will beminimized when y(x) is equal to E[t|x], in which case this term will vanish. Thisis simply the result that we derived previously and that shows that the optimal leastsquares predictor is given by the conditional mean. The second term is the varianceof the distribution of t, averaged over x. It represents the intrinsic variability ofthe target data and can be regarded as noise. Because it is independent of y(x), it
represents the irreducible minimum value of the loss function.
As with the classiﬁcation problem, we can either determine the appropriate probabilities and then use these to make optimal decisions, or we can build models thatmake decisions directly. Indeed, we can identify three distinct approaches to solving
regression problems given, in order of decreasing complexity, by:(a) First solve the inference problem of determining the joint density p(x, t). Then
normalize to ﬁnd the conditional density p(t|x), and ﬁnally marginalize to ﬁnd
the conditional mean given by (1.89).

48

1. INTRODUCTION(b) First solve the inference problem of determining the conditional density p(t|x),
and then subsequently marginalize to ﬁnd the conditional mean given by (1.89).
(c) Find a regression function y(x) directly from the training data.

Section 5.6

Exercise 1.27

The relative merits of these three approaches follow the same lines as for classiﬁcation problems above.
The squared loss is not the only possible choice of loss function for regression.
Indeed, there are situations in which squared loss can lead to very poor results andwhere we need to develop more sophisticated approaches. An important example
concerns situations in which the conditional distribution p(t|x) is multimodal, as
often arises in the solution of inverse problems. Here we consider brieﬂy one simple
generalization of the squared loss, called the Minkowski loss, whose expectation is
given by
E[Lq ] =|y(x) − t|q p(x, t) dx dt(1.91)
which reduces to the expected squared loss for q = 2. The function |y − t|q isplotted against y − t for various values of q in Figure 1.29. The minimum of E[Lq ]is given by the conditional mean for q = 2, the conditional median for q = 1, andthe conditional mode for q → 0.

1.6. Information Theory

Exercise 1.28

In this chapter, we have discussed a variety of concepts from probability theory anddecision theory that will form the foundations for much of the subsequent discussion
in this book. We close this chapter by introducing some additional concepts fromthe ﬁeld of information theory, which will also prove useful in our development ofpattern recognition and machine learning techniques. Again, we shall focus only on
the key concepts, and we refer the reader elsewhere for more detailed discussions(Viterbi and Omura, 1979; Cover and Thomas, 1991; MacKay, 2003) .
We begin by considering a discrete random variable x and we ask how muchinformation is received when we observe a speciﬁc value for this variable. The
amount of information can be viewed as the ‘degree of surprise’ on learning thevalue of x. If we are told that a highly improbable event has just occurred, we will
have received more information than if we were told that some very likely event
has just occurred, and if we knew that the event was certain to happen we would
receive no information. Our measure of information content will therefore dependon the probability distribution p(x), and we therefore look for a quantity h(x) thatis a monotonic function of the probability p(x) and that expresses the information
content. The form of h(·) can be found by noting that if we have two events x
and y that are unrelated, then the information gain from observing both of themshould be the sum of the information gained from each of them separately, so thath(x, y) = h(x) + h(y). Two unrelated events will be statistically independent andso p(x, y) = p(x)p(y). From these two relationships, it is easily shown that h(x)must be given by the logarithm of p(x) and so we have

49

1.6. Information Theory22
q=1|y − t|q|y − t|qq = 0.3
1

0
−2

−1

0
y −t

1

1

0
−2

2

2

−1

1

2

1

2

2
q = 10|y − t|qq=2|y − t|q

0
y −t

1

0
−2

−1

Figure 1.29

0
y −t

1

2

1

0
−2

−1

0
y −t

Plots of the quantity Lq = |y − t|q for various values of q.

h(x) = − log2 p(x)(1.92)

where the negative sign ensures that information is positive or zero. Note that low
probability events x correspond to high information content. The choice of basis
for the logarithm is arbitrary, and for the moment we shall adopt the convention
prevalent in information theory of using logarithms to the base of 2. In this case, as
we shall see shortly, the units of h(x) are bits (‘binary digits’).
Now suppose that a sender wishes to transmit the value of a random variable toa receiver. The average amount of information that they transmit in the process isobtained by taking the expectation of (1.92) with respect to the distribution p(x) andis given byp(x) log2 p(x).
(1.93)
H[x] = −xThis important quantity is called the entropy of the random variable x. Note that
limp→0 p ln p = 0 and so we shall take p(x) ln p(x) = 0 whenever we encounter avalue for x such that p(x) = 0.
So far we have given a rather heuristic motivation for the deﬁnition of informa-

50

1. INTRODUCTION
tion (1.92) and the corresponding entropy (1.93). We now show that these deﬁnitionsindeed possess useful properties. Consider a random variable x having 8 possible
states, each of which is equally likely. In order to communicate the value of x toa receiver, we would need to transmit a message of length 3 bits. Notice that theentropy of this variable is given by
H[x] = −8 ×1
1
log2 = 3 bits.
88

Now consider an example (Cover and Thomas, 1991) of a variable having 8 possible states {a, b, c, d, e, f, g, h} for which the respective probabilities are given by
1
1
1
1
1
, 64
, 64
, 64
, 64
). The entropy in this case is given by( 12 , 14 , 18 , 16
1 1
1 1
1
1
1
1
4
1
log2
−
log2
= 2 bits.
H[x] = − log2 − log2 − log2 −
2
2 4
4 8
8 16
16 64
64
We see that the nonuniform distribution has a smaller entropy than the uniform one,and we shall gain some insight into this shortly when we discuss the interpretation ofentropy in terms of disorder. For the moment, let us consider how we would transmit
the identity of the variable’s state to a receiver. We could do this, as before, using
a 3-bit number. However, we can take advantage of the nonuniform distribution byusing shorter codes for the more probable events, at the expense of longer codes forthe less probable events, in the hope of getting a shorter average code length. Thiscan be done by representing the states {a, b, c, d, e, f, g, h} using, for instance, thefollowing set of code strings: 0, 10, 110, 1110, 111100, 111101, 111110, 111111.
The average length of the code that has to be transmitted is thenaverage code length =

1
1
1
1
1
×1+ ×2+ ×3+
×4+4×× 6 = 2 bits
2
4
8
16
64

which again is the same as the entropy of the random variable. Note that shorter code
strings cannot be used because it must be possible to disambiguate a concatenationof such strings into its component parts. For instance, 11001110 decodes uniquely
into the state sequence c, a, d.
This relation between entropy and shortest coding length is a general one. The
noiseless coding theorem (Shannon, 1948) states that the entropy is a lower boundon the number of bits needed to transmit the state of a random variable.
From now on, we shall switch to the use of natural logarithms in deﬁning entropy, as this will provide a more convenient link with ideas elsewhere in this book.
In this case, the entropy is measured in units of ‘nats’ instead of bits, which differ
simply by a factor of ln 2.
We have introduced the concept of entropy in terms of the average amount of
information needed to specify the state of a random variable. In fact, the concept ofentropy has much earlier origins in physics where it was introduced in the context
of equilibrium thermodynamics and later given a deeper interpretation as a measureof disorder through developments in statistical mechanics. We can understand thisalternative view of entropy by considering a set of N identical objects that are to be
divided amongst a set of bins, such that there are ni objects in the ith bin. Consider

1.6. Information Theory

51

the number of different ways of allocating the objects to the bins. There are N
ways to choose the ﬁrst object, (N − 1) ways to choose the second object, andso on, leading to a total of N ! ways to allocate all N objects to the bins, where N !
(pronounced ‘factorial N ’) denotes the product N × (N − 1) × · · · × 2 × 1. However,
we don’t wish to distinguish between rearrangements of objects within each bin. In
the ith bin there are ni ! ways of reordering the objects, and so the total number ofways of allocating the N objects to the bins is given byN!
W =
i ni !

(1.94)

which is called the multiplicity. The entropy is then deﬁned as the logarithm of themultiplicity scaled by an appropriate constant
H=

1
1 
1
ln ni !.
ln W =
ln N ! −NN
N(1.95)i

We now consider the limit N → ∞, in which the fractions ni /N are held ﬁxed, andapply Stirling’s approximationln N !  N ln N − N
which gives
H = − limN →∞

  ni 
i

N

ln

n 
i

N

=−

(1.96)

pi ln pi

(1.97)i
where we have used i ni = N . Here pi = limN →∞ (ni /N ) is the probabilityof an object being assigned to the ith bin. In physics terminology, the speciﬁc arrangements of objects in the bins is called a microstate, and the overall distributionof occupation numbers, expressed through the ratios ni /N , is called a macrostate.
The multiplicity W is also known as the weight of the macrostate.
We can interpret the bins as the states xi of a discrete random variable X, wherep(X = xi ) = pi . The entropy of the random variable X is thenH[p] = −p(xi ) ln p(xi ).
(1.98)i

Appendix E

Distributions p(xi ) that are sharply peaked around a few values will have a relatively
low entropy, whereas those that are spread more evenly across many values will
have higher entropy, as illustrated in Figure 1.30. Because 0  pi  1, the entropy
is nonnegative, and it will equal its minimum value of 0 when one of the pi =1 and all other pj=i = 0. The maximum entropy conﬁguration can be found by
maximizing H using a Lagrange multiplier to enforce the normalization constraint
on the probabilities. Thus we maximize




 =−
H
p(xi ) ln p(xi ) + λp(xi ) − 1(1.99)ii

52

1. INTRODUCTION
0.5

0.5

H = 3.09
probabilities

probabilities

H = 1.77

0.25

0

0.25

0

Figure 1.30 Histograms of two probability distributions over 30 bins illustrating the higher value of the entropy
H for the broader distribution. The largest entropy would arise from a uniform distribution that would give H =− ln(1/30) = 3.40.

Exercise 1.29

from which we ﬁnd that all of the p(xi ) are equal and are given by p(xi ) = 1/M
where M is the total number of states xi . The corresponding value of the entropy
is then H = ln M . This result can also be derived from Jensen’s inequality (to be
discussed shortly). To verify that the stationary point is indeed a maximum, we canevaluate the second derivative of the entropy, which gives∂H1= −Iij∂p(xi )∂p(xj )pi(1.100)where Iij are the elements of the identity matrix.
We can extend the deﬁnition of entropy to include distributions p(x) over continuous variables x as follows. First divide x into bins of width ∆. Then, assumingp(x) is continuous, the mean value theorem (Weisstein, 1999) tells us that, for eachsuch bin, there must exist a value xi such that
 (i+1)∆p(x) dx = p(xi )∆.
(1.101)i∆

We can now quantize the continuous variable x by assigning any value x to the value
xi whenever x falls in the ith bin. The probability of observing the value xi is thenp(xi )∆. This gives a discrete distribution for which the entropy takes the formp(xi )∆ ln (p(xi )∆) = −p(xi )∆ ln p(xi ) − ln ∆(1.102)H∆ = −iiwhere we have used i p(xi )∆ = 1, which follows from (1.101). We now omit
the second term − ln ∆ on the right-hand side of (1.102) and then consider the limit1.6. Information Theory53∆ → 0. The ﬁrst term on the right-hand side of (1.102) will approach the integral ofp(x) ln p(x) in this limit so that




lim
p(xi )∆ ln p(xi ) = − p(x) ln p(x) dx(1.103)∆→0iwhere the quantity on the right-hand side is called the differential entropy. We seethat the discrete and continuous forms of the entropy differ by a quantity ln ∆, which
diverges in the limit ∆ → 0. This reﬂects the fact that to specify a continuous
variable very precisely requires a large number of bits. For a density deﬁned overmultiple continuous variables, denoted collectively by the vector x, the differential
entropy is given by
H[x] = − p(x) ln p(x) dx.
(1.104)In the case of discrete distributions, we saw that the maximum entropy conﬁguration corresponded to an equal distribution of probabilities across the possible
states of the variable. Let us now consider the maximum entropy conﬁguration for
a continuous variable. In order for this maximum to be well deﬁned, it will be necessary to constrain the ﬁrst and second moments of p(x) as well as preserving thenormalization constraint. We therefore maximize the differential entropy with the

Ludwig Boltzmann
1844–1906
Ludwig Eduard Boltzmann was anAustrian physicist who created theﬁeld of statistical mechanics. Prior
to Boltzmann, the concept of entropy was already known fromclassical thermodynamics where it
quantiﬁes the fact that when we take energy from asystem, not all of that energy is typically available
to do useful work. Boltzmann showed that the thermodynamic entropy S, a macroscopic quantity, could
be related to the statistical properties at the microscopic level. This is expressed through the famous
equation S = k ln W in which W represents thenumber of possible microstates in a macrostate, andk  1.38 × 10−23 (in units of Joules per Kelvin) is
known as Boltzmann’s constant. Boltzmann’s ideas
were disputed by many scientists of they day. One difﬁculty they saw arose from the second law of thermo-

dynamics, which states that the entropy of a closed
system tends to increase with time. By contrast, at
the microscopic level the classical Newtonian equations of physics are reversible, and so they found it
difﬁcult to see how the latter could explain the former. They didn’t fully appreciate Boltzmann’s arguments, which were statistical in nature and which concluded not that entropy could never decrease overtime but simply that with overwhelming probability it
would generally increase. Boltzmann even had a longrunning dispute with the editor of the leading German
physics journal who refused to let him refer to atoms
and molecules as anything other than convenient theoretical constructs. The continued attacks on his work
lead to bouts of depression, and eventually he committed suicide. Shortly after Boltzmann’s death, newexperiments by Perrin on colloidal suspensions veriﬁed his theories and conﬁrmed the value of the Boltzmann constant. The equation S = k ln W is carved on
Boltzmann’s tombstone.

54

1. INTRODUCTION
three constraints

 ∞p(x) dx = 1(1.105)xp(x) dx = µ(1.106)

−∞
 ∞
−∞

 ∞
−∞

Appendix E(x − µ)2 p(x) dx = σ 2 .

The constrained maximization can be performed using Lagrange multipliers so that
we maximize the following functional with respect to p(x)
 ∞

 ∞
−
p(x) ln p(x) dx + λ1
p(x) dx − 1
−∞
−∞
 ∞

 ∞

2
2
+λ2
xp(x) dx − µ + λ3
(x − µ) p(x) dx − σ .
−∞Appendix D

Exercise 1.34−∞

Using the calculus of variations, we set the derivative of this functional to zero giving
(1.108)p(x) = exp −1 + λ1 + λ2 x + λ3 (x − µ)2 .
The Lagrange multipliers can be found by back substitution of this result into thethree constraint equations, leading ﬁnally to the resultp(x) =

Exercise 1.35

(1.107)(x − µ)2
1
exp
−
2σ 2
(2πσ 2 )1/2

(1.109)and so the distribution that maximizes the differential entropy is the Gaussian. Notethat we did not constrain the distribution to be nonnegative when we maximized theentropy. However, because the resulting distribution is indeed nonnegative, we seewith hindsight that such a constraint is not necessary.
If we evaluate the differential entropy of the Gaussian, we obtain
H[x] =

1
1 + ln(2πσ 2 ) .
2(1.110)Thus we see again that the entropy increases as the distribution becomes broader,i.e., as σ 2 increases. This result also shows that the differential entropy, unlike thediscrete entropy, can be negative, because H(x) < 0 in (1.110) for σ 2 < 1/(2πe).
Suppose we have a joint distribution p(x, y) from which we draw pairs of valuesof x and y. If a value of x is already known, then the additional information needed
to specify the corresponding value of y is given by − ln p(y|x). Thus the average
additional information needed to specify y can be written asH[y|x] = −p(y, x) ln p(y|x) dy dx(1.111)

1.6. Information Theory

Exercise 1.37

55

which is called the conditional entropy of y given x. It is easily seen, using the
product rule, that the conditional entropy satisﬁes the relation
H[x, y] = H[y|x] + H[x](1.112)where H[x, y] is the differential entropy of p(x, y) and H[x] is the differential entropy of the marginal distribution p(x). Thus the information needed to describe x
and y is given by the sum of the information needed to describe x alone plus the
additional information required to specify y given x.

1.6.1 Relative entropy and mutual information
So far in this section, we have introduced a number of concepts from information
theory, including the key notion of entropy. We now start to relate these ideas topattern recognition. Consider some unknown distribution p(x), and suppose that
we have modelled this using an approximating distribution q(x). If we use q(x) toconstruct a coding scheme for the purpose of transmitting values of x to a receiver,then the average additional amount of information (in nats) required to specify thevalue of x (assuming we choose an efﬁcient coding scheme) as a result of using q(x)instead of the true distribution p(x) is given by KL(pq) = − p(x) ln q(x) dx − − p(x) ln p(x) dx
q(x)= − p(x) ln
dx.
(1.113)p(x)This is known as the relative entropy or Kullback-Leibler divergence, or KL divergence (Kullback and Leibler, 1951), between the distributions p(x) and q(x). Notethat it is not a symmetrical quantity, that is to say KL(pq) ≡ KL(qp).
We now show that the Kullback-Leibler divergence satisﬁes KL(pq)  0 withequality if, and only if, p(x) = q(x). To do this we ﬁrst introduce the concept of
convex functions. A function f (x) is said to be convex if it has the property thatevery chord lies on or above the function, as shown in Figure 1.31. Any value of x
in the interval from x = a to x = b can be written in the form λa + (1 − λ)b where0  λ  1. The corresponding point on the chord is given by λf (a) + (1 − λ)f (b),Claude Shannon
1916–2001
After graduating from Michigan andMIT, Shannon joined the AT&T Bell
Telephone laboratories in 1941. His
paper ‘A Mathematical Theory ofCommunication’ published in the
Bell System Technical Journal in1948 laid the foundations for modern information the-ory. This paper introduced the word ‘bit’, and his concept that information could be sent as a stream of 1s
and 0s paved the way for the communications revolution. It is said that von Neumann recommended to
Shannon that he use the term entropy, not only because of its similarity to the quantity used in physics,
but also because “nobody knows what entropy reallyis, so in any discussion you will always have an advantage”.

56

1. INTRODUCTION
Figure 1.31

A convex function f (x) is one for which every chord (shown in blue) lies on or above
the function (shown in red).

f (x)
chord

a

xλ

b

x

and the corresponding value of the function is f (λa + (1 − λ)b). Convexity thenimpliesf (λa + (1 − λ)b)  λf (a) + (1 − λ)f (b).
(1.114)
Exercise 1.36

Exercise 1.38

This is equivalent to the requirement that the second derivative of the function be
everywhere positive. Examples of convex functions are x ln x (for x > 0) and x2 . A
function is called strictly convex if the equality is satisﬁed only for λ = 0 and λ = 1.
If a function has the opposite property, namely that every chord lies on or below thefunction, it is called concave, with a corresponding deﬁnition for strictly concave. If
a function f (x) is convex, then −f (x) will be concave.
Using the technique of proof by induction, we can show from (1.114) that a
convex function f (x) satisﬁes
M

M


λi xi λi f (xi )(1.115)
f



i=1

i=1

where λi  0 and i λi = 1, for any set of points {xi }. The result (1.115) is
known as Jensen’s inequality. If we interpret the λi as the probability distribution
over a discrete variable x taking the values {xi }, then (1.115) can be writtenf (E[x])  E[f (x)](1.116)

where E[·] denotes the expectation. For continuous variables, Jensen’s inequality
takes the form

 
f

xp(x) dxf (x)p(x) dx.

(1.117)

We can apply Jensen’s inequality in the form (1.117) to the Kullback-Leibler
divergence (1.113) to giveq(x)dx  − ln q(x) dx = 0(1.118)KL(pq) = − p(x) lnp(x)1.6. Information Theory

57

where we have used the fact that − ln x is a convex function, together with the normalization condition q(x) dx = 1. In fact, − ln x is a strictly convex function,
so the equality will hold if, and only if, q(x) = p(x) for all x. Thus we can interpret the Kullback-Leibler divergence as a measure of the dissimilarity of the two
distributions p(x) and q(x).
We see that there is an intimate relationship between data compression and density estimation (i.e., the problem of modelling an unknown probability distribution)
because the most efﬁcient compression is achieved when we know the true distribution. If we use a distribution that is different from the true one, then we mustnecessarily have a less efﬁcient coding, and on average the additional informationthat must be transmitted is (at least) equal to the Kullback-Leibler divergence between the two distributions.
Suppose that data is being generated from an unknown distribution p(x) that wewish to model. We can try to approximate this distribution using some parametric
distribution q(x|θ), governed by a set of adjustable parameters θ, for example a
multivariate Gaussian. One way to determine θ is to minimize the Kullback-Leibler
divergence between p(x) and q(x|θ) with respect to θ. We cannot do this directly
because we don’t know p(x). Suppose, however, that we have observed a ﬁnite setof training points xn , for n = 1, . . . , N , drawn from p(x). Then the expectation
with respect to p(x) can be approximated by a ﬁnite sum over these points, using(1.35), so thatN{− ln q(xn |θ) + ln p(xn )} .
(1.119)KL(pq) 
n=1

The second term on the right-hand side of (1.119) is independent of θ, and the ﬁrst
term is the negative log likelihood function for θ under the distribution q(x|θ) evaluated using the training set. Thus we see that minimizing this Kullback-Leibler
divergence is equivalent to maximizing the likelihood function.
Now consider the joint distribution between two sets of variables x and y givenby p(x, y). If the sets of variables are independent, then their joint distribution will
factorize into the product of their marginals p(x, y) = p(x)p(y). If the variables are
not independent, we can gain some idea of whether they are ‘close’ to being independent by considering the Kullback-Leibler divergence between the joint distribution
and the product of the marginals, given by
I[x, y] ≡ KL(p(x, y)p(x)p(y))

p(x)p(y)p(x, y) ln
dx dy
= −
p(x, y)Exercise 1.41

(1.120)which is called the mutual information between the variables x and y. From theproperties of the Kullback-Leibler divergence, we see that I(x, y)  0 with equality if, and only if, x and y are independent. Using the sum and product rules ofprobability, we see that the mutual information is related to the conditional entropy
throughI[x, y] = H[x] − H[x|y] = H[y] − H[y|x].
(1.121)

58

1. INTRODUCTION
Thus we can view the mutual information as the reduction in the uncertainty about x
by virtue of being told the value of y (or vice versa). From a Bayesian perspective,
we can view p(x) as the prior distribution for x and p(x|y) as the posterior distribution after we have observed new data y. The mutual information therefore represents
the reduction in uncertainty about x as a consequence of the new observation y.

Exercises

1.1 () www Consider the sum-of-squares error function given by (1.2) in which
the function y(x, w) is given by the polynomial (1.1). Show that the coefﬁcientsw = {wi } that minimize this error function are given by the solution to the following
set of linear equationsM
Aij wj = Ti
(1.122)
j =0

where
Aij =

N

n=1

i+ j

(xn )

,

Ti =

N


(xn )i tn .

(1.123)n=1

Here a sufﬁx i or j denotes the index of a component, whereas (x)i denotes x raised
to the power of i.
1.2 () Write down the set of coupled linear equations, analogous to (1.122), satisﬁedby the coefﬁcients wi which minimize the regularized sum-of-squares error function
given by (1.4).
1.3 ( ) Suppose that we have three coloured boxes r (red), b (blue), and g (green).
Box r contains 3 apples, 4 oranges, and 3 limes, box b contains 1 apple, 1 orange,
and 0 limes, and box g contains 3 apples, 3 oranges, and 4 limes. If a box is chosen
at random with probabilities p(r) = 0.2, p(b) = 0.2, p(g) = 0.6, and a piece offruit is removed from the box (with equal probability of selecting any of the items inthe box), then what is the probability of selecting an apple? If we observe that the
selected fruit is in fact an orange, what is the probability that it came from the greenbox?
1.4 ( ) www Consider a probability density px (x) deﬁned over a continuous variable x, and suppose that we make a nonlinear change of variable using x = g(y),so that the density transforms according to (1.27). By differentiating (1.27), showthat the location 
y of the maximum of the density in y is not in general related to thelocation 
x of the maximum of the density over x by the simple functional relationx = g(y ) as a consequence of the Jacobian factor. This shows that the maximumof a probability density (in contrast to a simple function) is dependent on the choiceof variable. Verify that, in the case of a linear transformation, the location of the
maximum transforms in the same way as the variable itself.
1.5 () Using the deﬁnition (1.38) show that var[f (x)] satisﬁes (1.39).

Exercises59

1.6 () Show that if two variables x and y are independent, then their covariance iszero.
1.7 ( ) www In this exercise, we prove the normalization condition (1.48) for theunivariate Gaussian. To do this consider, the integral


 ∞
1 2
I=
exp − 2 x
dx
(1.124)2σ−∞which we can evaluate by ﬁrst writing its square in the form
 ∞ ∞
1
1
I2 =exp − 2 x2 − 2 y 2 dx dy.
2σ
2σ
−∞ −∞

(1.125)

Now make the transformation from Cartesian coordinates (x, y) to polar coordinates(r, θ) and then substitute u = r2 . Show that, by performing the integrals over θ andu, and then taking the square root of both sides, we obtain
I = 2πσ 21 /2.

(1.126)

Finally, use this result to show that the Gaussian distribution N (x|µ, σ 2 ) is normalized.
1.8 ( ) www By using a change of variables, verify that the univariate Gaussian
distribution given by (1.46) satisﬁes (1.49). Next, by differentiating both sides of thenormalization condition ∞N x|µ, σ 2 dx = 1(1.127)−∞

with respect to σ 2 , verify that the Gaussian satisﬁes (1.50). Finally, show that (1.51)
holds.
1.9 () www Show that the mode (i.e. the maximum) of the Gaussian distribution(1.46) is given by µ. Similarly, show that the mode of the multivariate Gaussian(1.52) is given by µ.
1.10 () www Suppose that the two variables x and z are statistically independent.
Show that the mean and variance of their sum satisﬁes
E[x + z] = E[x] + E[z]var[x + z] = var[x] + var[z].

(1.128)(1.129)1.11 () By setting the derivatives of the log likelihood function (1.54) with respect to µ
and σ 2 equal to zero, verify the results (1.55) and (1.56).

60

1. INTRODUCTION
1.12 ( ) www

Using the results (1.49) and (1.50), show thatE[xn xm ] = µ2 + Inm σ 2(1.130)where xn and xm denote data points sampled from a Gaussian distribution with mean
µ and variance σ 2 , and Inm satisﬁes Inm = 1 if n = m and Inm = 0 otherwise.
Hence prove the results (1.57) and (1.58).
1.13 () Suppose that the variance of a Gaussian is estimated using the result (1.56) but
with the maximum likelihood estimate µML replaced with the true value µ of themean. Show that this estimator has the property that its expectation is given by thetrue variance σ 2 .
1.14 ( ) Show that an arbitrary square matrix with elements wij can be written in
S
A
S
A
+ wij
where wij
and wij
are symmetric and anti-symmetric
the form wij = wijSSAAmatrices, respectively, satisfying wij = wji and wij= −wjifor all i and j. Now
consider the second order term in a higher order polynomial in D dimensions, given
by
D 
D

wij xi xj .
(1.131)i=1 j =1

Show thatD Dwij xi xj =i=1 j =1

D 
D


S
wij
xi xj

(1.132)i=1 j =1

so that the contribution from the anti-symmetric matrix vanishes. We therefore see
that, without loss of generality, the matrix of coefﬁcients wij can be chosen to besymmetric, and so not all of the D2 elements of this matrix can be chosen indepenS
is given
dently. Show that the number of independent parameters in the matrix wij
by D(D + 1)/2.
1.15 (  ) www In this exercise and the next, we explore how the number of independent parameters in a polynomial grows with the order M of the polynomial and withthe dimensionality D of the input space. We start by writing down the M th orderterm for a polynomial in D dimensions in the form
D 
D


···

i1 =1 i2 =1D

wi1 i2 ···iM xi1 xi2 · · · xiM .

(1.133)iM =1

The coefﬁcients wi1 i2 ···iM comprise DM elements, but the number of independent
parameters is signiﬁcantly fewer due to the many interchange symmetries of thefactor xi1 xi2 · · · xiM . Begin by showing that the redundancy in the coefﬁcients canbe removed by rewriting this M th order term in the formi1D i1 =1 i2 =1

iM − 1

···



iM =1

w
 i1 i2 ···iM xi1 xi2 · · · xiM .

(1.134)

Exercises61 coefﬁcients and w coefﬁcients need
Note that the precise relationship between the w
not be made explicit. Use this result to show that the number of independent parameters n(D, M ), which appear at order M , satisﬁes the following recursion relationn(D, M ) =Dn(i, M − 1).

(1.135)i=1

Next use proof by induction to show that the following result holdsD
i=1(i + M − 2)!
(D + M − 1)!
=(i − 1)! (M − 1)!
(D − 1)! M !

(1.136)

which can be done by ﬁrst proving the result for D = 1 and arbitrary M by making
use of the result 0! = 1, then assuming it is correct for dimension D and verifyingthat it is correct for dimension D + 1. Finally, use the two previous results, together
with proof by induction, to shown(D, M ) =(D + M − 1)!
.
(D − 1)! M !

(1.137)

To do this, ﬁrst show that the result is true for M = 2, and any value of D  1,
by comparison with the result of Exercise 1.14. Then make use of (1.135), together
with (1.136), to show that, if the result holds at order M − 1, then it will also hold atorder M
1.16 (  ) In Exercise 1.15, we proved the result (1.135) for the number of independent
parameters in the M th order term of a D-dimensional polynomial. We now ﬁnd an
expression for the total number N (D, M ) of independent parameters in all of theterms up to and including the M 6th order. First show that N (D, M ) satisﬁes
N (D, M ) =M
n(D, m)(1.138)m=0where n(D, m) is the number of independent parameters in the term of order m.
Now make use of the result (1.137), together with proof by induction, to show that(D + M )!
.
(1.139)D! M !
This can be done by ﬁrst proving that the result holds for M = 0 and arbitraryD  1, then assuming that it holds at order M , and hence showing that it holds atorder M + 1. Finally, make use of Stirling’s approximation in the form
N (d, M ) =n!  nn e−n(1.140)for large n to show that, for D
M , the quantity N (D, M ) grows like DM ,D
and for M
D it grows like M . Consider a cubic (M = 3) polynomial in D
dimensions, and evaluate numerically the total number of independent parametersfor (i) D = 10 and (ii) D = 100, which correspond to typical small-scale andmedium-scale machine learning applications.

62

1. INTRODUCTION
1.17 ( ) www

The gamma function is deﬁned by ∞ux−1 e−u du.
Γ(x) ≡(1.141)0

Using integration by parts, prove the relation Γ(x + 1) = xΓ(x). Show also that
Γ(1) = 1 and hence that Γ(x + 1) = x! when x is an integer.
1.18 ( ) www We can use the result (1.126) to derive an expression for the surface
area SD , and the volume VD , of a sphere of unit radius in D dimensions. To do this,consider the following result, which is obtained by transforming from Cartesian topolar coordinates
D  ∞

i=1

−∞

−x2i

e

 ∞
dxi = SDe−r rD−1 dr.
2(1.142)0

Using the deﬁnition (1.141) of the Gamma function, together with (1.126), evaluate
both sides of this equation, and hence show thatSD =2π D/2.
Γ(D/2)(1.143)

Next, by integrating with respect to radius from 0 to 1, show that the volume of theunit sphere in D dimensions is given by
VD =

SD
.
D

(1.144)√
Finally, use the results Γ(1) = 1 and Γ(3/2) = π/2 to show that (1.143) and(1.144) reduce to the usual expressions for D = 2 and D = 3.
1.19 ( ) Consider a sphere of radius a in D-dimensions together with the concentric
hypercube of side 2a, so that the sphere touches the hypercube at the centres of eachof its sides. By using the results of Exercise 1.18, show that the ratio of the volumeof the sphere to the volume of the cube is given by
π D/2
volume of sphere=.
D−
volume of cube
D2 1 Γ(D/2)(1.145)

Now make use of Stirling’s formula in the form
Γ(x + 1)  (2π)1/2 e−x xx+1/2(1.146)

which is valid for x
1, to show that, as D → ∞, the ratio (1.145) goes to zero.
Show also that the ratio of the distance from the centre of the hypercube√ to one of
the corners, divided by the perpendicular distance to one of the sides, is D, which
therefore goes to ∞ as D → ∞. From these results we see that, in a space of highdimensionality, most of the volume of a cube is concentrated in the large number ofcorners, which themselves become very long ‘spikes’!

Exercises

63

1.20 ( ) www In this exercise, we explore the behaviour of the Gaussian distribution
in high-dimensional spaces. Consider a Gaussian distribution in D dimensions given
by


1
x2
p(x) =
exp−.
(1.147)2σ 2(2πσ 2 )D/2
We wish to ﬁnd the density with respect to radius in polar coordinates in which thedirection variables have been integrated out. To do this, show that the integral of
the probability density over a thin shell of radius r and thickness , where
1, isgiven by p(r) where
SD rD−1r2p(r) =
exp − 2
(1.148)2σ(2πσ 2 )D/2
where SD is the surface area of a unit sphere in D dimensions. √Show that the functionr  Dσ. By consideringp(r) has a single stationary point located, for large D, at r + ) wherer, show that for large D,
p(


3 2
p(
r + ) = p(r) exp − 2(1.149)2σr is a maximum of the radial probability density and also that p(r)which shows that 
r with length scale σ. We havedecays exponentially away from its maximum at 
already seen that σr for large D, and so we see that most of the probability
mass is concentrated in a thin shell at large radius. Finally, show that the probability
r by a factor of exp(D/2).
density p(x) is larger at the origin than at the radius 
We therefore see that most of the probability mass in a high-dimensional Gaussian
distribution is located at a different radius from the region of high probability density.
This property of distributions in spaces of high dimensionality will have importantconsequences when we consider Bayesian inference of model parameters in later
chapters.
1.21 ( ) Consider two nonnegative numbers a and b, and show that, if a  b, thena  (ab)1/2 . Use this result to show that, if the decision regions of a two-classclassiﬁcation problem are chosen to minimize the probability of misclassiﬁcation,
this probability will satisfy

1 /2
dx.
(1.150)p(mistake)  {p(x, C1 )p(x, C2 )}1.22 () www Given a loss matrix with elements Lkj , the expected risk is minimizedif, for each x, we choose the class that minimizes (1.81). Verify that, when theloss matrix is given by Lkj = 1 − Ikj , where Ikj are the elements of the identity
matrix, this reduces to the criterion of choosing the class having the largest posterior
probability. What is the interpretation of this form of loss matrix?
1.23 () Derive the criterion for minimizing the expected loss when there is a generalloss matrix and general prior probabilities for the classes.

64

1. INTRODUCTION
1.24 ( ) www Consider a classiﬁcation problem in which the loss incurred when
an input vector from class Ck is classiﬁed as belonging to class Cj is given by theloss matrix Lkj , and for which the loss incurred in selecting the reject option is λ.
Find the decision criterion that will give the minimum expected loss. Verify that thisreduces to the reject criterion discussed in Section 1.5.3 when the loss matrix is given
by Lkj = 1 − Ikj . What is the relationship between λ and the rejection threshold θ?
1.25 () www Consider the generalization of the squared loss function (1.87) for asingle target variable t to the case of multiple target variables described by the vectort given byy(x) − t2 p(x, t) dx dt.
(1.151)
E[L(t, y(x))] =Using the calculus of variations, show that the function y(x) for which this expected
loss is minimized is given by y(x) = Et [t|x]. Show that this result reduces to (1.89)
for the case of a single target variable t.
1.26 () By expansion of the square in (1.151), derive a result analogous to (1.90) andhence show that the function y(x) that minimizes the expected squared loss for thecase of a vector t of target variables is again given by the conditional expectation oft.
1.27 ( ) www Consider the expected loss for regression problems under the Lq loss
function given by (1.91). Write down the condition that y(x) must satisfy in orderto minimize E[Lq ]. Show that, for q = 1, this solution represents the conditional
median, i.e., the function y(x) such that the probability mass for t < y(x) is thesame as for t  y(x). Also show that the minimum expected Lq loss for q → 0 is
given by the conditional mode, i.e., by the function y(x) equal to the value of t thatmaximizes p(t|x) for each x.
1.28 () In Section 1.6, we introduced the idea of entropy h(x) as the information gainedon observing the value of a random variable x having distribution p(x). We saw
that, for independent variables x and y for which p(x, y) = p(x)p(y), the entropy
functions are additive, so that h(x, y) = h(x) + h(y). In this exercise, we derive therelation between h and p in the form of a function h(p). First show that h(p2 ) =2h(p), and hence by induction that h(pn ) = nh(p) where n is a positive integer.
Hence show that h(pn/m ) = (n/m)h(p) where m is also a positive integer. Thisimplies that h(px ) = xh(p) where x is a positive rational number, and hence bycontinuity when it is a positive real number. Finally, show that this implies h(p)must take the form h(p) ∝ ln p.
1.29 () www Consider an M -state discrete random variable x, and use Jensen’s inequality in the form (1.115) to show that the entropy of its distribution p(x) satisﬁes
H[x]  ln M .
1.30 ( ) Evaluate the Kullback-Leibler divergence (1.113) between two Gaussians
p(x) = N (x|µ, σ 2 ) and q(x) = N (x|m, s2 ).

Exercises
Table 1.3

The joint distribution p(x, y) for two binary variables
x and y used in Exercise 1.39.

65

y
x

0
1

0
1/3
0

1
1/3
1/3

1.31 ( ) www Consider two variables x and y having joint distribution p(x, y). Show
that the differential entropy of this pair of variables satisﬁes
H[x, y]  H[x] + H[y](1.152)

with equality if, and only if, x and y are statistically independent.
1.32 () Consider a vector x of continuous variables with distribution p(x) and corresponding entropy H[x]. Suppose that we make a nonsingular linear transformationof x to obtain a new variable y = Ax. Show that the corresponding entropy is given
by H[y] = H[x] + ln |A| where |A| denotes the determinant of A.
1.33 ( ) Suppose that the conditional entropy H[y|x] between two discrete randomvariables x and y is zero. Show that, for all values of x such that p(x) > 0, thevariable y must be a function of x, in other words for each x there is only one valueof y such that p(y|x) = 0.
1.34 ( ) www Use the calculus of variations to show that the stationary point of thefunctional (1.108) is given by (1.108). Then use the constraints (1.105), (1.106),and (1.107) to eliminate the Lagrange multipliers and hence show that the maximum
entropy solution is given by the Gaussian (1.109).
1.35 () www Use the results (1.106) and (1.107) to show that the entropy of theunivariate Gaussian (1.109) is given by (1.110).
1.36 () A strictly convex function is deﬁned as one for which every chord lies above
the function. Show that this is equivalent to the condition that the second derivativeof the function be positive.
1.37 () Using the deﬁnition (1.111) together with the product rule of probability, prove
the result (1.112).
1.38 ( ) www Using proof by induction, show that the inequality (1.114) for convex
functions implies the result (1.115).
1.39 (  ) Consider two binary variables x and y having the joint distribution given in
Table 1.3.
Evaluate the following quantities(a) H[x](b) H[y](c) H[y|x](d) H[x|y](e) H[x, y](f) I[x, y].

Draw a diagram to show the relationship between these various quantities.

66

1. INTRODUCTION
1.40 () By applying Jensen’s inequality (1.115) with f (x) = ln x, show that the arithmetic mean of a set of real numbers is never less than their geometrical mean.
1.41 () www Using the sum and product rules of probability, show that the mutual
information I(x, y) satisﬁes the relation (1.121).

2
Probability
Distributions

In Chapter 1, we emphasized the central role played by probability theory in thesolution of pattern recognition problems. We turn now to an exploration of someparticular examples of probability distributions and their properties. As well as being of great interest in their own right, these distributions can form building blocks
for more complex models and will be used extensively throughout the book. The
distributions introduced in this chapter will also serve another important purpose,
namely to provide us with the opportunity to discuss some key statistical concepts,
such as Bayesian inference, in the context of simple models before we encounterthem in more complex situations in later chapters.
One role for the distributions discussed in this chapter is to model the probability distribution p(x) of a random variable x, given a ﬁnite set x1 , . . . , xN ofobservations. This problem is known as density estimation. For the purposes ofthis chapter, we shall assume that the data points are independent and identically
distributed. It should be emphasized that the problem of density estimation is fun-

67

68

2. PROBABILITY DISTRIBUTIONS
damentally ill-posed, because there are inﬁnitely many probability distributions that
could have given rise to the observed ﬁnite data set. Indeed, any distribution p(x)that is nonzero at each of the data points x1 , . . . , xN is a potential candidate. The
issue of choosing an appropriate distribution relates to the problem of model selection that has already been encountered in the context of polynomial curve ﬁtting in
Chapter 1 and that is a central issue in pattern recognition.
We begin by considering the binomial and multinomial distributions for discreterandom variables and the Gaussian distribution for continuous random variables.
These are speciﬁc examples of parametric distributions, so-called because they aregoverned by a small number of adaptive parameters, such as the mean and variance inthe case of a Gaussian for example. To apply such models to the problem of density
estimation, we need a procedure for determining suitable values for the parameters,
given an observed data set. In a frequentist treatment, we choose speciﬁc values
for the parameters by optimizing some criterion, such as the likelihood function. By
contrast, in a Bayesian treatment we introduce prior distributions over the parameters
and then use Bayes’ theorem to compute the corresponding posterior distribution
given the observed data.
We shall see that an important role is played by conjugate priors, that lead toposterior distributions having the same functional form as the prior, and that therefore lead to a greatly simpliﬁed Bayesian analysis. For example, the conjugate priorfor the parameters of the multinomial distribution is called the Dirichlet distribution,
while the conjugate prior for the mean of a Gaussian is another Gaussian. All of these
distributions are examples of the exponential family of distributions, which possessa number of important properties, and which will be discussed in some detail.
One limitation of the parametric approach is that it assumes a speciﬁc functional
form for the distribution, which may turn out to be inappropriate for a particularapplication. An alternative approach is given by nonparametric density estimation
methods in which the form of the distribution typically depends on the size of the data
set. Such models still contain parameters, but these control the model complexity
rather than the form of the distribution. We end this chapter by considering three
nonparametric methods based respectively on histograms, nearest-neighbours, andkernels.

2.1. Binary Variables
We begin by considering a single binary random variable x ∈ {0, 1}. For example,
x might describe the outcome of ﬂipping a coin, with x = 1 representing ‘heads’,
and x = 0 representing ‘tails’. We can imagine that this is a damaged coin so thatthe probability of landing heads is not necessarily the same as that of landing tails.
The probability of x = 1 will be denoted by the parameter µ so thatp(x = 1|µ) = µ(2.1)

2.1. Binary Variables

69

where 0  µ  1, from which it follows that p(x = 0|µ) = 1 − µ. The probability
distribution over x can therefore be written in the form
Bern(x|µ) = µx (1 − µ)1−x
Exercise 2.1(2.2)

which is known as the Bernoulli distribution. It is easily veriﬁed that this distribution
is normalized and that it has mean and variance given byE[x] = µvar[x] = µ(1 − µ).

(2.3)
(2.4)

Now suppose we have a data set D = {x1 , . . . , xN } of observed values of x.
We can construct the likelihood function, which is a function of µ, on the assumptionthat the observations are drawn independently from p(x|µ), so thatp(D|µ) =Np(xn |µ) =

n=1

N


µxn (1 − µ)1−xn .

(2.5)

n=1

In a frequentist setting, we can estimate a value for µ by maximizing the likelihood
function, or equivalently by maximizing the logarithm of the likelihood. In the caseof the Bernoulli distribution, the log likelihood function is given byln p(D|µ) =Nln p(xn |µ) =n=1

Section 2.4N{xn ln µ + (1 − xn ) ln(1 − µ)} .

(2.6)

n=1

At this point, it is worth noting that thelog likelihood function depends on the N
observations xn only through their sum n xn . This sum provides an example of a
sufﬁcient statistic for the data under this distribution, and we shall study the important role of sufﬁcient statistics in some detail. If we set the derivative of ln p(D|µ)
with respect to µ equal to zero, we obtain the maximum likelihood estimator
1 
xn
N
N

µML =

(2.7)

n=1

Jacob Bernoulli
1654–1705
Jacob Bernoulli, also known as
Jacques or James Bernoulli, was a
Swiss mathematician and was theﬁrst of many in the Bernoulli family
to pursue a career in science andmathematics. Although compelled
to study philosophy and theology against his will byhis parents, he travelled extensively after graduatingin order to meet with many of the leading scientists of

his time, including Boyle and Hooke in England. Whenhe returned to Switzerland, he taught mechanics andbecame Professor of Mathematics at Basel in 1687.
Unfortunately, rivalry between Jacob and his younger
brother Johann turned an initially productive collaboration into a bitter and public dispute. Jacob’s most signiﬁcant contributions to mathematics appeared in The
Art of Conjecture published in 1713, eight years after
his death, which deals with topics in probability theory including what has become known as the Bernoulli
distribution.

70

2. PROBABILITY DISTRIBUTIONS
Figure 2.1

Histogram plot of the binomial dis0.3
tribution (2.9) as a function of m for
N = 10 and µ = 0.25.

0.2

0.1

0

0

1

2

3

4

5
m

6

7

8

9

10

which is also known as the sample mean. If we denote the number of observationsof x = 1 (heads) within this data set by m, then we can write (2.7) in the form
µML =

m
N

(2.8)

so that the probability of landing heads is given, in this maximum likelihood framework, by the fraction of observations of heads in the data set.
Now suppose we ﬂip a coin, say, 3 times and happen to observe 3 heads. Then
N = m = 3 and µML = 1. In this case, the maximum likelihood result would
predict that all future observations should give heads. Common sense tells us that
this is unreasonable, and in fact this is an extreme example of the over-ﬁtting associated with maximum likelihood. We shall see shortly how to arrive at more sensible
conclusions through the introduction of a prior distribution over µ.
We can also work out the distribution of the number m of observations of x = 1,
given that the data set has size N . This is called the binomial distribution, andfrom (2.5) we see that it is proportional to µm (1 − µ)N −m . In order to obtain thenormalization coefﬁcient we note that out of N coin ﬂips, we have to add up allof the possible ways of obtaining m heads, so that the binomial distribution can be
written N mµ (1 − µ)N −m(2.9)
Bin(m|N, µ) =m
where

Exercise 2.3

 
N
N!
≡
(N − m)!m!
m(2.10)

is the number of ways of choosing m objects out of a total of N identical objects.
Figure 2.1 shows a plot of the binomial distribution for N = 10 and µ = 0.25.
The mean and variance of the binomial distribution can be found by using theresult of Exercise 1.10, which shows that for independent events the mean of thesum is the sum of the means, and the variance of the sum is the sum of the variances.
Because m = x1 + . . . + xN , and for each observation the mean and variance are2.1. Binary Variables

71

given by (2.3) and (2.4), respectively, we have
E[m] ≡N

mBin(m|N, µ) = N µ(2.11)

2(2.12)

m=0

var[m] ≡N(m − E[m]) Bin(m|N, µ) = N µ(1 − µ).

m=0

Exercise 2.4

These results can also be proved directly using calculus.

2.1.1 The beta distribution
We have seen in (2.8) that the maximum likelihood setting for the parameter µin the Bernoulli distribution, and hence in the binomial distribution, is given by thefraction of the observations in the data set having x = 1. As we have already noted,
this can give severely over-ﬁtted results for small data sets. In order to develop a
Bayesian treatment for this problem, we need to introduce a prior distribution p(µ)
over the parameter µ. Here we consider a form of prior distribution that has a simple
interpretation as well as some useful analytical properties. To motivate this prior,
we note that the likelihood function takes the form of the product of factors of theform µx (1 − µ)1−x . If we choose a prior to be proportional to powers of µ and(1 − µ), then the posterior distribution, which is proportional to the product of theprior and the likelihood function, will have the same functional form as the prior.
This property is called conjugacy and we will see several examples of it later in thischapter. We therefore choose a prior, called the beta distribution, given by
Beta(µ|a, b) =Exercise 2.5

Γ(a + b) a−1µ (1 − µ)b−1
Γ(a)Γ(b)(2.13)

where Γ(x) is the gamma function deﬁned by (1.141), and the coefﬁcient in (2.13)
ensures that the beta distribution is normalized, so that
 1
Beta(µ|a, b) dµ = 1.
(2.14)
0

Exercise 2.6

The mean and variance of the beta distribution are given bya
E[µ] =
a+b
ab
var[µ] =.
2(a + b) (a + b + 1)

(2.15)(2.16)The parameters a and b are often called hyperparameters because they control the
distribution of the parameter µ. Figure 2.2 shows plots of the beta distribution forvarious values of the hyperparameters.
The posterior distribution of µ is now obtained by multiplying the beta prior(2.13) by the binomial likelihood function (2.9) and normalizing. Keeping only thefactors that depend on µ, we see that this posterior distribution has the formp(µ|m, l, a, b) ∝ µm+a−1 (1 − µ)l+b−1(2.17)

72

2. PROBABILITY DISTRIBUTIONS
3

3

a=1

a = 0.1

b=1

b = 0.1
2

2

1

1

0

0

0.5

µ

1

3

0

0

µ

1

0.5

µ

1

3
a=2

a=8

b=3

b=4

2

2

1

1

0

0.5

0

0.5

µ

1

0

0

Figure 2.2 Plots of the beta distribution Beta(µ|a, b) given by (2.13) as a function of µ for various values of thehyperparameters a and b.

where l = N − m, and therefore corresponds to the number of ‘tails’ in the coinexample. We see that (2.17) has the same functional dependence on µ as the priordistribution, reﬂecting the conjugacy properties of the prior with respect to the likelihood function. Indeed, it is simply another beta distribution, and its normalization
coefﬁcient can therefore be obtained by comparison with (2.13) to givep(µ|m, l, a, b) =Γ(m + a + l + b) m+a−1(1 − µ)l+b−1 .
µΓ(m + a)Γ(l + b)

(2.18)We see that the effect of observing a data set of m observations of x = 1 andl observations of x = 0 has been to increase the value of a by m, and the value ofb by l, in going from the prior distribution to the posterior distribution. This allows
us to provide a simple interpretation of the hyperparameters a and b in the prior as
an effective number of observations of x = 1 and x = 0, respectively. Note thata and b need not be integers. Furthermore, the posterior distribution can act as theprior if we subsequently observe additional data. To see this, we can imagine taking
observations one at a time and after each observation updating the current posterior

73

2.1. Binary Variables
2

2

prior

1

0

2

likelihood function

1

0

0.5
µ

1

0

posterior

1

0

0.5
µ

0

1

0

0.5
µ

1

Figure 2.3 Illustration of one step of sequential Bayesian inference. The prior is given by a beta distribution
with parameters a = 2, b = 2, and the likelihood function, given by (2.9) with N = m = 1, corresponds to asingle observation of x = 1, so that the posterior is given by a beta distribution with parameters a = 3, b = 2.

Section 2.3.5

distribution by multiplying by the likelihood function for the new observation andthen normalizing to obtain the new, revised posterior distribution. At each stage, theposterior is a beta distribution with some total number of (prior and actual) observed
values for x = 1 and x = 0 given by the parameters a and b. Incorporation of anadditional observation of x = 1 simply corresponds to incrementing the value of aby 1, whereas for an observation of x = 0 we increment b by 1. Figure 2.3 illustrates
one step in this process.
We see that this sequential approach to learning arises naturally when we adopt
a Bayesian viewpoint. It is independent of the choice of prior and of the likelihood
function and depends only on the assumption of i.i.d. data. Sequential methods makeuse of observations one at a time, or in small batches, and then discard them before
the next observations are used. They can be used, for example, in real-time learning
scenarios where a steady stream of data is arriving, and predictions must be madebefore all of the data is seen. Because they do not require the whole data set to be
stored or loaded into memory, sequential methods are also useful for large data sets.
Maximum likelihood methods can also be cast into a sequential framework.
If our goal is to predict, as best we can, the outcome of the next trial, then wemust evaluate the predictive distribution of x, given the observed data set D. Fromthe sum and product rules of probability, this takes the form
 1
 1
p(x = 1|µ)p(µ|D) dµ =µp(µ|D) dµ = E[µ|D]. (2.19)p(x = 1|D) =0

0

Using the result (2.18) for the posterior distribution p(µ|D), together with the result(2.15) for the mean of the beta distribution, we obtainp(x = 1|D) =m+am+a+l+b(2.20)

which has a simple interpretation as the total fraction of observations (both real observations and ﬁctitious prior observations) that correspond to x = 1. Note that in
the limit of an inﬁnitely large data set m, l → ∞ the result (2.20) reduces to the
maximum likelihood result (2.8). As we shall see, it is a very general property thatthe Bayesian and maximum likelihood results will agree in the limit of an inﬁnitely

74

2. PROBABILITY DISTRIBUTIONS

Exercise 2.7

Exercise 2.8

large data set. For a ﬁnite data set, the posterior mean for µ always lies between theprior mean and the maximum likelihood estimate for µ corresponding to the relative
frequencies of events given by (2.7).
From Figure 2.2, we see that as the number of observations increases, so theposterior distribution becomes more sharply peaked. This can also be seen fromthe result (2.16) for the variance of the beta distribution, in which we see that thevariance goes to zero for a → ∞ or b → ∞. In fact, we might wonder whether it isa general property of Bayesian learning that, as we observe more and more data, theuncertainty represented by the posterior distribution will steadily decrease.
To address this, we can take a frequentist view of Bayesian learning and show
that, on average, such a property does indeed hold. Consider a general Bayesianinference problem for a parameter θ for which we have observed a data set D, described by the joint distribution p(θ, D). The following resultEθ [θ] = ED [Eθ [θ|D]]
where(2.21)

Eθ [θ] ≡p(θ)θ dθ 
ED [Eθ [θ|D]] ≡θp(θ|D) dθ p(D) dD(2.22)(2.23)

says that the posterior mean of θ, averaged over the distribution generating the data,is equal to the prior mean of θ. Similarly, we can show thatvarθ [θ] = ED [varθ [θ|D]] + varD [Eθ [θ|D]] .

(2.24)The term on the left-hand side of (2.24) is the prior variance of θ. On the righthand side, the ﬁrst term is the average posterior variance of θ, and the second termmeasures the variance in the posterior mean of θ. Because this variance is a positive
quantity, this result shows that, on average, the posterior variance of θ is smaller thanthe prior variance. The reduction in variance is greater if the variance in the posteriormean is greater. Note, however, that this result only holds on average, and that for aparticular observed data set it is possible for the posterior variance to be larger thanthe prior variance.

2.2. Multinomial Variables
Binary variables can be used to describe quantities that can take one of two possible
values. Often, however, we encounter discrete variables that can take on one of K
possible mutually exclusive states. Although there are various alternative ways toexpress such variables, we shall see shortly that a particularly convenient representation is the 1-of-K scheme in which the variable is represented by a K-dimensional
vector x in which one of the elements xk equals 1, and all remaining elements equal2.2. Multinomial Variables

75

0. So, for instance if we have a variable that can take K = 6 states and a particularobservation of the variable happens to correspond to the state where x3 = 1, then x
will be represented by(2.25)
x = (0, 0, 1, 0, 0, 0)T .
KNote that such vectors satisfy k=1 xk = 1. If we denote the probability of xk = 1
by the parameter µk , then the distribution of x is givenp(x|µ) =

K


µxkk

(2.26)k=1
T
whereµ = (µ1 , . . . , µK ) , and the parameters µk are constrained to satisfy µk  0
and k µk = 1, because they represent probabilities. The distribution (2.26) can be
regarded as a generalization of the Bernoulli distribution to more than two outcomes.
It is easily seen that the distribution is normalizedp(x|µ) =x

and thatE[x|µ] =Kµk = 1(2.27)

k=1

p(x|µ)x = (µ1 , . . . , µM )T = µ.

(2.28)x

Now consider a data set D of N independent observations x1 , . . . , xN . The
corresponding likelihood function takes the formp(D|µ) =N K
n=1 k=1µxknk =KKP
( n xnk )  mk
µk
=
µk .
k=1

(2.29)

k=1

We see that the likelihood function depends on the N data points only through theK quantities
xnk(2.30)
mk =nSection 2.4

Appendix E

which represent the number of observations of xk = 1. These are called the sufﬁcient
statistics for this distribution.
In order to ﬁnd the maximum likelihood solution for µ, we need to maximize
ln p(D|µ) with respect to µk taking account of the constraint that the µk must sum
to one. This can be achieved using a Lagrange multiplier λ and maximizing
K

K


mk ln µk + λµk − 1 .
(2.31)
k=1

k=1

Setting the derivative of (2.31) with respect to µk to zero, we obtainµk = −mk /λ.

(2.32)

76

2. PROBABILITY DISTRIBUTIONS
We
 can solve for the Lagrange multiplier λ by substituting (2.32) into the constraint
k µk = 1 to give λ = −N . Thus we obtain the maximum likelihood solution in
the form
mk=(2.33)µMLkN
which is the fraction of the N observations for which xk = 1.
We can consider the joint distribution of the quantities m1 , . . . , mK , conditionedon the parameters µ and on the total number N of observations. From (2.29) thistakes the form


K
N
k
Mult(m1 , m2 , . . . , mK |µ, N ) =µm(2.34)k
m1 m2 . . . mK
k=1

which is known as the multinomial distribution. The normalization coefﬁcient is thenumber of ways of partitioning N objects into K groups of size m1 , . . . , mK and isgiven by
NN!
.
(2.35)=
m1 m2 . . . mK
m1 !m2 ! . . . mK !
Note that the variables mk are subject to the constraintK

mk = N.

(2.36)

k=1

2.2.1 The Dirichlet distribution
We now introduce a family of prior distributions for the parameters {µk } ofthe multinomial distribution (2.34). By inspection of the form of the multinomial
distribution, we see that the conjugate prior is given byp(µ|α) ∝

Exercise 2.9

K


k −1
µα
k

(2.37)

k=1

where 0  µk  1 and k µk = 1. Here α1 , . . . , αK are the parameters of the
distribution, and α denotes (α1 , . . . , αK )T . Note that, because of the summation
constraint, the distribution over the space of the {µk } is conﬁned to a simplex ofdimensionality K − 1, as illustrated for K = 3 in Figure 2.4.
The normalized form for this distribution is byΓ(α0 )k −1µα
Dir(µ|α) =k
Γ(α1 ) · · · Γ(αK )K(2.38)

k=1

which is called the Dirichlet distribution. Here Γ(x) is the gamma function deﬁnedby (1.141) while
K

α0 =
αk .
(2.39)
k=1

2.2. Multinomial Variables
Figure 2.4

77µ2

The Dirichlet distribution over three variables µ1 , µ2 , µ3
is conﬁned to a simplex (a bounded linear manifold) of
the form shown,Pas a consequence of the constraints
0  µk  1 and k µk = 1.

µ1µ3

Plots of the Dirichlet distribution over the simplex, for various settings of the parameters αk , are shown in Figure 2.5.
Multiplying the prior (2.38) by the likelihood function (2.34), we obtain theposterior distribution for the parameters {µk } in the formp(µ|D, α) ∝ p(D|µ)p(µ|α) ∝Kk +mk −1µα.
k(2.40)

k=1

We see that the posterior distribution again takes the form of a Dirichlet distribution,
conﬁrming that the Dirichlet is indeed a conjugate prior for the multinomial. This
allows us to determine the normalization coefﬁcient by comparison with (2.38) sothat
p(µ|D, α) = Dir(µ|α + m)Γ(α0 + N )k +mk −1µαkΓ(α1 + m1 ) · · · Γ(αK + mK )K=

(2.41)

k=1

where we have denoted m = (m1 , . . . , mK )T . As for the case of the binomial
distribution with its beta prior, we can interpret the parameters αk of the Dirichletprior as an effective number of observations of xk = 1.
Note that two-state quantities can either be represented as binary variables andLejeune Dirichlet
1805–1859
Johann Peter Gustav Lejeune
Dirichlet was a modest and reserved mathematician who madecontributions in number theory, mechanics, and astronomy, and who
gave the ﬁrst rigorous analysis of
Fourier series. His family originated from Richelet
in Belgium, and the name Lejeune Dirichlet comesfrom ‘le jeune de Richelet’ (the young person from
Richelet). Dirichlet’s ﬁrst paper, which was publishedin 1825, brought him instant fame. It concerned Fermat’s last theorem, which claims that there are no
positive integer solutions to xn + y n = z n for n > 2.
Dirichlet gave a partial proof for the case n = 5, which
was sent to Legendre for review and who in turn completed the proof. Later, Dirichlet gave a complete proof
for n = 14, although a full proof of Fermat’s last theorem for arbitrary n had to wait until the work of Andrew
Wiles in the closing years of the 20th century.

78

2. PROBABILITY DISTRIBUTIONS

Figure 2.5 Plots of the Dirichlet distribution over three variables, where the two horizontal axes are coordinatesin the plane of the simplex and the vertical axis corresponds to the value of the density. Here {αk } = 0.1 on theleft plot, {αk } = 1 in the centre plot, and {αk } = 10 in the right plot.

modelled using the binomial distribution (2.9) or as 1-of-2 variables and modelledusing the multinomial distribution (2.34) with K = 2.

2.3. The Gaussian Distribution
The Gaussian, also known as the normal distribution, is a widely used model for the
distribution of continuous variables. In the case of a single variable x, the Gaussian
distribution can be written in the form


1
1
2
N (x|µ, σ 2 ) =
exp
−
(x
−
µ)
(2.42)
1/22σ 2(2πσ 2 )where µ is the mean and σ 2 is the variance. For a D-dimensional vector x, themultivariate Gaussian distribution takes the form


1
1
1
T −1
exp
−
Σ
(x
−
µ)
(2.43)
N (x|µ, Σ) =
(x
−
µ)
2
(2π)D/2 |Σ|1/2

Section 1.6
Exercise 2.14

where µ is a D-dimensional mean vector, Σ is a D × D covariance matrix, and |Σ|denotes the determinant of Σ.
The Gaussian distribution arises in many different contexts and can be motivatedfrom a variety of different perspectives. For example, we have already seen that fora single real variable, the distribution that maximizes the entropy is the Gaussian.
This property applies also to the multivariate Gaussian.
Another situation in which the Gaussian distribution arises is when we consider
the sum of multiple random variables. The central limit theorem (due to Laplace)tells us that, subject to certain mild conditions, the sum of a set of random variables,
which is of course itself a random variable, has a distribution that becomes increasingly Gaussian as the number of terms in the sum increases (Walker, 1969). We can

79

2.3. The Gaussian Distribution
3

3

N =1

3

N =2

2

2

2

1

1

1

0

0

0.5

1

0

0

0.5

1

0

N = 100

0.5

1Figure 2.6 Histogram plots of the mean of N uniformly distributed numbers for various values of N . We
observe that as N increases, the distribution tends towards a Gaussian.

Appendix C

illustrate this by considering N variables x1 , . . . , xN each of which has a uniform
distribution over the interval [0, 1] and then considering the distribution of the mean
(x1 + · · · + xN )/N . For large N , this distribution tends to a Gaussian, as illustrated
in Figure 2.6. In practice, the convergence to a Gaussian as N increases can bevery rapid. One consequence of this result is that the binomial distribution (2.9),
which is a distribution over m deﬁned by the sum of N observations of the randombinary variable x, will tend to a Gaussian as N → ∞ (see Figure 2.1 for the case of
N = 10).
The Gaussian distribution has many important analytical properties, and we shallconsider several of these in detail. As a result, this section will be rather more technically involved than some of the earlier sections, and will require familiarity withvarious matrix identities. However, we strongly encourage the reader to become proﬁcient in manipulating Gaussian distributions using the techniques presented here as
this will prove invaluable in understanding the more complex models presented inlater chapters.
We begin by considering the geometrical form of the Gaussian distribution. The

Carl Friedrich Gauss
1777–1855
It is said that when Gauss went
to elementary school at age 7, his
teacher Büttner, trying to keep theclass occupied, asked the pupils to
sum the integers from 1 to 100. To
the teacher’s amazement, Gauss
arrived at the answer in a matter of moments by notingthat the sum can be represented as 50 pairs (1 + 100,
2+99, etc.) each of which added to 101, giving the answer 5,050. It is now believed that the problem which
was actually set was of the same form but somewhat
harder in that the sequence had a larger starting value
and a larger increment. Gauss was a German math-ematician and scientist with a reputation for being a
hard-working perfectionist. One of his many contributions was to show that least squares can be derived
under the assumption of normally distributed errors.
He also created an early formulation of non-Euclidean
geometry (a self-consistent geometrical theory that violates the axioms of Euclid) but was reluctant to discuss it openly for fear that his reputation might suffer
if it were seen that he believed in such a geometry.
At one point, Gauss was asked to conduct a geodetic
survey of the state of Hanover, which led to his formulation of the normal distribution, now also known
as the Gaussian. After his death, a study of his diaries revealed that he had discovered several important mathematical results years or even decades before they were published by others.

80

2. PROBABILITY DISTRIBUTIONS
functional dependence of the Gaussian on x is through the quadratic form
∆2 = (x − µ)T Σ−1 (x − µ)Exercise 2.17

which appears in the exponent. The quantity ∆ is called the Mahalanobis distancefrom µ to x and reduces to the Euclidean distance when Σ is the identity matrix. The
Gaussian distribution will be constant on surfaces in x-space for which this quadratic
form is constant.
First of all, we note that the matrix Σ can be taken to be symmetric, without
loss of generality, because any antisymmetric component would disappear from theexponent. Now consider the eigenvector equation for the covariance matrix
Σui = λi ui

Exercise 2.18

(2.44)(2.45)

where i = 1, . . . , D. Because Σ is a real, symmetric matrix its eigenvalues will bereal, and its eigenvectors can be chosen to form an orthonormal set, so thatuT
i uj = Iij(2.46)

where Iij is the i, j element of the identity matrix and satisﬁes
1, if i = j
0, otherwise.

Iij =Exercise 2.19

(2.47)

The covariance matrix Σ can be expressed as an expansion in terms of its eigenvectors in the formD
λ i ui uT(2.48)
Σ=ii=1and similarly the inverse covariance matrix Σ−1 can be expressed as
−1

Σ

=

D

1
i=1

λi

ui uT
i .

(2.49)

Substituting (2.49) into (2.44), the quadratic form becomes
2

∆ =

D

y2
i

i=1

where we have deﬁned

λi

y i = uTi (x − µ).

(2.50)(2.51)

We can interpret {yi } as a new coordinate system deﬁned by the orthonormal vectors
ui that are shifted and rotated with respect to the original xi coordinates. Forming
the vector y = (y1 , . . . , yD )T , we havey = U(x − µ)(2.52)

81

2.3. The Gaussian Distribution
Figure 2.7

The red curve shows the ellip- x2
tical surface of constant probability density for a Gaussian ina two-dimensional space x =(x1 , x2 ) on which the density
is exp(−1/2) of its value at
x = µ. The major axes ofthe ellipse are deﬁned by the
eigenvectors ui of the covariance matrix, with corresponding eigenvalues λi .

u2
u1
y2
y1
µ
1/2
λ2
1/2

λ1

x1

Appendix C

where U is a matrix whose rows are given by uT
i . From (2.46) it follows that U isan orthogonal matrix, i.e., it satisﬁes UUT = I, and hence also UT U = I, where I
is the identity matrix.
The quadratic form, and hence the Gaussian density, will be constant on surfaces
for which (2.51) is constant. If all of the eigenvalues λi are positive, then thesesurfaces represent ellipsoids, with their centres at µ and their axes oriented along ui ,1 /2
and with scaling factors in the directions of the axes given by λi , as illustrated in
Figure 2.7.
For the Gaussian distribution to be well deﬁned, it is necessary for all of theeigenvalues λi of the covariance matrix to be strictly positive, otherwise the distribution cannot be properly normalized. A matrix whose eigenvalues are strictly
positive is said to be positive deﬁnite. In Chapter 12, we will encounter Gaussian
distributions for which one or more of the eigenvalues are zero, in which case the
distribution is singular and is conﬁned to a subspace of lower dimensionality. If allof the eigenvalues are nonnegative, then the covariance matrix is said to be positive
semideﬁnite.
Now consider the form of the Gaussian distribution in the new coordinate systemdeﬁned by the yi . In going from the x to the y coordinate system, we have a Jacobianmatrix J with elements given by
Jij =

∂xi
= Uji
∂yj

(2.53)

where Uji are the elements of the matrix UT . Using the orthonormality property ofthe matrix U, we see that the square of the determinant of the Jacobian matrix is 2  |J|2 = UT  = UT  |U| = UT U = |I| = 1(2.54)and hence |J| = 1. Also, the determinant |Σ| of the covariance matrix can be written

82

2. PROBABILITY DISTRIBUTIONS
as the product of its eigenvalues, and hence
1 /2

|Σ|

=

D


1 /2

λj .

(2.55)

j =1

Thus in the yj coordinate system, the Gaussian distribution takes the formp(y) = p(x)|J| =

D

j =1

yj2
1
exp −
2λj
(2πλj )1/2

(2.56)

which is the product of D independent univariate Gaussian distributions. The eigenvectors therefore deﬁne a new set of shifted and rotated coordinates with respect
to which the joint probability distribution factorizes into a product of independent
distributions. The integral of the distribution in the y coordinate system is thenp(y) dy =

D  ∞

j =1

yj2
1
exp
−
1 /2
2λj
−∞ (2πλj )dyj = 1

(2.57)

where we have used the result (1.48) for the normalization of the univariate Gaussian.
This conﬁrms that the multivariate Gaussian (2.43) is indeed normalized.
We now look at the moments of the Gaussian distribution and thereby provide aninterpretation of the parameters µ and Σ. The expectation of x under the Gaussian
distribution is given by1
1
1
exp − (x − µ)T Σ−1 (x − µ) x dx
E[x] =
D/
2
1
/
2
2
(2π)
|Σ|

1
1
1
=
exp − zT Σ−1 z (z + µ) dz(2.58)
D/
2
1
/
2
2
(2π)
|Σ|
where we have changed variables using z = x − µ. We now note that the exponent
is an even function of the components of z and, because the integrals over these aretaken over the range (−∞, ∞), the term in z in the factor (z + µ) will vanish bysymmetry. Thus
E[x] = µ(2.59)and so we refer to µ as the mean of the Gaussian distribution.
We now consider second order moments of the Gaussian. In the univariate case,we considered the second order moment given by E[x2 ]. For the multivariate Gaussian, there are D2 second order moments given by E[xi xj ], which we can group
together to form the matrix E[xxT ]. This matrix can be written as

1
1
1
T
E[xx ] =exp − (x − µ)T Σ−1 (x − µ) xxT dx2(2π)D/2 |Σ|1/2

1
1
1
=
exp − zT Σ−1 z (z + µ)(z + µ)T dz
D/
2
1
/
2
2
(2π)
|Σ|

2.3. The Gaussian Distribution

83

where again we have changed variables using z = x − µ. Note that the cross-terms
involving µzT and µT z will again vanish by symmetry. The term µµT is constantand can be taken outside the integral, which itself is unity because the Gaussian
distribution is normalized. Consider the term involving zzT . Again, we can makeuse of the eigenvector expansion of the covariance matrix given by (2.45), together
with the completeness of the set of eigenvectors, to writez=D

y j uj(2.60)

j =1

where yj = uT
j z, which gives
1
1
D/
2
(2π)
|Σ|1/2=1
exp − zT Σ−1 z zzT dz
2
 D


D
D
 y2
1 
1
k
ui uT
exp −
yi yj dyj2λk(2π)D/2 |Σ|1/2
i=1 j =1

=

D


ui uT
i λi = Σ

k=1(2.61)

i=1

where we have made use of the eigenvector equation (2.45), together with the factthat the integral on the right-hand side of the middle line vanishes by symmetry
unless i = j, and in the ﬁnal line we have made use of the results (1.50) and (2.55),
together with (2.48). Thus we haveE[xxT ] = µµT + Σ.

(2.62)For single random variables, we subtracted the mean before taking second moments in order to deﬁne a variance. Similarly, in the multivariate case it is again
convenient to subtract off the mean, giving rise to the covariance of a random vector
x deﬁned by(2.63)
cov[x] = E (x − E[x])(x − E[x])T .
For the speciﬁc case of a Gaussian distribution, we can make use of E[x] = µ,together with the result (2.62), to givecov[x] = Σ.

Exercise 2.21

(2.64)

Because the parameter matrix Σ governs the covariance of x under the Gaussian
distribution, it is called the covariance matrix.
Although the Gaussian distribution (2.43) is widely used as a density model, it
suffers from some signiﬁcant limitations. Consider the number of free parameters in
the distribution. A general symmetric covariance matrix Σ will have D(D + 1)/2
independent parameters, and there are another D independent parameters in µ, giving D(D + 3)/2 parameters in total. For large D, the total number of parameters84

2. PROBABILITY DISTRIBUTIONS

Figure 2.8 Contours of constant x2
probability density for a Gaussian
distribution in two dimensions inwhich the covariance matrix is (a) ofgeneral form, (b) diagonal, in which
the elliptical contours are alignedwith the coordinate axes, and (c)proportional to the identity matrix, in
which the contours are concentric
circles.

Section 8.3

Section 13.3

x2

x2

x1
(a)

x1
(b)

x1
(c)

therefore grows quadratically with D, and the computational task of manipulating
and inverting large matrices can become prohibitive. One way to address this problem is to use restricted forms of the covariance matrix. If we consider covariancematrices that are diagonal, so that Σ = diag(σi2 ), we then have a total of 2D independent parameters in the density model. The corresponding contours of constantdensity are given by axis-aligned ellipsoids. We could further restrict the covariancematrix to be proportional to the identity matrix, Σ = σ 2 I, known as an isotropic covariance, giving D + 1 independent parameters in the model and spherical surfacesof constant density. The three possibilities of general, diagonal, and isotropic covariance matrices are illustrated in Figure 2.8. Unfortunately, whereas such approaches
limit the number of degrees of freedom in the distribution and make inversion of thecovariance matrix a much faster operation, they also greatly restrict the form of theprobability density and limit its ability to capture interesting correlations in the data.
A further limitation of the Gaussian distribution is that it is intrinsically unimodal (i.e., has a single maximum) and so is unable to provide a good approximation
to multimodal distributions. Thus the Gaussian distribution can be both too ﬂexible,
in the sense of having too many parameters, while also being too limited in the rangeof distributions that it can adequately represent. We will see later that the introduction of latent variables, also called hidden variables or unobserved variables, allows
both of these problems to be addressed. In particular, a rich family of multimodal
distributions is obtained by introducing discrete latent variables leading to mixturesof Gaussians, as discussed in Section 2.3.9. Similarly, the introduction of continuous
latent variables, as described in Chapter 12, leads to models in which the number offree parameters can be controlled independently of the dimensionality D of the data
space while still allowing the model to capture the dominant correlations in the data
set. Indeed, these two approaches can be combined and further extended to derivea very rich set of hierarchical models that can be adapted to a broad range of practical applications. For instance, the Gaussian version of the Markov random ﬁeld,
which is widely used as a probabilistic model of images, is a Gaussian distribution
over the joint space of pixel intensities but rendered tractable through the impositionof considerable structure reﬂecting the spatial organization of the pixels. Similarly,
the linear dynamical system, used to model time series data for applications suchas tracking, is also a joint Gaussian distribution over a potentially large number ofobserved and latent variables and again is tractable due to the structure imposed on
the distribution. A powerful framework for expressing the form and properties of2.3. The Gaussian Distribution

85

such complex distributions is that of probabilistic graphical models, which will form
the subject of Chapter 8.

2.3.1 Conditional Gaussian distributions
An important property of the multivariate Gaussian distribution is that if two
sets of variables are jointly Gaussian, then the conditional distribution of one set
conditioned on the other is again Gaussian. Similarly, the marginal distribution ofeither set is also Gaussian.
Consider ﬁrst the case of conditional distributions. Suppose x is a D-dimensional
vector with Gaussian distribution N (x|µ, Σ) and that we partition x into two disjoint subsets xa and xb . Without loss of generality, we can take xa to form the ﬁrst
M components of x, with xb comprising the remaining D − M components, so that
 
xa
x=
.
(2.65)xb
We also deﬁne corresponding partitions of the mean vector µ given by
 
µa
µ=
µb
and of the covariance matrix Σ given byΣaa ΣabΣ=.
Σba Σbb(2.66)(2.67)

Note that the symmetry ΣT = Σ of the covariance matrix implies that Σaa and Σbb
are symmetric, while Σba = ΣTab .
In many situations, it will be convenient to work with the inverse of the covariance matrix(2.68)Λ ≡ Σ−1
which is known as the precision matrix. In fact, we shall see that some properties
of Gaussian distributions are most naturally expressed in terms of the covariance,
whereas others take a simpler form when viewed in terms of the precision. We
therefore also introduce the partitioned form of the precision matrixΛaa Λab(2.69)
Λ=Λba Λbb
Exercise 2.22

corresponding to the partitioning (2.65) of the vector x. Because the inverse of asymmetric matrix is also symmetric, we see that Λaa and Λbb are symmetric, whileΛT
ab = Λba . It should be stressed at this point that, for instance, Λaa is not simply
given by the inverse of Σaa . In fact, we shall shortly examine the relation between
the inverse of a partitioned matrix and the inverses of its partitions.
Let us begin by ﬁnding an expression for the conditional distribution p(xa |xb ).
From the product rule of probability, we see that this conditional distribution can be86

2. PROBABILITY DISTRIBUTIONS
evaluated from the joint distribution p(x) = p(xa , xb ) simply by ﬁxing xb to theobserved value and normalizing the resulting expression to obtain a valid probability
distribution over xa . Instead of performing this normalization explicitly, we canobtain the solution more efﬁciently by considering the quadratic form in the exponentof the Gaussian distribution given by (2.44) and then reinstating the normalization
coefﬁcient at the end of the calculation. If we make use of the partitioning (2.65),(2.66), and (2.69), we obtain
1− (x − µ)T Σ−1 (x − µ) =2
11
− (xa − µa )T Λaa (xa − µa ) − (xa − µa )T Λab (xb − µb )
2
2
1
1
T
− (xb − µb ) Λba (xa − µa ) − (xb − µb )T Λbb (xb − µb ). (2.70)
2
2
We see that as a function of xa , this is again a quadratic form, and hence the corresponding conditional distribution p(xa |xb ) will be Gaussian. Because this distribution is completely characterized by its mean and its covariance, our goal will beto identify expressions for the mean and covariance of p(xa |xb ) by inspection of(2.70).
This is an example of a rather common operation associated with Gaussian
distributions, sometimes called ‘completing the square’, in which we are given aquadratic form deﬁning the exponent terms in a Gaussian distribution, and we needto determine the corresponding mean and covariance. Such problems can be solved
straightforwardly by noting that the exponent in a general Gaussian distribution
N (x|µ, Σ) can be written11− (x − µ)T Σ−1 (x − µ) = − xT Σ−1 x + xT Σ−1 µ + const
22(2.71)

where ‘const’ denotes terms which are independent of x, and we have made use of
the symmetry of Σ. Thus if we take our general quadratic form and express it in
the form given by the right-hand side of (2.71), then we can immediately equate thematrix of coefﬁcients entering the second order term in x to the inverse covariancematrix Σ−1 and the coefﬁcient of the linear term in x to Σ−1 µ, from which we canobtain µ.
Now let us apply this procedure to the conditional Gaussian distribution p(xa |xb )for which the quadratic form in the exponent is given by (2.70). We will denote themean and covariance of this distribution by µa|b and Σa|b , respectively. Consider
the functional dependence of (2.70) on xa in which xb is regarded as a constant. If
we pick out all terms that are second order in xa , we have1
− xT
Λaa xa2 a(2.72)

from which we can immediately conclude that the covariance (inverse precision) ofp(xa |xb ) is given by1
Σa|b = Λ−(2.73)aa .

2.3. The Gaussian Distribution

87

Now consider all of the terms in (2.70) that are linear in xa
xTa {Λaa µa − Λab (xb − µb )}(2.74)

where we have used ΛT
ba = Λab . From our discussion of the general form (2.71),
1
the coefﬁcient of xa in this expression must equal Σ−
a|b µa|b and henceµa|b= Σa|b {Λaa µa − Λab (xb − µb )}1= µa − Λ −aa Λab (xb − µb )Exercise 2.24

(2.75)

where we have made use of (2.73).
The results (2.73) and (2.75) are expressed in terms of the partitioned precision
matrix of the original joint distribution p(xa , xb ). We can also express these resultsin terms of the corresponding partitioned covariance matrix. To do this, we make useof the following identity for the inverse of a partitioned matrix

−1 

A B
M
−MBD−1=(2.76)C D−D−1 CM D−1 + D−1 CMBD−1
where we have deﬁned

M = (A − BD−1 C)−1 .

(2.77)−1

The quantity M is known as the Schur complement of the matrix on the left-handside of (2.76) with respect to the submatrix D. Using the deﬁnition

−1 

Σaa ΣabΛaa Λab=(2.78)
Σba ΣbbΛba Λbband making use of (2.76), we have1−1
Λaa = (Σaa − Σab Σ−bb Σba )(2.79)

Λab =(2.80)11−1−(Σaa − Σab Σ−Σab Σ−bb Σba )bb .

From these we obtain the following expressions for the mean and covariance of theconditional distribution p(xa |xb )Section 8.1.4µa|b1= µa + Σab Σ−
bb (xb − µb )(2.81)

Σa|b1= Σaa − Σab Σ−bb Σba .

(2.82)

Comparing (2.73) and (2.82), we see that the conditional distribution p(xa |xb ) takes
a simpler form when expressed in terms of the partitioned precision matrix thanwhen it is expressed in terms of the partitioned covariance matrix. Note that themean of the conditional distribution p(xa |xb ), given by (2.81), is a linear function ofxb and that the covariance, given by (2.82), is independent of xa . This represents an
example of a linear-Gaussian model.

88

2. PROBABILITY DISTRIBUTIONS

2.3.2 Marginal Gaussian distributions
We have seen that if a joint distribution p(xa , xb ) is Gaussian, then the conditional distribution p(xa |xb ) will again be Gaussian. Now we turn to a discussion of
the marginal distribution given by(2.83)p(xa ) = p(xa , xb ) dxb
which, as we shall see, is also Gaussian. Once again, our strategy for evaluating this
distribution efﬁciently will be to focus on the quadratic form in the exponent of thejoint distribution and thereby to identify the mean and covariance of the marginal
distribution p(xa ).
The quadratic form for the joint distribution can be expressed, using the partitioned precision matrix, in the form (2.70). Because our goal is to integrate out
xb , this is most easily achieved by ﬁrst considering the terms involving xb and thencompleting the square in order to facilitate integration. Picking out just those terms
that involve xb , we have1
1 T −1
1
−1
−1
T
T
− xT
b Λbb xb +xb m = − (xb −Λbb m) Λbb (xb −Λbb m)+ m Λbb m (2.84)
222
where we have deﬁned
m = Λbb µb − Λba (xa − µa ).

(2.85)We see that the dependence on xb has been cast into the standard quadratic form of a
Gaussian distribution corresponding to the ﬁrst term on the right-hand side of (2.84),
plus a term that does not depend on xb (but that does depend on xa ). Thus, when
we take the exponential of this quadratic form, we see that the integration over xb
required by (2.83) will take the form

1
1
−1
T
exp − (xb − Λ−dxb .
(2.86)bb m) Λbb (xb − Λbb m)
2This integration is easily performed by noting that it is the integral over an unnormalized Gaussian, and so the result will be the reciprocal of the normalization coefﬁcient. We know from the form of the normalized Gaussian given by (2.43), that
this coefﬁcient is independent of the mean and depends only on the determinant of
the covariance matrix. Thus, by completing the square with respect to xb , we canintegrate out xb and the only term remaining from the contributions on the left-handside of (2.84) that depends on xa is the last term on the right-hand side of (2.84) inwhich m is given by (2.85). Combining this term with the remaining terms from

2.3. The Gaussian Distribution

89(2.70) that depend on xa , we obtain1T
1[Λbb µb − Λba (xa − µa )] Λ−bb [Λbb µb − Λba (xa − µa )]2
1− xT
Λaa xa + xTa (Λaa µa + Λab µb ) + const
2 a1
1(Λaa − Λab Λ−= − xTbb Λba )xa
2 a
−1
−1
+xT
µa + const(2.87)a (Λaa − Λab Λbb Λba )where ‘const’ denotes quantities independent of xa . Again, by comparison with
(2.71), we see that the covariance of the marginal distribution of p(xa ) is given by1−1.
Σa = (Λaa − Λab Λ−bb Λba )(2.88)

Similarly, the mean is given by1
Σa (Λaa − Λab Λ−bb Λba )µa = µa(2.89)

where we have used (2.88). The covariance in (2.88) is expressed in terms of thepartitioned precision matrix given by (2.69). We can rewrite this in terms of thecorresponding partitioning of the covariance matrix given by (2.67), as we did forthe conditional distribution. These partitioned matrices are related by

−1 

Λaa ΛabΣaa Σab=(2.90)
Λba ΛbbΣba Σbb
Making use of (2.76), we then have1Λaa − Λab Λ−bb Λba−1= Σaa .

(2.91)

Thus we obtain the intuitively satisfying result that the marginal distribution p(xa )has mean and covariance given by
E[xa ] = µa
cov[xa ] = Σaa .

(2.92)(2.93)We see that for a marginal distribution, the mean and covariance are most simply expressed in terms of the partitioned covariance matrix, in contrast to the conditional
distribution for which the partitioned precision matrix gives rise to simpler expressions.
Our results for the marginal and conditional distributions of a partitioned Gaussian are summarized below.
Partitioned Gaussians
Given a joint Gaussian distribution N (x|µ, Σ) with Λ ≡ Σ−1 and
 
 
xa
µa
x=
,
µ=
xb
µb

(2.94)

90

2. PROBABILITY DISTRIBUTIONS
1

10

xb

p(xa |xb = 0.7)

xb = 0.7

0.5

5p(xa , xb )p(xa )

0

0

0.5

0

1

xa

0

0.5

xa

1

Figure 2.9 The plot on the left shows the contours of a Gaussian distribution p(xa , xb ) over two variables, andthe plot on the right shows the marginal distribution p(xa ) (blue curve) and the conditional distribution p(xa |xb )for xb = 0.7 (red curve).

Σ=Σaa Σab,Σba ΣbbΛ=Λaa Λab.
Λba Λbb(2.95)

Conditional distribution:1p(xa |xb ) = N (x|µa|b , Λ−
aa )

µa|b

=

1
µa − Λ−aa Λab (xb − µb ).

(2.96)(2.97)

Marginal distribution:p(xa ) = N (xa |µa , Σaa ).

(2.98)

We illustrate the idea of conditional and marginal distributions associated witha multivariate Gaussian using an example involving two variables in Figure 2.9.

2.3.3 Bayes’ theorem for Gaussian variables
In Sections 2.3.1 and 2.3.2, we considered a Gaussian p(x) in which we partitioned the vector x into two subvectors x = (xa , xb ) and then found expressions forthe conditional distribution p(xa |xb ) and the marginal distribution p(xa ). We notedthat the mean of the conditional distribution p(xa |xb ) was a linear function of xb .
Here we shall suppose that we are given a Gaussian marginal distribution p(x) and a
Gaussian conditional distribution p(y|x) in which p(y|x) has a mean that is a linear
function of x, and a covariance which is independent of x. This is an example of2.3. The Gaussian Distribution

91

a linear Gaussian model (Roweis and Ghahramani, 1999), which we shall study ingreater generality in Section 8.1.4. We wish to ﬁnd the marginal distribution p(y)and the conditional distribution p(x|y). This is a problem that will arise frequently
in subsequent chapters, and it will prove convenient to derive the general results here.
We shall take the marginal and conditional distributions to bep(x) = N x|µ, Λ−1(2.99)−1p(y|x) = N y|Ax + b, L

(2.100)where µ, A, and b are parameters governing the means, and Λ and L are precision
matrices. If x has dimensionality M and y has dimensionality D, then the matrix A
has size D × M .
First we ﬁnd an expression for the joint distribution over x and y. To do this, we
deﬁne
 
x
(2.101)z=yand then consider the log of the joint distributionln p(z) = ln p(x) + ln p(y|x)
1= − (x − µ)T Λ(x − µ)21
− (y − Ax − b)T L(y − Ax − b) + const
2(2.102)where ‘const’ denotes terms independent of x and y. As before, we see that this is aquadratic function of the components of z, and hence p(z) is Gaussian distribution.
To ﬁnd the precision of this Gaussian, we consider the second order terms in (2.102),
which can be written as1
1
1
1− xT (Λ + AT LA)x − yT Ly + yT LAx + xT AT Ly
2222 T  
11 xx
Λ + AT LA −AT L= − zT Rz
= −
y
y
−LA
L
2
2

(2.103)and so the Gaussian distribution over z has precision (inverse covariance) matrix
given by
Λ + AT LA −AT L.
(2.104)
R=−LA
L
Exercise 2.29

The covariance matrix is found by taking the inverse of the precision, which can bedone using the matrix inversion formula (2.76) to give −1ΛΛ−1 AT−1.
(2.105)cov[z] = R =AΛ−1 L−1 + AΛ−1 AT

92

2. PROBABILITY DISTRIBUTIONS
Similarly, we can ﬁnd the mean of the Gaussian distribution over z by identifying the linear terms in (2.102), which are given by T 
xΛµ − AT Lb
T
T T
Tx Λµ − x A Lb + y Lb =.
(2.106)y
Lb
Using our earlier result (2.71) obtained by completing the square over the quadratic
form of a multivariate Gaussian, we ﬁnd that the mean of z is given byT−1 Λµ − A Lb.
(2.107)
E[z] = R
Lb

Exercise 2.30

Making use of (2.105), we then obtain
E[z] =

Section 2.3




µ
.
Aµ + bNext we ﬁnd an expression for the marginal distribution p(y) in which we havemarginalized over x. Recall that the marginal distribution over a subset of the components of a Gaussian random vector takes a particularly simple form when expressed in terms of the partitioned covariance matrix. Speciﬁcally, its mean andcovariance are given by (2.92) and (2.93), respectively. Making use of (2.105) and(2.108) we see that the mean and covariance of the marginal distribution p(y) are
given byE[y] = Aµ + b
cov[y] = L−1 + AΛ−1 AT .

Section 2.3(2.108)(2.109)(2.110)A special case of this result is when A = I, in which case it reduces to the convolution of two Gaussians, for which we see that the mean of the convolution is the sumof the mean of the two Gaussians, and the covariance of the convolution is the sumof their covariances.
Finally, we seek an expression for the conditional p(x|y). Recall that the results
for the conditional distribution are most easily expressed in terms of the partitioned
precision matrix, using (2.73) and (2.75). Applying these results to (2.105) and(2.108) we see that the conditional distribution p(x|y) has mean and covariancegiven by

(2.111)
E[x|y] = (Λ + AT LA)−1 AT L(y − b) + Λµ
cov[x|y] = (Λ + AT LA)−1 .

(2.112)The evaluation of this conditional can be seen as an example of Bayes’ theorem.
We can interpret the distribution p(x) as a prior distribution over x. If the variable
y is observed, then the conditional distribution p(x|y) represents the corresponding
posterior distribution over x. Having found the marginal and conditional distributions, we effectively expressed the joint distribution p(z) = p(x)p(y|x) in the formp(x|y)p(y). These results are summarized below.

2.3. The Gaussian Distribution

93

Marginal and Conditional Gaussians
Given a marginal Gaussian distribution for x and a conditional Gaussian distribution for y given x in the form
p(x) = N (x|µ, Λ−1 )p(y|x) = N (y|Ax + b, L−1 )(2.113)(2.114)the marginal distribution of y and the conditional distribution of x given y are
given byp(y) = N (y|Aµ + b, L−1 + AΛ−1 AT )p(x|y) = N (x|Σ{AT L(y − b) + Λµ}, Σ)(2.115)(2.116)

Σ = (Λ + AT LA)−1 .

(2.117)where

2.3.4 Maximum likelihood for the Gaussian
Given a data set X = (x1 , . . . , xN )T in which the observations {xn } are assumed to be drawn independently from a multivariate Gaussian distribution, we canestimate the parameters of the distribution by maximum likelihood. The log likelihood function is given byN
1
ND
ln(2π)− ln |Σ|−(xn −µ)T Σ−1 (xn −µ). (2.118)222N

ln p(X|µ, Σ) = −n=1

By simple rearrangement, we see that the likelihood function depends on the data set
only through the two quantities
N


N


xn ,

n=1

Appendix C

xn xT
n.

(2.119)n=1

These are known as the sufﬁcient statistics for the Gaussian distribution. Using(C.19), the derivative of the log likelihood with respect to µ is given by∂ln p(X|µ, Σ) =
Σ−1 (xn − µ)∂µN(2.120)n=1

and setting this derivative to zero, we obtain the solution for the maximum likelihood
estimate of the mean given by
1 
xn
N
N

µML =

n=1

(2.121)

94

2. PROBABILITY DISTRIBUTIONS

Exercise 2.34

which is the mean of the observed set of data points. The maximization of (2.118)
with respect to Σ is rather more involved. The simplest approach is to ignore thesymmetry constraint and show that the resulting solution is symmetric as required.
Alternative derivations of this result, which impose the symmetry and positive deﬁniteness constraints explicitly, can be found in Magnus and Neudecker (1999). The
result is as expected and takes the form
1 
ΣML =
(xn − µML )(xn − µML )TN
N(2.122)n=1

Exercise 2.35

which involves µML because this is the result of a joint maximization with respectto µ and Σ. Note that the solution (2.121) for µML does not depend on ΣML , and so
we can ﬁrst evaluate µML and then use this to evaluate ΣML .
If we evaluate the expectations of the maximum likelihood solutions under thetrue distribution, we obtain the following results
E[µML ] = µN −1Σ.
E[ΣML ] =N

(2.123)(2.124)We see that the expectation of the maximum likelihood estimate for the mean is equal
to the true mean. However, the maximum likelihood estimate for the covariance has
an expectation that is less than the true value, and hence it is biased. We can correct
 given by
this bias by deﬁning a different estimator Σ

=
Σ

1 
(xn − µML )(xn − µML )T .
N −1
N(2.125)n=1

 is equal to Σ.
Clearly from (2.122) and (2.124), the expectation of Σ

2.3.5 Sequential estimation
Our discussion of the maximum likelihood solution for the parameters of a Gaussian distribution provides a convenient opportunity to give a more general discussionof the topic of sequential estimation for maximum likelihood. Sequential methods
allow data points to be processed one at a time and then discarded and are importantfor on-line applications, and also where large data sets are involved so that batchprocessing of all data points at once is infeasible.
Consider the result (2.121) for the maximum likelihood estimator of the mean(N )µML , which we will denote by µML when it is based on N observations. If we2.3. The Gaussian Distribution
Figure 2.10

A schematic illustration of two correlated random variables z and θ, together with theregression function f (θ) given by the conditional expectation E[z|θ]. The RobbinsMonro algorithm provides a general sequential procedure for ﬁnding the root θ of suchfunctions.

95

z

f (θ)
θ
θ



dissect out the contribution from the ﬁnal data point xN , we obtain
(N )

µML

1 
xn
N
N

=

n=1

1
1 
xn
xN +
N
N
N −1

=

n=1

1
N − 1 (N −1)
=
xN +
µML
N
N
1
(N −1)(N −1)= µML + (xN − µML ).
N(2.126)This result has a nice interpretation, as follows. After observing N − 1 data points(N −1)
we have estimated µ by µML . We now observe data point xN , and we obtain our(N )revised estimate µML by moving the old estimate a small amount, proportional to(N −1)
1/N , in the direction of the ‘error signal’ (xN − µML ). Note that, as N increases,
so the contribution from successive data points gets smaller.
The result (2.126) will clearly give the same answer as the batch result (2.121)
because the two formulae are equivalent. However, we will not always be able to derive a sequential algorithm by this route, and so we seek a more general formulationof sequential learning, which leads us to the Robbins-Monro algorithm. Consider apair of random variables θ and z governed by a joint distribution p(z, θ). The conditional expectation of z given θ deﬁnes a deterministic function f (θ) that is givenby(2.127)f (θ) ≡ E[z|θ] = zp(z|θ) dz
and is illustrated schematically in Figure 2.10. Functions deﬁned in this way arecalled regression functions.
Our goal is to ﬁnd the root θ at which f (θ ) = 0. If we had a large data setof observations of z and θ, then we could model the regression function directly andthen obtain an estimate of its root. Suppose, however, that we observe values ofz one at a time and we wish to ﬁnd a corresponding sequential estimation schemefor θ . The following general procedure for solving such problems was given by96

2. PROBABILITY DISTRIBUTIONS
Robbins and Monro (1951). We shall assume that the conditional variance of z is
ﬁnite so thatE (z − f )2 | θ < ∞(2.128)and we shall also, without loss of generality, consider the case where f (θ) > 0 forθ > θ and f (θ) < 0 for θ < θ , as is the case in Figure 2.10. The Robbins-Monroprocedure then deﬁnes a sequence of successive estimates of the root θ given byθ(N ) = θ(N −1) + aN −1 z(θ(N −1) )(2.129)where z(θ(N ) ) is an observed value of z when θ takes the value θ(N ) . The coefﬁcients{aN } represent a sequence of positive numbers that satisfy the conditions
lim aN= 0(2.130)aN= ∞(2.131)a2N< ∞.

(2.132)

N →∞
∞

N =1
∞

N =1

It can then be shown (Robbins and Monro, 1951; Fukunaga, 1990) that the sequenceof estimates given by (2.129) does indeed converge to the root with probability one.
Note that the ﬁrst condition (2.130) ensures that the successive corrections decreasein magnitude so that the process can converge to a limiting value. The second condition (2.131) is required to ensure that the algorithm does not converge short of theroot, and the third condition (2.132) is needed to ensure that the accumulated noise
has ﬁnite variance and hence does not spoil convergence.
Now let us consider how a general maximum likelihood problem can be solved
sequentially using the Robbins-Monro algorithm. By deﬁnition, the maximum likelihood solution θML is a stationary point of the log likelihood function and hence
satisﬁes


N

1 
∂

ln p(xn |θ) = 0.
(2.133)
∂θ N

n=1

θML

Exchanging the derivative and the summation, and taking the limit N → ∞ we have



N
1  ∂
∂
lim
ln p(xn |θ) = Exln p(x|θ)N →∞ N∂θ∂θ

(2.134)n=1

and so we see that ﬁnding the maximum likelihood solution corresponds to ﬁnding the root of a regression function. We can therefore apply the Robbins-Monroprocedure, which now takes the formθ(N ) = θ(N −1) + aN −1∂
ln p(xN |θ(N −1) ).
∂θ(N −1)(2.135)

2.3. The Gaussian Distribution
Figure 2.11

In the case of a Gaussian distribution, with θ zcorresponding to the mean µ, the regressionfunction illustrated in Figure 2.10 takes the formof a straight line, as shown in red. In thiscase, the random variable z corresponds to thederivative of the log likelihood function and isgiven by (x − µML )/σ 2 , and its expectation thatdeﬁnes the regression function is a straight line
given by (µ − µML )/σ 2 . The root of the regression function corresponds to the maximum likelihood estimator µML .

97p(z|µ)µMLµ

As a speciﬁc example, we consider once again the sequential estimation of themean of a Gaussian distribution, in which case the parameter θ(N ) is the estimate(N )µML of the mean of the Gaussian, and the random variable z is given byz=∂
1ln p(x|µML , σ 2 ) = 2 (x − µML ).
∂µML
σ(2.136)

Thus the distribution of z is Gaussian with mean µ − µML , as illustrated in Figure 2.11. Substituting (2.136) into (2.135), we obtain the univariate form of (2.126),
provided we choose the coefﬁcients aN to have the form aN = σ 2 /N . Note that
although we have focussed on the case of a single variable, the same technique,together with the same restrictions (2.130)–(2.132) on the coefﬁcients aN , apply
equally to the multivariate case (Blum, 1965).

2.3.6 Bayesian inference for the Gaussian
The maximum likelihood framework gave point estimates for the parameters µand Σ. Now we develop a Bayesian treatment by introducing prior distributions
over these parameters. Let us begin with a simple example in which we consider asingle Gaussian random variable x. We shall suppose that the variance σ 2 is known,
and we consider the task of inferring the mean µ given a set of N observations
X = {x1 , . . . , xN }. The likelihood function, that is the probability of the observed
data given µ, viewed as a function of µ, is given by


N
N

1
1 
p(xn |µ) =exp − 2(xn − µ)2 .
(2.137)p(X|µ) =2σ(2πσ 2 )N/2
n=1

n=1

Again we emphasize that the likelihood function p(X|µ) is not a probability distribution over µ and is not normalized.
We see that the likelihood function takes the form of the exponential of a quadratic form in µ. Thus if we choose a prior p(µ) given by a Gaussian, it will be a

98

2. PROBABILITY DISTRIBUTIONS
conjugate distribution for this likelihood function because the corresponding posterior will be a product of two exponentials of quadratic functions of µ and hence will
also be Gaussian. We therefore take our prior distribution to bep(µ) = N µ|µ0 , σ02(2.138)and the posterior distribution is given byp(µ|X) ∝ p(X|µ)p(µ).
Exercise 2.38

(2.139)Simple manipulation involving completing the square in the exponent shows that theposterior distribution is given by2
p(µ|X) = N µ|µN , σN

(2.140)

where
µN

=

1
2
σN

=

σ2
N σ02
µ
+
µML
0
N σ02 + σ 2
N σ02 + σ 2
1
N
+ 2
2
σ0
σ

(2.141)(2.142)in which µML is the maximum likelihood solution for µ given by the sample mean
1 
xn .
N
N

µML =

(2.143)n=1

It is worth spending a moment studying the form of the posterior mean andvariance. First of all, we note that the mean of the posterior distribution given by(2.141) is a compromise between the prior mean µ0 and the maximum likelihood
solution µML . If the number of observed data points N = 0, then (2.141) reduces
to the prior mean as expected. For N → ∞, the posterior mean is given by the
maximum likelihood solution. Similarly, consider the result (2.142) for the varianceof the posterior distribution. We see that this is most naturally expressed in termsof the inverse variance, which is called the precision. Furthermore, the precisions
are additive, so that the precision of the posterior is given by the precision of theprior plus one contribution of the data precision from each of the observed data
points. As we increase the number of observed data points, the precision steadily
increases, corresponding to a posterior distribution with steadily decreasing variance.
With no observed data points, we have the prior variance, whereas if the number of2goes to zero and the posterior distribution
data points N → ∞, the variance σN
becomes inﬁnitely peaked around the maximum likelihood solution. We therefore
see that the maximum likelihood result of a point estimate for µ given by (2.143) isrecovered precisely from the Bayesian formalism in the limit of an inﬁnite numberof observations. Note also that for ﬁnite N , if we take the limit σ02 → ∞ in which theprior has inﬁnite variance then the posterior mean (2.141) reduces to the maximum2= σ 2 /N .
likelihood result, while from (2.142) the posterior variance is given by σN

2.3. The Gaussian Distribution
Figure 2.12

Exercise 2.40
Section 2.3.5

Illustration of Bayesian inference for5
the mean µ of a Gaussian distribution, in which the variance is assumed to be known. The curves
show the prior distribution over µ(the curve labelled N = 0), whichin this case is itself Gaussian, alongwith the posterior distribution givenby (2.140) for increasing numbers N
of data points. The data points are
generated from a Gaussian of mean
0.8 and variance 0.1, and the prior ischosen to have mean 0. In both theprior and the likelihood function, the 0−1
variance is set to the true value.

99

N = 10
N =2
N =1
N =0

0

1

We illustrate our analysis of Bayesian inference for the mean of a Gaussian
distribution in Figure 2.12. The generalization of this result to the case of a Ddimensional Gaussian random variable x with known covariance and unknown mean
is straightforward.
We have already seen how the maximum likelihood expression for the mean ofa Gaussian can be re-cast as a sequential update formula in which the mean after
observing N data points was expressed in terms of the mean after observing N − 1data points together with the contribution from data point xN . In fact, the Bayesianparadigm leads very naturally to a sequential view of the inference problem. To see
this in the context of the inference of the mean of a Gaussian, we write the posterior
distribution with the contribution from the ﬁnal data point xN separated out so thatN−1
p(µ|D) ∝ p(µ)
p(xn |µ) p(xN |µ).
(2.144)n=1

The term in square brackets is (up to a normalization coefﬁcient) just the posterior
distribution after observing N − 1 data points. We see that this can be viewed asa prior distribution, which is combined using Bayes’ theorem with the likelihood
function associated with data point xN to arrive at the posterior distribution after
observing N data points. This sequential view of Bayesian inference is very generaland applies to any problem in which the observed data are assumed to be independentand identically distributed.
So far, we have assumed that the variance of the Gaussian distribution over thedata is known and our goal is to infer the mean. Now let us suppose that the mean
is known and we wish to infer the variance. Again, our calculations will be greatly
simpliﬁed if we choose a conjugate form for the prior distribution. It turns out to be
most convenient to work with the precision λ ≡ 1/σ 2 . The likelihood function for λtakes the form


N
N

λ
−1
N/2
2
p(X|λ) =N (xn |µ, λ ) ∝ λexp −(xn − µ) .
(2.145)
2
n=1

n=1

100

2. PROBABILITY DISTRIBUTIONS

2

2

2

a = 0.1
b = 0.1

a=1
b=1

1

0

a=4
b=6

1

0

Figure 2.13
a and b.

λ

1

2

0

1

λ

0

1

0

2

λ

0

1

2

Plot of the gamma distribution Gam(λ|a, b) deﬁned by (2.146) for various values of the parameters

The corresponding conjugate prior should therefore be proportional to the productof a power of λ and the exponential of a linear function of λ. This corresponds to
the gamma distribution which is deﬁned byGam(λ|a, b) =Exercise 2.41
Exercise 2.42

1 a a−1exp(−bλ).
b λΓ(a)(2.146)

Here Γ(a) is the gamma function that is deﬁned by (1.141) and that ensures that
(2.146) is correctly normalized. The gamma distribution has a ﬁnite integral if a > 0,
and the distribution itself is ﬁnite if a  1. It is plotted, for various values of a andb, in Figure 2.13. The mean and variance of the gamma distribution are given byE[λ] =
var[λ] =

a
b
a
.
b2

(2.147)(2.148)

Consider a prior distribution Gam(λ|a0 , b0 ). If we multiply by the likelihood
function (2.145), then we obtain a posterior distributionNλ
a0 −1 N/22p(λ|X) ∝ λ
λexp −b0 λ −(xn − µ)(2.149)2n=1which we recognize as a gamma distribution of the form Gam(λ|aN , bN ) whereaN= a0 +bN= b0 +

N
2
N
1
2

n=1

(2.150)(xn − µ)2 = b0 +

N 2
σ
2 ML

(2.151)

2is the maximum likelihood estimator of the variance. Note that in (2.149)
where σML
there is no need to keep track of the normalization constants in the prior and thelikelihood function because, if required, the correct coefﬁcient can be found at theend using the normalized form (2.146) for the gamma distribution.

2.3. The Gaussian Distribution

Section 2.2

101

From (2.150), we see that the effect of observing N data points is to increase
the value of the coefﬁcient a by N/2. Thus we can interpret the parameter a0 in
the prior in terms of 2a0 ‘effective’ prior observations. Similarly, from (2.151) we
2
2
see that the N data points contribute N σML/2 to the parameter b, where σMListhe variance, and so we can interpret the parameter b0 in the prior as arising fromthe 2a0 ‘effective’ prior observations having variance 2b0 /(2a0 ) = b0 /a0 . Recall
that we made an analogous interpretation for the Dirichlet prior. These distributions
are examples of the exponential family, and we shall see that the interpretation ofa conjugate prior in terms of effective ﬁctitious data points is a general one for the
exponential family of distributions.
Instead of working with the precision, we can consider the variance itself. The
conjugate prior in this case is called the inverse gamma distribution, although weshall not discuss this further because we will ﬁnd it more convenient to work withthe precision.
Now suppose that both the mean and the precision are unknown. To ﬁnd a
conjugate prior, we consider the dependence of the likelihood function on µ and λ

1/2
N 

λ

λ
exp − (xn − µ)2
2
n=1




N
N
N
2


λµ
λ
∝
λ1/2 exp −exp λµ
xn −
x2n .
2
2

p(X|µ, λ) =

2π

n=1

(2.152)n=1

We now wish to identify a prior distribution p(µ, λ) that has the same functional
dependence on µ and λ as the likelihood function and that should therefore take the
form


β
λµ2
1 /2
exp {cλµ − dλ}p(µ, λ) ∝ λ exp −
2


βλ
c2
2
β/2
λ
(2.153)= exp − (µ − c/β) λ exp − d −22βwhere c, d, and β are constants. Since we can always write p(µ, λ) = p(µ|λ)p(λ),
we can ﬁnd p(µ|λ) and p(λ) by inspection. In particular, we see that p(µ|λ) is a
Gaussian whose precision is a linear function of λ and that p(λ) is a gamma distribution, so that the normalized prior takes the formp(µ, λ) = N (µ|µ0 , (βλ)−1 )Gam(λ|a, b)(2.154)

where we have deﬁned new constants given by µ0 = c/β, a = 1 + β/2, b =d−c2 /2β. The distribution (2.154) is called the normal-gamma or Gaussian-gamma
distribution and is plotted in Figure 2.14. Note that this is not simply the productof an independent Gaussian prior over µ and a gamma prior over λ, because theprecision of µ is a linear function of λ. Even if we chose a prior in which µ and λwere independent, the posterior distribution would exhibit a coupling between the
precision of µ and the value of λ.

102

2. PROBABILITY DISTRIBUTIONS
Figure 2.14

Contour plot of the normal-gamma
distribution (2.154) for parametervalues µ0 = 0, β = 2, a = 5 andb = 6.

2

λ 1

0
−2

Exercise 2.450µ2In the case of the multivariate Gaussian distribution N x|µ, Λ−1 for a Ddimensional variable x, the conjugate prior distribution for the mean µ, assuming
the precision is known, is again a Gaussian. For known mean and unknown precision
matrix Λ, the conjugate prior is the Wishart distribution given by1(ν−D−1)/2−1
exp − Tr(W Λ)(2.155)W(Λ|W, ν) = B|Λ|2
where ν is called the number of degrees of freedom of the distribution, W is a D ×D
scale matrix, and Tr(·) denotes the trace. The normalization constant B is given by
B(W, ν) = |W|−ν/2


−1
D

ν
+
1
−
i
2νD/2 π D(D−1)/4Γ.
2(2.156)i=1

Again, it is also possible to deﬁne a conjugate prior over the covariance matrix itself,
rather than over the precision matrix, which leads to the inverse Wishart distribution, although we shall not discuss this further. If both the mean and the precision
are unknown, then, following a similar line of reasoning to the univariate case, theconjugate prior is given byp(µ, Λ|µ0 , β, W, ν) = N (µ|µ0 , (βΛ)−1 ) W(Λ|W, ν)(2.157)

which is known as the normal-Wishart or Gaussian-Wishart distribution.

2.3.7 Student’s t-distribution
Section 2.3.6
Exercise 2.46

We have seen that the conjugate prior for the precision of a Gaussian is given
by a gamma distribution. If we have a univariate Gaussian N (x|µ, τ −1 ) together
with a Gamma prior Gam(τ |a, b) and we integrate out the precision, we obtain themarginal distribution of x in the form

103

2.3. The Gaussian Distribution
Figure 2.15

Plot of Student’s t-distribution (2.159)
0.5
for µ = 0 and λ = 1 for various valuesof ν. The limit ν → ∞ corresponds
to a Gaussian distribution with mean 0.4µ and precision λ.

ν→∞ν = 1.0
ν = 0.1

0.3
0.2
0.1
0
−5

 ∞
p(x|µ, a, b) =05

N (x|µ, τ −1 )Gam(τ |a, b) dτ

(2.158)

 ∞
0

=
=

 τ

ba e(−bτ ) τ a−1  τ 1/2
exp − (x − µ)2 dτ
Γ(a)
2π
2
0
 1/2 a
2 −a−1/2
1(x − µ)bb+
Γ(a + 1/2)
Γ(a) 2π2where we have made the change of variable z = τ [b + (x − µ)2 /2]. By convention
we deﬁne new parameters given by ν = 2a and λ = a/b, in terms of which the
distribution p(x|µ, a, b) takes the form
St(x|µ, λ, ν) =Exercise 2.47

Exercise 12.24

Γ(ν/2 + 1/2)
Γ(ν/2)λπν

1/2 1+

λ(x − µ)2ν−ν/2−1/2(2.159)which is known as Student’s t-distribution. The parameter λ is sometimes called theprecision of the t-distribution, even though it is not in general equal to the inverseof the variance. The parameter ν is called the degrees of freedom, and its effect is
illustrated in Figure 2.15. For the particular case of ν = 1, the t-distribution reduces
to the Cauchy distribution, while in the limit ν → ∞ the t-distribution St(x|µ, λ, ν)
becomes a Gaussian N (x|µ, λ−1 ) with mean µ and precision λ.
From (2.158), we see that Student’s t-distribution is obtained by adding up an
inﬁnite number of Gaussian distributions having the same mean but different precisions. This can be interpreted as an inﬁnite mixture of Gaussians (Gaussian mixtures
will be discussed in detail in Section 2.3.9. The result is a distribution that in general has longer ‘tails’ than a Gaussian, as was seen in Figure 2.15. This gives the tdistribution an important property called robustness, which means that it is much lesssensitive than the Gaussian to the presence of a few data points which are outliers.
The robustness of the t-distribution is illustrated in Figure 2.16, which compares the
maximum likelihood solutions for a Gaussian and a t-distribution. Note that the maximum likelihood solution for the t-distribution can be found using the expectationmaximization (EM) algorithm. Here we see that the effect of a small number of

104

2. PROBABILITY DISTRIBUTIONS

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0
−5

0

5

10

(a)

0
−5

0

5

10

(b)

Figure 2.16 Illustration of the robustness of Student’s t-distribution compared to a Gaussian. (a) Histogram
distribution of 30 data points drawn from a Gaussian distribution, together with the maximum likelihood ﬁt obtained from a t-distribution (red curve) and a Gaussian (green curve, largely hidden by the red curve). Because
the t-distribution contains the Gaussian as a special case it gives almost the same solution as the Gaussian.
(b) The same data set but with three additional outlying data points showing how the Gaussian (green curve) isstrongly distorted by the outliers, whereas the t-distribution (red curve) is relatively unaffected.

outliers is much less signiﬁcant for the t-distribution than for the Gaussian. Outliers
can arise in practical applications either because the process that generates the data
corresponds to a distribution having a heavy tail or simply through mislabelled data.
Robustness is also an important property for regression problems. Unsurprisingly,
the least squares approach to regression does not exhibit robustness, because it corresponds to maximum likelihood under a (conditional) Gaussian distribution. Bybasing a regression model on a heavy-tailed distribution such as a t-distribution, we
obtain a more robust model.
If we go back to (2.158) and substitute the alternative parameters ν = 2a, λ =a/b, and η = τ b/a, we see that the t-distribution can be written in the form ∞N x|µ, (ηλ)−1 Gam(η|ν/2, ν/2) dη.
(2.160)
St(x|µ, λ, ν) =0

We can then generalize this to a multivariate Gaussian N (x|µ, Λ) to obtain the corresponding multivariate Student’s t-distribution in the form ∞N (x|µ, (ηΛ)−1 )Gam(η|ν/2, ν/2) dη.
(2.161)
St(x|µ, Λ, ν) =0

Exercise 2.48

Using the same technique as for the univariate case, we can evaluate this integral to
give

2.3. The Gaussian Distribution−D/2−ν/2Γ(D/2 + ν/2) |Λ|1/2∆2
St(x|µ, Λ, ν) =1+
Γ(ν/2)
ν(πν)D/2105(2.162)where D is the dimensionality of x, and ∆2 is the squared Mahalanobis distancedeﬁned by(2.163)∆2 = (x − µ)T Λ(x − µ).
Exercise 2.49

This is the multivariate form of Student’s t-distribution and satisﬁes the following
properties
E[x] = µ,νΛ−1 ,cov[x] =(ν − 2)mode[x] = µ

if

ν>1

(2.164)if

ν>2(2.165)(2.166)

with corresponding results for the univariate case.

2.3.8 Periodic variables
Although Gaussian distributions are of great practical signiﬁcance, both in theirown right and as building blocks for more complex probabilistic models, there aresituations in which they are inappropriate as density models for continuous variables. One important case, which arises in practical applications, is that of periodic
variables.
An example of a periodic variable would be the wind direction at a particulargeographical location. We might, for instance, measure values of wind direction on anumber of days and wish to summarize this using a parametric distribution. Anotherexample is calendar time, where we may be interested in modelling quantities thatare believed to be periodic over 24 hours or over an annual cycle. Such quantities
can conveniently be represented using an angular (polar) coordinate 0  θ < 2π.
We might be tempted to treat periodic variables by choosing some direction
as the origin and then applying a conventional distribution such as the Gaussian.
Such an approach, however, would give results that were strongly dependent on thearbitrary choice of origin. Suppose, for instance, that we have two observations at
θ1 = 1◦ and θ2 = 359◦ , and we model them using a standard univariate Gaussian
distribution. If we choose the origin at 0◦ , then the sample mean of this data set
will be 180◦ with standard deviation 179◦ , whereas if we choose the origin at 180◦ ,then the mean will be 0◦ and the standard deviation will be 1◦ . We clearly need todevelop a special approach for the treatment of periodic variables.
Let us consider the problem of evaluating the mean of a set of observations
D = {θ1 , . . . , θN } of a periodic variable. From now on, we shall assume that θ ismeasured in radians. We have already seen that the simple average (θ1 +· · ·+θN )/N
will be strongly coordinate dependent. To ﬁnd an invariant measure of the mean, wenote that the observations can be viewed as points on the unit circle and can therefore
be described instead by two-dimensional unit vectors x1 , . . . , xN where xn  = 1
for n = 1, . . . , N , as illustrated in Figure 2.17. We can average the vectors {xn }1062. PROBABILITY DISTRIBUTIONS
Figure 2.17x2Illustration of the representation of values θn of a periodic variable as twodimensional vectors xn living on the unit
circle. Also shown is the average x of
those vectors.

x4

x3

x̄
r̄

x2

θ̄
x1

instead to give

1 
xn
N

x1

N

x=

(2.167)n=1

and then ﬁnd the corresponding angle θ of this average. Clearly, this deﬁnition will
ensure that the location of the mean is independent of the origin of the angular coordinate. Note that x will typically lie inside the unit circle. The Cartesian coordinatesof the observations are given by xn = (cos θn , sin θn ), and we can write the Cartesian coordinates of the sample mean in the form x = (r cos θ, r sin θ). Substituting
into (2.167) and equating the x1 and x2 components then gives
1 
r cos θ =cos θn ,
N
N

n=1

1 
r sin θ =sin θn .
NN(2.168)n=1

Taking the ratio, and using the identity tan θ = sin θ/ cos θ, we can solve for θ togivesin θn−1n
θ = tan.
(2.169)n cos θn
Shortly, we shall see how this result arises naturally as the maximum likelihood
estimator for an appropriately deﬁned distribution over a periodic variable.
We now consider a periodic generalization of the Gaussian called the von Mises
distribution. Here we shall limit our attention to univariate distributions, although
periodic distributions can also be found over hyperspheres of arbitrary dimension.
For an extensive discussion of periodic distributions, see Mardia and Jupp (2000).
By convention, we will consider distributions p(θ) that have period 2π. Any
probability density p(θ) deﬁned over θ must not only be nonnegative and integrate

107

2.3. The Gaussian Distribution
Figure 2.18

The von Mises distribution can be derived by considering
a two-dimensional Gaussian of the form (2.173), whose
density contours are shown in blue and conditioning on
the unit circle shown in red.

x2
p(x)

x1
r=1

to one, but it must also be periodic. Thus p(θ) must satisfy the three conditionsp(θ)  0(2.170)p(θ) dθ = 1(2.171)p(θ + 2π) = p(θ).

(2.172) 2π0From (2.172), it follows that p(θ + M 2π) = p(θ) for any integer M .
We can easily obtain a Gaussian-like distribution that satisﬁes these three properties as follows. Consider a Gaussian distribution over two variables x = (x1 , x2 )having mean µ = (µ1 , µ2 ) and a covariance matrix Σ = σ 2 I where I is the 2 × 2
identity matrix, so thatp(x1 , x2 ) =1(x1 − µ1 )2 + (x2 − µ2 )2
exp
−
2πσ 2
2σ 2

.

(2.173)The contours of constant p(x) are circles, as illustrated in Figure 2.18. Now suppose
we consider the value of this distribution along a circle of ﬁxed radius. Then by construction this distribution will be periodic, although it will not be normalized. We candetermine the form of this distribution by transforming from Cartesian coordinates(x1 , x2 ) to polar coordinates (r, θ) so thatx1 = r cos θ,

x2 = r sin θ.

(2.174)We also map the mean µ into polar coordinates by writingµ1 = r0 cos θ0 ,µ2 = r0 sin θ0 .

(2.175)

Next we substitute these transformations into the two-dimensional Gaussian distribution (2.173), and then condition on the unit circle r = 1, noting that we are interested
only in the dependence on θ. Focussing on the exponent in the Gaussian distribution
we have
1 − 2 (r cos θ − r0 cos θ0 )2 + (r sin θ − r0 sin θ0 )22σ
1 = − 2 1 + r02 − 2r0 cos θ cos θ0 − 2r0 sin θ sin θ02σr0=
cos(θ − θ0 ) + const(2.176)σ2108

2. PROBABILITY DISTRIBUTIONS

π/4

m = 5, θ0 = π/4

3π/4m = 1, θ0 = 3π/40
2πm = 5, θ0 = π/4m = 1, θ0 = 3π/4Figure 2.19 The von Mises distribution plotted for two different parameter values, shown as a Cartesian plot
on the left and as the corresponding polar plot on the right.

Exercise 2.51

where ‘const’ denotes terms independent of θ, and we have made use of the following
trigonometrical identities
cos2 A + sin2 A = 1
cos A cos B + sin A sin B = cos(A − B).

(2.177)(2.178)If we now deﬁne m = r0 /σ 2 , we obtain our ﬁnal expression for the distribution ofp(θ) along the unit circle r = 1 in the formp(θ|θ0 , m) =1
exp {m cos(θ − θ0 )}2πI0 (m)(2.179)which is called the von Mises distribution, or the circular normal. Here the parameter θ0 corresponds to the mean of the distribution, while m, which is known as
the concentration parameter, is analogous to the inverse variance (precision) for theGaussian. The normalization coefﬁcient in (2.179) is expressed in terms of I0 (m),which is the zeroth-order Bessel function of the ﬁrst kind (Abramowitz and Stegun,1965) and is deﬁned by 2π
1
exp {m cos θ} dθ.
(2.180)I0 (m) =2π 0Exercise 2.52

For large m, the distribution becomes approximately Gaussian. The von Mises distribution is plotted in Figure 2.19, and the function I0 (m) is plotted in Figure 2.20.
Now consider the maximum likelihood estimators for the parameters θ0 and m
for the von Mises distribution. The log likelihood function is given byln p(D|θ0 , m) = −N ln(2π) − N ln I0 (m) + m

N

n=1

cos(θn − θ0 ).

(2.181)

2.3. The Gaussian Distribution
3000

109

1

2000
I0 (m)

A(m) 0.5
1000

0Figure 2.20
(2.186).

0

5
m

0

10

0

5
m

10

Plot of the Bessel function I0 (m) deﬁned by (2.180), together with the function A(m) deﬁned by

Setting the derivative with respect to θ0 equal to zero givesN

sin(θn − θ0 ) = 0.

(2.182)n=1

To solve for θ0 , we make use of the trigonometric identitysin(A − B) = cos B sin A − cos A sin B
Exercise 2.53

(2.183)
sin θnn(2.184)n cos θnwhich we recognize as the result (2.169) obtained earlier for the mean of the observations viewed in a two-dimensional Cartesian space.
Similarly, maximizing (2.181) with respect to m, and making use of I0 (m) =I1 (m) (Abramowitz and Stegun, 1965), we havefrom which we obtain

θ0ML = tan−11 cos(θn − θ0ML )N
N

A(m) =(2.185)n=1

where we have substituted for the maximum likelihood solution for θ0ML (recallingthat we are performing a joint optimization over θ and m), and we have deﬁned
A(m) =I1 (m).
I0 (m)(2.186)

The function A(m) is plotted in Figure 2.20. Making use of the trigonometric identity (2.178), we can write (2.185) in the form




N
N
1 
1 
ML
A(mML ) =cos θn cos θ0 −sin θn sin θ0ML .
(2.187)
N
N
n=1

n=1

110

2. PROBABILITY DISTRIBUTIONS

Figure 2.21 Plots of the ‘old faithful’ data in which the blue curves
show contours of constant probability density.
On the left is asingle Gaussian distribution which
has been ﬁtted to the data using maximum likelihood. Note that
this distribution fails to capture thetwo clumps in the data and indeed
places much of its probability massin the central region between theclumps where the data are relatively
sparse. On the right the distribution
is given by a linear combination of
two Gaussians which has been ﬁtted
to the data by maximum likelihoodusing techniques discussed Chapter 9, and which gives a better representation of the data.

100

100

80

80

60

60

40

1

2

3

4

5

6

40

1

2

3

4

5

6

The right-hand side of (2.187) is easily evaluated, and the function A(m) can beinverted numerically.
For completeness, we mention brieﬂy some alternative techniques for the construction of periodic distributions. The simplest approach is to use a histogram ofobservations in which the angular coordinate is divided into ﬁxed bins. This has thevirtue of simplicity and ﬂexibility but also suffers from signiﬁcant limitations, as weshall see when we discuss histogram methods in more detail in Section 2.5. Anotherapproach starts, like the von Mises distribution, from a Gaussian distribution over a
Euclidean space but now marginalizes onto the unit circle rather than conditioning(Mardia and Jupp, 2000). However, this leads to more complex forms of distribution
and will not be discussed further. Finally, any valid distribution over the real axis
(such as a Gaussian) can be turned into a periodic distribution by mapping successive intervals of width 2π onto the periodic variable (0, 2π), which corresponds to‘wrapping’ the real axis around unit circle. Again, the resulting distribution is more
complex to handle than the von Mises distribution.
One limitation of the von Mises distribution is that it is unimodal. By forming
mixtures of von Mises distributions, we obtain a ﬂexible framework for modelling
periodic variables that can handle multimodality. For an example of a machine learning application that makes use of von Mises distributions, see Lawrence et al. (2002),
and for extensions to modelling conditional densities for regression problems, see
Bishop and Nabney (1996).

2.3.9 Mixtures of Gaussians

Appendix A

While the Gaussian distribution has some important analytical properties, it suffers from signiﬁcant limitations when it comes to modelling real data sets. Consider
the example shown in Figure 2.21. This is known as the ‘Old Faithful’ data set,and comprises 272 measurements of the eruption of the Old Faithful geyser at Yellowstone National Park in the USA. Each measurement comprises the duration of2.3. The Gaussian Distribution
Figure 2.22

111

Example of a Gaussian mixture distribution p(x)in one dimension showing three Gaussians(each scaled by a coefﬁcient) in blue andtheir sum in red.

x

the eruption in minutes (horizontal axis) and the time in minutes to the next eruption (vertical axis). We see that the data set forms two dominant clumps, and thata simple Gaussian distribution is unable to capture this structure, whereas a linear
superposition of two Gaussians gives a better characterization of the data set.
Such superpositions, formed by taking linear combinations of more basic distributions such as Gaussians, can be formulated as probabilistic models known as
mixture distributions (McLachlan and Basford, 1988; McLachlan and Peel, 2000).
In Figure 2.22 we see that a linear combination of Gaussians can give rise to very
complex densities. By using a sufﬁcient number of Gaussians, and by adjusting theirmeans and covariances as well as the coefﬁcients in the linear combination, almost
any continuous density can be approximated to arbitrary accuracy.
We therefore consider a superposition of K Gaussian densities of the form
p(x) =K

πk N (x|µk , Σk )(2.188)k=1

Section 9.3.3

which is called a mixture of Gaussians. Each Gaussian density N (x|µk , Σk ) is
called a component of the mixture and has its own mean µk and covariance Σk .
Contour and surface plots for a Gaussian mixture having 3 components are shown in
Figure 2.23.
In this section we shall consider Gaussian components to illustrate the framework of mixture models. More generally, mixture models can comprise linear combinations of other distributions. For instance, in Section 9.3.3 we shall consider
mixtures of Bernoulli distributions as an example of a mixture model for discretevariables.
The parameters πk in (2.188) are called mixing coefﬁcients. If we integrate bothsides of (2.188) with respect to x, and note that both p(x) and the individual Gaussiancomponents are normalized, we obtainK

πk = 1.

(2.189)k=1

Also, the requirement that p(x)  0, together with N (x|µk , Σk )  0, implies
πk  0 for all k. Combining this with the condition (2.189) we obtain0  πk  1.

(2.190)112

1

2. PROBABILITY DISTRIBUTIONS

1

(a)

0.5

0.2

(b)

0.5

0.3

0.5
0

0
0

0.5

1

0

0.5

1

Figure 2.23 Illustration of a mixture of 3 Gaussians in a two-dimensional space. (a) Contours of constantdensity for each of the mixture components, in which the 3 components are denoted red, blue and green, andthe values of the mixing coefﬁcients are shown below each component. (b) Contours of the marginal probability
density p(x) of the mixture distribution. (c) A surface plot of the distribution p(x).

We therefore see that the mixing coefﬁcients satisfy the requirements to be probabilities.
From the sum and product rules, the marginal density is given byp(x) =

K

p(k)p(x|k)(2.191)k=1

which is equivalent to (2.188) in which we can view πk = p(k) as the prior probability of picking the k th component, and the density N (x|µk , Σk ) = p(x|k) as
the probability of x conditioned on k. As we shall see in later chapters, an important role is played by the posterior probabilities p(k|x), which are also known as
responsibilities. From Bayes’ theorem these are given by
γk (x) ≡ p(k|x)p(k)p(x|k)= l p(l)p(x|l)πk N (x|µk , Σk ).
= l πl N (x|µl , Σl )(2.192)We shall discuss the probabilistic interpretation of the mixture distribution in greater
detail in Chapter 9.
The form of the Gaussian mixture distribution is governed by the parameters π,µ and Σ, where we have used the notation π ≡ {π1 , . . . , πK }, µ ≡ {µ1 , . . . , µK }and Σ ≡ {Σ1 , . . . ΣK }. One way to set the values of these parameters is to use
maximum likelihood. From (2.188) the log of the likelihood function is given by
K

N


ln p(X|π, µ, Σ) =ln
πk N (xn |µk , Σk )(2.193)n=1

k=1

2.4. The Exponential Family

113

where X = {x1 , . . . , xN }. We immediately see that the situation is now much
more complex than with a single Gaussian, due to the presence of the summation
over k inside the logarithm. As a result, the maximum likelihood solution for theparameters no longer has a closed-form analytical solution. One approach to maximizing the likelihood function is to use iterative numerical optimization techniques(Fletcher, 1987; Nocedal and Wright, 1999; Bishop and Nabney, 2008). Alternatively we can employ a powerful framework called expectation maximization, which
will be discussed at length in Chapter 9.

2.4. The Exponential Family
The probability distributions that we have studied so far in this chapter (with the
exception of the Gaussian mixture) are speciﬁc examples of a broad class of distributions called the exponential family (Duda and Hart, 1973; Bernardo and Smith,1994). Members of the exponential family have many important properties in common, and it is illuminating to discuss these properties in some generality.
The exponential family of distributions over x, given parameters η, is deﬁned tobe the set of distributions of the form

(2.194)p(x|η) = h(x)g(η) exp η T u(x)where x may be scalar or vector, and may be discrete or continuous. Here η arecalled the natural parameters of the distribution, and u(x) is some function of x.
The function g(η) can be interpreted as the coefﬁcient that ensures that the distribution is normalized and therefore satisﬁes(2.195)g(η) h(x) exp η T u(x) dx = 1
where the integration is replaced by summation if x is a discrete variable.
We begin by taking some examples of the distributions introduced earlier inthe chapter and showing that they are indeed members of the exponential family.
Consider ﬁrst the Bernoulli distributionp(x|µ) = Bern(x|µ) = µx (1 − µ)1−x .

(2.196)

Expressing the right-hand side as the exponential of the logarithm, we havep(x|µ) = exp {x ln µ + (1 − x) ln(1 − µ)}µ= (1 − µ) exp ln
x .
1−µComparison with (2.194) allows us to identifyµη = ln1−µ(2.197)(2.198)

114

2. PROBABILITY DISTRIBUTIONS
which we can solve for µ to give µ = σ(η), whereσ(η) =

1
1 + exp(−η)(2.199)is called the logistic sigmoid function. Thus we can write the Bernoulli distribution
using the standard representation (2.194) in the formp(x|η) = σ(−η) exp(ηx)(2.200)where we have used 1 − σ(η) = σ(−η), which is easily proved from (2.199). Comparison with (2.194) shows thatu(x) = xh(x) = 1
g(η) = σ(−η).

(2.201)(2.202)(2.203)

Next consider the multinomial distribution that, for a single observation x, takes
the formMMp(x|µ) =µxkk = exp
xk ln µk(2.204)k=1

k=1

where x = (x1 , . . . , xN ) . Again, we can write this in the standard representation(2.194) so that
(2.205)p(x|η) = exp(η T x)
T

where ηk = ln µk , and we have deﬁned η = (η1 , . . . , ηM )T . Again, comparing with
(2.194) we haveu(x) = xh(x) = 1
g(η) = 1.

(2.206)(2.207)(2.208)Note that the parameters ηk are not independent because the parameters µk are subject to the constraintMµk = 1(2.209)k=1

so that, given any M − 1 of the parameters µk , the value of the remaining parameter
is ﬁxed. In some circumstances, it will be convenient to remove this constraint byexpressing the distribution in terms of only M − 1 parameters. This can be achievedby using the relationship (2.209) to eliminate µM by expressing it in terms of theremaining {µk } where k = 1, . . . , M − 1, thereby leaving M − 1 parameters. Notethat these remaining parameters are still subject to the constraints
0  µk  1,

M
−1

k=1

µk  1.

(2.210)2.4. The Exponential Family

115

Making use of the constraint (2.209), the multinomial distribution in this representation then becomes
M


exp
xk ln µk
k=1

= exp

M −1




= exp

M −1


1−

xk ln µk +

k=1

M
−1

k=1


xk ln

1−

k=1

We now identify

µk
M −1
j =1

xk


ln 1 −



µj


ln



M
−1



µk

k=1


+ ln 1 −

M
−1



µk

. (2.211)

k=1


1−

µk


j µj

= ηk

(2.212)

which we can solve for µk by ﬁrst summing both sides over k and then rearranging
and back-substituting to giveµk =exp(ηk )
.
1 + j exp(ηj )(2.213)This is called the softmax function, or the normalized exponential. In this representation, the multinomial distribution therefore takes the form

−1
M
−1

exp(ηk )exp(η T x).
(2.214)p(x|η) = 1 +k=1This is the standard form of the exponential family, with parameter vector η =(η1 , . . . , ηM −1 )T in whichu(x) = xh(x) = 1
g(η) =

1+

M−1


(2.215)(2.216)−1exp(ηk ).

(2.217)

k=1

Finally, let us consider the Gaussian distribution. For the univariate Gaussian,
we havep(x|µ, σ 2 ) ==1
1
exp − 2 (x − µ)2
2
1
/
2
2σ
(2πσ )
11µ
1
exp − 2 x2 + 2 x − 2 µ22σσ2σ(2πσ 2 )1/2

(2.218)(2.219)116

2. PROBABILITY DISTRIBUTIONS

Exercise 2.57

which, after some simple rearrangement, can be cast in the standard exponential
family form (2.194) with


µ/σ 2
η =
(2.220)−1/2σ 2 
xu(x) =(2.221)x2
h(x) = (2π)−1/2g(η) = (−2η2 )1/2 exp

η12(2.222)

4η2.

(2.223)2.4.1 Maximum likelihood and sufﬁcient statistics
Let us now consider the problem of estimating the parameter vector η in the general exponential family distribution (2.194) using the technique of maximum likelihood. Taking the gradient of both sides of (2.195) with respect to η, we have
∇g(η) h(x) exp η T u(x) dx+ g(η) h(x) exp η T u(x) u(x) dx = 0.
(2.224)
Rearranging, and making use again of (2.195) then gives



1
−
∇g(η) = g(η) h(x) exp η T u(x) u(x) dx = E[u(x)]g(η)(2.225)where we have used (2.194). We therefore obtain the result−∇ ln g(η) = E[u(x)].
Exercise 2.58

(2.226)Note that the covariance of u(x) can be expressed in terms of the second derivativesof g(η), and similarly for higher order moments. Thus, provided we can normalize a
distribution from the exponential family, we can always ﬁnd its moments by simple
differentiation.
Now consider a set of independent identically distributed data denoted by X ={x1 , . . . , xn }, for which the likelihood function is given by
N



N


h(xn ) g(η)N exp η Tu(xn ) .
(2.227)p(X|η) =n=1

n=1

Setting the gradient of ln p(X|η) with respect to η to zero, we get the following
condition to be satisﬁed by the maximum likelihood estimator η ML
1 u(xn )
NN−∇ ln g(η ML ) =n=1(2.228)

2.4. The Exponential Family

117

which can in principle be solved to obtain η ML . We see that thesolution for the
maximum likelihood estimator depends on the data only through n u(xn ), whichis therefore called the sufﬁcient statistic of the distribution (2.194). We do not needto store the entire data set itself but only the value of the sufﬁcient statistic. For
the Bernoulli distribution, for example, the function u(x) is given just by x andso we need only keep the sum of the data points {xn }, whereas for the Gaussianu(x) = (x, x2 )T , and so we should keep both the sum of {xn } and the sum of {x2n }.
If we consider the limit N → ∞, then the right-hand side of (2.228) becomesE[u(x)], and so by comparing with (2.226) we see that in this limit η ML will equal
the true value η.
In fact, this sufﬁciency property holds also for Bayesian inference, although
we shall defer discussion of this until Chapter 8 when we have equipped ourselveswith the tools of graphical models and can thereby gain a deeper insight into theseimportant concepts.

2.4.2 Conjugate priors
We have already encountered the concept of a conjugate prior several times, forexample in the context of the Bernoulli distribution (for which the conjugate prioris the beta distribution) or the Gaussian (where the conjugate prior for the mean isa Gaussian, and the conjugate prior for the precision is the Wishart distribution). In
general, for a given probability distribution p(x|η), we can seek a prior p(η) that isconjugate to the likelihood function, so that the posterior distribution has the samefunctional form as the prior. For any member of the exponential family (2.194), there
exists a conjugate prior that can be written in the form

(2.229)p(η|χ, ν) = f (χ, ν)g(η)ν exp νη T χ
where f (χ, ν) is a normalization coefﬁcient, and g(η) is the same function as appears in (2.194). To see that this is indeed conjugate, let us multiply the prior (2.229)
by the likelihood function (2.227) to obtain the posterior distribution, up to a normalization coefﬁcient, in the form
 N


u(xn ) + νχ.
(2.230)p(η|X, χ, ν) ∝ g(η)ν +N exp η T
n=1

This again takes the same functional form as the prior (2.229), conﬁrming conjugacy.
Furthermore, we see that the parameter ν can be interpreted as a effective number ofpseudo-observations in the prior, each of which has a value for the sufﬁcient statistic
u(x) given by χ.

2.4.3 Noninformative priors
In some applications of probabilistic inference, we may have prior knowledge
that can be conveniently expressed through the prior distribution. For example, if
the prior assigns zero probability to some value of variable, then the posterior distribution will necessarily also assign zero probability to that value, irrespective of

118

2. PROBABILITY DISTRIBUTIONS
any subsequent observations of data. In many cases, however, we may have little
idea of what form the distribution should take. We may then seek a form of priordistribution, called a noninformative prior, which is intended to have as little inﬂuence on the posterior distribution as possible (Jeffries, 1946; Box and Tao, 1973;
Bernardo and Smith, 1994). This is sometimes referred to as ‘letting the data speak
for themselves’.
If we have a distribution p(x|λ) governed by a parameter λ, we might be tempted
to propose a prior distribution p(λ) = const as a suitable prior. If λ is a discretevariable with K states, this simply amounts to setting the prior probability of eachstate to 1/K. In the case of continuous parameters, however, there are two potential
difﬁculties with this approach. The ﬁrst is that, if the domain of λ is unbounded,this prior distribution cannot be correctly normalized because the integral over λdiverges. Such priors are called improper. In practice, improper priors can often
be used provided the corresponding posterior distribution is proper, i.e., that it canbe correctly normalized. For instance, if we put a uniform prior distribution overthe mean of a Gaussian, then the posterior distribution for the mean, once we haveobserved at least one data point, will be proper.
A second difﬁculty arises from the transformation behaviour of a probability
density under a nonlinear change of variables, given by (1.27). If a function h(λ)
is constant, and we change variables to λ = η 2 , then h(η) = h(η 2 ) will also beconstant. However, if we choose the density pλ (λ) to be constant, then the densityof η will be given, from (1.27), by  dλ 
 = pλ (η 2 )2η ∝ η(2.231)pη (η) = pλ (λ) dη and so the density over η will not be constant. This issue does not arise when we use
maximum likelihood, because the likelihood function p(x|λ) is a simple function of
λ and so we are free to use any convenient parameterization. If, however, we are tochoose a prior distribution that is constant, we must take care to use an appropriate
representation for the parameters.
Here we consider two simple examples of noninformative priors (Berger, 1985).
First of all, if a density takes the form
p(x|µ) = f (x − µ)(2.232)then the parameter µ is known as a location parameter. This family of densities
x = x + c,
exhibits translation invariance because if we shift x by a constant to give thenx|µ) = f (x−µ)(2.233)p( = µ + c. Thus the density takes the same form in thewhere we have deﬁned µ
new variable as in the original one, and so the density is independent of the choiceof origin. We would like to choose a prior distribution that reﬂects this translation
invariance property, and so we choose a prior that assigns equal probability mass to2.4. The Exponential Family

119

an interval A  µ  B as to the shifted interval A − c  µ  B − c. This implies
 B
 B−c
 B
p(µ) dµ =p(µ) dµ =p(µ − c) dµ(2.234)AA−cA

and because this must hold for all choices of A and B, we havep(µ − c) = p(µ)(2.235)which implies that p(µ) is constant. An example of a location parameter would be
the mean µ of a Gaussian distribution. As we have seen, the conjugate prior distribution for µ in this case is a Gaussian p(µ|µ0 , σ02 ) = N (µ|µ0 , σ02 ), and we obtain anoninformative prior by taking the limit σ02 → ∞. Indeed, from (2.141) and (2.142)
we see that this gives a posterior distribution over µ in which the contributions fromthe prior vanish.
As a second example, consider a density of the form
1 xp(x|σ) = f(2.236)σσ
Exercise 2.59

where σ > 0. Note that this will be a normalized density provided f (x) is correctly
normalized. The parameter σ is known as a scale parameter, and the density exhibits
x = cx, thenscale invariance because if we scale x by a constant to give 
 
1
x

x|σ
) = f(2.237)p(σσwhere we have deﬁned σ = cσ. This transformation corresponds to a change ofscale, for example from meters to kilometers if x is a length, and we would liketo choose a prior distribution that reﬂects this scale invariance. If we consider aninterval A  σ  B, and a scaled interval A/c  σ  B/c, then the prior should
assign equal probability mass to these two intervals. Thus we have
 B
 B/c
 B  1
1p(σ) dσ =p(σ) dσ =pσ
dσ(2.238)cc
A
A/cA
and because this must hold for choices of A and B, we have 
1
1p(σ) = pσcc(2.239)and hence p(σ) ∝ 1/σ. Note that again this is an improper prior because the integralof the distribution over 0  σ  ∞ is divergent. It is sometimes also convenientto think of the prior distribution for a scale parameter in terms of the density of thelog of the parameter. Using the transformation rule (1.27) for densities we see thatp(ln σ) = const. Thus, for this prior there is the same probability mass in the range
1  σ  10 as in the range 10  σ  100 and in 100  σ  1000.

120

2. PROBABILITY DISTRIBUTIONS
An example of a scale parameter would be the standard deviation σ of a Gaussian
distribution, after we have taken account of the location parameter µ, because
N (x|µ, σ 2 ) ∝ σ −1 exp −(x/σ)2(2.240)Section 2.3

where x = x − µ. As discussed earlier, it is often more convenient to work in termsof the precision λ = 1/σ 2 rather than σ itself. Using the transformation rule fordensities, we see that a distribution p(σ) ∝ 1/σ corresponds to a distribution over λof the form p(λ) ∝ 1/λ. We have seen that the conjugate prior for λ was the gamma
distribution Gam(λ|a0 , b0 ) given by (2.146). The noninformative prior is obtained
as the special case a0 = b0 = 0. Again, if we examine the results (2.150) and (2.151)
for the posterior distribution of λ, we see that for a0 = b0 = 0, the posterior depends
only on terms arising from the data and not from the prior.

2.5. Nonparametric Methods
Throughout this chapter, we have focussed on the use of probability distributions
having speciﬁc functional forms governed by a small number of parameters whose
values are to be determined from a data set. This is called the parametric approachto density modelling. An important limitation of this approach is that the chosendensity might be a poor model of the distribution that generates the data, which can
result in poor predictive performance. For instance, if the process that generates thedata is multimodal, then this aspect of the distribution can never be captured by aGaussian, which is necessarily unimodal.
In this ﬁnal section, we consider some nonparametric approaches to density estimation that make few assumptions about the form of the distribution. Here we shallfocus mainly on simple frequentist methods. The reader should be aware, however,that nonparametric Bayesian methods are attracting increasing interest (Walker et al.,
1999; Neal, 2000; Müller and Quintana, 2004; Teh et al., 2006).
Let us start with a discussion of histogram methods for density estimation, which
we have already encountered in the context of marginal and conditional distributionsin Figure 1.11 and in the context of the central limit theorem in Figure 2.6. Here we
explore the properties of histogram density models in more detail, focussing on thecase of a single continuous variable x. Standard histograms simply partition x into
distinct bins of width ∆i and then count the number ni of observations of x fallingin bin i. In order to turn this count into a normalized probability density, we simply
divide by the total number N of observations and by the width ∆i of the bins to
obtain probability values for each bin given by
pi =

ni
N ∆i

(2.241)

for which it is easily seen that p(x) dx = 1. This gives a model for the densityp(x) that is constant over the width of each bin, and often the bins are chosen to havethe same width ∆i = ∆.

2.5. Nonparametric Methods
Figure 2.24

An illustration of the histogram approach 5
to density estimation, in which a data set∆ = 0.04
of 50 data points is generated from the
distribution shown by the green curve.
Histogram density estimates, based on 0 0(2.241), with a common bin width ∆ are 5∆ = 0.08
shown for various values of ∆.

0
50Section 1.4

0

121

0.5

1

0.5

1

0.5

1

∆ = 0.25

0

In Figure 2.24, we show an example of histogram density estimation. Herethe data is drawn from the distribution, corresponding to the green curve, which isformed from a mixture of two Gaussians. Also shown are three examples of histogram density estimates corresponding to three different choices for the bin width∆. We see that when ∆ is very small (top ﬁgure), the resulting density model is very
spiky, with a lot of structure that is not present in the underlying distribution thatgenerated the data set. Conversely, if ∆ is too large (bottom ﬁgure) then the result is
a model that is too smooth and that consequently fails to capture the bimodal property of the green curve. The best results are obtained for some intermediate valueof ∆ (middle ﬁgure). In principle, a histogram density model is also dependent on
the choice of edge location for the bins, though this is typically much less signiﬁcantthan the value of ∆.
Note that the histogram method has the property (unlike the methods to be discussed shortly) that, once the histogram has been computed, the data set itself canbe discarded, which can be advantageous if the data set is large. Also, the histogramapproach is easily applied if the data points are arriving sequentially.
In practice, the histogram technique can be useful for obtaining a quick visualization of data in one or two dimensions but is unsuited to most density estimation
applications. One obvious problem is that the estimated density has discontinuities
that are due to the bin edges rather than any property of the underlying distributionthat generated the data. Another major limitation of the histogram approach is its
scaling with dimensionality. If we divide each variable in a D-dimensional space
into M bins, then the total number of bins will be M D . This exponential scaling
with D is an example of the curse of dimensionality. In a space of high dimensionality, the quantity of data needed to provide meaningful estimates of local probability
density would be prohibitive.
The histogram approach to density estimation does, however, teach us two important lessons. First, to estimate the probability density at a particular location,
we should consider the data points that lie within some local neighbourhood of thatpoint. Note that the concept of locality requires that we assume some form of distance measure, and here we have been assuming Euclidean distance. For histograms,

122

2. PROBABILITY DISTRIBUTIONS
this neighbourhood property was deﬁned by the bins, and there is a natural ‘smoothing’ parameter describing the spatial extent of the local region, in this case the bin
width. Second, the value of the smoothing parameter should be neither too large nor
too small in order to obtain good results. This is reminiscent of the choice of model
complexity in polynomial curve ﬁtting discussed in Chapter 1 where the degree Mof the polynomial, or alternatively the value α of the regularization parameter, wasoptimal for some intermediate value, neither too large nor too small. Armed withthese insights, we turn now to a discussion of two widely used nonparametric techniques for density estimation, kernel estimators and nearest neighbours, which havebetter scaling with dimensionality than the simple histogram model.

2.5.1 Kernel density estimators
Let us suppose that observations are being drawn from some unknown probability density p(x) in some D-dimensional space, which we shall take to be Euclidean,
and we wish to estimate the value of p(x). From our earlier discussion of locality,
let us consider some small region R containing x. The probability mass associatedwith this region is given byp(x) dx.
(2.242)P =R

Section 2.1

Now suppose that we have collected a data set comprising N observations drawn
from p(x). Because each data point has a probability P of falling within R, the total
number K of points that lie inside R will be distributed according to the binomial
distributionN!
(2.243)
P K (1 − P )1−K .
Bin(K|N, P ) =K!(N − K)!
Using (2.11), we see that the mean fraction of points falling inside the region is
E[K/N ] = P , and similarly using (2.12) we see that the variance around this mean
is var[K/N ] = P (1 − P )/N . For large N , this distribution will be sharply peakedaround the mean and so
K  N P.
(2.244)If, however, we also assume that the region R is sufﬁciently small that the probability
density p(x) is roughly constant over the region, then we haveP  p(x)V(2.245)where V is the volume of R. Combining (2.244) and (2.245), we obtain our density
estimate in the form
K.
(2.246)p(x) =
NV
Note that the validity of (2.246) depends on two contradictory assumptions, namelythat the region R be sufﬁciently small that the density is approximately constant over
the region and yet sufﬁciently large (in relation to the value of that density) that the
number K of points falling inside the region is sufﬁcient for the binomial distribution
to be sharply peaked.

2.5. Nonparametric Methods

123

We can exploit the result (2.246) in two different ways. Either we can ﬁx K anddetermine the value of V from the data, which gives rise to the K-nearest-neighbour
technique discussed shortly, or we can ﬁx V and determine K from the data, giving rise to the kernel approach. It can be shown that both the K-nearest-neighbour
density estimator and the kernel density estimator converge to the true probability
density in the limit N → ∞ provided V shrinks suitably with N , and K grows withN (Duda and Hart, 1973).
We begin by discussing the kernel method in detail, and to start with we takethe region R to be a small hypercube centred on the point x at which we wish to
determine the probability density. In order to count the number K of points fallingwithin this region, it is convenient to deﬁne the following function
k(u) =

1, |ui |  1/2,
0, otherwisei = 1, . . . , D,

(2.247)

which represents a unit cube centred on the origin. The function k(u) is an example
of a kernel function, and in this context is also called a Parzen window. From (2.247),
the quantity k((x − xn )/h) will be one if the data point xn lies inside a cube of sideh centred on x, and zero otherwise. The total number of data points lying inside thiscube will therefore beNx − x nk.
(2.248)K=hn=1

Substituting this expression into (2.246) then gives the following result for the estimated density at x
N
1  1  x − xn p(x) =
k(2.249)
N
hD
h
n=1

D

where we have used V = h for the volume of a hypercube of side h in D dimensions. Using the symmetry of the function k(u), we can now re-interpret thisequation, not as a single cube centred on x but as the sum over N cubes centred on
the N data points xn .
As it stands, the kernel density estimator (2.249) will suffer from one of the sameproblems that the histogram method suffered from, namely the presence of artiﬁcial
discontinuities, in this case at the boundaries of the cubes. We can obtain a smootherdensity model if we choose a smoother kernel function, and a common choice is theGaussian, which gives rise to the following kernel density model1 1
x − xn 2
exp
−
N
2h2
(2πh2 )1/2
N

p(x) =(2.250)n=1

where h represents the standard deviation of the Gaussian components. Thus ourdensity model is obtained by placing a Gaussian over each data point and then adding
up the contributions over the whole data set, and then dividing by N so that the density is correctly normalized. In Figure 2.25, we apply the model (2.250) to the data

124

2. PROBABILITY DISTRIBUTIONS
Figure 2.25

Illustration of the kernel density model(2.250) applied to the same data set used
to demonstrate the histogram approach in
Figure 2.24. We see that h acts as asmoothing parameter and that if it is set
too small (top panel), the result is a very
noisy density model, whereas if it is set
too large (bottom panel), then the bimodal
nature of the underlying distribution fromwhich the data is generated (shown by thegreen curve) is washed out. The best density model is obtained for some intermediate value of h (middle panel).

5

0
5

0
5

0

h = 0.005

0

0.5

1

0.5

1

0.5

1

h = 0.070
h = 0.20

set used earlier to demonstrate the histogram technique. We see that, as expected,
the parameter h plays the role of a smoothing parameter, and there is a trade-off
between sensitivity to noise at small h and over-smoothing at large h. Again, theoptimization of h is a problem in model complexity, analogous to the choice of bin
width in histogram density estimation, or the degree of the polynomial used in curve
ﬁtting.
We can choose any other kernel function k(u) in (2.249) subject to the conditionsk(u)  0,(2.251)k(u) du = 1(2.252)

which ensure that the resulting probability distribution is nonnegative everywhere
and integrates to one. The class of density model given by (2.249) is called a kernel
density estimator, or Parzen estimator. It has a great merit that there is no computation involved in the ‘training’ phase because this simply requires storage of the
training set. However, this is also one of its great weaknesses because the computational cost of evaluating the density grows linearly with the size of the data set.

2.5.2 Nearest-neighbour methods
One of the difﬁculties with the kernel approach to density estimation is that theparameter h governing the kernel width is ﬁxed for all kernels. In regions of highdata density, a large value of h may lead to over-smoothing and a washing out ofstructure that might otherwise be extracted from the data. However, reducing h maylead to noisy estimates elsewhere in data space where the density is smaller. Thus
the optimal choice for h may be dependent on location within the data space. Thisissue is addressed by nearest-neighbour methods for density estimation.
We therefore return to our general result (2.246) for local density estimation,and instead of ﬁxing V and determining the value of K from the data, we consider
a ﬁxed value of K and use the data to ﬁnd an appropriate value for V . To do this,we consider a small sphere centred on the point x at which we wish to estimate the2.5. Nonparametric Methods
Figure 2.26

Illustration of K-nearest-neighbour density estimation using the same data set
as in Figures 2.25 and 2.24. We seethat the parameter K governs the degreeof smoothing, so that a small value ofK leads to a very noisy density model(top panel), whereas a large value (bottom panel) smoothes out the bimodal nature of the true distribution (shown by thegreen curve) from which the data set was
generated.

5

0
5

0
5

0

Exercise 2.61

125

K =1

0

0.5

1

0.5

1

0.5

1

K =5

0
K = 300density p(x), and we allow the radius of the sphere to grow until it contains precisely
K data points. The estimate of the density p(x) is then given by (2.246) with V set tothe volume of the resulting sphere. This technique is known as K nearest neighbours
and is illustrated in Figure 2.26, for various choices of the parameter K, using thesame data set as used in Figure 2.24 and Figure 2.25. We see that the value of K
now governs the degree of smoothing and that again there is an optimum choice forK that is neither too large nor too small. Note that the model produced by K nearest
neighbours is not a true density model because the integral over all space diverges.
We close this chapter by showing how the K-nearest-neighbour technique fordensity estimation can be extended to the problem of classiﬁcation. To do this, weapply the K-nearest-neighbour density estimation technique to each class separately
and then make use of Bayes’ theorem. Let us suppose that wehave a data set comprising Nk points in class Ck with N points in total, so that k Nk = N . If wewish to classify a new point x, we draw a sphere centred on x containing precisely
K points irrespective of their class. Suppose this sphere has volume V and contains
Kk points from class Ck . Then (2.246) provides an estimate of the density associatedwith each classKk
p(x|Ck ) =.
(2.253)
Nk V
Similarly, the unconditional density is given byp(x) =

K
NV(2.254)p(Ck ) =Nk.
N(2.255)

while the class priors are given by

We can now combine (2.253), (2.254), and (2.255) using Bayes’ theorem to obtain
the posterior probability of class membershipp(Ck |x) =Kk
p(x|Ck )p(Ck )
=
.
p(x)
K

(2.256)126

2. PROBABILITY DISTRIBUTIONS

Figure 2.27 (a) In the K-nearestneighbour classiﬁer, a new point,
shown by the black diamond, is classiﬁed according to the majority classmembership of the K closest training data points, in this case K =3.
(b) In the nearest-neighbour(K = 1) approach to classiﬁcation,
the resulting decision boundary iscomposed of hyperplanes that form
perpendicular bisectors of pairs of
points from different classes.

x2

x2

x1

x1

(a)

(b)

If we wish to minimize the probability of misclassiﬁcation, this is done by assigning
the test point x to the class having the largest posterior probability, corresponding to
the largest value of Kk /K. Thus to classify a new point, we identify the K nearest
points from the training data set and then assign the new point to the class having thelargest number of representatives amongst this set. Ties can be broken at random.
The particular case of K = 1 is called the nearest-neighbour rule, because a test
point is simply assigned to the same class as the nearest point from the training set.
These concepts are illustrated in Figure 2.27.
In Figure 2.28, we show the results of applying the K-nearest-neighbour algorithm to the oil ﬂow data, introduced in Chapter 1, for various values of K. As
expected, we see that K controls the degree of smoothing, so that small K produces
many small regions of each class, whereas large K leads to fewer larger regions.

K =1

K =3

2

K = 31

2

x7

2

x7
1

0

x7
1

0

1

x6

2

0

1

0

1

x6

2

0

0

1

x6

2

Figure 2.28 Plot of 200 data points from the oil data set showing values of x6 plotted against x7 , where thered, green, and blue points correspond to the ‘laminar’, ‘annular’, and ‘homogeneous’ classes, respectively. Also
shown are the classiﬁcations of the input space given by the K-nearest-neighbour algorithm for various valuesof K.

Exercises

127

An interesting property of the nearest-neighbour (K = 1) classiﬁer is that, in thelimit N → ∞, the error rate is never more than twice the minimum achievable error
rate of an optimal classiﬁer, i.e., one that uses the true class distributions (Cover andHart, 1967) .
As discussed so far, both the K-nearest-neighbour method, and the kernel density estimator, require the entire training data set to be stored, leading to expensive
computation if the data set is large. This effect can be offset, at the expense of some
additional one-off computation, by constructing tree-based search structures to allow(approximate) near neighbours to be found efﬁciently without doing an exhaustive
search of the data set. Nevertheless, these nonparametric methods are still severely
limited. On the other hand, we have seen that simple parametric models are very
restricted in terms of the forms of distribution that they can represent. We therefore
need to ﬁnd density models that are very ﬂexible and yet for which the complexityof the models can be controlled independently of the size of the training set, and weshall see in subsequent chapters how to achieve this.

Exercises

2.1 ( ) www
ertiesVerify that the Bernoulli distribution (2.2) satisﬁes the following prop1
p(x|µ) = 1(2.257)

E[x] = µvar[x] = µ(1 − µ).

(2.258)(2.259)x=0Show that the entropy H[x] of a Bernoulli distributed random binary variable x is
given byH[x] = −µ ln µ − (1 − µ) ln(1 − µ).
(2.260)2.2 ( ) The form of the Bernoulli distribution given by (2.2) is not symmetric between the two values of x. In some situations, it will be more convenient to use anequivalent formulation for which x ∈ {−1, 1}, in which case the distribution can be
written(1−x)/2 
(1+x)/21−µ1+µ(2.261)p(x|µ) =22
where µ ∈ [−1, 1]. Show that the distribution (2.261) is normalized, and evaluate its
mean, variance, and entropy.
2.3 ( ) www In this exercise, we prove that the binomial distribution (2.9) is normalized. First use the deﬁnition (2.10) of the number of combinations of m identicalobjects chosen from a total of N to show that
  
 

N
N
N +1
+
=
.
(2.262)
m
m−1
m

128

2. PROBABILITY DISTRIBUTIONS
Use this result to prove by induction the following result
N  

N m
(1 + x) =xm
N(2.263)m=0which is known as the binomial theorem, and which is valid for all real values of x.
Finally, show that the binomial distribution is normalized, so that
N  

N m
µ (1 − µ)N −m = 1m(2.264)m=0

which can be done by ﬁrst pulling out a factor (1 − µ)N out of the summation andthen making use of the binomial theorem.
2.4 ( ) Show that the mean of the binomial distribution is given by (2.11). To do this,differentiate both sides of the normalization condition (2.264) with respect to µ andthen rearrange to obtain an expression for the mean of n. Similarly, by differentiating(2.264) twice with respect to µ and making use of the result (2.11) for the mean of
the binomial distribution prove the result (2.12) for the variance of the binomial.
2.5 ( ) www In this exercise, we prove that the beta distribution, given by (2.13), iscorrectly normalized, so that (2.14) holds. This is equivalent to showing that
 1
Γ(a)Γ(b)µa−1 (1 − µ)b−1 dµ =.
(2.265)Γ(a + b)0
From the deﬁnition (1.141) of the gamma function, we have
 ∞
 ∞
a−1
exp(−x)xdx
exp(−y)y b−1 dy.
Γ(a)Γ(b) =0(2.266)0

Use this expression to prove (2.265) as follows. First bring the integral over y inside
the integrand of the integral over x, next make the change of variable t = y + x
where x is ﬁxed, then interchange the order of the x and t integrations, and ﬁnallymake the change of variable x = tµ where t is ﬁxed.
2.6 ( ) Make use of the result (2.265) to show that the mean, variance, and mode of thebeta distribution (2.13) are given respectively byE[µ] =
var[µ] =mode[µ] =aa+b(2.267)ab(a + b)2 (a + b + 1)
a−1
.
a+b−2

(2.268)(2.269)

Exercises

129

2.7 ( ) Consider a binomial random variable x given by (2.9), with prior distribution
for µ given by the beta distribution (2.13), and suppose we have observed m occurrences of x = 1 and l occurrences of x = 0. Show that the posterior mean value of xlies between the prior mean and the maximum likelihood estimate for µ. To do this,show that the posterior mean can be written as λ times the prior mean plus (1 − λ)times the maximum likelihood estimate, where 0  λ  1. This illustrates the concept of the posterior distribution being a compromise between the prior distribution
and the maximum likelihood solution.
2.8 ( ) Consider two variables x and y with joint distribution p(x, y). Prove the following two results
E[x] = Ey [Ex [x|y]]var[x] = Ey [varx [x|y]] + vary [Ex [x|y]] .

(2.270)(2.271)

Here Ex [x|y] denotes the expectation of x under the conditional distribution p(x|y),with a similar notation for the conditional variance.
2.9 () www . In this exercise, we prove the normalization of the Dirichlet distribution (2.38) using induction. We have already shown in Exercise 2.5 that thebeta distribution, which is a special case of the Dirichlet for M = 2, is normalized.
We now assume that the Dirichlet distribution is normalized for M − 1 variables
and prove that it is normalized for M variables. To do this, consider the Dirichlet
M
distribution over M variables, and take account of the constraint k=1 µk = 1 byeliminating µM , so that the Dirichlet is writtenpM (µ1 , . . . , µM −1 ) = CM

M
−1



k −1
µα
k

1−

k=1

M
−1


αM −1µj(2.272)j =1

and our goal is to ﬁnd an expression for CM . To do this, integrate over µM −1 , taking
care over the limits of integration, and then make a change of variable so that thisintegral has limits 0 and 1. By assuming the correct result for CM −1 and making useof (2.265), derive the expression for CM .
2.10 ( ) Using the property Γ(x + 1) = xΓ(x) of the gamma function, derive thefollowing results for the mean, variance, and covariance of the Dirichlet distribution
given by (2.38)αjα0αj (α0 − αj )var[µj ] =α02 (α0 + 1)
αj α l,cov[µj µl ] = − 2
α0 (α0 + 1)
E[µj ] =where α0 is deﬁned by (2.39).

(2.273)(2.274)j=l(2.275)130

2. PROBABILITY DISTRIBUTIONS
2.11 ( ) www By expressing the expectation of ln µj under the Dirichlet distribution(2.38) as a derivative with respect to αj , show thatE[ln µj ] = ψ(αj ) − ψ(α0 )(2.276)

where α0 is given by (2.39) andψ(a) ≡dln Γ(a)da(2.277)is the digamma function.
2.12 ( ) The uniform distribution for a continuous variable x is deﬁned by
U(x|a, b) =

1
,
b−a

a  x  b.

(2.278)

Verify that this distribution is normalized, and ﬁnd expressions for its mean andvariance.
2.13 ( ) Evaluate the Kullback-Leibler divergence (1.113) between two Gaussians
p(x) = N (x|µ, Σ) and q(x) = N (x|m, L).
2.14 ( ) www This exercise demonstrates that the multivariate distribution with maximum entropy, for a given covariance, is a Gaussian. The entropy of a distribution
p(x) is given by(2.279)H[x] = − p(x) ln p(x) dx.
We wish to maximize H[x] over all distributions p(x) subject to the constraints thatp(x) be normalized and that it have a speciﬁc mean and covariance, so thatp(x) dx = 1(2.280)
p(x)x dx = µ(2.281)p(x)(x − µ)(x − µ)T dx = Σ.
(2.282)
By performing a variational maximization of (2.279) and using Lagrange multipliers
to enforce the constraints (2.280), (2.281), and (2.282), show that the maximum
likelihood distribution is given by the Gaussian (2.43).
2.15 () Show that the entropy of the multivariate Gaussian N (x|µ, Σ) is given by
H[x] =

D1
ln |Σ| + (1 + ln(2π))
22where D is the dimensionality of x.

(2.283)

Exercises131

2.16 () www Consider two random variables x1 and x2 having Gaussian distributions with means µ1 , µ2 and precisions τ1 , τ2 respectively. Derive an expression
for the differential entropy of the variable x = x1 + x2 . To do this, ﬁrst ﬁnd the
distribution of x by using the relation ∞p(x|x2 )p(x2 ) dx2(2.284)p(x) =−∞and completing the square in the exponent. Then observe that this represents the
convolution of two Gaussian distributions, which itself will be Gaussian, and ﬁnallymake use of the result (1.110) for the entropy of the univariate Gaussian.
2.17 ( ) www Consider the multivariate Gaussian distribution given by (2.43). By
writing the precision matrix (inverse covariance matrix) Σ−1 as the sum of a symmetric and an anti-symmetric matrix, show that the anti-symmetric term does notappear in the exponent of the Gaussian, and hence that the precision matrix may betaken to be symmetric without loss of generality. Because the inverse of a symmetricmatrix is also symmetric (see Exercise 2.22), it follows that the covariance matrix
may also be chosen to be symmetric without loss of generality.
2.18 () Consider a real, symmetric matrix Σ whose eigenvalue equation is givenby (2.45). By taking the complex conjugate of this equation and subtracting theoriginal equation, and then forming the inner product with eigenvector ui , show thatthe eigenvalues λi are real. Similarly, use the symmetry property of Σ to show that
two eigenvectors ui and uj will be orthogonal provided λj = λi . Finally, show thatwithout loss of generality, the set of eigenvectors can be chosen to be orthonormal,
so that they satisfy (2.46), even if some of the eigenvalues are zero.
2.19 ( ) Show that a real, symmetric matrix Σ having the eigenvector equation (2.45)
can be expressed as an expansion in the eigenvectors, with coefﬁcients given by the
eigenvalues, of the form (2.48). Similarly, show that the inverse matrix Σ−1 has arepresentation of the form (2.49).
2.20 ( ) www A positive deﬁnite matrix Σ can be deﬁned as one for which thequadratic form
(2.285)aT Σais positive for any real value of the vector a. Show that a necessary and sufﬁcientcondition for Σ to be positive deﬁnite is that all of the eigenvalues λi of Σ, deﬁnedby (2.45), are positive.
2.21 ( ) Show that a real, symmetric matrix of size D × D has D(D + 1)/2 independent
parameters.
2.22 ( ) www

Show that the inverse of a symmetric matrix is itself symmetric.

2.23 ( ) By diagonalizing the coordinate system using the eigenvector expansion (2.45),
show that the volume contained within the hyperellipsoid corresponding to a constant

132

2. PROBABILITY DISTRIBUTIONS
Mahalanobis distance ∆ is given by
VD |Σ|1/2 ∆D(2.286)

where VD is the volume of the unit sphere in D dimensions, and the Mahalanobis
distance is deﬁned by (2.44).
2.24 () www

Prove the identity (2.76) by multiplying both sides by the matrix
A B(2.287)C D

and making use of the deﬁnition (2.77).
2.25 ( ) In Sections 2.3.1 and 2.3.2, we considered the conditional and marginal distributions for a multivariate Gaussian. More generally, we can consider a partitioningof the components of x into three groups xa , xb , and xc , with a corresponding partitioning of the mean vector µ and of the covariance matrix Σ in the form
 


µa
Σaa Σab ΣacΣ = Σba Σbb Σbc .
(2.288)µ = µb ,µcΣca Σcb Σcc
By making use of the results of Section 2.3, ﬁnd an expression for the conditional
distribution p(xa |xb ) in which xc has been marginalized out.
2.26 ( ) A very useful result from linear algebra is the Woodbury matrix inversion
formula given by(A + BCD)−1 = A−1 − A−1 B(C−1 + DA−1 B)−1 DA−1 .

(2.289)By multiplying both sides by (A + BCD) prove the correctness of this result.
2.27 ( ) Let x and z be two independent random vectors, so that p(x, z) = p(x)p(z).
Show that the mean of their sum y = x + z is given by the sum of the means of eachof the variable separately. Similarly, show that the covariance matrix of y is given by
the sum of the covariance matrices of x and z. Conﬁrm that this result agrees withthat of Exercise 1.10.
2.28 () www

Consider a joint distribution over the variable
 
x
z=
y

(2.290)whose mean and covariance are given by (2.108) and (2.105) respectively. By making use of the results (2.92) and (2.93) show that the marginal distribution p(x) is
given (2.99). Similarly, by making use of the results (2.81) and (2.82) show that theconditional distribution p(y|x) is given by (2.100).

Exercises

133

2.29 ( ) Using the partitioned matrix inversion formula (2.76), show that the inverse of
the precision matrix (2.104) is given by the covariance matrix (2.105).
2.30 ( ) By starting from (2.107) and making use of the result (2.105), verify the result(2.108).
2.31 ( ) Consider two multidimensional random vectors x and z having Gaussian
distributions p(x) = N (x|µx , Σx ) and p(z) = N (z|µz , Σz ) respectively, together
with their sum y = x+z. Use the results (2.109) and (2.110) to ﬁnd an expression forthe marginal distribution p(y) by considering the linear-Gaussian model comprising
the product of the marginal distribution p(x) and the conditional distribution p(y|x).
2.32 () wwwThis exercise and the next provide practice at manipulating thequadratic forms that arise in linear-Gaussian models, as well as giving an independent check of results derived in the main text. Consider a joint distribution p(x, y)deﬁned by the marginal and conditional distributions given by (2.99) and (2.100).
By examining the quadratic form in the exponent of the joint distribution, and using
the technique of ‘completing the square’ discussed in Section 2.3, ﬁnd expressions
for the mean and covariance of the marginal distribution p(y) in which the variable
x has been integrated out. To do this, make use of the Woodbury matrix inversion
formula (2.289). Verify that these results agree with (2.109) and (2.110) obtainedusing the results of Chapter 2.
2.33 () Consider the same joint distribution as in Exercise 2.32, but now use thetechnique of completing the square to ﬁnd expressions for the mean and covarianceof the conditional distribution p(x|y). Again, verify that these agree with the corresponding expressions (2.111) and (2.112).
2.34 ( ) www To ﬁnd the maximum likelihood solution for the covariance matrix
of a multivariate Gaussian, we need to maximize the log likelihood function (2.118)
with respect to Σ, noting that the covariance matrix must be symmetric and positivedeﬁnite. Here we proceed by ignoring these constraints and doing a straightforward
maximization. Using the results (C.21), (C.26), and (C.28) from Appendix C, showthat the covariance matrix Σ that maximizes the log likelihood function (2.118) is
given by the sample covariance (2.122). We note that the ﬁnal result is necessarily
symmetric and positive deﬁnite (provided the sample covariance is nonsingular).
2.35 ( ) Use the result (2.59) to prove (2.62). Now, using the results (2.59), and (2.62),
show thatE[xn xm ] = µµT + Inm Σ(2.291)
where xn denotes a data point sampled from a Gaussian distribution with mean µand covariance Σ, and Inm denotes the (n, m) element of the identity matrix. Hence
prove the result (2.124).
2.36 ( ) www Using an analogous procedure to that used to obtain (2.126), derive
an expression for the sequential estimation of the variance of a univariate Gaussian

134

2. PROBABILITY DISTRIBUTIONS
distribution, by starting with the maximum likelihood expression1 (xn − µ)2 .
N
N

2
σML
=

(2.292)n=1

Verify that substituting the expression for a Gaussian distribution into the RobbinsMonro sequential estimation formula (2.135) gives a result of the same form, andhence obtain an expression for the corresponding coefﬁcients aN .
2.37 ( ) Using an analogous procedure to that used to obtain (2.126), derive an expression for the sequential estimation of the covariance of a multivariate Gaussian
distribution, by starting with the maximum likelihood expression (2.122). Verify that
substituting the expression for a Gaussian distribution into the Robbins-Monro sequential estimation formula (2.135) gives a result of the same form, and hence obtainan expression for the corresponding coefﬁcients aN .
2.38 ( ) Use the technique of completing the square for the quadratic form in the exponent to derive the results (2.141) and (2.142).
2.39 ( ) Starting from the results (2.141) and (2.142) for the posterior distributionof the mean of a Gaussian random variable, dissect out the contributions from theﬁrst N − 1 data points and hence obtain expressions for the sequential update of2. Now derive the same results starting from the posterior distributionµN and σN2p(µ|x1 , . . . , xN −1 ) = N (µ|µN −1 , σN−1 ) and multiplying by the likelihood func2
tion p(xN |µ) = N (xN |µ, σ ) and then completing the square and normalizing toobtain the posterior distribution after N observations.
2.40 ( ) www Consider a D-dimensional Gaussian random variable x with distribution N (x|µ, Σ) in which the covariance Σ is known and for which we wish to infer
the mean µ from a set of observations X = {x1 , . . . , xN }. Given a prior distributionp(µ) = N (µ|µ0 , Σ0 ), ﬁnd the corresponding posterior distribution p(µ|X).
2.41 ( ) Use the deﬁnition of the gamma function (1.141) to show that the gamma distribution (2.146) is normalized.
2.42 () Evaluate the mean, variance, and mode of the gamma distribution (2.146).

2.43 ( ) The following distributionp(x|σ 2 , q) =



|x|q
exp
−
2σ 2
2(2σ 2 )1/q Γ(1/q)
q(2.293)is a generalization of the univariate Gaussian distribution. Show that this distribution
is normalized so that ∞p(x|σ 2 , q) dx = 1(2.294)−∞and that it reduces to the Gaussian when q = 2. Consider a regression model inwhich the target variable is given by t = y(x, w) +  and  is a random noise

Exercises

135

variable drawn from the distribution (2.293). Show that the log likelihood function
over w and σ 2 , for an observed data set of input vectors X = {x1 , . . . , xN } andcorresponding target variables t = (t1 , . . . , tN )T , is given by1 N|y(xn , w) − tn |q −ln(2σ 2 ) + const
ln p(t|X, w, σ ) = − 2
2σ
q
N

2

(2.295)n=1

where ‘const’ denotes terms independent of both w and σ 2 . Note that, as a functionof w, this is the Lq error function considered in Section 1.5.5.
2.44 ( ) Consider a univariate Gaussian distribution N (x|µ, τ −1 ) having conjugate
Gaussian-gamma prior given by (2.154), and a data set x = {x1 , . . . , xN } of i.i.d.
observations. Show that the posterior distribution is also a Gaussian-gamma distribution of the same functional form as the prior, and write down expressions for theparameters of this posterior distribution.
2.45 ( ) Verify that the Wishart distribution deﬁned by (2.155) is indeed a conjugate
prior for the precision matrix of a multivariate Gaussian.
2.46 ( ) www

Verify that evaluating the integral in (2.158) leads to the result (2.159).

2.47 ( ) www Show that in the limit ν → ∞, the t-distribution (2.159) becomes a
Gaussian. Hint: ignore the normalization coefﬁcient, and simply look at the dependence on x.
2.48 ( ) By following analogous steps to those used to derive the univariate Student’s
t-distribution (2.159), verify the result (2.162) for the multivariate form of the Student’s t-distribution, by marginalizing over the variable η in (2.161). Using thedeﬁnition (2.161), show by exchanging integration variables that the multivariate
t-distribution is correctly normalized.
2.49 ( ) By using the deﬁnition (2.161) of the multivariate Student’s t-distribution as a
convolution of a Gaussian with a gamma distribution, verify the properties (2.164),(2.165), and (2.166) for the multivariate t-distribution deﬁned by (2.162).
2.50 ( ) Show that in the limit ν → ∞, the multivariate Student’s t-distribution (2.162)
reduces to a Gaussian with mean µ and precision Λ.
2.51 ( ) www The various trigonometric identities used in the discussion of periodic
variables in this chapter can be proven easily from the relation
exp(iA) = cos A + i sin A(2.296)

in which i is the square root of minus one. By considering the identity
exp(iA) exp(−iA) = 1(2.297)

prove the result (2.177). Similarly, using the identitycos(A − B) =exp{i(A − B)}(2.298)136

2. PROBABILITY DISTRIBUTIONS
where
denotes the real part, prove (2.178). Finally, by using sin(A − B) =exp{i(A − B)}, where denotes the imaginary part, prove the result (2.183).
2.52 ( ) For large m, the von Mises distribution (2.179) becomes sharply peakedaround the mode θ0 . By deﬁning ξ = m1/2 (θ − θ0 ) and making the Taylor expansion of the cosine function given byα2+ O(α4 )2show that as m → ∞, the von Mises distribution tends to a Gaussian.
cos α = 1 −(2.299)2.53 ( ) Using the trigonometric identity (2.183), show that solution of (2.182) for θ0 isgiven by (2.184).
2.54 ( ) By computing ﬁrst and second derivatives of the von Mises distribution (2.179),
and using I0 (m) > 0 for m > 0, show that the maximum of the distribution occurs
when θ = θ0 and that the minimum occurs when θ = θ0 + π (mod 2π).
2.55 ( ) By making use of the result (2.168), together with (2.184) and the trigonometricidentity (2.178), show that the maximum likelihood solution mML for the concentration of the von Mises distribution satisﬁes A(mML ) = r where r is the radius of themean of the observations viewed as unit vectors in the two-dimensional Euclidean
plane, as illustrated in Figure 2.17.
2.56 ( ) www Express the beta distribution (2.13), the gamma distribution (2.146),
and the von Mises distribution (2.179) as members of the exponential family (2.194)
and thereby identify their natural parameters.
2.57 ( ) Verify that the multivariate Gaussian distribution can be cast in exponential
family form (2.194) and derive expressions for η, u(x), h(x) and g(η) analogous to(2.220)–(2.223).
2.58 ( ) The result (2.226) showed that the negative gradient of ln g(η) for the exponential family is given by the expectation of u(x). By taking the second derivatives of(2.195), show that−∇∇ ln g(η) = E[u(x)u(x)T ] − E[u(x)]E[u(x)T ] = cov[u(x)].

(2.300)2.59 ( ) By changing variables using y = x/σ, show that the density (2.236) will becorrectly normalized, provided f (x) is correctly normalized.
2.60 ( ) www Consider a histogram-like density model in which the space x is divided into ﬁxed regions for which the density p(x) takes the constant value hi overthe ith region, and that the volume of region i is denoted ∆i . Suppose we have a setof N observations of x such that ni of these observations fall in region i. Using a
Lagrange multiplier to enforce the normalization constraint on the density, derive an
expression for the maximum likelihood estimator for the {hi }.
2.61 ( ) Show that the K-nearest-neighbour density model deﬁnes an improper distribution whose integral over all space is divergent.

3

LinearModels for
Regression

The focus so far in this book has been on unsupervised learning, including topics
such as density estimation and data clustering. We turn now to a discussion of supervised learning, starting with regression. The goal of regression is to predict the valueof one or more continuous target variables t given the value of a D-dimensional vector x of input variables. We have already encountered an example of a regression
problem when we considered polynomial curve ﬁtting in Chapter 1. The polynomial
is a speciﬁc example of a broad class of functions called linear regression models,
which share the property of being linear functions of the adjustable parameters, andwhich will form the focus of this chapter. The simplest form of linear regression
models are also linear functions of the input variables. However, we can obtain amuch more useful class of functions by taking linear combinations of a ﬁxed set of
nonlinear functions of the input variables, known as basis functions. Such models
are linear functions of the parameters, which gives them simple analytical properties,
and yet can be nonlinear with respect to the input variables.

137

138

3. LINEAR MODELS FOR REGRESSION
Given a training data set comprising N observations {xn }, where n = 1, . . . , N ,together with corresponding target values {tn }, the goal is to predict the value of t
for a new value of x. In the simplest approach, this can be done by directly constructing an appropriate function y(x) whose values for new inputs x constitute thepredictions for the corresponding values of t. More generally, from a probabilistic
perspective, we aim to model the predictive distribution p(t|x) because this expresses
our uncertainty about the value of t for each value of x. From this conditional distribution we can make predictions of t, for any new value of x, in such a way as tominimize the expected value of a suitably chosen loss function. As discussed in Section 1.5.5, a common choice of loss function for real-valued variables is the squared
loss, for which the optimal solution is given by the conditional expectation of t.
Although linear models have signiﬁcant limitations as practical techniques forpattern recognition, particularly for problems involving input spaces of high dimensionality, they have nice analytical properties and form the foundation for more sophisticated models to be discussed in later chapters.

3.1. Linear Basis Function Models
The simplest linear model for regression is one that involves a linear combination of
the input variables
y(x, w) = w0 + w1 x1 + . . . + wD xD(3.1)

where x = (x1 , . . . , xD )T . This is often simply known as linear regression. The key
property of this model is that it is a linear function of the parameters w0 , . . . , wD . It isalso, however, a linear function of the input variables xi , and this imposes signiﬁcantlimitations on the model. We therefore extend the class of models by considering
linear combinations of ﬁxed nonlinear functions of the input variables, of the formy(x, w) = w0 +M−1wj φj (x)

(3.2)

j =1

where φj (x) are known as basis functions. By denoting the maximum value of theindex j by M − 1, the total number of parameters in this model will be M .
The parameter w0 allows for any ﬁxed offset in the data and is sometimes calleda bias parameter (not to be confused with ‘bias’ in a statistical sense). It is often
convenient to deﬁne an additional dummy ‘basis function’ φ0 (x) = 1 so thaty(x, w) =M−1

wj φj (x) = wT φ(x)

(3.3)

j =0

where w = (w0 , . . . , wM −1 )T and φ = (φ0 , . . . , φM −1 )T . In many practical applications of pattern recognition, we will apply some form of ﬁxed pre-processing,3.1. Linear Basis Function Models

139

or feature extraction, to the original data variables. If the original variables comprise the vector x, then the features can be expressed in terms of the basis functions{φj (x)}.
By using nonlinear basis functions, we allow the function y(x, w) to be a nonlinear function of the input vector x. Functions of the form (3.2) are called linear
models, however, because this function is linear in w. It is this linearity in the parameters that will greatly simplify the analysis of this class of models. However, italso leads to some signiﬁcant limitations, as we discuss in Section 3.6.
The example of polynomial regression considered in Chapter 1 is a particular
example of this model in which there is a single input variable x, and the basis functions take the form of powers of x so that φj (x) = xj . One limitation of polynomial
basis functions is that they are global functions of the input variable, so that changes
in one region of input space affect all other regions. This can be resolved by dividing
the input space up into regions and ﬁt a different polynomial in each region, leading
to spline functions (Hastie et al., 2001).
There are many other possible choices for the basis functions, for example
φj (x) = exp −(x − µj )2
2s2

(3.4)

where the µj govern the locations of the basis functions in input space, and the parameter s governs their spatial scale. These are usually referred to as ‘Gaussian’basis functions, although it should be noted that they are not required to have a probabilistic interpretation, and in particular the normalization coefﬁcient is unimportantbecause these basis functions will be multiplied by adaptive parameters wj .
Another possibility is the sigmoidal basis function of the form
x − µ j(3.5)
φj (x) = σswhere σ(a) is the logistic sigmoid function deﬁned by
σ(a) =

1.
1 + exp(−a)(3.6)

Equivalently, we can use the ‘tanh’ function because this is related to the logistic
sigmoid by tanh(a) = 2σ(a) − 1, and so a general linear combination of logistic
sigmoid functions is equivalent to a general linear combination of ‘tanh’ functions.
These various choices of basis function are illustrated in Figure 3.1.
Yet another possible choice of basis function is the Fourier basis, which leads toan expansion in sinusoidal functions. Each basis function represents a speciﬁc frequency and has inﬁnite spatial extent. By contrast, basis functions that are localized
to ﬁnite regions of input space necessarily comprise a spectrum of different spatial
frequencies. In many signal processing applications, it is of interest to consider basis functions that are localized in both space and frequency, leading to a class offunctions known as wavelets. These are also deﬁned to be mutually orthogonal, tosimplify their application. Wavelets are most applicable when the input values live140

3. LINEAR MODELS FOR REGRESSION
1

1

1

0.5

0.75

0.75

0

0.5

0.5

−0.5

0.25

0.25

−1
−1

0

1

0

−1

0

1

0
−1

0

1

Figure 3.1 Examples of basis functions, showing polynomials on the left, Gaussians of the form (3.4) in thecentre, and sigmoidal of the form (3.5) on the right.

on a regular lattice, such as the successive time points in a temporal sequence, or the
pixels in an image. Useful texts on wavelets include Ogden (1997), Mallat (1999),
and Vidakovic (1999).
Most of the discussion in this chapter, however, is independent of the particular
choice of basis function set, and so for most of our discussion we shall not specify
the particular form of the basis functions, except for the purposes of numerical illustration. Indeed, much of our discussion will be equally applicable to the situationin which the vector φ(x) of basis functions is simply the identity φ(x) = x. Furthermore, in order to keep the notation simple, we shall focus on the case of a single
target variable t. However, in Section 3.1.5, we consider brieﬂy the modiﬁcations
needed to deal with multiple target variables.

3.1.1 Maximum likelihood and least squares
In Chapter 1, we ﬁtted polynomial functions to data sets by minimizing a sumof-squares error function. We also showed that this error function could be motivatedas the maximum likelihood solution under an assumed Gaussian noise model. Let
us return to this discussion and consider the least squares approach, and its relation
to maximum likelihood, in more detail.
As before, we assume that the target variable t is given by a deterministic function y(x, w) with additive Gaussian noise so thatt = y(x, w) + (3.7)

where  is a zero mean Gaussian random variable with precision (inverse variance)
β. Thus we can writep(t|x, w, β) = N (t|y(x, w), β −1 ).
Section 1.5.5(3.8)

Recall that, if we assume a squared loss function, then the optimal prediction, for a
new value of x, will be given by the conditional mean of the target variable. In thecase of a Gaussian conditional distribution of the form (3.8), the conditional mean

3.1. Linear Basis Function Models
will be simply141
E[t|x] =tp(t|x) dt = y(x, w).

(3.9)

Note that the Gaussian noise assumption implies that the conditional distribution oft given x is unimodal, which may be inappropriate for some applications. An extension to mixtures of conditional Gaussian distributions, which permit multimodal
conditional distributions, will be discussed in Section 14.5.1.
Now consider a data set of inputs X = {x1 , . . . , xN } with corresponding target
values t1 , . . . , tN . We group the target variables {tn } into a column vector that wedenote by t where the typeface is chosen to distinguish it from a single observationof a multivariate target, which would be denoted t. Making the assumption thatthese data points are drawn independently from the distribution (3.8), we obtain thefollowing expression for the likelihood function, which is a function of the adjustableparameters w and β, in the formp(t|X, w, β) =NN (tn |wT φ(xn ), β −1 )(3.10)

n=1

where we have used (3.3). Note that in supervised learning problems such as regression (and classiﬁcation), we are not seeking to model the distribution of the input
variables. Thus x will always appear in the set of conditioning variables, and sofrom now on we will drop the explicit x from expressions such as p(t|x, w, β) in order to keep the notation uncluttered. Taking the logarithm of the likelihood function,and making use of the standard form (1.46) for the univariate Gaussian, we haveln p(t|w, β) =Nln N (tn |wT φ(xn ), β −1 )

n=1

=

N
N
ln β −ln(2π) − βED (w)22(3.11)

where the sum-of-squares error function is deﬁned by1
ED (w) ={tn − wT φ(xn )}2 .
2N(3.12)

n=1

Having written down the likelihood function, we can use maximum likelihood todetermine w and β. Consider ﬁrst the maximization with respect to w. As observed
already in Section 1.2.5, we see that maximization of the likelihood function under aconditional Gaussian noise distribution for a linear model is equivalent to minimizing
a sum-of-squares error function given by ED (w). The gradient of the log likelihood
function (3.11) takes the form
∇ ln p(t|w, β) =

N


n=1


tn − wT φ(xn ) φ(xn )T .

(3.13)

142

3. LINEAR MODELS FOR REGRESSION
Setting this gradient to zero gives0=Ntn φ(xn ) − w
T

T

N


n=1


φ(xn )φ(xn )T.

(3.14)

n=1

Solving for w we obtain−1

wML = ΦT Φ

ΦT t

(3.15)

which are known as the normal equations for the least squares problem. Here Φ is an
N ×M matrix, called the design matrix, whose elements are given by Φnj = φj (xn ),so that⎛⎞
φ0 (x1 ) φ1 (x1 ) · · · φM −1 (x1 )⎜ φ0 (x2 ) φ1 (x2 ) · · · φM −1 (x2 ) ⎟⎟.
Φ=⎜(3.16)
..
..
..
..
⎝
⎠
.
.
.
.
φ0 (xN ) φ1 (xN ) · · ·The quantityΦ† ≡ ΦT Φ−1

φM −1 (xN )ΦT(3.17)

is known as the Moore-Penrose pseudo-inverse of the matrix Φ (Rao and Mitra,
1971; Golub and Van Loan, 1996). It can be regarded as a generalization of thenotion of matrix inverse to nonsquare matrices. Indeed, if Φ is square and invertible,
then using the property (AB)−1 = B−1 A−1 we see that Φ† ≡ Φ−1 .
At this point, we can gain some insight into the role of the bias parameter w0 . If
we make the bias parameter explicit, then the error function (3.12) becomes
ED (w) =
1
{tn − w0 −wj φj (xn )}2 .
2
N

M −1

n=1

j =1

(3.18)

Setting the derivative with respect to w0 equal to zero, and solving for w0 , we obtainw0 = t −

M
−1


wj φj

(3.19)

j =1

where we have deﬁned
1 
t=
tn ,
N

1 
φj =
φj (xn ).
N

N

N

n=1

(3.20)

n=1

Thus the bias w0 compensates for the difference between the averages (over the
training set) of the target values and the weighted sum of the averages of the basis
function values.
We can also maximize the log likelihood function (3.11) with respect to the noise
precision parameter β, giving
1
βML

1 
T
{tn − wML
φ(xn )}2
N
N

=

n=1

(3.21)

143

3.1. Linear Basis Function Models
Figure 3.2

Geometrical interpretation of the least-squares
solution, in an N -dimensional space whose axes
are the values of t1 , . . . , tN . The least-squares
regression function is obtained by ﬁnding the orthogonal projection of the data vector t onto thesubspace spanned by the basis functions φj (x)in which each basis function is viewed as a vector ϕj of length N with elements φj (xn ).

S

ϕ1

t
y

ϕ2

and so we see that the inverse of the noise precision is given by the residual varianceof the target values around the regression function.

3.1.2 Geometry of least squares

Exercise 3.2

At this point, it is instructive to consider the geometrical interpretation of theleast-squares solution. To do this we consider an N -dimensional space whose axes
are given by the tn , so that t = (t1 , . . . , tN )T is a vector in this space. Each basis
function φj (xn ), evaluated at the N data points, can also be represented as a vector in
the same space, denoted by ϕj , as illustrated in Figure 3.2. Note that ϕj corresponds
to the j th column of Φ, whereas φ(xn ) corresponds to the nth row of Φ. If thenumber M of basis functions is smaller than the number N of data points, then the
M vectors φj (xn ) will span a linear subspace S of dimensionality M . We deﬁne
y to be an N -dimensional vector whose nth element is given by y(xn , w), wheren = 1, . . . , N . Because y is an arbitrary linear combination of the vectors ϕj , it canlive anywhere in the M -dimensional subspace. The sum-of-squares error (3.12) isthen equal (up to a factor of 1/2) to the squared Euclidean distance between y andt. Thus the least-squares solution for w corresponds to that choice of y that lies insubspace S and that is closest to t. Intuitively, from Figure 3.2, we anticipate that
this solution corresponds to the orthogonal projection of t onto the subspace S. This
is indeed the case, as can easily be veriﬁed by noting that the solution for y is givenby ΦwML , and then conﬁrming that this takes the form of an orthogonal projection.
In practice, a direct solution of the normal equations can lead to numerical difﬁculties when ΦT Φ is close to singular. In particular, when two or more of the basis
vectors ϕj are co-linear, or nearly so, the resulting parameter values can have large
magnitudes. Such near degeneracies will not be uncommon when dealing with real
data sets. The resulting numerical difﬁculties can be addressed using the techniqueof singular value decomposition, or SVD (Press et al., 1992; Bishop and Nabney,2008). Note that the addition of a regularization term ensures that the matrix is nonsingular, even in the presence of degeneracies.

3.1.3 Sequential learning
Batch techniques, such as the maximum likelihood solution (3.15), which involve processing the entire training set in one go, can be computationally costly forlarge data sets. As we have discussed in Chapter 1, if the data set is sufﬁciently large,
it may be worthwhile to use sequential algorithms, also known as on-line algorithms,

144

3. LINEAR MODELS FOR REGRESSION
in which the data points are considered one at a time, and the model parameters updated after each such presentation. Sequential learning is also appropriate for realtime applications in which the data observations are arriving in a continuous stream,and predictions must be made before all of the data points are seen.
We can obtain a sequential learning algorithm by applying the technique ofstochastic gradient descent, also known as sequential gradient descent, as follows. If
the error function comprises a sum over data points E = n En , then after presentation of pattern n, the stochastic gradient descent algorithm updates the parameter
vector w usingw(τ +1) = w(τ ) − η∇En
(3.22)
where τ denotes the iteration number, and η is a learning rate parameter. We shalldiscuss the choice of value for η shortly. The value of w is initialized to some starting
vector w(0) . For the case of the sum-of-squares error function (3.12), this givesw(τ +1) = w(τ ) + η(tn − w(τ )T φn )φn(3.23)

where φn = φ(xn ). This is known as least-mean-squares or the LMS algorithm.
The value of η needs to be chosen with care to ensure that the algorithm converges(Bishop and Nabney, 2008).

3.1.4 Regularized least squares
In Section 1.1, we introduced the idea of adding a regularization term to anerror function in order to control over-ﬁtting, so that the total error function to beminimized takes the form
(3.24)ED (w) + λEW (w)where λ is the regularization coefﬁcient that controls the relative importance of thedata-dependent error ED (w) and the regularization term EW (w). One of the simplest forms of regularizer is given by the sum-of-squares of the weight vector elements
1
EW (w) = wT w.
(3.25)
2
If we also consider the sum-of-squares error function given by1
{tn − wT φ(xn )}2E(w) =
2N(3.26)

n=1

then the total error function becomesλ1
{tn − wT φ(xn )}2 + wT w.
2
2
N(3.27)

n=1

This particular choice of regularizer is known in the machine learning literature as
weight decay because in sequential learning algorithms, it encourages weight values
to decay towards zero, unless supported by the data. In statistics, it provides an example of a parameter shrinkage method because it shrinks parameter values towards

3.1. Linear Basis Function Models

q = 0.5
Figure 3.3

q=1

q=2

145

q=4

Contours of the regularization term in (3.29) for various values of the parameter q.

zero. It has the advantage that the error function remains a quadratic function ofw, and so its exact minimizer can be found in closed form. Speciﬁcally, setting thegradient of (3.27) with respect to w to zero, and solving for w as before, we obtainw = λI + ΦT Φ

−1

ΦT t.

(3.28)

This represents a simple extension of the least-squares solution (3.15).
A more general regularizer is sometimes used, for which the regularized error
takes the form
N
M
1
λ
{tn − wT φ(xn )}2 +|wj |q(3.29)
2
2
n=1

Exercise 3.5

j =1

where q = 2 corresponds to the quadratic regularizer (3.27). Figure 3.3 shows contours of the regularization function for different values of q.
The case of q = 1 is know as the lasso in the statistics literature (Tibshirani,1996). It has the property that if λ is sufﬁciently large, some of the coefﬁcients
wj are driven to zero, leading to a sparse model in which the corresponding basis
functions play no role. To see this, we ﬁrst note that minimizing (3.29) is equivalent
to minimizing the unregularized sum-of-squares error (3.12) subject to the constraintM|wj |q  η(3.30)

j =1

Appendix E

for an appropriate value of the parameter η, where the two approaches can be relatedusing Lagrange multipliers. The origin of the sparsity can be seen from Figure 3.4,
which shows that the minimum of the error function, subject to the constraint (3.30).
As λ is increased, so an increasing number of parameters are driven to zero.
Regularization allows complex models to be trained on data sets of limited size
without severe over-ﬁtting, essentially by limiting the effective model complexity.
However, the problem of determining the optimal model complexity is then shiftedfrom one of ﬁnding the appropriate number of basis functions to one of determininga suitable value of the regularization coefﬁcient λ. We shall return to the issue ofmodel complexity later in this chapter.

146

3. LINEAR MODELS FOR REGRESSION

Figure 3.4 Plot of the contoursof the unregularized error function(blue) along with the constraint region (3.30) for the quadratic regularizer q = 2 on the left and the lasso
regularizer q = 1 on the right, inwhich the optimum value for the parameter vector w is denoted by w .
The lasso gives a sparse solution inwhich w1 = 0.

w2

w2

w

w

w1

w1

For the remainder of this chapter we shall focus on the quadratic regularizer(3.27) both for its practical importance and its analytical tractability.

3.1.5 Multiple outputs
So far, we have considered the case of a single target variable t. In some applications, we may wish to predict K > 1 target variables, which we denote collectively
by the target vector t. This could be done by introducing a different set of basis functions for each component of t, leading to multiple, independent regression problems.
However, a more interesting, and more common, approach is to use the same set ofbasis functions to model all of the components of the target vector so thaty(x, w) = WT φ(x)(3.31)

where y is a K-dimensional column vector, W is an M × K matrix of parameters,and φ(x) is an M -dimensional column vector with elements φj (x), with φ0 (x) = 1
as before. Suppose we take the conditional distribution of the target vector to be an
isotropic Gaussian of the formp(t|x, W, β) = N (t|WT φ(x), β −1 I).

(3.32)

If we have a set of observations t1 , . . . , tN , we can combine these into a matrix T
of size N × K such that the nth row is given by tT
n . Similarly, we can combine theinput vectors x1 , . . . , xN into a matrix X. The log likelihood function is then givenbyln p(T|X, W, β) =Nln N (tn |WT φ(xn ), β −1 I)

n=1

=

NK
ln
2



β
2π


−

N
'
β '
'tn − WT φ(xn )'2 . (3.33)
2
n=1

3.2. The Bias-Variance Decomposition147

As before, we can maximize this function with respect to W, giving
WML = ΦT Φ

−1

ΦT T.

(3.34)

If we examine this result for each target variable tk , we have
wk = ΦT Φ

Exercise 3.6−1

ΦT tk = Φ† tk(3.35)

where tk is an N -dimensional column vector with components tnk for n = 1, . . . N .
Thus the solution to the regression problem decouples between the different target
variables, and we need only compute a single pseudo-inverse matrix Φ† , which isshared by all of the vectors wk .
The extension to general Gaussian noise distributions having arbitrary covariance matrices is straightforward. Again, this leads to a decoupling into K independent regression problems. This result is unsurprising because the parameters W
deﬁne only the mean of the Gaussian noise distribution, and we know from Section 2.3.4 that the maximum likelihood solution for the mean of a multivariate Gaussian is independent of the covariance. From now on, we shall therefore consider asingle target variable t for simplicity.

3.2. The Bias-Variance Decomposition
So far in our discussion of linear models for regression, we have assumed that theform and number of basis functions are both ﬁxed. As we have seen in Chapter 1,
the use of maximum likelihood, or equivalently least squares, can lead to severe
over-ﬁtting if complex models are trained using data sets of limited size. However,limiting the number of basis functions in order to avoid over-ﬁtting has the side
effect of limiting the ﬂexibility of the model to capture interesting and important
trends in the data. Although the introduction of regularization terms can control
over-ﬁtting for models with many parameters, this raises the question of how todetermine a suitable value for the regularization coefﬁcient λ. Seeking the solutionthat minimizes the regularized error function with respect to both the weight vector
w and the regularization coefﬁcient λ is clearly not the right approach since thisleads to the unregularized solution with λ = 0.
As we have seen in earlier chapters, the phenomenon of over-ﬁtting is really an
unfortunate property of maximum likelihood and does not arise when we marginalize
over parameters in a Bayesian setting. In this chapter, we shall consider the Bayesianview of model complexity in some depth. Before doing so, however, it is instructive
to consider a frequentist viewpoint of the model complexity issue, known as the biasvariance trade-off. Although we shall introduce this concept in the context of linear
basis function models, where it is easy to illustrate the ideas using simple examples,the discussion has more general applicability.
In Section 1.5.5, when we discussed decision theory for regression problems,
we considered various loss functions each of which leads to a corresponding optimal
prediction once we are given the conditional distribution p(t|x). A popular choice is

148

3. LINEAR MODELS FOR REGRESSION
the squared loss function, for which the optimal prediction is given by the conditional
expectation, which we denote by h(x) and which is given byh(x) = E[t|x] = tp(t|x) dt.
(3.36)At this point, it is worth distinguishing between the squared loss function arisingfrom decision theory and the sum-of-squares error function that arose in the maximum likelihood estimation of model parameters. We might use more sophisticatedtechniques than least squares, for example regularization or a fully Bayesian approach, to determine the conditional distribution p(t|x). These can all be combinedwith the squared loss function for the purpose of making predictions.
We showed in Section 1.5.5 that the expected squared loss can be written in the
form


2
E[L] = {y(x) − h(x)} p(x) dx + {h(x) − t}2 p(x, t) dx dt.
(3.37)
Recall that the second term, which is independent of y(x), arises from the intrinsic
noise on the data and represents the minimum achievable value of the expected loss.
The ﬁrst term depends on our choice for the function y(x), and we will seek a solution for y(x) which makes this term a minimum. Because it is nonnegative, thesmallest that we can hope to make this term is zero. If we had an unlimited supply ofdata (and unlimited computational resources), we could in principle ﬁnd the regression function h(x) to any desired degree of accuracy, and this would represent the
optimal choice for y(x). However, in practice we have a data set D containing only
a ﬁnite number N of data points, and consequently we do not know the regressionfunction h(x) exactly.
If we model the h(x) using a parametric function y(x, w) governed by a parameter vector w, then from a Bayesian perspective the uncertainty in our model isexpressed through a posterior distribution over w. A frequentist treatment, however,involves making a point estimate of w based on the data set D, and tries instead
to interpret the uncertainty of this estimate through the following thought experiment. Suppose we had a large number of data sets each of size N and each drawn
independently from the distribution p(t, x). For any given data set D, we can run
our learning algorithm and obtain a prediction function y(x; D). Different data setsfrom the ensemble will give different functions and consequently different values ofthe squared loss. The performance of a particular learning algorithm is then assessedby taking the average over this ensemble of data sets.
Consider the integrand of the ﬁrst term in (3.37), which for a particular data set
D takes the form
(3.38){y(x; D) − h(x)}2 .
Because this quantity will be dependent on the particular data set D, we take its average over the ensemble of data sets. If we add and subtract the quantity ED [y(x; D)]3.2. The Bias-Variance Decomposition

149

inside the braces, and then expand, we obtain{y(x; D) − ED [y(x; D)] + ED [y(x; D)] − h(x)}2= {y(x; D) − ED [y(x; D)]}2 + {ED [y(x; D)] − h(x)}2+2{y(x; D) − ED [y(x; D)]}{ED [y(x; D)] − h(x)}.

(3.39)

We now take the expectation of this expression with respect to D and note that theﬁnal term will vanish, giving
ED {y(x; D) − h(x)}2
= {ED [y(x; D)] − h(x)}2 + ED {y(x; D) − ED [y(x; D)]}2 . (3.40)
(
)*
+ (
)*
+
(bias)2
variance
We see that the expected squared difference between y(x; D) and the regressionfunction h(x) can be expressed as the sum of two terms. The ﬁrst term, called thesquared bias, represents the extent to which the average prediction over all data setsdiffers from the desired regression function. The second term, called the variance,measures the extent to which the solutions for individual data sets vary around theiraverage, and hence this measures the extent to which the function y(x; D) is sensitive
to the particular choice of data set. We shall provide some intuition to support thesedeﬁnitions shortly when we consider a simple example.
So far, we have considered a single input value x. If we substitute this expansion
back into (3.37), we obtain the following decomposition of the expected squared loss
expected loss = (bias)2 + variance + noise
where(bias)

2=variance =noise =

Appendix A(3.41){ED [y(x; D)] − h(x)}2 p(x) dx(3.42)

ED {y(x; D) − ED [y(x; D)]}2 p(x) dx(3.43){h(x) − t}2 p(x, t) dx dt(3.44)and the bias and variance terms now refer to integrated quantities.
Our goal is to minimize the expected loss, which we have decomposed into thesum of a (squared) bias, a variance, and a constant noise term. As we shall see, there
is a trade-off between bias and variance, with very ﬂexible models having low bias
and high variance, and relatively rigid models having high bias and low variance.
The model with the optimal predictive capability is the one that leads to the bestbalance between bias and variance. This is illustrated by considering the sinusoidal
data set from Chapter 1. Here we generate 100 data sets, each containing N = 25
data points, independently from the sinusoidal curve h(x) = sin(2πx). The data
sets are indexed by l = 1, . . . , L, where L = 100, and for each data set D(l) we150

3. LINEAR MODELS FOR REGRESSION

11ln λ = 2.6

t

t

0

0

−1

−1

0

1

x

1

0

−1

−1

1

x

1

0

−1

−1

x

x

1

0

x

1

t

0

0

0

1

ln λ = −2.4

t

1

t

0

0

x

1

ln λ = −0.31

t

0

1

Figure 3.5 Illustration of the dependence of bias and variance on model complexity, governed by a regularization parameter λ, using the sinusoidal data set from Chapter 1. There are L = 100 data sets, each having N = 25
data points, and there are 24 Gaussian basis functions in the model so that the total number of parameters is
M = 25 including the bias parameter. The left column shows the result of ﬁtting the model to the data sets forvarious values of ln λ (for clarity, only 20 of the 100 ﬁts are shown). The right column shows the corresponding
average of the 100 ﬁts (red) along with the sinusoidal function from which the data sets were generated (green).

151

3.2. The Bias-Variance Decomposition
Figure 3.6

Plot of squared bias and variance,
together with their sum, corresponding to the results shown in Figure 3.5. Also shown is the average
test set error for a test data set sizeof 1000 points. The minimum valueof (bias)2 + variance occurs aroundln λ = −0.31, which is close to thevalue that gives the minimum error
on the test data.

0.15(bias)2
variance(bias)2 + variancetest error

0.12
0.09
0.06
0.03
0
−3

−2

−1

0

1

2

ln λ

ﬁt a model with 24 Gaussian basis functions by minimizing the regularized error
function (3.27) to give a prediction function y (l) (x) as shown in Figure 3.5. The
top row corresponds to a large value of the regularization coefﬁcient λ that gives low
variance (because the red curves in the left plot look similar) but high bias (because
the two curves in the right plot are very different). Conversely on the bottom row, forwhich λ is small, there is large variance (shown by the high variability between the
red curves in the left plot) but low bias (shown by the good ﬁt between the averagemodel ﬁt and the original sinusoidal function). Note that the result of averaging manysolutions for the complex model with M = 25 is a very good ﬁt to the regressionfunction, which suggests that averaging may be a beneﬁcial procedure. Indeed, aweighted averaging of multiple solutions lies at the heart of a Bayesian approach,although the averaging is with respect to the posterior distribution of parameters, notwith respect to multiple data sets.
We can also examine the bias-variance trade-off quantitatively for this example.
The average prediction is estimated from1  (l )y(x) =y (x)LL(3.45)

l=1

and the integrated squared bias and integrated variance are then given by1 
2{y(xn ) − h(xn )}N(3.46)

N
L
2
1  1   (l )y (xn ) − y(xn )NL(3.47)

N(bias)2=n=1

variance =

n=1

l=1

where the integral over x weighted by the distribution p(x) is approximated by aﬁnite sum over data points drawn from that distribution. These quantities, alongwith their sum, are plotted as a function of ln λ in Figure 3.6. We see that smallvalues of λ allow the model to become ﬁnely tuned to the noise on each individual

152

3. LINEAR MODELS FOR REGRESSION
data set leading to large variance. Conversely, a large value of λ pulls the weightparameters towards zero leading to large bias.
Although the bias-variance decomposition may provide some interesting insights into the model complexity issue from a frequentist perspective, it is of limited practical value, because the bias-variance decomposition is based on averages
with respect to ensembles of data sets, whereas in practice we have only the single
observed data set. If we had a large number of independent training sets of a given
size, we would be better off combining them into a single large training set, whichof course would reduce the level of over-ﬁtting for a given model complexity.
Given these limitations, we turn in the next section to a Bayesian treatment oflinear basis function models, which not only provides powerful insights into theissues of over-ﬁtting but which also leads to practical techniques for addressing thequestion model complexity.

3.3. Bayesian Linear Regression
In our discussion of maximum likelihood for setting the parameters of a linear regression model, we have seen that the effective model complexity, governed by the
number of basis functions, needs to be controlled according to the size of the data
set. Adding a regularization term to the log likelihood function means the effectivemodel complexity can then be controlled by the value of the regularization coefﬁcient, although the choice of the number and form of the basis functions is of course
still important in determining the overall behaviour of the model.
This leaves the issue of deciding the appropriate model complexity for the particular problem, which cannot be decided simply by maximizing the likelihood function, because this always leads to excessively complex models and over-ﬁtting. Independent hold-out data can be used to determine model complexity, as discussed
in Section 1.3, but this can be both computationally expensive and wasteful of valuable data. We therefore turn to a Bayesian treatment of linear regression, which will
avoid the over-ﬁtting problem of maximum likelihood, and which will also lead toautomatic methods of determining model complexity using the training data alone.
Again, for simplicity we will focus on the case of a single target variable t. Extension to multiple target variables is straightforward and follows the discussion ofSection 3.1.5.

3.3.1 Parameter distribution
We begin our discussion of the Bayesian treatment of linear regression by introducing a prior probability distribution over the model parameters w. For the moment, we shall treat the noise precision parameter β as a known constant. First notethat the likelihood function p(t|w) deﬁned by (3.10) is the exponential of a quadratic
function of w. The corresponding conjugate prior is therefore given by a Gaussian
distribution of the form
(3.48)
p(w) = N (w|m0 , S0 )having mean m0 and covariance S0 .

3.3. Bayesian Linear Regression

Exercise 3.7

153

Next we compute the posterior distribution, which is proportional to the productof the likelihood function and the prior. Due to the choice of a conjugate Gaussian prior distribution, the posterior will also be Gaussian. We can evaluate this
distribution by the usual procedure of completing the square in the exponential, andthen ﬁnding the normalization coefﬁcient using the standard result for a normalized
Gaussian. However, we have already done the necessary work in deriving the general result (2.116), which allows us to write down the posterior distribution directlyin the formp(w|t) = N (w|mN , SN )(3.49)
where

Exercise 3.8

mN

T1= SN S−0 m0 + βΦ t(3.50)

1
S−
N

T
1
S−
0 + βΦ Φ.

(3.51)=

Note that because the posterior distribution is Gaussian, its mode coincides with its
mean. Thus the maximum posterior weight vector is simply given by wMAP = mN .
If we consider an inﬁnitely broad prior S0 = α−1 I with α → 0, the mean mNof the posterior distribution reduces to the maximum likelihood value wML givenby (3.15). Similarly, if N = 0, then the posterior distribution reverts to the prior.
Furthermore, if data points arrive sequentially, then the posterior distribution at any
stage acts as the prior distribution for the subsequent data point, such that the newposterior distribution is again given by (3.49).
For the remainder of this chapter, we shall consider a particular form of Gaussian prior in order to simplify the treatment. Speciﬁcally, we consider a zero-mean
isotropic Gaussian governed by a single precision parameter α so thatp(w|α) = N (w|0, α−1 I)(3.52)and the corresponding posterior distribution over w is then given by (3.49) with
mN
1
S−
N

= βSN ΦT t
= αI + βΦT Φ.

(3.53)(3.54)The log of the posterior distribution is given by the sum of the log likelihood andthe log of the prior and, as a function of w, takes the formαβ{tn − wT φ(xn )}2 − wT w + const.
22N

ln p(w|t) = −(3.55)

n=1

Maximization of this posterior distribution with respect to w is therefore equivalent to the minimization of the sum-of-squares error function with the addition of aquadratic regularization term, corresponding to (3.27) with λ = α/β.
We can illustrate Bayesian learning in a linear basis function model, as well as
the sequential update of a posterior distribution, using a simple example involving
straight-line ﬁtting. Consider a single input variable x, a single target variable t and154

3. LINEAR MODELS FOR REGRESSIONa linear model of the form y(x, w) = w0 + w1 x. Because this has just two adaptive parameters, we can plot the prior and posterior distributions directly in parameter
space. We generate synthetic data from the function f (x, a) = a0 + a1 x with parameter values a0 = −0.3 and a1 = 0.5 by ﬁrst choosing values of xn from the uniform
distribution U(x|−1, 1), then evaluating f (xn , a), and ﬁnally adding Gaussian noisewith standard deviation of 0.2 to obtain the target values tn . Our goal is to recover
the values of a0 and a1 from such data, and we will explore the dependence on thesize of the data set. We assume here that the noise variance is known and hence we
set the precision parameter to its true value β = (1/0.2)2 = 25. Similarly, we ﬁx
the parameter α to 2.0. We shall shortly discuss strategies for determining α andβ from the training data. Figure 3.7 shows the results of Bayesian learning in thismodel as the size of the data set is increased and demonstrates the sequential natureof Bayesian learning in which the current posterior distribution forms the prior whena new data point is observed. It is worth taking time to study this ﬁgure in detail as
it illustrates several important aspects of Bayesian inference. The ﬁrst row of thisﬁgure corresponds to the situation before any data points are observed and shows aplot of the prior distribution in w space together with six samples of the functiony(x, w) in which the values of w are drawn from the prior. In the second row, we
see the situation after observing a single data point. The location (x, t) of the data
point is shown by a blue circle in the right-hand column. In the left-hand column is aplot of the likelihood function p(t|x, w) for this data point as a function of w. Notethat the likelihood function provides a soft constraint that the line must pass close to
the data point, where close is determined by the noise precision β. For comparison,
the true parameter values a0 = −0.3 and a1 = 0.5 used to generate the data set
are shown by a white cross in the plots in the left column of Figure 3.7. When we
multiply this likelihood function by the prior from the top row, and normalize, we
obtain the posterior distribution shown in the middle plot on the second row. Samples of the regression function y(x, w) obtained by drawing samples of w from thisposterior distribution are shown in the right-hand plot. Note that these sample lines
all pass close to the data point. The third row of this ﬁgure shows the effect of observing a second data point, again shown by a blue circle in the plot in the right-handcolumn. The corresponding likelihood function for this second data point alone isshown in the left plot. When we multiply this likelihood function by the posterior
distribution from the second row, we obtain the posterior distribution shown in the
middle plot of the third row. Note that this is exactly the same posterior distribution
as would be obtained by combining the original prior with the likelihood function
for the two data points. This posterior has now been inﬂuenced by two data points,
and because two points are sufﬁcient to deﬁne a line this already gives a relatively
compact posterior distribution. Samples from this posterior distribution give rise to
the functions shown in red in the third column, and we see that these functions passclose to both of the data points. The fourth row shows the effect of observing a totalof 20 data points. The left-hand plot shows the likelihood function for the 20th data
point alone, and the middle plot shows the resulting posterior distribution that has
now absorbed information from all 20 observations. Note how the posterior is muchsharper than in the third row. In the limit of an inﬁnite number of data points, the3.3. Bayesian Linear Regression

Figure 3.7

155

Illustration of sequential Bayesian learning for a simple linear model of the form y(x, w) =w0 + w1 x. A detailed description of this ﬁgure is given in the text.

156

3. LINEAR MODELS FOR REGRESSION
posterior distribution would become a delta function centred on the true parametervalues, shown by the white cross.
Other forms of prior over the parameters can be considered. For instance, we
can generalize the Gaussian prior to give  1/q
M
M
q α
1
α
p(w|α) =exp −|wj |q(3.56)2 2
Γ(1/q)2
j =1

in which q = 2 corresponds to the Gaussian distribution, and only in this case is theprior conjugate to the likelihood function (3.10). Finding the maximum of the posterior distribution over w corresponds to minimization of the regularized error function(3.29). In the case of the Gaussian prior, the mode of the posterior distribution was
equal to the mean, although this will no longer hold if q = 2.

3.3.2 Predictive distribution
In practice, we are not usually interested in the value of w itself but rather inmaking predictions of t for new values of x. This requires that we evaluate thepredictive distribution deﬁned by(3.57)
p(t|t, α, β) = p(t|w, β)p(w|t, α, β) dw

Exercise 3.10

in which t is the vector of target values from the training set, and we have omitted thecorresponding input vectors from the right-hand side of the conditioning statements
to simplify the notation. The conditional distribution p(t|x, w, β) of the target variable is given by (3.8), and the posterior weight distribution is given by (3.49). We
see that (3.57) involves the convolution of two Gaussian distributions, and so making
use of the result (2.115) from Section 8.1.4, we see that the predictive distribution
takes the form
2
(3.58)
p(t|x, t, α, β) = N (t|mT
N φ(x), σN (x))
2where the variance σN(x) of the predictive distribution is given by
2
(x) =
σN

Exercise 3.11

1
+ φ(x)T SN φ(x).
β(3.59)

The ﬁrst term in (3.59) represents the noise on the data whereas the second term
reﬂects the uncertainty associated with the parameters w. Because the noise process
and the distribution of w are independent Gaussians, their variances are additive.
Note that, as additional data points are observed, the posterior distribution becomes2
narrower. As a consequence it can be shown (Qazaz et al., 1997) that σN+1 (x) 2σN (x). In the limit N → ∞, the second term in (3.59) goes to zero, and the varianceof the predictive distribution arises solely from the additive noise governed by theparameter β.
As an illustration of the predictive distribution for Bayesian linear regression
models, let us return to the synthetic sinusoidal data set of Section 1.1. In Figure 3.8,

157

3.3. Bayesian Linear Regression

1

1

t

t
0

0

−1

−1

0

x

1

1

0

x

1

0

x

1

1

t

t
0

0

−1

−1

0

x

1

Figure 3.8 Examples of the predictive distribution (3.58) for a model consisting of 9 Gaussian basis functionsof the form (3.4) using the synthetic sinusoidal data set of Section 1.1. See the text for a detailed discussion.

we ﬁt a model comprising a linear combination of Gaussian basis functions to data
sets of various sizes and then look at the corresponding posterior distributions. Herethe green curves correspond to the function sin(2πx) from which the data points
were generated (with the addition of Gaussian noise). Data sets of size N = 1,
N = 2, N = 4, and N = 25 are shown in the four plots by the blue circles. For
each plot, the red curve shows the mean of the corresponding Gaussian predictive
distribution, and the red shaded region spans one standard deviation either side of
the mean. Note that the predictive uncertainty depends on x and is smallest in theneighbourhood of the data points. Also note that the level of uncertainty decreases
as more data points are observed.
The plots in Figure 3.8 only show the point-wise predictive variance as a function of x. In order to gain insight into the covariance between the predictions at
different values of x, we can draw samples from the posterior distribution over w,
and then plot the corresponding functions y(x, w), as shown in Figure 3.9.

158

3. LINEAR MODELS FOR REGRESSION

1

1

t

t
0

0

−1

−1

0

x

1

1

0

x

1

0

x

1

1

t

t
0

0

−1

−1

0

x

1

Figure 3.9 Plots of the function y(x, w) using samples from the posterior distributions over w corresponding to
the plots in Figure 3.8.

Section 6.4

Exercise 3.12
Exercise 3.13

If we used localized basis functions such as Gaussians, then in regions awayfrom the basis function centres, the contribution from the second term in the predictive variance (3.59) will go to zero, leaving only the noise contribution β −1 . Thus,
the model becomes very conﬁdent in its predictions when extrapolating outside theregion occupied by the basis functions, which is generally an undesirable behaviour.
This problem can be avoided by adopting an alternative Bayesian approach to regression known as a Gaussian process.
Note that, if both w and β are treated as unknown, then we can introduce a
conjugate prior distribution p(w, β) that, from the discussion in Section 2.3.6, will
be given by a Gaussian-gamma distribution (Denison et al., 2002). In this case, thepredictive distribution is a Student’s t-distribution.

3.3. Bayesian Linear Regression

159

Figure 3.10 The equivalent kernel k(x, x ) for the Gaussian basis
functions in Figure 3.1, shown asa plot of x versus x , together withthree slices through this matrix corresponding to three different valuesof x. The data set used to generate
this kernel comprised 200 values ofx equally spaced over the interval(−1, 1).

3.3.3 Equivalent kernel

Chapter 6

The posterior mean solution (3.53) for the linear basis function model has an interesting interpretation that will set the stage for kernel methods, including Gaussian
processes. If we substitute (3.53) into the expression (3.3), we see that the predictive
mean can be written in the formTTy(x, mN ) = mTN φ(x) = βφ(x) SN Φ t =N
βφ(x)T SN φ(xn )tn(3.60)

n=1

where SN is deﬁned by (3.51). Thus the mean of the predictive distribution at a point
x is given by a linear combination of the training set target variables tn , so that we
can writeNy(x, mN ) =k(x, xn )tn(3.61)
n=1

where the function

k(x, x ) = βφ(x)T SN φ(x )(3.62)

is known as the smoother matrix or the equivalent kernel. Regression functions, suchas this, which make predictions by taking linear combinations of the training set
target values are known as linear smoothers. Note that the equivalent kernel depends
on the input values xn from the data set because these appear in the deﬁnition of
SN . The equivalent kernel is illustrated for the case of Gaussian basis functions inFigure 3.10 in which the kernel functions k(x, x ) have been plotted as a function ofx for three different values of x. We see that they are localized around x, and so themean of the predictive distribution at x, given by y(x, mN ), is obtained by forming
a weighted combination of the target values in which data points close to x are given
higher weight than points further removed from x. Intuitively, it seems reasonable
that we should weight local evidence more strongly than distant evidence. Note that
this localization property holds not only for the localized Gaussian basis functions
but also for the nonlocal polynomial and sigmoidal basis functions, as illustrated in
Figure 3.11.

160

3. LINEAR MODELS FOR REGRESSION

Figure 3.11 Examples of equivalent kernels k(x, x ) for x = 0
plotted as a function of x , corresponding (left) to the polynomial basis functions and (right) to the sigmoidal basis functions shown in Figure 3.1. Note that these are localized functions of x even though thecorresponding basis functions arenonlocal.

0.04

0.04

0.02

0.02

0

0

−1

0

1

−1

0

1

Further insight into the role of the equivalent kernel can be obtained by considering the covariance between y(x) and y(x ), which is given by
cov[y(x), y(x )] = cov[φ(x)T w, wT φ(x )]= φ(x)T SN φ(x ) = β −1 k(x, x )(3.63)

where we have made use of (3.49) and (3.62). From the form of the equivalent
kernel, we see that the predictive mean at nearby points will be highly correlated,
whereas for more distant pairs of points the correlation will be smaller.
The predictive distribution shown in Figure 3.8 allows us to visualize the pointwise uncertainty in the predictions, governed by (3.59). However, by drawing samples from the posterior distribution over w, and plotting the corresponding model
functions y(x, w) as in Figure 3.9, we are visualizing the joint uncertainty in theposterior distribution between the y values at two (or more) x values, as governed by
the equivalent kernel.
The formulation of linear regression in terms of a kernel function suggests analternative approach to regression as follows. Instead of introducing a set of basis
functions, which implicitly determines an equivalent kernel, we can instead deﬁne
a localized kernel directly and use this to make predictions for new input vectors x,
given the observed training set. This leads to a practical framework for regression(and classiﬁcation) called Gaussian processes, which will be discussed in detail in
Section 6.4.
We have seen that the effective kernel deﬁnes the weights by which the training
set target values are combined in order to make a prediction at a new value of x, andit can be shown that these weights sum to one, in other wordsNk(x, xn ) = 1(3.64)

n=1

Exercise 3.14

for all values of x. This intuitively pleasing result can easily be proven informallyy (x)by noting that the summation is equivalent to considering the predictive mean 
for a set of target data in which tn = 1 for all n. Provided the basis functions arelinearly independent, that there are more data points than basis functions, and that
one of the basis functions is constant (corresponding to the bias parameter), then it isclear that we can ﬁt the training data exactly and hence that the predictive mean will

3.4. Bayesian Model Comparison

Chapter 6

161

y (x) = 1, from which we obtain (3.64). Note that the kernel function canbe simply 
be negative as well as positive, so although it satisﬁes a summation constraint, thecorresponding predictions are not necessarily convex combinations of the training
set target variables.
Finally, we note that the equivalent kernel (3.62) satisﬁes an important property
shared by kernel functions in general, namely that it can be expressed in the form an
inner product with respect to a vector ψ(x) of nonlinear functions, so thatk(x, z) = ψ(x)T ψ(z)(3.65)

1 /2

where ψ(x) = β 1/2 SN φ(x).

3.4. Bayesian Model Comparison

Section 1.5.4

In Chapter 1, we highlighted the problem of over-ﬁtting as well as the use of crossvalidation as a technique for setting the values of regularization parameters or forchoosing between alternative models. Here we consider the problem of model selection from a Bayesian perspective. In this section, our discussion will be very
general, and then in Section 3.5 we shall see how these ideas can be applied to thedetermination of regularization parameters in linear regression.
As we shall see, the over-ﬁtting associated with maximum likelihood can be
avoided by marginalizing (summing or integrating) over the model parameters instead of making point estimates of their values. Models can then be compared directly on the training data, without the need for a validation set. This allows all
available data to be used for training and avoids the multiple training runs for eachmodel associated with cross-validation. It also allows multiple complexity parameters to be determined simultaneously as part of the training process. For example,
in Chapter 7 we shall introduce the relevance vector machine, which is a Bayesianmodel having one complexity parameter for every training data point.
The Bayesian view of model comparison simply involves the use of probabilities
to represent uncertainty in the choice of model, along with a consistent applicationof the sum and product rules of probability. Suppose we wish to compare a set of L
models {Mi } where i = 1, . . . , L. Here a model refers to a probability distribution
over the observed data D. In the case of the polynomial curve-ﬁtting problem, thedistribution is deﬁned over the set of target values t, while the set of input values X
is assumed to be known. Other types of model deﬁne a joint distributions over X
and t. We shall suppose that the data is generated from one of these models but weare uncertain which one. Our uncertainty is expressed through a prior probability
distribution p(Mi ). Given a training set D, we then wish to evaluate the posterior
distribution(3.66)
p(Mi |D) ∝ p(Mi )p(D|Mi ).
The prior allows us to express a preference for different models. Let us simply
assume that all models are given equal prior probability. The interesting term isthe model evidence p(D|Mi ) which expresses the preference shown by the data for162

3. LINEAR MODELS FOR REGRESSION
different models, and we shall examine this term in more detail shortly. The model
evidence is sometimes also called the marginal likelihood because it can be viewed
as a likelihood function over the space of models, in which the parameters have been
marginalized out. The ratio of model evidences p(D|Mi )/p(D|Mj ) for two models
is known as a Bayes factor (Kass and Raftery, 1995).
Once we know the posterior distribution over models, the predictive distribution
is given, from the sum and product rules, byp(t|x, D) =Lp(t|x, Mi , D)p(Mi |D).

(3.67)

i=1

This is an example of a mixture distribution in which the overall predictive distribution is obtained by averaging the predictive distributions p(t|x, Mi , D) of individualmodels, weighted by the posterior probabilities p(Mi |D) of those models. For instance, if we have two models that are a-posteriori equally likely and one predicts
a narrow distribution around t = a while the other predicts a narrow distribution
around t = b, the overall predictive distribution will be a bimodal distribution withmodes at t = a and t = b, not a single model at t = (a + b)/2.
A simple approximation to model averaging is to use the single most probablemodel alone to make predictions. This is known as model selection.
For a model governed by a set of parameters w, the model evidence is given,from the sum and product rules of probability, by(3.68)
p(D|Mi ) = p(D|w, Mi )p(w|Mi ) dw.
Chapter 11

From a sampling perspective, the marginal likelihood can be viewed as the probability of generating the data set D from a model whose parameters are sampled at
random from the prior. It is also interesting to note that the evidence is precisely thenormalizing term that appears in the denominator in Bayes’ theorem when evaluating
the posterior distribution over parameters becausep(w|D, Mi ) =p(D|w, Mi )p(w|Mi ).
p(D|Mi )(3.69)

We can obtain some insight into the model evidence by making a simple approximation to the integral over parameters. Consider ﬁrst the case of a model having asingle parameter w. The posterior distribution over parameters is proportional top(D|w)p(w), where we omit the dependence on the model Mi to keep the notation
uncluttered. If we assume that the posterior distribution is sharply peaked around themost probable value wMAP , with width ∆wposterior , then we can approximate the integral by the value of the integrand at its maximum times the width of the peak. If wefurther assume that the prior is ﬂat with width ∆wprior so that p(w) = 1/∆wprior ,then we have
∆wposterior(3.70)
p(D) = p(D|w)p(w) dw  p(D|wMAP )∆wprior3.4. Bayesian Model Comparison
Figure 3.12

163

∆wposterior

We can obtain a rough approximation to
the model evidence if we assume thatthe posterior distribution over parameters is sharply peaked around its mode
wMAP .

w

wMAP∆wpriorand so taking logs we obtainln p(D)  ln p(D|wMAP ) + ln

∆wposterior∆wprior.

(3.71)

This approximation is illustrated in Figure 3.12. The ﬁrst term represents the ﬁt to
the data given by the most probable parameter values, and for a ﬂat prior this would
correspond to the log likelihood. The second term penalizes the model according to
its complexity. Because ∆wposterior < ∆wprior this term is negative, and it increases
in magnitude as the ratio ∆wposterior /∆wprior gets smaller. Thus, if parameters are
ﬁnely tuned to the data in the posterior distribution, then the penalty term is large.
For a model having a set of M parameters, we can make a similar approximationfor each parameter in turn. Assuming that all parameters have the same ratio of∆wposterior /∆wprior , we obtain∆wposteriorln p(D)  ln p(D|wMAP ) + M ln.
(3.72)∆wpriorSection 4.4.1

Thus, in this very simple approximation, the size of the complexity penalty increases
linearly with the number M of adaptive parameters in the model. As we increase
the complexity of the model, the ﬁrst term will typically decrease, because a more
complex model is better able to ﬁt the data, whereas the second term will increase
due to the dependence on M . The optimal model complexity, as determined by
the maximum evidence, will be given by a trade-off between these two competing
terms. We shall later develop a more reﬁned version of this approximation, based on
a Gaussian approximation to the posterior distribution.
We can gain further insight into Bayesian model comparison and understandhow the marginal likelihood can favour models of intermediate complexity by considering Figure 3.13. Here the horizontal axis is a one-dimensional representationof the space of possible data sets, so that each point on this axis corresponds to a
speciﬁc data set. We now consider three models M1 , M2 and M3 of successively
increasing complexity. Imagine running these models generatively to produce example data sets, and then looking at the distribution of data sets that result. Any given

164

3. LINEAR MODELS FOR REGRESSION
Figure 3.13

Schematic illustration of the
distribution of data sets for p(D)three models of different complexity, in which M1 is thesimplest and M3 is the mostcomplex. Note that the distributions are normalized. In
this example, for the particular observed data set D0 ,the model M2 with intermediate complexity has the largest
evidence.

M1
M2
M3

D0

D

model can generate a variety of different data sets since the parameters are governedby a prior probability distribution, and for any choice of the parameters there may
be random noise on the target variables. To generate a particular data set from a speciﬁc model, we ﬁrst choose the values of the parameters from their prior distributionp(w), and then for these parameter values we sample the data from p(D|w). A simple model (for example, based on a ﬁrst order polynomial) has little variability andso will generate data sets that are fairly similar to each other. Its distribution p(D)is therefore conﬁned to a relatively small region of the horizontal axis. By contrast,
a complex model (such as a ninth order polynomial) can generate a great variety ofdifferent data sets, and so its distribution p(D) is spread over a large region of thespace of data sets. Because the distributions p(D|Mi ) are normalized, we see thatthe particular data set D0 can have the highest value of the evidence for the modelof intermediate complexity. Essentially, the simpler model cannot ﬁt the data well,whereas the more complex model spreads its predictive probability over too broad a
range of data sets and so assigns relatively small probability to any one of them.
Implicit in the Bayesian model comparison framework is the assumption thatthe true distribution from which the data are generated is contained within the set ofmodels under consideration. Provided this is so, we can show that Bayesian modelcomparison will on average favour the correct model. To see this, consider twomodels M1 and M2 in which the truth corresponds to M1 . For a given ﬁnite dataset, it is possible for the Bayes factor to be larger for the incorrect model. However, if
we average the Bayes factor over the distribution of data sets, we obtain the expected
Bayes factor in the formp(D|M1 )p(D|M1 ) ln
dD
(3.73)
p(D|M2 )Section 1.6.1

where the average has been taken with respect to the true distribution of the data.
This quantity is an example of the Kullback-Leibler divergence and satisﬁes the property of always being positive unless the two distributions are equal in which case it
is zero. Thus on average the Bayes factor will always favour the correct model.
We have seen that the Bayesian framework avoids the problem of over-ﬁtting
and allows models to be compared on the basis of the training data alone. However,3.5. The Evidence Approximation165

a Bayesian approach, like any approach to pattern recognition, needs to make assumptions about the form of the model, and if these are invalid then the results canbe misleading. In particular, we see from Figure 3.12 that the model evidence canbe sensitive to many aspects of the prior, such as the behaviour in the tails. Indeed,
the evidence is not deﬁned if the prior is improper, as can be seen by noting thatan improper prior has an arbitrary scaling factor (in other words, the normalization
coefﬁcient is not deﬁned because the distribution cannot be normalized). If we consider a proper prior and then take a suitable limit in order to obtain an improper prior(for example, a Gaussian prior in which we take the limit of inﬁnite variance) thenthe evidence will go to zero, as can be seen from (3.70) and Figure 3.12. It may,
however, be possible to consider the evidence ratio between two models ﬁrst andthen take a limit to obtain a meaningful answer.
In a practical application, therefore, it will be wise to keep aside an independent
test set of data on which to evaluate the overall performance of the ﬁnal system.

3.5. The Evidence Approximation
In a fully Bayesian treatment of the linear basis function model, we would introduce prior distributions over the hyperparameters α and β and make predictions by
marginalizing with respect to these hyperparameters as well as with respect to theparameters w. However, although we can integrate analytically over either w orover the hyperparameters, the complete marginalization over all of these variables
is analytically intractable. Here we discuss an approximation in which we set thehyperparameters to speciﬁc values determined by maximizing the marginal likelihood function obtained by ﬁrst integrating over the parameters w. This framework
is known in the statistics literature as empirical Bayes (Bernardo and Smith, 1994;
Gelman et al., 2004), or type 2 maximum likelihood (Berger, 1985), or generalized
maximum likelihood (Wahba, 1975), and in the machine learning literature is also
called the evidence approximation (Gull, 1989; MacKay, 1992a).
If we introduce hyperpriors over α and β, the predictive distribution is obtainedby marginalizing over w, α and β so thatp(t|w, β)p(w|t, α, β)p(α, β|t) dw dα dβ(3.74)
p(t|t) =where p(t|w, β) is given by (3.8) and p(w|t, α, β) is given by (3.49) with mN and
SN deﬁned by (3.53) and (3.54) respectively. Here we have omitted the dependence
on the input variable x to keep the notation uncluttered. If the posterior distribution
 then the predictive distribution is and β,p(α, β|t) is sharply peaked around values αobtained simply by marginalizing over w in which α and β are ﬁxed to the values αand β, so that



 dw.
 , β) = p(t|w, β)p(w|t,α
 , β)(3.75)
p(t|t)  p(t|t, α

166

3. LINEAR MODELS FOR REGRESSION
From Bayes’ theorem, the posterior distribution for α and β is given byp(α, β|t) ∝ p(t|α, β)p(α, β).

(3.76)

 and
If the prior is relatively ﬂat, then in the evidence framework the values of α are obtained by maximizing the marginal likelihood function p(t|α, β). We shallβ
proceed by evaluating the marginal likelihood for the linear basis function model andthen ﬁnding its maxima. This will allow us to determine values for these hyperparameters from the training data alone, without recourse to cross-validation. Recall
that the ratio α/β is analogous to a regularization parameter.
As an aside it is worth noting that, if we deﬁne conjugate (Gamma) prior distributions over α and β, then the marginalization over these hyperparameters in (3.74)
can be performed analytically to give a Student’s t-distribution over w (see Section 2.3.7). Although the resulting integral over w is no longer analytically tractable,
it might be thought that approximating this integral, for example using the Laplaceapproximation discussed (Section 4.4) which is based on a local Gaussian approximation centred on the mode of the posterior distribution, might provide a practicalalternative to the evidence framework (Buntine and Weigend, 1991). However, theintegrand as a function of w typically has a strongly skewed mode so that the Laplaceapproximation fails to capture the bulk of the probability mass, leading to poorer results than those obtained by maximizing the evidence (MacKay, 1999).
Returning to the evidence framework, we note that there are two approaches that
we can take to the maximization of the log evidence. We can evaluate the evidence
function analytically and then set its derivative equal to zero to obtain re-estimation
equations for α and β, which we shall do in Section 3.5.2. Alternatively we use atechnique called the expectation maximization (EM) algorithm, which will be discussed in Section 9.3.4 where we shall also show that these two approaches converge
to the same solution.

3.5.1 Evaluation of the evidence function
The marginal likelihood function p(t|α, β) is obtained by integrating over theweight parameters w, so that
(3.77)
p(t|α, β) = p(t|w, β)p(w|α) dw.

Exercise 3.16

Exercise 3.17

One way to evaluate this integral is to make use once again of the result (2.115)
for the conditional distribution in a linear-Gaussian model. Here we shall evaluate
the integral instead by completing the square in the exponent and making use of thestandard form for the normalization coefﬁcient of a Gaussian.
From (3.11), (3.12), and (3.52), we can write the evidence function in the formp(t|α, β) =β2π

N/2 

α M/22π

exp {−E(w)} dw(3.78)

3.5. The Evidence Approximation

167

where M is the dimensionality of w, and we have deﬁned
E(w) = βED (w) + αEW (w)βα2=t − Φw + wT w.
2
2
Exercise 3.18

(3.79)We recognize (3.79) as being equal, up to a constant of proportionality, to the regularized sum-of-squares error function (3.27). We now complete the square over w
giving1
E(w) = E(mN ) + (w − mN )T A(w − mN )(3.80)
2
where we have introduced(3.81)
A = αI + βΦT Φtogether withβα2t − ΦmN  + mTmN .
(3.82)
2
2 N
Note that A corresponds to the matrix of second derivatives of the error function
E(mN ) =A = ∇∇E(w)(3.83)and is known as the Hessian matrix. Here we have also deﬁned mN given by
mN = βA−1 ΦT t.

Exercise 3.19

(3.84)

1
Using (3.54), we see that A = S−
N , and hence (3.84) is equivalent to the previous
deﬁnition (3.53), and therefore represents the mean of the posterior distribution.
The integral over w can now be evaluated simply by appealing to the standard
result for the normalization coefﬁcient of a multivariate Gaussian, giving
exp {−E(w)} dw1= exp{−E(mN )} exp − (w − mN )T A(w − mN ) dw
2

= exp{−E(mN )}(2π)M/2 |A|−1/2 .

(3.85)

Using (3.78) we can then write the log of the marginal likelihood in the formln p(t|α, β) =MNN1ln α +
ln β − E(mN ) − ln |A| −ln(2π)
2
2
2
2(3.86)

which is the required expression for the evidence function.
Returning to the polynomial regression problem, we can plot the model evidence
against the order of the polynomial, as shown in Figure 3.14. Here we have assumed
a prior of the form (1.65) with the parameter α ﬁxed at α = 5 × 10−3 . The formof this plot is very instructive. Referring back to Figure 1.4, we see that the M = 0
polynomial has very poor ﬁt to the data and consequently gives a relatively low value

168

3. LINEAR MODELS FOR REGRESSION
Figure 3.14

Plot of the model evidence versus
the order M , for the polynomial re- −18
gression model, showing that theevidence favours the model with−20
M = 3.

−22
−24
−26

0

2

4

6

8

M

for the evidence. Going to the M = 1 polynomial greatly improves the data ﬁt, andhence the evidence is signiﬁcantly higher. However, in going to M = 2, the dataﬁt is improved only very marginally, due to the fact that the underlying sinusoidal
function from which the data is generated is an odd function and so has no even terms
in a polynomial expansion. Indeed, Figure 1.5 shows that the residual data error is
reduced only slightly in going from M = 1 to M = 2. Because this richer model
suffers a greater complexity penalty, the evidence actually falls in going from M = 1
to M = 2. When we go to M = 3 we obtain a signiﬁcant further improvement indata ﬁt, as seen in Figure 1.4, and so the evidence is increased again, giving thehighest overall evidence for any of the polynomials. Further increases in the valueof M produce only small improvements in the ﬁt to the data but suffer increasing
complexity penalty, leading overall to a decrease in the evidence values. Looking
again at Figure 1.5, we see that the generalization error is roughly constant between
M = 3 and M = 8, and it would be difﬁcult to choose between these models on
the basis of this plot alone. The evidence values, however, show a clear preference
for M = 3, since this is the simplest model which gives a good explanation for theobserved data.

3.5.2 Maximizing the evidence function
Let us ﬁrst consider the maximization of p(t|α, β) with respect to α. This canbe done by ﬁrst deﬁning the following eigenvector equationβΦT Φ ui = λi ui .

(3.87)

From (3.81), it then follows that A has eigenvalues α + λi . Now consider the derivative of the term involving ln |A| in (3.86) with respect to α. We have

 1
d 
d
d
ln(λi + α) =ln |A| =ln (λi + α) =.
(3.88)dαdα
dα
λi + αi

i

i

Thus the stationary points of (3.86) with respect to α satisfy
1
1 1
M
− mT
.
0=
N mN −2α 2
2λi + αi(3.89)

3.5. The Evidence Approximation
Multiplying through by 2α and rearranging, we obtain
 1
= γ.
αmT
N mN = M − αλi + α169(3.90)i

Since there are M terms in the sum over i, the quantity γ can be written
 λi
γ=
.
α + λi(3.91)i

Exercise 3.20

The interpretation of the quantity γ will be discussed shortly. From (3.90) we seethat the value of α that maximizes the marginal likelihood satisﬁesγ
α= T.
(3.92)mN mN
Note that this is an implicit solution for α not only because γ depends on α, but also
because the mode mN of the posterior distribution itself depends on the choice of
α. We therefore adopt an iterative procedure in which we make an initial choice for
α and use this to ﬁnd mN , which is given by (3.53), and also to evaluate γ, which
is given by (3.91). These values are then used to re-estimate α using (3.92), and theprocess repeated until convergence. Note that because the matrix ΦT Φ is ﬁxed, we
can compute its eigenvalues once at the start and then simply multiply these by β toobtain the λi .
It should be emphasized that the value of α has been determined purely by looking at the training data. In contrast to maximum likelihood methods, no independent
data set is required in order to optimize the model complexity.
We can similarly maximize the log marginal likelihood (3.86) with respect to β.
To do this, we note that the eigenvalues λi deﬁned by (3.87) are proportional to β,
and hence dλi /dβ = λi /β givingdd γ
1  λiln |A| == .
(3.93)ln(λi + α) =dβ
dββ
λi + αβii

The stationary point of the marginal likelihood therefore satisﬁes
0=

N
2
N
1 
γ
−
tn − mT−N φ(xn )2β22β(3.94)

n=1

Exercise 3.22

and rearranging we obtain
N
2
1
1 
=
tn − m T.
N φ(xn )β
N −γ(3.95)

n=1

Again, this is an implicit solution for β and can be solved by choosing an initialvalue for β and then using this to calculate mN and γ and then re-estimate β using(3.95), repeating until convergence. If both α and β are to be determined from thedata, then their values can be re-estimated together after each update of γ.

170

3. LINEAR MODELS FOR REGRESSION

Figure 3.15 Contours of the likelihood function (red)and the prior (green) in which the axes in parameter
space have been rotated to align with the eigenvectors
ui of the Hessian. For α = 0, the mode of the posterior is given by the maximum likelihood solution wML ,whereas for nonzero α the mode is at wMAP = mN . In
the direction w1 the eigenvalue λ1 , deﬁned by (3.87), issmall compared with α and so the quantity λ1 /(λ1 + α)
is close to zero, and the corresponding MAP value of
w1 is also close to zero. By contrast, in the direction w2
the eigenvalue λ2 is large compared with α and so thequantity λ2 /(λ2 +α) is close to unity, and the MAP valueof w2 is close to its maximum likelihood value.

w2
u2
wML
wMAP

u1

w1

3.5.3 Effective number of parameters
The result (3.92) has an elegant interpretation (MacKay, 1992a), which provides
insight into the Bayesian solution for α. To see this, consider the contours of the likelihood function and the prior as illustrated in Figure 3.15. Here we have implicitlytransformed to a rotated set of axes in parameter space aligned with the eigenvectors ui deﬁned in (3.87). Contours of the likelihood function are then axis-aligned
ellipses. The eigenvalues λi measure the curvature of the likelihood function, andso in Figure 3.15 the eigenvalue λ1 is small compared with λ2 (because a smaller
curvature corresponds to a greater elongation of the contours of the likelihood function). Because βΦT Φ is a positive deﬁnite matrix, it will have positive eigenvalues,
and so the ratio λi /(λi + α) will lie between 0 and 1. Consequently, the quantity γ
α,deﬁned by (3.91) will lie in the range 0  γ  M . For directions in which λithe corresponding parameter wi will be close to its maximum likelihood value, andthe ratio λi /(λi + α) will be close to 1. Such parameters are called well determinedbecause their values are tightly constrained by the data. Conversely, for directions
α, the corresponding parameters wi will be close to zero, as will thein which λiratios λi /(λi + α). These are directions in which the likelihood function is relatively
insensitive to the parameter value and so the parameter has been set to a small value
by the prior. The quantity γ deﬁned by (3.91) therefore measures the effective total
number of well determined parameters.
We can obtain some insight into the result (3.95) for re-estimating β by comparing it with the corresponding maximum likelihood result given by (3.21). Bothof these formulae express the variance (the inverse precision) as an average of thesquared differences between the targets and the model predictions. However, theydiffer in that the number of data points N in the denominator of the maximum likelihood result is replaced by N − γ in the Bayesian result. We recall from (1.56) thatthe maximum likelihood estimate of the variance for a Gaussian distribution over a3.5. The Evidence Approximation

171

single variable x is given by1 (xn − µML )2
N
N

2
σML
=

(3.96)

n=1

and that this estimate is biased because the maximum likelihood solution µML for
the mean has ﬁtted some of the noise on the data. In effect, this has used up one
degree of freedom in the model. The corresponding unbiased estimate is given by(1.59) and takes the form1 (xn − µML )2 .
N −1
N

2
σMAP
=

(3.97)

n=1

We shall see in Section 10.1.3 that this result can be obtained from a Bayesian treatment in which we marginalize over the unknown mean. The factor of N − 1 in the
denominator of the Bayesian result takes account of the fact that one degree of freedom has been used in ﬁtting the mean and removes the bias of maximum likelihood.
Now consider the corresponding results for the linear regression model. The meanof the target distribution is now given by the function wT φ(x), which contains M
parameters. However, not all of these parameters are tuned to the data. The effective
number of parameters that are determined by the data is γ, with the remaining M −γ
parameters set to small values by the prior. This is reﬂected in the Bayesian result
for the variance that has a factor N − γ in the denominator, thereby correcting forthe bias of the maximum likelihood result.
We can illustrate the evidence framework for setting hyperparameters using thesinusoidal synthetic data set from Section 1.1, together with the Gaussian basis function model comprising 9 basis functions, so that the total number of parameters in
the model is given by M = 10 including the bias. Here, for simplicity of illustration, we have set β to its true value of 11.1 and then used the evidence framework to
determine α, as shown in Figure 3.16.
We can also see how the parameter α controls the magnitude of the parameters{wi }, by plotting the individual parameters versus the effective number γ of parameters, as shown in Figure 3.17.
If we consider the limit N
M in which the number of data points is large in
relation to the number of parameters, then from (3.87) all of the parameters will bewell determined by the data because ΦT Φ involves an implicit sum over data points,
and so the eigenvalues λi increase with the size of the data set. In this case, γ = M ,and the re-estimation equations for α and β become
α =
β

=

M
2EW (mN )N
2ED (mN )(3.98)(3.99)

where EW and ED are deﬁned by (3.25) and (3.26), respectively. These results
can be used as an easy-to-compute approximation to the full evidence re-estimation

172

3. LINEAR MODELS FOR REGRESSION

−5

0
ln α

5

−5

0
ln α

5

Figure 3.16 The left plot shows γ (red curve) and 2αEW (mN ) (blue curve) versus ln α for the sinusoidal
synthetic data set. It is the intersection of these two curves that deﬁnes the optimum value for α given by theevidence procedure. The right plot shows the corresponding graph of log evidence ln p(t|α, β) versus ln α (redcurve) showing that the peak coincides with the crossing point of the curves in the left plot. Also shown is thetest set error (blue curve) showing that the evidence maximum occurs close to the point of best generalization.

formulae, because they do not require evaluation of the eigenvalue spectrum of the
Hessian.
Figure 3.17

Plot of the 10 parameters wi
from the Gaussian basis function
2
model versus the effective number of parameters γ, in which the w i
hyperparameter α is varied in the1range 0  α  ∞ causing γ tovary in the range 0  γ  M .
0

0
8
4
5
2
6
3

−1

1
7

−2

9

0

2

4

6

γ

8

10

3.6. Limitations of Fixed Basis Functions
Throughout this chapter, we have focussed on models comprising a linear combination of ﬁxed, nonlinear basis functions. We have seen that the assumption of linearity
in the parameters led to a range of useful properties including closed-form solutions
to the least-squares problem, as well as a tractable Bayesian treatment. Furthermore,
for a suitable choice of basis functions, we can model arbitrary nonlinearities in the

Exercises

173

mapping from input variables to targets. In the next chapter, we shall study an analogous class of models for classiﬁcation.
It might appear, therefore, that such linear models constitute a general purpose
framework for solving problems in pattern recognition. Unfortunately, there aresome signiﬁcant shortcomings with linear models, which will cause us to turn inlater chapters to more complex models such as support vector machines and neuralnetworks.
The difﬁculty stems from the assumption that the basis functions φj (x) are ﬁxed
before the training data set is observed and is a manifestation of the curse of dimensionality discussed in Section 1.4. As a consequence, the number of basis functions
needs to grow rapidly, often exponentially, with the dimensionality D of the input
space.
Fortunately, there are two properties of real data sets that we can exploit to help
alleviate this problem. First of all, the data vectors {xn } typically lie close to a nonlinear manifold whose intrinsic dimensionality is smaller than that of the input space
as a result of strong correlations between the input variables. We will see an exampleof this when we consider images of handwritten digits in Chapter 12. If we are using
localized basis functions, we can arrange that they are scattered in input space only
in regions containing data. This approach is used in radial basis function networks
and also in support vector and relevance vector machines. Neural network models,
which use adaptive basis functions having sigmoidal nonlinearities, can adapt theparameters so that the regions of input space over which the basis functions varycorresponds to the data manifold. The second property is that target variables may
have signiﬁcant dependence on only a small number of possible directions within thedata manifold. Neural networks can exploit this property by choosing the directionsin input space to which the basis functions respond.

Exercises

3.1 ( ) www Show that the ‘tanh’ function and the logistic sigmoid function (3.6)
are related bytanh(a) = 2σ(2a) − 1.
(3.100)Hence show that a general linear combination of logistic sigmoid functions of theformMx − µ jwj σ(3.101)y(x, w) = w0 +s
j =1

is equivalent to a linear combination of ‘tanh’ functions of the formy(x, u) = u0 +

M

j =1

uj tanhx − µ js(3.102)and ﬁnd expressions to relate the new parameters {u1 , . . . , uM } to the original parameters {w1 , . . . , wM }.

174

3. LINEAR MODELS FOR REGRESSION
3.2 () Show that the matrix

Φ(ΦT Φ)−1 ΦT(3.103)takes any vector v and projects it onto the space spanned by the columns of Φ. Use
this result to show that the least-squares solution (3.15) corresponds to an orthogonal
projection of the vector t onto the manifold S as shown in Figure 3.2.
3.3 ( ) Consider a data set in which each data point tn is associated with a weighting
factor rn > 0, so that the sum-of-squares error function becomes
ED (w) =N
21 
rn tn − wT φ(xn ) .
2(3.104)n=1

Find an expression for the solution w that minimizes this error function. Give twoalternative interpretations of the weighted sum-of-squares error function in terms of(i) data dependent noise variance and (ii) replicated data points.
3.4 ( ) www

Consider a linear model of the formy(x, w) = w0 +

D


wi xi

(3.105)i=1

together with a sum-of-squares error function of the form
12
ED (w) ={y(xn , w) − tn } .
2N(3.106)n=1

Now suppose that Gaussian noise i with zero mean and variance σ 2 is added independently to each of the input variables xi . By making use of E[i ] = 0 andE[i j ] = δij σ 2 , show that minimizing ED averaged over the noise distribution isequivalent to minimizing the sum-of-squares error for noise-free input variables withthe addition of a weight-decay regularization term, in which the bias parameter w0
is omitted from the regularizer.
3.5 ( ) www Using the technique of Lagrange multipliers, discussed in Appendix E,
show that minimization of the regularized error function (3.29) is equivalent to minimizing the unregularized sum-of-squares error (3.12) subject to the constraint (3.30).
Discuss the relationship between the parameters η and λ.
3.6 ( ) www Consider a linear basis function regression model for a multivariate
target variable t having a Gaussian distribution of the form

wherep(t|W, Σ) = N (t|y(x, W), Σ)(3.107)y(x, W) = WT φ(x)(3.108)

Exercises

175

together with a training data set comprising input basis vectors φ(xn ) and corresponding target vectors tn , with n = 1, . . . , N . Show that the maximum likelihood
solution WML for the parameter matrix W has the property that each column isgiven by an expression of the form (3.15), which was the solution for an isotropic
noise distribution. Note that this is independent of the covariance matrix Σ. Show
that the maximum likelihood solution for Σ is given by1 
Ttn − WML
φ(xn )
Σ=NN

T
tn − WML
φ(xn )

T.

(3.109)n=1

3.7 ( ) By using the technique of completing the square, verify the result (3.49) for theposterior distribution of the parameters w in the linear basis function model in whichmN and SN are deﬁned by (3.50) and (3.51) respectively.
3.8 ( ) www Consider the linear basis function model in Section 3.1, and supposethat we have already observed N data points, so that the posterior distribution over
w is given by (3.49). This posterior can be regarded as the prior for the next observation. By considering an additional data point (xN +1 , tN +1 ), and by completingthe square in the exponential, show that the resulting posterior distribution is again
given by (3.49) but with SN replaced by SN +1 and mN replaced by mN +1 .
3.9 ( ) Repeat the previous exercise but instead of completing the square by hand,make use of the general result for linear-Gaussian models given by (2.116).
3.10 ( ) www By making use of the result (2.115) to evaluate the integral in (3.57),
verify that the predictive distribution for the Bayesian linear regression model isgiven by (3.58) in which the input-dependent variance is given by (3.59).
3.11 ( ) We have seen that, as the size of a data set increases, the uncertainty associatedwith the posterior distribution over model parameters decreases. Make use of thematrix identity (Appendix C)
M + vvT−1= M−1 −

(M−1 v) vT M−1
1 + vT M−1 v(3.110)2(x) associated with the linear regression function
to show that the uncertainty σNgiven by (3.59) satisﬁes22
σN(3.111)+1 (x)  σN (x).

3.12 ( ) We saw in Section 2.3.6 that the conjugate prior for a Gaussian distribution
with unknown mean and unknown precision (inverse variance) is a normal-gamma
distribution. This property also holds for the case of the conditional Gaussian distribution p(t|x, w, β) of the linear regression model. If we consider the likelihood
function (3.10), then the conjugate prior for w and β is given byp(w, β) = N (w|m0 , β −1 S0 )Gam(β|a0 , b0 ).

(3.112)

176

3. LINEAR MODELS FOR REGRESSIONShow that the corresponding posterior distribution takes the same functional form,
so thatp(w, β|t) = N (w|mN , β −1 SN )Gam(β|aN , bN )(3.113)and ﬁnd expressions for the posterior parameters mN , SN , aN , and bN .
3.13 ( ) Show that the predictive distribution p(t|x, t) for the model discussed in Exercise 3.12 is given by a Student’s t-distribution of the formp(t|x, t) = St(t|µ, λ, ν)(3.114)and obtain expressions for µ, λ and ν.
3.14 ( ) In this exercise, we explore in more detail the properties of the equivalent
kernel deﬁned by (3.62), where SN is deﬁned by (3.54). Suppose that the basis
functions φj (x) are linearly independent and that the number N of data points isgreater than the number M of basis functions. Furthermore, let one of the basis
functions be constant, say φ0 (x) = 1. By taking suitable linear combinations ofthese basis functions, we can construct a new basis set ψj (x) spanning the same
space but that are orthonormal, so thatNψj (xn )ψk (xn ) = Ijk(3.115)n=1

where Ijk is deﬁned to be 1 if j = k and 0 otherwise, and we take ψ0 (x) = 1. Show
that for α = 0, the equivalent kernel can be written as k(x, x ) = ψ(x)T ψ(x )where ψ = (ψ1 , . . . , ψM )T . Use this result to show that the kernel satisﬁes thesummation constraintNk(x, xn ) = 1.
(3.116)n=1

3.15 ( ) www Consider a linear basis function model for regression in which the parameters α and β are set using the evidence framework. Show that the function
E(mN ) deﬁned by (3.82) satisﬁes the relation 2E(mN ) = N .
3.16 ( ) Derive the result (3.86) for the log evidence function p(t|α, β) of the linear
regression model by making use of (2.115) to evaluate the integral (3.77) directly.
3.17 ( ) Show that the evidence function for the Bayesian linear regression model canbe written in the form (3.78) in which E(w) is deﬁned by (3.79).
3.18 ( ) www By completing the square over w, show that the error function (3.79)
in Bayesian linear regression can be written in the form (3.80).
3.19 ( ) Show that the integration over w in the Bayesian linear regression model gives
the result (3.85). Hence show that the log marginal likelihood is given by (3.86).

Exercises

177

3.20 ( ) www Starting from (3.86) verify all of the steps needed to show that maximization of the log marginal likelihood function (3.86) with respect to α leads to there-estimation equation (3.92).
3.21 ( ) An alternative way to derive the result (3.92) for the optimal value of α in theevidence framework is to make use of the identity


d
−1 d
ln |A| = Tr A
A .
(3.117)dαdαProve this identity by considering the eigenvalue expansion of a real, symmetricmatrix A, and making use of the standard results for the determinant and trace ofA expressed in terms of its eigenvalues (Appendix C). Then make use of (3.117) toderive (3.92) starting from (3.86).
3.22 ( ) Starting from (3.86) verify all of the steps needed to show that maximization of the log marginal likelihood function (3.86) with respect to β leads to there-estimation equation (3.95).
3.23 ( ) www Show that the marginal probability of the data, in other words themodel evidence, for the model described in Exercise 3.12 is given byp(t) =

ba0 0 Γ(aN ) |SN |1/21(2π)N/2 baNN Γ(a0 ) |S0 |1/2

(3.118)by ﬁrst marginalizing with respect to w and then with respect to β.
3.24 () Repeat the previous exercise but now use Bayes’ theorem in the formp(t) =

p(t|w, β)p(w, β)p(w, β|t)(3.119)and then substitute for the prior and posterior distributions and the likelihood function in order to derive the result (3.118).

4

LinearModels forClassiﬁcation

In the previous chapter, we explored a class of regression models having particularly
simple analytical and computational properties. We now discuss an analogous classof models for solving classiﬁcation problems. The goal in classiﬁcation is to take an
input vector x and to assign it to one of K discrete classes Ck where k = 1, . . . , K.
In the most common scenario, the classes are taken to be disjoint, so that each input isassigned to one and only one class. The input space is thereby divided into decision
regions whose boundaries are called decision boundaries or decision surfaces. In
this chapter, we consider linear models for classiﬁcation, by which we mean that thedecision surfaces are linear functions of the input vector x and hence are deﬁnedby (D − 1)-dimensional hyperplanes within the D-dimensional input space. Data
sets whose classes can be separated exactly by linear decision surfaces are said to belinearly separable.
For regression problems, the target variable t was simply the vector of real numbers whose values we wish to predict. In the case of classiﬁcation, there are various

179

180

4. LINEAR MODELS FOR CLASSIFICATION
ways of using target values to represent class labels. For probabilistic models, themost convenient, in the case of two-class problems, is the binary representation inwhich there is a single target variable t ∈ {0, 1} such that t = 1 represents class C1
and t = 0 represents class C2 . We can interpret the value of t as the probability thatthe class is C1 , with the values of probability taking only the extreme values of 0 and
1. For K > 2 classes, it is convenient to use a 1-of-K coding scheme in which t isa vector of length K such that if the class is Cj , then all elements tk of t are zero
except element tj , which takes the value 1. For instance, if we have K = 5 classes,
then a pattern from class 2 would be given the target vectort = (0, 1, 0, 0, 0)T .

(4.1)

Again, we can interpret the value of tk as the probability that the class is Ck . For
nonprobabilistic models, alternative choices of target variable representation will
sometimes prove convenient.
In Chapter 1, we identiﬁed three distinct approaches to the classiﬁcation problem. The simplest involves constructing a discriminant function that directly assigns
each vector x to a speciﬁc class. A more powerful approach, however, models theconditional probability distribution p(Ck |x) in an inference stage, and then subsequently uses this distribution to make optimal decisions. By separating inference
and decision, we gain numerous beneﬁts, as discussed in Section 1.5.4. There are
two different approaches to determining the conditional probabilities p(Ck |x). One
technique is to model them directly, for example by representing them as parametricmodels and then optimizing the parameters using a training set. Alternatively, we
can adopt a generative approach in which we model the class-conditional densities
given by p(x|Ck ), together with the prior probabilities p(Ck ) for the classes, and thenwe compute the required posterior probabilities using Bayes’ theoremp(Ck |x) =p(x|Ck )p(Ck )
.
p(x)

(4.2)

We shall discuss examples of all three approaches in this chapter.
In the linear regression models considered in Chapter 3, the model predictiony(x, w) was given by a linear function of the parameters w. In the simplest case,the model is also linear in the input variables and therefore takes the form y(x) =wT x + w0 , so that y is a real number. For classiﬁcation problems, however, we wish
to predict discrete class labels, or more generally posterior probabilities that lie in
the range (0, 1). To achieve this, we consider a generalization of this model in which
we transform the linear function of w using a nonlinear function f ( · ) so thaty(x) = f wT x + w0 .

(4.3)

In the machine learning literature f ( · ) is known as an activation function, whereas
its inverse is called a link function in the statistics literature. The decision surfaces
correspond to y(x) = constant, so that wT x + w0 = constant and hence the decision surfaces are linear functions of x, even if the function f (·) is nonlinear. For thisreason, the class of models described by (4.3) are called generalized linear models

4.1. Discriminant Functions

181(McCullagh and Nelder, 1989). Note, however, that in contrast to the models used
for regression, they are no longer linear in the parameters due to the presence of thenonlinear function f (·). This will lead to more complex analytical and computational properties than for linear regression models. Nevertheless, these models arestill relatively simple compared to the more general nonlinear models that will be
studied in subsequent chapters.
The algorithms discussed in this chapter will be equally applicable if we ﬁrst
make a ﬁxed nonlinear transformation of the input variables using a vector of basis
functions φ(x) as we did for regression models in Chapter 3. We begin by considering classiﬁcation directly in the original input space x, while in Section 4.3 we shallﬁnd it convenient to switch to a notation involving basis functions for consistency
with later chapters.

4.1. Discriminant Functions
A discriminant is a function that takes an input vector x and assigns it to one of K
classes, denoted Ck . In this chapter, we shall restrict attention to linear discriminants,
namely those for which the decision surfaces are hyperplanes. To simplify the discussion, we consider ﬁrst the case of two classes and then investigate the extension
to K > 2 classes.

4.1.1 Two classes
The simplest representation of a linear discriminant function is obtained by taking a linear function of the input vector so thaty(x) = wT x + w0

(4.4)

where w is called a weight vector, and w0 is a bias (not to be confused with bias in
the statistical sense). The negative of the bias is sometimes called a threshold. An
input vector x is assigned to class C1 if y(x)  0 and to class C2 otherwise. The corresponding decision boundary is therefore deﬁned by the relation y(x) = 0, whichcorresponds to a (D − 1)-dimensional hyperplane within the D-dimensional input
space. Consider two points xA and xB both of which lie on the decision surface.
Because y(xA ) = y(xB ) = 0, we have wT (xA − xB ) = 0 and hence the vector w isorthogonal to every vector lying within the decision surface, and so w determines theorientation of the decision surface. Similarly, if x is a point on the decision surface,then y(x) = 0, and so the normal distance from the origin to the decision surface isgiven by
wT x
w0
=−
.
(4.5)
w
w
We therefore see that the bias parameter w0 determines the location of the decision
surface. These properties are illustrated for the case of D = 2 in Figure 4.1.
Furthermore, we note that the value of y(x) gives a signed measure of the perpendicular distance r of the point x from the decision surface. To see this, consider

182

4. LINEAR MODELS FOR CLASSIFICATION

Figure 4.1 Illustration of the geometry of ax2
y>0
linear discriminant function in two dimensions.
y=0
The decision surface, shown in red, is perpenR1
dicular to w, and its displacement from the y < 0origin is controlled by the bias parameter w0 .
R2Also, the signed orthogonal distance of a general point x from the decision surface is givenby y(x)/w.

x
w

y(x)
w

x⊥
x1
−w0
w

an arbitrary point x and let x⊥ be its orthogonal projection onto the decision surface,
so thatw.
(4.6)
x = x⊥ + r
w
Multiplying both sides of this result by wT and adding w0 , and making use of y(x) =wT x + w0 and y(x⊥ ) = wT x⊥ + w0 = 0, we have
r=

y(x)
.
w

(4.7)

This result is illustrated in Figure 4.1.
As with the linear regression models in Chapter 3, it is sometimes convenient
to use a more compact notation in which we introduce an additional dummy ‘input’ = (w0 , w) and x
 = (x0 , x) so thatvalue x0 = 1 and then deﬁne w

 Tx.
y(x) = w(4.8)

In this case, the decision surfaces are D-dimensional hyperplanes passing throughthe origin of the D + 1-dimensional expanded input space.

4.1.2 Multiple classes
Now consider the extension of linear discriminants to K > 2 classes. We might
be tempted be to build a K-class discriminant by combining a number of two-classdiscriminant functions. However, this leads to some serious difﬁculties (Duda andHart, 1973) as we now show.
Consider the use of K −1 classiﬁers each of which solves a two-class problem of
separating points in a particular class Ck from points not in that class. This is known
as a one-versus-the-rest classiﬁer. The left-hand example in Figure 4.2 shows an183

4.1. Discriminant Functions

C3

C1
?
R1
R1
R2
C1

R3

R3

C1

C2

?

C2

C3

R2

not C1

C2

not C2

Figure 4.2 Attempting to construct a K class discriminant from a set of two class discriminants leads to ambiguous regions, shown in green. On the left is an example involving the use of two discriminants designed to
distinguish points in class Ck from points not in class Ck . On the right is an example involving three discriminantfunctions each of which is used to separate a pair of classes Ck and Cj .

example involving three classes where this approach leads to regions of input spacethat are ambiguously classiﬁed.
An alternative is to introduce K(K − 1)/2 binary discriminant functions, one
for every possible pair of classes. This is known as a one-versus-one classiﬁer. Eachpoint is then classiﬁed according to a majority vote amongst the discriminant functions. However, this too runs into the problem of ambiguous regions, as illustrated
in the right-hand diagram of Figure 4.2.
We can avoid these difﬁculties by considering a single K-class discriminantcomprising K linear functions of the formyk (x) = wkT x + wk0(4.9)

and then assigning a point x to class Ck if yk (x) > yj (x) for all j = k. The decisionboundary between class Ck and class Cj is therefore given by yk (x) = yj (x) andhence corresponds to a (D − 1)-dimensional hyperplane deﬁned by(wk − wj )T x + (wk0 − wj 0 ) = 0.

(4.10)

This has the same form as the decision boundary for the two-class case discussed in
Section 4.1.1, and so analogous geometrical properties apply.
The decision regions of such a discriminant are always singly connected andconvex. To see this, consider two points xA and xB both of which lie inside decision
 that lies on the line connecting
region Rk , as illustrated in Figure 4.3. Any point x
xA and xB can be expressed in the formx
 = λxA + (1 − λ)xB

(4.11)

184

4. LINEAR MODELS FOR CLASSIFICATION
Figure 4.3

Illustration of the decision regions for a multiclass linear discriminant, with the decisionboundaries shown in red. If two points xAand xB both lie inside the same decision reb that lies on the line
gion Rk , then any point x
connecting these two points must also lie in
Rk , and hence the decision region must be
singly connected and convex.

Rj
Ri
Rk
xA

xB
x̂

where 0  λ  1. From the linearity of the discriminant functions, it follows thatyk (x) = λyk (xA ) + (1 − λ)yk (xB ).

(4.12)

Because both xA and xB lie inside Rk , it follows that yk (xA ) > yj (xA ), and) > yj (x), and so x
 also liesyk (xB ) > yj (xB ), for all j = k, and hence yk (xinside Rk . Thus Rk is singly connected and convex.
Note that for two classes, we can either employ the formalism discussed here,
based on two discriminant functions y1 (x) and y2 (x), or else use the simpler but
equivalent formulation described in Section 4.1.1 based on a single discriminantfunction y(x).
We now explore three approaches to learning the parameters of linear discriminant functions, based on least squares, Fisher’s linear discriminant, and the perceptron algorithm.

4.1.3 Least squares for classiﬁcation
In Chapter 3, we considered models that were linear functions of the parameters, and we saw that the minimization of a sum-of-squares error function led to asimple closed-form solution for the parameter values. It is therefore tempting to see
if we can apply the same formalism to classiﬁcation problems. Consider a generalclassiﬁcation problem with K classes, with a 1-of-K binary coding scheme for thetarget vector t. One justiﬁcation for using least squares in such a context is that it
approximates the conditional expectation E[t|x] of the target values given the input
vector. For the binary coding scheme, this conditional expectation is given by thevector of posterior class probabilities. Unfortunately, however, these probabilities
are typically approximated rather poorly, indeed the approximations can have values
outside the range (0, 1), due to the limited ﬂexibility of a linear model as we shallsee shortly.
Each class Ck is described by its own linear model so thatyk (x) = wkT x + wk0(4.13)

where k = 1, . . . , K. We can conveniently group these together using vector notation so that, Tx(4.14)
y(x) = W

4.1. Discriminant Functions

185, is a matrix whose k th column comprises the D + 1-dimensional vector
where Ww
 k = (wk0 , wkT )T and x
 is the corresponding augmented input vector (1, xT )T witha dummy input x0 = 1. This representation was discussed in detail in Section 3.1. A
 kT x is largest.
new input x is then assigned to the class for which the output yk = w,
We now determine the parameter matrix W by minimizing a sum-of-squares
error function, as we did for regression in Chapter 3. Consider a training data set{xn , tn } where n = 1, . . . , N , and deﬁne a matrix T whose nth row is the vector tT
n,
th
T

n . The sum-of-squares error function
together with a matrix X whose n row is x
can then be written as, = 1 Tr (XW, − T)T (XW, − T) .
ED (W)(4.15)
2, to zero, and rearranging, we then obtain the
Setting the derivative with respect to W
, in the form
solution for W, = (X
 T X)
 −1 X
 TT = X †TW(4.16)

 † is the pseudo-inverse of the matrix X,
 as discussed in Section 3.1.1. We
where X
then obtain the discriminant function in the form
 T
† x
, Tx
 = TT X
.
(4.17)
y(x) = W
An interesting property of least-squares solutions with multiple target variables
is that if every target vector in the training set satisﬁes some linear constraint
aT t n + b = 0
Exercise 4.2

Section 2.3.7(4.18)

for some constants a and b, then the model prediction for any value of x will satisfy
the same constraint so that
(4.19)aT y(x) + b = 0.
Thus if we use a 1-of-K coding scheme for K classes, then the predictions madeby the model will have the property that the elements of y(x) will sum to 1 for any
value of x. However, this summation constraint alone is not sufﬁcient to allow themodel outputs to be interpreted as probabilities because they are not constrained tolie within the interval (0, 1).
The least-squares approach gives an exact closed-form solution for the discriminant function parameters. However, even as a discriminant function (where we use it
to make decisions directly and dispense with any probabilistic interpretation) it suffers from some severe problems. We have already seen that least-squares solutions
lack robustness to outliers, and this applies equally to the classiﬁcation application,
as illustrated in Figure 4.4. Here we see that the additional data points in the righthand ﬁgure produce a signiﬁcant change in the location of the decision boundary,even though these point would be correctly classiﬁed by the original decision boundary in the left-hand ﬁgure. The sum-of-squares error function penalizes predictions
that are ‘too correct’ in that they lie a long way on the correct side of the decision

186

4. LINEAR MODELS FOR CLASSIFICATION

4

4

2

2

0

0

−2

−2

−4

−4

−6

−6

−8

−8

−4

−2

0

2

4

6

8

−4

−2

0

2

4

6

8

Figure 4.4 The left plot shows data from two classes, denoted by red crosses and blue circles, together withthe decision boundary found by least squares (magenta curve) and also by the logistic regression model (green
curve), which is discussed later in Section 4.3.2. The right-hand plot shows the corresponding results obtained
when extra data points are added at the bottom left of the diagram, showing that least squares is highly sensitive
to outliers, unlike logistic regression.

boundary. In Section 7.1.2, we shall consider several alternative error functions forclassiﬁcation and we shall see that they do not suffer from this difﬁculty.
However, problems with least squares can be more severe than simply lack ofrobustness, as illustrated in Figure 4.5. This shows a synthetic data set drawn fromthree classes in a two-dimensional input space (x1 , x2 ), having the property that linear decision boundaries can give excellent separation between the classes. Indeed,
the technique of logistic regression, described later in this chapter, gives a satisfactory solution as seen in the right-hand plot. However, the least-squares solution gives
poor results, with only a small region of the input space assigned to the green class.
The failure of least squares should not surprise us when we recall that it corresponds to maximum likelihood under the assumption of a Gaussian conditional
distribution, whereas binary target vectors clearly have a distribution that is far fromGaussian. By adopting more appropriate probabilistic models, we shall obtain classiﬁcation techniques with much better properties than least squares. For the moment,
however, we continue to explore alternative nonprobabilistic methods for setting theparameters in the linear classiﬁcation models.

4.1.4 Fisher’s linear discriminant
One way to view a linear classiﬁcation model is in terms of dimensionality
reduction. Consider ﬁrst the case of two classes, and suppose we take the D-

187

4.1. Discriminant Functions
6

6

4

4

2

2

0

0

−2

−2

−4

−4

−6
−6

−4

−2

0

2

4

−6
−6

6

−4

−2

0

2

4

6

Figure 4.5 Example of a synthetic data set comprising three classes, with training data points denoted in red(×), green (+), and blue (◦). Lines denote the decision boundaries, and the background colours denote therespective classes of the decision regions. On the left is the result of using a least-squares discriminant. We seethat the region of input space assigned to the green class is too small and so most of the points from this classare misclassiﬁed. On the right is the result of using logistic regressions as described in Section 4.3.2 showing
correct classiﬁcation of the training data.

dimensional input vector x and project it down to one dimension using
y = wT x.

(4.20)

If we place a threshold on y and classify y  −w0 as class C1 , and otherwise class
C2 , then we obtain our standard linear classiﬁer discussed in the previous section.
In general, the projection onto one dimension leads to a considerable loss of information, and classes that are well separated in the original D-dimensional space may
become strongly overlapping in one dimension. However, by adjusting the components of the weight vector w, we can select a projection that maximizes the classseparation. To begin with, consider a two-class problem in which there are N1 pointsof class C1 and N2 points of class C2 , so that the mean vectors of the two classes are
given by
1 
1 
xn ,
m2 =
xn .
(4.21)
m1 =N1
N2n ∈ C1n ∈ C2The simplest measure of the separation of the classes, when projected onto w, is theseparation of the projected class means. This suggests that we might choose w so as
to maximize(4.22)
m2 − m1 = wT (m2 − m1 )where

mk = w T mk

(4.23)

188

4. LINEAR MODELS FOR CLASSIFICATION

4

4

2

2

0

0

−2

−2

−2

2

6

−2

2

6

Figure 4.6 The left plot shows samples from two classes (depicted in red and blue) along with the histograms
resulting from projection onto the line joining the class means. Note that there is considerable class overlap in
the projected space. The right plot shows the corresponding projection based on the Fisher linear discriminant,showing the greatly improved class separation.

Appendix E
Exercise 4.4

is the mean of the projected data from class Ck . However, this expression can bemade arbitrarily large simply by increasing the magnitude of
w. To solve thisproblem, we could constrain w to have unit length, so that i wi2 = 1. Usinga Lagrange multiplier to perform the constrained maximization, we then ﬁnd thatw ∝ (m2 − m1 ). There is still a problem with this approach, however, as illustrated
in Figure 4.6. This shows two classes that are well separated in the original twodimensional space (x1 , x2 ) but that have considerable overlap when projected onto
the line joining their means. This difﬁculty arises from the strongly nondiagonalcovariances of the class distributions. The idea proposed by Fisher is to maximize
a function that will give a large separation between the projected class means while
also giving a small variance within each class, thereby minimizing the class overlap.
The projection formula (4.20) transforms the set of labelled data points in x
into a labelled set in the one-dimensional space y. The within-class variance of thetransformed data from class Ck is therefore given bys2k =(yn − mk )2(4.24)
n∈Ck

where yn = wT xn . We can deﬁne the total within-class variance for the whole
data set to be simply s21 + s22 . The Fisher criterion is deﬁned to be the ratio of the
between-class variance to the within-class variance and is given by
J(w) =

Exercise 4.5(m2 − m1 )2.
s21 + s22

(4.25)

We can make the dependence on w explicit by using (4.20), (4.23), and (4.24) torewrite the Fisher criterion in the form

189

4.1. Discriminant Functions

J(w) =

w T SB w
w T SW w(4.26)

where SB is the between-class covariance matrix and is given by
SB = (m2 − m1 )(m2 − m1 )Tand SW is the total within-class covariance matrix, given by(xn − m1 )(xn − m1 )T +(xn − m2 )(xn − m2 )T .
SW =
n∈C1

(4.27)(4.28)

n∈C2

Differentiating (4.26) with respect to w, we ﬁnd that J(w) is maximized when
(wT SB w)SW w = (wT SW w)SB w.

(4.29)

From (4.27), we see that SB w is always in the direction of (m2 −m1 ). Furthermore,
we do not care about the magnitude of w, only its direction, and so we can drop the1scalar factors (wT SB w) and (wT SW w). Multiplying both sides of (4.29) by S−W
we then obtain
1(4.30)w ∝ S−W (m2 − m1 ).
Note that if the within-class covariance is isotropic, so that SW is proportional to theunit matrix, we ﬁnd that w is proportional to the difference of the class means, as
discussed above.
The result (4.30) is known as Fisher’s linear discriminant, although strictly it
is not a discriminant but rather a speciﬁc choice of direction for projection of thedata down to one dimension. However, the projected data can subsequently be used
to construct a discriminant, by choosing a threshold y0 so that we classify a new
point as belonging to C1 if y(x)  y0 and classify it as belonging to C2 otherwise.
For example, we can model the class-conditional densities p(y|Ck ) using Gaussian
distributions and then use the techniques of Section 1.2.4 to ﬁnd the parametersof the Gaussian distributions by maximum likelihood. Having found Gaussian approximations to the projected classes, the formalism of Section 1.5.1 then gives anexpression for the optimal threshold. Some justiﬁcation for the Gaussian assumption
comes from the central limit theorem by noting that y = wT x is the sum of a set of
random variables.

4.1.5 Relation to least squares
The least-squares approach to the determination of a linear discriminant was
based on the goal of making the model predictions as close as possible to a set oftarget values. By contrast, the Fisher criterion was derived by requiring maximum
class separation in the output space. It is interesting to see the relationship between
these two approaches. In particular, we shall show that, for the two-class problem,
the Fisher criterion can be obtained as a special case of least squares.
So far we have considered 1-of-K coding for the target values. If, however, we
adopt a slightly different target coding scheme, then the least-squares solution for190

4. LINEAR MODELS FOR CLASSIFICATION
the weights becomes equivalent to the Fisher solution (Duda and Hart, 1973). In
particular, we shall take the targets for class C1 to be N/N1 , where N1 is the numberof patterns in class C1 , and N is the total number of patterns. This target value
approximates the reciprocal of the prior probability for class C1 . For class C2 , weshall take the targets to be −N/N2 , where N2 is the number of patterns in class C2 .
The sum-of-squares error function can be written
1 T
2
w xn + w0 − tn .
2NE=

(4.31)

n=1

Setting the derivatives of E with respect to w0 and w to zero, we obtain respectivelyN

wT xn + w0 − tn

= 0(4.32)

wT xn + w0 − tn xn= 0.

(4.33)

n=1
N

n=1

From (4.32), and making use of our choice of target coding scheme for the tn , weobtain an expression for the bias in the form
w0 = −wT mwhere we have usedN

tn = N 1

n=1

N
N
− N2
=0
N1
N2

(4.34)(4.35)

and where m is the mean of the total data set and is given by11 
xn = (N1 m1 + N2 m2 ).
m=
N
N
N

(4.36)

n=1

Exercise 4.6

After some straightforward algebra, and again making use of the choice of tn , thesecond equation (4.33) becomes


N1 N2
SW +
(4.37)SB w = N (m1 − m2 )Nwhere SW is deﬁned by (4.28), SB is deﬁned by (4.27), and we have substituted forthe bias using (4.34). Using (4.27), we note that SB w is always in the direction of(m2 − m1 ). Thus we can write1
w ∝ S−W (m2 − m1 )(4.38)

where we have ignored irrelevant scale factors. Thus the weight vector coincides
with that found from the Fisher criterion. In addition, we have also found an expression for the bias value w0 given by (4.34). This tells us that a new vector x should beclassiﬁed as belonging to class C1 if y(x) = wT (x − m) > 0 and class C2 otherwise.

4.1. Discriminant Functions

191

4.1.6 Fisher’s discriminant for multiple classes
We now consider the generalization of the Fisher discriminant to K > 2 classes,
and we shall assume that the dimensionality D of the input space is greater than thenumber K of classes. Next, we introduce D > 1 linear ‘features’ yk = wkT x, wherek = 1, . . . , D . These feature values can conveniently be grouped together to forma vector y. Similarly, the weight vectors {wk } can be considered to be the columnsof a matrix W, so that
(4.39)
y = WT x.
Note that again we are not including any bias parameters in the deﬁnition of y. The
generalization of the within-class covariance matrix to the case of K classes follows
from (4.28) to giveKSk
(4.40)
SW =
k=1

where
Sk



=

(xn − mk )(xn − mk )T(4.41)

n∈Ck

mk

1 
xn
Nk

=

(4.42)n∈Ck

and Nk is the number of patterns in class Ck . In order to ﬁnd a generalization of the
between-class covariance matrix, we follow Duda and Hart (1973) and consider ﬁrst
the total covariance matrix
ST =

N


(xn − m)(xn − m)T(4.43)

n=1

where m is the mean of the total data set
1 
1 
xn =
Nk mk
N
N
N

m=

n=1



K

(4.44)

k=1

and N = k Nk is the total number of data points. The total covariance matrix canbe decomposed into the sum of the within-class covariance matrix, given by (4.40)
and (4.41), plus an additional matrix SB , which we identify as a measure of the
between-class covariance(4.45)
ST = SW + S B
where
SB =

K

k=1

Nk (mk − m)(mk − m)T .

(4.46)

192

4. LINEAR MODELS FOR CLASSIFICATION
These covariance matrices have been deﬁned in the original x-space. We can now
deﬁne similar matrices in the projected D -dimensional y-space
sW =

K 


(yn − µk )(yn − µk )T(4.47)

k=1 n∈Ck

and
sB =

K


Nk (µk − µ)(µk − µ)T(4.48)

k=1

where

1 
µk =
yn ,
Nk
n∈Ck

1 
µ=
Nk µk .
N
K

(4.49)

k=1

Again we wish to construct a scalar that is large when the between-class covarianceis large and when the within-class covariance is small. There are now many possible
choices of criterion (Fukunaga, 1990). One example is given by 1 (4.50)
J(W) = Tr s−
W sB .
This criterion can then be rewritten as an explicit function of the projection matrix
W in the form

(4.51)
J(w) = Tr (WSW WT )−1 (WSB WT ) .
Maximization of such criteria is straightforward, though somewhat involved, and isdiscussed at length in Fukunaga (1990). The weight values are determined by those1eigenvectors of S−
W SB that correspond to the D largest eigenvalues.
There is one important result that is common to all such criteria, which is worthemphasizing. We ﬁrst note from (4.46) that SB is composed of the sum of K matrices, each of which is an outer product of two vectors and therefore of rank 1. In
addition, only (K − 1) of these matrices are independent as a result of the constraint(4.44). Thus, SB has rank at most equal to (K − 1) and so there are at most (K − 1)
nonzero eigenvalues. This shows that the projection onto the (K − 1)-dimensional
subspace spanned by the eigenvectors of SB does not alter the value of J(w), andso we are therefore unable to ﬁnd more than (K − 1) linear ‘features’ by this means(Fukunaga, 1990).

4.1.7 The perceptron algorithm
Another example of a linear discriminant model is the perceptron of Rosenblatt(1962), which occupies an important place in the history of pattern recognition algorithms. It corresponds to a two-class model in which the input vector x is ﬁrst
transformed using a ﬁxed nonlinear transformation to give a feature vector φ(x),and this is then used to construct a generalized linear model of the formy(x) = f wT φ(x)(4.52)

4.1. Discriminant Functions

193

where the nonlinear activation function f (·) is given by a step function of the formf (a) =

+1, a  0
−1, a < 0.

(4.53)The vector φ(x) will typically include a bias component φ0 (x) = 1. In earlier
discussions of two-class classiﬁcation problems, we have focussed on a target coding
scheme in which t ∈ {0, 1}, which is appropriate in the context of probabilistic
models. For the perceptron, however, it is more convenient to use target values
t = +1 for class C1 and t = −1 for class C2 , which matches the choice of activation
function.
The algorithm used to determine the parameters w of the perceptron can mosteasily be motivated by error function minimization. A natural choice of error function would be the total number of misclassiﬁed patterns. However, this does not leadto a simple learning algorithm because the error is a piecewise constant functionof w, with discontinuities wherever a change in w causes the decision boundary tomove across one of the data points. Methods based on changing w using the gradient of the error function cannot then be applied, because the gradient is zero almost
everywhere.
We therefore consider an alternative error function known as the perceptron criterion. To derive this, we note that we are seeking a weight vector w such thatpatterns xn in class C1 will have wT φ(xn ) > 0, whereas patterns xn in class C2
have wT φ(xn ) < 0. Using the t ∈ {−1, +1} target coding scheme it follows that
we would like all patterns to satisfy wT φ(xn )tn > 0. The perceptron criterion
associates zero error with any pattern that is correctly classiﬁed, whereas for a misclassiﬁed pattern xn it tries to minimize the quantity −wT φ(xn )tn . The perceptron
criterion is therefore given bywT φn tn
(4.54)EP (w) = −n∈MFrank Rosenblatt
1928–1969
Rosenblatt’s perceptron played an
important role in the history of machine learning. Initially, Rosenblatt
simulated the perceptron on an IBM
704 computer at Cornell in 1957,
but by the early 1960s he had built
special-purpose hardware that provided a direct, parallel implementation of perceptron learning. Many of
his ideas were encapsulated in “Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms” published in 1962. Rosenblatt’s work was
criticized by Marvin Minksy, whose objections were
published in the book “Perceptrons”, co-authored with

Seymour Papert. This book was widely misinterpreted at the time as showing that neural networks
were fatally ﬂawed and could only learn solutions forlinearly separable problems. In fact, it only proved
such limitations in the case of single-layer networks
such as the perceptron and merely conjectured (incorrectly) that they applied to more general networkmodels. Unfortunately, however, this book contributed
to the substantial decline in research funding for neural computing, a situation that was not reversed until the mid-1980s. Today, there are many hundreds,
if not thousands, of applications of neural networks
in widespread use, with examples in areas such as
handwriting recognition and information retrieval being used routinely by millions of people.

194

4. LINEAR MODELS FOR CLASSIFICATION

Section 3.1.3

where M denotes the set of all misclassiﬁed patterns. The contribution to the error
associated with a particular misclassiﬁed pattern is a linear function of w in regionsof w space where the pattern is misclassiﬁed and zero in regions where it is correctlyclassiﬁed. The total error function is therefore piecewise linear.
We now apply the stochastic gradient descent algorithm to this error function.
The change in the weight vector w is then given byw(τ +1) = w(τ ) − η∇EP (w) = w(τ ) + ηφn tn(4.55)

where η is the learning rate parameter and τ is an integer that indexes the steps ofthe algorithm. Because the perceptron function y(x, w) is unchanged if we multiply
w by a constant, we can set the learning rate parameter η equal to 1 without ofgenerality. Note that, as the weight vector evolves during training, the set of patterns
that are misclassiﬁed will change.
The perceptron learning algorithm has a simple interpretation, as follows. We
cycle through the training patterns in turn, and for each pattern xn we evaluate theperceptron function (4.52). If the pattern is correctly classiﬁed, then the weightvector remains unchanged, whereas if it is incorrectly classiﬁed, then for class C1
we add the vector φ(xn ) onto the current estimate of weight vector w while forclass C2 we subtract the vector φ(xn ) from w. The perceptron learning algorithm isillustrated in Figure 4.7.
If we consider the effect of a single update in the perceptron learning algorithm,
we see that the contribution to the error from a misclassiﬁed pattern will be reduced
because from (4.55) we have−w(τ +1)T φn tn = −w(τ )T φn tn − (φn tn )T φn tn < −w(τ )T φn tn(4.56)

where we have set η = 1, and made use of φn tn 2 > 0. Of course, this does
not imply that the contribution to the error function from the other misclassiﬁed
patterns will have been reduced. Furthermore, the change in weight vector may havecaused some previously correctly classiﬁed patterns to become misclassiﬁed. Thus
the perceptron learning rule is not guaranteed to reduce the total error function ateach stage.
However, the perceptron convergence theorem states that if there exists an exact solution (in other words, if the training data set is linearly separable), then theperceptron learning algorithm is guaranteed to ﬁnd an exact solution in a ﬁnite number of steps. Proofs of this theorem can be found for example in Rosenblatt (1962),
Block (1962), Nilsson (1965), Minsky and Papert (1969), Hertz et al. (1991), andBishop (1995a). Note, however, that the number of steps required to achieve convergence could still be substantial, and in practice, until convergence is achieved,
we will not be able to distinguish between a nonseparable problem and one that is
simply slow to converge.
Even when the data set is linearly separable, there may be many solutions, andwhich one is found will depend on the initialization of the parameters and on the order of presentation of the data points. Furthermore, for data sets that are not linearly
separable, the perceptron learning algorithm will never converge.

4.1. Discriminant Functions

1

1

0.5

0.5

0

0

−0.5

−0.5

−1
−1

−0.5

0

0.5

1

−1
−1

1

1

0.5

0.5

0

0

−0.5

−0.5

−1
−1

−0.5

0

0.5

1

−1
−1

195

−0.5

0

0.5

1

−0.5

0

0.5

1

Figure 4.7 Illustration of the convergence of the perceptron learning algorithm, showing data points from twoclasses (red and blue) in a two-dimensional feature space (φ1 , φ2 ). The top left plot shows the initial parameter
vector w shown as a black arrow together with the corresponding decision boundary (black line), in which thearrow points towards the decision region which classiﬁed as belonging to the red class. The data point circled
in green is misclassiﬁed and so its feature vector is added to the current weight vector, giving the new decisionboundary shown in the top right plot. The bottom left plot shows the next misclassiﬁed point to be considered,indicated by the green circle, and its feature vector is again added to the weight vector giving the decisionboundary shown in the bottom right plot for which all data points are correctly classiﬁed.

196

4. LINEAR MODELS FOR CLASSIFICATION

Figure 4.8 Illustration of the Mark 1 perceptron hardware. The photograph on the left shows how the inputs
were obtained using a simple camera system in which an input scene, in this case a printed character, wasilluminated by powerful lights, and an image focussed onto a 20 × 20 array of cadmium sulphide photocells,
giving a primitive 400 pixel image. The perceptron also had a patch board, shown in the middle photograph,
which allowed different conﬁgurations of input features to be tried. Often these were wired up at random todemonstrate the ability of the perceptron to learn without the need for precise wiring, in contrast to a modern
digital computer. The photograph on the right shows one of the racks of adaptive weights. Each weight was
implemented using a rotary variable resistor, also called a potentiometer, driven by an electric motor thereby
allowing the value of the weight to be adjusted automatically by the learning algorithm.

Aside from difﬁculties with the learning algorithm, the perceptron does not provide probabilistic outputs, nor does it generalize readily to K > 2 classes. The mostimportant limitation, however, arises from the fact that (in common with all of themodels discussed in this chapter and the previous one) it is based on linear combinations of ﬁxed basis functions. More detailed discussions of the limitations ofperceptrons can be found in Minsky and Papert (1969) and Bishop (1995a).
Analogue hardware implementations of the perceptron were built by Rosenblatt,based on motor-driven variable resistors to implement the adaptive parameters wj .
These are illustrated in Figure 4.8. The inputs were obtained from a simple camera
system based on an array of photo-sensors, while the basis functions φ could bechosen in a variety of ways, for example based on simple ﬁxed functions of randomly
chosen subsets of pixels from the input image. Typical applications involved learning
to discriminate simple shapes or characters.
At the same time that the perceptron was being developed, a closely related
system called the adaline, which is short for ‘adaptive linear element’, was being
explored by Widrow and co-workers. The functional form of the model was the same
as for the perceptron, but a different approach to training was adopted (Widrow and
Hoff, 1960; Widrow and Lehr, 1990).

4.2. Probabilistic Generative Models
We turn next to a probabilistic view of classiﬁcation and show how models withlinear decision boundaries arise from simple assumptions about the distribution of
the data. In Section 1.5.4, we discussed the distinction between the discriminative
and the generative approaches to classiﬁcation. Here we shall adopt a generative197

4.2. Probabilistic Generative Models
Figure 4.9

Plot of the logistic sigmoid function
1
σ(a) deﬁned by (4.59), shown inred, together with the scaled probit function Φ(λa), for λ2 = π/8,
shown in dashed blue, where Φ(a)is deﬁned by (4.114). The scaling factor π/8 is chosen so that thederivatives of the two curves are 0.5
equal for a = 0.

0

−5

0

5

approach in which we model the class-conditional densities p(x|Ck ), as well as theclass priors p(Ck ), and then use these to compute posterior probabilities p(Ck |x)
through Bayes’ theorem.
Consider ﬁrst of all the case of two classes. The posterior probability for class
C1 can be written asp(C1 |x) ==p(x|C1 )p(C1 )p(x|C1 )p(C1 ) + p(x|C2 )p(C2 )1= σ(a)
1 + exp(−a)

where we have deﬁned
a = lnp(x|C1 )p(C1 )p(x|C2 )p(C2 )(4.57)(4.58)and σ(a) is the logistic sigmoid function deﬁned by
σ(a) =

1
1 + exp(−a)(4.59)

which is plotted in Figure 4.9. The term ‘sigmoid’ means S-shaped. This type offunction is sometimes also called a ‘squashing function’ because it maps the wholereal axis into a ﬁnite interval. The logistic sigmoid has been encountered alreadyin earlier chapters and plays an important role in many classiﬁcation algorithms. It
satisﬁes the following symmetry propertyσ(−a) = 1 − σ(a)as is easily veriﬁed. The inverse of the logistic sigmoid is given by σ 
a = ln1−σ(4.60)(4.61)and is known as the logit function. It represents the log of the ratio of probabilitiesln [p(C1 |x)/p(C2 |x)] for the two classes, also known as the log odds.

198

4. LINEAR MODELS FOR CLASSIFICATION
Note that in (4.57) we have simply rewritten the posterior probabilities in an
equivalent form, and so the appearance of the logistic sigmoid may seem rather vacuous. However, it will have signiﬁcance provided a(x) takes a simple functional
form. We shall shortly consider situations in which a(x) is a linear function of x, in
which case the posterior probability is governed by a generalized linear model.
For the case of K > 2 classes, we havep(Ck |x) ==p(x|Ck )p(Ck )j p(x|Cj )p(Cj )exp(ak )j exp(aj )(4.62)

which is known as the normalized exponential and can be regarded as a multiclassgeneralization of the logistic sigmoid. Here the quantities ak are deﬁned by
ak = ln p(x|Ck )p(Ck ).

(4.63)The normalized exponential is also known as the softmax function, as it represents
aj for all j = k, thena smoothed version of the ‘max’ function because, if ak
p(Ck |x)  1, and p(Cj |x)  0.
We now investigate the consequences of choosing speciﬁc forms for the classconditional densities, looking ﬁrst at continuous input variables x and then discussing brieﬂy the case of discrete inputs.

4.2.1 Continuous inputs
Let us assume that the class-conditional densities are Gaussian and then explore
the resulting form for the posterior probabilities. To start with, we shall assume that
all classes share the same covariance matrix. Thus the density for class Ck is givenbyp(x|Ck ) =1
1
1
exp − (x − µk )T Σ−1 (x − µk ) .
D/
2
1
/
2
2
(2π)
|Σ|

(4.64)

Consider ﬁrst the case of two classes. From (4.57) and (4.58), we havep(C1 |x) = σ(wT x + w0 )(4.65)

w = Σ−1 (µ1 − µ2 )1
1
p(C1 )−1−1µ1 + µTµ2 + lnw0 = − µT
.
1Σ
2Σ
2
2
p(C2 )(4.66)

where we have deﬁned(4.67)We see that the quadratic terms in x from the exponents of the Gaussian densities
have cancelled (due to the assumption of common covariance matrices) leading toa linear function of x in the argument of the logistic sigmoid. This result is illustrated for the case of a two-dimensional input space x in Figure 4.10. The resulting

4.2. Probabilistic Generative Models

199

Figure 4.10 The left-hand plot shows the class-conditional densities for two classes, denoted red and blue.
On the right is the corresponding posterior probability p(C1 |x), which is given by a logistic sigmoid of a linear
function of x. The surface in the right-hand plot is coloured using a proportion of red ink given by p(C1 |x) and a
proportion of blue ink given by p(C2 |x) = 1 − p(C1 |x).

decision boundaries correspond to surfaces along which the posterior probabilitiesp(Ck |x) are constant and so will be given by linear functions of x, and therefore
the decision boundaries are linear in input space. The prior probabilities p(Ck ) enter
only through the bias parameter w0 so that changes in the priors have the effect ofmaking parallel shifts of the decision boundary and more generally of the parallel
contours of constant posterior probability.
For the general case of K classes we have, from (4.62) and (4.63),
ak (x) = wkT x + wk0(4.68)= Σ−1 µk1
= − µTΣ−1 µk + ln p(Ck ).
2 k(4.69)

where we have deﬁned
wk
wk0(4.70)We see that the ak (x) are again linear functions of x as a consequence of the cancellation of the quadratic terms due to the shared covariances. The resulting decisionboundaries, corresponding to the minimum misclassiﬁcation rate, will occur when
two of the posterior probabilities (the two largest) are equal, and so will be deﬁned
by linear functions of x, and so again we have a generalized linear model.
If we relax the assumption of a shared covariance matrix and allow each classconditional density p(x|Ck ) to have its own covariance matrix Σk , then the earlier
cancellations will no longer occur, and we will obtain quadratic functions of x, giving rise to a quadratic discriminant. The linear and quadratic decision boundaries
are illustrated in Figure 4.11.

200

4. LINEAR MODELS FOR CLASSIFICATION
2.5
2
1.5
1
0.5
0

−0.5
−1
−1.5
−2
−2.5

−2

−1

0

1

2

Figure 4.11 The left-hand plot shows the class-conditional densities for three classes each having a Gaussian
distribution, coloured red, green, and blue, in which the red and green classes have the same covariance matrix.
The right-hand plot shows the corresponding posterior probabilities, in which the RGB colour vector represents
the posterior probabilities for the respective three classes. The decision boundaries are also shown. Notice thatthe boundary between the red and green classes, which have the same covariance matrix, is linear, whereas
those between the other pairs of classes are quadratic.

4.2.2 Maximum likelihood solution
Once we have speciﬁed a parametric functional form for the class-conditional
densities p(x|Ck ), we can then determine the values of the parameters, together withthe prior class probabilities p(Ck ), using maximum likelihood. This requires a data
set comprising observations of x along with their corresponding class labels.
Consider ﬁrst the case of two classes, each having a Gaussian class-conditional
density with a shared covariance matrix, and suppose we have a data set {xn , tn }where n = 1, . . . , N . Here tn = 1 denotes class C1 and tn = 0 denotes class C2 . We
denote the prior class probability p(C1 ) = π, so that p(C2 ) = 1 − π. For a data point
xn from class C1 , we have tn = 1 and hencep(xn , C1 ) = p(C1 )p(xn |C1 ) = πN (xn |µ1 , Σ).
Similarly for class C2 , we have tn = 0 and hencep(xn , C2 ) = p(C2 )p(xn |C2 ) = (1 − π)N (xn |µ2 , Σ).
Thus the likelihood function is given byp(t|π, µ1 , µ2 , Σ) =

N


t

1−tn

[πN (xn |µ1 , Σ)] n [(1 − π)N (xn |µ2 , Σ)](4.71)

n=1

where t = (t1 , . . . , tN )T . As usual, it is convenient to maximize the log of thelikelihood function. Consider ﬁrst the maximization with respect to π. The terms in

201

4.2. Probabilistic Generative Models
the log likelihood function that depend on π areN{tn ln π + (1 − tn ) ln(1 − π)} .

(4.72)

n=1

Setting the derivative with respect to π equal to zero and rearranging, we obtain
N1
1 
N1
=
π=
tn =
N
N
N1 + N2N(4.73)

n=1

Exercise 4.9

where N1 denotes the total number of data points in class C1 , and N2 denotes the total
number of data points in class C2 . Thus the maximum likelihood estimate for π is
simply the fraction of points in class C1 as expected. This result is easily generalized
to the multiclass case where again the maximum likelihood estimate of the priorprobability associated with class Ck is given by the fraction of the training set points
assigned to that class.
Now consider the maximization with respect to µ1 . Again we can pick out of
the log likelihood function those terms that depend on µ1 giving
N

n=1

1
tn ln N (xn |µ1 , Σ) = −tn (xn − µ1 )T Σ−1 (xn − µ1 ) + const. (4.74)
2
N

n=1

Setting the derivative with respect to µ1 to zero and rearranging, we obtain
1 
µ1 =
tn xn
N1
N

(4.75)

n=1

which is simply the mean of all the input vectors xn assigned to class C1 . By asimilar argument, the corresponding result for µ2 is given by1 (1 − tn )xn
N2
N

µ2 =

(4.76)

n=1

which again is the mean of all the input vectors xn assigned to class C2 .
Finally, consider the maximum likelihood solution for the shared covariancematrix Σ. Picking out the terms in the log likelihood function that depend on Σ, we
have
1
1
tn ln |Σ| −tn (xn − µ1 )T Σ−1 (xn − µ1 )
−
2
2
N

N

n=1

n=1

N
N
1
1
(1 − tn ) ln |Σ| −(1 − tn )(xn − µ2 )T Σ−1 (xn − µ2 )
−
2
2
n=1
n=1
N
N  −1 = − ln |Σ| − Tr Σ S(4.77)
22

202

4. LINEAR MODELS FOR CLASSIFICATION
where we have deﬁned
S =
S1 =
S2 =

N1
N2
S1 +
S2
N
N

1
(xn − µ1 )(xn − µ1 )T
N1
n∈C1
1 
(xn − µ2 )(xn − µ2 )T .
N2

(4.78)(4.79)(4.80)n∈C2

Exercise 4.10
Section 2.3.7

Using the standard result for the maximum likelihood solution for a Gaussian distribution, we see that Σ = S, which represents a weighted average of the covariancematrices associated with each of the two classes separately.
This result is easily extended to the K class problem to obtain the corresponding
maximum likelihood solutions for the parameters in which each class-conditional
density is Gaussian with a shared covariance matrix. Note that the approach of ﬁtting
Gaussian distributions to the classes is not robust to outliers, because the maximum
likelihood estimation of a Gaussian is not robust.

4.2.3 Discrete features

Section 8.2.2

Let us now consider the case of discrete feature values xi . For simplicity, we
begin by looking at binary feature values xi ∈ {0, 1} and discuss the extension tomore general discrete features shortly. If there are D inputs, then a general distribution would correspond to a table of 2D numbers for each class, containing 2D − 1
independent variables (due to the summation constraint). Because this grows exponentially with the number of features, we might seek a more restricted representation. Here we will make the naive Bayes assumption in which the feature values aretreated as independent, conditioned on the class Ck . Thus we have class-conditional
distributions of the formp(x|Ck ) =Dµxkii (1 − µki )1−xi(4.81)

i=1

which contain D independent parameters for each class. Substituting into (4.63) then
gives
D

ak (x) ={xi ln µki + (1 − xi ) ln(1 − µki )} + ln p(Ck )(4.82)
i=1

Exercise 4.11

which again are linear functions of the input values xi . For the case of K = 2 classes,
we can alternatively consider the logistic sigmoid formulation given by (4.57). Analogous results are obtained for discrete variables each of which can take M > 2
states.

4.2.4 Exponential family
As we have seen, for both Gaussian distributed and discrete inputs, the posterior
class probabilities are given by generalized linear models with logistic sigmoid (K =

4.3. Probabilistic Discriminative Models

203

2 classes) or softmax (K  2 classes) activation functions. These are particular casesof a more general result obtained by assuming that the class-conditional densities
p(x|Ck ) are members of the exponential family of distributions.
Using the form (2.194) for members of the exponential family, we see that the
distribution of x can be written in the formp(x|λk ) = h(x)g(λk ) exp λT(4.83)k u(x) .
We now restrict attention to the subclass of such distributions for which u(x) = x.
Then we make use of (2.236) to introduce a scaling parameter s, so that we obtain
the restricted set of exponential family class-conditional densities of the form
 
1
1
1 T
(4.84)x g(λk ) exp
λ x .
p(x|λk , s) = hsss kNote that we are allowing each class to have its own parameter vector λk but we areassuming that the classes share the same scale parameter s.
For the two-class problem, we substitute this expression for the class-conditional
densities into (4.58) and we see that the posterior class probability is again given bya logistic sigmoid acting on a linear function a(x) which is given bya(x) = (λ1 − λ2 )T x + ln g(λ1 ) − ln g(λ2 ) + ln p(C1 ) − ln p(C2 ).

(4.85)

Similarly, for the K-class problem, we substitute the class-conditional density expression into (4.63) to giveak (x) = λT
k x + ln g(λk ) + ln p(Ck )(4.86)and so again is a linear function of x.

4.3. Probabilistic Discriminative Models
For the two-class classiﬁcation problem, we have seen that the posterior probabilityof class C1 can be written as a logistic sigmoid acting on a linear function of x, for awide choice of class-conditional distributions p(x|Ck ). Similarly, for the multiclasscase, the posterior probability of class Ck is given by a softmax transformation of alinear function of x. For speciﬁc choices of the class-conditional densities p(x|Ck ),we have used maximum likelihood to determine the parameters of the densities aswell as the class priors p(Ck ) and then used Bayes’ theorem to ﬁnd the posterior classprobabilities.
However, an alternative approach is to use the functional form of the generalized
linear model explicitly and to determine its parameters directly by using maximum
likelihood. We shall see that there is an efﬁcient algorithm ﬁnding such solutions
known as iterative reweighted least squares, or IRLS.
The indirect approach to ﬁnding the parameters of a generalized linear model,
by ﬁtting class-conditional densities and class priors separately and then applying

204

4. LINEAR MODELS FOR CLASSIFICATION

1
1
φ2

x2

0

0.5

−1
0
−1

0

x1

1

0

0.5

φ1

1

Figure 4.12 Illustration of the role of nonlinear basis functions in linear classiﬁcation models. The left plotshows the original input space (x1 , x2 ) together with data points from two classes labelled red and blue. Two‘Gaussian’ basis functions φ1 (x) and φ2 (x) are deﬁned in this space with centres shown by the green crosses
and with contours shown by the green circles. The right-hand plot shows the corresponding feature space(φ1 , φ2 ) together with the linear decision boundary obtained given by a logistic regression model of the form
discussed in Section 4.3.2. This corresponds to a nonlinear decision boundary in the original input space,shown by the black curve in the left-hand plot.

Bayes’ theorem, represents an example of generative modelling, because we could
take such a model and generate synthetic data by drawing values of x from themarginal distribution p(x). In the direct approach, we are maximizing a likelihood
function deﬁned through the conditional distribution p(Ck |x), which represents aform of discriminative training. One advantage of the discriminative approach isthat there will typically be fewer adaptive parameters to be determined, as we shallsee shortly. It may also lead to improved predictive performance, particularly when
the class-conditional density assumptions give a poor approximation to the true distributions.

4.3.1 Fixed basis functions
So far in this chapter, we have considered classiﬁcation models that work directly with the original input vector x. However, all of the algorithms are equallyapplicable if we ﬁrst make a ﬁxed nonlinear transformation of the inputs using avector of basis functions φ(x). The resulting decision boundaries will be linear in
the feature space φ, and these correspond to nonlinear decision boundaries in theoriginal x space, as illustrated in Figure 4.12. Classes that are linearly separable
in the feature space φ(x) need not be linearly separable in the original observation
space x. Note that as in our discussion of linear models for regression, one of the4.3. Probabilistic Discriminative Models

Section 3.6

205

basis functions is typically set to a constant, say φ0 (x) = 1, so that the corresponding parameter w0 plays the role of a bias. For the remainder of this chapter, we shall
include a ﬁxed basis function transformation φ(x), as this will highlight some usefulsimilarities to the regression models discussed in Chapter 3.
For many problems of practical interest, there is signiﬁcant overlap between
the class-conditional densities p(x|Ck ). This corresponds to posterior probabilitiesp(Ck |x), which, for at least some values of x, are not 0 or 1. In such cases, the optimal solution is obtained by modelling the posterior probabilities accurately and thenapplying standard decision theory, as discussed in Chapter 1. Note that nonlinear
transformations φ(x) cannot remove such class overlap. Indeed, they can increase
the level of overlap, or create overlap where none existed in the original observation
space. However, suitable choices of nonlinearity can make the process of modelling
the posterior probabilities easier.
Such ﬁxed basis function models have important limitations, and these will be
resolved in later chapters by allowing the basis functions themselves to adapt to thedata. Notwithstanding these limitations, models with ﬁxed nonlinear basis functions
play an important role in applications, and a discussion of such models will introduce many of the key concepts needed for an understanding of their more complex
counterparts.

4.3.2 Logistic regression
We begin our treatment of generalized linear models by considering the problemof two-class classiﬁcation. In our discussion of generative approaches in Section 4.2,
we saw that under rather general assumptions, the posterior probability of class C1
can be written as a logistic sigmoid acting on a linear function of the feature vector
φ so that(4.87)p(C1 |φ) = y(φ) = σ wT φ

Exercise 4.12

with p(C2 |φ) = 1 − p(C1 |φ). Here σ(·) is the logistic sigmoid function deﬁned by(4.59). In the terminology of statistics, this model is known as logistic regression,
although it should be emphasized that this is a model for classiﬁcation rather thanregression.
For an M -dimensional feature space φ, this model has M adjustable parameters.
By contrast, if we had ﬁtted Gaussian class conditional densities using maximum
likelihood, we would have used 2M parameters for the means and M (M + 1)/2
parameters for the (shared) covariance matrix. Together with the class prior p(C1 ),this gives a total of M (M +5)/2+1 parameters, which grows quadratically with M ,in contrast to the linear dependence on M of the number of parameters in logistic
regression. For large values of M , there is a clear advantage in working with thelogistic regression model directly.
We now use maximum likelihood to determine the parameters of the logistic
regression model. To do this, we shall make use of the derivative of the logistic sigmoid function, which can conveniently be expressed in terms of the sigmoid function
itselfdσ= σ(1 − σ).
(4.88)da206

4. LINEAR MODELS FOR CLASSIFICATION
For a data set {φn , tn }, where tn ∈ {0, 1} and φn = φ(xn ), with n =1, . . . , N , the likelihood function can be writtenp(t|w) =

N


1−tn

yntn {1 − yn }(4.89)

n=1

where t = (t1 , . . . , tN )T and yn = p(C1 |φn ). As usual, we can deﬁne an error
function by taking the negative logarithm of the likelihood, which gives the crossentropy error function in the form
E(w) = − ln p(t|w) = −N{tn ln yn + (1 − tn ) ln(1 − yn )}(4.90)

n=1

Exercise 4.13

where yn = σ(an ) and an = wT φn . Taking the gradient of the error function withrespect to w, we obtain∇E(w) =

N
(yn − tn )φn(4.91)

n=1

Section 3.1.1

Exercise 4.14

where we have made use of (4.88). We see that the factor involving the derivativeof the logistic sigmoid has cancelled, leading to a simpliﬁed form for the gradientof the log likelihood. In particular, the contribution to the gradient from data pointn is given by the ‘error’ yn − tn between the target value and the prediction of themodel, times the basis function vector φn . Furthermore, comparison with (3.13)
shows that this takes precisely the same form as the gradient of the sum-of-squares
error function for the linear regression model.
If desired, we could make use of the result (4.91) to give a sequential algorithm
in which patterns are presented one at a time, in which each of the weight vectors is
updated using (3.22) in which ∇En is the nth term in (4.91).
It is worth noting that maximum likelihood can exhibit severe over-ﬁtting fordata sets that are linearly separable. This arises because the maximum likelihood solution occurs when the hyperplane corresponding to σ = 0.5, equivalent to wT φ =0, separates the two classes and the magnitude of w goes to inﬁnity. In this case, thelogistic sigmoid function becomes inﬁnitely steep in feature space, corresponding toa Heaviside step function, so that every training point from each class k is assigned
a posterior probability p(Ck |x) = 1. Furthermore, there is typically a continuumof such solutions because any separating hyperplane will give rise to the same posterior probabilities at the training data points, as will be seen later in Figure 10.13.
Maximum likelihood provides no way to favour one such solution over another, andwhich solution is found in practice will depend on the choice of optimization algorithm and on the parameter initialization. Note that the problem will arise even if
the number of data points is large compared with the number of parameters in themodel, so long as the training data set is linearly separable. The singularity can be
avoided by inclusion of a prior and ﬁnding a MAP solution for w, or equivalently byadding a regularization term to the error function.

4.3. Probabilistic Discriminative Models

207

4.3.3 Iterative reweighted least squares
In the case of the linear regression models discussed in Chapter 3, the maximum likelihood solution, on the assumption of a Gaussian noise model, leads to aclosed-form solution. This was a consequence of the quadratic dependence of thelog likelihood function on the parameter vector w. For logistic regression, there
is no longer a closed-form solution, due to the nonlinearity of the logistic sigmoid
function. However, the departure from a quadratic form is not substantial. To be
precise, the error function is concave, as we shall see shortly, and hence has a uniqueminimum. Furthermore, the error function can be minimized by an efﬁcient iterativetechnique based on the Newton-Raphson iterative optimization scheme, which uses a
local quadratic approximation to the log likelihood function. The Newton-Raphson
update, for minimizing a function E(w), takes the form (Fletcher, 1987; Bishop andNabney, 2008)
w(new) = w(old) − H−1 ∇E(w).
(4.92)
where H is the Hessian matrix whose elements comprise the second derivatives ofE(w) with respect to the components of w.
Let us ﬁrst of all apply the Newton-Raphson method to the linear regression
model (3.3) with the sum-of-squares error function (3.12). The gradient and Hessianof this error function are given by∇E(w) =

N
(wT φn − tn )φn = ΦT Φw − ΦT t

(4.93)

T
φn φTn =Φ Φ

(4.94)n=1

H = ∇∇E(w) =

N

n=1

Section 3.1.1

where Φ is the N × M design matrix, whose nth row is given by φT
n . The NewtonRaphson update then takes the formw(new) = w(old) − (ΦT Φ)−1 ΦT Φw(old) − ΦT t
= (ΦT Φ)−1 ΦT t

(4.95)

which we recognize as the standard least-squares solution. Note that the error function in this case is quadratic and hence the Newton-Raphson formula gives the exact
solution in one step.
Now let us apply the Newton-Raphson update to the cross-entropy error function(4.90) for the logistic regression model. From (4.91) we see that the gradient andHessian of this error function are given by∇E(w) =

N
(yn − tn )φn = ΦT (y − t)

(4.96)n=1

H = ∇∇E(w) =

N

n=1

T
yn (1 − yn )φn φT
n = Φ RΦ

(4.97)

208

4. LINEAR MODELS FOR CLASSIFICATION
where we have made use of (4.88). Also, we have introduced the N × N diagonal
matrix R with elements
Rnn = yn (1 − yn ).
(4.98)

Exercise 4.15

We see that the Hessian is no longer constant but depends on w through the weighting matrix R, corresponding to the fact that the error function is no longer quadratic.
Using the property 0 < yn < 1, which follows from the form of the logistic sigmoid
function, we see that uT Hu > 0 for an arbitrary vector u, and so the Hessian matrix
H is positive deﬁnite. It follows that the error function is a concave function of w
and hence has a unique minimum.
The Newton-Raphson update formula for the logistic regression model then becomesw(new)= w(old) − (ΦT RΦ)−1 ΦT (y − t)
= (ΦT RΦ)−1 ΦT RΦw(old) − ΦT (y − t)
= (ΦT RΦ)−1 ΦT Rz(4.99)

where z is an N -dimensional vector with elementsz = Φw(old) − R−1 (y − t).

(4.100)We see that the update formula (4.99) takes the form of a set of normal equations for aweighted least-squares problem. Because the weighing matrix R is not constant but
depends on the parameter vector w, we must apply the normal equations iteratively,
each time using the new weight vector w to compute a revised weighing matrix
R. For this reason, the algorithm is known as iterative reweighted least squares, or
IRLS (Rubin, 1983). As in the weighted least-squares problem, the elements of the
diagonal weighting matrix R can be interpreted as variances because the mean andvariance of t in the logistic regression model are given byE[t] = σ(x) = yvar[t] = E[t2 ] − E[t]2 = σ(x) − σ(x)2 = y(1 − y)(4.101)(4.102)where we have used the property t2 = t for t ∈ {0, 1}. In fact, we can interpret IRLS
as the solution to a linearized problem in the space of the variable a = wT φ. The
quantity zn , which corresponds to the nth element of z, can then be given a simple
interpretation as an effective target value in this space obtained by making a local
linear approximation to the logistic sigmoid function around the current operating
point w(old)dan 
(old)an (w)  an (w)+(tn − yn )dy n w(old)(yn − tn )(old)−(4.103)= zn .
= φTnw
yn (1 − yn )

4.3. Probabilistic Discriminative Models

209

4.3.4 Multiclass logistic regression
Section 4.2

In our discussion of generative models for multiclass classiﬁcation, we haveseen that for a large class of distributions, the posterior probabilities are given by asoftmax transformation of linear functions of the feature variables, so thatexp(ak )p(Ck |φ) = yk (φ) = 
j exp(aj )(4.104)

where the ‘activations’ ak are given by
ak = wkT φ.

Exercise 4.17

(4.105)

There we used maximum likelihood to determine separately the class-conditional
densities and the class priors and then found the corresponding posterior probabilities
using Bayes’ theorem, thereby implicitly determining the parameters {wk }. Here we
consider the use of maximum likelihood to determine the parameters {wk } of thismodel directly. To do this, we will require the derivatives of yk with respect to all of
the activations aj . These are given by∂yk= yk (Ikj − yj )∂aj

(4.106)

where Ikj are the elements of the identity matrix.
Next we write down the likelihood function. This is most easily done using
the 1-of-K coding scheme in which the target vector tn for a feature vector φnbelonging to class Ck is a binary vector with all elements zero except for element k,which equals one. The likelihood function is then given byp(T|w1 , . . . , wK ) =

N 
K


p(Ck |φn )tnk =n=1 k=1N K
n=1 k=1

tnk
ynk(4.107)

where ynk = yk (φn ), and T is an N × K matrix of target variables with elements
tnk . Taking the negative logarithm then gives
E(w1 , . . . , wK ) = − ln p(T|w1 , . . . , wK ) = −

N 
K


tnk ln ynk(4.108)n=1 k=1

Exercise 4.18

which is known as the cross-entropy error function for the multiclass classiﬁcation
problem.
We now take the gradient of the error function with respect to one of the parameter vectors wj . Making use of the result (4.106) for the derivatives of the softmax
function, we obtain∇wj E(w1 , . . . , wK ) =

N

n=1

(ynj − tnj ) φn(4.109)210

4. LINEAR MODELS FOR CLASSIFICATION


where we have made use of k tnk = 1. Once again, we see the same form arising
for the gradient as was found for the sum-of-squares error function with the linear
model and the cross-entropy error for the logistic regression model, namely the product of the error (ynj − tnj ) times the basis function φn . Again, we could use thisto formulate a sequential algorithm in which patterns are presented one at a time, inwhich each of the weight vectors is updated using (3.22).
We have seen that the derivative of the log likelihood function for a linear regression model with respect to the parameter vector w for a data point n took the formof the ‘error’ yn − tn times the feature vector φn . Similarly, for the combinationof logistic sigmoid activation function and cross-entropy error function (4.90), andfor the softmax activation function with the multiclass cross-entropy error function(4.108), we again obtain this same simple form. This is an example of a more general
result, as we shall see in Section 4.3.6.
To ﬁnd a batch algorithm, we again appeal to the Newton-Raphson update toobtain the corresponding IRLS algorithm for the multiclass problem. This requires
evaluation of the Hessian matrix that comprises blocks of size M × M in whichblock j, k is given by∇wk ∇wj E(w1 , . . . , wK ) = −Nynk (Ikj − ynj )φn φTn.

(4.110)n=1

Exercise 4.20

As with the two-class problem, the Hessian matrix for the multiclass logistic regression model is positive deﬁnite and so the error function again has a unique minimum.
Practical details of IRLS for the multiclass case can be found in Bishop and Nabney(2008).

4.3.5 Probit regression
We have seen that, for a broad range of class-conditional distributions, describedby the exponential family, the resulting posterior class probabilities are given by alogistic (or softmax) transformation acting on a linear function of the feature variables. However, not all choices of class-conditional density give rise to such a simple
form for the posterior probabilities (for instance, if the class-conditional densities aremodelled using Gaussian mixtures). This suggests that it might be worth exploringother types of discriminative probabilistic model. For the purposes of this chapter,
however, we shall return to the two-class case, and again remain within the framework of generalized linear models so thatp(t = 1|a) = f (a)(4.111)

where a = wT φ, and f (·) is the activation function.
One way to motivate an alternative choice for the link function is to consider anoisy threshold model, as follows. For each input φn , we evaluate an = wT φn andthen we set the target value according to
tn = 1 if an  θ
tn = 0 otherwise.

(4.112)

211

4.3. Probabilistic Discriminative Models
Figure 4.13 Schematic example of a probability density p(θ)
shown by the blue curve, given in this example by a mixtureof two Gaussians, along with its cumulative distribution function
f (a), shown by the red curve. Note that the value of the blue
curve at any point, such as that indicated by the vertical green
line, corresponds to the slope of the red curve at the same point.
Conversely, the value of the red curve at this point corresponds
to the area under the blue curve indicated by the shaded green
region. In the stochastic threshold model, the class label takes
the value t = 1 if the value of a = wT φ exceeds a threshold, otherwise it takes the value t = 0. This is equivalent to an activation
function given by the cumulative distribution function f (a).

1
0.8
0.6
0.4
0.2
0

0

1

2

3

4

If the value of θ is drawn from a probability density p(θ), then the corresponding
activation function will be given by the cumulative distribution function
 ap(θ) dθ(4.113)f (a) =−∞

as illustrated in Figure 4.13.
As a speciﬁc example, suppose that the density p(θ) is given by a zero mean,
unit variance Gaussian. The corresponding cumulative distribution function is givenbyaΦ(a) =−∞

N (θ|0, 1) dθ

(4.114)

which is known as the probit function. It has a sigmoidal shape and is comparedwith the logistic sigmoid function in Figure 4.9. Note that the use of a more general Gaussian distribution does not change the model because this is equivalent toa re-scaling of the linear coefﬁcients w. Many numerical packages provide for theevaluation of a closely related function deﬁned by a2
exp(−θ2 /2) dθ
(4.115)erf(a) = √π 0
Exercise 4.21

and known as the erf function or error function (not to be confused with the error
function of a machine learning model). It is related to the probit function by
Φ(a) =

1
2

1
1 + √ erf(a) .
2(4.116)

The generalized linear model based on a probit activation function is known as probit
regression.
We can determine the parameters of this model using maximum likelihood, by astraightforward extension of the ideas discussed earlier. In practice, the results foundusing probit regression tend to be similar to those of logistic regression. We shall,

212

4. LINEAR MODELS FOR CLASSIFICATION
however, ﬁnd another use for the probit model when we discuss Bayesian treatmentsof logistic regression in Section 4.5.
One issue that can occur in practical applications is that of outliers, which canarise for instance through errors in measuring the input vector x or through mislabelling of the target value t. Because such points can lie a long way to the wrong sideof the ideal decision boundary, they can seriously distort the classiﬁer. Note that thelogistic and probit regression models behave differently in this respect because thetails of the logistic sigmoid decay asymptotically like exp(−x) for x → ∞, whereas
for the probit activation function they decay like exp(−x2 ), and so the probit model
can be signiﬁcantly more sensitive to outliers.
However, both the logistic and the probit models assume the data is correctly
labelled. The effect of mislabelling is easily incorporated into a probabilistic modelby introducing a probability  that the target value t has been ﬂipped to the wrong
value (Opper and Winther, 2000a), leading to a target value distribution for data point
x of the formp(t|x) = (1 − )σ(x) + (1 − σ(x))=  + (1 − 2)σ(x)(4.117)

where σ(x) is the activation function with input vector x. Here  may be set inadvance, or it may be treated as a hyperparameter whose value is inferred from thedata.

4.3.6 Canonical link functions
For the linear regression model with a Gaussian noise distribution, the error
function, corresponding to the negative log likelihood, is given by (3.12). If we takethe derivative with respect to the parameter vector w of the contribution to the error
function from a data point n, this takes the form of the ‘error’ yn − tn times thefeature vector φn , where yn = wT φn . Similarly, for the combination of the logistic
sigmoid activation function and the cross-entropy error function (4.90), and for thesoftmax activation function with the multiclass cross-entropy error function (4.108),
we again obtain this same simple form. We now show that this is a general resultof assuming a conditional distribution for the target variable from the exponential
family, along with a corresponding choice for the activation function known as the
canonical link function.
We again make use of the restricted form (4.84) of exponential family distributions. Note that here we are applying the assumption of exponential family distribution to the target variable t, in contrast to Section 4.2.4 where we applied it to the
input vector x. We therefore consider conditional distributions of the target variableof the form ηt 
1 t
g(η) exp.
(4.118)p(t|η, s) = hsssUsing the same line of argument as led to the derivation of the result (2.226), we seethat the conditional mean of t, which we denote by y, is given byy ≡ E[t|η] = −sdln g(η).
dη(4.119)

4.4. The Laplace Approximation

213

Thus y and η must related, and we denote this relation through η = ψ(y).
Following Nelder and Wedderburn (1972), we deﬁne a generalized linear model
to be one for which y is a nonlinear function of a linear combination of the input (orfeature) variables so thaty = f (wT φ)(4.120)where f (·) is known as the activation function in the machine learning literature, andf −1 (·) is known as the link function in statistics.
Now consider the log likelihood function for this model, which, as a function of
η, is given byln p(t|η, s) =N

ln p(tn |η, s) =

n=1

N 

ηn tn ln g(ηn ) ++ consts(4.121)n=1where we are assuming that all observations share a common scale parameter (whichcorresponds to the noise variance for a Gaussian distribution for instance) and so s
is independent of n. The derivative of the log likelihood with respect to the model
parameters w is then given by∇w ln p(t|η, s) =

N


d
tn
ln g(ηn ) +
dηn
s

n=1

=

N

1
n=1

s

dηn dyn∇an
dyn dan{tn − yn } ψ  (yn )f  (an )φn(4.122)where an = wT φn , and we have used yn = f (an ) together with the result (4.119)
for E[t|η]. We now see that there is a considerable simpliﬁcation if we choose aparticular form for the link function f −1 (y) given by
f −1 (y) = ψ(y)(4.123)

which gives f (ψ(y)) = y and hence f  (ψ)ψ  (y) = 1. Also, because a = f −1 (y),we have a = ψ and hence f  (a)ψ  (y) = 1. In this case, the gradient of the error
function reduces toN1
∇ ln E(w) ={yn − tn }φn .
(4.124)sn=1

For the Gaussian s = β−1, whereas for the logistic model s = 1.

4.4. The Laplace Approximation
In Section 4.5 we shall discuss the Bayesian treatment of logistic regression. As
we shall see, this is more complex than the Bayesian treatment of linear regression
models, discussed in Sections 3.3 and 3.5. In particular, we cannot integrate exactly

214

Chapter 10
Chapter 11

4. LINEAR MODELS FOR CLASSIFICATION
over the parameter vector w since the posterior distribution is no longer Gaussian.
It is therefore necessary to introduce some form of approximation. Later in thebook we shall consider a range of techniques based on analytical approximations
and numerical sampling.
Here we introduce a simple, but widely used, framework called the Laplace approximation, that aims to ﬁnd a Gaussian approximation to a probability density
deﬁned over a set of continuous variables. Consider ﬁrst the case of a single continuous variable z, and suppose the distribution p(z) is deﬁned byp(z) =

1f (z)Z(4.125)
where Z = f (z) dz is the normalization coefﬁcient. We shall suppose that thevalue of Z is unknown. In the Laplace method the goal is to ﬁnd a Gaussian approximation q(z) which is centred on a mode of the distribution p(z). The ﬁrst step is toﬁnd a mode of p(z), in other words a point z0 such that p (z0 ) = 0, or equivalentlydf (z) = 0.
(4.126)dz z=z0
A Gaussian distribution has the property that its logarithm is a quadratic functionof the variables. We therefore consider a Taylor expansion of ln f (z) centred on themode z0 so that1
ln f (z)  ln f (z0 ) − A(z − z0 )2(4.127)
2
where


d2
A = − 2 ln f (z)
.
(4.128)dz
z =z0Note that the ﬁrst-order term in the Taylor expansion does not appear since z0 is alocal maximum of the distribution. Taking the exponential we obtainAf (z)  f (z0 ) exp − (z − z0 )2 .
2

(4.129)We can then obtain a normalized distribution q(z) by making use of the standard
result for the normalization of a Gaussian, so thatq(z) =

A
2π

1/2

A
exp − (z − z0 )2 .
2

(4.130)

The Laplace approximation is illustrated in Figure 4.14. Note that the Gaussian
approximation will only be well deﬁned if its precision A > 0, in other words thestationary point z0 must be a local maximum, so that the second derivative of f (z)at the point z0 is negative.

215

4.4. The Laplace Approximation
0.8

40

0.6

30

0.4

20

0.2

10

0
−2

−1

0

1

2

3

4

0
−2

−1

0

1

2

3

4

Figure 4.14 Illustration of the Laplace approximation applied to the distribution p(z) ∝ exp(−z 2 /2)σ(20z + 4)
where σ(z) is the logistic sigmoid function deﬁned by σ(z) = (1 + e−z )−1 . The left plot shows the normalized
distribution p(z) in yellow, together with the Laplace approximation centred on the mode z0 of p(z) in red. The
right plot shows the negative logarithms of the corresponding curves.

We can extend the Laplace method to approximate a distribution p(z) = f (z)/Zdeﬁned over an M -dimensional space z. At a stationary point z0 the gradient ∇f (z)will vanish. Expanding around this stationary point we have1
ln f (z)  ln f (z0 ) − (z − z0 )T A(z − z0 )2

(4.131)

where the M × M Hessian matrix A is deﬁned by
A = − ∇∇ ln f (z)|z=z0(4.132)and ∇ is the gradient operator. Taking the exponential of both sides we obtain
1f (z)  f (z0 ) exp − (z − z0 )T A(z − z0 ) .
2(4.133)The distribution q(z) is proportional to f (z) and the appropriate normalization coefﬁcient can be found by inspection, using the standard result (2.43) for a normalized
multivariate Gaussian, giving
q(z) =|A|1/2
1
exp − (z − z0 )T A(z − z0 )2(2π)M/2= N (z|z0 , A−1 )(4.134)

where |A| denotes the determinant of A. This Gaussian distribution will be welldeﬁned provided its precision matrix, given by A, is positive deﬁnite, which implies
that the stationary point z0 must be a local maximum, not a minimum or a saddle
point.
In order to apply the Laplace approximation we ﬁrst need to ﬁnd the mode z0 ,and then evaluate the Hessian matrix at that mode. In practice a mode will typically be found by running some form of numerical optimization algorithm (Bishop

216

4. LINEAR MODELS FOR CLASSIFICATION
and Nabney, 2008). Many of the distributions encountered in practice will be multimodal and so there will be different Laplace approximations according to whichmode is being considered. Note that the normalization constant Z of the true distribution does not need to be known in order to apply the Laplace method. As a resultof the central limit theorem, the posterior distribution for a model is expected to
become increasingly better approximated by a Gaussian as the number of observed
data points is increased, and so we would expect the Laplace approximation to be
most useful in situations where the number of data points is relatively large.
One major weakness of the Laplace approximation is that, since it is based on a
Gaussian distribution, it is only directly applicable to real variables. In other cases
it may be possible to apply the Laplace approximation to a transformation of thevariable. For instance if 0  τ < ∞ then we can consider a Laplace approximationof ln τ . The most serious limitation of the Laplace framework, however, is that
it is based purely on the aspects of the true distribution at a speciﬁc value of thevariable, and so can fail to capture important global properties. In Chapter 10 weshall consider alternative approaches which adopt a more global perspective.

4.4.1 Model comparison and BIC
As well as approximating the distribution p(z) we can also obtain an approximation to the normalization constant Z. Using the approximation (4.133) we haveZ =f (z) dz1
 f (z0 ) exp − (z − z0 )T A(z − z0 ) dz
2
= f (z0 )(2π)M/2|A|1/2(4.135)where we have noted that the integrand is Gaussian and made use of the standard
result (2.43) for a normalized Gaussian distribution. We can use the result (4.135) toobtain an approximation to the model evidence which, as discussed in Section 3.4,
plays a central role in Bayesian model comparison.
Consider a data set D and a set of models {Mi } having parameters {θ i }. For
each model we deﬁne a likelihood function p(D|θ i , Mi ). If we introduce a priorp(θ i |Mi ) over the parameters, then we are interested in computing the model evidence p(D|Mi ) for the various models. From now on we omit the conditioning on
Mi to keep the notation uncluttered. From Bayes’ theorem the model evidence isgiven byp(D) = p(D|θ)p(θ) dθ.
(4.136)
Exercise 4.22

Identifying f (θ) = p(D|θ)p(θ) and Z = p(D), and applying the result (4.135), we
obtain1M
ln p(D)  ln p(D|θ MAP ) + ln p(θ MAP ) +ln(2π) − ln |A|(4.137)2
2()*+
Occam factor

4.5. Bayesian Logistic Regression

217

where θ MAP is the value of θ at the mode of the posterior distribution, and A is the
Hessian matrix of second derivatives of the negative log posterior
A = −∇∇ ln p(D|θ MAP )p(θ MAP ) = −∇∇ ln p(θ MAP |D).

Exercise 4.23

The ﬁrst term on the right hand side of (4.137) represents the log likelihood evaluated using the optimized parameters, while the remaining three terms comprise the‘Occam factor’ which penalizes model complexity.
If we assume that the Gaussian prior distribution over parameters is broad, andthat the Hessian has full rank, then we can approximate (4.137) very roughly using1
ln p(D)  ln p(D|θ MAP ) − M ln N
2Section 3.5.3(4.138)(4.139)where N is the number of data points, M is the number of parameters in θ andwe have omitted additive constants. This is known as the Bayesian Information
Criterion (BIC) or the Schwarz criterion (Schwarz, 1978). Note that, compared to
AIC given by (1.73), this penalizes model complexity more heavily.
Complexity measures such as AIC and BIC have the virtue of being easy toevaluate, but can also give misleading results. In particular, the assumption that the
Hessian matrix has full rank is often not valid since many of the parameters are not‘well-determined’. We can use the result (4.137) to obtain a more accurate estimateof the model evidence starting from the Laplace approximation, as we illustrate in
the context of neural networks in Section 5.7.

4.5. Bayesian Logistic Regression
We now turn to a Bayesian treatment of logistic regression. Exact Bayesian inference for logistic regression is intractable. In particular, evaluation of the posterior
distribution would require normalization of the product of a prior distribution and alikelihood function that itself comprises a product of logistic sigmoid functions, one
for every data point. Evaluation of the predictive distribution is similarly intractable.
Here we consider the application of the Laplace approximation to the problem ofBayesian logistic regression (Spiegelhalter and Lauritzen, 1990; MacKay, 1992b).

4.5.1 Laplace approximation
Recall from Section 4.4 that the Laplace approximation is obtained by ﬁnding
the mode of the posterior distribution and then ﬁtting a Gaussian centred at that
mode. This requires evaluation of the second derivatives of the log posterior, whichis equivalent to ﬁnding the Hessian matrix.
Because we seek a Gaussian representation for the posterior distribution, it isnatural to begin with a Gaussian prior, which we write in the general formp(w) = N (w|m0 , S0 )(4.140)218

4. LINEAR MODELS FOR CLASSIFICATION
where m0 and S0 are ﬁxed hyperparameters. The posterior distribution over w is
given byp(w|t) ∝ p(w)p(t|w)(4.141)
where t = (t1 , . . . , tN )T . Taking the log of both sides, and substituting for the priordistribution using (4.140), and for the likelihood function using (4.89), we obtain11ln p(w|t) = − (w − m0 )T S−
0 (w − m0 )2N+{tn ln yn + (1 − tn ) ln(1 − yn )} + const (4.142)
n=1

where yn = σ(wT φn ). To obtain a Gaussian approximation to the posterior distribution, we ﬁrst maximize the posterior distribution to give the MAP (maximum
posterior) solution wMAP , which deﬁnes the mean of the Gaussian. The covarianceis then given by the inverse of the matrix of second derivatives of the negative log
likelihood, which takes the form
1
SN = −∇∇ ln p(w|t) = S−
0 +

N


yn (1 − yn )φn φTn.

(4.143)n=1

The Gaussian approximation to the posterior distribution therefore takes the formq(w) = N (w|wMAP , SN ).

(4.144)

Having obtained a Gaussian approximation to the posterior distribution, there
remains the task of marginalizing with respect to this distribution in order to makepredictions.

4.5.2 Predictive distribution
The predictive distribution for class C1 , given a new feature vector φ(x), is
obtained by marginalizing with respect to the posterior distribution p(w|t), which isitself approximated by a Gaussian distribution q(w) so that
(4.145)p(C1 |φ, t) = p(C1 |φ, w)p(w|t) dw  σ(wT φ)q(w) dw
with the corresponding probability for class C2 given by p(C2 |φ, t) = 1 − p(C1 |φ, t).
To evaluate the predictive distribution, we ﬁrst note that the function σ(wT φ) depends on w only through its projection onto φ. Denoting a = wT φ, we haveTσ(w φ) = δ(a − wT φ)σ(a) da(4.146)
where δ(·) is the Dirac delta function. From this we obtain
σ(wT φ)q(w) dw = σ(a)p(a) da(4.147)

4.5. Bayesian Logistic Regression

219



where
p(a) =

δ(a − wT φ)q(w) dw.

(4.148)

We can evaluate p(a) by noting that the delta function imposes a linear constraint
on w and so forms a marginal distribution from the joint distribution q(w) by integrating out all directions orthogonal to φ. Because q(w) is Gaussian, we know from
Section 2.3.2 that the marginal distribution will also be Gaussian. We can evaluate
the mean and covariance of this distribution by taking moments, and interchanging
the order of integration over a and w, so thatT
φ
(4.149)µa = E[a] = p(a)a da = q(w)wT φ dw = wMAP
where we have used the result (4.144) for the variational posterior distribution q(w).
Similarly2
σa = var[a] = p(a) a2 − E[a]2 da2q(w) (wT φ)2 − (mT
dw = φT SN φ.
(4.150)=N φ)Note that the distribution of a takes the same form as the predictive distribution(3.58) for the linear regression model, with the noise variance set to zero. Thus ourvariational approximation to the predictive distribution becomes(4.151)p(C1 |t) = σ(a)p(a) da = σ(a)N (a|µa , σa2 ) da.

Exercise 4.24

Exercise 4.25

Exercise 4.26

This result can also be derived directly by making use of the results for the marginalof a Gaussian distribution given in Section 2.3.2.
The integral over a represents the convolution of a Gaussian with a logistic sigmoid, and cannot be evaluated analytically. We can, however, obtain a good approximation (Spiegelhalter and Lauritzen, 1990; MacKay, 1992b; Barber and Bishop,
1998a) by making use of the close similarity between the logistic sigmoid function
σ(a) deﬁned by (4.59) and the probit function Φ(a) deﬁned by (4.114). In order toobtain the best approximation to the logistic function we need to re-scale the horizontal axis, so that we approximate σ(a) by Φ(λa). We can ﬁnd a suitable value of
λ by requiring that the two functions have the same slope at the origin, which gives
λ2 = π/8. The similarity of the logistic sigmoid and the probit function, for thischoice of λ, is illustrated in Figure 4.9.
The advantage of using a probit function is that its convolution with a Gaussian
can be expressed analytically in terms of another probit function. Speciﬁcally we
can show that



µ
2
Φ(λa)N (a|µ, σ ) da = Φ.
(4.152)(λ−2 + σ 2 )1/2

220

4. LINEAR MODELS FOR CLASSIFICATION
We now apply the approximation σ(a)  Φ(λa) to the probit functions appearing
on both sides of this equation, leading to the following approximation for the convolution of a logistic sigmoid with a Gaussian
σ(a)N (a|µ, σ 2 ) da  σ κ(σ 2 )µ(4.153)where we have deﬁned

κ(σ 2 ) = (1 + πσ 2 /8)−1/2 .

(4.154)

Applying this result to (4.151) we obtain the approximate predictive distribution
in the form
(4.155)p(C1 |φ, t) = σ κ(σa2 )µa
where µa and σa2 are deﬁned by (4.149) and (4.150), respectively, and κ(σa2 ) is deﬁned by (4.154).
Note that the decision boundary corresponding to p(C1 |φ, t) = 0.5 is given byµa = 0, which is the same as the decision boundary obtained by using the MAP
value for w. Thus if the decision criterion is based on minimizing misclassiﬁcation rate, with equal prior probabilities, then the marginalization over w has no effect. However, for more complex decision criteria it will play an important role.
Marginalization of the logistic sigmoid model under a Gaussian approximation to
the posterior distribution will be illustrated in the context of variational inference inFigure 10.13.

Exercises

4.1 ( ) Given a set of data points {xn }, we can deﬁne the convex hull to be the set of
all points x given by

x=
αn xn
(4.156)nwhere αn  0 and n αn = 1. Consider a second set of points {yn } together with
their corresponding convex hull. By deﬁnition, the two sets of points will be linearly and a scalar w0 such that w
 T xn + w0 > 0 for all
separable if there exists a vector w
Txn , and w
 yn + w0 < 0 for all yn . Show that if their convex hulls intersect, the two
sets of points cannot be linearly separable, and conversely that if they are linearly
separable, their convex hulls do not intersect.
4.2 ( ) www Consider the minimization of a sum-of-squares error function (4.15),
and suppose that all of the target vectors in the training set satisfy a linear constraint
aT t n + b = 0(4.157)

where tn corresponds to the nth row of the matrix T in (4.15). Show that as aconsequence of this constraint, the elements of the model prediction y(x) given by
the least-squares solution (4.17) also satisfy this constraint, so thataT y(x) + b = 0.

(4.158)

Exercises

221

To do so, assume that one of the basis functions φ0 (x) = 1 so that the corresponding
parameter w0 plays the role of a bias.
4.3 ( ) Extend the result of Exercise 4.2 to show that if multiple linear constraints
are satisﬁed simultaneously by the target vectors, then the same constraints will also
be satisﬁed by the least-squares prediction of a linear model.
4.4 ( ) www Show that maximization of the class separation criterion given by (4.23)
with respect to w, using a Lagrange multiplier to enforce the constraint wT w = 1,
leads to the result that w ∝ (m2 − m1 ).
4.5 ( ) By making use of (4.20), (4.23), and (4.24), show that the Fisher criterion (4.25)
can be written in the form (4.26).
4.6 ( ) Using the deﬁnitions of the between-class and within-class covariance matrices
given by (4.27) and (4.28), respectively, together with (4.34) and (4.36) and the
choice of target values described in Section 4.1.5, show that the expression (4.33)
that minimizes the sum-of-squares error function can be written in the form (4.37).
4.7 ( ) www Show that the logistic sigmoid function (4.59) satisﬁes the propertyσ(−a) = 1 − σ(a) and that its inverse is given by σ −1 (y) = ln {y/(1 − y)}.
4.8 ( ) Using (4.57) and (4.58), derive the result (4.65) for the posterior class probability
in the two-class generative model with Gaussian densities, and verify the results(4.66) and (4.67) for the parameters w and w0 .
4.9 ( ) www Consider a generative classiﬁcation model for K classes deﬁned byprior class probabilities p(Ck ) = πk and general class-conditional densities p(φ|Ck )where φ is the input feature vector. Suppose we are given a training data set {φn , tn }where n = 1, . . . , N , and tn is a binary target vector of length K that uses the 1-ofK coding scheme, so that it has components tnj = Ijk if pattern n is from class Ck .
Assuming that the data points are drawn independently from this model, show thatthe maximum-likelihood solution for the prior probabilities is given by
πk =

Nk
N

(4.159)where Nk is the number of data points assigned to class Ck .
4.10 ( ) Consider the classiﬁcation model of Exercise 4.9 and now suppose that theclass-conditional densities are given by Gaussian distributions with a shared covariance matrix, so thatp(φ|Ck ) = N (φ|µk , Σ).
(4.160)Show that the maximum likelihood solution for the mean of the Gaussian distribution
for class Ck is given byN
1 
tnk φn(4.161)
µk =
Nk
n=1

222

4. LINEAR MODELS FOR CLASSIFICATION
which represents the mean of those feature vectors assigned to class Ck . Similarly,
show that the maximum likelihood solution for the shared covariance matrix is given
by
K

Nk
Σ=
(4.162)
Sk
N
k=1

where

1 
tnk (φn − µk )(φn − µk )T .
Nk
N

Sk =

(4.163)n=1

Thus Σ is given by a weighted average of the covariances of the data associated witheach class, in which the weighting coefﬁcients are given by the prior probabilities of
the classes.
4.11 ( ) Consider a classiﬁcation problem with K classes for which the feature vector
φ has M components each of which can take L discrete states. Let the values of thecomponents be represented by a 1-of-L binary coding scheme. Further suppose that,conditioned on the class Ck , the M components of φ are independent, so that theclass-conditional density factorizes with respect to the feature vector components.
Show that the quantities ak given by (4.63), which appear in the argument to thesoftmax function describing the posterior class probabilities, are linear functions of
the components of φ. Note that this represents an example of the naive Bayes model
which is discussed in Section 8.2.2.
4.12 ( ) www Verify the relation (4.88) for the derivative of the logistic sigmoid function deﬁned by (4.59).
4.13 ( ) www By making use of the result (4.88) for the derivative of the logistic sigmoid, show that the derivative of the error function (4.90) for the logistic regression
model is given by (4.91).
4.14 ( ) Show that for a linearly separable data set, the maximum likelihood solution
for the logistic regression model is obtained by ﬁnding a vector w whose decisionboundary wT φ(x) = 0 separates the classes and then taking the magnitude of w to
inﬁnity.
4.15 ( ) Show that the Hessian matrix H for the logistic regression model, given by(4.97), is positive deﬁnite. Here R is a diagonal matrix with elements yn (1 − yn ),and yn is the output of the logistic regression model for input vector xn . Hence showthat the error function is a concave function of w and that it has a unique minimum.
4.16 ( ) Consider a binary classiﬁcation problem in which each observation xn is known
to belong to one of two classes, corresponding to t = 0 and t = 1, and suppose thatthe procedure for collecting training data is imperfect, so that training points are
sometimes mislabelled. For every data point xn , instead of having a value t for theclass label, we have instead a value πn representing the probability that tn = 1.
Given a probabilistic model p(t = 1|φ), write down the log likelihood function
appropriate to such a data set.

Exercises

223

4.17 ( ) www Show that the derivatives of the softmax activation function (4.104),
where the ak are deﬁned by (4.105), are given by (4.106).
4.18 ( ) Using the result (4.91) for the derivatives of the softmax activation function,
show that the gradients of the cross-entropy error (4.108) are given by (4.109).
4.19 ( ) www Write down expressions for the gradient of the log likelihood, as wellas the corresponding Hessian matrix, for the probit regression model deﬁned in Section 4.3.5. These are the quantities that would be required to train such a model using
IRLS.
4.20 ( ) Show that the Hessian matrix for the multiclass logistic regression problem,
deﬁned by (4.110), is positive semideﬁnite. Note that the full Hessian matrix for
this problem is of size M K × M K, where M is the number of parameters and K
is the number of classes. To prove the positive semideﬁnite property, consider the
product uT Hu where u is an arbitrary vector of length M K, and then apply Jensen’s
inequality.
4.21 ( ) Show that the probit function (4.114) and the erf function (4.115) are related by(4.116).
4.22 ( ) Using the result (4.135), derive the expression (4.137) for the log model evidence under the Laplace approximation.
4.23 ( ) www In this exercise, we derive the BIC result (4.139) starting from the
Laplace approximation to the model evidence given by (4.137). Show that if theprior over parameters is Gaussian of the form p(θ) = N (θ|m, V0 ), the log model
evidence under the Laplace approximation takes the form11
ln p(D)  ln p(D|θ MAP ) − (θ MAP − m)T V0−1 (θ MAP − m) − ln |H| + const
2
2
where H is the matrix of second derivatives of the log likelihood ln p(D|θ) evaluatedat θ MAP . Now assume that the prior is broad so that V0−1 is small and the second
term on the right-hand side above can be neglected. Furthermore, consider the caseof independent, identically distributed data so that H is the sum of terms one for eachdata point. Show that the log model evidence can then be written approximately inthe form of the BIC expression (4.139).
4.24 ( ) Use the results from Section 2.3.2 to derive the result (4.151) for the marginalization of the logistic regression model with respect to a Gaussian posterior distribution over the parameters w.
4.25 ( ) Suppose we wish to approximate the logistic sigmoid σ(a) deﬁned by (4.59)
by a scaled probit function Φ(λa), where Φ(a) is deﬁned by (4.114). Show that if
λ is chosen so that the derivatives of the two functions are equal at a = 0, thenλ2 = π/8.

224

4. LINEAR MODELS FOR CLASSIFICATION
4.26 ( ) In this exercise, we prove the relation (4.152) for the convolution of a probit
function with a Gaussian distribution. To do this, show that the derivative of the lefthand side with respect to µ is equal to the derivative of the right-hand side, and thenintegrate both sides with respect to µ and then show that the constant of integration
vanishes. Note that before differentiating the left-hand side, it is convenient ﬁrst
to introduce a change of variable given by a = µ + σz so that the integral over ais replaced by an integral over z. When we differentiate the left-hand side of therelation (4.152), we will then obtain a Gaussian integral over z that can be evaluatedanalytically.

5
NeuralNetworks

In Chapters 3 and 4 we considered models for regression and classiﬁcation that comprised linear combinations of ﬁxed basis functions. We saw that such models haveuseful analytical and computational properties but that their practical applicability
was limited by the curse of dimensionality. In order to apply such models to largescale problems, it is necessary to adapt the basis functions to the data.
Support vector machines (SVMs), discussed in Chapter 7, address this by ﬁrst
deﬁning basis functions that are centred on the training data points and then selecting
a subset of these during training. One advantage of SVMs is that, although thetraining involves nonlinear optimization, the objective function is convex, and so thesolution of the optimization problem is relatively straightforward. The number ofbasis functions in the resulting models is generally much smaller than the number oftraining points, although it is often still relatively large and typically increases withthe size of the training set. The relevance vector machine, discussed in Section 7.2,
also chooses a subset from a ﬁxed set of basis functions and typically results in much

225

226

5. NEURAL NETWORKS
sparser models. Unlike the SVM it also produces probabilistic outputs, although thisis at the expense of a nonconvex optimization during training.
An alternative approach is to ﬁx the number of basis functions in advance butallow them to be adaptive, in other words to use parametric forms for the basis functions in which the parameter values are adapted during training. The most successfulmodel of this type in the context of pattern recognition is the feed-forward neuralnetwork, also known as the multilayer perceptron, discussed in this chapter. In fact,‘multilayer perceptron’ is really a misnomer, because the model comprises multiple layers of logistic regression models (with continuous nonlinearities) rather thanmultiple perceptrons (with discontinuous nonlinearities). For many applications, theresulting model can be signiﬁcantly more compact, and hence faster to evaluate, thana support vector machine having the same generalization performance. The price tobe paid for this compactness, as with the relevance vector machine, is that the likelihood function, which forms the basis for network training, is no longer a convex
function of the model parameters. In practice, however, it is often worth investing
substantial computational resources during the training phase in order to obtain acompact model that is fast at processing new data.
The term ‘neural network’ has its origins in attempts to ﬁnd mathematical representations of information processing in biological systems (McCulloch and Pitts,
1943; Widrow and Hoff, 1960; Rosenblatt, 1962; Rumelhart et al., 1986). Indeed,
it has been used very broadly to cover a wide range of different models, many ofwhich have been the subject of exaggerated claims regarding their biological plausibility. From the perspective of practical applications of pattern recognition, however, biological realism would impose entirely unnecessary constraints. Our focus inthis chapter is therefore on neural networks as efﬁcient models for statistical patternrecognition. In particular, we shall restrict our attention to the speciﬁc class of neural networks that have proven to be of greatest practical value, namely the multilayer
perceptron.
We begin by considering the functional form of the network model, including
the speciﬁc parameterization of the basis functions, and we then discuss the problem of determining the network parameters within a maximum likelihood framework, which involves the solution of a nonlinear optimization problem. This requires
the evaluation of derivatives of the log likelihood function with respect to the network parameters, and we shall see how these can be obtained efﬁciently using thetechnique of error backpropagation. We shall also show how the backpropagation
framework can be extended to allow other derivatives to be evaluated, such as theJacobian and Hessian matrices. Next we discuss various approaches to regularization of neural network training and the relationships between them. We also consider
some extensions to the neural network model, and in particular we describe a general framework for modelling conditional probability distributions known as mixture
density networks. Finally, we discuss the use of Bayesian treatments of neural networks. Additional background on neural network models can be found in Bishop(1995a).

5.1. Feed-forward Network Functions

227

5.1. Feed-forward Network Functions
The linear models for regression and classiﬁcation discussed in Chapters 3 and 4, respectively, are based on linear combinations of ﬁxed nonlinear basis functions φj (x)and take the formM
wj φj (x)(5.1)
y(x, w) = f
j =1

where f (·) is a nonlinear activation function in the case of classiﬁcation and is theidentity in the case of regression. Our goal is to extend this model by making thebasis functions φj (x) depend on parameters and then to allow these parameters tobe adjusted, along with the coefﬁcients {wj }, during training. There are, of course,many ways to construct parametric nonlinear basis functions. Neural networks use
basis functions that follow the same form as (5.1), so that each basis function is itself
a nonlinear function of a linear combination of the inputs, where the coefﬁcients inthe linear combination are adaptive parameters.
This leads to the basic neural network model, which can be described a seriesof functional transformations. First we construct M linear combinations of the input
variables x1 , . . . , xD in the form
aj =

D


(1)

(1)

wji xi + wj 0

(5.2)

i=1

where j = 1, . . . , M , and the superscript (1) indicates that the corresponding param(1)eters are in the ﬁrst ‘layer’ of the network. We shall refer to the parameters wji as(1)weights and the parameters wj 0 as biases, following the nomenclature of Chapter 3.
The quantities aj are known as activations. Each of them is then transformed using
a differentiable, nonlinear activation function h(·) to givezj = h(aj ).

Exercise 5.1(5.3)

These quantities correspond to the outputs of the basis functions in (5.1) that, in thecontext of neural networks, are called hidden units. The nonlinear functions h(·) are
generally chosen to be sigmoidal functions such as the logistic sigmoid or the ‘tanh’
function. Following (5.1), these values are again linearly combined to give output
unit activations
M

(2)
(2)
wkj zj + wk0
(5.4)
ak =
j =1

where k = 1, . . . , K, and K is the total number of outputs. This transformation cor(2)responds to the second layer of the network, and again the wk0 are bias parameters.
Finally, the output unit activations are transformed using an appropriate activation
function to give a set of network outputs yk . The choice of activation function isdetermined by the nature of the data and the assumed distribution of target variables

228

5. NEURAL NETWORKS
Figure 5.1

Network diagram for the twolayer neural network corresponding to (5.7). The input,
hidden, and output variables
are represented by nodes, andxD
the weight parameters are represented by links between thenodes, in which the bias parameters are denoted by links inputs
coming from additional input
and hidden variables x0 andz0 . Arrows denote the direcx1
tion of information ﬂow throughthe network during forward
propagation.

(1)

wM D

hidden units
zM
(2)

wKM
yK
outputs
y1

z1

x0

(2)

w10

z0

and follows the same considerations as for linear models discussed in Chapters 3 and4. Thus for standard regression problems, the activation function is the identity sothat yk = ak . Similarly, for multiple binary classiﬁcation problems, each output unit
activation is transformed using a logistic sigmoid function so thatyk = σ(ak )where
σ(a) =

1.
1 + exp(−a)

(5.5)

(5.6)

Finally, for multiclass problems, a softmax activation function of the form (4.62)
is used. The choice of output unit activation function is discussed in detail in Section 5.2.
We can combine these various stages to give the overall network function that,for sigmoidal output unit activation functions, takes the form
M
D


 (2)
 (1)
(1)
(2)
wkj h
wji xi + wj 0 + wk0(5.7)
yk (x, w) = σ
j =1

i=1

where the set of all weight and bias parameters have been grouped together into avector w. Thus the neural network model is simply a nonlinear function from a setof input variables {xi } to a set of output variables {yk } controlled by a vector w ofadjustable parameters.
This function can be represented in the form of a network diagram as shownin Figure 5.1. The process of evaluating (5.7) can then be interpreted as a forward
propagation of information through the network. It should be emphasized that these
diagrams do not represent probabilistic graphical models of the kind to be considered in Chapter 8 because the internal nodes represent deterministic variables rather
than stochastic ones. For this reason, we have adopted a slightly different graphical5.1. Feed-forward Network Functions

229

notation for the two kinds of model. We shall see later how to give a probabilistic
interpretation to a neural network.
As discussed in Section 3.1, the bias parameters in (5.2) can be absorbed into
the set of weight parameters by deﬁning an additional input variable x0 whose value
is clamped at x0 = 1, so that (5.2) takes the form
aj =

D


(1)

wji xi .

(5.8)

i=0

We can similarly absorb the second-layer biases into the second-layer weights, sothat the overall network function becomes
M
D

 (2)
 (1)
yk (x, w) = σ
wkj h
wji xi
.
(5.9)
j =0

i=0

As can be seen from Figure 5.1, the neural network model comprises two stagesof processing, each of which resembles the perceptron model of Section 4.1.7, andfor this reason the neural network is also known as the multilayer perceptron, or
MLP. A key difference compared to the perceptron, however, is that the neural network uses continuous sigmoidal nonlinearities in the hidden units, whereas the perceptron uses step-function nonlinearities. This means that the neural network function is differentiable with respect to the network parameters, and this property willplay a central role in network training.
If the activation functions of all the hidden units in a network are taken to belinear, then for any such network we can always ﬁnd an equivalent network without
hidden units. This follows from the fact that the composition of successive linear
transformations is itself a linear transformation. However, if the number of hidden
units is smaller than either the number of input or output units, then the transformations that the network can generate are not the most general possible linear transformations from inputs to outputs because information is lost in the dimensionality
reduction at the hidden units. In Section 12.4.2, we show that networks of linear
units give rise to principal component analysis. In general, however, there is little
interest in multilayer networks of linear units.
The network architecture shown in Figure 5.1 is the most commonly used one
in practice. However, it is easily generalized, for instance by considering additional
layers of processing each consisting of a weighted linear combination of the form
(5.4) followed by an element-wise transformation using a nonlinear activation function. Note that there is some confusion in the literature regarding the terminology
for counting the number of layers in such networks. Thus the network in Figure 5.1
may be described as a 3-layer network (which counts the number of layers of units,
and treats the inputs as units) or sometimes as a single-hidden-layer network (whichcounts the number of layers of hidden units). We recommend a terminology in which
Figure 5.1 is called a two-layer network, because it is the number of layers of adaptive weights that is important for determining the network properties.
Another generalization of the network architecture is to include skip-layer connections, each of which is associated with a corresponding adaptive parameter. For

230

5. NEURAL NETWORKS
Figure 5.2

Example of a neural network having ageneral feed-forward topology. Note thateach hidden and output unit has an
associated bias parameter (omitted forclarity).

z2
y2

x2
inputsz1

outputsy1

x1
z3

instance, in a two-layer network these would go directly from inputs to outputs. In
principle, a network with sigmoidal hidden units can always mimic skip layer connections (for bounded input values) by using a sufﬁciently small ﬁrst-layer weight
that, over its operating range, the hidden unit is effectively linear, and then compensating with a large weight value from the hidden unit to the output. In practice,
however, it may be advantageous to include skip-layer connections explicitly.
Furthermore, the network can be sparse, with not all possible connections withina layer being present. We shall see an example of a sparse network architecture when
we consider convolutional neural networks in Section 5.5.6.
Because there is a direct correspondence between a network diagram and its
mathematical function, we can develop more general network mappings by considering more complex network diagrams. However, these must be restricted to afeed-forward architecture, in other words to one having no closed directed cycles, toensure that the outputs are deterministic functions of the inputs. This is illustrated
with a simple example in Figure 5.2. Each (hidden or output) unit in such a network
computes a function given by
wkj zj(5.10)
zk = hjwhere the sum runs over all units that send connections to unit k (and a bias parameter is included in the summation). For a given set of values applied to the inputs of
the network, successive application of (5.10) allows the activations of all units in the
network to be evaluated including those of the output units.
The approximation properties of feed-forward networks have been widely studied (Funahashi, 1989; Cybenko, 1989; Hornik et al., 1989; Stinchecombe and White,
1989; Cotter, 1990; Ito, 1991; Hornik, 1991; Kreinovich, 1991; Ripley, 1996) andfound to be very general. Neural networks are therefore said to be universal approximators. For example, a two-layer network with linear outputs can uniformly
approximate any continuous function on a compact input domain to arbitrary accuracy provided the network has a sufﬁciently large number of hidden units. This result
holds for a wide range of hidden unit activation functions, but excluding polynomials. Although such theorems are reassuring, the key problem is how to ﬁnd suitableparameter values given a set of training data, and in later sections of this chapter we

5.1. Feed-forward Network Functions
Figure 5.3 Illustration of the capability of a multilayer perceptron
to approximate four different functions comprising (a) f (x) = x2 , (b)f (x) = sin(x), (c), f (x) = |x|,and (d) f (x) = H(x) where H(x)is the Heaviside step function. In
each case, N = 50 data points,
shown as blue dots, have been sampled uniformly in x over the interval(−1, 1) and the corresponding values of f (x) evaluated. These data
points are then used to train a twolayer network having 3 hidden units
with ‘tanh’ activation functions andlinear output units. The resulting
network functions are shown by the
red curves, and the outputs of thethree hidden units are shown by thethree dashed curves.

(a)

(b)

(c)

(d)

231

will show that there exist effective solutions to this problem based on both maximum
likelihood and Bayesian approaches.
The capability of a two-layer network to model a broad range of functions is
illustrated in Figure 5.3. This ﬁgure also shows how individual hidden units work
collaboratively to approximate the ﬁnal function. The role of hidden units in a simple
classiﬁcation problem is illustrated in Figure 5.4 using the synthetic classiﬁcation
data set described in Appendix A.

5.1.1 Weight-space symmetries
One property of feed-forward networks, which will play a role when we consider
Bayesian model comparison, is that multiple distinct choices for the weight vector
w can all give rise to the same mapping function from inputs to outputs (Chen et al.,1993). Consider a two-layer network of the form shown in Figure 5.1 with M hidden
units having ‘tanh’ activation functions and full connectivity in both layers. If wechange the sign of all of the weights and the bias feeding into a particular hidden
unit, then, for a given input pattern, the sign of the activation of the hidden unit will
be reversed, because ‘tanh’ is an odd function, so that tanh(−a) = − tanh(a). Thistransformation can be exactly compensated by changing the sign of all of the weights
leading out of that hidden unit. Thus, by changing the signs of a particular group of
weights (and a bias), the input–output mapping function represented by the network
is unchanged, and so we have found two different weight vectors that give rise to
the same mapping function. For M hidden units, there will be M such ‘sign-ﬂip’

232

5. NEURAL NETWORKS
Figure 5.4

Example of the solution of a simple two- 3
class classiﬁcation problem involving
synthetic data using a neural network
having two inputs, two hidden units with 2‘tanh’ activation functions, and a single
output having a logistic sigmoid activa- 1
tion function. The dashed blue lines
show the z = 0.5 contours for each of 0
the hidden units, and the red line shows
the y = 0.5 decision surface for the net- −1
work. For comparison, the green line
denotes the optimal decision boundary −2computed from the distributions used togenerate the data.

−2

−1

0

1

2

symmetries, and thus any given weight vector will be one of a set 2M equivalentweight vectors .
Similarly, imagine that we interchange the values of all of the weights (and thebias) leading both into and out of a particular hidden unit with the corresponding
values of the weights (and bias) associated with a different hidden unit. Again, thisclearly leaves the network input–output mapping function unchanged, but it corresponds to a different choice of weight vector. For M hidden units, any given weightvector will belong to a set of M ! equivalent weight vectors associated with this interchange symmetry, corresponding to the M ! different orderings of the hidden units.
The network will therefore have an overall weight-space symmetry factor of M !2M .
For networks with more than two layers of weights, the total level of symmetry will
be given by the product of such factors, one for each layer of hidden units.
It turns out that these factors account for all of the symmetries in weight space(except for possible accidental symmetries due to speciﬁc choices for the weight values). Furthermore, the existence of these symmetries is not a particular property ofthe ‘tanh’ function but applies to a wide range of activation functions (Ku̇rková andKainen, 1994). In many cases, these symmetries in weight space are of little practical consequence, although in Section 5.7 we shall encounter a situation in which weneed to take them into account.

5.2. Network Training
So far, we have viewed neural networks as a general class of parametric nonlinear
functions from a vector x of input variables to a vector y of output variables. A
simple approach to the problem of determining the network parameters is to make ananalogy with the discussion of polynomial curve ﬁtting in Section 1.1, and therefore
to minimize a sum-of-squares error function. Given a training set comprising a setof input vectors {xn }, where n = 1, . . . , N , together with a corresponding set of

5.2. Network Training

233

target vectors {tn }, we minimize the error function1
y(xn , w) − tn 2 .
2NE(w) =(5.11)

n=1

However, we can provide a much more general view of network training by ﬁrst
giving a probabilistic interpretation to the network outputs. We have already seen
many advantages of using probabilistic predictions in Section 1.5.4. Here it will also
provide us with a clearer motivation both for the choice of output unit nonlinearity
and the choice of error function.
We start by discussing regression problems, and for the moment we consider
a single target variable t that can take any real value. Following the discussions
in Section 1.2.5 and 3.1, we assume that t has a Gaussian distribution with an xdependent mean, which is given by the output of the neural network, so thatp(t|x, w) = N t|y(x, w), β −1(5.12)

where β is the precision (inverse variance) of the Gaussian noise. Of course thisis a somewhat restrictive assumption, and in Section 5.6 we shall see how to extend
this approach to allow for more general conditional distributions. For the conditional
distribution given by (5.12), it is sufﬁcient to take the output unit activation function
to be the identity, because such a network can approximate any continuous function
from x to y. Given a data set of N independent, identically distributed observations
X = {x1 , . . . , xN }, along with corresponding target values t = {t1 , . . . , tN }, we
can construct the corresponding likelihood function
p(t|X, w, β) =Np(tn |xn , w, β).

n=1Taking the negative logarithm, we obtain the error functionβNN{y(xn , w) − tn }2 −ln β +ln(2π)
2
2
2
N(5.13)

n=1

which can be used to learn the parameters w and β. In Section 5.7, we shall discuss the Bayesian treatment of neural networks, while here we consider a maximum
likelihood approach. Note that in the neural networks literature, it is usual to consider the minimization of an error function rather than the maximization of the (log)likelihood, and so here we shall follow this convention. Consider ﬁrst the determination of w. Maximizing the likelihood function is equivalent to minimizing thesum-of-squares error function given by1
{y(xn , w) − tn }22N

E(w) =

n=1(5.14)

234

5. NEURAL NETWORKS
where we have discarded additive and multiplicative constants. The value of w foundby minimizing E(w) will be denoted wML because it corresponds to the maximum
likelihood solution. In practice, the nonlinearity of the network function y(xn , w)
causes the error E(w) to be nonconvex, and so in practice local maxima of thelikelihood may be found, corresponding to local minima of the error function, as
discussed in Section 5.2.1.
Having found wML , the value of β can be found by minimizing the negative log
likelihood to give
N
1
1 
=
{y(xn , wML ) − tn }2 .
(5.15)βMLN
n=1

Note that this can be evaluated once the iterative optimization required to ﬁnd wML
is completed. If we have multiple target variables, and we assume that they are independent conditional on x and w with shared noise precision β, then the conditional
distribution of the target values is given byp(t|x, w) = N t|y(x, w), β −1 I .

Exercise 5.2

Following the same argument as for a single target variable, we see that the maximum
likelihood weights are determined by minimizing the sum-of-squares error function(5.11). The noise precision is then given by
1
βML

Exercise 5.3(5.16)1 
y(xn , wML ) − tn 2
NKN=(5.17)

n=1

where K is the number of target variables. The assumption of independence can be
dropped at the expense of a slightly more complex optimization problem.
Recall from Section 4.3.6 that there is a natural pairing of the error function(given by the negative log likelihood) and the output unit activation function. In the
regression case, we can view the network as having an output activation function thatis the identity, so that yk = ak . The corresponding sum-of-squares error function
has the property∂E= yk − tk(5.18)∂ak
which we shall make use of when discussing error backpropagation in Section 5.3.
Now consider the case of binary classiﬁcation in which we have a single target
variable t such that t = 1 denotes class C1 and t = 0 denotes class C2 . Following
the discussion of canonical link functions in Section 4.3.6, we consider a network
having a single output whose activation function is a logistic sigmoid
y = σ(a) ≡1
1 + exp(−a)(5.19)

so that 0  y(x, w)  1. We can interpret y(x, w) as the conditional probabilityp(C1 |x), with p(C2 |x) given by 1 − y(x, w). The conditional distribution of targets
given inputs is then a Bernoulli distribution of the form
1−tp(t|x, w) = y(x, w)t {1 − y(x, w)}.

(5.20)

5.2. Network Training

235

If we consider a training set of independent observations, then the error function,
which is given by the negative log likelihood, is then a cross-entropy error functionof the formN
{tn ln yn + (1 − tn ) ln(1 − yn )}(5.21)
E(w) = −n=1

Exercise 5.4

where yn denotes y(xn , w). Note that there is no analogue of the noise precision βbecause the target values are assumed to be correctly labelled. However, the modelis easily extended to allow for labelling errors. Simard et al. (2003) found that using
the cross-entropy error function instead of the sum-of-squares for a classiﬁcation
problem leads to faster training as well as improved generalization.
If we have K separate binary classiﬁcations to perform, then we can use a network having K outputs each of which has a logistic sigmoid activation function.
Associated with each output is a binary class label tk ∈ {0, 1}, where k = 1, . . . , K.
If we assume that the class labels are independent, given the input vector, then theconditional distribution of the targets isp(t|x, w) =

K


1−tk

yk (x, w)tk [1 − yk (x, w)].

(5.22)

k=1

Exercise 5.5

Taking the negative logarithm of the corresponding likelihood function then gives
the following error function
E(w) = −N K{tnk ln ynk + (1 − tnk ) ln(1 − ynk )}(5.23)

n=1 k=1

Exercise 5.6

where ynk denotes yk (xn , w). Again, the derivative of the error function with respect to the activation for a particular output unit takes the form (5.18) just as in the
regression case.
It is interesting to contrast the neural network solution to this problem with thecorresponding approach based on a linear classiﬁcation model of the kind discussed
in Chapter 4. Suppose that we are using a standard two-layer network of the kindshown in Figure 5.1. We see that the weight parameters in the ﬁrst layer of the
network are shared between the various outputs, whereas in the linear model eachclassiﬁcation problem is solved independently. The ﬁrst layer of the network can
be viewed as performing a nonlinear feature extraction, and the sharing of features
between the different outputs can save on computation and can also lead to improved
generalization.
Finally, we consider the standard multiclass classiﬁcation problem in which each
input is assigned to one of K mutually exclusive classes. The binary target variables
tk ∈ {0, 1} have a 1-of-K coding scheme indicating the class, and the network
outputs are interpreted as yk (x, w) = p(tk = 1|x), leading to the following error
functionN Ktkn ln yk (xn , w).
(5.24)
E(w) = −n=1 k=1

236

5. NEURAL NETWORKS
Figure 5.5

Geometrical view of the error function E(w) asa surface sitting over weight space. Point wA isa local minimum and wB is the global minimum.
At any point wC , the local gradient of the error
surface is given by the vector ∇E.

E(w)

w1
wA
w2

wB

wC
∇E

Following the discussion of Section 4.3.4, we see that the output unit activation
function, which corresponds to the canonical link, is given by the softmax function
exp(ak (x, w))yk (x, w) = 
exp(aj (x, w))(5.25)

j

Exercise 5.7

which satisﬁes 0  yk  1 and k yk = 1. Note that the yk (x, w) are unchanged
if a constant is added to all of the ak (x, w), causing the error function to be constantfor some directions in weight space. This degeneracy is removed if an appropriate
regularization term (Section 5.5) is added to the error function.
Once again, the derivative of the error function with respect to the activation for
a particular output unit takes the familiar form (5.18).
In summary, there is a natural choice of both output unit activation function
and matching error function, according to the type of problem being solved. For regression we use linear outputs and a sum-of-squares error, for (multiple independent)
binary classiﬁcations we use logistic sigmoid outputs and a cross-entropy error function, and for multiclass classiﬁcation we use softmax outputs with the corresponding
multiclass cross-entropy error function. For classiﬁcation problems involving twoclasses, we can use a single logistic sigmoid output, or alternatively we can use a
network with two outputs having a softmax output activation function.

5.2.1 Parameter optimization
We turn next to the task of ﬁnding a weight vector w which minimizes thechosen function E(w). At this point, it is useful to have a geometrical picture of theerror function, which we can view as a surface sitting over weight space as shown in
Figure 5.5. First note that if we make a small step in weight space from w to w + δw
then the change in the error function is δE  δwT ∇E(w), where the vector ∇E(w)points in the direction of greatest rate of increase of the error function. Because theerror E(w) is a smooth continuous function of w, its smallest value will occur at a

5.2. Network Training

237

point in weight space such that the gradient of the error function vanishes, so that
∇E(w) = 0

Section 5.1.1(5.26)

as otherwise we could make a small step in the direction of −∇E(w) and thereby
further reduce the error. Points at which the gradient vanishes are called stationary
points, and may be further classiﬁed into minima, maxima, and saddle points.
Our goal is to ﬁnd a vector w such that E(w) takes its smallest value. However, the error function typically has a highly nonlinear dependence on the weights
and bias parameters, and so there will be many points in weight space at which thegradient vanishes (or is numerically very small). Indeed, from the discussion in Section 5.1.1 we see that for any point w that is a local minimum, there will be otherpoints in weight space that are equivalent minima. For instance, in a two-layer network of the kind shown in Figure 5.1, with M hidden units, each point in weightspace is a member of a family of M !2M equivalent points.
Furthermore, there will typically be multiple inequivalent stationary points andin particular multiple inequivalent minima. A minimum that corresponds to thesmallest value of the error function for any weight vector is said to be a global
minimum. Any other minima corresponding to higher values of the error function
are said to be local minima. For a successful application of neural networks, it may
not be necessary to ﬁnd the global minimum (and in general it will not be known
whether the global minimum has been found) but it may be necessary to compare
several local minima in order to ﬁnd a sufﬁciently good solution.
Because there is clearly no hope of ﬁnding an analytical solution to the equation ∇E(w) = 0 we resort to iterative numerical procedures. The optimization ofcontinuous nonlinear functions is a widely studied problem and there exists an extensive literature on how to solve it efﬁciently. Most techniques involve choosing
some initial value w(0) for the weight vector and then moving through weight spacein a succession of steps of the formw(τ +1) = w(τ ) + ∆w(τ )(5.27)

where τ labels the iteration step. Different algorithms involve different choices forthe weight vector update ∆w(τ ) . Many algorithms make use of gradient informationand therefore require that, after each update, the value of ∇E(w) is evaluated at
the new weight vector w(τ +1) . In order to understand the importance of gradient
information, it is useful to consider a local approximation to the error function basedon a Taylor expansion.

5.2.2 Local quadratic approximation
Insight into the optimization problem, and into the various techniques for solving it, can be obtained by considering a local quadratic approximation to the error
function.
 in weight space
Consider the Taylor expansion of E(w) around some point w
1
 + (w − w)
 T b + (w − w)
 T H(w − w)E(w)  E(w)2

(5.28)

238

5. NEURAL NETWORKS
where cubic and higher terms have been omitted. Here b is deﬁned to be the gradientof E evaluated at w

(5.29)b ≡ ∇E|w=w
b
and the Hessian matrix H = ∇∇E has elements

∂E 
.
(H)ij ≡∂wi ∂wj w=w
b

(5.30)

From (5.28), the corresponding local approximation to the gradient is given by∇E  b + H(w − w).
(5.31)

 these expressions will give reasonable
For points w that are sufﬁciently close to w,
approximations for the error and its gradient.
Consider the particular case of a local quadratic approximation around a pointw that is a minimum of the error function. In this case there is no linear term,
because ∇E = 0 at w , and (5.28) becomes1
E(w) = E(w ) + (w − w )T H(w − w )2

(5.32)

where the Hessian H is evaluated at w . In order to interpret this geometrically,
consider the eigenvalue equation for the Hessian matrix
Hui = λi ui(5.33)

where the eigenvectors ui form a complete orthonormal set (Appendix C) so thatuT
i uj = δij .

(5.34)

We now expand (w − w ) as a linear combination of the eigenvectors in the form
αi u i .
(5.35)
w − w =i

This can be regarded as a transformation of the coordinate system in which the origin
is translated to the point w , and the axes are rotated to align with the eigenvectors(through the orthogonal matrix whose columns are the ui ), and is discussed in moredetail in Appendix C. Substituting (5.35) into (5.32), and using (5.33) and (5.34),
allows the error function to be written in the form1λi αi2 .
(5.36)
E(w) = E(w ) +2i

A matrix H is said to be positive deﬁnite if, and only if,vT Hv > 0

for all v.

(5.37)

239

5.2. Network Training
Figure 5.6

In the neighbourhood of a min- w2
imum w , the error function
can be approximated by aquadratic. Contours of constant error are then ellipses
whose axes are aligned withthe eigenvectors ui of the Hes−1/2
λ2sian matrix, with lengths thatare inversely proportional to thesquare roots of the corresponding eigenvectors λi .

u2

u1
w
−1/2

λ1

w1

Because the eigenvectors {ui } form a complete set, an arbitrary vector v can be
written in the formv=
ci ui .
(5.38)i

From (5.33) and (5.34), we then havevT Hv =

c2i λi(5.39)i

Exercise 5.10
Exercise 5.11

Exercise 5.12

and so H will be positive deﬁnite if, and only if, all of its eigenvalues are positive.
In the new coordinate system, whose basis vectors are given by the eigenvectors{ui }, the contours of constant E are ellipses centred on the origin, as illustrated
in Figure 5.6. For a one-dimensional weight space, a stationary point w will be aminimum if∂ 2 E > 0.
(5.40)∂w2 w
The corresponding result in D-dimensions is that the Hessian matrix, evaluated atw , should be positive deﬁnite.

5.2.3 Use of gradient information

Exercise 5.13

As we shall see in Section 5.3, it is possible to evaluate the gradient of an error
function efﬁciently by means of the backpropagation procedure. The use of thisgradient information can lead to signiﬁcant improvements in the speed with which
the minima of the error function can be located. We can see why this is so, as follows.
In the quadratic approximation to the error function, given in (5.28), the error
surface is speciﬁed by the quantities b and H, which contain a total of W (W +3)/2 independent elements (because the matrix H is symmetric), where W is thedimensionality of w (i.e., the total number of adaptive parameters in the network).
The location of the minimum of this quadratic approximation therefore depends onO(W 2 ) parameters, and we should not expect to be able to locate the minimum until
we have gathered O(W 2 ) independent pieces of information. If we do not makeuse of gradient information, we would expect to have to perform O(W 2 ) function

240

5. NEURAL NETWORKS
evaluations, each of which would require O(W ) steps. Thus, the computational
effort needed to ﬁnd the minimum using such an approach would be O(W 3 ).
Now compare this with an algorithm that makes use of the gradient information.
Because each evaluation of ∇E brings W items of information, we might hope toﬁnd the minimum of the function in O(W ) gradient evaluations. As we shall see,by using error backpropagation, each such evaluation takes only O(W ) steps and so
the minimum can now be found in O(W 2 ) steps. For this reason, the use of gradient
information forms the basis of practical algorithms for training neural networks.

5.2.4 Gradient descent optimization
The simplest approach to using gradient information is to choose the weight
update in (5.27) to comprise a small step in the direction of the negative gradient, sothatw(τ +1) = w(τ ) − η∇E(w(τ ) )(5.41)
where the parameter η > 0 is known as the learning rate. After each such update, thegradient is re-evaluated for the new weight vector and the process repeated. Note thatthe error function is deﬁned with respect to a training set, and so each step requiresthat the entire training set be processed in order to evaluate ∇E. Techniques that
use the whole data set at once are called batch methods. At each step the weightvector is moved in the direction of the greatest rate of decrease of the error function,and so this approach is known as gradient descent or steepest descent. Althoughsuch an approach might intuitively seem reasonable, in fact it turns out to be a poor
algorithm, for reasons discussed in Bishop and Nabney (2008).
For batch optimization, there are more efﬁcient methods, such as conjugate gradients and quasi-Newton methods, which are much more robust and much fasterthan simple gradient descent (Gill et al., 1981; Fletcher, 1987; Nocedal and Wright,
1999). Unlike gradient descent, these algorithms have the property that the error
function always decreases at each iteration unless the weight vector has arrived at a
local or global minimum.
In order to ﬁnd a sufﬁciently good minimum, it may be necessary to run a
gradient-based algorithm multiple times, each time using a different randomly chosen starting point, and comparing the resulting performance on an independent validation set.
There is, however, an on-line version of gradient descent that has proved usefulin practice for training neural networks on large data sets (Le Cun et al., 1989).
Error functions based on maximum likelihood for a set of independent observations
comprise a sum of terms, one for each data point
E(w) =N

En (w).

(5.42)

n=1

On-line gradient descent, also known as sequential gradient descent or stochastic
gradient descent, makes an update to the weight vector based on one data point at atime, so that
(5.43)w(τ +1) = w(τ ) − η∇En (w(τ ) ).

5.3. Error Backpropagation

241

This update is repeated by cycling through the data either in sequence or by selecting
points at random with replacement. There are of course intermediate scenarios inwhich the updates are based on batches of data points.
One advantage of on-line methods compared to batch methods is that the former
handle redundancy in the data much more efﬁciently. To see, this consider an extreme example in which we take a data set and double its size by duplicating every
data point. Note that this simply multiplies the error function by a factor of 2 and so
is equivalent to using the original error function. Batch methods will require double
the computational effort to evaluate the batch error function gradient, whereas online methods will be unaffected. Another property of on-line gradient descent is thepossibility of escaping from local minima, since a stationary point with respect tothe error function for the whole data set will generally not be a stationary point foreach data point individually.
Nonlinear optimization algorithms, and their practical application to neural network training, are discussed in detail in Bishop and Nabney (2008).

5.3. Error Backpropagation
Our goal in this section is to ﬁnd an efﬁcient technique for evaluating the gradientof an error function E(w) for a feed-forward neural network. We shall see that
this can be achieved using a local message passing scheme in which information issent alternately forwards and backwards through the network and is known as error
backpropagation, or sometimes simply as backprop.
It should be noted that the term backpropagation is used in the neural computing literature to mean a variety of different things. For instance, the multilayer
perceptron architecture is sometimes called a backpropagation network. The term
backpropagation is also used to describe the training of a multilayer perceptron using gradient descent applied to a sum-of-squares error function. In order to clarify
the terminology, it is useful to consider the nature of the training process more carefully. Most training algorithms involve an iterative procedure for minimization of anerror function, with adjustments to the weights being made in a sequence of steps. At
each such step, we can distinguish between two distinct stages. In the ﬁrst stage, thederivatives of the error function with respect to the weights must be evaluated. As
we shall see, the important contribution of the backpropagation technique is in providing a computationally efﬁcient method for evaluating such derivatives. Because
it is at this stage that errors are propagated backwards through the network, we shalluse the term backpropagation speciﬁcally to describe the evaluation of derivatives.
In the second stage, the derivatives are then used to compute the adjustments to bemade to the weights. The simplest such technique, and the one originally consideredby Rumelhart et al. (1986), involves gradient descent. It is important to recognizethat the two stages are distinct. Thus, the ﬁrst stage, namely the propagation of errors backwards through the network in order to evaluate derivatives, can be applied
to many other kinds of network and not just the multilayer perceptron. It can also be
applied to error functions other that just the simple sum-of-squares, and to the eval-

242

5. NEURAL NETWORKS
uation of other derivatives such as the Jacobian and Hessian matrices, as we shallsee later in this chapter. Similarly, the second stage of weight adjustment using the
calculated derivatives can be tackled using a variety of optimization schemes, manyof which are substantially more powerful than simple gradient descent.

5.3.1 Evaluation of error-function derivatives
We now derive the backpropagation algorithm for a general network having arbitrary feed-forward topology, arbitrary differentiable nonlinear activation functions,and a broad class of error function. The resulting formulae will then be illustratedusing a simple layered network structure having a single layer of sigmoidal hidden
units together with a sum-of-squares error.
Many error functions of practical interest, for instance those deﬁned by maximum likelihood for a set of i.i.d. data, comprise a sum of terms, one for each data
point in the training set, so that
E(w) =N

En (w).

(5.44)

n=1

Here we shall consider the problem of evaluating ∇En (w) for one such term in theerror function. This may be used directly for sequential optimization, or the results
can be accumulated over the training set in the case of batch methods.
Consider ﬁrst a simple linear model in which the outputs yk are linear combinations of the input variables xi so that
wki xi(5.45)yk =itogether with an error function that, for a particular input pattern n, takes the form1(ynk − tnk )2(5.46)En =2kwhere ynk = yk (xn , w). The gradient of this error function with respect to a weight
wji is given by∂En= (ynj − tnj )xni
(5.47)∂wji
which can be interpreted as a ‘local’ computation involving the product of an ‘error
signal’ ynj − tnj associated with the output end of the link wji and the variable xni
associated with the input end of the link. In Section 4.3.2, we saw how a similar
formula arises with the logistic sigmoid activation function together with the crossentropy error function, and similarly for the softmax activation function together
with its matching cross-entropy error function. We shall now see how this simple
result extends to the more complex setting of multilayer feed-forward networks.
In a general feed-forward network, each unit computes a weighted sum of its
inputs of the form
aj =wji zi(5.48)i5.3. Error Backpropagation

243

where zi is the activation of a unit, or input, that sends a connection to unit j, and wji
is the weight associated with that connection. In Section 5.1, we saw that biases canbe included in this sum by introducing an extra unit, or input, with activation ﬁxed
at +1. We therefore do not need to deal with biases explicitly. The sum in (5.48) istransformed by a nonlinear activation function h(·) to give the activation zj of unit jin the form
(5.49)
zj = h(aj ).
Note that one or more of the variables zi in the sum in (5.48) could be an input, andsimilarly, the unit j in (5.49) could be an output.
For each pattern in the training set, we shall suppose that we have supplied thecorresponding input vector to the network and calculated the activations of all of
the hidden and output units in the network by successive application of (5.48) and(5.49). This process is often called forward propagation because it can be regarded
as a forward ﬂow of information through the network.
Now consider the evaluation of the derivative of En with respect to a weightwji . The outputs of the various units will depend on the particular input pattern n.
However, in order to keep the notation uncluttered, we shall omit the subscript nfrom the network variables. First we note that En depends on the weight wji only
via the summed input aj to unit j. We can therefore apply the chain rule for partial
derivatives to give∂En
∂En ∂aj=.
(5.50)∂wji∂aj ∂wji
We now introduce a useful notation
δj ≡

∂En
∂aj

(5.51)

where the δ’s are often referred to as errors for reasons we shall see shortly. Using(5.48), we can write∂aj= zi .
(5.52)∂wji
Substituting (5.51) and (5.52) into (5.50), we then obtain∂En= δ j zi .
∂wji

(5.53)

Equation (5.53) tells us that the required derivative is obtained simply by multiplying
the value of δ for the unit at the output end of the weight by the value of z for the unit
at the input end of the weight (where z = 1 in the case of a bias). Note that this takes
the same form as for the simple linear model considered at the start of this section.
Thus, in order to evaluate the derivatives, we need only to calculate the value of δjfor each hidden and output unit in the network, and then apply (5.53).
As we have seen already, for the output units, we have
δ k = yk − tk

(5.54)

244

5. NEURAL NETWORKS
Figure 5.7

Illustration of the calculation of δj for hidden unit j bybackpropagation of the δ’s from those units k to which zi
unit j sends connections. The blue arrow denotes the
direction of information ﬂow during forward propagation,and the red arrows indicate the backward propagationof error information.

wji

δj
zj

δk
wkj
δ1

provided we are using the canonical link as the output-unit activation function. To
evaluate the δ’s for hidden units, we again make use of the chain rule for partial
derivatives,∂En  ∂En ∂akδj ≡=(5.55)∂aj∂ak ∂ajkwhere the sum runs over all units k to which unit j sends connections. The arrangement of units and weights is illustrated in Figure 5.7. Note that the units labelled kcould include other hidden units and/or output units. In writing down (5.55), we aremaking use of the fact that variations in aj give rise to variations in the error function only through variations in the variables ak . If we now substitute the deﬁnitionof δ given by (5.51) into (5.55), and make use of (5.48) and (5.49), we obtain thefollowing backpropagation formula
wkj δk
(5.56)
δj = h (aj )kwhich tells us that the value of δ for a particular hidden unit can be obtained bypropagating the δ’s backwards from units higher up in the network, as illustrated
in Figure 5.7. Note that the summation in (5.56) is taken over the ﬁrst index on
wkj (corresponding to backward propagation of information through the network),
whereas in the forward propagation equation (5.10) it is taken over the second index.
Because we already know the values of the δ’s for the output units, it follows thatby recursively applying (5.56) we can evaluate the δ’s for all of the hidden units in a
feed-forward network, regardless of its topology.
The backpropagation procedure can therefore be summarized as follows.
Error Backpropagation
1. Apply an input vector xn to the network and forward propagate throughthe network using (5.48) and (5.49) to ﬁnd the activations of all the hidden
and output units.
2. Evaluate the δk for all the output units using (5.54).
3. Backpropagate the δ’s using (5.56) to obtain δj for each hidden unit in the
network.
4. Use (5.53) to evaluate the required derivatives.

5.3. Error Backpropagation

245

For batch methods, the derivative of the total error E can then be obtained byrepeating the above steps for each pattern in the training set and then summing overall patterns:
 ∂En
∂E
=
.
(5.57)∂wji∂wjin

In the above derivation we have implicitly assumed that each hidden or output unit in
the network has the same activation function h(·). The derivation is easily generalized, however, to allow different units to have individual activation functions, simply
by keeping track of which form of h(·) goes with which unit.

5.3.2 A simple example
The above derivation of the backpropagation procedure allowed for generalforms for the error function, the activation functions, and the network topology. In
order to illustrate the application of this algorithm, we shall consider a particular
example. This is chosen both for its simplicity and for its practical importance, because many applications of neural networks reported in the literature make use ofthis type of network. Speciﬁcally, we shall consider a two-layer network of the form
illustrated in Figure 5.1, together with a sum-of-squares error, in which the output
units have linear activation functions, so that yk = ak , while the hidden units havelogistic sigmoid activation functions given byh(a) ≡ tanh(a)(5.58)

whereea − e−a.
(5.59)
ea + e−a
A useful feature of this function is that its derivative can be expressed in a particularly simple form:h (a) = 1 − h(a)2 .
(5.60)We also consider a standard sum-of-squares error function, so that for pattern n theerror is given byK1
(yk − tk )2(5.61)
En =2tanh(a) =k=1

where yk is the activation of output unit k, and tk is the corresponding target, for aparticular input pattern xn .
For each pattern in the training set in turn, we ﬁrst perform a forward propagation
using
D


wji xi(1)(5.62)

zj= tanh(aj )(5.63)

yk

=

aj

=

i=0
M

j =0

(2)

wkj zj .

(5.64)

246

5. NEURAL NETWORKS
Next we compute the δ’s for each output unit using
δ k = yk − tk .

(5.65)

Then we backpropagate these to obtain δs for the hidden units using
δj = (1 − zj2 )K

wkj δk .

(5.66)

k=1

Finally, the derivatives with respect to the ﬁrst-layer and second-layer weights are
given by∂En∂En
= δj xi ,= δ k zj .
(5.67)
(1)
(2)
∂wji
∂wkj

5.3.3 Efﬁciency of backpropagation
One of the most important aspects of backpropagation is its computational efﬁciency. To understand this, let us examine how the number of computer operations
required to evaluate the derivatives of the error function scales with the total number
W of weights and biases in the network. A single evaluation of the error function(for a given input pattern) would require O(W ) operations, for sufﬁciently large W .
This follows from the fact that, except for a network with very sparse connections,the number of weights is typically much greater than the number of units, and so thebulk of the computational effort in forward propagation is concerned with evaluating the sums in (5.48), with the evaluation of the activation functions representing asmall overhead. Each term in the sum in (5.48) requires one multiplication and one
addition, leading to an overall computational cost that is O(W ).
An alternative approach to backpropagation for computing the derivatives of theerror function is to use ﬁnite differences. This can be done by perturbing each weightin turn, and approximating the derivatives by the expression
En (wji + ) − En (wji )∂En=+ O()∂wji

(5.68)

where 1. In a software simulation, the accuracy of the approximation to thederivatives can be improved by making  smaller, until numerical roundoff problems
arise. The accuracy of the ﬁnite differences method can be improved signiﬁcantly
by using symmetrical central differences of the form
En (wji + ) − En (wji − )∂En=
+ O(2 ).
∂wji
2Exercise 5.14

(5.69)In this case, the O() corrections cancel, as can be veriﬁed by Taylor expansion on
the right-hand side of (5.69), and so the residual corrections are O(2 ). The numberof computational steps is, however, roughly doubled compared with (5.68).
The main problem with numerical differentiation is that the highly desirable
O(W ) scaling has been lost. Each forward propagation requires O(W ) steps, and5.3. Error Backpropagation
Figure 5.8

Illustration of a modular patternrecognition system in which theJacobian matrix can be used u
to backpropagate error signals
from the outputs through to earlier modules in the system.

247

v
y

x

w

z

there are W weights in the network each of which must be perturbed individually, sothat the overall scaling is O(W 2 ).
However, numerical differentiation plays an important role in practice, because acomparison of the derivatives calculated by backpropagation with those obtained using central differences provides a powerful check on the correctness of any software
implementation of the backpropagation algorithm. When training networks in practice, derivatives should be evaluated using backpropagation, because this gives thegreatest accuracy and numerical efﬁciency. However, the results should be comparedwith numerical differentiation using (5.69) for some test cases in order to check thecorrectness of the implementation.

5.3.4 The Jacobian matrix
We have seen how the derivatives of an error function with respect to the weights
can be obtained by the propagation of errors backwards through the network. The
technique of backpropagation can also be applied to the calculation of other derivatives. Here we consider the evaluation of the Jacobian matrix, whose elements are
given by the derivatives of the network outputs with respect to the inputs
Jki ≡

∂yk
∂xi

(5.70)

where each such derivative is evaluated with all other inputs held ﬁxed. Jacobianmatrices play a useful role in systems built from a number of distinct modules, as
illustrated in Figure 5.8. Each module can comprise a ﬁxed or adaptive function,
which can be linear or nonlinear, so long as it is differentiable. Suppose we wish
to minimize an error function E with respect to the parameter w in Figure 5.8. The
derivative of the error function is given by ∂E ∂yk ∂zj∂E=∂w∂yk ∂zj ∂w(5.71)

k,jin which the Jacobian matrix for the red module in Figure 5.8 appears in the middle
term.
Because the Jacobian matrix provides a measure of the local sensitivity of theoutputs to changes in each of the input variables, it also allows any known errors ∆xi

248

5. NEURAL NETWORKS
associated with the inputs to be propagated through the trained network in order toestimate their contribution ∆yk to the errors at the outputs, through the relation
 ∂yk
∆xi
(5.72)∆yk ∂xiiwhich is valid provided the |∆xi | are small. In general, the network mapping represented by a trained neural network will be nonlinear, and so the elements of theJacobian matrix will not be constants but will depend on the particular input vector
used. Thus (5.72) is valid only for small perturbations of the inputs, and the Jacobian
itself must be re-evaluated for each new input vector.
The Jacobian matrix can be evaluated using a backpropagation procedure that issimilar to the one derived earlier for evaluating the derivatives of an error function
with respect to the weights. We start by writing the element Jki in the form ∂yk ∂aj
∂yk
=
Jki =
∂xi
∂aj ∂xi
j

=


j

wji

∂yk
∂aj

(5.73)

where we have made use of (5.48). The sum in (5.73) runs over all units j to which
the input unit i sends connections (for example, over all units in the ﬁrst hidden
layer in the layered topology considered earlier). We now write down a recursive
backpropagation formula to determine the derivatives ∂yk /∂aj ∂yk ∂al∂yk=∂aj
∂al ∂ajl∂yk= h (aj )wlj(5.74)∂all

where the sum runs over all units l to which unit j sends connections (corresponding
to the ﬁrst index of wlj ). Again, we have made use of (5.48) and (5.49). This
backpropagation starts at the output units for which the required derivatives can befound directly from the functional form of the output-unit activation function. For
instance, if we have individual sigmoidal activation functions at each output unit,then∂yk= δkj σ  (aj )(5.75)∂aj
whereas for softmax outputs we have∂yk
= δkj yk − yk yj .
∂aj

(5.76)

We can summarize the procedure for evaluating the Jacobian matrix as follows.
Apply the input vector corresponding to the point in input space at which the Jacobian matrix is to be found, and forward propagate in the usual way to obtain the5.4. The Hessian Matrix

Exercise 5.15

249

activations of all of the hidden and output units in the network. Next, for each row
k of the Jacobian matrix, corresponding to the output unit k, backpropagate using
the recursive relation (5.74), starting with (5.75) or (5.76), for all of the hidden units
in the network. Finally, use (5.73) to do the backpropagation to the inputs. The
Jacobian can also be evaluated using an alternative forward propagation formalism,
which can be derived in an analogous way to the backpropagation approach given
here.
Again, the implementation of such algorithms can be checked by using numerical differentiation in the form∂ykyk (xi + ) − yk (xi − )+ O(2 )=∂xi
2

(5.77)

which involves 2D forward propagations for a network having D inputs.

5.4. The Hessian Matrix
We have shown how the technique of backpropagation can be used to obtain the ﬁrst
derivatives of an error function with respect to the weights in the network. Backpropagation can also be used to evaluate the second derivatives of the error, given
by∂2E.
(5.78)∂wji ∂wlk
Note that it is sometimes convenient to consider all of the weight and bias parameters
as elements wi of a single vector, denoted w, in which case the second derivatives
form the elements Hij of the Hessian matrix H, where i, j ∈ {1, . . . , W } and W is
the total number of weights and biases. The Hessian plays an important role in many
aspects of neural computing, including the following:
1. Several nonlinear optimization algorithms used for training neural networks
are based on considerations of the second-order properties of the error surface,which are controlled by the Hessian matrix (Bishop and Nabney, 2008).
2. The Hessian forms the basis of a fast procedure for re-training a feed-forward
network following a small change in the training data (Bishop, 1991).
3. The inverse of the Hessian has been used to identify the least signiﬁcant weights
in a network as part of network ‘pruning’ algorithms (Le Cun et al., 1990).
4. The Hessian plays a central role in the Laplace approximation for a Bayesianneural network (see Section 5.7). Its inverse is used to determine the predictive distribution for a trained network, its eigenvalues determine the values ofhyperparameters, and its determinant is used to evaluate the model evidence.
Various approximation schemes have been used to evaluate the Hessian matrix
for a neural network. However, the Hessian can also be calculated exactly using an
extension of the backpropagation technique.

250

5. NEURAL NETWORKS
An important consideration for many applications of the Hessian is the efﬁciency
with which it can be evaluated. If there are W parameters (weights and biases) in the
network, then the Hessian matrix has dimensions W × W and so the computational
effort needed to evaluate the Hessian will scale like O(W 2 ) for each pattern in thedata set. As we shall see, there are efﬁcient methods for evaluating the Hessian
whose scaling is indeed O(W 2 ).

5.4.1 Diagonal approximation
Some of the applications for the Hessian matrix discussed above require theinverse of the Hessian, rather than the Hessian itself. For this reason, there has
been some interest in using a diagonal approximation to the Hessian, in other words
one that simply replaces the off-diagonal elements with zeros, because its inverse istrivial to evaluate. Again, we shall consider an error function
 that consists of a sumof terms, one for each pattern in the data set, so that E = n En . The Hessian canthen be obtained by considering one pattern at a time, and then summing the resultsover all patterns. From (5.48), the diagonal elements of the Hessian, for pattern n,
can be written∂ 2 En 2∂ 2 En=z .
(5.79)
2∂wji∂a2j iUsing (5.48) and (5.49), the second derivatives on the right-hand side of (5.79) can
be found recursively using the chain rule of differential calculus to give a backpropagation equation of the form
∂ 2 En∂E n∂ 2 En

2

j
=
h
(a
)
w
w
+
h
(a
)
w
.
j
kj
k
j
kj
∂a2j
∂ak ∂ak
∂ak

k

k

(5.80)k

If we now neglect off-diagonal elements in the second-derivative terms, we obtain(Becker and Le Cun, 1989; Le Cun et al., 1990)
2


∂En
∂ 2 En22 ∂ En

=
h
(a
)
w
+
h
(a
)
wkj
.
j
j
kj
2
2
∂aj
∂ak
∂ak

k

(5.81)k

Note that the number of computational steps required to evaluate this approximationis O(W ), where W is the total number of weight and bias parameters in the network,
compared with O(W 2 ) for the full Hessian.
Ricotti et al. (1988) also used the diagonal approximation to the Hessian, butthey retained all terms in the evaluation of ∂ 2 En /∂a2j and so obtained exact expressions for the diagonal terms. Note that this no longer has O(W ) scaling. The major
problem with diagonal approximations, however, is that in practice the Hessian is
typically found to be strongly nondiagonal, and so these approximations, which are
driven mainly be computational convenience, must be treated with care.

5.4. The Hessian Matrix

251

5.4.2 Outer product approximation
When neural networks are applied to regression problems, it is common to use
a sum-of-squares error function of the form1(yn − tn )22
NE=(5.82)

n=1

Exercise 5.16

where we have considered the case of a single output in order to keep the notation
simple (the extension to several outputs is straightforward). We can then write theHessian matrix in the form
H = ∇∇E =N∇yn ∇yn +n=1

Exercise 5.17N(yn − tn )∇∇yn .

(5.83)

n=1

If the network has been trained on the data set, and its outputs yn happen to be very
close to the target values tn , then the second term in (5.83) will be small and canbe neglected. More generally, however, it may be appropriate to neglect this termby the following argument. Recall from Section 1.5.5 that the optimal function thatminimizes a sum-of-squares loss is the conditional average of the target data. The
quantity (yn − tn ) is then a random variable with zero mean. If we assume that its
value is uncorrelated with the value of the second derivative term on the right-handside of (5.83), then the whole term will average to zero in the summation over n.
By neglecting the second term in (5.83), we arrive at the Levenberg–Marquardt
approximation or outer product approximation (because the Hessian matrix is built
up from a sum of outer products of vectors), given by
H

N


bn bT
n

(5.84)

n=1

Exercise 5.19

where bn = ∇yn = ∇an because the activation function for the output units is
simply the identity. Evaluation of the outer product approximation for the Hessian
is straightforward as it only involves ﬁrst derivatives of the error function, which
can be evaluated efﬁciently in O(W ) steps using standard backpropagation. The
elements of the matrix can then be found in O(W 2 ) steps by simple multiplication.
It is important to emphasize that this approximation is only likely to be valid for a
network that has been trained appropriately, and that for a general network mapping
the second derivative terms on the right-hand side of (5.83) will typically not be
negligible.
In the case of the cross-entropy error function for a network with logistic sigmoid
output-unit activation functions, the corresponding approximation is given byHNyn (1 − yn )bn bT
n.

(5.85)

n=1

Exercise 5.20

An analogous result can be obtained for multiclass networks having softmax outputunit activation functions.

252

5. NEURAL NETWORKS

5.4.3 Inverse Hessian
We can use the outer-product approximation to develop a computationally efﬁcient procedure for approximating the inverse of the Hessian (Hassibi and Stork,
1993). First we write the outer-product approximation in matrix notation as
HN =

N


bn bT
n

(5.86)

n=1

where bn ≡ ∇w an is the contribution to the gradient of the output unit activation
arising from data point n. We now derive a sequential procedure for building up the
Hessian by including data points one at a time. Suppose we have already obtained
the inverse Hessian using the ﬁrst L data points. By separating off the contribution
from data point L + 1, we obtain
HL+1 = HL + bL+1 bT
L+1 .

(5.87)In order to evaluate the inverse of the Hessian, we now consider the matrix identity(M−1 v) vT M−1(5.88)
1 + vT M−1 v
where I is the unit matrix, which is simply a special case of the Woodbury identity(C.7). If we now identify HL with M and bL+1 with v, we obtainM + vvT−1= M−1 −

1
−1H−
L+1 = HL −Exercise 5.21

1
−1
T
H−
L bL+1 bL+1 HL−1
1 + bT
L+1 HL bL+1.

(5.89)In this way, data points are sequentially absorbed until L+1 = N and the whole data
set has been processed. This result therefore represents a procedure for evaluating
the inverse of the Hessian using a single pass through the data set. The initial matrix
H0 is chosen to be αI, where α is a small quantity, so that the algorithm actuallyﬁnds the inverse of H + αI. The results are not particularly sensitive to the precise
value of α. Extension of this algorithm to networks having more than one output isstraightforward.
We note here that the Hessian matrix can sometimes be calculated indirectly as
part of the network training algorithm. In particular, quasi-Newton nonlinear optimization algorithms gradually build up an approximation to the inverse of the Hessian during training. Such algorithms are discussed in detail in Bishop and Nabney(2008).

5.4.4 Finite differences
As in the case of the ﬁrst derivatives of the error function, we can ﬁnd the second
derivatives by using ﬁnite differences, with accuracy limited by numerical precision.
If we perturb each possible pair of weights in turn, we obtain1∂2E= 2 {E(wji + , wlk + ) − E(wji + , wlk − )∂wji ∂wlk
4
−E(wji − , wlk + ) + E(wji − , wlk − )} + O(2 ).

(5.90)

5.4. The Hessian Matrix

253

Again, by using a symmetrical central differences formulation, we ensure that theresidual errors are O(2 ) rather than O(). Because there are W 2 elements in the
Hessian matrix, and because the evaluation of each element requires four forward
propagations each needing O(W ) operations (per pattern), we see that this approach
will require O(W 3 ) operations to evaluate the complete Hessian. It therefore haspoor scaling properties, although in practice it is very useful as a check on the software implementation of backpropagation methods.
A more efﬁcient version of numerical differentiation can be found by applying
central differences to the ﬁrst derivatives of the error function, which are themselvescalculated using backpropagation. This gives∂2E
1=∂wji ∂wlk
2

∂E∂E(wlk + ) −(wlk − )
∂wji
∂wji

+ O(2 ).

(5.91)

Because there are now only W weights to be perturbed, and because the gradientscan be evaluated in O(W ) steps, we see that this method gives the Hessian in O(W 2 )operations.

5.4.5 Exact evaluation of the Hessian

Exercise 5.22

So far, we have considered various approximation schemes for evaluating the
Hessian matrix or its inverse. The Hessian can also be evaluated exactly, for a network of arbitrary feed-forward topology, using extension of the technique of backpropagation used to evaluate ﬁrst derivatives, which shares many of its desirable
features including computational efﬁciency (Bishop, 1991; Bishop, 1992). It can beapplied to any differentiable error function that can be expressed as a function of
the network outputs and to networks having arbitrary differentiable activation functions. The number of computational steps needed to evaluate the Hessian scales
like O(W 2 ). Similar algorithms have also been considered by Buntine and Weigend(1993).
Here we consider the speciﬁc case of a network having two layers of weights,
for which the required equations are easily derived. We shall use indices i and i
to denote inputs, indices j and j  to denoted hidden units, and indices k and k  todenote outputs. We ﬁrst deﬁne
δk =

∂En
,
∂ak

Mkk ≡∂ 2 En∂ak ∂ak

(5.92)

where En is the contribution to the error from data point n. The Hessian matrix for
this network can then be considered in three separate blocks as follows.
1. Both weights in the second layer:∂ 2 En(2)(2)∂wkj ∂wk j = zj zj  Mkk .

(5.93)

254

5. NEURAL NETWORKS
2. Both weights in the ﬁrst layer:∂ 2 En(1)= xi xi h (aj  )Ijj (1)∂wji ∂wj  i

+xi xi h (aj  )h (aj )




k

(2)

wkj  δk

k
(2)

(2)

wk j  wkj Mkk .

(5.94)k

3. One weight in each layer:∂ 2 En(1)

(2)∂wji ∂wkj Exercise 5.23

= xi h (aj  )δk Ijj  + zj




(2)

wk j  Hkk.

(5.95)k

Here Ijj  is the j, j  element of the identity matrix. If one or both of the weights isa bias term, then the corresponding expressions are obtained simply by setting theappropriate activation(s) to 1. Inclusion of skip-layer connections is straightforward.

5.4.6 Fast multiplication by the Hessian
For many applications of the Hessian, the quantity of interest is not the Hessian
matrix H itself but the product of H with some vector v. We have seen that theevaluation of the Hessian takes O(W 2 ) operations, and it also requires storage that isO(W 2 ). The vector vT H that we wish to calculate, however, has only W elements,
so instead of computing the Hessian as an intermediate step, we can instead try toﬁnd an efﬁcient approach to evaluating vT H directly in a way that requires onlyO(W ) operations.
To do this, we ﬁrst note thatvT H = vT ∇(∇E)(5.96)

where ∇ denotes the gradient operator in weight space. We can then write down
the standard forward-propagation and backpropagation equations for the evaluationof ∇E and apply (5.96) to these equations to give a set of forward-propagation andbackpropagation equations for the evaluation of vT H (Møller, 1993; Pearlmutter,1994). This corresponds to acting on the original forward-propagation and backpropagation equations with a differential operator vT ∇. Pearlmutter (1994) used thenotation R{·} to denote the operator vT ∇, and we shall follow this convention. The
analysis is straightforward and makes use of the usual rules of differential calculus,
together with the result
R{w} = v.
(5.97)The technique is best illustrated with a simple example, and again we choose a
two-layer network of the form shown in Figure 5.1, with linear output units and asum-of-squares error function. As before, we consider the contribution to the error
function from one pattern in the data set. The required vector is then obtained as5.4. The Hessian Matrix

255

usual by summing over the contributions from each of the patterns separately. For
the two-layer network, the forward-propagation equations are given byaj =wji xi(5.98)izjyk= h(aj )
=
wkj zj .

(5.99)(5.100)j

We now act on these equations using the R{·} operator to obtain a set of forward
propagation equations in the form
R{aj } =vji xi(5.101)iR{zj } = h (aj )R{aj }
wkj R{zj } +vkj zjR{yk } =j(5.102)(5.103)jwhere vji is the element of the vector v that corresponds to the weight wji . Quantities of the form R{zj }, R{aj } and R{yk } are to be regarded as new variables
whose values are found using the above equations.
Because we are considering a sum-of-squares error function, we have the following standard backpropagation expressions:δkδj= yk − tk= h (aj )wkj δk .

(5.104)(5.105)k

Again, we act on these equations with the R{·} operator to obtain a set of backpropagation equations in the formR{δk } = R{yk }R{δj } = h (aj )R{aj }+ h (aj )(5.106)wkj δkkvkj δk + h (aj )k

wkj R{δk }.

(5.107)k

Finally, we have the usual equations for the ﬁrst derivatives of the error
∂E
∂wkj
∂E
∂wji

= δ k zj

(5.108)= δj xi(5.109)256

5. NEURAL NETWORKS
and acting on these with the R{·} operator, we obtain expressions for the elementsof the vector vT H
R

∂E
∂wkj

= R{δk }zj + δk R{zj }(5.110)

R

∂E
∂wji

= xi R{δj }.

(5.111)

The implementation of this algorithm involves the introduction of additionalvariables R{aj }, R{zj } and R{δj } for the hidden units and R{δk } and R{yk }for the output units. For each input pattern, the values of these quantities can befound using the above results, and the elements of vT H are then given by (5.110)
and (5.111). An elegant aspect of this technique is that the equations for evaluatingvT H mirror closely those for standard forward and backward propagation, and so theextension of existing software to compute this product is typically straightforward.
If desired, the technique can be used to evaluate the full Hessian matrix bychoosing the vector v to be given successively by a series of unit vectors of theform (0, 0, . . . , 1, . . . , 0) each of which picks out one column of the Hessian. This
leads to a formalism that is analytically equivalent to the backpropagation procedureof Bishop (1992), as described in Section 5.4.5, though with some loss of efﬁciency
due to redundant calculations.

5.5. Regularization in Neural Networks
The number of input and outputs units in a neural network is generally determinedby the dimensionality of the data set, whereas the number M of hidden units is a freeparameter that can be adjusted to give the best predictive performance. Note that M
controls the number of parameters (weights and biases) in the network, and so wemight expect that in a maximum likelihood setting there will be an optimum valueof M that gives the best generalization performance, corresponding to the optimum
balance between under-ﬁtting and over-ﬁtting. Figure 5.9 shows an example of theeffect of different values of M for the sinusoidal regression problem.
The generalization error, however, is not a simple function of M due to thepresence of local minima in the error function, as illustrated in Figure 5.10. Here
we see the effect of choosing multiple random initializations for the weight vector
for a range of values of M . The overall best validation set performance in thiscase occurred for a particular solution having M = 8. In practice, one approach tochoosing M is in fact to plot a graph of the kind shown in Figure 5.10 and then tochoose the speciﬁc solution having the smallest validation set error.
There are, however, other ways to control the complexity of a neural networkmodel in order to avoid over-ﬁtting. From our discussion of polynomial curve ﬁtting
in Chapter 1, we see that an alternative approach is to choose a relatively large value
for M and then to control complexity by the addition of a regularization term to theerror function. The simplest regularizer is the quadratic, giving a regularized error

257

5.5. Regularization in Neural Networks

M =1

1

M =3

1

0

0

0

−1

−1

−1

0

1

0

M = 101

1

01

Figure 5.9 Examples of two-layer networks trained on 10 data points drawn from the sinusoidal data set. The
graphs show the result of ﬁtting networks having M = 1, 3 and 10 hidden units, respectively, by minimizing asum-of-squares error function using a scaled conjugate-gradient algorithm.

of the form

λ

E(w)
= E(w) + wT w.
(5.112)2This regularizer is also known as weight decay and has been discussed at lengthin Chapter 3. The effective model complexity is then determined by the choice ofthe regularization coefﬁcient λ. As we have seen previously, this regularizer can beinterpreted as the negative logarithm of a zero-mean Gaussian prior distribution over
the weight vector w.

5.5.1 Consistent Gaussian priors
One of the limitations of simple weight decay in the form (5.112) is that isinconsistent with certain scaling properties of network mappings. To illustrate this,consider a multilayer perceptron network having two layers of weights and linear
output units, which performs a mapping from a set of input variables {xi } to a setof output variables {yk }. The activations of the hidden units in the ﬁrst hidden layer
Figure 5.10

Plot of the sum-of-squares test-set
error for the polynomial data set versus the number of hidden units in the
network, with 30 random starts foreach network size, showing the effect of local minima. For each newstart, the weight vector was initialized by sampling from an isotropic
Gaussian distribution having a meanof zero and a variance of 10.

160
140
120
100
80
60
0

2

4

6

8

10

258

5. NEURAL NETWORKS
take the form
zj = h
wji xi + wj 0

(5.113)

while the activations of the output units are given by
wkj zj + wk0 .
yk =(5.114)ij

Suppose we perform a linear transformation of the input data of the form
xi = axi + b.
xi → 

Exercise 5.24

(5.115)

Then we can arrange for the mapping performed by the network to be unchanged
by making a corresponding linear transformation of the weights and biases from the
inputs to the units in the hidden layer of the formwji → w
 ji =1wjia

 j 0 = wj 0 −wj 0 → w(5.116)b
wji .
a(5.117)i

Similarly, a linear transformation of the output variables of the network of the form
yk = cyk + d
yk → 

(5.118)can be achieved by making a transformation of the second-layer weights and biasesusing kj = cwkj
wkj → w
 k0 = cwk0 + d.
wk0 → w

(5.119)(5.120)If we train one network using the original data and one network using data for which
the input and/or target variables are transformed by one of the above linear transformations, then consistency requires that we should obtain equivalent networks thatdiffer only by the linear transformation of the weights as given. Any regularizer
should be consistent with this property, otherwise it arbitrarily favours one solution
over another, equivalent one. Clearly, simple weight decay (5.112), that treats allweights and biases on an equal footing, does not satisfy this property.
We therefore look for a regularizer which is invariant under the linear transformations (5.116), (5.117), (5.119) and (5.120). These require that the regularizer
should be invariant to re-scaling of the weights and to shifts of the biases. Such aregularizer is given byλ 1  2 λ2  2w +w(5.121)
2
2
w∈W1

w∈W2

where W1 denotes the set of weights in the ﬁrst layer, W2 denotes the set of weights
in the second layer, and biases are excluded from the summations. This regularizer

5.5. Regularization in Neural Networks

259

will remain unchanged under the weight transformations provided the regularization
parameters are re-scaled using λ1 → a1/2 λ1 and λ2 → c−1/2 λ2 .
The regularizer (5.121) corresponds to a prior of the form
α 1  2 α2  2w −w .
(5.122)p(w|α1 , α2 ) ∝ exp −
2
2
w∈W1

w∈W2

Note that priors of this form are improper (they cannot be normalized) because thebias parameters are unconstrained. The use of improper priors can lead to difﬁculties
in selecting regularization coefﬁcients and in model comparison within the Bayesianframework, because the corresponding evidence is zero. It is therefore common to
include separate priors for the biases (which then break shift invariance) having theirown hyperparameters. We can illustrate the effect of the resulting four hyperparameters by drawing samples from the prior and plotting the corresponding network
functions, as shown in Figure 5.11.
More generally, we can consider priors in which the weights are divided into
any number of groups Wk so that1p(w) ∝ exp −αk w2k(5.123)2k

wherew2k =

wj2 .

(5.124)j∈Wk

As a special case of this prior, if we choose the groups to correspond to the sets
of weights associated with each of the input units, and we optimize the marginal
likelihood with respect to the corresponding parameters αk , we obtain automatic
relevance determination as discussed in Section 7.2.2.

5.5.2 Early stopping
An alternative to regularization as a way of controlling the effective complexityof a network is the procedure of early stopping. The training of nonlinear networkmodels corresponds to an iterative reduction of the error function deﬁned with respect to a set of training data. For many of the optimization algorithms used for
network training, such as conjugate gradients, the error is a nonincreasing functionof the iteration index. However, the error measured with respect to independent data,
generally called a validation set, often shows a decrease at ﬁrst, followed by an increase as the network starts to over-ﬁt. Training can therefore be stopped at the pointof smallest error with respect to the validation data set, as indicated in Figure 5.12,
in order to obtain a network having good generalization performance.
The behaviour of the network in this case is sometimes explained qualitatively
in terms of the effective number of degrees of freedom in the network, in which thisnumber starts out small and then to grows during the training process, corresponding
to a steady increase in the effective complexity of the model. Halting training before

260

5. NEURAL NETWORKS

4

b
w
b
αw
1 = 1, α 1 = 1, α 2 = 1, α 2 = 1

40

2

20

0

0

−2

−20

−4

−40

−6
−1
5

−0.5

0

0.5

1

b
w
b
αw
1 = 1000, α 1 = 100, α 2 = 1, α 2 = 1

−60
−1
5

0

0

−5

−5

−10
−1

−0.5

0

0.5

1

−10
−1

b
w
b
αw
1 = 1, α 1 = 1, α 2 = 10, α 2 = 1

−0.5

0

0.5

1

b
w
b
αw
1 = 1000, α 1 = 1000, α 2 = 1, α 2 = 1

−0.5

0

0.5

1

Figure 5.11 Illustration of the effect of the hyperparameters governing the prior distribution over weights andbiases in a two-layer network having a single input, a single linear output, and 12 hidden units having ‘tanh’activation functions. The priors are governed by four hyperparameters α1b , α1w , α2b , and α2w , which represent
the precisions of the Gaussian distributions of the ﬁrst-layer biases, ﬁrst-layer weights, second-layer biases, andsecond-layer weights, respectively. We see that the parameter α2w governs the vertical scale of functions (notethe different vertical axis ranges on the top two diagrams), α1w governs the horizontal scale of variations in thefunction values, and α1b governs the horizontal range over which variations occur. The parameter α2b , whose
effect is not illustrated here, governs the range of vertical offsets of the functions.

Exercise 5.25

a minimum of the training error has been reached then represents a way of limiting
the effective network complexity.
In the case of a quadratic error function, we can verify this insight, and showthat early stopping should exhibit similar behaviour to regularization using a simple weight-decay term. This can be understood from Figure 5.13, in which the axes
in weight space have been rotated to be parallel to the eigenvectors of the Hessian
matrix. If, in the absence of weight decay, the weight vector starts at the origin andproceeds during training along a path that follows the local negative gradient vector, then the weight vector will move initially parallel to the w2 axis through a point
corresponding roughly to w
 and then move towards the minimum of the error function wML . This follows from the shape of the error surface and the widely differing
 is therefore similar to weight
eigenvalues of the Hessian. Stopping at a point near w
decay. The relationship between early stopping and weight decay can be made quantitative, thereby showing that the quantity τ η (where τ is the iteration index, and η
is the learning rate parameter) plays the role of the reciprocal of the regularization

5.5. Regularization in Neural Networks

261

0.45

0.25
0.4
0.2

0.15

0

10

20

30

40

50

0.35

0

10

20

30

40

50

Figure 5.12 An illustration of the behaviour of training set error (left) and validation set error (right) during atypical training session, as a function of the iteration step, for the sinusoidal data set. The goal of achieving
the best generalization performance suggests that training should be stopped at the point shown by the verticaldashed lines, corresponding to the minimum of the validation set error.

parameter λ. The effective number of parameters in the network therefore grows
during the course of training.

5.5.3 Invariances
In many applications of pattern recognition, it is known that predictions should
be unchanged, or invariant, under one or more transformations of the input variables. For example, in the classiﬁcation of objects in two-dimensional images, suchas handwritten digits, a particular object should be assigned the same classiﬁcation
irrespective of its position within the image (translation invariance) or of its size
(scale invariance). Such transformations produce signiﬁcant changes in the raw
data, expressed in terms of the intensities at each of the pixels in the image, andyet should give rise to the same output from the classiﬁcation system. Similarly
in speech recognition, small levels of nonlinear warping along the time axis, which
preserve temporal ordering, should not change the interpretation of the signal.
If sufﬁciently large numbers of training patterns are available, then an adaptivemodel such as a neural network can learn the invariance, at least approximately. This
involves including within the training set a sufﬁciently large number of examples ofthe effects of the various transformations. Thus, for translation invariance in an image, the training set should include examples of objects at many different positions.
This approach may be impractical, however, if the number of training examples
is limited, or if there are several invariants (because the number of combinations oftransformations grows exponentially with the number of such transformations). We
therefore seek alternative approaches for encouraging an adaptive model to exhibit
the required invariances. These can broadly be divided into four categories:
1. The training set is augmented using replicas of the training patterns, transformed according to the desired invariances. For instance, in our digit recognition example, we could make multiple copies of each example in which the262

5. NEURAL NETWORKS
Figure 5.13

A schematic illustration of why
early stopping can give similar
results to weight decay in thecase of a quadratic error function. The ellipse shows a contour of constant error, and wML
denotes the minimum of the error function. If the weight vector
starts at the origin and moves according to the local negative gradient direction, then it will follow
the path shown by the curve. By
stopping training early, a weight
e is found that is qualvector w
itatively similar to that obtainedwith a simple weight-decay regularizer and training to the minimum of the regularized error, as
can be seen by comparing with
Figure 3.15.

w2

wML

w

w1

digit is shifted to a different position in each image.
2. A regularization term is added to the error function that penalizes changes in
the model output when the input is transformed. This leads to the technique oftangent propagation, discussed in Section 5.5.4.
3. Invariance is built into the pre-processing by extracting features that are invariant under the required transformations. Any subsequent regression or classiﬁcation system that uses such features as inputs will necessarily also respectthese invariances.
4. The ﬁnal option is to build the invariance properties into the structure of a neural network (or into the deﬁnition of a kernel function in the case of techniquessuch as the relevance vector machine). One way to achieve this is through theuse of local receptive ﬁelds and shared weights, as discussed in the context ofconvolutional neural networks in Section 5.5.6.
Approach 1 is often relatively easy to implement and can be used to encourage complex invariances such as those illustrated in Figure 5.14. For sequential training
algorithms, this can be done by transforming each input pattern before it is presented
to the model so that, if the patterns are being recycled, a different transformation(drawn from an appropriate distribution) is added each time. For batch methods, asimilar effect can be achieved by replicating each data point a number of times andtransforming each copy independently. The use of such augmented data can lead tosigniﬁcant improvements in generalization (Simard et al., 2003), although it can also
be computationally costly.
Approach 2 leaves the data set unchanged but modiﬁes the error function throughthe addition of a regularizer. In Section 5.5.5, we shall show that this approach is
closely related to approach 2.

263

5.5. Regularization in Neural Networks

Figure 5.14 Illustration of the synthetic warping of a handwritten digit. The original image is shown on theleft. On the right, the top row shows three examples of warped digits, with the corresponding displacementﬁelds shown on the bottom row. These displacement ﬁelds are generated by sampling random displacements∆x, ∆y ∈ (0, 1) at each pixel and then smoothing by convolution with Gaussians of width 0.01, 30 and 60
respectively.

One advantage of approach 3 is that it can correctly extrapolate well beyond therange of transformations included in the training set. However, it can be difﬁcult
to ﬁnd hand-crafted features with the required invariances that do not also discard
information that can be useful for discrimination.

5.5.4 Tangent propagation
We can use regularization to encourage models to be invariant to transformationsof the input through the technique of tangent propagation (Simard et al., 1992).
Consider the effect of a transformation on a particular input vector xn . Provided thetransformation is continuous (such as translation or rotation, but not mirror reﬂection
for instance), then the transformed pattern will sweep out a manifold M within the
D-dimensional input space. This is illustrated in Figure 5.15, for the case of D =2 for simplicity. Suppose the transformation is governed by a single parameter ξ(which might be rotation angle for instance). Then the subspace M swept out by xn
Figure 5.15

Illustration of a two-dimensional input space x2
showing the effect of a continuous transformation on a particular input vector xn . A onedimensional transformation, parameterized by
the continuous variable ξ, applied to xn causes
it to sweep out a one-dimensional manifold M.
Locally, the effect of the transformation can beapproximated by the tangent vector τ n .

M

τn
xn

ξ

x1

264

5. NEURAL NETWORKS
will be one-dimensional, and will be parameterized by ξ. Let the vector that resultsfrom acting on xn by this transformation be denoted by s(xn , ξ), which is deﬁned
so that s(x, 0) = x. Then the tangent to the curve M is given by the directional
derivative τ = ∂s/∂ξ, and the tangent vector at the point xn is given by∂s(xn , ξ) τn =.
(5.125)∂ξξ =0Under a transformation of the input vector, the network output vector will, in general,change. The derivative of output k with respect to ξ is given by


D
D



∂yk ∂xi ∂yk ==Jki τi(5.126)∂ξ ξ=0∂xi ∂ξ 
i=1

ξ =0

i=1

where Jki is the (k, i) element of the Jacobian matrix J, as discussed in Section 5.3.4.
The result (5.126) can be used to modify the standard error function, so as to encourage local invariance in the neighbourhood of the data points, by the addition to theoriginal error function E of a regularization function Ω to give a total error functionof the form
 = E + λΩE
(5.127)
where λ is a regularization coefﬁcient and
1 
Ω=
2
n

Exercise 5.26

k



D
2
 2
∂ynk 
1  
=Jnki τni .
∂ξ ξ=02nk(5.128)i=1

The regularization function will be zero when the network mapping function is invariant under the transformation in the neighbourhood of each pattern vector, andthe value of the parameter λ determines the balance between ﬁtting the training data
and learning the invariance property.
In a practical implementation, the tangent vector τ n can be approximated using ﬁnite differences, by subtracting the original vector xn from the corresponding
vector after transformation using a small value of ξ, and then dividing by ξ. This is
illustrated in Figure 5.16.
The regularization function depends on the network weights through the Jacobian J. A backpropagation formalism for computing the derivatives of the regularizer with respect to the network weights is easily obtained by extension of thetechniques introduced in Section 5.3.
If the transformation is governed by L parameters (e.g., L = 3 for the case oftranslations combined with in-plane rotations in a two-dimensional image), then the
manifold M will have dimensionality L, and the corresponding regularizer is givenby the sum of terms of the form (5.128), one for each transformation. If severaltransformations are considered at the same time, and the network mapping is madeinvariant to each separately, then it will be (locally) invariant to combinations of thetransformations (Simard et al., 1992).

5.5. Regularization in Neural Networks
Figure 5.16 Illustration showing(a) the original image x of a handwritten digit, (b) the tangent vector
τ corresponding to an inﬁnitesimalclockwise rotation, (c) the result of
adding a small contribution from thetangent vector to the original image
giving x + τ with = 15 degrees,and (d) the true image rotated forcomparison.

(a)

(b)

(c)

(d)

265

A related technique, called tangent distance, can be used to build invarianceproperties into distance-based methods such as nearest-neighbour classiﬁers (Simard
et al., 1993).

5.5.5 Training with transformed data
We have seen that one way to encourage invariance of a model to a set of transformations is to expand the training set using transformed versions of the original
input patterns. Here we show that this approach is closely related to the technique oftangent propagation (Bishop, 1995b; Leen, 1995).
As in Section 5.5.4, we shall consider a transformation governed by a single
parameter ξ and described by the function s(x, ξ), with s(x, 0) = x. We shallalso consider a sum-of-squares error function. The error function for untransformed
inputs can be written (in the inﬁnite data set limit) in the form1E={y(x) − t}2 p(t|x)p(x) dx dt(5.129)2
as discussed in Section 1.5.5. Here we have considered a network having a single
output, in order to keep the notation uncluttered. If we now consider an inﬁnite
number of copies of each data point, each of which is perturbed by the transformation

266

5. NEURAL NETWORKS
in which the parameter ξ is drawn from a distribution p(ξ), then the error function
deﬁned over this expanded data set can be written as

1

E=
{y(s(x, ξ)) − t}2 p(t|x)p(x)p(ξ) dx dt dξ.
(5.130)2
We now assume that the distribution p(ξ) has zero mean with small variance, so that
we are only considering small transformations of the original input vectors. We canthen expand the transformation function as a Taylor series in powers of ξ to give




ξ2 ∂2
∂

s(x, ξ)
+s(x, ξ)
+ O(ξ 3 )s(x, ξ) = s(x, 0) + ξ
2
∂ξ
2
∂ξ
ξ =0
ξ =0
1
= x + ξτ + ξ 2 τ  + O(ξ 3 )2
where τ  denotes the second derivative of s(x, ξ) with respect to ξ evaluated at ξ = 0.
This allows us to expand the model function to givey(s(x, ξ)) = y(x) + ξτ T ∇y(x) +.
ξ2 -  T(τ ) ∇y(x) + τ T ∇∇y(x)τ + O(ξ 3 ).
2

Substituting into the mean error function (5.130) and expanding, we then have

 = 1
E
{y(x) − t}2 p(t|x)p(x) dx dt2
{y(x) − t}τ T ∇y(x)p(t|x)p(x) dx dt+ E[ξ] 
1  T2{y(x) − t}(τ ) ∇y(x) + τ T ∇∇y(x)τ
+ E[ξ ]
2.
2T
+ τ ∇y(x)p(t|x)p(x) dx dt + O(ξ 3 ).
Because the distribution of transformations has zero mean we have E[ξ] = 0. Also,
we shall denote E[ξ 2 ] by λ. Omitting terms of O(ξ 3 ), the average error function thenbecomes = E + λΩE
(5.131)
where E is the original sum-of-squares error, and the regularization term Ω takes the
form
 

1  TΩ ={y(x) − E[t|x]}(τ ) ∇y(x) + τ T ∇∇y(x)τ
22+ τ T ∇y(x)p(x) dx(5.132)in which we have performed the integration over t.

5.5. Regularization in Neural Networks

267

We can further simplify this regularization term as follows. In Section 1.5.5 wesaw that the function that minimizes the sum-of-squares error is given by the conditional average E[t|x] of the target values t. From (5.131) we see that the regularized
error will equal the unregularized sum-of-squares plus terms which are O(ξ), and so
the network function that minimizes the total error will have the formy(x) = E[t|x] + O(ξ).

(5.133)Thus, to leading order in ξ, the ﬁrst term in the regularizer vanishes and we are left
with

1
2
Ω=
τ T ∇y(x) p(x) dx(5.134)
2

Exercise 5.27

which is equivalent to the tangent propagation regularizer (5.128).
If we consider the special case in which the transformation of the inputs simply
consists of the addition of random noise, so that x → x + ξ, then the regularizer
takes the form12
∇y(x) p(x) dx(5.135)Ω=
2which is known as Tikhonov regularization (Tikhonov and Arsenin, 1977; Bishop,
1995b). Derivatives of this regularizer with respect to the network weights can befound using an extended backpropagation algorithm (Bishop, 1993). We see that, forsmall noise amplitudes, Tikhonov regularization is related to the addition of random
noise to the inputs, which has been shown to improve generalization in appropriate
circumstances (Sietsma and Dow, 1991).

5.5.6 Convolutional networks
Another approach to creating models that are invariant to certain transformationof the inputs is to build the invariance properties into the structure of a neural network. This is the basis for the convolutional neural network (Le Cun et al., 1989;
LeCun et al., 1998), which has been widely applied to image data.
Consider the speciﬁc task of recognizing handwritten digits. Each input image
comprises a set of pixel intensity values, and the desired output is a posterior probability distribution over the ten digit classes. We know that the identity of the digit isinvariant under translations and scaling as well as (small) rotations. Furthermore, thenetwork must also exhibit invariance to more subtle transformations such as elastic
deformations of the kind illustrated in Figure 5.14. One simple approach would be totreat the image as the input to a fully connected network, such as the kind shown in
Figure 5.1. Given a sufﬁciently large training set, such a network could in principle
yield a good solution to this problem and would learn the appropriate invariances by
example.
However, this approach ignores a key property of images, which is that nearby
pixels are more strongly correlated than more distant pixels. Many of the modern
approaches to computer vision exploit this property by extracting local features that
depend only on small subregions of the image. Information from such features canthen be merged in later stages of processing in order to detect higher-order features268

5. NEURAL NETWORKS

Input image
Figure 5.17

Convolutional layer

Sub-sampling
layer

Diagram illustrating part of a convolutional neural network, showing a layer of convolutional units followed by a layer of subsampling units. Several successive pairs of suchlayers may be used.

and ultimately to yield information about the image as whole. Also, local featuresthat are useful in one region of the image are likely to be useful in other regions of
the image, for instance if the object of interest is translated.
These notions are incorporated into convolutional neural networks through threemechanisms: (i) local receptive ﬁelds, (ii) weight sharing, and (iii) subsampling. The
structure of a convolutional network is illustrated in Figure 5.17. In the convolutional
layer the units are organized into planes, each of which is called a feature map. Units
in a feature map each take inputs only from a small subregion of the image, and allof the units in a feature map are constrained to share the same weight values. For
instance, a feature map might consist of 100 units arranged in a 10 × 10 grid, witheach unit taking inputs from a 5 × 5 pixel patch of the image. The whole feature map
therefore has 25 adjustable weight parameters plus one adjustable bias parameter.
Input values from a patch are linearly combined using the weights and the bias, andthe result transformed by a sigmoidal nonlinearity using (5.1). If we think of the units
as feature detectors, then all of the units in a feature map detect the same pattern but
at different locations in the input image. Due to the weight sharing, the evaluationof the activations of these units is equivalent to a convolution of the image pixel
intensities with a ‘kernel’ comprising the weight parameters. If the input image isshifted, the activations of the feature map will be shifted by the same amount but willotherwise be unchanged. This provides the basis for the (approximate) invariance of5.5. Regularization in Neural Networks

Exercise 5.28

269

the network outputs to translations and distortions of the input image. Because wewill typically need to detect multiple features in order to build an effective model,
there will generally be multiple feature maps in the convolutional layer, each having
its own set of weight and bias parameters.
The outputs of the convolutional units form the inputs to the subsampling layerof the network. For each feature map in the convolutional layer, there is a plane ofunits in the subsampling layer and each unit takes inputs from a small receptive ﬁeld
in the corresponding feature map of the convolutional layer. These units performsubsampling. For instance, each subsampling unit might take inputs from a 2 × 2
unit region in the corresponding feature map and would compute the average of
those inputs, multiplied by an adaptive weight with the addition of an adaptive bias
parameter, and then transformed using a sigmoidal nonlinear activation function.
The receptive ﬁelds are chosen to be contiguous and nonoverlapping so that there
are half the number of rows and columns in the subsampling layer compared withthe convolutional layer. In this way, the response of a unit in the subsampling layer
will be relatively insensitive to small shifts of the image in the corresponding regionsof the input space.
In a practical architecture, there may be several pairs of convolutional and subsampling layers. At each stage there is a larger degree of invariance to input transformations compared to the previous layer. There may be several feature maps in agiven convolutional layer for each plane of units in the previous subsampling layer,
so that the gradual reduction in spatial resolution is then compensated by an increasing number of features. The ﬁnal layer of the network would typically be a fully
connected, fully adaptive layer, with a softmax output nonlinearity in the case ofmulticlass classiﬁcation.
The whole network can be trained by error minimization using backpropagation
to evaluate the gradient of the error function. This involves a slight modiﬁcationof the usual backpropagation algorithm to ensure that the shared-weight constraints
are satisﬁed. Due to the use of local receptive ﬁelds, the number of weights inthe network is smaller than if the network were fully connected. Furthermore, thenumber of independent parameters to be learned from the data is much smaller still,
due to the substantial numbers of constraints on the weights.

5.5.7 Soft weight sharing
One way to reduce the effective complexity of a network with a large number
of weights is to constrain weights within certain groups to be equal. This is thetechnique of weight sharing that was discussed in Section 5.5.6 as a way of building
translation invariance into networks used for image interpretation. It is only applicable, however, to particular problems in which the form of the constraints can be
speciﬁed in advance. Here we consider a form of soft weight sharing (Nowlan and
Hinton, 1992) in which the hard constraint of equal weights is replaced by a formof regularization in which groups of weights are encouraged to have similar values.
Furthermore, the division of weights into groups, the mean weight value for eachgroup, and the spread of values within the groups are all determined as part of thelearning process.

270

5. NEURAL NETWORKS

Section 2.3.9

Recall that the simple weight decay regularizer, given in (5.112), can be viewed
as the negative log of a Gaussian prior distribution over the weights. We can encourage the weight values to form several groups, rather than just one group, by considering instead a probability distribution that is a mixture of Gaussians. The centresand variances of the Gaussian components, as well as the mixing coefﬁcients, will beconsidered as adjustable parameters to be determined as part of the learning process.
Thus, we have a probability density of the formp(w) =p(wi )(5.136)i

wherep(wi ) =Mπj N (wi |µj , σj2 )(5.137)j =1

and πj are the mixing coefﬁcients. Taking the negative logarithm then leads to aregularization function of the form
M



2
ln
πj N (wi |µj , σj ) .
(5.138)Ω(w) = −ij =1

The total error function is then given byE(w)= E(w) + λΩ(w)(5.139)where λ is the regularization coefﬁcient. This error is minimized both with respect
to the weights wi and with respect to the parameters {πj , µj , σj } of the mixture
model. If the weights were constant, then the parameters of the mixture model could
be determined by using the EM algorithm discussed in Chapter 9. However, the distribution of weights is itself evolving during the learning process, and so to avoid numerical instability, a joint optimization is performed simultaneously over the weights
and the mixture-model parameters. This can be done using a standard optimization
algorithm such as conjugate gradients or quasi-Newton methods.
In order to minimize the total error function, it is necessary to be able to evaluate
its derivatives with respect to the various adjustable parameters. To do this it is convenient to regard the {πj } as prior probabilities and to introduce the corresponding
posterior probabilities which, following (2.192), are given by Bayes’ theorem in theform
πj N (w|µj , σj2 ).
(5.140)γj (w) =2k πk N (w|µk , σk )Exercise 5.29

The derivatives of the total error function with respect to the weights are then givenby
(wi − µj )
∂E
∂E
=
+λ
γj (wi ).
(5.141)
∂wi
∂wi
σj2
j

5.5. Regularization in Neural Networks

Exercise 5.30

271

The effect of the regularization term is therefore to pull each weight towards thecentre of the j th Gaussian, with a force proportional to the posterior probability ofthat Gaussian for the given weight. This is precisely the kind of effect that we are
seeking.
Derivatives of the error with respect to the centres of the Gaussians are alsoeasily computed to give(µi − wj )∂E
=λγj (wi )∂µj
σj2(5.142)i

Exercise 5.31

which has a simple intuitive interpretation, because it pushes µj towards an average of the weight values, weighted by the posterior probabilities that the respective
weight parameters were generated by component j. Similarly, the derivatives withrespect to the variances are given by(wi − µj )21∂E=λγj (wi )−(5.143)
∂σj
σj
σj3
i

which drives σj towards the weighted average of the squared deviations of the weights
around the corresponding centre µj , where the weighting coefﬁcients are again givenby the posterior probability that each weight is generated by component j. Note thatin a practical implementation, new variables ηj deﬁned by
σj2 = exp(ηj )(5.144)are introduced, and the minimization is performed with respect to the ηj . This ensures that the parameters σj remain positive. It also has the effect of discouraging
pathological solutions in which one or more of the σj goes to zero, corresponding
to a Gaussian component collapsing onto one of the weight parameter values. Suchsolutions are discussed in more detail in the context of Gaussian mixture models in
Section 9.2.1.
For the derivatives with respect to the mixing coefﬁcients πj , we need to takeaccount of the constraints
πj = 1,0  πi  1(5.145)jwhich follow from the interpretation of the πj as prior probabilities. This can bedone by expressing the mixing coefﬁcients in terms of a set of auxiliary variables{ηj } using the softmax function given byexp(ηj )πj = M.
k=1 exp(ηk )Exercise 5.32

(5.146)The derivatives of the regularized error function with respect to the {ηj } then takethe form

272

5. NEURAL NETWORKS

Figure 5.18 The left ﬁgure shows a two-link robot arm,in which the Cartesian coordinates (x1 , x2 ) of the end effector are determined uniquely by the two joint angles θ1and θ2 and the (ﬁxed) lengths L1 and L2 of the arms. Thisis know as the forward kinematics of the arm. In practice, we have to ﬁnd the joint angles that will give rise to adesired end effector position and, as shown in the right ﬁgure, this inverse kinematics has two solutions corresponding to ‘elbow up’ and ‘elbow down’.

(x1 , x2 )L2

(x1 , x2 )

θ2
L1

elbow
up

θ1



∂E
=
{πj − γj (wi )} .
∂ηj

elbow
down

(5.147)i

We see that πj is therefore driven towards the average posterior probability for component j.

5.6. Mixture Density Networks

Exercise 5.33

The goal of supervised learning is to model a conditional distribution p(t|x), which
for many simple regression problems is chosen to be Gaussian. However, practicalmachine learning problems can often have signiﬁcantly non-Gaussian distributions.
These can arise, for example, with inverse problems in which the distribution can be
multimodal, in which case the Gaussian assumption can lead to very poor predictions.
As a simple example of an inverse problem, consider the kinematics of a robot
arm, as illustrated in Figure 5.18. The forward problem involves ﬁnding the end effector position given the joint angles and has a unique solution. However, in practice
we wish to move the end effector of the robot to a speciﬁc position, and to do this wemust set appropriate joint angles. We therefore need to solve the inverse problem,
which has two solutions as seen in Figure 5.18.
Forward problems often corresponds to causality in a physical system and generally have a unique solution. For instance, a speciﬁc pattern of symptoms in thehuman body may be caused by the presence of a particular disease. In pattern recognition, however, we typically have to solve an inverse problem, such as trying topredict the presence of a disease given a set of symptoms. If the forward problem
involves a many-to-one mapping, then the inverse problem will have multiple solutions. For instance, several different diseases may result in the same symptoms.
In the robotics example, the kinematics is deﬁned by geometrical equations, andthe multimodality is readily apparent. However, in many machine learning problems
the presence of multimodality, particularly in problems involving spaces of high dimensionality, can be less obvious. For tutorial purposes, however, we shall consider
a simple toy problem for which we can easily visualize the multimodality. Data forthis problem is generated by sampling a variable x uniformly over the interval (0, 1),
to give a set of values {xn }, and the corresponding target values tn are obtained5.6. Mixture Density Networks
Figure 5.19 On the left is the data
set for a simple ‘forward problem’ inwhich the red curve shows the resultof ﬁtting a two-layer neural networkby minimizing the sum-of-squares
error function. The corresponding
inverse problem, shown on the right,
is obtained by exchanging the rolesof x and t. Here the same network trained again by minimizing thesum-of-squares error function gives
a very poor ﬁt to the data due to themultimodality of the data set.

1

1

0

0
0

1

0

273

1

by computing the function xn + 0.3 sin(2πxn ) and then adding uniform noise overthe interval (−0.1, 0.1). The inverse problem is then obtained by keeping the samedata points but exchanging the roles of x and t. Figure 5.19 shows the data sets forthe forward and inverse problems, along with the results of ﬁtting two-layer neuralnetworks having 6 hidden units and a single linear output unit by minimizing a sumof-squares error function. Least squares corresponds to maximum likelihood undera Gaussian assumption. We see that this leads to a very poor model for the highly
non-Gaussian inverse problem.
We therefore seek a general framework for modelling conditional probability
distributions. This can be achieved by using a mixture model for p(t|x) in which
both the mixing coefﬁcients as well as the component densities are ﬂexible functionsof the input vector x, giving rise to the mixture density network. For any given valueof x, the mixture model provides a general formalism for modelling an arbitrary
conditional density function p(t|x). Provided we consider a sufﬁciently ﬂexible
network, we then have a framework for approximating arbitrary conditional distributions.
Here we shall develop the model explicitly for Gaussian components, so thatp(t|x) =Kπk (x)N t|µk (x), σk2 (x) .

(5.148)k=1This is an example of a heteroscedastic model since the noise variance on the data
is a function of the input vector x. Instead of Gaussians, we can use other distributions for the components, such as Bernoulli distributions if the target variables arebinary rather than continuous. We have also specialized to the case of isotropic covariances for the components, although the mixture density network can readily be
extended to allow for general covariance matrices by representing the covariances
using a Cholesky factorization (Williams, 1996). Even with isotropic components,
the conditional distribution p(t|x) does not assume factorization with respect to thecomponents of t (in contrast to the standard sum-of-squares regression model) as aconsequence of the mixture distribution.
We now take the various parameters of the mixture model, namely the mixing
coefﬁcients πk (x), the means µk (x), and the variances σk2 (x), to be governed by

274

5. NEURAL NETWORKSp(t|x)
xD

θM
θ

x1

θ1
t

Figure 5.20

The mixture density network can represent general conditional probability densities p(t|x)
by considering a parametric mixture model for the distribution of t whose parameters are
determined by the outputs of a neural network that takes x as its input vector.

the outputs of a conventional neural network that takes x as its input. The structureof this mixture density network is illustrated in Figure 5.20. The mixture density
network is closely related to the mixture of experts discussed in Section 14.5.3. The
principle difference is that in the mixture density network the same function is used
to predict the parameters of all of the component densities as well as the mixing coefﬁcients, and so the nonlinear hidden units are shared amongst the input-dependentfunctions.
The neural network in Figure 5.20 can, for example, be a two-layer network
having sigmoidal (‘tanh’) hidden units. If there are L components in the mixture
model (5.148), and if t has K components, then the network will have L output unit
activations denoted by aπk that determine the mixing coefﬁcients πk (x), K outputs
denoted by aσk that determine the kernel widths σk (x), and L × K outputs denotedby aµkj that determine the components µkj (x) of the kernel centres µk (x). The total
number of network outputs is given by (K + 2)L, as compared with the usual K
outputs for a network, which simply predicts the conditional means of the target
variables.
The mixing coefﬁcients must satisfy the constraintsKπk (x) = 1,

0  πk (x)  1(5.149)k=1

which can be achieved using a set of softmax outputs
exp(aπk ).
πk (x) = Kπl=1 exp(al )(5.150)Similarly, the variances must satisfy σk2 (x)  0 and so can be represented in termsof the exponentials of the corresponding network activations using
σk (x) = exp(aσk ).

(5.151)

Finally, because the means µk (x) have real components, they can be represented

5.6. Mixture Density Networks

275

directly by the network output activationsµkj (x) = aµkj .

(5.152)

The adaptive parameters of the mixture density network comprise the vector w
of weights and biases in the neural network, that can be set by maximum likelihood,or equivalently by minimizing an error function deﬁned to be the negative logarithmof the likelihood. For independent data, this error function takes the form

 k
N


ln
πk (xn , w)N tn |µk (xn , w), σk2 (xn , w)
(5.153)E(w) = −n=1

k=1

where we have made the dependencies on w explicit.
In order to minimize the error function, we need to calculate the derivatives ofthe error E(w) with respect to the components of w. These can be evaluated byusing the standard backpropagation procedure, provided we obtain suitable expressions for the derivatives of the error with respect to the output-unit activations. These
represent error signals δ for each pattern and for each output unit, and can be backpropagated to the hidden units and the error function derivatives evaluated in theusual way. Because the error function (5.153) is composed of a sum of terms, one
for each training data point, we can consider the derivatives for a particular patternn and then ﬁnd the derivatives of E by summing over all patterns.
Because we are dealing with mixture distributions, it is convenient to view themixing coefﬁcients πk (x) as x-dependent prior probabilities and to introduce thecorresponding posterior probabilities given byπk Nnkγk (t|x) = K
l=1 πl Nnl

Exercise 5.34

Exercise 5.35

where Nnk denotes N (tn |µk (xn ), σk2 (xn )).
The derivatives with respect to the network output activations governing the mixing coefﬁcients are given by∂En= πk − γk .
(5.155)∂aπkSimilarly, the derivatives with respect to the output activations controlling the component means are given by
∂En
= γk
∂aµkl

Exercise 5.36

(5.154)µkl − tl
σk2.

(5.156)

Finally, the derivatives with respect to the output activations controlling the component variances are given by
∂En
= −γk
∂aσk

t − µk 2
1
−
3
σk
σk

.

(5.157)276

5. NEURAL NETWORKS

Figure 5.21 (a) Plot of the mixing
coefﬁcients πk (x) as a function ofx for the three kernel functions in a
mixture density network trained on
the data shown in Figure 5.19. The
model has three Gaussian components, and uses a two-layer multilayer perceptron with ﬁve ‘tanh’ sigmoidal units in the hidden layer, andnine outputs (corresponding to the 3means and 3 variances of the Gaussian components and the 3 mixing
coefﬁcients). At both small and large
values of x, where the conditional
probability density of the target data
is unimodal, only one of the kernels has a high value for its priorprobability, while at intermediate values of x, where the conditional density is trimodal, the three mixing coefﬁcients have comparable values.
(b) Plots of the means µk (x) using
the same colour coding as for the
mixing coefﬁcients. (c) Plot of thecontours of the corresponding conditional probability density of the target data for the same mixture density network. (d) Plot of the approximate conditional mode, shownby the red points, of the conditional
density.

1

1

0

0
0

1

0

(a)

(b)

1

1

0

0
0

1

1

0

(c)

1
(d)

We illustrate the use of a mixture density network by returning to the toy example of an inverse problem shown in Figure 5.19. Plots of the mixing coefﬁcients πk (x), the means µk (x), and the conditional density contours correspondingto p(t|x), are shown in Figure 5.21. The outputs of the neural network, and hence theparameters in the mixture model, are necessarily continuous single-valued functionsof the input variables. However, we see from Figure 5.21(c) that the model is able toproduce a conditional density that is unimodal for some values of x and trimodal forother values by modulating the amplitudes of the mixing components πk (x).
Once a mixture density network has been trained, it can predict the conditional
density function of the target data for any given value of the input vector. Thisconditional density represents a complete description of the generator of the data, so
far as the problem of predicting the value of the output vector is concerned. From
this density function we can calculate more speciﬁc quantities that may be of interest
in different applications. One of the simplest of these is the mean, corresponding to
the conditional average of the target data, and is given by
E [t|x] =tp(t|x) dt =

K

k=1

πk (x)µk (x)(5.158)

5.7. Bayesian Neural Networks

Exercise 5.37

277

where we have used (5.148). Because a standard network trained by least squares
is approximating the conditional mean, we see that a mixture density network can
reproduce the conventional least-squares result as a special case. Of course, as wehave already noted, for a multimodal distribution the conditional mean is of limited
value.
We can similarly evaluate the variance of the density function about the conditional average, to give2s2 (x) = E t − E[t|x] |x(5.159)
⎧
⎫
'
'
K
K
'
'2 ⎬
⎨


'
'
2
πk (x) σk (x) + 'µk (x) −πl (x)µl (x)'(5.160)
=
'
' ⎭
⎩
k=1

l=1

where we have used (5.148) and (5.158). This is more general than the corresponding
least-squares result because the variance is a function of x.
We have seen that for multimodal distributions, the conditional mean can givea poor representation of the data. For instance, in controlling the simple robot arm
shown in Figure 5.18, we need to pick one of the two possible joint angle settingsin order to achieve the desired end-effector location, whereas the average of the twosolutions is not itself a solution. In such cases, the conditional mode may be ofmore value. Because the conditional mode for the mixture density network does nothave a simple analytical solution, this would require numerical iteration. A simple
alternative is to take the mean of the most probable component (i.e., the one with thelargest mixing coefﬁcient) at each value of x. This is shown for the toy data set in
Figure 5.21(d).

5.7. Bayesian Neural Networks
So far, our discussion of neural networks has focussed on the use of maximum likelihood to determine the network parameters (weights and biases). Regularized maximum likelihood can be interpreted as a MAP (maximum posterior) approach inwhich the regularizer can be viewed as the logarithm of a prior parameter distribution. However, in a Bayesian treatment we need to marginalize over the distributionof parameters in order to make predictions.
In Section 3.3, we developed a Bayesian solution for a simple linear regression
model under the assumption of Gaussian noise. We saw that the posterior distribution, which is Gaussian, could be evaluated exactly and that the predictive distribution could also be found in closed form. In the case of a multilayered network, thehighly nonlinear dependence of the network function on the parameter values meansthat an exact Bayesian treatment can no longer be found. In fact, the log of the posterior distribution will be nonconvex, corresponding to the multiple local minima inthe error function.
The technique of variational inference, to be discussed in Chapter 10, has been
applied to Bayesian neural networks using a factorized Gaussian approximation

278

5. NEURAL NETWORKS
to the posterior distribution (Hinton and van Camp, 1993) and also using a fullcovariance Gaussian (Barber and Bishop, 1998a; Barber and Bishop, 1998b). The
most complete treatment, however, has been based on the Laplace approximation(MacKay, 1992c; MacKay, 1992b) and forms the basis for the discussion given here.
We will approximate the posterior distribution by a Gaussian, centred at a mode of
the true posterior. Furthermore, we shall assume that the covariance of this Gaussian is small so that the network function is approximately linear with respect to theparameters over the region of parameter space for which the posterior probability issigniﬁcantly nonzero. With these two approximations, we will obtain models thatare analogous to the linear regression and classiﬁcation models discussed in earlier
chapters and so we can exploit the results obtained there. We can then make use of
the evidence framework to provide point estimates for the hyperparameters and tocompare alternative models (for example, networks having different numbers of hidden units). To start with, we shall discuss the regression case and then later consider
the modiﬁcations needed for solving classiﬁcation tasks.

5.7.1 Posterior parameter distribution
Consider the problem of predicting a single continuous target variable t froma vector x of inputs (the extension to multiple targets is straightforward). We shallsuppose that the conditional distribution p(t|x) is Gaussian, with an x-dependent
mean given by the output of a neural network model y(x, w), and with precision(inverse variance) βp(t|x, w, β) = N (t|y(x, w), β −1 ).

(5.161)

Similarly, we shall choose a prior distribution over the weights w that is Gaussian of
the form(5.162)p(w|α) = N (w|0, α−1 I).
For an i.i.d. data set of N observations x1 , . . . , xN , with a corresponding set of target
values D = {t1 , . . . , tN }, the likelihood function is given byp(D|w, β) =NN (tn |y(xn , w), β −1 )(5.163)n=1

and so the resulting posterior distribution is thenp(w|D, α, β) ∝ p(w|α)p(D|w, β).

(5.164)

which, as a consequence of the nonlinear dependence of y(x, w) on w, will be nonGaussian.
We can ﬁnd a Gaussian approximation to the posterior distribution by using the
Laplace approximation. To do this, we must ﬁrst ﬁnd a (local) maximum of theposterior, and this must be done using iterative numerical optimization. As usual, it
is convenient to maximize the logarithm of the posterior, which can be written in the5.7. Bayesian Neural Networks

279

form
α
β
2
ln p(w|D) = − wT w −{y(xn , w) − tn } + const
2
2
N(5.165)n=1

which corresponds to a regularized sum-of-squares error function. Assuming forthe moment that α and β are ﬁxed, we can ﬁnd a maximum of the posterior, which
we denote wMAP , by standard nonlinear optimization algorithms such as conjugate
gradients, using error backpropagation to evaluate the required derivatives.
Having found a mode wMAP , we can then build a local Gaussian approximationby evaluating the matrix of second derivatives of the negative log posterior distribution. From (5.165), this is given by
A = −∇∇ ln p(w|D, α, β) = αI + βH

(5.166)

where H is the Hessian matrix comprising the second derivatives of the sum-ofsquares error function with respect to the components of w. Algorithms for computing and approximating the Hessian were discussed in Section 5.4. The corresponding
Gaussian approximation to the posterior is then given from (4.134) byq(w|D) = N (w|wMAP , A−1 ).

(5.167)

Similarly, the predictive distribution is obtained by marginalizing with respect
to this posterior distribution
(5.168)p(t|x, D) = p(t|x, w)q(w|D) dw.
However, even with the Gaussian approximation to the posterior, this integration isstill analytically intractable due to the nonlinearity of the network function y(x, w)
as a function of w. To make progress, we now assume that the posterior distribution
has small variance compared with the characteristic scales of w over which y(x, w)
is varying. This allows us to make a Taylor series expansion of the network function
around wMAP and retain only the linear terms
y(x, w)  y(x, wMAP ) + gT (w − wMAP )where we have deﬁned

g = ∇w y(x, w)|w=wMAP .

(5.169)(5.170)With this approximation, we now have a linear-Gaussian model with a Gaussian
distribution for p(w) and a Gaussian for p(t|w) whose mean is a linear function of
w of the formp(t|x, w, β)  N t|y(x, wMAP ) + gT (w − wMAP ), β −1 .
Exercise 5.38

(5.171)

We can therefore make use of the general result (2.115) for the marginal p(t) to givep(t|x, D, α, β) = N t|y(x, wMAP ), σ 2 (x)(5.172)

280

5. NEURAL NETWORKS
where the input-dependent variance is given byσ 2 (x) = β −1 + gT A−1 g.

(5.173)We see that the predictive distribution p(t|x, D) is a Gaussian whose mean is givenby the network function y(x, wMAP ) with the parameter set to their MAP value. The
variance has two terms, the ﬁrst of which arises from the intrinsic noise on the target
variable, whereas the second is an x-dependent term that expresses the uncertainty
in the interpolant due to the uncertainty in the model parameters w. This should
be compared with the corresponding predictive distribution for the linear regression
model, given by (3.58) and (3.59).

5.7.2 Hyperparameter optimization
So far, we have assumed that the hyperparameters α and β are ﬁxed and known.
We can make use of the evidence framework, discussed in Section 3.5, together with
the Gaussian approximation to the posterior obtained using the Laplace approximation, to obtain a practical procedure for choosing the values of such hyperparameters.
The marginal likelihood, or evidence, for the hyperparameters is obtained byintegrating over the network weights(5.174)p(D|α, β) = p(D|w, β)p(w|α) dw.
Exercise 5.39

This is easily evaluated by making use of the Laplace approximation result (4.135).
Taking logarithms then givesln p(D|α, β)  −E(wMAP ) −1
WNN
ln |A| +ln α +
ln β −ln(2π) (5.175)2222

where W is the total number of parameters in w, and the regularized error function
is deﬁned byβ
α T
2{y(xn , wMAP ) − tn } + wMAP
wMAP .
2
2
N

E(wMAP ) =(5.176)n=1

We see that this takes the same form as the corresponding result (3.86) for the linear
regression model.
In the evidence framework, we make point estimates for α and β by maximizing
ln p(D|α, β). Consider ﬁrst the maximization with respect to α, which can be doneby analogy with the linear regression case discussed in Section 3.5.2. We ﬁrst deﬁne
the eigenvalue equation(5.177)βHui = λi ui
where H is the Hessian matrix comprising the second derivatives of the sum-ofsquares error function, evaluated at w = wMAP . By analogy with (3.92), we obtain
α=

γ
T
wMAP
wMAP

(5.178)

5.7. Bayesian Neural Networks
Section 3.5.3

281

where γ represents the effective number of parameters and is deﬁned by
γ=

W

i=1

λi
.
α + λi(5.179)Note that this result was exact for the linear regression case. For the nonlinear neuralnetwork, however, it ignores the fact that changes in α will cause changes in the
Hessian H, which in turn will change the eigenvalues. We have therefore implicitly
ignored terms involving the derivatives of λi with respect to α.
Similarly, from (3.95) we see that maximizing the evidence with respect to β
gives the re-estimation formula1
1 {y(xn , wMAP ) − tn }2 .
=
β
N −γ
N

(5.180)n=1

Section 5.1.1

As with the linear model, we need to alternate between re-estimation of the hyperparameters α and β and updating of the posterior distribution. The situation witha neural network model is more complex, however, due to the multimodality of theposterior distribution. As a consequence, the solution for wMAP found by maximizing the log posterior will depend on the initialization of w. Solutions that differ only
as a consequence of the interchange and sign reversal symmetries in the hidden units
are identical so far as predictions are concerned, and it is irrelevant which of theequivalent solutions is found. However, there may be inequivalent solutions as well,and these will generally yield different values for the optimized hyperparameters.
In order to compare different models, for example neural networks having different numbers of hidden units, we need to evaluate the model evidence p(D). This canbe approximated by taking (5.175) and substituting the values of α and β obtained
from the iterative optimization of these hyperparameters. A more careful evaluation
is obtained by marginalizing over α and β, again by making a Gaussian approximation (MacKay, 1992c; Bishop, 1995a). In either case, it is necessary to evaluate thedeterminant |A| of the Hessian matrix. This can be problematic in practice because
the determinant, unlike the trace, is sensitive to the small eigenvalues that are often
difﬁcult to determine accurately.
The Laplace approximation is based on a local quadratic expansion around amode of the posterior distribution over weights. We have seen in Section 5.1.1 thatany given mode in a two-layer network is a member of a set of M !2M equivalentmodes that differ by interchange and sign-change symmetries, where M is the number of hidden units. When comparing networks having different numbers of hidden units, this can be taken into account by multiplying the evidence by a factor of
M !2M .

5.7.3 Bayesian neural networks for classiﬁcation
So far, we have used the Laplace approximation to develop a Bayesian treatment of neural network regression models. We now discuss the modiﬁcations to282

5. NEURAL NETWORKS

Exercise 5.40

this framework that arise when it is applied to classiﬁcation. Here we shall consider a network having a single logistic sigmoid output corresponding to a two-classclassiﬁcation problem. The extension to networks with multiclass softmax outputs
is straightforward. We shall build extensively on the analogous results for linear
classiﬁcation models discussed in Section 4.5, and so we encourage the reader tofamiliarize themselves with that material before studying this section.
The log likelihood function for this model is given byln p(D|w) == 1N {tn ln yn + (1 − tn ) ln(1 − yn )}(5.181)n

Exercise 5.41

where tn ∈ {0, 1} are the target values, and yn ≡ y(xn , w). Note that there is nohyperparameter β, because the data points are assumed to be correctly labelled. As
before, the prior is taken to be an isotropic Gaussian of the form (5.162).
The ﬁrst stage in applying the Laplace framework to this model is to initialize
the hyperparameter α, and then to determine the parameter vector w by maximizing
the log posterior distribution. This is equivalent to minimizing the regularized error
functionα(5.182)E(w) = − ln p(D|w) + wT w
2and can be achieved using error backpropagation combined with standard optimization algorithms, as discussed in Section 5.3.
Having found a solution wMAP for the weight vector, the next step is to evaluate the Hessian matrix H comprising the second derivatives of the negative log
likelihood function. This can be done, for instance, using the exact method of Section 5.4.5, or using the outer product approximation given by (5.85). The second
derivatives of the negative log posterior can again be written in the form (5.166), and
the Gaussian approximation to the posterior is then given by (5.167).
To optimize the hyperparameter α, we again maximize the marginal likelihood,which is easily shown to take the formln p(D|α)  −E(wMAP ) −1
W
ln |A| +ln α + const
2
2(5.183)where the regularized error function is deﬁned by
E(wMAP ) = −

N

n=1

{tn ln yn + (1 − tn ) ln(1 − yn )} +α T
w
wMAP (5.184)
2 MAP

in which yn ≡ y(xn , wMAP ). Maximizing this evidence function with respect to αagain leads to the re-estimation equation given by (5.178).
The use of the evidence procedure to determine α is illustrated in Figure 5.22
for the synthetic two-dimensional data discussed in Appendix A.
Finally, we need the predictive distribution, which is deﬁned by (5.168). Again,
this integration is intractable due to the nonlinearity of the network function. The

283

5.7. Bayesian Neural Networks
Figure 5.22

Illustration of the evidence framework
3
applied to a synthetic two-class data set.
The green curve shows the optimal decision boundary, the black curve shows 2
the result of ﬁtting a two-layer networkwith 8 hidden units by maximum likeli- 1
hood, and the red curve shows the result of including a regularizer in which 0
α is optimized using the evidence procedure, starting from the initial value −1
α = 0. Note that the evidence procedure greatly reduces the over-ﬁtting of −2
the network.

−2

−1

0

1

2

simplest approximation is to assume that the posterior distribution is very narrowand hence make the approximationp(t|x, D)  p(t|x, wMAP ).

(5.185)

We can improve on this, however, by taking account of the variance of the posterior
distribution. In this case, a linear approximation for the network outputs, as was usedin the case of regression, would be inappropriate due to the logistic sigmoid outputunit activation function that constrains the output to lie in the range (0, 1). Instead,
we make a linear approximation for the output unit activation in the forma(x, w)  aMAP (x) + bT (w − wMAP )(5.186)

where aMAP (x) = a(x, wMAP ), and the vector b ≡ ∇a(x, wMAP ) can be found by
backpropagation.
Because we now have a Gaussian approximation for the posterior distribution
over w, and a model for a that is a linear function of w, we can now appeal to theresults of Section 4.5.2. The distribution of output unit activation values, induced by
the distribution over network weights, is given by
p(a|x, D) = δ a − aMAP (x) − bT (x)(w − wMAP ) q(w|D) dw (5.187)
where q(w|D) is the Gaussian approximation to the posterior distribution given by(5.167). From Section 4.5.2, we see that this distribution is Gaussian with mean
aMAP ≡ a(x, wMAP ), and variance
σa2 (x) = bT (x)A−1 b(x).

(5.188)

Finally, to obtain the predictive distribution, we must marginalize over a using(5.189)p(t = 1|x, D) = σ(a)p(a|x, D) da.

284

5. NEURAL NETWORKS
3

3

2

2

1

1

0

0

−1

−1

−2

−2
−2

−1

0

1

2

−2

−1

0

1

2

Figure 5.23 An illustration of the Laplace approximation for a Bayesian neural network having 8 hidden units
with ‘tanh’ activation functions and a single logistic-sigmoid output unit. The weight parameters were found using
scaled conjugate gradients, and the hyperparameter α was optimized using the evidence framework. On the left
is the result of using the simple approximation (5.185) based on a point estimate wMAP of the parameters,
in which the green curve shows the y = 0.5 decision boundary, and the other contours correspond to output
probabilities of y = 0.1, 0.3, 0.7, and 0.9. On the right is the corresponding result obtained using (5.190). Notethat the effect of marginalization is to spread out the contours and to make the predictions less conﬁdent, sothat at each input point x, the posterior probabilities are shifted towards 0.5, while the y = 0.5 contour itself isunaffected.

The convolution of a Gaussian with a logistic sigmoid is intractable. We therefore
apply the approximation (4.153) to (5.189) givingp(t = 1|x, D) = σ κ(σa2 )bT wMAP

(5.190)where κ(·) is deﬁned by (4.154). Recall that both σa2 and b are functions of x.
Figure 5.23 shows an example of this framework applied to the synthetic classiﬁcation data set described in Appendix A.

Exercises

5.1 ( ) Consider a two-layer network function of the form (5.7) in which the hiddenunit nonlinear activation functions g(·) are given by logistic sigmoid functions of theform−1(5.191)σ(a) = {1 + exp(−a)} .
Show that there exists an equivalent network, which computes exactly the same function, but with hidden unit activation functions given by tanh(a) where the tanh function is deﬁned by (5.59). Hint: ﬁrst ﬁnd the relation between σ(a) and tanh(a), andthen show that the parameters of the two networks differ by linear transformations.
5.2 ( ) www Show that maximizing the likelihood function under the conditional
distribution (5.16) for a multioutput neural network is equivalent to minimizing thesum-of-squares error function (5.11).

Exercises285

5.3 ( ) Consider a regression problem involving multiple target variables in which it
is assumed that the distribution of the targets, conditioned on the input vector x, is a
Gaussian of the formp(t|x, w) = N (t|y(x, w), Σ)(5.192)where y(x, w) is the output of a neural network with input vector x and weightvector w, and Σ is the covariance of the assumed Gaussian noise on the targets.
Given a set of independent observations of x and t, write down the error functionthat must be minimized in order to ﬁnd the maximum likelihood solution for w, if
we assume that Σ is ﬁxed and known. Now assume that Σ is also to be determinedfrom the data, and write down an expression for the maximum likelihood solutionfor Σ. Note that the optimizations of w and Σ are now coupled, in contrast to thecase of independent target variables discussed in Section 5.2.
5.4 ( ) Consider a binary classiﬁcation problem in which the target values are t ∈{0, 1}, with a network output y(x, w) that represents p(t = 1|x), and suppose that
there is a probability  that the class label on a training data point has been incorrectlyset. Assuming independent and identically distributed data, write down the error
function corresponding to the negative log likelihood. Verify that the error function(5.21) is obtained when  = 0. Note that this error function makes the model robust
to incorrectly labelled data, in contrast to the usual error function.
5.5 ( ) www Show that maximizing likelihood for a multiclass neural network modelin which the network outputs have the interpretation yk (x, w) = p(tk = 1|x) is
equivalent to the minimization of the cross-entropy error function (5.24).
5.6 ( ) www Show the derivative of the error function (5.21) with respect to theactivation ak for an output unit having a logistic sigmoid activation function satisﬁes(5.18).
5.7 ( ) Show the derivative of the error function (5.24) with respect to the activation ak
for output units having a softmax activation function satisﬁes (5.18).
5.8 ( ) We saw in (4.88) that the derivative of the logistic sigmoid activation function
can be expressed in terms of the function value itself. Derive the corresponding result
for the ‘tanh’ activation function deﬁned by (5.59).
5.9 ( ) www The error function (5.21) for binary classiﬁcation problems was derived for a network having a logistic-sigmoid output activation function, so that0  y(x, w)  1, and data having target values t ∈ {0, 1}. Derive the corresponding error function if we consider a network having an output −1  y(x, w)  1
and target values t = 1 for class C1 and t = −1 for class C2 . What would be theappropriate choice of output unit activation function?
5.10 ( ) www Consider a Hessian matrix H with eigenvector equation (5.33). By
setting the vector v in (5.39) equal to each of the eigenvectors ui in turn, show that
H is positive deﬁnite if, and only if, all of its eigenvalues are positive.

286

5. NEURAL NETWORKS
5.11 ( ) www Consider a quadratic error function deﬁned by (5.32), in which theHessian matrix H has an eigenvalue equation given by (5.33). Show that the contours of constant error are ellipses whose axes are aligned with the eigenvectors ui ,with lengths that are inversely proportional to the square root of the corresponding
eigenvalues λi .
5.12 ( ) www By considering the local Taylor expansion (5.32) of an error function
about a stationary point w , show that the necessary and sufﬁcient condition for thestationary point to be a local minimum of the error function is that the Hessian matrix
 = w , be positive deﬁnite.
H, deﬁned by (5.30) with w
5.13 ( ) Show that as a consequence of the symmetry of the Hessian matrix H, thenumber of independent elements in the quadratic error function (5.28) is given by
W (W + 3)/2.
5.14 ( ) By making a Taylor expansion, verify that the terms that are O() cancel on theright-hand side of (5.69).
5.15 ( ) In Section 5.3.4, we derived a procedure for evaluating the Jacobian matrix of aneural network using a backpropagation procedure. Derive an alternative formalism
for ﬁnding the Jacobian based on forward propagation equations.
5.16 ( ) The outer product approximation to the Hessian matrix for a neural network
using a sum-of-squares error function is given by (5.84). Extend this result to thecase of multiple outputs.
5.17 ( ) Consider a squared loss function of the form12{y(x, w) − t} p(x, t) dx dt
E=2(5.193)where y(x, w) is a parametric function such as a neural network. The result (1.89)
shows that the function y(x, w) that minimizes this error is given by the conditional
expectation of t given x. Use this result to show that the second derivative of E withrespect to two elements wr and ws of the vector w, is given by

∂y ∂y
∂2E
=
p(x) dx.
(5.194)∂wr ∂ws∂wr ∂ws
Note that, for a ﬁnite sample from p(x), we obtain (5.84).
5.18 ( ) Consider a two-layer network of the form shown in Figure 5.1 with the additionof extra parameters corresponding to skip-layer connections that go directly fromthe inputs to the outputs. By extending the discussion of Section 5.3.2, write down
the equations for the derivatives of the error function with respect to these additionalparameters.
5.19 ( ) www Derive the expression (5.85) for the outer product approximation to
the Hessian matrix for a network having a single output with a logistic sigmoid
output-unit activation function and a cross-entropy error function, corresponding to
the result (5.84) for the sum-of-squares error function.

Exercises

287

5.20 ( ) Derive an expression for the outer product approximation to the Hessian matrix
for a network having K outputs with a softmax output-unit activation function anda cross-entropy error function, corresponding to the result (5.84) for the sum-ofsquares error function.
5.21 () Extend the expression (5.86) for the outer product approximation of the Hessian matrix to the case of K > 1 output units. Hence, derive a recursive expression
analogous to (5.87) for incrementing the number N of patterns and a similar expression for incrementing the number K of outputs. Use these results, together with theidentity (5.88), to ﬁnd sequential update expressions analogous to (5.89) for ﬁnding
the inverse of the Hessian by incrementally including both extra patterns and extra
outputs.
5.22 ( ) Derive the results (5.93), (5.94), and (5.95) for the elements of the Hessian
matrix of a two-layer feed-forward network by application of the chain rule of calculus.
5.23 ( ) Extend the results of Section 5.4.5 for the exact Hessian of a two-layer network
to include skip-layer connections that go directly from inputs to outputs.
5.24 ( ) Verify that the network function deﬁned by (5.113) and (5.114) is invariant under the transformation (5.115) applied to the inputs, provided the weights and biases
are simultaneously transformed using (5.116) and (5.117). Similarly, show that the
network outputs can be transformed according (5.118) by applying the transformation (5.119) and (5.120) to the second-layer weights and biases.
5.25 () www

Consider a quadratic error function of the form
1
E = E0 + (w − w )T H(w − w )2

(5.195)where w represents the minimum, and the Hessian matrix H is positive deﬁnite andconstant. Suppose the initial weight vector w(0) is chosen to be at the origin and is
updated using simple gradient descentw(τ ) = w(τ −1) − ρ∇E(5.196)

where τ denotes the step number, and ρ is the learning rate (which is assumed to besmall). Show that, after τ steps, the components of the weight vector parallel to theeigenvectors of H can be written
(τ )

wj

= {1 − (1 − ρηj )τ } wj

(5.197)

where wj = wT uj , and uj and ηj are the eigenvectors and eigenvalues, respectively,of H so that
(5.198)
Huj = ηj uj .
Show that as τ → ∞, this gives w(τ ) → w as expected, provided |1 − ρηj | < 1.
Now suppose that training is halted after a ﬁnite number τ of steps. Show that the288

5. NEURAL NETWORKS
components of the weight vector parallel to the eigenvectors of the Hessian satisfy
(τ )

wj

 wj

(τ )

|wj |

when ηj(ρτ )−1(5.199)|wj | when ηj(ρτ )−1 .

(5.200)Compare this result with the discussion in Section 3.5.3 of regularization with simple
weight decay, and hence show that (ρτ )−1 is analogous to the regularization parameter λ. The above results also show that the effective number of parameters in the
network, as deﬁned by (3.91), grows as the training progresses.
5.26 ( ) Consider a multilayer perceptron with arbitrary feed-forward topology, whichis to be trained by minimizing the tangent propagation error function (5.127) inwhich the regularizing function is given by (5.128). Show that the regularization
term Ω can be written as a sum over patterns of terms of the form12
(Gyk )(5.201)Ωn =2kwhere G is a differential operator deﬁned by
 ∂
G≡
τi
.
∂xi

(5.202)i

By acting on the forward propagation equations
zj = h(aj ),

aj =



wji zi(5.203)iwith the operator G, show that Ωn can be evaluated by forward propagation using
the following equations:βj =wji αi .
(5.204)αj = h (aj )βj ,iwhere we have deﬁned the new variables
αj ≡ Gzj ,βj ≡ Gaj .

(5.205)

Now show that the derivatives of Ωn with respect to a weight wrs in the network canbe written in the form∂Ωn=αk {φkr zs + δkr αs }(5.206)∂wrs
kwhere we have deﬁned
δkr ≡

∂yk
,
∂ar

φkr ≡ Gδkr .

(5.207)

Write down the backpropagation equations for δkr , and hence derive a set of backpropagation equations for the evaluation of the φkr .

Exercises289

5.27 ( ) www Consider the framework for training with transformed data in thespecial case in which the transformation consists simply of the addition of random
noise x → x + ξ where ξ has a Gaussian distribution with zero mean and unit
covariance. By following an argument analogous to that of Section 5.5.5, show thatthe resulting regularizer reduces to the Tikhonov form (5.135).
5.28 ( ) www Consider a neural network, such as the convolutional network discussed
in Section 5.5.6, in which multiple weights are constrained to have the same value.
Discuss how the standard backpropagation algorithm must be modiﬁed in order toensure that such constraints are satisﬁed when evaluating the derivatives of an error
function with respect to the adjustable parameters in the network.
5.29 ( ) www

Verify the result (5.141).

5.30 ( ) Verify the result (5.142).
5.31 ( ) Verify the result (5.143).
5.32 ( ) Show that the derivatives of the mixing coefﬁcients {πk }, deﬁned by (5.146),
with respect to the auxiliary parameters {ηj } are given by∂πk= δjk πj − πj πk .
(5.208)∂ηj
Hence, by making use of the constraint k πk = 1, derive the result (5.147).
5.33 ( ) Write down a pair of equations that express the Cartesian coordinates (x1 , x2 )for the robot arm shown in Figure 5.18 in terms of the joint angles θ1 and θ2 andthe lengths L1 and L2 of the links. Assume the origin of the coordinate system isgiven by the attachment point of the lower arm. These equations deﬁne the ‘forward
kinematics’ of the robot arm.
5.34 ( ) www Derive the result (5.155) for the derivative of the error function withrespect to the network output activations controlling the mixing coefﬁcients in the
mixture density network.
5.35 ( ) Derive the result (5.156) for the derivative of the error function with respect
to the network output activations controlling the component means in the mixture
density network.
5.36 ( ) Derive the result (5.157) for the derivative of the error function with respect tothe network output activations controlling the component variances in the mixture
density network.
5.37 ( ) Verify the results (5.158) and (5.160) for the conditional mean and variance ofthe mixture density network model.
5.38 ( ) Using the general result (2.115), derive the predictive distribution (5.172) for
the Laplace approximation to the Bayesian neural network model.

290

5. NEURAL NETWORKS
5.39 ( ) www Make use of the Laplace approximation result (4.135) to show that theevidence function for the hyperparameters α and β in the Bayesian neural networkmodel can be approximated by (5.175).
5.40 ( ) www Outline the modiﬁcations needed to the framework for Bayesian neuralnetworks, discussed in Section 5.7.3, to handle multiclass problems using networks
having softmax output-unit activation functions.
5.41 ( ) By following analogous steps to those given in Section 5.7.1 for regression
networks, derive the result (5.183) for the marginal likelihood in the case of a network having a cross-entropy error function and logistic-sigmoid output-unit activation function.

6
KernelMethods

Chapter 5

Section 2.5.1

In Chapters 3 and 4, we considered linear parametric models for regression andclassiﬁcation in which the form of the mapping y(x, w) from input x to output yis governed by a vector w of adaptive parameters. During the learning phase, a
set of training data is used either to obtain a point estimate of the parameter vector
or to determine a posterior distribution over this vector. The training data is thendiscarded, and predictions for new inputs are based purely on the learned parameter
vector w. This approach is also used in nonlinear parametric models such as neuralnetworks.
However, there is a class of pattern recognition techniques, in which the training
data points, or a subset of them, are kept and used also during the prediction phase.
For instance, the Parzen probability density model comprised a linear combinationof ‘kernel’ functions each one centred on one of the training data points. Similarly,
in Section 2.5.2 we introduced a simple technique for classiﬁcation called nearest
neighbours, which involved assigning to each new test vector the same label as the

291

292

6. KERNEL METHODS
closest example from the training set. These are examples of memory-based methods
that involve storing the entire training set in order to make predictions for future data
points. They typically require a metric to be deﬁned that measures the similarity of
any two vectors in input space, and are generally fast to ‘train’ but slow at making
predictions for test data points.
Many linear parametric models can be re-cast into an equivalent ‘dual representation’ in which the predictions are also based on linear combinations of a kernel
function evaluated at the training data points. As we shall see, for models which are
based on a ﬁxed nonlinear feature space mapping φ(x), the kernel function is givenby the relation
k(x, x ) = φ(x)T φ(x ).
(6.1)

Chapter 7

Section 12.3

Section 6.3

From this deﬁnition, we see that the kernel is a symmetric function of its arguments
so that k(x, x ) = k(x , x). The kernel concept was introduced into the ﬁeld of pattern recognition by Aizerman et al. (1964) in the context of the method of potentialfunctions, so-called because of an analogy with electrostatics. Although neglected
for many years, it was re-introduced into machine learning in the context of largemargin classiﬁers by Boser et al. (1992) giving rise to the technique of support
vector machines. Since then, there has been considerable interest in this topic, bothin terms of theory and applications. One of the most signiﬁcant developments has
been the extension of kernels to handle symbolic objects, thereby greatly expanding
the range of problems that can be addressed.
The simplest example of a kernel function is obtained by considering the identity
mapping for the feature space in (6.1) so that φ(x) = x, in which case k(x, x ) =xT x . We shall refer to this as the linear kernel.
The concept of a kernel formulated as an inner product in a feature space allows
us to build interesting extensions of many well-known algorithms by making use of
the kernel trick, also known as kernel substitution. The general idea is that, if we havean algorithm formulated in such a way that the input vector x enters only in the formof scalar products, then we can replace that scalar product with some other choice of
kernel. For instance, the technique of kernel substitution can be applied to principal
component analysis in order to develop a nonlinear variant of PCA (Schölkopf et al.,1998). Other examples of kernel substitution include nearest-neighbour classiﬁers
and the kernel Fisher discriminant (Mika et al., 1999; Roth and Steinhage, 2000;
Baudat and Anouar, 2000).
There are numerous forms of kernel functions in common use, and we shall encounter several examples in this chapter. Many have the property of being a function
only of the difference between the arguments, so that k(x, x ) = k(x − x ), which
are known as stationary kernels because they are invariant to translations in input
space. A further specialization involves homogeneous kernels, also known as radial basis functions, which depend only on the magnitude of the distance (typicallyEuclidean) between the arguments so that k(x, x ) = k(x − x ).
For recent textbooks on kernel methods, see Schölkopf and Smola (2002), Herbrich (2002), and Shawe-Taylor and Cristianini (2004).

6.1. Dual Representations

293

6.1. Dual Representations
Many linear models for regression and classiﬁcation can be reformulated in terms ofa dual representation in which the kernel function arises naturally. This concept willplay an important role when we consider support vector machines in the next chapter.
Here we consider a linear regression model whose parameters are determined byminimizing a regularized sum-of-squares error function given by2 λ
1  T
J(w) =w φ(xn ) − tn + wT w
2
2
N

(6.2)

n=1

where λ  0. If we set the gradient of J(w) with respect to w equal to zero, we seethat the solution for w takes the form of a linear combination of the vectors φ(xn ),with coefﬁcients that are functions of w, of the formw=−

1  Tw φ(xn ) − tn φ(xn ) =an φ(xn ) = ΦT a
λ
N

N

n=1

n=1

(6.3)

where Φ is the design matrix, whose nth row is given by φ(xn )T . Here the vectora = (a1 , . . . , aN )T , and we have deﬁned

1 Tw φ(xn ) − tn .
(6.4)
λ
Instead of working with the parameter vector w, we can now reformulate the leastsquares algorithm in terms of the parameter vector a, giving rise to a dual representation. If we substitute w = ΦT a into J(w), we obtainan = −

J(a) =1 T
1λa ΦΦT ΦΦT a − aT ΦΦT t + tT t + aT ΦΦT a
2
2
2

(6.5)

where t = (t1 , . . . , tN )T . We now deﬁne the Gram matrix K = ΦΦT , which is an
N × N symmetric matrix with elements
Knm = φ(xn )T φ(xm ) = k(xn , xm )(6.6)

where we have introduced the kernel function k(x, x ) deﬁned by (6.1). In terms of
the Gram matrix, the sum-of-squares error function can be written as
J(a) =

1λ1 T
a KKa − aT Kt + tT t + aT Ka.
2
2
2

(6.7)

Setting the gradient of J(a) with respect to a to zero, we obtain the following solution−1
a = (K + λIN ) t.
(6.8)

294

6. KERNEL METHODS
If we substitute this back into the linear regression model, we obtain the following
prediction for a new input x−1y(x) = wT φ(x) = aT Φφ(x) = k(x)T (K + λIN )Exercise 6.1

Exercise 6.2

t

(6.9)

where we have deﬁned the vector k(x) with elements kn (x) = k(xn , x). Thus we
see that the dual formulation allows the solution to the least-squares problem to be
expressed entirely in terms of the kernel function k(x, x ). This is known as a dual
formulation because, by noting that the solution for a can be expressed as a linear
combination of the elements of φ(x), we recover the original formulation in terms of
the parameter vector w. Note that the prediction at x is given by a linear combinationof the target values from the training set. In fact, we have already obtained this result,using a slightly different notation, in Section 3.3.3.
In the dual formulation, we determine the parameter vector a by inverting an
N × N matrix, whereas in the original parameter space formulation we had to invert
an M × M matrix in order to determine w. Because N is typically much larger
than M , the dual formulation does not seem to be particularly useful. However, theadvantage of the dual formulation, as we shall see, is that it is expressed entirely interms of the kernel function k(x, x ). We can therefore work directly in terms ofkernels and avoid the explicit introduction of the feature vector φ(x), which allows
us implicitly to use feature spaces of high, even inﬁnite, dimensionality.
The existence of a dual representation based on the Gram matrix is a property ofmany linear models, including the perceptron. In Section 6.4, we will develop a duality between probabilistic linear models for regression and the technique of Gaussian
processes. Duality will also play an important role when we discuss support vector
machines in Chapter 7.

6.2. Constructing Kernels
In order to exploit kernel substitution, we need to be able to construct valid kernel
functions. One approach is to choose a feature space mapping φ(x) and then use
this to ﬁnd the corresponding kernel, as is illustrated in Figure 6.1. Here the kernel
function is deﬁned for a one-dimensional input space byk(x, x ) = φ(x)T φ(x ) =Mφi (x)φi (x )(6.10)

i=1

where φi (x) are the basis functions.
An alternative approach is to construct kernel functions directly. In this case,
we must ensure that the function we choose is a valid kernel, in other words that itcorresponds to a scalar product in some (perhaps inﬁnite dimensional) feature space.
As a simple example, consider a kernel function given byk(x, z) = xT z

2.

(6.11)

6.2. Constructing Kernels
1

1

1

0.5

0.75

0.75

0

0.5

0.5

−0.5

0.25

0.25

−1
−1

0

1

0

−1

0

1

0
−1

0.04

0.04

0.04

0.02

0.02

0.02

0

0

0

−1

0

1

−1

0

1

−1

295

0

1

0

1

Figure 6.1 Illustration of the construction of kernel functions starting from a corresponding set of basis functions. In each column the lower plot shows the kernel function k(x, x ) deﬁned by (6.10) plotted as a function ofx for x = 0, while the upper plot shows the corresponding basis functions given by polynomials (left column),‘Gaussians’ (centre column), and logistic sigmoids (right column).

If we take the particular case of a two-dimensional input space x = (x1 , x2 ) we
can expand out the terms and thereby identify the corresponding nonlinear feature
mappingk(x, z) =Appendix C

xT z

2

= (x1 z1 + x2 z2 )2= x21 z12 + 2x1 z1 x2 z2 + x22 z22√√= (x21 , 2x1 x2 , x22 )(z12 , 2z1 z2 , z22 )T= φ(x)T φ(z).
(6.12)√We see that the feature mapping takes the form φ(x) = (x21 , 2x1 x2 , x22 )T and
therefore comprises all possible second order terms, with a speciﬁc weighting between them.
More generally, however, we need a simple way to test whether a function constitutes a valid kernel without having to construct the function φ(x) explicitly. A
necessary and sufﬁcient condition for a function k(x, x ) to be a valid kernel (ShaweTaylor and Cristianini, 2004) is that the Gram matrix K, whose elements are given byk(xn , xm ), should be positive semideﬁnite for all possible choices of the set {xn }.
Note that a positive semideﬁnite matrix is not the same thing as a matrix whose
elements are nonnegative.
One powerful technique for constructing new kernels is to build them out of
simpler kernels as building blocks. This can be done using the following properties:

296

6. KERNEL METHODSTechniques for Constructing New Kernels.
Given valid kernels k1 (x, x ) and k2 (x, x ), the following new kernels will also
be valid:k(x, x )k(x, x )k(x, x )k(x, x )k(x, x )k(x, x )k(x, x )k(x, x )k(x, x )k(x, x )

=
=
=
=
=
=
=
=
=
=

ck1 (x, x )f (x)k1 (x, x )f (x )q (k1 (x, x ))exp (k1 (x, x ))k1 (x, x ) + k2 (x, x )k1 (x, x )k2 (x, x )k3 (φ(x), φ(x ))xT Ax
ka (xa , xa ) + kb (xb , xb )ka (xa , xa )kb (xb , xb )(6.13)(6.14)(6.15)(6.16)(6.17)(6.18)(6.19)(6.20)(6.21)(6.22)

where c > 0 is a constant, f (·) is any function, q(·) is a polynomial with nonnegative coefﬁcients, φ(x) is a function from x to RM , k3 (·, ·) is a valid kernel in
RM , A is a symmetric positive semideﬁnite matrix, xa and xb are variables (notnecessarily disjoint) with x = (xa , xb ), and ka and kb are valid kernel functions
over their respective spaces.
Equipped with these properties, we can now embark on the construction of more
complex kernels appropriate to speciﬁc applications. We require that the kernelk(x, x ) be symmetric and positive semideﬁnite and that it expresses the appropriate
form of similarity between x and x according to the intended application. Here we
consider a few common examples of kernel functions. For a more extensive discussion of ‘kernel engineering’, see Shawe-Taylor and Cristianini (2004).
2
We saw that the simple polynomial kernel k(x, x ) = xT x contains only
terms of degree two. If we consider the slightly generalized kernel k(x, x ) =2
xT x + c with c > 0, then the corresponding feature mapping φ(x) contains conM

stant and linear terms as well as terms of order two. Similarly, k(x, x ) = xT x
contains all monomials of order M . For instance, if x and x are two images, then
the kernel represents a particular weighted sum of all possible products of M pixels
in the ﬁrst image with M pixels in the second image. This can similarly be generM
alized to include all terms up to degree M by considering k(x, x ) = xT x + cwith c > 0. Using the results (6.17) and (6.18) for combining kernels we see thatthese will all be valid kernel functions.
Another commonly used kernel takes the formk(x, x ) = exp −x − x 2 /2σ 2(6.23)and is often called a ‘Gaussian’ kernel. Note, however, that in this context it is
not interpreted as a probability density, and hence the normalization coefﬁcient is6.2. Constructing Kernels

297

omitted. We can see that this is a valid kernel by expanding the squarex − x 2 = xT x + (x )T x − 2xT x

(6.24)

to givek(x, x ) = exp −xT x/2σ 2 exp xT x /σ 2 exp −(x )T x /2σ 2

Exercise 6.11

(6.25)and then making use of (6.14) and (6.16), together with the validity of the linear
kernel k(x, x ) = xT x . Note that the feature vector that corresponds to the Gaussian
kernel has inﬁnite dimensionality.
The Gaussian kernel is not restricted to the use of Euclidean distance. If we use
kernel substitution in (6.24) to replace xT x with a nonlinear kernel κ(x, x ), we
obtaink(x, x ) = exp −1(κ(x, x) + κ(x , x ) − 2κ(x, x )) .
2σ 2(6.26)

An important contribution to arise from the kernel viewpoint has been the extension to inputs that are symbolic, rather than simply vectors of real numbers. Kernel
functions can be deﬁned over objects as diverse as graphs, sets, strings, and text documents. Consider, for instance, a ﬁxed set and deﬁne a nonvectorial space consistingof all possible subsets of this set. If A1 and A2 are two such subsets then one simple
choice of kernel would bek(A1 , A2 ) = 2|A1 ∩A2 |Exercise 6.12

(6.27)

where A1 ∩ A2 denotes the intersection of sets A1 and A2 , and |A| denotes thenumber of subsets in A. This is a valid kernel function because it can be shown tocorrespond to an inner product in a feature space.
One powerful approach to the construction of kernels starts from a probabilistic
generative model (Haussler, 1999), which allows us to apply generative models in a
discriminative setting. Generative models can deal naturally with missing data andin the case of hidden Markov models can handle sequences of varying length. By
contrast, discriminative models generally give better performance on discriminative
tasks than generative models. It is therefore of some interest to combine these two
approaches (Lasserre et al., 2006). One way to combine them is to use a generativemodel to deﬁne a kernel, and then use this kernel in a discriminative approach.
Given a generative model p(x) we can deﬁne a kernel byk(x, x ) = p(x)p(x ).

(6.28)

This is clearly a valid kernel function because we can interpret it as an inner productin the one-dimensional feature space deﬁned by the mapping p(x). It says that two
inputs x and x are similar if they both have high probabilities. We can use (6.13) and(6.17) to extend this class of kernels by considering sums over products of different
probability distributions, with positive weighting coefﬁcients p(i), of the formp(x|i)p(x |i)p(i).
(6.29)
k(x, x ) =i

298

6. KERNEL METHODS

Section 9.2

Section 13.2

This is equivalent, up to an overall multiplicative constant, to a mixture distributionin which the components factorize, with the index i playing the role of a ‘latent’variable. Two inputs x and x will give a large value for the kernel function, andhence appear similar, if they have signiﬁcant probability under a range of different
components. Taking the limit of an inﬁnite sum, we can also consider kernels of theformk(x, x ) = p(x|z)p(x |z)p(z) dz(6.30)
where z is a continuous latent variable.
Now suppose that our data consists of ordered sequences of length L so thatan observation is given by X = {x1 , . . . , xL }. A popular generative model forsequences is the hidden Markov model, which expresses the distribution p(X) as amarginalization over a corresponding sequence of hidden states Z = {z1 , . . . , zL }.
We can use this approach to deﬁne a kernel function measuring the similarity of two
sequences X and X by extending the mixture representation (6.29) to givek(X, X ) =p(X|Z)p(X |Z)p(Z)(6.31)
Z

so that both observed sequences are generated by the same hidden sequence Z. Thismodel can easily be extended to allow sequences of differing length to be compared.
An alternative technique for using generative models to deﬁne kernel functions
is known as the Fisher kernel (Jaakkola and Haussler, 1999). Consider a parametric
generative model p(x|θ) where θ denotes the vector of parameters. The goal is toﬁnd a kernel that measures the similarity of two input vectors x and x induced by thegenerative model. Jaakkola and Haussler (1999) consider the gradient with respectto θ, which deﬁnes a vector in a ‘feature’ space having the same dimensionality as
θ. In particular, they consider the Fisher scoreg(θ, x) = ∇θ ln p(x|θ)(6.32)

from which the Fisher kernel is deﬁned byk(x, x ) = g(θ, x)T F−1 g(θ, x ).
Here F is the Fisher information matrix, given by
F = Ex g(θ, x)g(θ, x)T

Exercise 6.13

(6.33)(6.34)

where the expectation is with respect to x under the distribution p(x|θ). This canbe motivated from the perspective of information geometry (Amari, 1998), which
considers the differential geometry of the space of model parameters. Here we simply note that the presence of the Fisher information matrix causes this kernel to beinvariant under a nonlinear re-parameterization of the density model θ → ψ(θ).
In practice, it is often infeasible to evaluate the Fisher information matrix. One
approach is simply to replace the expectation in the deﬁnition of the Fisher information with the sample average, giving
1 g(θ, xn )g(θ, xn )T .
N
N

F

n=1

(6.35)

6.3. Radial Basis Function Networks

Section 12.1.3

299

This is the covariance matrix of the Fisher scores, and so the Fisher kernel corresponds to a whitening of these scores. More simply, we can just omit the Fisher
information matrix altogether and use the noninvariant kernelk(x, x ) = g(θ, x)T g(θ, x ).

(6.36)

An application of Fisher kernels to document retrieval is given by Hofmann (2000).
A ﬁnal example of a kernel function is the sigmoidal kernel given byk(x, x ) = tanh axT x + b

Section 6.4.7(6.37)

whose Gram matrix in general is not positive semideﬁnite. This form of kernelhas, however, been used in practice (Vapnik, 1995), possibly because it gives kernel
expansions such as the support vector machine a superﬁcial resemblance to neuralnetwork models. As we shall see, in the limit of an inﬁnite number of basis functions,a Bayesian neural network with an appropriate prior reduces to a Gaussian process,thereby providing a deeper link between neural networks and kernel methods.

6.3. Radial Basis Function Networks
In Chapter 3, we discussed regression models based on linear combinations of ﬁxed
basis functions, although we did not discuss in detail what form those basis functions
might take. One choice that has been widely used is that of radial basis functions,which have the property that each basis function depends only on the radial distance(typically Euclidean) from a centre µj , so that φj (x) = h(x − µj ).
Historically, radial basis functions were introduced for the purpose of exact function interpolation (Powell, 1987). Given a set of input vectors {x1 , . . . , xN } alongwith corresponding target values {t1 , . . . , tN }, the goal is to ﬁnd a smooth function
f (x) that ﬁts every target value exactly, so that f (xn ) = tn for n = 1, . . . , N . Thisis achieved by expressing f (x) as a linear combination of radial basis functions, one
centred on every data point
f (x) =N

wn h(x − xn ).

(6.38)

n=1

The values of the coefﬁcients {wn } are found by least squares, and because there
are the same number of coefﬁcients as there are constraints, the result is a functionthat ﬁts every target value exactly. In pattern recognition applications, however, thetarget values are generally noisy, and exact interpolation is undesirable because thiscorresponds to an over-ﬁtted solution.
Expansions in radial basis functions also arise from regularization theory (Poggio and Girosi, 1990; Bishop, 1995a). For a sum-of-squares error function with aregularizer deﬁned in terms of a differential operator, the optimal solution is given
by an expansion in the Green’s functions of the operator (which are analogous to theeigenvectors of a discrete matrix), again with one basis function centred on each data

300

6. KERNEL METHODS
point. If the differential operator is isotropic then the Green’s functions depend only
on the radial distance from the corresponding data point. Due to the presence of theregularizer, the solution no longer interpolates the training data exactly.
Another motivation for radial basis functions comes from a consideration of
the interpolation problem when the input (rather than the target) variables are noisy(Webb, 1994; Bishop, 1995a). If the noise on the input variable x is describedby a variable ξ having a distribution ν(ξ), then the sum-of-squares error function
becomesN 
12E={y(xn + ξ) − tn } ν(ξ) dξ.
(6.39)
2
n=1

Appendix D
Exercise 6.17

Using the calculus of variations, we can optimize with respect to the function f (x)to giveNtn h(x − xn )(6.40)y(xn ) =n=1

where the basis functions are given byh(x − xn ) =ν(x − xn )N.

(6.41)

ν(x − xn )n=1

We see that there is one basis function centred on every data point. This is known as
the Nadaraya-Watson model and will be derived again from a different perspective
in Section 6.3.1. If the noise distribution ν(ξ) is isotropic, so that it is a function
only of ξ, then the basis functions will be radial.

Note that the basis functions (6.41) are normalized, so that n h(x − xn ) = 1
for any value of x. The effect of such normalization is shown in Figure 6.2. Normalization is sometimes used in practice as it avoids having regions of input space where
all of the basis functions take small values, which would necessarily lead to predictions in such regions that are either small or controlled purely by the bias parameter.
Another situation in which expansions in normalized radial basis functions arise
is in the application of kernel density estimation to the problem of regression, as weshall discuss in Section 6.3.1.
Because there is one basis function associated with every data point, the corresponding model can be computationally costly to evaluate when making predictions
for new data points. Models have therefore been proposed (Broomhead and Lowe,1988; Moody and Darken, 1989; Poggio and Girosi, 1990), which retain the expansion in radial basis functions but where the number M of basis functions is smaller
than the number N of data points. Typically, the number of basis functions, and thelocations µi of their centres, are determined based on the input data {xn } alone. The
basis functions are then kept ﬁxed and the coefﬁcients {wi } are determined by leastsquares by solving the usual set of linear equations, as discussed in Section 3.1.1.

6.3. Radial Basis Function Networks
1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0
−1

−0.5

0

0.5

1

0
−1

−0.5

0

0.5

301

1

Figure 6.2 Plot of a set of Gaussian basis functions on the left, together with the corresponding normalizedbasis functions on the right.

Section 9.1

One of the simplest ways of choosing basis function centres is to use a randomly
chosen subset of the data points. A more systematic approach is called orthogonal
least squares (Chen et al., 1991). This is a sequential selection process in which ateach step the next data point to be chosen as a basis function centre corresponds tothe one that gives the greatest reduction in the sum-of-squares error. Values for the
expansion coefﬁcients are determined as part of the algorithm. Clustering algorithms
such as K-means have also been used, which give a set of basis function centres thatno longer coincide with training data points.

6.3.1 Nadaraya-Watson model

Section 2.5.1

In Section 3.3.3, we saw that the prediction of a linear regression model for a
new input x takes the form of a linear combination of the training set target values
with coefﬁcients given by the ‘equivalent kernel’ (3.62) where the equivalent kernel
satisﬁes the summation constraint (3.64).
We can motivate the kernel regression model (3.61) from a different perspective,
starting with kernel density estimation. Suppose we have a training set {xn , tn } andwe use a Parzen density estimator to model the joint distribution p(x, t), so that1 f (x − xn , t − tn )NNp(x, t) =(6.42)

n=1

where f (x, t) is the component density function, and there is one such component
centred on each data point. We now ﬁnd an expression for the regression function
y(x), corresponding to the conditional average of the target variable conditioned on

302

6. KERNEL METHODS
the input variable, which is given by ∞tp(t|x) dt
y(x) = E[t|x] =−∞
tp(x, t) dt=p(x, t) dttf (x − xn , t − tn ) dt
=

n



.

(6.43)f (x − xm , t − tm ) dtm

We now assume for simplicity that the component density functions have zero meanso that ∞f (x, t)t dt = 0(6.44)−∞

for all values of x. Using a simple change of variable, we then obtaing(x − xn )tnn

y(x) =

g(x − xm )

m



=

k(x, xn )tn(6.45)nwhere n, m = 1, . . . , N and the kernel function k(x, xn ) is given byg(x − xn )k(x, xn ) = g(x − xm )(6.46)m ∞and we have deﬁned

f (x, t) dt.

g(x) =(6.47)−∞The result (6.45) is known as the Nadaraya-Watson model, or kernel regression(Nadaraya, 1964; Watson, 1964). For a localized kernel function, it has the property of giving more weight to the data points xn that are close to x. Note that thekernel (6.46) satisﬁes the summation constraintN
n=1

k(x, xn ) = 1.

6.4. Gaussian Processes

303

Figure 6.3 Illustration of the Nadaraya-Watson kernel
1.5
regression model using isotropic Gaussian kernels, for thesinusoidal data set. The original sine function is shown
1
by the green curve, the data points are shown in blue,and each is the centre of an isotropic Gaussian kernel.
The resulting regression function, given by the condi- 0.5
tional mean, is shown by the red line, along with the two0
standard-deviation region for the conditional distributionp(t|x) shown by the red shading. The blue ellipse aroundeach data point shows one standard deviation contour for −0.5
the corresponding kernel. These appear noncircular due−1
to the different scales on the horizontal and vertical axes.

−1.5
0

0.2

0.4

0.6

0.8

1

In fact, this model deﬁnes not only a conditional expectation but also a fullconditional distribution given byf (x − xn , t − tn )p(t, x)n= (6.48)
p(t|x) = p(t, x) dt
f (x − xm , t − tm ) dtm

Exercise 6.18

from which other expectations can be evaluated.
As an illustration we consider the case of a single input variable x in whichf (x, t) is given by a zero-mean isotropic Gaussian over the variable z = (x, t) withvariance σ 2 . The corresponding conditional distribution (6.48) is given by a Gaussian mixture, and is shown, together with the conditional mean, for the sinusoidal
synthetic data set in Figure 6.3.
An obvious extension of this model is to allow for more ﬂexible forms of Gaussian components, for instance having different variance parameters for the input andtarget variables. More generally, we could model the joint distribution p(t, x) using
a Gaussian mixture model, trained using techniques discussed in Chapter 9 (Ghahramani and Jordan, 1994), and then ﬁnd the corresponding conditional distributionp(t|x). In this latter case we no longer have a representation in terms of kernel functions evaluated at the training set data points. However, the number of components
in the mixture model can be smaller than the number of training set points, resulting
in a model that is faster to evaluate for test data points. We have thereby accepted anincreased computational cost during the training phase in order to have a model thatis faster at making predictions.

6.4. Gaussian Processes
In Section 6.1, we introduced kernels by applying the concept of duality to a nonprobabilistic model for regression. Here we extend the role of kernels to probabilis-

304

6. KERNEL METHODS
tic discriminative models, leading to the framework of Gaussian processes. We shallthereby see how kernels arise naturally in a Bayesian setting.
In Chapter 3, we considered linear regression models of the form y(x, w) =wT φ(x) in which w is a vector of parameters and φ(x) is a vector of ﬁxed nonlinear
basis functions that depend on the input vector x. We showed that a prior distribution
over w induced a corresponding prior distribution over functions y(x, w). Given a
training data set, we then evaluated the posterior distribution over w and thereby
obtained the corresponding posterior distribution over regression functions, which
in turn (with the addition of noise) implies a predictive distribution p(t|x) for new
input vectors x.
In the Gaussian process viewpoint, we dispense with the parametric model andinstead deﬁne a prior probability distribution over functions directly. At ﬁrst sight, it
might seem difﬁcult to work with a distribution over the uncountably inﬁnite space offunctions. However, as we shall see, for a ﬁnite training set we only need to consider
the values of the function at the discrete set of input values xn corresponding to thetraining set and test set data points, and so in practice we can work in a ﬁnite space.
Models equivalent to Gaussian processes have been widely studied in many different ﬁelds. For instance, in the geostatistics literature Gaussian process regression
is known as kriging (Cressie, 1993). Similarly, ARMA (autoregressive moving average) models, Kalman ﬁlters, and radial basis function networks can all be viewed asforms of Gaussian process models. Reviews of Gaussian processes from a machinelearning perspective can be found in MacKay (1998), Williams (1999), and MacKay(2003), and a comparison of Gaussian process models with alternative approaches isgiven in Rasmussen (1996). See also Rasmussen and Williams (2006) for a recent
textbook on Gaussian processes.

6.4.1 Linear regression revisited
In order to motivate the Gaussian process viewpoint, let us return to the linear
regression example and re-derive the predictive distribution by working in termsof distributions over functions y(x, w). This will provide a speciﬁc example of a
Gaussian process.
Consider a model deﬁned in terms of a linear combination of M ﬁxed basis
functions given by the elements of the vector φ(x) so thaty(x) = wT φ(x)(6.49)

where x is the input vector and w is the M -dimensional weight vector. Now consider
a prior distribution over w given by an isotropic Gaussian of the formp(w) = N (w|0, α−1 I)(6.50)

governed by the hyperparameter α, which represents the precision (inverse variance)of the distribution. For any given value of w, the deﬁnition (6.49) deﬁnes a particular function of x. The probability distribution over w deﬁned by (6.50) therefore
induces a probability distribution over functions y(x). In practice, we wish to evaluate this function at speciﬁc values of x, for example at the training data points

6.4. Gaussian Processes

305

x1 , . . . , xN . We are therefore interested in the joint distribution of the function values y(x1 ), . . . , y(xN ), which we denote by the vector y with elements yn = y(xn )for n = 1, . . . , N . From (6.49), this vector is given byy = Φw

Exercise 2.31

(6.51)

where Φ is the design matrix with elements Φnk = φk (xn ). We can ﬁnd the probability distribution of y as follows. First of all we note that y is a linear combination ofGaussian distributed variables given by the elements of w and hence is itself Gaussian. We therefore need only to ﬁnd its mean and covariance, which are given from(6.50) by
E[y] = ΦE[w] = 0(6.52)
 T
1(6.53)
cov[y] = E yy = ΦE wwT ΦT = ΦΦT = K
αwhere K is the Gram matrix with elements1
Knm = k(xn , xm ) = φ(xn )T φ(xm )(6.54)
αand k(x, x ) is the kernel function.
This model provides us with a particular example of a Gaussian process. In general, a Gaussian process is deﬁned as a probability distribution over functions y(x)such that the set of values of y(x) evaluated at an arbitrary set of points x1 , . . . , xN
jointly have a Gaussian distribution. In cases where the input vector x is two dimensional, this may also be known as a Gaussian random ﬁeld. More generally, astochastic process y(x) is speciﬁed by giving the joint probability distribution forany ﬁnite set of values y(x1 ), . . . , y(xN ) in a consistent manner.
A key point about Gaussian stochastic processes is that the joint distribution
over N variables y1 , . . . , yN is speciﬁed completely by the second-order statistics,
namely the mean and the covariance. In most applications, we will not have any
prior knowledge about the mean of y(x) and so by symmetry we take it to be zero.
This is equivalent to choosing the mean of the prior over weight values p(w|α) tobe zero in the basis function viewpoint. The speciﬁcation of the Gaussian process isthen completed by giving the covariance of y(x) evaluated at any two values of x,
which is given by the kernel function
E [y(xn )y(xm )] = k(xn , xm ).

(6.55)For the speciﬁc case of a Gaussian process deﬁned by the linear regression model(6.49) with a weight prior (6.50), the kernel function is given by (6.54).
We can also deﬁne the kernel function directly, rather than indirectly through achoice of basis function. Figure 6.4 shows samples of functions drawn from Gaussian processes for two different choices of kernel function. The ﬁrst of these is a
‘Gaussian’ kernel of the form (6.23), and the second is the exponential kernel givenby(6.56)
k(x, x ) = exp (−θ |x − x |)which corresponds to the Ornstein-Uhlenbeck process originally introduced by Uhlenbeck and Ornstein (1930) to describe Brownian motion.

306

6. KERNEL METHODS

Figure 6.4 Samples from Gaussian processes for a ‘Gaussian’ kernel (left) and an exponential kernel(right).

3

3

1.5

1.5

0

0

−1.5

−1.5

−3
−1

−0.5

0

0.5

1

−3
−1

−0.5

0

0.5

1

6.4.2 Gaussian processes for regression
In order to apply Gaussian process models to the problem of regression, we needto take account of the noise on the observed target values, which are given byt n = y n + n

(6.57)

where yn = y(xn ), and n is a random noise variable whose value is chosen independently for each observation n. Here we shall consider noise processes that havea Gaussian distribution, so thatp(tn |yn ) = N (tn |yn , β −1 )(6.58)

where β is a hyperparameter representing the precision of the noise. Because thenoise is independent for each data point, the joint distribution of the target valuest = (t1 , . . . , tN )T conditioned on the values of y = (y1 , . . . , yN )T is given by anisotropic Gaussian of the formp(t|y) = N (t|y, β −1 IN )(6.59)

where IN denotes the N × N unit matrix. From the deﬁnition of a Gaussian process,the marginal distribution p(y) is given by a Gaussian whose mean is zero and whose
covariance is deﬁned by a Gram matrix K so thatp(y) = N (y|0, K).

(6.60)The kernel function that determines K is typically chosen to express the property
that, for points xn and xm that are similar, the corresponding values y(xn ) andy(xm ) will be more strongly correlated than for dissimilar points. Here the notionof similarity will depend on the application.
In order to ﬁnd the marginal distribution p(t), conditioned on the input values
x1 , . . . , xN , we need to integrate over y. This can be done by making use of theresults from Section 2.3.3 for the linear-Gaussian model. Using (2.115), we see thatthe marginal distribution of t is given by(6.61)
p(t) = p(t|y)p(y) dy = N (t|0, C)

6.4. Gaussian Processes

307

where the covariance matrix C has elements
C(xn , xm ) = k(xn , xm ) + β −1 δnm .

(6.62)

This result reﬂects the fact that the two Gaussian sources of randomness, namelythat associated with y(x) and that associated with , are independent and so theircovariances simply add.
One widely used kernel function for Gaussian process regression is given by the
exponential of a quadratic form, with the addition of constant and linear terms togivek(xn , xm ) = θ0 exp −θ1xn − xm 22+ θ2 + θ3 xT
n xm .

(6.63)

Note that the term involving θ3 corresponds to a parametric model that is a linear
function of the input variables. Samples from this prior are plotted for various valuesof the parameters θ0 , . . . , θ3 in Figure 6.5, and Figure 6.6 shows a set of points sampled from the joint distribution (6.60) along with the corresponding values deﬁnedby (6.61).
So far, we have used the Gaussian process viewpoint to build a model of thejoint distribution over sets of data points. Our goal in regression, however, is tomake predictions of the target variables for new inputs, given a set of training data.
Let us suppose that tN = (t1 , . . . , tN )T , corresponding to input values x1 , . . . , xN ,comprise the observed training set, and our goal is to predict the target variable tN +1
for a new input vector xN +1 . This requires that we evaluate the predictive distribution p(tN +1 |tN ). Note that this distribution is conditioned also on the variables
x1 , . . . , xN and xN +1 . However, to keep the notation simple we will not show these
conditioning variables explicitly.
To ﬁnd the conditional distribution p(tN +1 |t), we begin by writing down the
joint distribution p(tN +1 ), where tN +1 denotes the vector (t1 , . . . , tN , tN +1 )T . We
then apply the results from Section 2.3.1 to obtain the required conditional distribution, as illustrated in Figure 6.7.
From (6.61), the joint distribution over t1 , . . . , tN +1 will be given byp(tN +1 ) = N (tN +1 |0, CN +1 )(6.64)

where CN +1 is an (N + 1) × (N + 1) covariance matrix with elements given by(6.62). Because this joint distribution is Gaussian, we can apply the results from
Section 2.3.1 to ﬁnd the conditional Gaussian distribution. To do this, we partition
the covariance matrix as follows

CN k(6.65)
CN +1 =
kT c
where CN is the N × N covariance matrix with elements given by (6.62) for n, m =1, . . . , N , the vector k has elements k(xn , xN +1 ) for n = 1, . . . , N , and the scalar

308

6. KERNEL METHODS(1.00, 4.00, 0.00, 0.00)(9.00, 4.00, 0.00, 0.00)(1.00, 64.00, 0.00, 0.00)

3

9

3

1.5

4.5

1.5

0

0

0

−1.5

−4.5

−1.5

−3
−1

−0.5

0

0.5

1

−9
−1

(1.00, 0.25, 0.00, 0.00)

−0.5

0

0.5

−3
−1

1

9

4

1.5

4.5

2

0

0

0

−1.5

−4.5

−2

−0.5

0

0.5

1

−9
−1

−0.5

0

0.5

0

0.5

1

(1.00, 4.00, 0.00, 5.00)(1.00, 4.00, 10.00, 0.00)

3

−3
−1

−0.5

−4
−1

1

−0.5

0

0.5

1

Figure 6.5 Samples from a Gaussian process prior deﬁned by the covariance function (6.63). The title above
each plot denotes (θ0 , θ1 , θ2 , θ3 ).

c = k(xN +1 , xN +1 ) + β −1 . Using the results (2.81) and (2.82), we see that the conditional distribution p(tN +1 |t) is a Gaussian distribution with mean and covariancegiven by1
m(xN +1 ) = kT C−
N t(6.66)σ (xN +1 ) = c − k(6.67)

2

T

1
C−
N k.

These are the key results that deﬁne Gaussian process regression. Because the vector
k is a function of the test point input value xN +1 , we see that the predictive distribution is a Gaussian whose mean and variance both depend on xN +1 . An example ofGaussian process regression is shown in Figure 6.8.
The only restriction on the kernel function is that the covariance matrix given by(6.62) must be positive deﬁnite. If λi is an eigenvalue of K, then the corresponding
eigenvalue of C will be λi + β −1 . It is therefore sufﬁcient that the kernel matrixk(xn , xm ) be positive semideﬁnite for any pair of points xn and xm , so that λi  0,
because any eigenvalue λi that is zero will still give rise to a positive eigenvalue
for C because β > 0. This is the same restriction on the kernel function discussed
earlier, and so we can again exploit all of the techniques in Section 6.2 to construct309

6.4. Gaussian Processes
Figure 6.6

Illustration of the sampling of data
points {tn } from a Gaussian process.
The blue curve shows a sample function from the Gaussian process priorover functions, and the red points
show the values of yn obtained byevaluating the function at a set of input values {xn }. The corresponding values of {tn }, shown in green,
are obtained by adding independent
Gaussian noise to each of the {yn }.

3

t

0

−3
−1

0

x

1

suitable kernels.
Note that the mean (6.66) of the predictive distribution can be written, as a function of xN +1 , in the formm(xN +1 ) =Nan k(xn , xN +1 )(6.68)

n=1

Exercise 6.21

1
where an is the nth component of C−
N t. Thus, if the kernel function k(xn , xm )depends only on the distance xn − xm , then we obtain an expansion in radialbasis functions.
The results (6.66) and (6.67) deﬁne the predictive distribution for Gaussian process regression with an arbitrary kernel function k(xn , xm ). In the particular case inwhich the kernel function k(x, x ) is deﬁned in terms of a ﬁnite set of basis functions,
we can derive the results obtained previously in Section 3.3.2 for linear regression
starting from the Gaussian process viewpoint.
For such models, we can therefore obtain the predictive distribution either bytaking a parameter space viewpoint and using the linear regression result or by taking
a function space viewpoint and using the Gaussian process result.
The central computational operation in using Gaussian processes will involvethe inversion of a matrix of size N × N , for which standard methods require O(N 3 )computations. By contrast, in the basis function model we have to invert a matrix
SN of size M × M , which has O(M 3 ) computational complexity. Note that for
both viewpoints, the matrix inversion must be performed once for the given training
set. For each new test point, both methods require a vector-matrix multiply, which
has cost O(N 2 ) in the Gaussian process case and O(M 2 ) for the linear basis function model. If the number M of basis functions is smaller than the number N ofdata points, it will be computationally more efﬁcient to work in the basis function

310

6. KERNEL METHODS
Figure 6.7

Illustration of the mechanism ofGaussian process regression forthe case of one training point andone test point, in which the red ellipses show contours of the joint distribution p(t1 , t2 ). Here t1 is thetraining data point, and conditioning on the value of t1 , corresponding to the vertical blue line, we obtain p(t2 |t1 ) shown as a function of
t2 by the green curve.

t2

1
m(x2 )
0

t1

−1

−1

Exercise 6.23
Figure 6.8

01

framework. However, an advantage of a Gaussian processes viewpoint is that we
can consider covariance functions that can only be expressed in terms of an inﬁnite
number of basis functions.
For large training data sets, however, the direct application of Gaussian process
methods can become infeasible, and so a range of approximation schemes have beendeveloped that have better scaling with training set size than the exact approach(Gibbs, 1997; Tresp, 2001; Smola and Bartlett, 2001; Williams and Seeger, 2001;
Csató and Opper, 2002; Seeger et al., 2003). Practical issues in the application ofGaussian processes are discussed in Bishop and Nabney (2008).
We have introduced Gaussian process regression for the case of a single target variable. The extension of this formalism to multiple target variables, known
as co-kriging (Cressie, 1993), is straightforward. Various other extensions of GausIllustration of Gaussian process regression applied to the sinusoidal
1
data set in Figure A.6 in which thethree right-most data points havebeen omitted. The green curve 0.5
shows the sinusoidal function fromwhich the data points, shown in0blue, are obtained by sampling andaddition of Gaussian noise. The
red line shows the mean of the −0.5
Gaussian process predictive distribution, and the shaded region cor−1
responds to plus and minus twostandard deviations. Notice how
the uncertainty increases in the re0gion to the right of the data points.

0.2

0.4

0.6

0.8

1

6.4. Gaussian Processes

311

sian process regression have also been considered, for purposes such as modellingthe distribution over low-dimensional manifolds for unsupervised learning (Bishop
et al., 1998a) and the solution of stochastic differential equations (Graepel, 2003).

6.4.3 Learning the hyperparameters

Section 3.5

The predictions of a Gaussian process model will depend, in part, on the choiceof covariance function. In practice, rather than ﬁxing the covariance function, we
may prefer to use a parametric family of functions and then infer the parametervalues from the data. These parameters govern such things as the length scale of thecorrelations and the precision of the noise and correspond to the hyperparameters ina standard parametric model.
Techniques for learning the hyperparameters are based on the evaluation of thelikelihood function p(t|θ) where θ denotes the hyperparameters of the Gaussian process model. The simplest approach is to make a point estimate of θ by maximizing
the log likelihood function. Because θ represents a set of hyperparameters for the
regression problem, this can be viewed as analogous to the type 2 maximum likelihood procedure for linear regression models. Maximization of the log likelihood
can be done using efﬁcient gradient-based optimization algorithms such as conjugate
gradients (Fletcher, 1987; Nocedal and Wright, 1999; Bishop and Nabney, 2008).
The log likelihood function for a Gaussian process regression model is easily
evaluated using the standard form for a multivariate Gaussian distribution, giving
1N11ln(2π).
ln p(t|θ) = − ln |CN | − tT C−
N t−
2
2
2

(6.69)For nonlinear optimization, we also need the gradient of the log likelihood function with respect to the parameter vector θ. We shall assume that evaluation of thederivatives of CN is straightforward, as would be the case for the covariance functions considered in this chapter. Making use of the result (C.21) for the derivative of1
C−
N , together with the result (C.22) for the derivative of ln |CN |, we obtain


1
1
∂
−1 ∂CN1 ∂CN
ln p(t|θ) = − Tr CN+ tT C−C−1 t.
(6.70)
N
∂θi
2
∂θi
2
∂θi N
Because ln p(t|θ) will in general be a nonconvex function, it can have multiple maxima.
It is straightforward to introduce a prior over θ and to maximize the log posterior using gradient-based methods. In a fully Bayesian treatment, we need to evaluatemarginals over θ weighted by the product of the prior p(θ) and the likelihood function p(t|θ). In general, however, exact marginalization will be intractable, and wemust resort to approximations.
The Gaussian process regression model gives a predictive distribution whosemean and variance are functions of the input vector x. However, we have assumed
that the contribution to the predictive variance arising from the additive noise, governed by the parameter β, is a constant. For some problems, known as heteroscedastic, the noise variance itself will also depend on x. To model this, we can extend the

312

6. KERNEL METHODS

Figure 6.9 Samples from the ARD
prior for Gaussian processes, inwhich the kernel function is given by(6.71). The left plot corresponds toη1 = η2 = 1, and the right plot corresponds to η1 = 1, η2 = 0.01.

Gaussian process framework by introducing a second Gaussian process to represent
the dependence of β on the input x (Goldberg et al., 1998). Because β is a variance,and hence nonnegative, we use the Gaussian process to model ln β(x).

6.4.4 Automatic relevance determination
In the previous section, we saw how maximum likelihood could be used to determine a value for the correlation length-scale parameter in a Gaussian process.
This technique can usefully be extended by incorporating a separate parameter foreach input variable (Rasmussen and Williams, 2006). The result, as we shall see, isthat the optimization of these parameters by maximum likelihood allows the relative
importance of different inputs to be inferred from the data. This represents an example in the Gaussian process context of automatic relevance determination, or ARD,
which was originally formulated in the framework of neural networks (MacKay,
1994; Neal, 1996). The mechanism by which appropriate inputs are preferred is
discussed in Section 7.2.2.
Consider a Gaussian process with a two-dimensional input space x = (x1 , x2 ),having a kernel function of the form


2

1
k(x, x ) = θ0 exp −ηi (xi − xi )2 .
(6.71)
2
i=1

Samples from the resulting prior over functions y(x) are shown for two different
settings of the precision parameters ηi in Figure 6.9. We see that, as a particular parameter ηi becomes small, the function becomes relatively insensitive to thecorresponding input variable xi . By adapting these parameters to a data set using
maximum likelihood, it becomes possible to detect input variables that have little
effect on the predictive distribution, because the corresponding values of ηi will besmall. This can be useful in practice because it allows such inputs to be discarded.
ARD is illustrated using a simple synthetic data set having three inputs x1 , x2 and x3(Nabney, 2002) in Figure 6.10. The target variable t, is generated by sampling 100
values of x1 from a Gaussian, evaluating the function sin(2πx1 ), and then adding

6.4. Gaussian Processes
Figure 6.10

Illustration of automatic relevance determination in a Gaussian process for a synthetic problem having three inputs x1 , x2 ,and x3 , for which the curves
show the corresponding values of
the hyperparameters η1 (red), η2(green), and η3 (blue) as a function of the number of iterations
when optimizing the marginal
likelihood. Details are given in
the text. Note the logarithmic
scale on the vertical axis.

313

2

10

0

10

−2

10

−4

10

0

20

40

60

80

100

Gaussian noise. Values of x2 are given by copying the corresponding values of x1
and adding noise, and values of x3 are sampled from an independent Gaussian distribution. Thus x1 is a good predictor of t, x2 is a more noisy predictor of t, and x3
has only chance correlations with t. The marginal likelihood for a Gaussian processwith ARD parameters η1 , η2 , η3 is optimized using the scaled conjugate gradientsalgorithm. We see from Figure 6.10 that η1 converges to a relatively large value, η2converges to a much smaller value, and η3 becomes very small indicating that x3 isirrelevant for predicting t.
The ARD framework is easily incorporated into the exponential-quadratic kernel(6.63) to give the following form of kernel function, which has been found useful for
applications of Gaussian processes to a range of regression problems


D
D

1
2
ηi (xni − xmi ) + θ2 + θ3
xni xmi (6.72)k(xn , xm ) = θ0 exp −
2
i=1

i=1

where D is the dimensionality of the input space.

6.4.5 Gaussian processes for classiﬁcation
In a probabilistic approach to classiﬁcation, our goal is to model the posterior
probabilities of the target variable for a new input vector, given a set of training
data. These probabilities must lie in the interval (0, 1), whereas a Gaussian processmodel makes predictions that lie on the entire real axis. However, we can easily
adapt Gaussian processes to classiﬁcation problems by transforming the output of
the Gaussian process using an appropriate nonlinear activation function.
Consider ﬁrst the two-class problem with a target variable t ∈ {0, 1}. If we deﬁne a Gaussian process over a function a(x) and then transform the function using
a logistic sigmoid y = σ(a), given by (4.59), then we will obtain a non-Gaussianstochastic process over functions y(x) where y ∈ (0, 1). This is illustrated for thecase of a one-dimensional input space in Figure 6.11 in which the probability distri-314

6. KERNEL METHODS
10

1

5

0.75

0

0.5

−5

0.25

−10
−1

−0.5

0

0.5

1

0
−1

−0.5

0

0.5

1

Figure 6.11 The left plot shows a sample from a Gaussian process prior over functions a(x), and the right plotshows the result of transforming this sample using a logistic sigmoid function.

bution over the target variable t is then given by the Bernoulli distributionp(t|a) = σ(a)t (1 − σ(a))1−t .

(6.73)As usual, we denote the training set inputs by x1 , . . . , xN with corresponding
observed target variables t = (t1 , . . . , tN )T . We also consider a single test point
xN +1 with target value tN +1 . Our goal is to determine the predictive distributionp(tN +1 |t), where we have left the conditioning on the input variables implicit. To dothis we introduce a Gaussian process prior over the vector aN +1 , which has components a(x1 ), . . . , a(xN +1 ). This in turn deﬁnes a non-Gaussian process over tN +1 ,and by conditioning on the training data tN we obtain the required predictive distribution. The Gaussian process prior for aN +1 takes the formp(aN +1 ) = N (aN +1 |0, CN +1 ).

(6.74)

Unlike the regression case, the covariance matrix no longer includes a noise term
because we assume that all of the training data points are correctly labelled. However, for numerical reasons it is convenient to introduce a noise-like term governedby a parameter ν that ensures that the covariance matrix is positive deﬁnite. Thus
the covariance matrix CN +1 has elements given by
C(xn , xm ) = k(xn , xm ) + νδnm

(6.75)

where k(xn , xm ) is any positive semideﬁnite kernel function of the kind consideredin Section 6.2, and the value of ν is typically ﬁxed in advance. We shall assume thatthe kernel function k(x, x ) is governed by a vector θ of parameters, and we shalllater discuss how θ may be learned from the training data.
For two-class problems, it is sufﬁcient to predict p(tN +1 = 1|tN ) because thevalue of p(tN +1 = 0|tN ) is then given by 1 − p(tN +1 = 1|tN ). The required6.4. Gaussian Processes
predictive distribution is given byp(tN +1 = 1|tN ) = p(tN +1 = 1|aN +1 )p(aN +1 |tN ) daN +1

Section 2.3

Section 10.1

Section 10.7

315

(6.76)

where p(tN +1 = 1|aN +1 ) = σ(aN +1 ).
This integral is analytically intractable, and so may be approximated using sampling methods (Neal, 1997). Alternatively, we can consider techniques based on
an analytical approximation. In Section 4.5.2, we derived the approximate formula(4.153) for the convolution of a logistic sigmoid with a Gaussian distribution. We
can use this result to evaluate the integral in (6.76) provided we have a Gaussian
approximation to the posterior distribution p(aN +1 |tN ). The usual justiﬁcation for a
Gaussian approximation to a posterior distribution is that the true posterior will tend
to a Gaussian as the number of data points increases as a consequence of the central
limit theorem. In the case of Gaussian processes, the number of variables grows with
the number of data points, and so this argument does not apply directly. However, if
we consider increasing the number of data points falling in a ﬁxed region of x space,then the corresponding uncertainty in the function a(x) will decrease, again leading
asymptotically to a Gaussian (Williams and Barber, 1998).
Three different approaches to obtaining a Gaussian approximation have been
considered. One technique is based on variational inference (Gibbs and MacKay,2000) and makes use of the local variational bound (10.144) on the logistic sigmoid.
This allows the product of sigmoid functions to be approximated by a product ofGaussians thereby allowing the marginalization over aN to be performed analytically. The approach also yields a lower bound on the likelihood function p(tN |θ).
The variational framework for Gaussian process classiﬁcation can also be extended
to multiclass (K > 2) problems by using a Gaussian approximation to the softmax
function (Gibbs, 1997).
A second approach uses expectation propagation (Opper and Winther, 2000b;
Minka, 2001b; Seeger, 2003). Because the true posterior distribution is unimodal, as
we shall see shortly, the expectation propagation approach can give good results.

6.4.6 Laplace approximation
Section 4.4

The third approach to Gaussian process classiﬁcation is based on the Laplaceapproximation, which we now consider in detail. In order to evaluate the predictive
distribution (6.76), we seek a Gaussian approximation to the posterior distribution
over aN +1 , which, using Bayes’ theorem, is given byp(aN +1 |tN ) =p(aN +1 , aN |tN ) daN
1=
p(aN +1 , aN )p(tN |aN +1 , aN ) daNp(tN )
1=p(aN +1 |aN )p(aN )p(tN |aN ) daNp(tN )
=p(aN +1 |aN )p(aN |tN ) daN(6.77)

316

6. KERNEL METHODS
where we have used p(tN |aN +1 , aN ) = p(tN |aN ). The conditional distributionp(aN +1 |aN ) is obtained by invoking the results (6.66) and (6.67) for Gaussian process regression, to give1
T −1
p(aN +1 |aN ) = N (aN +1 |kT C−
N aN , c − k CN k).

(6.78)

We can therefore evaluate the integral in (6.77) by ﬁnding a Laplace approximation
for the posterior distribution p(aN |tN ), and then using the standard result for the
convolution of two Gaussian distributions.
The prior p(aN ) is given by a zero-mean Gaussian process with covariance matrix CN , and the data term (assuming independence of the data points) is given byp(tN |aN ) =

N


tn

1−tn

σ(an ) (1 − σ(an ))

n=1

=

N


ean tn σ(−an ).

(6.79)

n=1

We then obtain the Laplace approximation by Taylor expanding the logarithm ofp(aN |tN ), which up to an additive normalization constant is given by the quantity
Ψ(aN ) = ln p(aN ) + ln p(tN |aN )1N1ln(2π) − ln |CN | + tT
C−1 aN −= − aT
N aN
2 N N
2
2
N

−
ln(1 + ean ) + const.

(6.80)

n=1

First we need to ﬁnd the mode of the posterior distribution, and this requires that weevaluate the gradient of Ψ(aN ), which is given by1∇Ψ(aN ) = tN − σ N − C−N aNSection 4.3.3

where σ N is a vector with elements σ(an ). We cannot simply ﬁnd the mode bysetting this gradient to zero, because σ N depends nonlinearly on aN , and so weresort to an iterative scheme based on the Newton-Raphson method, which gives riseto an iterative reweighted least squares (IRLS) algorithm. This requires the second
derivatives of Ψ(aN ), which we also require for the Laplace approximation anyway,
and which are given by1∇∇Ψ(aN ) = −WN − C−N

Exercise 6.24

(6.81)(6.82)

where WN is a diagonal matrix with elements σ(an )(1 − σ(an )), and we have used
the result (4.88) for the derivative of the logistic sigmoid function. Note that these
diagonal elements lie in the range (0, 1/4), and hence WN is a positive deﬁnite
matrix. Because CN (and hence its inverse) is positive deﬁnite by construction, andbecause the sum of two positive deﬁnite matrices is also positive deﬁnite, we seethat the Hessian matrix A = −∇∇Ψ(aN ) is positive deﬁnite and so the posterior
distribution p(aN |tN ) is log convex and therefore has a single mode that is the global

6.4. Gaussian Processes

Exercise 6.25

317

maximum. The posterior distribution is not Gaussian, however, because the Hessian
is a function of aN .
Using the Newton-Raphson formula (4.92), the iterative update equation for aN
is given by−1{tN − σ N + WN aN } .
anew
N = CN (I + WN CN )(6.83)

These equations are iterated until they converge to the mode which we denote byaN . At the mode, the gradient ∇Ψ(aN ) will vanish, and hence aN will satisfyaN = CN (tN − σ N ).

(6.84)

Once we have found the mode aN of the posterior, we can evaluate the Hessian
matrix given by1(6.85)
H = −∇∇Ψ(aN ) = WN + C−
N
where the elements of WN are evaluated using aN . This deﬁnes our Gaussian approximation to the posterior distribution p(aN |tN ) given by
q(aN ) = N (aN |aN , H−1 ).

Exercise 6.26

(6.86)

We can now combine this with (6.78) and hence evaluate the integral (6.77). Because
this corresponds to a linear-Gaussian model, we can use the general result (2.115) togive
E[aN +1 |tN ] = kT (tN − σ N )−1+ CN )−1 k.
var[aN +1 |tN ] = c − kT (WN(6.87)(6.88)

Now that we have a Gaussian distribution for p(aN +1 |tN ), we can approximate
the integral (6.76) using the result (4.153). As with the Bayesian logistic regression
model of Section 4.5, if we are only interested in the decision boundary corresponding to p(tN +1 |tN ) = 0.5, then we need only consider the mean and we can ignore
the effect of the variance.
We also need to determine the parameters θ of the covariance function. One
approach is to maximize the likelihood function given by p(tN |θ) for which we needexpressions for the log likelihood and its gradient. If desired, suitable regularization
terms can also be added, leading to a penalized maximum likelihood solution. The
likelihood function is deﬁned byp(tN |θ) = p(tN |aN )p(aN |θ) daN .
(6.89)
This integral is analytically intractable, so again we make use of the Laplace approximation. Using the result (4.135), we obtain the following approximation for the logof the likelihood functionln p(tN |θ) = Ψ(aN ) −1
N1
ln |WN + C−
ln(2π)
N |+2
2(6.90)

318

6. KERNEL METHODS
where Ψ(aN ) = ln p(aN |θ) + ln p(tN |aN ). We also need to evaluate the gradientof ln p(tN |θ) with respect to the parameter vector θ. Note that changes in θ willcause changes in aN , leading to additional terms in the gradient. Thus, when we
differentiate (6.90) with respect to θ, we obtain two sets of terms, the ﬁrst arisingfrom the dependence of the covariance matrix CN on θ, and the rest arising fromdependence of aN on θ.
The terms arising from the explicit dependence on θ can be found by using(6.80) together with the results (C.21) and (C.22), and are given by∂ ln p(tN |θ)∂θj=1 T −1 ∂CN −1 a C
C a
2 N N ∂θj N N


∂CN
1
−1
− Tr (I + CN WN ) WN.
2
∂θj

(6.91)

To compute the terms arising from the dependence of aN on θ, we note that
the Laplace approximation has been constructed such that Ψ(aN ) has zero gradientat aN = aN , and so Ψ(aN ) gives no contribution to the gradient as a result of its
dependence on aN . This leaves the following contribution to the derivative withrespect to a component θj of θ11  ∂ ln |WN + C−
N | ∂an
−
2
∂an
∂θj

N

n=1

1 
∂a
(I + CN WN )−1 CN nn σn (1 − σn )(1 − 2σn ) n
2
∂θj
N

=−

(6.92)

n=1

where σn = σ(an ), and again we have used the result (C.22) together with thedeﬁnition of WN . We can evaluate the derivative of aN with respect to θj by differentiating the relation (6.84) with respect to θj to give
∂an
∂a
∂CN
=
(tN − σ N ) − CN WN n .
∂θj
∂θj
∂θj

(6.93)

Rearranging then gives∂an∂CN= (I + WN CN )−1(tN − σ N ).
∂θj∂θjAppendix A(6.94)

Combining (6.91), (6.92), and (6.94), we can evaluate the gradient of the log
likelihood function, which can be used with standard nonlinear optimization algorithms in order to determine a value for θ.
We can illustrate the application of the Laplace approximation for Gaussian processes using the synthetic two-class data set shown in Figure 6.12. Extension of the
Laplace approximation to Gaussian processes involving K > 2 classes, using thesoftmax activation function, is straightforward (Williams and Barber, 1998).

6.4. Gaussian Processes

319

2

0

−2
−2

0

2

Figure 6.12 Illustration of the use of a Gaussian process for classiﬁcation, showing the data on the left together
with the optimal decision boundary from the true distribution in green, and the decision boundary from theGaussian process classiﬁer in black. On the right is the predicted posterior probability for the blue and redclasses together with the Gaussian process decision boundary.

6.4.7 Connection to neural networks
We have seen that the range of functions which can be represented by a neuralnetwork is governed by the number M of hidden units, and that, for sufﬁciently
large M , a two-layer network can approximate any given function with arbitrary
accuracy. In the framework of maximum likelihood, the number of hidden units
needs to be limited (to a level dependent on the size of the training set) in order
to avoid over-ﬁtting. However, from a Bayesian perspective it makes little sense tolimit the number of parameters in the network according to the size of the trainingset.
In a Bayesian neural network, the prior distribution over the parameter vector
w, in conjunction with the network function f (x, w), produces a prior distribution
over functions from y(x) where y is the vector of network outputs. Neal (1996)
has shown that, for a broad class of prior distributions over w, the distribution offunctions generated by a neural network will tend to a Gaussian process in the limit
M → ∞. It should be noted, however, that in this limit the output variables of theneural network become independent. One of the great merits of neural networks isthat the outputs share the hidden units and so they can ‘borrow statistical strength’
from each other, that is, the weights associated with each hidden unit are inﬂuenced
by all of the output variables not just by one of them. This property is therefore lostin the Gaussian process limit.
We have seen that a Gaussian process is determined by its covariance (kernel)
function. Williams (1998) has given explicit forms for the covariance in the case of
two speciﬁc choices for the hidden unit activation function (probit and Gaussian).
These kernel functions k(x, x ) are nonstationary, i.e. they cannot be expressed asa function of the difference x − x , as a consequence of the Gaussian weight priorbeing centred on zero which breaks translation invariance in weight space.

320

6. KERNEL METHODS
By working directly with the covariance function we have implicitly marginalized over the distribution of weights. If the weight prior is governed by hyperparameters, then their values will determine the length scales of the distribution overfunctions, as can be understood by studying the examples in Figure 5.11 for the caseof a ﬁnite number of hidden units. Note that we cannot marginalize out the hyperparameters analytically, and must instead resort to techniques of the kind discussed in
Section 6.4.

Exercises

6.1 ( ) www Consider the dual formulation of the least squares linear regression
problem given in Section 6.1. Show that the solution for the components an ofthe vector a can be expressed as a linear combination of the elements of the vector
φ(xn ). Denoting these coefﬁcients by the vector w, show that the dual of the dual
formulation is given by the original representation in terms of the parameter vectorw.
6.2 ( ) In this exercise, we develop a dual formulation of the perceptron learningalgorithm. Using the perceptron learning rule (4.55), show that the learned weightvector w can be written as a linear combination of the vectors tn φ(xn ) where tn ∈{−1, +1}. Denote the coefﬁcients of this linear combination by αn and derive aformulation of the perceptron learning algorithm, and the predictive function for theperceptron, in terms of the αn . Show that the feature vector φ(x) enters only in theform of the kernel function k(x, x ) = φ(x)T φ(x ).
6.3 ( ) The nearest-neighbour classiﬁer (Section 2.5.2) assigns a new input vector x
to the same class as that of the nearest input vector xn from the training set, wherein the simplest case, the distance is deﬁned by the Euclidean metric x − xn 2 . By
expressing this rule in terms of scalar products and then making use of kernel substitution, formulate the nearest-neighbour classiﬁer for a general nonlinear kernel.
6.4 ( ) In Appendix C, we give an example of a matrix that has positive elements butthat has a negative eigenvalue and hence that is not positive deﬁnite. Find an exampleof the converse property, namely a 2 × 2 matrix with positive eigenvalues yet thathas at least one negative element.
6.5 ( ) www

Verify the results (6.13) and (6.14) for constructing valid kernels.

6.6 ( ) Verify the results (6.15) and (6.16) for constructing valid kernels.
6.7 ( ) www

Verify the results (6.17) and (6.18) for constructing valid kernels.

6.8 ( ) Verify the results (6.19) and (6.20) for constructing valid kernels.
6.9 ( ) Verify the results (6.21) and (6.22) for constructing valid kernels.
6.10 ( ) Show that an excellent choice of kernel for learning a function f (x) is givenby k(x, x ) = f (x)f (x ) by showing that a linear learning machine based on thiskernel will always ﬁnd a solution proportional to f (x).

Exercises

321

6.11 ( ) By making use of the expansion (6.25), and then expanding the middle factor
as a power series, show that the Gaussian kernel (6.23) can be expressed as the innerproduct of an inﬁnite-dimensional feature vector.
6.12 ( ) www Consider the space of all possible subsets A of a given ﬁxed set D.
Show that the kernel function (6.27) corresponds to an inner product in a feature
space of dimensionality 2|D| deﬁned by the mapping φ(A) where A is a subset of D
and the element φU (A), indexed by the subset U , is given byφU (A) =1, if U ⊆ A;
0, otherwise.

(6.95)

Here U ⊆ A denotes that U is either a subset of A or is equal to A.
6.13 ( ) Show that the Fisher kernel, deﬁned by (6.33), remains invariant if we makea nonlinear transformation of the parameter vector θ → ψ(θ), where the function
ψ(·) is invertible and differentiable.
6.14 ( ) www Write down the form of the Fisher kernel, deﬁned by (6.33), for thecase of a distribution p(x|µ) = N (x|µ, S) that is Gaussian with mean µ and ﬁxed
covariance S.
6.15 ( ) By considering the determinant of a 2 × 2 Gram matrix, show that a positivedeﬁnite kernel function k(x, x ) satisﬁes the Cauchy-Schwartz inequalityk(x1 , x2 )2  k(x1 , x1 )k(x2 , x2 ).

(6.96)

6.16 ( ) Consider a parametric model governed by the parameter vector w together
with a data set of input values x1 , . . . , xN and a nonlinear feature mapping φ(x).
Suppose that the dependence of the error function on w takes the form
J(w) = f (wT φ(x1 ), . . . , wT φ(xN )) + g(wT w)(6.97)

where g(·) is a monotonically increasing function. By writing w in the formw=Nαn φ(xn ) + w⊥

(6.98)

n=1

show that the value of w that minimizes J(w) takes the form of a linear combinationof the basis functions φ(xn ) for n = 1, . . . , N .
6.17 ( ) www Consider the sum-of-squares error function (6.39) for data having
noisy inputs, where ν(ξ) is the distribution of the noise. Use the calculus of variations to minimize this error function with respect to the function y(x), and henceshow that the optimal solution is given by an expansion of the form (6.40) in which
the basis functions are given by (6.41).

322

6. KERNEL METHODS
6.18 ( ) Consider a Nadaraya-Watson model with one input variable x and one target
variable t having Gaussian components with isotropic covariances, so that the covariance matrix is given by σ 2 I where I is the unit matrix. Write down expressions
for the conditional density p(t|x) and for the conditional mean E[t|x] and variancevar[t|x], in terms of the kernel function k(x, xn ).
6.19 ( ) Another viewpoint on kernel regression comes from a consideration of regression problems in which the input variables as well as the target variables arecorrupted with additive noise. Suppose each target value tn is generated as usual
by taking a function y(zn ) evaluated at a point zn , and adding Gaussian noise. The
value of zn is not directly observed, however, but only a noise corrupted version
xn = zn + ξ n where the random variable ξ is governed by some distribution g(ξ).
Consider a set of observations {xn , tn }, where n = 1, . . . , N , together with a corresponding sum-of-squares error function deﬁned by averaging over the distributionof input noise to give
1
E=
2
N


2

{y(xn − ξ n ) − tn } g(ξ n ) dξn .

(6.99)

n=1

By minimizing E with respect to the function y(z) using the calculus of variations(Appendix D), show that optimal solution for y(x) is given by a Nadaraya-Watson
kernel regression solution of the form (6.45) with a kernel of the form (6.46).
6.20 () www

Verify the results (6.66) and (6.67).

6.21 ( ) www Consider a Gaussian process regression model in which the kernel
function is deﬁned in terms of a ﬁxed set of nonlinear basis functions. Show that thepredictive distribution is identical to the result (3.58) obtained in Section 3.3.2 for theBayesian linear regression model. To do this, note that both models have Gaussianpredictive distributions, and so it is only necessary to show that the conditional mean
and variance are the same. For the mean, make use of the matrix identity (C.6), andfor the variance, make use of the matrix identity (C.7).
6.22 ( ) Consider a regression problem with N training set input vectors x1 , . . . , xN
and L test set input vectors xN +1 , . . . , xN +L , and suppose we deﬁne a Gaussian
process prior over functions t(x). Derive an expression for the joint predictive distribution for t(xN +1 ), . . . , t(xN +L ), given the values of t(x1 ), . . . , t(xN ). Show themarginal of this distribution for one of the test observations tj where N + 1  j 
N + L is given by the usual Gaussian process regression result (6.66) and (6.67).
6.23 ( ) www Consider a Gaussian process regression model in which the target
variable t has dimensionality D. Write down the conditional distribution of tN +1
for a test input vector xN +1 , given a training set of input vectors x1 , . . . , xN +1 andcorresponding target observations t1 , . . . , tN .
6.24 ( ) Show that a diagonal matrix W whose elements satisfy 0 < Wii < 1 is positivedeﬁnite. Show that the sum of two positive deﬁnite matrices is itself positive deﬁnite.

Exercises323

6.25 ( ) www Using the Newton-Raphson formula (4.92), derive the iterative update
formula (6.83) for ﬁnding the mode aN of the posterior distribution in the Gaussian
process classiﬁcation model.
6.26 ( ) Using the result (2.115), derive the expressions (6.87) and (6.88) for the mean
and variance of the posterior distribution p(aN +1 |tN ) in the Gaussian process classiﬁcation model.
6.27 () Derive the result (6.90) for the log likelihood function in the Laplace approximation framework for Gaussian process classiﬁcation. Similarly, derive the results(6.91), (6.92), and (6.94) for the terms in the gradient of the log likelihood.

7
Sparse Kernel
Machines

In the previous chapter, we explored a variety of learning algorithms based on nonlinear kernels. One of the signiﬁcant limitations of many such algorithms is thatthe kernel function k(xn , xm ) must be evaluated for all possible pairs xn and xmof training points, which can be computationally infeasible during training and canlead to excessive computation times when making predictions for new data points.
In this chapter we shall look at kernel-based algorithms that have sparse solutions,so that predictions for new inputs depend only on the kernel function evaluated at a
subset of the training data points.
We begin by looking in some detail at the support vector machine (SVM), whichbecame popular in some years ago for solving problems in classiﬁcation, regression,and novelty detection. An important property of support vector machines is that thedetermination of the model parameters corresponds to a convex optimization problem, and so any local solution is also a global optimum. Because the discussion of
support vector machines makes extensive use of Lagrange multipliers, the reader is

325

326

7. SPARSE KERNEL MACHINES

Section 7.2

encouraged to review the key concepts covered in Appendix E. Additional information on support vector machines can be found in Vapnik (1995), Burges (1998),
Cristianini and Shawe-Taylor (2000), Müller et al. (2001), Schölkopf and Smola(2002), and Herbrich (2002).
The SVM is a decision machine and so does not provide posterior probabilities.
We have already discussed some of the beneﬁts of determining probabilities in Section 1.5.4. An alternative sparse kernel technique, known as the relevance vector
machine (RVM), is based on a Bayesian formulation and provides posterior probabilistic outputs, as well as having typically much sparser solutions than the SVM.

7.1. Maximum Margin Classiﬁers
We begin our discussion of support vector machines by returning to the two-classclassiﬁcation problem using linear models of the formy(x) = wT φ(x) + b

Section 7.1.5(7.1)

where φ(x) denotes a ﬁxed feature-space transformation, and we have made thebias parameter b explicit. Note that we shall shortly introduce a dual representation
expressed in terms of kernel functions, which avoids having to work explicitly infeature space. The training data set comprises N input vectors x1 , . . . , xN , withcorresponding target values t1 , . . . , tN where tn ∈ {−1, 1}, and new data points x
are classiﬁed according to the sign of y(x).
We shall assume for the moment that the training data set is linearly separable infeature space, so that by deﬁnition there exists at least one choice of the parameters
w and b such that a function of the form (7.1) satisﬁes y(xn ) > 0 for points having
tn = +1 and y(xn ) < 0 for points having tn = −1, so that tn y(xn ) > 0 for all
training data points.
There may of course exist many such solutions that separate the classes exactly.
In Section 4.1.7, we described the perceptron algorithm that is guaranteed to ﬁnd
a solution in a ﬁnite number of steps. The solution that it ﬁnds, however, will bedependent on the (arbitrary) initial values chosen for w and b as well as on theorder in which the data points are presented. If there are multiple solutions all ofwhich classify the training data set exactly, then we should try to ﬁnd the one thatwill give the smallest generalization error. The support vector machine approaches
this problem through the concept of the margin, which is deﬁned to be the smallest
distance between the decision boundary and any of the samples, as illustrated in
Figure 7.1.
In support vector machines the decision boundary is chosen to be the one forwhich the margin is maximized. The maximum margin solution can be motivated using computational learning theory, also known as statistical learning theory. However, a simple insight into the origins of maximum margin has been given by Tongand Koller (2000) who consider a framework for classiﬁcation based on a hybrid ofgenerative and discriminative approaches. They ﬁrst model the distribution over input vectors x for each class using a Parzen density estimator with Gaussian kernels

7.1. Maximum Margin Classiﬁers

327

y = −1y=1
y=0
y = −1y=0
y=1

marginFigure 7.1 The margin is deﬁned as the perpendicular distance between the decision boundary and the closestof the data points, as shown on the left ﬁgure. Maximizing the margin leads to a particular choice of decisionboundary, as shown on the right. The location of this boundary is determined by a subset of the data points,
known as support vectors, which are indicated by the circles.

having a common parameter σ 2 . Together with the class priors, this deﬁnes an optimal misclassiﬁcation-rate decision boundary. However, instead of using this optimal
boundary, they determine the best hyperplane by minimizing the probability of error
relative to the learned density model. In the limit σ 2 → 0, the optimal hyperplaneis shown to be the one having maximum margin. The intuition behind this result isthat as σ 2 is reduced, the hyperplane is increasingly dominated by nearby data points
relative to more distant ones. In the limit, the hyperplane becomes independent ofdata points that are not support vectors.
We shall see in Figure 10.13 that marginalization with respect to the prior distribution of the parameters in a Bayesian approach for a simple linearly separable data
set leads to a decision boundary that lies in the middle of the region separating thedata points. The large margin solution has similar behaviour.
Recall from Figure 4.1 that the perpendicular distance of a point x from a hyperplane deﬁned by y(x) = 0 where y(x) takes the form (7.1) is given by |y(x)|/w.
Furthermore, we are only interested in solutions for which all data points are correctly classiﬁed, so that tn y(xn ) > 0 for all n. Thus the distance of a point xn to thedecision surface is given bytn y(xn )tn (wT φ(xn ) + b)
=
.
w
w

(7.2)

The margin is given by the perpendicular distance to the closest point xn from thedata set, and we wish to optimize the parameters w and b in order to maximize thisdistance. Thus the maximum margin solution is found by solving
arg maxw,b1
min tn wT φ(xn ) + b
w n

(7.3)

where we have taken the factor 1/w outside the optimization over n because w

328

7. SPARSE KERNEL MACHINES
does not depend on n. Direct solution of this optimization problem would be very
complex, and so we shall convert it into an equivalent problem that is much easier
to solve. To do this we note that if we make the rescaling w → κw and b → κb,then the distance from any point xn to the decision surface, given by tn y(xn )/w,is unchanged. We can use this freedom to set
tn wT φ(xn ) + b = 1(7.4)

for the point that is closest to the surface. In this case, all data points will satisfy theconstraints
tn wT φ(xn ) + b  1,
n = 1, . . . , N.
(7.5)
This is known as the canonical representation of the decision hyperplane. In thecase of data points for which the equality holds, the constraints are said to be active,
whereas for the remainder they are said to be inactive. By deﬁnition, there will
always be at least one active constraint, because there will always be a closest point,
and once the margin has been maximized there will be at least two active constraints.
The optimization problem then simply requires that we maximize w−1 , which isequivalent to minimizing w2 , and so we have to solve the optimization problem
1
arg min w2
2
w,b

Appendix E(7.6)

subject to the constraints given by (7.5). The factor of 1/2 in (7.6) is included forlater convenience. This is an example of a quadratic programming problem in which
we are trying to minimize a quadratic function subject to a set of linear inequality
constraints. It appears that the bias parameter b has disappeared from the optimization. However, it is determined implicitly via the constraints, because these requirethat changes to w be compensated by changes to b. We shall see how this works
shortly.
In order to solve this constrained optimization problem, we introduce Lagrange
multipliers an  0, with one multiplier an for each of the constraints in (7.5), giving
the Lagrangian function
N



1
2
an tn (wT φ(xn ) + b) − 1
L(w, b, a) = w −
2

(7.7)

n=1

where a = (a1 , . . . , aN )T . Note the minus sign in front of the Lagrange multiplierterm, because we are minimizing with respect to w and b, and maximizing withrespect to a. Setting the derivatives of L(w, b, a) with respect to w and b equal tozero, we obtain the following two conditions
w =

N


an tn φ(xn )(7.8)

an tn .

(7.9)

n=1

0 =

N

n=1

7.1. Maximum Margin Classiﬁers

329

Eliminating w and b from L(w, b, a) using these conditions then gives the dual
representation of the maximum margin problem in which we maximize


L(a)
=

N


1
an am tn tm k(xn , xm )
2
N

an −

n=1

N

(7.10)

n=1 m=1

with respect to a subject to the constraints
an
N


 0,

n = 1, . . . , N,an tn = 0.

(7.11)(7.12)

n=1

Here the kernel function is deﬁned by k(x, x ) = φ(x)T φ(x ). Again, this takes theform of a quadratic programming problem in which we optimize a quadratic functionof a subject to a set of inequality constraints. We shall discuss techniques for solving
such quadratic programming problems in Section 7.1.1.
The solution to a quadratic programming problem in M variables in general has
computational complexity that is O(M 3 ). In going to the dual formulation we haveturned the original optimization problem, which involved minimizing (7.6) over M
variables, into the dual problem (7.10), which has N variables. For a ﬁxed set ofbasis functions whose number M is smaller than the number N of data points, themove to the dual problem appears disadvantageous. However, it allows the model tobe reformulated using kernels, and so the maximum margin classiﬁer can be applied
efﬁciently to feature spaces whose dimensionality exceeds the number of data points,
including inﬁnite feature spaces. The kernel formulation also makes clear the roleof the constraint that the kernel function k(x, x ) be positive deﬁnite, because this
is bounded below, giving rise to a wellensures that the Lagrangian function L(a)deﬁned optimization problem.
In order to classify new data points using the trained model, we evaluate the signof y(x) deﬁned by (7.1). This can be expressed in terms of the parameters {an } andthe kernel function by substituting for w using (7.8) to givey(x) =

N

an tn k(x, xn ) + b.

(7.13)

n=1

Joseph-Louis Lagrange
1736–1813
Although widely considered to bea French mathematician, Lagrange
was born in Turin in Italy. By the ageof nineteen, he had already madeimportant contributions mathematics and had been appointed as Professor at the Royal Artillery School in Turin. For many

years, Euler worked hard to persuade Lagrange tomove to Berlin, which he eventually did in 1766 where
he succeeded Euler as Director of Mathematics at
the Berlin Academy. Later he moved to Paris, narrowly escaping with his life during the French revolution thanks to the personal intervention of Lavoisier(the French chemist who discovered oxygen) who himself was later executed at the guillotine. Lagrange
made key contributions to the calculus of variations
and the foundations of dynamics.

330

7. SPARSE KERNEL MACHINESIn Appendix E, we show that a constrained optimization of this form satisﬁes the
Karush-Kuhn-Tucker (KKT) conditions, which in this case require that the following
three properties holdan  0tn y(xn ) − 1  0
an {tn y(xn ) − 1} = 0.

(7.14)(7.15)(7.16)

Thus for every data point, either an = 0 or tn y(xn ) = 1. Any data point for
which an = 0 will not appear in the sum in (7.13) and hence plays no role in making
predictions for new data points. The remaining data points are called support vectors,
and because they satisfy tn y(xn ) = 1, they correspond to points that lie on the
maximum margin hyperplanes in feature space, as illustrated in Figure 7.1. This
property is central to the practical applicability of support vector machines. Oncethe model is trained, a signiﬁcant proportion of the data points can be discarded andonly the support vectors retained.
Having solved the quadratic programming problem and found a value for a, we
can then determine the value of the threshold parameter b by noting that any support
vector xn satisﬁes tn y(xn ) = 1. Using (7.13) this gives
am tm k(xn , xm ) + b = 1(7.17)tnm∈S

where S denotes the set of indices of the support vectors. Although we can solve
this equation for b using an arbitrarily chosen support vector xn , a numerically morestable solution is obtained by ﬁrst multiplying through by tn , making use of t2n = 1,
and then averaging these equations over all support vectors and solving for b to give



1 
b=
tn −
am tm k(xn , xm )(7.18)NSn∈S

m∈S

where NS is the total number of support vectors.
For later comparison with alternative models, we can express the maximummargin classiﬁer in terms of the minimization of an error function, with a simple
quadratic regularizer, in the formN
E∞ (y(xn )tn − 1) + λw2(7.19)

n=1

where E∞ (z) is a function that is zero if z  0 and ∞ otherwise and ensures thatthe constraints (7.5) are satisﬁed. Note that as long as the regularization parametersatisﬁes λ > 0, its precise value plays no role.
Figure 7.2 shows an example of the classiﬁcation resulting from training a support vector machine on a simple synthetic data set using a Gaussian kernel of the7.1. Maximum Margin Classiﬁers
Figure 7.2

331

Example of synthetic data fromtwo classes in two dimensions
showing contours of constanty(x) obtained from a support
vector machine having a Gaussian kernel function. Also shown
are the decision boundary, themargin boundaries, and the support vectors.

form (6.23). Although the data set is not linearly separable in the two-dimensional
data space x, it is linearly separable in the nonlinear feature space deﬁned implicitly
by the nonlinear kernel function. Thus the training data points are perfectly separated
in the original data space.
This example also provides a geometrical insight into the origin of sparsity in
the SVM. The maximum margin hyperplane is deﬁned by the location of the support
vectors. Other data points can be moved around freely (so long as they remain outside the margin region) without changing the decision boundary, and so the solution
will be independent of such data points.

7.1.1 Overlapping class distributions
So far, we have assumed that the training data points are linearly separable in thefeature space φ(x). The resulting support vector machine will give exact separationof the training data in the original input space x, although the corresponding decisionboundary will be nonlinear. In practice, however, the class-conditional distributions
may overlap, in which case exact separation of the training data can lead to poor
generalization.
We therefore need a way to modify the support vector machine so as to allow
some of the training points to be misclassiﬁed. From (7.19) we see that in the caseof separable classes, we implicitly used an error function that gave inﬁnite error
if a data point was misclassiﬁed and zero error if it was classiﬁed correctly, andthen optimized the model parameters to maximize the margin. We now modify thisapproach so that data points are allowed to be on the ‘wrong side’ of the margin
boundary, but with a penalty that increases with the distance from that boundary. For
the subsequent optimization problem, it is convenient to make this penalty a linear
function of this distance. To do this, we introduce slack variables, ξn  0 wheren = 1, . . . , N , with one slack variable for each training data point (Bennett, 1992;
Cortes and Vapnik, 1995). These are deﬁned by ξn = 0 for data points that are on or
inside the correct margin boundary and ξn = |tn − y(xn )| for other points. Thus adata point that is on the decision boundary y(xn ) = 0 will have ξn = 1, and points

332

7. SPARSE KERNEL MACHINES
Figure 7.3

Illustration of the slack variables ξn  0.
Data points with circles around them aresupport vectors.

y = −1
y=0
y=1

ξ>1

ξ<1
ξ=0
ξ=0

with ξn > 1 will be misclassiﬁed. The exact classiﬁcation constraints (7.5) are thenreplaced withn = 1, . . . , N(7.20)tn y(xn )  1 − ξn ,in which the slack variables are constrained to satisfy ξn  0. Data points for whichξn = 0 are correctly classiﬁed and are either on the margin or on the correct sideof the margin. Points for which 0 < ξn  1 lie inside the margin, but on the correct side of the decision boundary, and those data points for which ξn > 1 lie on
the wrong side of the decision boundary and are misclassiﬁed, as illustrated in Figure 7.3. This is sometimes described as relaxing the hard margin constraint to give asoft margin and allows some of the training set data points to be misclassiﬁed. Notethat while slack variables allow for overlapping class distributions, this framework is
still sensitive to outliers because the penalty for misclassiﬁcation increases linearlywith ξ.
Our goal is now to maximize the margin while softly penalizing points that lie
on the wrong side of the margin boundary. We therefore minimize
C

N

n=1

1
ξn + w2
2(7.21)

where the parameter C > 0 controls the trade-off between the slack variable penalty
and the margin. Because any point that is misclassiﬁed has ξn > 1, it follows thatn ξn is an upper bound on the number of misclassiﬁed points. The parameter C is
therefore analogous to (the inverse of) a regularization coefﬁcient because it controls
the trade-off between minimizing training errors and controlling model complexity.
In the limit C → ∞, we will recover the earlier support vector machine for separable
data.
We now wish to minimize (7.21) subject to the constraints (7.20) together withξn  0. The corresponding Lagrangian is given by
L(w, b, a) =




1
ξn −
an {tn y(xn ) − 1 + ξn } −µn ξn (7.22)
w2 + C
2
N

N

N

n=1

n=1

n=1

7.1. Maximum Margin Classiﬁers

Appendix E

333

where {an  0} and {µn  0} are Lagrange multipliers. The corresponding set of
KKT conditions are given byantn y(xn ) − 1 + ξnan (tn y(xn ) − 1 + ξn )
µn
ξn
µn ξn



=


=

0
0
0
0
0
0

(7.23)(7.24)(7.25)(7.26)(7.27)(7.28)

where n = 1, . . . , N .
We now optimize out w, b, and {ξn } making use of the deﬁnition (7.1) of y(x)to give∂L
an tn φ(xn )=0 ⇒ w=∂w
N(7.29)

n=1

∂L
=0 ⇒
∂b

N


an tn = 0(7.30)∂L= 0 ⇒ an = C − µn .
∂ξn(7.31)

n=1

Using these results to eliminate w, b, and {ξn } from the Lagrangian, we obtain the
dual Lagrangian in the form


L(a)
=

N

n=1

1
an am tn tm k(xn , xm )
2
N

an −

N

(7.32)

n=1 m=1

which is identical to the separable case, except that the constraints are somewhat
different. To see what these constraints are, we note that an  0 is required because
these are Lagrange multipliers. Furthermore, (7.31) together with µn  0 impliesan  C. We therefore have to minimize (7.32) with respect to the dual variables{an } subject to
0  an  CNan tn = 0(7.33)(7.34)

n=1

for n = 1, . . . , N , where (7.33) are known as box constraints. This again represents
a quadratic programming problem. If we substitute (7.29) into (7.1), we see thatpredictions for new data points are again made by using (7.13).
We can now interpret the resulting solution. As before, a subset of the data
points may have an = 0, in which case they do not contribute to the predictive

334

7. SPARSE KERNEL MACHINES
model (7.13). The remaining data points constitute the support vectors. These havean > 0 and hence from (7.25) must satisfy
tn y(xn ) = 1 − ξn .

(7.35)

If an < C, then (7.31) implies that µn > 0, which from (7.28) requires ξn = 0 andhence such points lie on the margin. Points with an = C can lie inside the marginand can either be correctly classiﬁed if ξn  1 or misclassiﬁed if ξn > 1.
To determine the parameter b in (7.1), we note that those support vectors forwhich 0 < an < C have ξn = 0 so that tn y(xn ) = 1 and hence will satisfy
am tm k(xn , xm ) + b = 1.
(7.36)tnm∈S

Again, a numerically stable solution is obtained by averaging to give



1 
b=
tn −
am tm k(xn , xm )NMn∈M

(7.37)

m∈S

where M denotes the set of indices of data points having 0 < an < C.
An alternative, equivalent formulation of the support vector machine, known as
the ν-SVM, has been proposed by Schölkopf et al. (2000). This involves maximizing
1

L(a)
=−
2

N 
N


an am tn tm k(xn , xm )(7.38)

n=1 m=1

subject to the constraints
0  an  1/NN

an tn = 0

(7.39)(7.40)n=1Nan  ν.

(7.41)

n=1

This approach has the advantage that the parameter ν, which replaces C, can beinterpreted as both an upper bound on the fraction of margin errors (points for whichξn > 0 and hence which lie on the wrong side of the margin boundary and which may
or may not be misclassiﬁed) and a lower bound on the fraction of support vectors. An
example of the ν-SVM applied to a synthetic data set is shown in Figure 7.4. Here
Gaussian kernels of the form exp (−γx − x 2 ) have been used, with γ = 0.45.
Although predictions for new inputs are made using only the support vectors,
the training phase (i.e., the determination of the parameters a and b) makes use of
the whole data set, and so it is important to have efﬁcient algorithms for solving

335

7.1. Maximum Margin Classiﬁers
Figure 7.4

Illustration of the ν-SVM applied
to a nonseparable data set in two
dimensions. The support vectors
are indicated by circles.

2

0

−2
−2

0

2


the quadratic programming problem. We ﬁrst note that the objective function L(a)given by (7.10) or (7.32) is quadratic and so any local optimum will also be a global
optimum provided the constraints deﬁne a convex region (which they do as a consequence of being linear). Direct solution of the quadratic programming problem using traditional techniques is often infeasible due to the demanding computation andmemory requirements, and so more practical approaches need to be found. The technique of chunking (Vapnik, 1982) exploits the fact that the value of the Lagrangianis unchanged if we remove the rows and columns of the kernel matrix corresponding
to Lagrange multipliers that have value zero. This allows the full quadratic programming problem to be broken down into a series of smaller ones, whose goal iseventually to identify all of the nonzero Lagrange multipliers and discard the others.
Chunking can be implemented using protected conjugate gradients (Burges, 1998).
Although chunking reduces the size of the matrix in the quadratic function from thenumber of data points squared to approximately the number of nonzero Lagrange
multipliers squared, even this may be too big to ﬁt in memory for large-scale applications. Decomposition methods (Osuna et al., 1996) also solve a series of smaller
quadratic programming problems but are designed so that each of these is of a ﬁxed
size, and so the technique can be applied to arbitrarily large data sets. However, it
still involves numerical solution of quadratic programming subproblems and thesecan be problematic and expensive. One of the most popular approaches to training
support vector machines is called sequential minimal optimization, or SMO (Platt,1999). It takes the concept of chunking to the extreme limit and considers just two
Lagrange multipliers at a time. In this case, the subproblem can be solved analytically, thereby avoiding numerical quadratic programming altogether. Heuristics are
given for choosing the pair of Lagrange multipliers to be considered at each step.
In practice, SMO is found to have a scaling with the number of data points that issomewhere between linear and quadratic depending on the particular application.
We have seen that kernel functions correspond to inner products in feature spacesthat can have high, or even inﬁnite, dimensionality. By working directly in terms of
the kernel function, without introducing the feature space explicitly, it might therefore seem that support vector machines somehow manage to avoid the curse of di-336

7. SPARSE KERNEL MACHINES

Section 1.4

mensionality. This is not the case, however, because there are constraints amongst
the feature values that restrict the effective dimensionality of feature space. To see
this consider a simple second-order polynomial kernel that we can expand in termsof its components
k(x, z) =1 + xT z

2= (1 + x1 z1 + x2 z2 )2= 1 + 2x1 z1 + 2x2 z2 + x21 z12 + 2x1 z1 x2 z2 + x22 z22
√
√
√
√
√
√
= (1, 2x1 , 2x2 , x21 , 2x1 x2 , x22 )(1, 2z1 , 2z2 , z12 , 2z1 z2 , z22 )T= φ(x)T φ(z).
(7.42)
This kernel function therefore represents an inner product in a feature space having
six dimensions, in which the mapping from input space to feature space is describedby the vector function φ(x). However, the coefﬁcients weighting these different
features are constrained to have speciﬁc forms. Thus any set of points in the original
two-dimensional space x would be constrained to lie exactly on a two-dimensional
nonlinear manifold embedded in the six-dimensional feature space.
We have already highlighted the fact that the support vector machine does notprovide probabilistic outputs but instead makes classiﬁcation decisions for new input vectors. Veropoulos et al. (1999) discuss modiﬁcations to the SVM to allow
the trade-off between false positive and false negative errors to be controlled. However, if we wish to use the SVM as a module in a larger probabilistic system, thenprobabilistic predictions of the class label t for new inputs x are required.
To address this issue, Platt (2000) has proposed ﬁtting a logistic sigmoid to the
outputs of a previously trained support vector machine. Speciﬁcally, the requiredconditional probability is assumed to be of the formp(t = 1|x) = σ (Ay(x) + B)

(7.43)

where y(x) is deﬁned by (7.1). Values for the parameters A and B are found by
minimizing the cross-entropy error function deﬁned by a training set consisting ofpairs of values y(xn ) and tn . The data used to ﬁt the sigmoid needs to be independentof that used to train the original SVM in order to avoid severe over-ﬁtting. This twostage approach is equivalent to assuming that the output y(x) of the support vector
machine represents the log-odds of x belonging to class t = 1. Because the SVM
training procedure is not speciﬁcally intended to encourage this, the SVM can givea poor approximation to the posterior probabilities (Tipping, 2001).

7.1.2 Relation to logistic regression

Section 4.3.2

As with the separable case, we can re-cast the SVM for nonseparable distributions in terms of the minimization of a regularized error function. This will also
allow us to highlight similarities, and differences, compared to the logistic regression
model.
We have seen that for data points that are on the correct side of the margin
boundary, and which therefore satisfy yn tn  1, we have ξn = 0, and for the7.1. Maximum Margin Classiﬁers
Figure 7.5

Plot of the ‘hinge’ error function usedin support vector machines, shownin blue, along with the error function
for logistic regression, rescaled by afactor of 1/ ln(2) so that it passes
through the point (0, 1), shown in red.
Also shown are the misclassiﬁcation
error in black and the squared error
in green.

337

E(z)

−2

−1

0

1

2

z

remaining points we have ξn = 1 − yn tn . Thus the objective function (7.21) can be
written (up to an overall multiplicative constant) in the formN

ESV (yn tn ) + λw2(7.44)

n=1

where λ = (2C)−1 , and ESV (·) is the hinge error function deﬁned by
ESV (yn tn ) = [1 − yn tn ]+(7.45)

where [ · ]+ denotes the positive part. The hinge error function, so-called becauseof its shape, is plotted in Figure 7.5. It can be viewed as an approximation to themisclassiﬁcation error, i.e., the error function that ideally we would like to minimize,
which is also shown in Figure 7.5.
When we considered the logistic regression model in Section 4.3.2, we found it
convenient to work with target variable t ∈ {0, 1}. For comparison with the support
vector machine, we ﬁrst reformulate maximum likelihood logistic regression using
the target variable t ∈ {−1, 1}. To do this, we note that p(t = 1|y) = σ(y) wherey(x) is given by (7.1), and σ(y) is the logistic sigmoid function deﬁned by (4.59). It
follows that p(t = −1|y) = 1 − σ(y) = σ(−y), where we have used the propertiesof the logistic sigmoid function, and so we can writep(t|y) = σ(yt).
Exercise 7.6(7.46)

From this we can construct an error function by taking the negative logarithm of thelikelihood function that, with a quadratic regularizer, takes the formN

ELR (yn tn ) + λw2 .

(7.47)n=1

where
ELR (yt) = ln (1 + exp(−yt)) .

(7.48)

338

7. SPARSE KERNEL MACHINES
For comparison with other error functions, we can divide by ln(2) so that the error
function passes through the point (0, 1). This rescaled error function is also plottedin Figure 7.5 and we see that it has a similar form to the support vector error function.
The key difference is that the ﬂat region in ESV (yt) leads to sparse solutions.
Both the logistic error and the hinge loss can be viewed as continuous approximations to the misclassiﬁcation error. Another continuous error function that has
sometimes been used to solve classiﬁcation problems is the squared error, whichis again plotted in Figure 7.5. It has the property, however, of placing increasingemphasis on data points that are correctly classiﬁed but that are a long way fromthe decision boundary on the correct side. Such points will be strongly weighted at
the expense of misclassiﬁed points, and so if the objective is to minimize the misclassiﬁcation rate, then a monotonically decreasing error function would be a better
choice.

7.1.3 Multiclass SVMs
The support vector machine is fundamentally a two-class classiﬁer. In practice,
however, we often have to tackle problems involving K > 2 classes. Various methods have therefore been proposed for combining multiple two-class SVMs in orderto build a multiclass classiﬁer.
One commonly used approach (Vapnik, 1998) is to construct K separate SVMs,
in which the k th model yk (x) is trained using the data from class Ck as the positive
examples and the data from the remaining K − 1 classes as the negative examples.
This is known as the one-versus-the-rest approach. However, in Figure 4.2 we sawthat using the decisions of the individual classiﬁers can lead to inconsistent results
in which an input is assigned to multiple classes simultaneously. This problem issometimes addressed by making predictions for new inputs x using
y(x) = max yk (x).
k(7.49)

Unfortunately, this heuristic approach suffers from the problem that the different
classiﬁers were trained on different tasks, and there is no guarantee that the realvalued quantities yk (x) for different classiﬁers will have appropriate scales.
Another problem with the one-versus-the-rest approach is that the training sets
are imbalanced. For instance, if we have ten classes each with equal numbers of
training data points, then the individual classiﬁers are trained on data sets comprising
90% negative examples and only 10% positive examples, and the symmetry of theoriginal problem is lost. A variant of the one-versus-the-rest scheme was proposedby Lee et al. (2001) who modify the target values so that the positive class has target+1 and the negative class has target −1/(K − 1).
Weston and Watkins (1999) deﬁne a single objective function for training all
K SVMs simultaneously, based on maximizing the margin from each to remainingclasses. However, this can result in much slower training because, instead of solving
K separate optimization problems each over N data points with an overall cost ofO(KN 2 ), a single optimization problem of size (K − 1)N must be solved giving an
overall cost of O(K 2 N 2 ).

7.1. Maximum Margin Classiﬁers

339

Another approach is to train K(K − 1)/2 different 2-class SVMs on all possible
pairs of classes, and then to classify test points according to which class has the highest number of ‘votes’, an approach that is sometimes called one-versus-one. Again,
we saw in Figure 4.2 that this can lead to ambiguities in the resulting classiﬁcation.
Also, for large K this approach requires signiﬁcantly more training time than the
one-versus-the-rest approach. Similarly, to evaluate test points, signiﬁcantly morecomputation is required.
The latter problem can be alleviated by organizing the pairwise classiﬁers intoa directed acyclic graph (not to be confused with a probabilistic graphical model)
leading to the DAGSVM (Platt et al., 2000). For K classes, the DAGSVM has a totalof K(K − 1)/2 classiﬁers, and to classify a new test point only K − 1 pairwiseclassiﬁers need to be evaluated, with the particular classiﬁers used depending on
which path through the graph is traversed.
A different approach to multiclass classiﬁcation, based on error-correcting output codes, was developed by Dietterich and Bakiri (1995) and applied to support
vector machines by Allwein et al. (2000). This can be viewed as a generalization of
the voting scheme of the one-versus-one approach in which more general partitionsof the classes are used to train the individual classiﬁers. The K classes themselves
are represented as particular sets of responses from the two-class classiﬁers chosen,
and together with a suitable decoding scheme, this gives robustness to errors and to
ambiguity in the outputs of the individual classiﬁers. Although the application of
SVMs to multiclass classiﬁcation problems remains an open issue, in practice theone-versus-the-rest approach is the most widely used in spite of its ad-hoc formulation and its practical limitations.
There are also single-class support vector machines, which solve an unsupervised learning problem related to probability density estimation. Instead of modelling the density of data, however, these methods aim to ﬁnd a smooth boundary
enclosing a region of high density. The boundary is chosen to represent a quantile ofthe density, that is, the probability that a data point drawn from the distribution willland inside that region is given by a ﬁxed number between 0 and 1 that is speciﬁed inadvance. This is a more restricted problem than estimating the full density but may
be sufﬁcient in speciﬁc applications. Two approaches to this problem using support
vector machines have been proposed. The algorithm of Schölkopf et al. (2001) tries
to ﬁnd a hyperplane that separates all but a ﬁxed fraction ν of the training data fromthe origin while at the same time maximizing the distance (margin) of the hyperplanefrom the origin, while Tax and Duin (1999) look for the smallest sphere in feature
space that contains all but a fraction ν of the data points. For kernels k(x, x ) thatare functions only of x − x , the two algorithms are equivalent.

7.1.4 SVMs for regression
Section 3.1.4

We now extend support vector machines to regression problems while at thesame time preserving the property of sparseness. In simple linear regression, we340

7. SPARSE KERNEL MACHINES
Figure 7.6

Plot of an -insensitive error function (inred) in which the error increases linearly with distance beyond the insensitive region. Also shown for comparison is the quadratic error function (ingreen).

E(z)

−

0



z

minimize a regularized error function given by1
λ2{yn − tn } + w2 .
2
2
N(7.50)

n=1

To obtain sparse solutions, the quadratic error function is replaced by an -insensitive
error function (Vapnik, 1995), which gives zero error if the absolute difference between the prediction y(x) and the target t is less than  where  > 0. A simple
example of an -insensitive error function, having a linear cost associated with errors
outside the insensitive region, is given by
E (y(x) − t) =0,
if |y(x) − t| < ;
|y(x) − t| − , otherwise(7.51)and is illustrated in Figure 7.6.
We therefore minimize a regularized error function given by
C

N

n=1

1
E (y(xn ) − tn ) + w2
2

(7.52)

where y(x) is given by (7.1). By convention the (inverse) regularization parameter,
denoted C, appears in front of the error term.
As before, we can re-express the optimization problem by introducing slack
variables. For each data point xn , we now need two slack variables ξn  0 andξn  0, where ξn > 0 corresponds to a point for which tn > y(xn ) + , and 
ξn > 0corresponds to a point for which tn < y(xn ) − , as illustrated in Figure 7.7.
The condition for a target point to lie inside the -tube is that yn −   tn 
yn +, where yn = y(xn ). Introducing the slack variables allows points to lie outside
the tube provided the slack variables are nonzero, and the corresponding conditions
are
tn
tn

 y(xn ) +  + ξn
 y(xn ) −  − ξn .

(7.53)(7.54)

7.1. Maximum Margin Classiﬁers
Figure 7.7

Illustration of SVM regression, showingy(x)the regression curve together with the insensitive ‘tube’. Also shown are examples of the slack variables ξ and b
ξ. Points
above the -tube have ξ > 0 and b
ξ = 0,
points below the -tube have ξ = 0 andb
ξ > 0, and points inside the -tube have
ξ =b
ξ = 0.

341
y+

ξ>0

y
y−

ξ > 0x

The error function for support vector regression can then be written as
C

N

n=1

1
(ξn + ξn ) + w2
2

(7.55)

which must be minimized subject to the constraints ξn  0 and 
ξn  0 as well as(7.53) and (7.54). This can be achieved by introducing Lagrange multipliers an  0,
an  0, µn  0, and µn  0 and optimizing the Lagrangian
L = C

N

n=1

−

N



1
(ξn + ξn ) + w2 −(µn ξn + µ
 n
ξn )
2
N

n=1

an ( + ξn + yn − tn ) −

n=1

N



an ( + ξn − yn + tn ). (7.56)

n=1

We now substitute for y(x) using (7.1) and then set the derivatives of the Laξn to zero, giving
grangian with respect to w, b, ξn , and ∂L(an − an )φ(xn )=0 ⇒ w=∂w(7.57)N∂L(an − an ) = 0=0 ⇒∂b(7.58)

N

n=1

n=1

∂L
= 0 ⇒ an + µn = C∂ξn∂L=0 ⇒ 
an + µn = C.
∂ξn
Exercise 7.7(7.59)(7.60)

Using these results to eliminate the corresponding variables from the Lagrangian, we
see that the dual problem involves maximizing

342

7. SPARSE KERNEL MACHINES
1
 
L(a,
a) = −
2
−

N 
N


(an − an )(am − am )k(xn , xm )n=1 m=1NN(an + an ) +(an − an )tnn=1(7.61)

n=1

with respect to {an } and {
an }, where we have introduced the kernel k(x, x ) =T
φ(x) φ(x ). Again, this is a constrained maximization, and to ﬁnd the constraintsan  0 are both required because these are Lagrange
we note that an  0 and 
multipliers. Also µn  0 and µn  0 together with (7.59) and (7.60), requirean  C and an  C, and so again we have the box constraints
0  an  Can  C0(7.62)(7.63)

together with the condition (7.58).
Substituting (7.57) into (7.1), we see that predictions for new inputs can be made
using
N

(an − an )k(x, xn ) + b(7.64)
y(x) =
n=1

which is again expressed in terms of the kernel function.
The corresponding Karush-Kuhn-Tucker (KKT) conditions, which state that at
the solution the product of the dual variables and the constraints must vanish, aregiven byan ( + ξn + yn − tn ) = 0an ( + ξn − yn + tn ) = 0(C − an )ξn = 0(C − an )
ξn = 0.

(7.65)(7.66)(7.67)(7.68)

From these we can obtain several useful results. First of all, we note that a coefﬁcientan can only be nonzero if  + ξn + yn − tn = 0, which implies that the data point
either lies on the upper boundary of the -tube (ξn = 0) or lies above the upperboundary (ξn > 0). Similarly, a nonzero value for 
an implies  + 
ξn − yn + tn = 0,
and such points must lie either on or below the lower boundary of the -tube.
ξn − yn + tn = 0
Furthermore, the two constraints  + ξn + yn − tn = 0 and  + 
are incompatible, as is easily seen by adding them together and noting that ξn andξn are nonnegative while  is strictly positive, and so for every data point xn , either
an or an (or both) must be zero.
The support vectors are those data points that contribute to predictions given by
an = 0. These are points that
(7.64), in other words those for which either an = 0 or lie on the boundary of the -tube or outside the tube. All points within the tube have

7.1. Maximum Margin Classiﬁers

343

an = 0. We again have a sparse solution, and the only terms that have to bean = 
evaluated in the predictive model (7.64) are those that involve the support vectors.
The parameter b can be found by considering a data point for which 0 < an <C, which from (7.67) must have ξn = 0, and from (7.65) must therefore satisfy
 + yn − tn = 0. Using (7.1) and solving for b, we obtainb = tn −  − wT φ(xn )N= tn −  −(am − am )k(xn , xm )(7.69)

m=1

where we have used (7.57). We can obtain an analogous result by considering a point
for which 0 < 
an < C. In practice, it is better to average over all such estimates ofb.
As with the classiﬁcation case, there is an alternative formulation of the SVM
for regression in which the parameter governing complexity has a more intuitive
interpretation (Schölkopf et al., 2000). In particular, instead of ﬁxing the width  ofthe insensitive region, we ﬁx instead a parameter ν that bounds the fraction of points
lying outside the tube. This involves maximizing
1
 
L(a,
a) = −
2
+

N 
N


(an − an )(am − am )k(xn , xm )n=1 m=1N(an − an )tn(7.70)

0  an  C/Nan  C/N
0(7.71)(7.72)

n=1

subject to the constraintsN(an − an ) = 0(7.73)(an + an )  νC.

(7.74)

n=1
N

n=1

Appendix A

It can be shown that there are at most νN data points falling outside the insensitive
tube, while at least νN data points are support vectors and so lie either on the tube
or outside it.
The use of a support vector machine to solve a regression problem is illustratedusing the sinusoidal data set in Figure 7.8. Here the parameters ν and C have beenchosen by hand. In practice, their values would typically be determined by crossvalidation.

344

7. SPARSE KERNEL MACHINES
Figure 7.8

Illustration of the ν-SVM for regression applied to the sinusoidal
synthetic data set using Gaussian
kernels. The predicted regression
curve is shown by the red line, andthe -insensitive tube corresponds
to the shaded region. Also, thedata points are shown in green,
and those with support vectors
are indicated by blue circles.

1
t
0

−1

0

x

1

7.1.5 Computational learning theory
Historically, support vector machines have largely been motivated and analysedusing a theoretical framework known as computational learning theory, also sometimes called statistical learning theory (Anthony and Biggs, 1992; Kearns and Vazirani, 1994; Vapnik, 1995; Vapnik, 1998). This has its origins with Valiant (1984)
who formulated the probably approximately correct, or PAC, learning framework.
The goal of the PAC framework is to understand how large a data set needs to be inorder to give good generalization. It also gives bounds for the computational cost oflearning, although we do not consider these here.
Suppose that a data set D of size N is drawn from some joint distribution p(x, t)
where x is the input variable and t represents the class label, and that we restrict
attention to ‘noise free’ situations in which the class labels are determined by some(unknown) deterministic function t = g(x). In PAC learning we say that a functionf (x; D), drawn from a space F of such functions on the basis of the training set
D, has good generalization if its expected error rate is below some pre-speciﬁed
threshold , so that
(7.75)
Ex,t [I (f (x; D) = t)] < 
where I(·) is the indicator function, and the expectation is with respect to the distribution p(x, t). The quantity on the left-hand side is a random variable, because
it depends on the training set D, and the PAC framework requires that (7.75) holds,
with probability greater than 1 − δ, for a data set D drawn randomly from p(x, t).
Here δ is another pre-speciﬁed parameter, and the terminology ‘probably approximately correct’ comes from the requirement that with high probability (greater than
1 − δ), the error rate be small (less than ). For a given choice of model space F, andfor given parameters  and δ, PAC learning aims to provide bounds on the minimum
size N of data set needed to meet this criterion. A key quantity in PAC learning isthe Vapnik-Chervonenkis dimension, or VC dimension, which provides a measure of
the complexity of a space of functions, and which allows the PAC framework to beextended to spaces containing an inﬁnite number of functions.
The bounds derived within the PAC framework are often described as worst-

7.2. Relevance Vector Machines345

case, because they apply to any choice for the distribution p(x, t), so long as boththe training and the test examples are drawn (independently) from the same distribution, and for any choice for the function f (x) so long as it belongs to F. In real-world
applications of machine learning, we deal with distributions that have signiﬁcant regularity, for example in which large regions of input space carry the same class label.
As a consequence of the lack of any assumptions about the form of the distribution,
the PAC bounds are very conservative, in other words they strongly over-estimate
the size of data sets required to achieve a given generalization performance. For thisreason, PAC bounds have found few, if any, practical applications.
One attempt to improve the tightness of the PAC bounds is the PAC-Bayesianframework (McAllester, 2003), which considers a distribution over the space F offunctions, somewhat analogous to the prior in a Bayesian treatment. This still considers any possible choice for p(x, t), and so although the bounds are tighter, they
are still very conservative.

7.2. Relevance Vector Machines
Support vector machines have been used in a variety of classiﬁcation and regression applications. Nevertheless, they suffer from a number of limitations, severalof which have been highlighted already in this chapter. In particular, the outputs ofan SVM represent decisions rather than posterior probabilities. Also, the SVM was
originally formulated for two classes, and the extension to K > 2 classes is problematic. There is a complexity parameter C, or ν (as well as a parameter  in the caseof regression), that must be found using a hold-out method such as cross-validation.
Finally, predictions are expressed as linear combinations of kernel functions that are
centred on training data points and that are required to be positive deﬁnite.
The relevance vector machine or RVM (Tipping, 2001) is a Bayesian sparse kernel technique for regression and classiﬁcation that shares many of the characteristicsof the SVM whilst avoiding its principal limitations. Additionally, it typically leads
to much sparser models resulting in correspondingly faster performance on test datawhilst maintaining comparable generalization error.
In contrast to the SVM we shall ﬁnd it more convenient to introduce the regression form of the RVM ﬁrst and then consider the extension to classiﬁcation tasks.

7.2.1 RVM for regression
The relevance vector machine for regression is a linear model of the form studied
in Chapter 3 but with a modiﬁed prior that results in sparse solutions. The model
deﬁnes a conditional distribution for a real-valued target variable t, given an input
vector x, which takes the formp(t|x, w, β) = N (t|y(x), β −1 )(7.76)

346

7. SPARSE KERNEL MACHINES
where β = σ −2 is the noise precision (inverse noise variance), and the mean is given
by a linear model of the formy(x) =

M

wi φi (x) = wT φ(x)(7.77)

i=1

with ﬁxed nonlinear basis functions φi (x), which will typically include a constantterm so that the corresponding weight parameter represents a ‘bias’.
The relevance vector machine is a speciﬁc instance of this model, which is intended to mirror the structure of the support vector machine. In particular, the basis
functions are given by kernels, with one kernel associated with each of the data
points from the training set. The general expression (7.77) then takes the SVM-likeformN
y(x) =
wn k(x, xn ) + b(7.78)
n=1

where b is a bias parameter. The number of parameters in this case is M = N + 1,
and y(x) has the same form as the predictive model (7.64) for the SVM, except thatthe coefﬁcients an are here denoted wn . It should be emphasized that the subsequent
analysis is valid for arbitrary choices of basis function, and for generality we shallwork with the form (7.77). In contrast to the SVM, there is no restriction to positivedeﬁnite kernels, nor are the basis functions tied in either number or location to thetraining data points.
Suppose we are given a set of N observations of the input vector x, which wedenote collectively by a data matrix X whose nth row is xT
n with n = 1, . . . , N . The
corresponding target values are given by t = (t1 , . . . , tN )T . Thus, the likelihood
function is given byp(t|X, w, β) =Np(tn |xn , w, β −1 ).

(7.79)

n=1

Next we introduce a prior distribution over the parameter vector w and as in
Chapter 3, we shall consider a zero-mean Gaussian prior. However, the key difference in the RVM is that we introduce a separate hyperparameter αi for each of theweight parameters wi instead of a single shared hyperparameter. Thus the weightprior takes the formM
N (wi |0, αi−1 )(7.80)
p(w|α) =i=1

where αi represents the precision of the corresponding parameter wi , and α denotes(α1 , . . . , αM )T . We shall see that, when we maximize the evidence with respect
to these hyperparameters, a signiﬁcant proportion of them go to inﬁnity, and thecorresponding weight parameters have posterior distributions that are concentrated
at zero. The basis functions associated with these parameters therefore play no role

7.2. Relevance Vector Machines347

in the predictions made by the model and so are effectively pruned out, resulting ina sparse model.
Using the result (3.49) for linear regression models, we see that the posterior
distribution for the weights is again Gaussian and takes the formp(w|t, X, α, β) = N (w|m, Σ)(7.81)

where the mean and covariance are given by
m = βΣΦT tΣ =Section 3.5

Exercise 7.10

(7.82)

A + βΦ ΦT−1(7.83)

where Φ is the N × M design matrix with elements Φni = φi (xn ), and A =diag(αi ). Note that in the speciﬁc case of the model (7.78), we have Φ = K, where
K is the symmetric (N + 1) × (N + 1) kernel matrix with elements k(xn , xm ).
The values of α and β are determined using type-2 maximum likelihood, also
known as the evidence approximation, in which we maximize the marginal likelihood function obtained by integrating out the weight parameters(7.84)
p(t|X, α, β) = p(t|X, w, β)p(w|α) dw.
Because this represents the convolution of two Gaussians, it is readily evaluated togive the log marginal likelihood in the formln p(t|X, α, β) = ln N (t|0, C)1
N ln(2π) + ln |C| + tT C−1 t= −2(7.85)

where t = (t1 , . . . , tN )T , and we have deﬁned the N × N matrix C given by
C = β −1 I + ΦA−1 ΦT .

Exercise 7.12

Section 3.5.3(7.86)

Our goal is now to maximize (7.85) with respect to the hyperparameters α andβ. This requires only a small modiﬁcation to the results obtained in Section 3.5 forthe evidence approximation in the linear regression model. Again, we can identify
two approaches. In the ﬁrst, we simply set the required derivatives of the marginal
likelihood to zero and obtain the following re-estimation equations
αinew

=

(β new )−1

=

γi
m2i
t − Φm2
N − i γi(7.87)(7.88)

where mi is the ith component of the posterior mean m deﬁned by (7.82). The
quantity γi measures how well the corresponding parameter wi is determined by thedata and is deﬁned by348

7. SPARSE KERNEL MACHINES
γi = 1 − αi Σii(7.89)

in which Σii is the i diagonal component of the posterior covariance Σ given by(7.83). Learning therefore proceeds by choosing initial values for α and β, evaluating the mean and covariance of the posterior using (7.82) and (7.83), respectively,
and then alternately re-estimating the hyperparameters, using (7.87) and (7.88), andre-estimating the posterior mean and covariance, using (7.82) and (7.83), until a suitable convergence criterion is satisﬁed.
The second approach is to use the EM algorithm, and is discussed in Section 9.3.4. These two approaches to ﬁnding the values of the hyperparameters thatmaximize the evidence are formally equivalent. Numerically, however, it is foundthat the direct optimization approach corresponding to (7.87) and (7.88) gives somewhat faster convergence (Tipping, 2001).
As a result of the optimization, we ﬁnd that a proportion of the hyperparameters{αi } are driven to large (in principle inﬁnite) values, and so the weight parameters
wi corresponding to these hyperparameters have posterior distributions with mean
and variance both zero. Thus those parameters, and the corresponding basis functions φi (x), are removed from the model and play no role in making predictions for
new inputs. In the case of models of the form (7.78), the inputs xn corresponding to
the remaining nonzero weights are called relevance vectors, because they are identiﬁed through the mechanism of automatic relevance determination, and are analogous to the support vectors of an SVM. It is worth emphasizing, however, that thismechanism for achieving sparsity in probabilistic models through automatic relevance determination is quite general and can be applied to any model expressed as
an adaptive linear combination of basis functions.
Having found values α and β  for the hyperparameters that maximize themarginal likelihood, we can evaluate the predictive distribution over t for a new
input x. Using (7.76) and (7.81), this is given by
p(t|x, w, β  )p(w|X, t, α , β  ) dw
p(t|x, X, t, α , β ) =th

Exercise 9.23

Section 7.2.2

Exercise 7.14

= N t|mT φ(x), σ 2 (x) .

(7.90)

Thus the predictive mean is given by (7.76) with w set equal to the posterior meanm, and the variance of the predictive distribution is given byσ 2 (x) = (β  )−1 + φ(x)T Σφ(x)Section 6.4.2(7.91)

where Σ is given by (7.83) in which α and β are set to their optimized values α andβ  . This is just the familiar result (3.59) obtained in the context of linear regression.
Recall that for localized basis functions, the predictive variance for linear regression
models becomes small in regions of input space where there are no basis functions.
In the case of an RVM with the basis functions centred on data points, the model will
therefore become increasingly certain of its predictions when extrapolating outside
the domain of the data (Rasmussen and Quiñonero-Candela, 2005), which of course
is undesirable. The predictive distribution in Gaussian process regression does not349

7.2. Relevance Vector Machines
Figure 7.9

Illustration of RVM regression using the same data set, and thesame Gaussian kernel functions,
as used in Figure 7.8 for the
ν-SVM regression model. The
mean of the predictive distribution for the RVM is shown by the
red line, and the one standarddeviation predictive distribution isshown by the shaded region.
Also, the data points are shownin green, and the relevance vectors are indicated by blue circles.
Note that there are only 3 relevance vectors compared to 7 support vectors for the ν-SVM in Figure 7.8.

1
t
0

−1

0

x

1

suffer from this problem. However, the computational cost of making predictions
with a Gaussian processes is typically much higher than with an RVM.
Figure 7.9 shows an example of the RVM applied to the sinusoidal regression
data set. Here the noise precision parameter β is also determined through evidence
maximization. We see that the number of relevance vectors in the RVM is significantly smaller than the number of support vectors used by the SVM. For a wide
range of regression and classiﬁcation tasks, the RVM is found to give models thatare typically an order of magnitude more compact than the corresponding support
vector machine, resulting in a signiﬁcant improvement in the speed of processing on
test data. Remarkably, this greater sparsity is achieved with little or no reduction in
generalization error compared with the corresponding SVM.
The principal disadvantage of the RVM compared to the SVM is that training
involves optimizing a nonconvex function, and training times can be longer than for acomparable SVM. For a model with M basis functions, the RVM requires inversion
of a matrix of size M × M , which in general requires O(M 3 ) computation. In thespeciﬁc case of the SVM-like model (7.78), we have M = N +1. As we have noted,
there are techniques for training SVMs whose cost is roughly quadratic in N . Of
course, in the case of the RVM we always have the option of starting with a smaller
number of basis functions than N + 1. More signiﬁcantly, in the relevance vector
machine the parameters governing complexity and noise variance are determined
automatically from a single training run, whereas in the support vector machine theparameters C and  (or ν) are generally found using cross-validation, which involves
multiple training runs. Furthermore, in the next section we shall derive an alternative
procedure for training the relevance vector machine that improves training speed
signiﬁcantly.

7.2.2 Analysis of sparsity
We have noted earlier that the mechanism of automatic relevance determination
causes a subset of parameters to be driven to zero. We now examine in more detail350

7. SPARSE KERNEL MACHINES
t2

t2
t

C

C

t1

ϕ

t

t1

Figure 7.10 Illustration of the mechanism for sparsity in a Bayesian linear regression model, showing a training
set vector of target values given by t = (t1 , t2 )T , indicated by the cross, for a model with one basis vector
ϕ = (φ(x1 ), φ(x2 ))T , which is poorly aligned with the target data vector t. On the left we see a model having
only isotropic noise, so that C = β −1 I, corresponding to α = ∞, with β set to its most probable value. On
the right we see the same model but with a ﬁnite value of α. In each case the red ellipse corresponds to unit
Mahalanobis distance, with |C| taking the same value for both plots, while the dashed green circle shows thecontrition arising from the noise term β −1 . We see that any ﬁnite value of α reduces the probability of theobserved data, and so for the most probable solution the basis vector is removed.

the mechanism of sparsity in the context of the relevance vector machine. In theprocess, we will arrive at a signiﬁcantly faster procedure for optimizing the hyperparameters compared to the direct techniques given above.
Before proceeding with a mathematical analysis, we ﬁrst give some informal
insight into the origin of sparsity in Bayesian linear models. Consider a data set
comprising N = 2 observations t1 and t2 , together with a model having a single
basis function φ(x), with hyperparameter α, along with isotropic noise having precision β. From (7.85), the marginal likelihood is given by p(t|α, β) = N (t|0, C) in
which the covariance matrix takes the form
C=

1
1
I + ϕϕTβα(7.92)

where ϕ denotes the N -dimensional vector (φ(x1 ), φ(x2 ))T , and similarly t =(t1 , t2 )T . Notice that this is just a zero-mean Gaussian process model over t withcovariance C. Given a particular observation for t, our goal is to ﬁnd α and β  bymaximizing the marginal likelihood. We see from Figure 7.10 that, if there is a pooralignment between the direction of ϕ and that of the training data vector t, then thecorresponding hyperparameter α will be driven to ∞, and the basis vector will be
pruned from the model. This arises because any ﬁnite value for α will always assign
a lower probability to the data, thereby decreasing the value of the density at t, provided that β is set to its optimal value. We see that any ﬁnite value for α would cause
the distribution to be elongated in a direction away from the data, thereby increasing
the probability mass in regions away from the observed data and hence reducing thevalue of the density at the target data vector itself. For the more general case of M

7.2. Relevance Vector Machines351

basis vectors ϕ1 , . . . , ϕM a similar intuition holds, namely that if a particular basis
vector is poorly aligned with the data vector t, then it is likely to be pruned from themodel.
We now investigate the mechanism for sparsity from a more mathematical perspective, for a general case involving M basis functions. To motivate this analysis
we ﬁrst note that, in the result (7.87) for re-estimating the parameter αi , the terms on
the right-hand side are themselves also functions of αi . These results therefore represent implicit solutions, and iteration would be required even to determine a single
αi with all other αj for j = i ﬁxed.
This suggests a different approach to solving the optimization problem for the
RVM, in which we make explicit all of the dependence of the marginal likelihood(7.85) on a particular αi and then determine its stationary points explicitly (Faul andTipping, 2002; Tipping and Faul, 2003). To do this, we ﬁrst pull out the contribution
from αi in the matrix C deﬁned by (7.86) to give−1T
C = β −1 I +αj−1 ϕj ϕT
j + αi ϕi ϕi
j =i= C−i + αi−1 ϕi ϕT
i(7.93)

where ϕi denotes the ith column of Φ, in other words the N -dimensional vector withelements (φi (x1 ), . . . , φi (xN )), in contrast to φn , which denotes the nth row of Φ.
The matrix C−i represents the matrix C with the contribution from basis function iremoved. Using the matrix identities (C.7) and (C.15), the determinant and inverseof C can then be written−1|C| = |C−i ||1 + αi−1 ϕTi C−i ϕi |C−1

Exercise 7.15

1
T −1
C−
−i ϕi ϕi C−i
−1
C−i −
.
−1
αi + ϕTi C−i ϕi=(7.94)(7.95)

Using these results, we can then write the log marginal likelihood function (7.85) in
the form(7.96)
L(α) = L(α−i ) + λ(αi )where L(α−i ) is simply the log marginal likelihood with basis function ϕi omitted,
and the quantity λ(αi ) is deﬁned by


1
qi2
ln αi − ln (αi + si ) +(7.97)λ(αi ) =2
αi + siand contains all of the dependence on αi . Here we have introduced the two quantities
si
qi

−1
= ϕT
i C−i ϕi(7.98)

T(7.99)

=

1
ϕi C−
−i t.

Here si is called the sparsity and qi is known as the quality of ϕi , and as we shallsee, a large value of si relative to the value of qi means that the basis function ϕi

352

7. SPARSE KERNEL MACHINES

Figure 7.11 Plots of the log
marginal likelihood λ(αi ) versus
ln αi showing on the left, the single
maximum at a ﬁnite αi for qi2 = 4
and si = 1 (so that qi2 > si ) and on
the right, the maximum at αi = ∞for qi2 = 1 and si = 2 (so thatqi2 < si ).

2

2

0

0

−2

−2

−4

−4
−5

0

5

−5

0

5

is more likely to be pruned from the model. The ‘sparsity’ measures the extent towhich basis function ϕi overlaps with the other basis vectors in the model, and the‘quality’ represents a measure of the alignment of the basis vector ϕn with the error
between the training set values t = (t1 , . . . , tN )T and the vector y−i of predictions
that would result from the model with the vector ϕi excluded (Tipping and Faul,2003).
The stationary points of the marginal likelihood with respect to αi occur when
the derivative
α−1 s2 − (qi2 − si )dλ(αi )
= i i
(7.100)dαi
2(αi + si )2
is equal to zero. There are two possible forms for the solution. Recalling that αi  0,
we see that if qi2 < si , then αi → ∞ provides a solution. Conversely, if qi2 > si , wecan solve for αi to obtains2
αi = 2 i .
(7.101)qi − siExercise 7.16

These two solutions are illustrated in Figure 7.11. We see that the relative size ofthe quality and sparsity terms determines whether a particular basis vector will be
pruned from the model or not. A more complete analysis (Faul and Tipping, 2002),
based on the second derivatives of the marginal likelihood, conﬁrms these solutions
are indeed the unique maxima of λ(αi ).
Note that this approach has yielded a closed-form solution for αi , for given
values of the other hyperparameters. As well as providing insight into the origin ofsparsity in the RVM, this analysis also leads to a practical algorithm for optimizing
the hyperparameters that has signiﬁcant speed advantages. This uses a ﬁxed setof candidate basis vectors, and then cycles through them in turn to decide whether
each vector should be included in the model or not. The resulting sequential sparse
Bayesian learning algorithm is described below.
Sequential Sparse Bayesian Learning Algorithm
1. If solving a regression problem, initialize β.
2. Initialize using one basis function ϕ1 , with hyperparameter α1 set using(7.101), with the remaining hyperparameters αj for j = i initialized toinﬁnity, so that only ϕ1 is included in the model.

7.2. Relevance Vector Machines353

3. Evaluate Σ and m, along with qi and si for all basis functions.
4. Select a candidate basis function ϕi .
5. If qi2 > si , and αi < ∞, so that the basis vector ϕi is already included in
the model, then update αi using (7.101).
6. If qi2 > si , and αi = ∞, then add ϕi to the model, and evaluate hyperparameter αi using (7.101).
7. If qi2  si , and αi < ∞ then remove basis function ϕi from the model,
and set αi = ∞.
8. If solving a regression problem, update β.
9. If converged terminate, otherwise go to 3.
Note that if qi2  si and αi = ∞, then the basis function ϕi is already excluded
from the model and no action is required.
In practice, it is convenient to evaluate the quantities
Qi
Si

−1
= ϕT
t
i C
T −1
= ϕi C ϕi .

(7.102)(7.103)The quality and sparseness variables can then be expressed in the form
qi =
si =
Exercise 7.17

αi Qi
αi − S i
αi S i.
αi − S i(7.104)(7.105)Note that when αi = ∞, we have qi = Qi and si = Si . Using (C.7), we can write
Qi
Si

T
2 T
= βϕT
i t − β ϕi ΦΣΦ t
T
2 T
= βϕT
i ϕi − β ϕi ΦΣΦ ϕi(7.106)(7.107)

where Φ and Σ involve only those basis vectors that correspond to ﬁnite hyperparameters αi . At each stage the required computations therefore scale like O(M 3 ),where M is the number of active basis vectors in the model and is typically muchsmaller than the number N of training patterns.

7.2.3 RVM for classiﬁcation
We can extend the relevance vector machine framework to classiﬁcation problems by applying the ARD prior over weights to a probabilistic linear classiﬁcation
model of the kind studied in Chapter 4. To start with, we consider two-class problems with a binary target variable t ∈ {0, 1}. The model now takes the form of alinear combination of basis functions transformed by a logistic sigmoid functiony(x, w) = σ wT φ(x)(7.108)354

7. SPARSE KERNEL MACHINES

Section 4.4

where σ(·) is the logistic sigmoid function deﬁned by (4.59). If we introduce a
Gaussian prior over the weight vector w, then we obtain the model that has been
considered already in Chapter 4. The difference here is that in the RVM, this model
uses the ARD prior (7.80) in which there is a separate precision hyperparameterassociated with each weight parameter.
In contrast to the regression model, we can no longer integrate analytically overthe parameter vector w. Here we follow Tipping (2001) and use the Laplace approximation, which was applied to the closely related problem of Bayesian logistic
regression in Section 4.5.1.
We begin by initializing the hyperparameter vector α. For this given value ofα, we then build a Gaussian approximation to the posterior distribution and thereby
obtain an approximation to the marginal likelihood. Maximization of this approximate marginal likelihood then leads to a re-estimated value for α, and the process isrepeated until convergence.
Let us consider the Laplace approximation for this model in more detail. Fora ﬁxed value of α, the mode of the posterior distribution over w is obtained by
maximizing
ln p(w|t, α) = ln {p(t|w)p(w|α)} − ln p(t|α)
=

N

n=1

Exercise 7.18

1
{tn ln yn + (1 − tn ) ln(1 − yn )} − wT Aw + const (7.109)
2

where A = diag(αi ). This can be done using iterative reweighted least squares(IRLS) as discussed in Section 4.3.3. For this, we need the gradient vector andHessian matrix of the log posterior distribution, which from (7.109) are given by∇ ln p(w|t, α) = ΦT (t − y) − Aw

(7.110)∇∇ ln p(w|t, α) = − Φ BΦ + A(7.111)T

where B is an N × N diagonal matrix with elements bn = yn (1 − yn ), the vectory = (y1 , . . . , yN )T , and Φ is the design matrix with elements Φni = φi (xn ). Here
we have used the property (4.88) for the derivative of the logistic sigmoid function.
At convergence of the IRLS algorithm, the negative Hessian represents the inverse
covariance matrix for the Gaussian approximation to the posterior distribution.
The mode of the resulting approximation to the posterior distribution, corresponding to the mean of the Gaussian approximation, is obtained setting (7.110) tozero, giving the mean and covariance of the Laplace approximation in the formw = A−1 ΦT (t − y)Σ =Φ BΦ + AT−1(7.112).

(7.113)

We can now use this Laplace approximation to evaluate the marginal likelihood.
Using the general result (4.135) for an integral evaluated using the Laplace approxi-

7.2. Relevance Vector Machines
mation, we havep(t|α) =p(t|w)p(w|α) dw

 p(t|w )p(w |α)(2π)M/2 |Σ|1/2 .
Exercise 7.19

355

(7.114)

If we substitute for p(t|w ) and p(w |α) and then set the derivative of the marginal
likelihood with respect to αi equal to zero, we obtain
1
1
1− Σii = 0.
− (wi )2 +2
2αi 2

(7.115)Deﬁning γi = 1 − αi Σii and rearranging then gives
αinew =

γi(wi )2(7.116)

which is identical to the re-estimation formula (7.87) obtained for the regression
RVM.
If we deﬁnet = Φw + B−1 (t − y)(7.117)
we can write the approximate log marginal likelihood in the form
1
N ln(2π) + ln |C| + (t)T C−1tln p(t|α, β) = −2
where

Appendix A

Section 13.3

C = B + ΦAΦT .

(7.118)(7.119)This takes the same form as (7.85) in the regression case, and so we can apply thesame analysis of sparsity and obtain the same fast learning algorithm in which we
fully optimize a single hyperparameter αi at each step.
Figure 7.12 shows the relevance vector machine applied to a synthetic classiﬁcation data set. We see that the relevance vectors tend not to lie in the region of thedecision boundary, in contrast to the support vector machine. This is consistent with
our earlier discussion of sparsity in the RVM, because a basis function φi (x) centred
on a data point near the boundary will have a vector ϕi that is poorly aligned withthe training data vector t.
One of the potential advantages of the relevance vector machine compared with
the SVM is that it makes probabilistic predictions. For example, this allows the RVM
to be used to help construct an emission density in a nonlinear extension of the linear
dynamical system for tracking faces in video sequences (Williams et al., 2005).
So far, we have considered the RVM for binary classiﬁcation problems. For
K > 2 classes, we again make use of the probabilistic approach in Section 4.3.4 inwhich there are K linear models of the form
ak = wkT x(7.120)356

7. SPARSE KERNEL MACHINES

2

0

−2
−2

0

2

Figure 7.12 Example of the relevance vector machine applied to a synthetic data set, in which the left-hand plotshows the decision boundary and the data points, with the relevance vectors indicated by circles. Comparison
with the results shown in Figure 7.4 for the corresponding support vector machine shows that the RVM gives amuch sparser model. The right-hand plot shows the posterior probability given by the RVM output in which theproportion of red (blue) ink indicates the probability of that point belonging to the red (blue) class.

which are combined using a softmax function to give outputs
exp(ak )yk (x) = .
exp(aj )(7.121)j

The log likelihood function is then given byln p(T|w1 , . . . , wK ) =N K
n=1 k=1

tnk
ynk(7.122)where the target values tnk have a 1-of-K coding for each data point n, and T is amatrix with elements tnk . Again, the Laplace approximation can be used to optimize
the hyperparameters (Tipping, 2001), in which the model and its Hessian are foundusing IRLS. This gives a more principled approach to multiclass classiﬁcation thanthe pairwise method used in the support vector machine and also provides probabilistic predictions for new data points. The principal disadvantage is that the Hessian
matrix has size M K ×M K, where M is the number of active basis functions, which
gives an additional factor of K 3 in the computational cost of training compared withthe two-class RVM.
The principal disadvantage of the relevance vector machine is the relatively longtraining times compared with the SVM. This is offset, however, by the avoidance ofcross-validation runs to set the model complexity parameters. Furthermore, because
it yields sparser models, the computation time on test points, which is usually themore important consideration in practice, is typically much less.

357

Exercises

Exercises

7.1 ( ) www Suppose we have a data set of input vectors {xn } with corresponding
target values tn ∈ {−1, 1}, and suppose that we model the density of input vectors within each class separately using a Parzen kernel density estimator (see Section 2.5.1) with a kernel k(x, x ). Write down the minimum misclassiﬁcation-ratedecision rule assuming the two classes have equal prior probability. Show also that,
if the kernel is chosen to be k(x, x ) = xT x , then the classiﬁcation rule reduces tosimply assigning a new input vector to the class having the closest mean. Finally,show that, if the kernel takes the form k(x, x ) = φ(x)T φ(x ), that the classiﬁcation
is based on the closest mean in the feature space φ(x).
7.2 ( ) Show that, if the 1 on the right-hand side of the constraint (7.5) is replaced bysome arbitrary constant γ > 0, the solution for the maximum margin hyperplane isunchanged.
7.3 ( ) Show that, irrespective of the dimensionality of the data space, a data set
consisting of just two data points, one from each class, is sufﬁcient to determine thelocation of the maximum-margin hyperplane.
7.4 ( ) www Show that the value ρ of the margin for the maximum-margin hyperplane is given by


1
=
an
ρ2
N

(7.123)n=1

where {an } are given by maximizing (7.10) subject to the constraints (7.11) and(7.12).
7.5 () Show that the values of ρ and {an } in the previous exercise also satisfy1= 2L(a)ρ2(7.124)is deﬁned by (7.10). Similarly, show that
where L(a)1= w2 .
ρ2(7.125)7.6 ( ) Consider the logistic regression model with a target variable t ∈ {−1, 1}. If
we deﬁne p(t = 1|y) = σ(y) where y(x) is given by (7.1), show that the negative
log likelihood, with the addition of a quadratic regularization term, takes the form
(7.47).
7.7 ( ) Consider the Lagrangian (7.56) for the regression support vector machine. By
setting the derivatives of the Lagrangian with respect to w, b, ξn , and 
ξn to zero andthen back substituting to eliminate the corresponding variables, show that the dual
Lagrangian is given by (7.61).

358

7. SPARSE KERNEL MACHINES
7.8 ( ) www For the regression support vector machine considered in Section 7.1.4,
show that all training data points for which ξn > 0 will have an = C, and similarly
all points for which 
ξn > 0 will have an = C.
7.9 ( ) Verify the results (7.82) and (7.83) for the mean and covariance of the posterior
distribution over weights in the regression RVM.
7.10 ( ) www Derive the result (7.85) for the marginal likelihood function in the
regression RVM, by performing the Gaussian integral over w in (7.84) using thetechnique of completing the square in the exponential.
7.11 () Repeat the above exercise, but this time make use of the general result (2.115).

7.12 ( ) www Show that direct maximization of the log marginal likelihood (7.85) for
the regression relevance vector machine leads to the re-estimation equations (7.87)
and (7.88) where γi is deﬁned by (7.89).
7.13 ( ) In the evidence framework for RVM regression, we obtained the re-estimation
formulae (7.87) and (7.88) by maximizing the marginal likelihood given by (7.85).
Extend this approach by inclusion of hyperpriors given by gamma distributions of
the form (B.26) and obtain the corresponding re-estimation formulae for α and β bymaximizing the corresponding posterior probability p(t, α, β|X) with respect to α
and β.
7.14 ( ) Derive the result (7.90) for the predictive distribution in the relevance vector
machine for regression. Show that the predictive variance is given by (7.91).
7.15 ( ) www Using the results (7.94) and (7.95), show that the marginal likelihood(7.85) can be written in the form (7.96), where λ(αn ) is deﬁned by (7.97) and thesparsity and quality factors are deﬁned by (7.98) and (7.99), respectively.
7.16 ( ) By taking the second derivative of the log marginal likelihood (7.97) for the
regression RVM with respect to the hyperparameter αi , show that the stationary
point given by (7.101) is a maximum of the marginal likelihood.
7.17 ( ) Using (7.83) and (7.86), together with the matrix identity (C.7), show thatthe quantities Sn and Qn deﬁned by (7.102) and (7.103) can be written in the form
(7.106) and (7.107).
7.18 ( ) www Show that the gradient vector and Hessian matrix of the log posterior distribution (7.109) for the classiﬁcation relevance vector machine are given by(7.110) and (7.111).
7.19 ( ) Verify that maximization of the approximate log marginal likelihood function(7.114) for the classiﬁcation relevance vector machine leads to the result (7.116) for
re-estimation of the hyperparameters.

8Graphical
Models

Probabilities play a central role in modern pattern recognition. We have seen in
Chapter 1 that probability theory can be expressed in terms of two simple equations
corresponding to the sum rule and the product rule. All of the probabilistic inference and learning manipulations discussed in this book, no matter how complex,
amount to repeated application of these two equations. We could therefore proceed
to formulate and solve complicated probabilistic models purely by algebraic manipulation. However, we shall ﬁnd it highly advantageous to augment the analysis
using diagrammatic representations of probability distributions, called probabilistic
graphical models. These offer several useful properties:
1. They provide a simple way to visualize the structure of a probabilistic modeland can be used to design and motivate new models.
2. Insights into the properties of the model, including conditional independence
properties, can be obtained by inspection of the graph.
359

360

8. GRAPHICAL MODELS
3. Complex computations, required to perform inference and learning in sophisticated models, can be expressed in terms of graphical manipulations, in which
underlying mathematical expressions are carried along implicitly.
A graph comprises nodes (also called vertices) connected by links (also known
as edges or arcs). In a probabilistic graphical model, each node represents a randomvariable (or group of random variables), and the links express probabilistic relationships between these variables. The graph then captures the way in which the joint
distribution over all of the random variables can be decomposed into a product offactors each depending only on a subset of the variables. We shall begin by discussing Bayesian networks, also known as directed graphical models, in which thelinks of the graphs have a particular directionality indicated by arrows. The othermajor class of graphical models are Markov random ﬁelds, also known as undirected
graphical models, in which the links do not carry arrows and have no directionalsigniﬁcance. Directed graphs are useful for expressing causal relationships between
random variables, whereas undirected graphs are better suited to expressing soft constraints between random variables. For the purposes of solving inference problems,
it is often convenient to convert both directed and undirected graphs into a different
representation called a factor graph.
In this chapter, we shall focus on the key aspects of graphical models as needed
for applications in pattern recognition and machine learning. More general treatments of graphical models can be found in the books by Whittaker (1990), Lauritzen(1996), Jensen (1996), Castillo et al. (1997), Jordan (1999), Cowell et al. (1999),
and Jordan (2007).

8.1. Bayesian Networks
In order to motivate the use of directed graphs to describe probability distributions,consider ﬁrst an arbitrary joint distribution p(a, b, c) over three variables a, b, and c.
Note that at this stage, we do not need to specify anything further about these variables, such as whether they are discrete or continuous. Indeed, one of the powerfulaspects of graphical models is that a speciﬁc graph can make probabilistic statements
for a broad class of distributions. By application of the product rule of probability(1.11), we can write the joint distribution in the form
p(a, b, c) = p(c|a, b)p(a, b).

(8.1)

A second application of the product rule, this time to the second term on the righthand side of (8.1), gives
p(a, b, c) = p(c|a, b)p(b|a)p(a).

(8.2)

Note that this decomposition holds for any choice of the joint distribution. We now
represent the right-hand side of (8.2) in terms of a simple graphical model as follows.
First we introduce a node for each of the random variables a, b, and c and associateeach node with the corresponding conditional distribution on the right-hand side of

361

8.1. Bayesian Networks
Figure 8.1

A directed graphical model representing the joint probability distribution over three variables a, b, and c, corresponding to the decomposition on the right-hand side of (8.2).

abc(8.2). Then, for each conditional distribution we add directed links (arrows) to thegraph from the nodes corresponding to the variables on which the distribution isconditioned. Thus for the factor p(c|a, b), there will be links from nodes a and b tonode c, whereas for the factor p(a) there will be no incoming links. The result is thegraph shown in Figure 8.1. If there is a link going from a node a to a node b, then we
say that node a is the parent of node b, and we say that node b is the child of node a.
Note that we shall not make any formal distinction between a node and the variable
to which it corresponds but will simply use the same symbol to refer to both.
An interesting point to note about (8.2) is that the left-hand side is symmetricalwith respect to the three variables a, b, and c, whereas the right-hand side is not.
Indeed, in making the decomposition in (8.2), we have implicitly chosen a particularordering, namely a, b, c, and had we chosen a different ordering we would haveobtained a different decomposition and hence a different graphical representation.
We shall return to this point later.
For the moment let us extend the example of Figure 8.1 by considering the joint
distribution over K variables given by p(x1 , . . . , xK ). By repeated application of
the product rule of probability, this joint distribution can be written as a product ofconditional distributions, one for each of the variables
p(x1 , . . . , xK ) = p(xK |x1 , . . . , xK−1 ) . . . p(x2 |x1 )p(x1 ).

(8.3)

For a given choice of K, we can again represent this as a directed graph having K
nodes, one for each conditional distribution on the right-hand side of (8.3), with eachnode having incoming links from all lower numbered nodes. We say that this graphis fully connected because there is a link between every pair of nodes.
So far, we have worked with completely general joint distributions, so that thedecompositions, and their representations as fully connected graphs, will be applicable to any choice of distribution. As we shall see shortly, it is the absence of links
in the graph that conveys interesting information about the properties of the class of
distributions that the graph represents. Consider the graph shown in Figure 8.2. This
is not a fully connected graph because, for instance, there is no link from x1 to x2 orfrom x3 to x7 .
We shall now go from this graph to the corresponding representation of the joint
probability distribution written in terms of the product of a set of conditional distributions, one for each node in the graph. Each such conditional distribution will
be conditioned only on the parents of the corresponding node in the graph. For instance, x5 will be conditioned on x1 and x3 . The joint distribution of all 7 variables

362

8. GRAPHICAL MODELS
Figure 8.2

Example of a directed acyclic graph describing the joint
distribution over variables x1 , . . . , x7 . The corresponding
decomposition of the joint distribution is given by (8.4).

x1
x2

x3

x4

x5

x6

x7

is therefore given byp(x1 )p(x2 )p(x3 )p(x4 |x1 , x2 , x3 )p(x5 |x1 , x3 )p(x6 |x4 )p(x7 |x4 , x5 ).

(8.4)

The reader should take a moment to study carefully the correspondence between(8.4) and Figure 8.2.
We can now state in general terms the relationship between a given directed
graph and the corresponding distribution over the variables. The joint distribution
deﬁned by a graph is given by the product, over all of the nodes of the graph, ofa conditional distribution for each node conditioned on the variables corresponding
to the parents of that node in the graph. Thus, for a graph with K nodes, the joint
distribution is given byK
p(x) =p(xk |pak )
(8.5)
k=1

Exercise 8.1

Exercise 8.2

where pak denotes the set of parents of xk , and x = {x1 , . . . , xK }. This key
equation expresses the factorization properties of the joint distribution for a directed
graphical model. Although we have considered each node to correspond to a single
variable, we can equally well associate sets of variables and vector-valued variables
with the nodes of a graph. It is easy to show that the representation on the righthand side of (8.5) is always correctly normalized provided the individual conditional
distributions are normalized.
The directed graphs that we are considering are subject to an important restriction namely that there must be no directed cycles, in other words there are no closed
paths within the graph such that we can move from node to node along links following the direction of the arrows and end up back at the starting node. Such graphs arealso called directed acyclic graphs, or DAGs. This is equivalent to the statement that
there exists an ordering of the nodes such that there are no links that go from any
node to any lower numbered node.

8.1.1 Example: Polynomial regression
As an illustration of the use of directed graphs to describe probability distributions, we consider the Bayesian polynomial regression model introduced in Sec-

8.1. Bayesian Networks
Figure 8.3

363w

Directed graphical model representing the joint
distribution (8.6) corresponding to the Bayesianpolynomial regression model introduced in Section 1.2.6.

t1

tN

tion 1.2.6. The random variables in this model are the vector of polynomial coefﬁcients w and the observed data t = (t1 , . . . , tN )T . In addition, this model contains
the input data x = (x1 , . . . , xN )T , the noise variance σ 2 , and the hyperparameter α
representing the precision of the Gaussian prior over w, all of which are parametersof the model rather than random variables. Focussing just on the random variables
for the moment, we see that the joint distribution is given by the product of the priorp(w) and N conditional distributions p(tn |w) for n = 1, . . . , N so thatp(t, w) = p(w)Np(tn |w).

(8.6)

n=1

This joint distribution can be represented by a graphical model shown in Figure 8.3.
When we start to deal with more complex models later in the book, we shall ﬁnd
it inconvenient to have to write out multiple nodes of the form t1 , . . . , tN explicitly as
in Figure 8.3. We therefore introduce a graphical notation that allows such multiple
nodes to be expressed more compactly, in which we draw a single representative
node tn and then surround this with a box, called a plate, labelled with N indicatingthat there are N nodes of this kind. Re-writing the graph of Figure 8.3 in this way,
we obtain the graph shown in Figure 8.4.
We shall sometimes ﬁnd it helpful to make the parameters of a model, as well as
its stochastic variables, explicit. In this case, (8.6) becomesp(t, w|x, α, σ ) = p(w|α)
2Np(tn |w, xn , σ 2 ).

n=1

Correspondingly, we can make x and α explicit in the graphical representation. To
do this, we shall adopt the convention that random variables will be denoted by open
circles, and deterministic parameters will be denoted by smaller solid circles. If wetake the graph of Figure 8.4 and include the deterministic parameters, we obtain thegraph shown in Figure 8.5.
When we apply a graphical model to a problem in machine learning or patternrecognition, we will typically set some of the random variables to speciﬁc observed
Figure 8.4

An alternative, more compact, representation of the graphshown in Figure 8.3 in which we have introduced a plate(the box labelled N ) that represents N nodes of which onlya single example tn is shown explicitly.

w
tn

N

364

8. GRAPHICAL MODELS
Figure 8.5

This shows the same model as in Figure 8.4 butwith the deterministic parameters shown explicitly
by the smaller solid nodes.

xn

α

w
σ2

tn

N

values, for example the variables {tn } from the training set in the case of polynomial
curve ﬁtting. In a graphical model, we will denote such observed variables by shading the corresponding nodes. Thus the graph corresponding to Figure 8.5 in which
the variables {tn } are observed is shown in Figure 8.6. Note that the value of w is
not observed, and so w is an example of a latent variable, also known as a hidden
variable. Such variables play a crucial role in many probabilistic models and will
form the focus of Chapters 9 and 12.
Having observed the values {tn } we can, if desired, evaluate the posterior distribution of the polynomial coefﬁcients w as discussed in Section 1.2.5. For the
moment, we note that this involves a straightforward application of Bayes’ theorem
p(w|T) ∝ p(w)Np(tn |w)

(8.7)

n=1

where again we have omitted the deterministic parameters in order to keep the notation uncluttered.
In general, model parameters such as w are of little direct interest in themselves,because our ultimate goal is to make predictions for new input values. Suppose wex and we wish to ﬁnd the corresponding probability disare given a new input value tribution for t conditioned on the observed data. The graphical model that describes
this problem is shown in Figure 8.7, and the corresponding joint distribution of allof the random variables in this model, conditioned on the deterministic parameters,
is then given byN
2x, x, α, σ ) =p(tn |xn , w, σ 2 ) p(w|α)p(t|
x, w, σ 2 ).
(8.8)
p(t, t, w|n=1

Figure 8.6

As in Figure 8.5 but with the nodes {tn } shaded
to indicate that the corresponding random variables have been set to their observed (training set)values.

xn

α

w
σ

2

tn

N

365

8.1. Bayesian Networks
Figure 8.7

The polynomial regression model, corresponding
to Figure 8.6, showing also a new input value xbtogether with the corresponding model prediction
bt.

xn

α

w
tn

σ2

N

x̂
t̂

The required predictive distribution for t is then obtained, from the sum rule ofprobability, by integrating out the model parameters w so that2p(t|
x, x, t, α, σ ) ∝ p(t, t, w|
x, x, α, σ 2 ) dw
where we are implicitly setting the random variables in t to the speciﬁc values observed in the data set. The details of this calculation were discussed in Chapter 3.

8.1.2 Generative models
There are many situations in which we wish to draw samples from a given probability distribution. Although we shall devote the whole of Chapter 11 to a detailed
discussion of sampling methods, it is instructive to outline here one technique, called
ancestral sampling, which is particularly relevant to graphical models. Consider ajoint distribution p(x1 , . . . , xK ) over K variables that factorizes according to (8.5)
corresponding to a directed acyclic graph. We shall suppose that the variables havebeen ordered such that there are no links from any node to any lower numbered node,in other words each node has a higher number than any of its parents. Our goal is tox1 , . . . , xK from the joint distribution.
draw a sample To do this, we start with the lowest-numbered node and draw a sample from thex1 . We then work through each of the nodes in ordistribution p(x1 ), which we call 
der, so that for node n we draw a sample from the conditional distribution p(xn |pan )in which the parent variables have been set to their sampled values. Note that at eachstage, these parent values will always be available because they correspond to lowernumbered nodes that have already been sampled. Techniques for sampling from
speciﬁc distributions will be discussed in detail in Chapter 11. Once we have sampled from the ﬁnal variable xK , we will have achieved our objective of obtaining asample from the joint distribution. To obtain a sample from some marginal distribution corresponding to a subset of the variables, we simply take the sampled values
for the required nodes and ignore the sampled values for the remaining nodes. For
example, to draw a sample from the distribution p(x2 , x4 ), we simply sample fromx2 , 
x4 and discard the remaining
the full joint distribution and then retain the values values {xj =2,4 }.

366

8. GRAPHICAL MODELS
Figure 8.8

A graphical model representing the process by which Object
images of objects are created, in which the identityof an object (a discrete variable) and the position andorientation of that object (continuous variables) have
independent prior probabilities. The image (a vectorof pixel intensities) has a probability distribution thatis dependent on the identity of the object as well as
on its position and orientation.

Position Orientation

Image

For practical applications of probabilistic models, it will typically be the highernumbered variables corresponding to terminal nodes of the graph that represent theobservations, with lower-numbered nodes corresponding to latent variables. The
primary role of the latent variables is to allow a complicated distribution over theobserved variables to be represented in terms of a model constructed from simpler(typically exponential family) conditional distributions.
We can interpret such models as expressing the processes by which the observed
data arose. For instance, consider an object recognition task in which each observed
data point corresponds to an image (comprising a vector of pixel intensities) of oneof the objects. In this case, the latent variables might have an interpretation as the
position and orientation of the object. Given a particular observed image, our goal is
to ﬁnd the posterior distribution over objects, in which we integrate over all possible
positions and orientations. We can represent this problem using a graphical modelof the form show in Figure 8.8.
The graphical model captures the causal process (Pearl, 1988) by which the observed data was generated. For this reason, such models are often called generativemodels. By contrast, the polynomial regression model described by Figure 8.5 is
not generative because there is no probability distribution associated with the input
variable x, and so it is not possible to generate synthetic data points from this model.
We could make it generative by introducing a suitable prior distribution p(x), at theexpense of a more complex model.
The hidden variables in a probabilistic model need not, however, have any explicit physical interpretation but may be introduced simply to allow a more complex
joint distribution to be constructed from simpler components. In either case, thetechnique of ancestral sampling applied to a generative model mimics the creationof the observed data and would therefore give rise to ‘fantasy’ data whose probability
distribution (if the model were a perfect representation of reality) would be the sameas that of the observed data. In practice, producing synthetic observations from agenerative model can prove informative in understanding the form of the probability
distribution represented by that model.

8.1.3 Discrete variables
Section 2.4

We have discussed the importance of probability distributions that are membersof the exponential family, and we have seen that this family includes many wellknown distributions as particular cases. Although such distributions are relatively
simple, they form useful building blocks for constructing more complex probability

367

8.1. Bayesian Networks
Figure 8.9x1(a) This fully-connected graph describes a general distribution over two K-state discrete variables having a total of (a)K 2 − 1 parameters. (b) By dropping the link between thenodes, the number of parameters is reduced to 2(K − 1).
x1
(b)

x2
x2

distributions, and the framework of graphical models is very useful in expressing theway in which these building blocks are linked together.
Such models have particularly nice properties if we choose the relationship between each parent-child pair in a directed graph to be conjugate, and we shall explore several examples of this shortly. Two cases are particularly worthy of note,
namely when the parent and child node each correspond to discrete variables andwhen they each correspond to Gaussian variables, because in these two cases therelationship can be extended hierarchically to construct arbitrarily complex directed
acyclic graphs. We begin by examining the discrete case.
The probability distribution p(x|µ) for a single discrete variable x having K
possible states (using the 1-of-K representation) is given byp(x|µ) =

K


µxkk

(8.9)

k=1
T
and
 is governed by the parameters µ = (µ1 , . . . , µK ) . Due to the constraint
k µk = 1, only K − 1 values for µk need to be speciﬁed in order to deﬁne the
distribution.
Now suppose that we have two discrete variables, x1 and x2 , each of which has
K states, and we wish to model their joint distribution. We denote the probability ofobserving both x1k = 1 and x2l = 1 by the parameter µkl , where x1k denotes thek th component of x1 , and similarly for x2l . The joint distribution can be writtenp(x1 , x2 |µ) =K K
k=1 l=1µxkl1k x2l .

 
Because the parameters µkl are subject to the constraint k l µkl = 1, this distribution is governed by K 2 − 1 parameters. It is easily seen that the total number ofparameters that must be speciﬁed for an arbitrary joint distribution over M variables
is K M − 1 and therefore grows exponentially with the number M of variables.
Using the product rule, we can factor the joint distribution p(x1 , x2 ) in the formp(x2 |x1 )p(x1 ), which corresponds to a two-node graph with a link going from thex1 node to the x2 node as shown in Figure 8.9(a). The marginal distribution p(x1 )is governed by K − 1 parameters, as before, Similarly, the conditional distributionp(x2 |x1 ) requires the speciﬁcation of K − 1 parameters for each of the K possible
values of x1 . The total number of parameters that must be speciﬁed in the joint
distribution is therefore (K − 1) + K(K − 1) = K 2 − 1 as before.
Now suppose that the variables x1 and x2 were independent, corresponding to
the graphical model shown in Figure 8.9(b). Each variable is then described by368

8. GRAPHICAL MODELS

Figure 8.10 This chain of M discrete nodes, each x1
having K states, requires the speciﬁcation of K − 1 +(M − 1)K(K − 1) parameters, which grows linearlywith the length M of the chain. In contrast, a fully connected graph of M nodes would have K M − 1 parameters, which grows exponentially with M .

x2

xM

a separate multinomial distribution, and the total number of parameters would be2(K − 1). For a distribution over M independent discrete variables, each having K
states, the total number of parameters would be M (K − 1), which therefore grows
linearly with the number of variables. From a graphical perspective, we have reduced
the number of parameters by dropping links in the graph, at the expense of having a
restricted class of distributions.
More generally, if we have M discrete variables x1 , . . . , xM , we can model
the joint distribution using a directed graph with one variable corresponding to eachnode. The conditional distribution at each node is given by a set of nonnegative parameters subject to the usual normalization constraint. If the graph is fully connected
then we have a completely general distribution having K M − 1 parameters, whereas
if there are no links in the graph the joint distribution factorizes into the product ofthe marginals, and the total number of parameters is M (K − 1). Graphs having intermediate levels of connectivity allow for more general distributions than the fully
factorized one while requiring fewer parameters than the general joint distribution.
As an illustration, consider the chain of nodes shown in Figure 8.10. The marginal
distribution p(x1 ) requires K − 1 parameters, whereas each of the M − 1 conditional distributions p(xi |xi−1 ), for i = 2, . . . , M , requires K(K − 1) parameters.
This gives a total parameter count of K − 1 + (M − 1)K(K − 1), which is quadratic
in K and which grows linearly (rather than exponentially) with the length M of thechain.
An alternative way to reduce the number of independent parameters in a model
is by sharing parameters (also known as tying of parameters). For instance, in the
chain example of Figure 8.10, we can arrange that all of the conditional distributionsp(xi |xi−1 ), for i = 2, . . . , M , are governed by the same set of K(K −1) parameters.
Together with the K −1 parameters governing the distribution of x1 , this gives a totalof K 2 − 1 parameters that must be speciﬁed in order to deﬁne the joint distribution.
We can turn a graph over discrete variables into a Bayesian model by introducing Dirichlet priors for the parameters. From a graphical point of view, each node
then acquires an additional parent representing the Dirichlet distribution over the parameters associated with the corresponding discrete node. This is illustrated for the
chain model in Figure 8.11. The corresponding model in which we tie the parameters governing the conditional distributions p(xi |xi−1 ), for i = 2, . . . , M , is shownin Figure 8.12.
Another way of controlling the exponential growth in the number of parameters
in models of discrete variables is to use parameterized models for the conditional
distributions instead of complete tables of conditional probability values. To illustrate this idea, consider the graph in Figure 8.13 in which all of the nodes represent
binary variables. Each of the parent variables xi is governed by a single parame-

Figure 8.11

Figure 8.12

8.1. Bayesian Networks

369

An extension of the model of µ1
Figure 8.10 to include Dirichlet priors over the parameters governing the discrete
distributions.

µ2

µM

x1

x2

xM

As in Figure 8.11 but with a sin- µ1
gle set of parameters µ shared
amongst all of the conditional
distributions p(xi |xi−1 ).

x1Section 2.4

µ

x2

xM

ter µi representing the probability p(xi = 1), giving M parameters in total for theparent nodes. The conditional distribution p(y|x1 , . . . , xM ), however, would require
2M parameters representing the probability p(y = 1) for each of the 2M possible
settings of the parent variables. Thus in general the number of parameters required
to specify this conditional distribution will grow exponentially with M . We can obtain a more parsimonious form for the conditional distribution by using a logistic
sigmoid function acting on a linear combination of the parent variables, givingMwi xi = σ(wT x)(8.10)
p(y = 1|x1 , . . . , xM ) = σ w0 +i=1

where σ(a) = (1+exp(−a))−1 is the logistic sigmoid, x = (x0 , x1 , . . . , xM )T is an(M + 1)-dimensional vector of parent states augmented with an additional variable
x0 whose value is clamped to 1, and w = (w0 , w1 , . . . , wM )T is a vector of M + 1
parameters. This is a more restricted form of conditional distribution than the generalcase but is now governed by a number of parameters that grows linearly with M . In
this sense, it is analogous to the choice of a restrictive form of covariance matrix (forexample, a diagonal matrix) in a multivariate Gaussian distribution. The motivation
for the logistic sigmoid representation was discussed in Section 4.2.
Figure 8.13

A graph comprising M parents x1 , . . . , xM and a single child y, used to illustrate the idea of parameterized
conditional distributions for discrete variables.

xM

x1

y

370

8. GRAPHICAL MODELS

8.1.4 Linear-Gaussian models
In the previous section, we saw how to construct joint probability distributions
over a set of discrete variables by expressing the variables as nodes in a directed
acyclic graph. Here we show how a multivariate Gaussian can be expressed as adirected graph corresponding to a linear-Gaussian model over the component variables. This allows us to impose interesting structure on the distribution, with thegeneral Gaussian and the diagonal covariance Gaussian representing opposite extremes. Several widely used techniques are examples of linear-Gaussian models,such as probabilistic principal component analysis, factor analysis, and linear dynamical systems (Roweis and Ghahramani, 1999). We shall make extensive use of
the results of this section in later chapters when we consider some of these techniques
in detail.
Consider an arbitrary directed acyclic graph over D variables in which node irepresents a single continuous random variable xi having a Gaussian distribution.
The mean of this distribution is taken to be a linear combination of the states of its
parent nodes pai of node i
⎛ 
⎞


p(xi |pai ) = N ⎝xi 
wij xj + bi , vi ⎠(8.11)
j∈paiwhere wij and bi are parameters governing the mean, and vi is the variance of theconditional distribution for xi . The log of the joint distribution is then the log of the
product of these conditionals over all nodes in the graph and hence takes the formln p(x) =D

ln p(xi |pai )

i=1

= −

D

1
i=1

2vi

⎛
⎝xi −

(8.12)

⎞2
wij xj − bi ⎠ + const(8.13)j∈pai

where x = (x1 , . . . , xD )T and ‘const’ denotes terms independent of x. We see that
this is a quadratic function of the components of x, and hence the joint distribution
p(x) is a multivariate Gaussian.
We can determine the mean and covariance of the joint distribution recursively
as follows. Each variable xi has (conditional on the states of its parents) a Gaussian
distribution of the form (8.11) and so√
wij xj + bi + vi i(8.14)xi =j∈pai

where i is a zero mean, unit variance Gaussian random variable satisfying E[i ] = 0
and E[i j ] = Iij , where Iij is the i, j element of the identity matrix. Taking the
expectation of (8.14), we haveE[xi ] =wij E[xj ] + bi .
(8.15)
j∈pai

8.1. Bayesian Networks
Figure 8.14

A directed graph over three Gaussian variables,with one missing link.

x1

x2

371
x3

Thus we can ﬁnd the components of E[x] = (E[x1 ], . . . , E[xD ])T by starting at thelowest numbered node and working recursively through the graph (here we again
assume that the nodes are numbered such that each node has a higher number thanits parents). Similarly, we can use (8.14) and (8.15) to obtain the i, j element of thecovariance matrix for p(x) in the form of a recursion relation
cov[xi , xj ] = E [(xi − E[xi ])(xj − E[xj ])]
⎧
⎫⎤
⎡
⎨
⎬
√
wjk (xk − E[xk ]) + vj j ⎦= E ⎣(xi − E[xi ])
⎩
⎭
k∈paj

=
wjk cov[xi , xk ] + Iij vj(8.16)k∈paj

Section 2.3

Exercise 8.7

and so the covariance can similarly be evaluated recursively starting from the lowest
numbered node.
Let us consider two extreme cases. First of all, suppose that there are no links
in the graph, which therefore comprises D isolated nodes. In this case, there are no
parameters wij and so there are just D parameters bi and D parameters vi . From
the recursion relations (8.15) and (8.16), we see that the mean of p(x) is given by(b1 , . . . , bD )T and the covariance matrix is diagonal of the form diag(v1 , . . . , vD ).
The joint distribution has a total of 2D parameters and represents a set of D independent univariate Gaussian distributions.
Now consider a fully connected graph in which each node has all lower numbered nodes as parents. The matrix wij then has i − 1 entries on the ith row andhence is a lower triangular matrix (with no entries on the leading diagonal). Thenthe total number of parameters wij is obtained by taking the number D2 of elements
in a D × D matrix, subtracting D to account for the absence of elements on the leading diagonal, and then dividing by 2 because the matrix has elements only below thediagonal, giving a total of D(D − 1)/2. The total number of independent parameters{wij } and {vi } in the covariance matrix is therefore D(D + 1)/2 corresponding toa general symmetric covariance matrix.
Graphs having some intermediate level of complexity correspond to joint Gaussian distributions with partially constrained covariance matrices. Consider for example the graph shown in Figure 8.14, which has a link missing between variables
x1 and x3 . Using the recursion relations (8.15) and (8.16), we see that the mean andcovariance of the joint distribution are given byT(8.17)µ = (b1 , b2 + w21 b1 , b3 + w32 b2 + w32 w21 b1 )v1
w21 v1
w32 w21 v1
2
2
w21 v1v2 + w21v1
w32 (v2 + w21
v1 )Σ =. (8.18)2
2
2
w32 w21 v1 w32 (v2 + w21 v1 ) v3 + w32 (v2 + w21
v1 )

372

8. GRAPHICAL MODELS
We can readily extend the linear-Gaussian graphical model to the case in which
the nodes of the graph represent multivariate Gaussian variables. In this case, we canwrite the conditional distribution for node i in the form
⎛ 
⎞


p(xi |pai ) = N ⎝xi 
Wij xj + bi , Σi ⎠(8.19)
j∈pai

Section 2.3.6

where now Wij is a matrix (which is nonsquare if xi and xj have different dimensionalities). Again it is easy to verify that the joint distribution over all variables is
Gaussian.
Note that we have already encountered a speciﬁc example of the linear-Gaussian
relationship when we saw that the conjugate prior for the mean µ of a Gaussianvariable x is itself a Gaussian distribution over µ. The joint distribution over x andµ is therefore Gaussian. This corresponds to a simple two-node graph in which
the node representing µ is the parent of the node representing x. The mean of the
distribution over µ is a parameter controlling a prior, and so it can be viewed as a
hyperparameter. Because the value of this hyperparameter may itself be unknown,
we can again treat it from a Bayesian perspective by introducing a prior over thehyperparameter, sometimes called a hyperprior, which is again given by a Gaussian
distribution. This type of construction can be extended in principle to any level and is
an illustration of a hierarchical Bayesian model, of which we shall encounter further
examples in later chapters.

8.2. Conditional Independence
An important concept for probability distributions over multiple variables is that ofconditional independence (Dawid, 1980). Consider three variables a, b, and c, andsuppose that the conditional distribution of a, given b and c, is such that it does notdepend on the value of b, so thatp(a|b, c) = p(a|c).

(8.20)

We say that a is conditionally independent of b given c. This can be expressed in aslightly different way if we consider the joint distribution of a and b conditioned onc, which we can write in the form
p(a, b|c) = p(a|b, c)p(b|c)= p(a|c)p(b|c).

(8.21)

where we have used the product rule of probability together with (8.20). Thus we
see that, conditioned on c, the joint distribution of a and b factorizes into the product of the marginal distribution of a and the marginal distribution of b (again bothconditioned on c). This says that the variables a and b are statistically independent,
given c. Note that our deﬁnition of conditional independence will require that (8.20),

8.2. Conditional Independence
Figure 8.15

373

c

The ﬁrst of three examples of graphs over three variables
a, b, and c used to discuss conditional independence
properties of directed graphical models.

abor equivalently (8.21), must hold for every possible value of c, and not just for somevalues. We shall sometimes use a shorthand notation for conditional independence(Dawid, 1979) in whicha⊥⊥b|c(8.22)
denotes that a is conditionally independent of b given c and is equivalent to (8.20).
Conditional independence properties play an important role in using probabilistic models for pattern recognition by simplifying both the structure of a model andthe computations needed to perform inference and learning under that model. We
shall see examples of this shortly.
If we are given an expression for the joint distribution over a set of variables interms of a product of conditional distributions (i.e., the mathematical representation
underlying a directed graph), then we could in principle test whether any potential conditional independence property holds by repeated application of the sum andproduct rules of probability. In practice, such an approach would be very time consuming. An important and elegant feature of graphical models is that conditional
independence properties of the joint distribution can be read directly from the graphwithout having to perform any analytical manipulations. The general framework
for achieving this is called d-separation, where the ‘d’ stands for ‘directed’ (Pearl,1988). Here we shall motivate the concept of d-separation and give a general statement of the d-separation criterion. A formal proof can be found in Lauritzen (1996).

8.2.1 Three example graphs
We begin our discussion of the conditional independence properties of directed
graphs by considering three simple examples each involving graphs having just threenodes. Together, these will motivate and illustrate the key concepts of d-separation.
The ﬁrst of the three examples is shown in Figure 8.15, and the joint distribution
corresponding to this graph is easily written down using the general result (8.5) togivep(a, b, c) = p(a|c)p(b|c)p(c).
(8.23)
If none of the variables are observed, then we can investigate whether a and b are
independent by marginalizing both sides of (8.23) with respect to c to givep(a|c)p(b|c)p(c).
(8.24)p(a, b) =c

In general, this does not factorize into the product p(a)p(b), and soa⊥⊥b|∅(8.25)

374

8. GRAPHICAL MODELS
Figure 8.16c

As in Figure 8.15 but where we have conditioned on thevalue of variable c.

abwhere ∅ denotes the empty set, and the symbol ⊥⊥ means that the conditional independence property does not hold in general. Of course, it may hold for a particular
distribution by virtue of the speciﬁc numerical values associated with the various
conditional probabilities, but it does not follow in general from the structure of thegraph.
Now suppose we condition on the variable c, as represented by the graph of
Figure 8.16. From (8.23), we can easily write down the conditional distribution of aand b, given c, in the form
p(a, b, c)p(c)= p(a|c)p(b|c)

p(a, b|c) =and so we obtain the conditional independence propertya⊥⊥ b | c.
We can provide a simple graphical interpretation of this result by considering
the path from node a to node b via c. The node c is said to be tail-to-tail with respect to this path because the node is connected to the tails of the two arrows, andthe presence of such a path connecting nodes a and b causes these nodes to be dependent. However, when we condition on node c, as in Figure 8.16, the conditioned
node ‘blocks’ the path from a to b and causes a and b to become (conditionally)
independent.
We can similarly consider the graph shown in Figure 8.17. The joint distribution
corresponding to this graph is again obtained from our general formula (8.5) to givep(a, b, c) = p(a)p(c|a)p(b|c).

(8.26)First of all, suppose that none of the variables are observed. Again, we can test to
see if a and b are independent by marginalizing over c to givep(c|a)p(b|c) = p(a)p(b|a).
p(a, b) = p(a)cFigure 8.17

The second of our three examples of 3-node agraphs used to motivate the conditional independence framework for directed graphical models.

cb8.2. Conditional Independence
Figure 8.18

As in Figure 8.17 but now conditioning on node c.

a

c

375
b

which in general does not factorize into p(a)p(b), and soa ⊥⊥ b | ∅(8.27)

as before.
Now suppose we condition on node c, as shown in Figure 8.18. Using Bayes’
theorem, together with (8.26), we obtainp(a, b, c)p(c)p(a)p(c|a)p(b|c)=p(c)= p(a|c)p(b|c)

p(a, b|c) =and so again we obtain the conditional independence propertya ⊥⊥ b | c.
As before, we can interpret these results graphically. The node c is said to behead-to-tail with respect to the path from node a to node b. Such a path connects
nodes a and b and renders them dependent. If we now observe c, as in Figure 8.18,
then this observation ‘blocks’ the path from a to b and so we obtain the conditional
independence property a ⊥⊥ b | c.
Finally, we consider the third of our 3-node examples, shown by the graph in
Figure 8.19. As we shall see, this has a more subtle behaviour than the two previous
graphs.
The joint distribution can again be written down using our general result (8.5) togivep(a, b, c) = p(a)p(b)p(c|a, b).
(8.28)
Consider ﬁrst the case where none of the variables are observed. Marginalizing bothsides of (8.28) over c we obtainp(a, b) = p(a)p(b)Figure 8.19

The last of our three examples of 3-node graphs used to a
explore conditional independence properties in graphical models. This graph has rather different properties
from the two previous examples.

b

c

376

8. GRAPHICAL MODELS
Figure 8.20

As in Figure 8.19 but conditioning on the value of node ac. In this graph, the act of conditioning induces a dependence between a and b.

bcand so a and b are independent with no variables observed, in contrast to the two
previous examples. We can write this result asa⊥⊥ b | ∅.

(8.29)

Now suppose we condition on c, as indicated in Figure 8.20. The conditional distribution of a and b is then given byp(a, b|c) ==p(a, b, c)p(c)p(a)p(b)p(c|a, b)p(c)which in general does not factorize into the product p(a)p(b), and soa⊥⊥ b | c.

Exercise 8.10

Thus our third example has the opposite behaviour from the ﬁrst two. Graphically,
we say that node c is head-to-head with respect to the path from a to b because it
connects to the heads of the two arrows. When node c is unobserved, it ‘blocks’
the path, and the variables a and b are independent. However, conditioning on c
‘unblocks’ the path and renders a and b dependent.
There is one more subtlety associated with this third example that we need toconsider. First we introduce some more terminology. We say that node y is a descendant of node x if there is a path from x to y in which each step of the pathfollows the directions of the arrows. Then it can be shown that a head-to-head pathwill become unblocked if either the node, or any of its descendants, is observed.
In summary, a tail-to-tail node or a head-to-tail node leaves a path unblocked
unless it is observed in which case it blocks the path. By contrast, a head-to-headnode blocks a path if it is unobserved, but once the node, and/or at least one of its
descendants, is observed the path becomes unblocked.
It is worth spending a moment to understand further the unusual behaviour of thegraph of Figure 8.20. Consider a particular instance of such a graph corresponding
to a problem with three binary random variables relating to the fuel system on a car,as shown in Figure 8.21. The variables are called B, representing the state of abattery that is either charged (B = 1) or ﬂat (B = 0), F representing the state of
the fuel tank that is either full of fuel (F = 1) or empty (F = 0), and G, which isthe state of an electric fuel gauge and which indicates either full (G = 1) or empty

377

8.2. Conditional Independence
B

F

B

F

G

B

G

F

G

Figure 8.21 An example of a 3-node graph used to illustrate the phenomenon of ‘explaining away’. The threenodes represent the state of the battery (B), the state of the fuel tank (F ) and the reading on the electric fuelgauge (G). See the text for details.

(G = 0). The battery is either charged or ﬂat, and independently the fuel tank iseither full or empty, with prior probabilitiesp(B = 1) = 0.9p(F = 1) = 0.9.
Given the state of the fuel tank and the battery, the fuel gauge reads full with probabilities given byp(G = 1|B = 1, F = 1)p(G = 1|B = 1, F = 0)p(G = 1|B = 0, F = 1)p(G = 1|B = 0, F = 0)

=
=
=
=

0.8
0.2
0.2
0.1

so this is a rather unreliable fuel gauge! All remaining probabilities are determinedby the requirement that probabilities sum to one, and so we have a complete speciﬁcation of the probabilistic model.
Before we observe any data, the prior probability of the fuel tank being empty
is p(F = 0) = 0.1. Now suppose that we observe the fuel gauge and discover thatit reads empty, i.e., G = 0, corresponding to the middle graph in Figure 8.21. We
can use Bayes’ theorem to evaluate the posterior probability of the fuel tank being
empty. First we evaluate the denominator for Bayes’ theorem given byp(G = 0|B, F )p(B)p(F ) = 0.315(8.30)
p(G = 0) =B∈{0,1} F ∈{0,1}and similarly we evaluatep(G = 0|F = 0) =p(G = 0|B, F = 0)p(B) = 0.81

(8.31)

p(G = 0|F = 0)p(F = 0)
 0.257p(G = 0)(8.32)

B∈{0,1}and using these results we havep(F = 0|G = 0) =

378

8. GRAPHICAL MODELS
and so p(F = 0|G = 0) > p(F = 0). Thus observing that the gauge reads empty
makes it more likely that the tank is indeed empty, as we would intuitively expect.
Next suppose that we also check the state of the battery and ﬁnd that it is ﬂat, i.e.,
B = 0. We have now observed the states of both the fuel gauge and the battery, as
shown by the right-hand graph in Figure 8.21. The posterior probability that the fueltank is empty given the observations of both the fuel gauge and the battery state isthen given byp(G = 0|B = 0, F = 0)p(F = 0)p(F = 0|G = 0, B = 0) = 
 0.111 (8.33)
F ∈{0,1} p(G = 0|B = 0, F )p(F )where the prior probability p(B = 0) has cancelled between numerator and denominator. Thus the probability that the tank is empty has decreased (from 0.257 to
0.111) as a result of the observation of the state of the battery. This accords with ourintuition that ﬁnding out that the battery is ﬂat explains away the observation that thefuel gauge reads empty. We see that the state of the fuel tank and that of the battery
have indeed become dependent on each other as a result of observing the readingon the fuel gauge. In fact, this would also be the case if, instead of observing thefuel gauge directly, we observed the state of some descendant of G. Note that theprobability p(F = 0|G = 0, B = 0)  0.111 is greater than the prior probabilityp(F = 0) = 0.1 because the observation that the fuel gauge reads zero still provides
some evidence in favour of an empty fuel tank.

8.2.2 D-separation
We now give a general statement of the d-separation property (Pearl, 1988) for
directed graphs. Consider a general directed graph in which A, B, and C are arbitrary nonintersecting sets of nodes (whose union may be smaller than the complete
set of nodes in the graph). We wish to ascertain whether a particular conditional
independence statement A ⊥⊥ B | C is implied by a given directed acyclic graph. To
do so, we consider all possible paths from any node in A to any node in B. Any such
path is said to be blocked if it includes a node such that either(a) the arrows on the path meet either head-to-tail or tail-to-tail at the node, and thenode is in the set C, or(b) the arrows meet head-to-head at the node, and neither the node, nor any of its
descendants, is in the set C.
If all paths are blocked, then A is said to be d-separated from B by C, and the joint
distribution over all of the variables in the graph will satisfy A ⊥⊥ B | C.
The concept of d-separation is illustrated in Figure 8.22. In graph (a), the pathfrom a to b is not blocked by node f because it is a tail-to-tail node for this pathand is not observed, nor is it blocked by node e because, although the latter is ahead-to-head node, it has a descendant c because is in the conditioning set. Thus
the conditional independence statement a ⊥⊥ b | c does not follow from this graph.
In graph (b), the path from a to b is blocked by node f because this is a tail-to-tailnode that is observed, and so the conditional independence property a ⊥⊥ b | f will

379

8.2. Conditional Independence
Figure 8.22 Illustration of the concept of d-separation. See the text for
details.

f

a

e

f

a

e

b

c

b

c
(a)

Section 2.3(b)

be satisﬁed by any distribution that factorizes according to this graph. Note that thispath is also blocked by node e because e is a head-to-head node and neither it nor its
descendant are in the conditioning set.
For the purposes of d-separation, parameters such as α and σ 2 in Figure 8.5,
indicated by small ﬁlled circles, behave in the same was as observed nodes. However, there are no marginal distributions associated with such nodes. Consequently
parameter nodes never themselves have parents and so all paths through these nodes
will always be tail-to-tail and hence blocked. Consequently they play no role ind-separation.
Another example of conditional independence and d-separation is provided by
the concept of i.i.d. (independent identically distributed) data introduced in Section 1.2.4. Consider the problem of ﬁnding the posterior distribution for the meanof a univariate Gaussian distribution. This can be represented by the directed graphshown in Figure 8.23 in which the joint distribution is deﬁned by a prior p(µ) together with a set of conditional distributions p(xn |µ) for n = 1, . . . , N . In practice,
we observe D = {x1 , . . . , xN } and our goal is to infer µ. Suppose, for a moment,that we condition on µ and consider the joint distribution of the observations. Using
d-separation, we note that there is a unique path from any xi to any other xj =i andthat this path is tail-to-tail with respect to the observed node µ. Every such path isblocked and so the observations D = {x1 , . . . , xN } are independent given µ, so thatp(D|µ) =Np(xn |µ).

(8.34)

n=1

Figure 8.23

(a) Directed graph corresponding to the problem
of inferring the mean µ ofa univariate Gaussian distribution from observations
x1 , . . . , xN . (b) The samegraph drawn using the plate
notation.

µ
µ
N
xN

x1
(a)

xn

N
(b)

380

8. GRAPHICAL MODELS
Figure 8.24

A graphical representation of the ‘naive Bayes’model for classiﬁcation.
Conditioned on theclass label z, the components of the observed
vector x = (x1 , . . . , xD )T are assumed to be
independent.

z

x1

xD

However, if we integrate over µ, the observations are in general no longer independent ∞N
p(D|µ)p(µ) dµ =p(xn ).
(8.35)
p(D) =
0

n=1

Here µ is a latent variable, because its value is not observed.
Another example of a model representing i.i.d. data is the graph in Figure 8.7
corresponding to Bayesian polynomial regression. Here the stochastic nodes correspond to {tn }, w and t. We see that the node for w is tail-to-tail with respect tothe path from t to any one of the nodes tn and so we have the following conditional
independence propertyt ⊥⊥ tn | w.
(8.36)

Section 3.3

Thus, conditioned on the polynomial coefﬁcients w, the predictive distribution fort is independent of the training data {t1 , . . . , tN }. We can therefore ﬁrst use thetraining data to determine the posterior distribution over the coefﬁcients w and thenwe can discard the training data and use the posterior distribution for w to makex.
predictions of t for new input observations A related graphical structure arises in an approach to classiﬁcation called thenaive Bayes model, in which we use conditional independence assumptions to simplify the model structure. Suppose our observed variable consists of a D-dimensional
vector x = (x1 , . . . , xD )T , and we wish to assign observed values of x to one of K
classes. Using the 1-of-K encoding scheme, we can represent these classes by a Kdimensional binary vector z. We can then deﬁne a generative model by introducing
a multinomial prior p(z|µ) over the class labels, where the k th component µk of µis the prior probability of class Ck , together with a conditional distribution p(x|z)
for the observed vector x. The key assumption of the naive Bayes model is that,conditioned on the class z, the distributions of the input variables x1 , . . . , xD are independent. The graphical representation of this model is shown in Figure 8.24. We
see that observation of z blocks the path between xi and xj for j = i (because suchpaths are tail-to-tail at the node z) and so xi and xj are conditionally independent
given z. If, however, we marginalize out z (so that z is unobserved) the tail-to-tailpath from xi to xj is no longer blocked. This tells us that in general the marginaldensity p(x) will not factorize with respect to the components of x. We encountereda simple application of the naive Bayes model in the context of fusing data fromdifferent sources for medical diagnosis in Section 1.5.
If we are given a labelled training set, comprising inputs {x1 , . . . , xN } together
with their class labels, then we can ﬁt the naive Bayes model to the training data

8.2. Conditional Independence

381

using maximum likelihood assuming that the data are drawn independently fromthe model. The solution is obtained by ﬁtting the model for each class separately
using the correspondingly labelled data. As an example, suppose that the probability
density within each class is chosen to be Gaussian. In this case, the naive Bayes
assumption then implies that the covariance matrix for each Gaussian is diagonal,
and the contours of constant density within each class will be axis-aligned ellipsoids.
The marginal density, however, is given by a superposition of diagonal Gaussians(with weighting coefﬁcients given by the class priors) and so will no longer factorize
with respect to its components.
The naive Bayes assumption is helpful when the dimensionality D of the input
space is high, making density estimation in the full D-dimensional space more challenging. It is also useful if the input vector contains both discrete and continuous
variables, since each can be represented separately using appropriate models (e.g.,Bernoulli distributions for binary observations or Gaussians for real-valued variables). The conditional independence assumption of this model is clearly a strong
one that may lead to rather poor representations of the class-conditional densities.
Nevertheless, even if this assumption is not precisely satisﬁed, the model may still
give good classiﬁcation performance in practice because the decision boundaries canbe insensitive to some of the details in the class-conditional densities, as illustrated
in Figure 1.27.
We have seen that a particular directed graph represents a speciﬁc decompositionof a joint probability distribution into a product of conditional probabilities. The
graph also expresses a set of conditional independence statements obtained throughthe d-separation criterion, and the d-separation theorem is really an expression of theequivalence of these two properties. In order to make this clear, it is helpful to thinkof a directed graph as a ﬁlter. Suppose we consider a particular joint probability
distribution p(x) over the variables x corresponding to the (nonobserved) nodes of
the graph. The ﬁlter will allow this distribution to pass through if, and only if, it canbe expressed in terms of the factorization (8.5) implied by the graph. If we present tothe ﬁlter the set of all possible distributions p(x) over the set of variables x, then thesubset of distributions that are passed by the ﬁlter will be denoted DF, for directedfactorization. This is illustrated in Figure 8.25. Alternatively, we can use the graph asa different kind of ﬁlter by ﬁrst listing all of the conditional independence properties
obtained by applying the d-separation criterion to the graph, and then allowing a
distribution to pass only if it satisﬁes all of these properties. If we present all possible
distributions p(x) to this second kind of ﬁlter, then the d-separation theorem tells usthat the set of distributions that will be allowed through is precisely the set DF .
It should be emphasized that the conditional independence properties obtained
from d-separation apply to any probabilistic model described by that particular directed graph. This will be true, for instance, whether the variables are discrete orcontinuous or a combination of these. Again, we see that a particular graph is describing a whole family of probability distributions.
At one extreme we have a fully connected graph that exhibits no conditional independence properties at all, and which can represent any possible joint probability
distribution over the given variables. The set DF will contain all possible distribu-

382

8. GRAPHICAL MODELS

DF

p(x)

Figure 8.25

We can view a graphical model (in this case a directed graph) as a ﬁlter in which a probability distribution p(x) is allowed through the ﬁlter if, and only if, it satisﬁes the directedfactorization property (8.5). The set of all possible probability distributions p(x) that passthrough the ﬁlter is denoted DF. We can alternatively use the graph to ﬁlter distributions
according to whether they respect all of the conditional independencies implied by thed-separation properties of the graph. The d-separation theorem says that it is the same
set of distributions DF that will be allowed through this second kind of ﬁlter.

tions p(x). At the other extreme, we have the fully disconnected graph, i.e., one
having no links at all. This corresponds to joint distributions which factorize into the
product of the marginal distributions over the variables comprising the nodes of thegraph.
Note that for any given graph, the set of distributions DF will include any distributions that have additional independence properties beyond those described by
the graph. For instance, a fully factorized distribution will always be passed throughthe ﬁlter implied by any graph over the corresponding set of variables.
We end our discussion of conditional independence properties by exploring theconcept of a Markov blanket or Markov boundary. Consider a joint distributionp(x1 , . . . , xD ) represented by a directed graph having D nodes, and consider theconditional distribution of a particular node with variables xi conditioned on all of
the remaining variables xj =i . Using the factorization property (8.5), we can express
this conditional distribution in the formp(xi |x{j =i} ) ==p(x1 , . . . , xD )p(x1 , . . . , xD ) dxi
p(xk |pak )k p(xk |pak ) dxi

kin which the integral is replaced by a summation in the case of discrete variables. We
now observe that any factor p(xk |pak ) that does not have any functional dependenceon xi can be taken outside the integral over xi , and will therefore cancel between
numerator and denominator. The only factors that remain will be the conditional
distribution p(xi |pai ) for node xi itself, together with the conditional distributions
for any nodes xk such that node xi is in the conditioning set of p(xk |pak ), in otherwords for which xi is a parent of xk . The conditional p(xi |pai ) will depend on theparents of node xi , whereas the conditionals p(xk |pak ) will depend on the children

8.3. Markov Random Fields
Figure 8.26

The Markov blanket of a node xi comprises the setof parents, children and co-parents of the node. It
has the property that the conditional distribution ofxi , conditioned on all the remaining variables in thegraph, is dependent only on the variables in theMarkov blanket.

383

xi

of xi as well as on the co-parents, in other words variables corresponding to parents
of node xk other than node xi . The set of nodes comprising the parents, the children
and the co-parents is called the Markov blanket and is illustrated in Figure 8.26. We
can think of the Markov blanket of a node xi as being the minimal set of nodes thatisolates xi from the rest of the graph. Note that it is not sufﬁcient to include only theparents and children of node xi because the phenomenon of explaining away meansthat observations of the child nodes will not block paths to the co-parents. We must
therefore observe the co-parent nodes also.

8.3. Markov Random Fields
We have seen that directed graphical models specify a factorization of the joint distribution over a set of variables into a product of local conditional distributions. They
also deﬁne a set of conditional independence properties that must be satisﬁed by any
distribution that factorizes according to the graph. We turn now to the second major class of graphical models that are described by undirected graphs and that again
specify both a factorization and a set of conditional independence relations.
A Markov random ﬁeld, also known as a Markov network or an undirected
graphical model (Kindermann and Snell, 1980), has a set of nodes each of whichcorresponds to a variable or group of variables, as well as a set of links each ofwhich connects a pair of nodes. The links are undirected, that is they do not carryarrows. In the case of undirected graphs, it is convenient to begin with a discussionof conditional independence properties.

8.3.1 Conditional independence properties
Section 8.2

In the case of directed graphs, we saw that it was possible to test whether a particular conditional independence property holds by applying a graphical test called
d-separation. This involved testing whether or not the paths connecting two sets ofnodes were ‘blocked’. The deﬁnition of blocked, however, was somewhat subtle
due to the presence of paths having head-to-head nodes. We might ask whether it
is possible to deﬁne an alternative graphical semantics for probability distributions
such that conditional independence is determined by simple graph separation. This
is indeed the case and corresponds to undirected graphical models. By removing the384

8. GRAPHICAL MODELS
Figure 8.27

An example of an undirected graph inwhich every path from any node in set
A to any node in set B passes throughat least one node in set C. Consequently the conditional independence
property A ⊥⊥ B | C holds for any
probability distribution described by thisgraph.

C
B
A

directionality from the links of the graph, the asymmetry between parent and childnodes is removed, and so the subtleties associated with head-to-head nodes no longer
arise.
Suppose that in an undirected graph we identify three sets of nodes, denoted A,
B, and C, and that we consider the conditional independence propertyA⊥⊥ B | C.

(8.37)To test whether this property is satisﬁed by a probability distribution deﬁned by agraph we consider all possible paths that connect nodes in set A to nodes in set
B. If all such paths pass through one or more nodes in set C, then all such paths are‘blocked’ and so the conditional independence property holds. However, if there is atleast one such path that is not blocked, then the property does not necessarily hold, ormore precisely there will exist at least some distributions corresponding to the graphthat do not satisfy this conditional independence relation. This is illustrated with an
example in Figure 8.27. Note that this is exactly the same as the d-separation criterion except that there is no ‘explaining away’ phenomenon. Testing for conditional
independence in undirected graphs is therefore simpler than in directed graphs.
An alternative way to view the conditional independence test is to imagine removing all nodes in set C from the graph together with any links that connect tothose nodes. We then ask if there exists a path that connects any node in A to any
node in B. If there are no such paths, then the conditional independence property
must hold.
The Markov blanket for an undirected graph takes a particularly simple form,
because a node will be conditionally independent of all other nodes conditioned only
on the neighbouring nodes, as illustrated in Figure 8.28.

8.3.2 Factorization properties
We now seek a factorization rule for undirected graphs that will correspond to
the above conditional independence test. Again, this will involve expressing the joint
distribution p(x) as a product of functions deﬁned over sets of variables that are local
to the graph. We therefore need to decide what is the appropriate notion of locality
in this case.

8.3. Markov Random Fields
Figure 8.28

385

For an undirected graph, the Markov blanket of a nodexi consists of the set of neighbouring nodes. It has theproperty that the conditional distribution of xi , conditionedon all the remaining variables in the graph, is dependent
only on the variables in the Markov blanket.

If we consider two nodes xi and xj that are not connected by a link, then thesevariables must be conditionally independent given all other nodes in the graph. This
follows from the fact that there is no direct path between the two nodes, and all otherpaths pass through nodes that are observed, and hence those paths are blocked. Thisconditional independence property can be expressed asp(xi , xj |x\{i,j} ) = p(xi |x\{i,j} )p(xj |x\{i,j} )(8.38)

where x\{i,j} denotes the set x of all variables with xi and xj removed. The factorization of the joint distribution must therefore be such that xi and xj do not appear
in the same factor in order for the conditional independence property to hold for all
possible distributions belonging to the graph.
This leads us to consider a graphical concept called a clique, which is deﬁned
as a subset of the nodes in a graph such that there exists a link between all pairs of
nodes in the subset. In other words, the set of nodes in a clique is fully connected.
Furthermore, a maximal clique is a clique such that it is not possible to include anyother nodes from the graph in the set without it ceasing to be a clique. These concepts
are illustrated by the undirected graph over four variables shown in Figure 8.29. Thisgraph has ﬁve cliques of two nodes given by {x1 , x2 }, {x2 , x3 }, {x3 , x4 }, {x4 , x2 },and {x1 , x3 }, as well as two maximal cliques given by {x1 , x2 , x3 } and {x2 , x3 , x4 }.
The set {x1 , x2 , x3 , x4 } is not a clique because of the missing link from x1 to x4 .
We can therefore deﬁne the factors in the decomposition of the joint distribution
to be functions of the variables in the cliques. In fact, we can consider functionsof the maximal cliques, without loss of generality, because other cliques must be
subsets of maximal cliques. Thus, if {x1 , x2 , x3 } is a maximal clique and we deﬁne
an arbitrary function over this clique, then including another factor deﬁned over a
subset of these variables would be redundant.
Let us denote a clique by C and the set of variables in that clique by xC . Then
Figure 8.29

A four-node undirected graph showing a clique (outlined ingreen) and a maximal clique (outlined in blue).

x1
x2

x3
x4

386

8. GRAPHICAL MODELS
the joint distribution is written as a product of potential functions ψC (xC ) over the
maximal cliques of the graph1 p(x) =
ψC (xC ).
(8.39)Z
C

Here the quantity Z, sometimes called the partition function, is a normalization constant and is given byψC (xC )(8.40)Z=
x

C

which ensures that the distribution p(x) given by (8.39) is correctly normalized.
By considering only potential functions which satisfy ψC (xC )  0 we ensure thatp(x)  0. In (8.40) we have assumed that x comprises discrete variables, but theframework is equally applicable to continuous variables, or a combination of the two,in which the summation is replaced by the appropriate combination of summationand integration.
Note that we do not restrict the choice of potential functions to those that have a
speciﬁc probabilistic interpretation as marginal or conditional distributions. This is
in contrast to directed graphs in which each factor represents the conditional distribution of the corresponding variable, conditioned on the state of its parents. However,in special cases, for instance where the undirected graph is constructed by startingwith a directed graph, the potential functions may indeed have such an interpretation,
as we shall see shortly.
One consequence of the generality of the potential functions ψC (xC ) is that
their product will in general not be correctly normalized. We therefore have to introduce an explicit normalization factor given by (8.40). Recall that for directed
graphs, the joint distribution was automatically normalized as a consequence of thenormalization of each of the conditional distributions in the factorization.
The presence of this normalization constant is one of the major limitations of
undirected graphs. If we have a model with M discrete nodes each having K states,
then the evaluation of the normalization term involves summing over K M states andso (in the worst case) is exponential in the size of the model. The partition function
is needed for parameter learning because it will be a function of any parameters thatgovern the potential functions ψC (xC ). However, for evaluation of local conditional
distributions, the partition function is not needed because a conditional is the ratioof two marginals, and the partition function cancels between numerator and denominator when evaluating this ratio. Similarly, for evaluating local marginal probabilities we can work with the unnormalized joint distribution and then normalize themarginals explicitly at the end. Provided the marginals only involves a small numberof variables, the evaluation of their normalization coefﬁcient will be feasible.
So far, we have discussed the notion of conditional independence based on simple graph separation and we have proposed a factorization of the joint distributionthat is intended to correspond to this conditional independence structure. However,
we have not made any formal connection between conditional independence andfactorization for undirected graphs. To do so we need to restrict attention to potential functions ψC (xC ) that are strictly positive (i.e., never zero or negative for any

8.3. Markov Random Fields

387

choice of xC ). Given this restriction, we can make a precise relationship between
factorization and conditional independence.
To do this we again return to the concept of a graphical model as a ﬁlter, corresponding to Figure 8.25. Consider the set of all possible distributions deﬁned overa ﬁxed set of variables corresponding to the nodes of a particular undirected graph.
We can deﬁne UI to be the set of such distributions that are consistent with the setof conditional independence statements that can be read from the graph using graphseparation. Similarly, we can deﬁne UF to be the set of such distributions that canbe expressed as a factorization of the form (8.39) with respect to the maximal cliquesof the graph. The Hammersley-Clifford theorem (Clifford, 1990) states that the sets
UI and UF are identical.
Because we are restricted to potential functions which are strictly positive it isconvenient to express them as exponentials, so thatψC (xC ) = exp {−E(xC )}(8.41)

where E(xC ) is called an energy function, and the exponential representation iscalled the Boltzmann distribution. The joint distribution is deﬁned as the product ofpotentials, and so the total energy is obtained by adding the energies of each of the
maximal cliques.
In contrast to the factors in the joint distribution for a directed graph, the potentials in an undirected graph do not have a speciﬁc probabilistic interpretation.
Although this gives greater ﬂexibility in choosing the potential functions, because
there is no normalization constraint, it does raise the question of how to motivate achoice of potential function for a particular application. This can be done by viewing the potential function as expressing which conﬁgurations of the local variables
are preferred to others. Global conﬁgurations that have a relatively high probability
are those that ﬁnd a good balance in satisfying the (possibly conﬂicting) inﬂuencesof the clique potentials. We turn now to a speciﬁc example to illustrate the use ofundirected graphs.

8.3.3 Illustration: Image de-noising
We can illustrate the application of undirected graphs using an example of noise
removal from a binary image (Besag, 1974; Geman and Geman, 1984; Besag, 1986).
Although a very simple example, this is typical of more sophisticated applications.
Let the observed noisy image be described by an array of binary pixel values yi ∈{−1, +1}, where the index i = 1, . . . , D runs over all pixels. We shall supposethat the image is obtained by taking an unknown noise-free image, described bybinary pixel values xi ∈ {−1, +1} and randomly ﬂipping the sign of pixels withsome small probability. An example binary image, together with a noise corrupted
image obtained by ﬂipping the sign of the pixels with probability 10%, is shown in
Figure 8.30. Given the noisy image, our goal is to recover the original noise-free
image.
Because the noise level is small, we know that there will be a strong correlation
between xi and yi . We also know that neighbouring pixels xi and xj in an image
are strongly correlated. This prior knowledge can be captured using the Markov

388

8. GRAPHICAL MODELS

Figure 8.30 Illustration of image de-noising using a Markov random ﬁeld. The top row shows the original
binary image on the left and the corrupted image after randomly changing 10% of the pixels on the right. The
bottom row shows the restored images obtained using iterated conditional models (ICM) on the left and using
the graph-cut algorithm on the right. ICM produces an image where 96% of the pixels agree with the original
image, whereas the corresponding number for graph-cut is 99%.

random ﬁeld model whose undirected graph is shown in Figure 8.31. This graph has
two types of cliques, each of which contains two variables. The cliques of the form
{xi , yi } have an associated energy function that expresses the correlation between
these variables. We choose a very simple energy function for these cliques of theform −ηxi yi where η is a positive constant. This has the desired effect of giving alower energy (thus encouraging a higher probability) when xi and yi have the same
sign and a higher energy when they have the opposite sign.
The remaining cliques comprise pairs of variables {xi , xj } where i and j areindices of neighbouring pixels. Again, we want the energy to be lower when the
pixels have the same sign than when they have the opposite sign, and so we choose
an energy given by −βxi xj where β is a positive constant.
Because a potential function is an arbitrary, nonnegative function over a maximal
clique, we can multiply it by any nonnegative functions of subsets of the clique, or8.3. Markov Random Fields
Figure 8.31

An undirected graphical model representing a
Markov random ﬁeld for image de-noising, inwhich xi is a binary variable denoting the stateof pixel i in the unknown noise-free image, and yi
denotes the corresponding value of pixel i in theobserved noisy image.

389

yi

xi

equivalently we can add the corresponding energies. In this example, this allows us
to add an extra term hxi for each pixel i in the noise-free image. Such a term has
the effect of biasing the model towards pixel values that have one particular sign inpreference to the other.
The complete energy function for the model then takes the formxi − βxi xj − ηxi yi
(8.42)E(x, y) = h
i

{i,j}

i

which deﬁnes a joint distribution over x and y given byp(x, y) =Exercise 8.13

1exp{−E(x, y)}.
Z

(8.43)

We now ﬁx the elements of y to the observed values given by the pixels of thenoisy image, which implicitly deﬁnes a conditional distribution p(x|y) over noisefree images. This is an example of the Ising model, which has been widely studied instatistical physics. For the purposes of image restoration, we wish to ﬁnd an image x
having a high probability (ideally the maximum probability). To do this we shall use
a simple iterative technique called iterated conditional modes, or ICM (Kittler andFöglein, 1984), which is simply an application of coordinate-wise gradient ascent.
The idea is ﬁrst to initialize the variables {xi }, which we do by simply setting xi =yi for all i. Then we take one node xj at a time and we evaluate the total energy
for the two possible states xj = +1 and xj = −1, keeping all other node variables
ﬁxed, and set xj to whichever state has the lower energy. This will either leave
the probability unchanged, if xj is unchanged, or will increase it. Because only
one variable is changed, this is a simple local computation that can be performed
efﬁciently. We then repeat the update for another site, and so on, until some suitablestopping criterion is satisﬁed. The nodes may be updated in a systematic way, forinstance by repeatedly raster scanning through the image, or by choosing nodes at
random.
If we have a sequence of updates in which every site is visited at least once,
and in which no changes to the variables are made, then by deﬁnition the algorithm

390

8. GRAPHICAL MODELS

x1
Figure 8.32 (a) Example of a directed
graph.
(b) The equivalent undirected (a)graph.

x2

xN −1

xN

x1

x2

xN

xN −1

(b)

Exercise 8.14
Section 8.4

will have converged to a local maximum of the probability. This need not, however,correspond to the global maximum.
For the purposes of this simple illustration, we have ﬁxed the parameters to be
β = 1.0, η = 2.1 and h = 0. Note that leaving h = 0 simply means that the priorprobabilities of the two states of xi are equal. Starting with the observed noisy image
as the initial conﬁguration, we run ICM until convergence, leading to the de-noised
image shown in the lower left panel of Figure 8.30. Note that if we set β = 0,
which effectively removes the links between neighbouring pixels, then the global
most probable solution is given by xi = yi for all i, corresponding to the observed
noisy image.
Later we shall discuss a more effective algorithm for ﬁnding high probability solutions called the max-product algorithm, which typically leads to better solutions,although this is still not guaranteed to ﬁnd the global maximum of the posterior distribution. However, for certain classes of model, including the one given by (8.42),
there exist efﬁcient algorithms based on graph cuts that are guaranteed to ﬁnd theglobal maximum (Greig et al., 1989; Boykov et al., 2001; Kolmogorov and Zabih,
2004). The lower right panel of Figure 8.30 shows the result of applying a graph-cut
algorithm to the de-noising problem.

8.3.4 Relation to directed graphs
We have introduced two graphical frameworks for representing probability distributions, corresponding to directed and undirected graphs, and it is instructive to
discuss the relation between these. Consider ﬁrst the problem of taking a model thatis speciﬁed using a directed graph and trying to convert it to an undirected graph. In
some cases this is straightforward, as in the simple example in Figure 8.32. Here thejoint distribution for the directed graph is given as a product of conditionals in theform(8.44)
p(x) = p(x1 )p(x2 |x1 )p(x3 |x2 ) · · · p(xN |xN −1 ).
Now let us convert this to an undirected graph representation, as shown in Figure 8.32. In the undirected graph, the maximal cliques are simply the pairs of neighbouring nodes, and so from (8.39) we wish to write the joint distribution in the form
p(x) =

1
ψ1,2 (x1 , x2 )ψ2,3 (x2 , x3 ) · · · ψN −1,N (xN −1 , xN ).
Z(8.45)

391

8.3. Markov Random Fields
Figure 8.33 Example of a simple
directed graph (a) and the corresponding moral graph (b).

x1

x3

x1

x2

x3
x2

x4
(a)

x4
(b)

This is easily done by identifying
ψ1,2 (x1 , x2 ) = p(x1 )p(x2 |x1 )
ψ2,3 (x2 , x3 ) = p(x3 |x2 )..
.
ψN −1,N (xN −1 , xN ) = p(xN |xN −1 )where we have absorbed the marginal p(x1 ) for the ﬁrst node into the ﬁrst potentialfunction. Note that in this case, the partition function Z = 1.
Let us consider how to generalize this construction, so that we can convert any
distribution speciﬁed by a factorization over a directed graph into one speciﬁed by afactorization over an undirected graph. This can be achieved if the clique potentialsof the undirected graph are given by the conditional distributions of the directed
graph. In order for this to be valid, we must ensure that the set of variables thatappears in each of the conditional distributions is a member of at least one clique ofthe undirected graph. For nodes on the directed graph having just one parent, this isachieved simply by replacing the directed link with an undirected link. However, fornodes in the directed graph having more than one parent, this is not sufﬁcient. These
are nodes that have ‘head-to-head’ paths encountered in our discussion of conditional
independence. Consider a simple directed graph over 4 nodes shown in Figure 8.33.
The joint distribution for the directed graph takes the form
p(x) = p(x1 )p(x2 )p(x3 )p(x4 |x1 , x2 , x3 ).

(8.46)We see that the factor p(x4 |x1 , x2 , x3 ) involves the four variables x1 , x2 , x3 , andx4 , and so these must all belong to a single clique if this conditional distribution is
to be absorbed into a clique potential. To ensure this, we add extra links between
all pairs of parents of the node x4 . Anachronistically, this process of ‘marrying
the parents’ has become known as moralization, and the resulting undirected graph,after dropping the arrows, is called the moral graph. It is important to observe thatthe moral graph in this example is fully connected and so exhibits no conditional
independence properties, in contrast to the original directed graph.
Thus in general to convert a directed graph into an undirected graph, we ﬁrst add
additional undirected links between all pairs of parents for each node in the graph and392

8. GRAPHICAL MODELS

Section 8.4

Section 8.2

Figure 8.34

then drop the arrows on the original links to give the moral graph. Then we initialize
all of the clique potentials of the moral graph to 1. We then take each conditional
distribution factor in the original directed graph and multiply it into one of the cliquepotentials. There will always exist at least one maximal clique that contains all of
the variables in the factor as a result of the moralization step. Note that in all cases
the partition function is given by Z = 1.
The process of converting a directed graph into an undirected graph plays animportant role in exact inference techniques such as the junction tree algorithm.
Converting from an undirected to a directed representation is much less common
and in general presents problems due to the normalization constraints.
We saw that in going from a directed to an undirected representation we had to
discard some conditional independence properties from the graph. Of course, we
could always trivially convert any distribution over a directed graph into one over anundirected graph by simply using a fully connected undirected graph. This would,
however, discard all conditional independence properties and so would be vacuous.
The process of moralization adds the fewest extra links and so retains the maximum
number of independence properties.
We have seen that the procedure for determining the conditional independence
properties is different between directed and undirected graphs. It turns out that thetwo types of graph can express different conditional independence properties, and
it is worth exploring this issue in more detail. To do so, we return to the view ofa speciﬁc (directed or undirected) graph as a ﬁlter, so that the set of all possible
distributions over the given variables could be reduced to a subset that respects theconditional independencies implied by the graph. A graph is said to be a D map(for ‘dependency map’) of a distribution if every conditional independence statement
satisﬁed by the distribution is reﬂected in the graph. Thus a completely disconnected
graph (no links) will be a trivial D map for any distribution.
Alternatively, we can consider a speciﬁc distribution and ask which graphs havethe appropriate conditional independence properties. If every conditional independence statement implied by a graph is satisﬁed by a speciﬁc distribution, then thegraph is said to be an I map (for ‘independence map’) of that distribution. Clearly afully connected graph will be a trivial I map for any distribution.
If it is the case that every conditional independence property of the distribution
is reﬂected in the graph, and vice versa, then the graph is said to be a perfect map for
Venn diagram illustrating the set of all distributions
P over a given set of variables, together with theset of distributions D that can be represented as aperfect map using a directed graph, and the set U
that can be represented as a perfect map using an
undirected graph.

D

U

P

393

8.4. Inference in Graphical Models
Figure 8.35

A directed graph whose conditional independence
properties cannot be expressed using an undirected
graph over the same three variables.

A

B

C

that distribution. A perfect map is therefore both an I map and a D map.
Consider the set of distributions such that for each distribution there exists a
directed graph that is a perfect map. This set is distinct from the set of distributions
such that for each distribution there exists an undirected graph that is a perfect map.
In addition there are distributions for which neither directed nor undirected graphs
offer a perfect map. This is illustrated as a Venn diagram in Figure 8.34.
Figure 8.35 shows an example of a directed graph that is a perfect map fora distribution satisfying the conditional independence properties A ⊥⊥ B | ∅ andA⊥⊥ B | C. There is no corresponding undirected graph over the same three variables that is a perfect map.
Conversely, consider the undirected graph over four variables shown in Figure 8.36. This graph exhibits the properties A ⊥⊥ B | ∅, C ⊥⊥ D | A ∪ B andA⊥⊥ B | C ∪ D. There is no directed graph over four variables that implies the same
set of conditional independence properties.
The graphical framework can be extended in a consistent way to graphs that
include both directed and undirected links. These are called chain graphs (Lauritzen
and Wermuth, 1989; Frydenberg, 1990), and contain the directed and undirected
graphs considered so far as special cases. Although such graphs can represent a
broader class of distributions than either directed or undirected alone, there remain
distributions for which even a chain graph cannot provide a perfect map. Chaingraphs are not discussed further in this book.
Figure 8.36

CAn undirected graph whose conditional independence
properties cannot be expressed in terms of a directed
graph over the same variables.

A

B

D

8.4. Inference in Graphical Models
We turn now to the problem of inference in graphical models, in which some of
the nodes in a graph are clamped to observed values, and we wish to compute theposterior distributions of one or more subsets of other nodes. As we shall see, we
can exploit the graphical structure both to ﬁnd efﬁcient algorithms for inference, and394

8. GRAPHICAL MODELS
Figure 8.37

A graphical representation of Bayes’ theorem. x
See the text for details.

x

x

y

y

y

(a)

(b)

(c)

to make the structure of those algorithms transparent. Speciﬁcally, we shall see thatmany algorithms can be expressed in terms of the propagation of local messagesaround the graph. In this section, we shall focus primarily on techniques for exact
inference, and in Chapter 10 we shall consider a number of approximate inference
algorithms.
To start with, let us consider the graphical interpretation of Bayes’ theorem.
Suppose we decompose the joint distribution p(x, y) over two variables x and y into
a product of factors in the form p(x, y) = p(x)p(y|x). This can be represented by
the directed graph shown in Figure 8.37(a). Now suppose we observe the value ofy, as indicated by the shaded node in Figure 8.37(b). We can view the marginal
distribution p(x) as a prior over the latent variable x, and our goal is to infer thecorresponding posterior distribution over x. Using the sum and product rules ofprobability we can evaluatep(y|x )p(x )(8.47)
p(y) =x

which can then be used in Bayes’ theorem to calculatep(x|y) =p(y|x)p(x).
p(y)(8.48)

Thus the joint distribution is now expressed in terms of p(y) and p(x|y). From agraphical perspective, the joint distribution p(x, y) is now represented by the graphshown in Figure 8.37(c), in which the direction of the arrow is reversed. This is thesimplest example of an inference problem for a graphical model.

8.4.1 Inference on a chain
Now consider a more complex problem involving the chain of nodes of the formshown in Figure 8.32. This example will lay the foundation for a discussion of exact
inference in more general graphs later in this section.
Speciﬁcally, we shall consider the undirected graph in Figure 8.32(b). We havealready seen that the directed chain can be transformed into an equivalent undirected
chain. Because the directed graph does not have any nodes with more than one
parent, this does not require the addition of any extra links, and the directed andundirected versions of this graph express exactly the same set of conditional independence statements.

8.4. Inference in Graphical Models

395

The joint distribution for this graph takes the form
p(x) =

1
ψ1,2 (x1 , x2 )ψ2,3 (x2 , x3 ) · · · ψN −1,N (xN −1 , xN ).
Z(8.49)We shall consider the speciﬁc case in which the N nodes represent discrete variables each having K states, in which case each potential function ψn−1,n (xn−1 , xn )comprises an K × K table, and so the joint distribution has (N − 1)K 2 parameters.
Let us consider the inference problem of ﬁnding the marginal distribution p(xn )for a speciﬁc node xn that is part way along the chain. Note that, for the moment,
there are no observed nodes. By deﬁnition, the required marginal is obtained bysumming the joint distribution over all variables except xn , so that

 

···
···
p(x).
(8.50)p(xn ) =x1

xn−1 xn+1

xN

In a naive implementation, we would ﬁrst evaluate the joint distribution andthen perform the summations explicitly. The joint distribution can be represented asa set of numbers, one for each possible value for x. Because there are N variables
each with K states, there are K N values for x and so evaluation and storage of thejoint distribution, as well as marginalization to obtain p(xn ), all involve storage andcomputation that scale exponentially with the length N of the chain.
We can, however, obtain a much more efﬁcient algorithm by exploiting the conditional independence properties of the graphical model. If we substitute the factorized expression (8.49) for the joint distribution into (8.50), then we can rearrange theorder of the summations and the multiplications to allow the required marginal to beevaluated much more efﬁciently. Consider for instance the summation over xN . The
potential ψN −1,N (xN −1 , xN ) is the only one that depends on xN , and so we canperform the summationψN −1,N (xN −1 , xN )(8.51)
xN

ﬁrst to give a function of xN −1 . We can then use this to perform the summation
over xN −1 , which will involve only this new function together with the potential
ψN −2,N −1 (xN −2 , xN −1 ), because this is the only other place that xN −1 appears.
Similarly, the summation over x1 involves only the potential ψ1,2 (x1 , x2 ) and so
can be performed separately to give a function of x2 , and so on. Because eachsummation effectively removes a variable from the distribution, this can be viewed
as the removal of a node from the graph.
If we group the potentials and summations together in this way, we can express

396

8. GRAPHICAL MODELS
the desired marginal in the form
1p(xn ) =
Z
⎡





⎣
ψn−1,n (xn−1 , xn ) · · ·ψ2,3 (x2 , x3 )
ψ1,2 (x1 , x2 )

(

xn−1

⎡
⎣
(



xn+1

x2


ψn,n+1 (xn , xn+1 ) · · ·)*µα (xn )


xN

x1

···⎦

+
⎤

ψN −1,N (xN −1 , xN ) · · · ⎦ .

)*µβ (xn )⎤(8.52)+The reader is encouraged to study this re-ordering carefully as the underlying idea
forms the basis for the later discussion of the general sum-product algorithm. Herethe key concept that we are exploiting is that multiplication is distributive over addition, so thatab + ac = a(b + c)
(8.53)
in which the left-hand side involves three arithmetic operations whereas the righthand side reduces this to two operations.
Let us work out the computational cost of evaluating the required marginal using
this re-ordered expression. We have to perform N − 1 summations each of which isover K states and each of which involves a function of two variables. For instance,
the summation over x1 involves only the function ψ1,2 (x1 , x2 ), which is a table of
K × K numbers. We have to sum this table over x1 for each value of x2 and so thishas O(K 2 ) cost. The resulting vector of K numbers is multiplied by the matrix of
numbers ψ2,3 (x2 , x3 ) and so is again O(K 2 ). Because there are N − 1 summations
and multiplications of this kind, the total cost of evaluating the marginal p(xn ) isO(N K 2 ). This is linear in the length of the chain, in contrast to the exponential costof a naive approach. We have therefore been able to exploit the many conditional
independence properties of this simple graph in order to obtain an efﬁcient calculation. If the graph had been fully connected, there would have been no conditional
independence properties, and we would have been forced to work directly with thefull joint distribution.
We now give a powerful interpretation of this calculation in terms of the passingof local messages around on the graph. From (8.52) we see that the expression for themarginal p(xn ) decomposes into the product of two factors times the normalization
constant1p(xn ) = µα (xn )µβ (xn ).
(8.54)Z
We shall interpret µα (xn ) as a message passed forwards along the chain from node
xn−1 to node xn . Similarly, µβ (xn ) can be viewed as a message passed backwards

8.4. Inference in Graphical Models
Figure 8.38 The marginal distributionp(xn ) for a node xn along the chain is obtained by multiplying the two messagesµα (xn ) and µβ (xn ), and then normalizing. These messages can themselves
be evaluated recursively by passing messages from both ends of the chain towards node xn .

µα (xn−1 )x1µβ (xn )µα (xn )

xn−1

xn

397

µβ (xn+1 )

xn+1

xN

along the chain to node xn from node xn+1 . Note that each of the messages comprises a set of K values, one for each choice of xn , and so the product of two messages should be interpreted as the point-wise multiplication of the elements of thetwo messages to give another set of K values.
The message µα (xn ) can be evaluated recursively because⎡⎤µα (xn ) =ψn−1,n (xn−1 , xn ) ⎣
···⎦
xn−1

=



xn−2

ψn−1,n (xn−1 , xn )µα (xn−1 ).

(8.55)

xn−1

We therefore ﬁrst evaluateµα (x2 ) =

ψ1,2 (x1 , x2 )(8.56)

x1

and then apply (8.55) repeatedly until we reach the desired node. Note carefully the
structure of the message passing equation. The outgoing message µα (xn ) in (8.55)
is obtained by multiplying the incoming message µα (xn−1 ) by the local potential
involving the node variable and the outgoing variable and then summing over thenode variable.
Similarly, the message µβ (xn ) can be evaluated recursively by starting with
node xN and using⎡⎤µβ (xn ) =ψn+1,n (xn+1 , xn ) ⎣
···⎦
xn+1

=



xn+2

ψn+1,n (xn+1 , xn )µβ (xn+1 ).

(8.57)

xn+1

This recursive message passing is illustrated in Figure 8.38. The normalization constant Z is easily evaluated by summing the right-hand side of (8.54) over all statesof xn , an operation that requires only O(K) computation.
Graphs of the form shown in Figure 8.38 are called Markov chains, and thecorresponding message passing equations represent an example of the ChapmanKolmogorov equations for Markov processes (Papoulis, 1984).

398

8. GRAPHICAL MODELS

Exercise 8.15

Chapter 9

Now suppose we wish to evaluate the marginals p(xn ) for every node n ∈{1, . . . , N } in the chain. Simply applying the above procedure separately for eachnode will have computational cost that is O(N 2 M 2 ). However, such an approach
would be very wasteful of computation. For instance, to ﬁnd p(x1 ) we need to propagate a message µβ (·) from node xN back to node x2 . Similarly, to evaluate p(x2 )we need to propagate a messages µβ (·) from node xN back to node x3 . This will
involve much duplicated computation because most of the messages will be identicalin the two cases.
Suppose instead we ﬁrst launch a message µβ (xN −1 ) starting from node xN
and propagate corresponding messages all the way back to node x1 , and suppose wesimilarly launch a message µα (x2 ) starting from node x1 and propagate the corresponding messages all the way forward to node xN . Provided we store all of theintermediate messages along the way, then any node can evaluate its marginal simply by applying (8.54). The computational cost is only twice that for ﬁnding themarginal of a single node, rather than N times as much. Observe that a message
has passed once in each direction across each link in the graph. Note also that thenormalization constant Z need be evaluated only once, using any convenient node.
If some of the nodes in the graph are observed, then the corresponding variables
are simply clamped to their observed values and there is no summation. To see
xn canthis, note that the effect of clamping a variable xn to an observed value 
be expressed by multiplying the joint distribution by (one or more copies of) anxn ), which takes the value 1 when xn = 
xn and the value
additional function I(xn , 
0 otherwise. One such function can then be absorbed into each of the potentials thatxn .
contain xn . Summations over xn then contain only one term in which xn = 
Now suppose we wish to calculate the joint distribution p(xn−1 , xn ) for twoneighbouring nodes on the chain. This is similar to the evaluation of the marginal
for a single node, except that there are now two variables that are not summed out.
A few moments thought will show that the required joint distribution can be written
in the form
1
(8.58)
p(xn−1 , xn ) = µα (xn−1 )ψn−1,n (xn−1 , xn )µβ (xn ).
Z
Thus we can obtain the joint distributions over all of the sets of variables in eachof the potentials directly once we have completed the message passing required toobtain the marginals.
This is a useful result because in practice we may wish to use parametric forms
for the clique potentials, or equivalently for the conditional distributions if we started
from a directed graph. In order to learn the parameters of these potentials in situations where not all of the variables are observed, we can employ the EM algorithm,
and it turns out that the local joint distributions of the cliques, conditioned on any
observed data, is precisely what is needed in the E step. We shall consider someexamples of this in detail in Chapter 13.

8.4.2 Trees
We have seen that exact inference on a graph comprising a chain of nodes can be
performed efﬁciently in time that is linear in the number of nodes, using an algorithm

399

8.4. Inference in Graphical Models
Figure 8.39 Examples of treestructured graphs, showing (a) an
undirected tree, (b) a directed tree,and (c) a directed polytree.

(a)

Exercise 8.18

(b)

(c)

that can be interpreted in terms of messages passed along the chain. More generally,
inference can be performed efﬁciently using local message passing on a broader
class of graphs called trees. In particular, we shall shortly generalize the message
passing formalism derived above for chains to give the sum-product algorithm, which
provides an efﬁcient framework for exact inference in tree-structured graphs.
In the case of an undirected graph, a tree is deﬁned as a graph in which there
is one, and only one, path between any pair of nodes. Such graphs therefore do nothave loops. In the case of directed graphs, a tree is deﬁned such that there is a single
node, called the root, which has no parents, and all other nodes have one parent. If
we convert a directed tree into an undirected graph, we see that the moralization step
will not add any links as all nodes have at most one parent, and as a consequence thecorresponding moralized graph will be an undirected tree. Examples of undirected
and directed trees are shown in Figure 8.39(a) and 8.39(b). Note that a distribution
represented as a directed tree can easily be converted into one represented by anundirected tree, and vice versa.
If there are nodes in a directed graph that have more than one parent, but there is
still only one path (ignoring the direction of the arrows) between any two nodes, thenthe graph is a called a polytree, as illustrated in Figure 8.39(c). Such a graph will
have more than one node with the property of having no parents, and furthermore,
the corresponding moralized undirected graph will have loops.

8.4.3 Factor graphs
The sum-product algorithm that we derive in the next section is applicable toundirected and directed trees and to polytrees. It can be cast in a particularly simple
and general form if we ﬁrst introduce a new graphical construction called a factor
graph (Frey, 1998; Kschischnang et al., 2001).
Both directed and undirected graphs allow a global function of several variables to be expressed as a product of factors over subsets of those variables. Factor
graphs make this decomposition explicit by introducing additional nodes for the factors themselves in addition to the nodes representing the variables. They also allow
us to be more explicit about the details of the factorization, as we shall see.
Let us write the joint distribution over a set of variables in the form of a productof factorsfs (xs )(8.59)
p(x) =swhere xs denotes a subset of the variables. For convenience, we shall denote the400

8. GRAPHICAL MODELS
Figure 8.40

x1

Example of a factor graph, which corresponds
to the factorization (8.60).

x2

fa

fb

x3

fc

fd

individual variables by xi , however, as in earlier discussions, these can comprisegroups of variables (such as vectors or matrices). Each factor fs is a function of acorresponding set of variables xs .
Directed graphs, whose factorization is deﬁned by (8.5), represent special casesof (8.59) in which the factors fs (xs ) are local conditional distributions. Similarly,
undirected graphs, given by (8.39), are a special case in which the factors are potential functions over the maximal cliques (the normalizing coefﬁcient 1/Z can beviewed as a factor deﬁned over the empty set of variables).
In a factor graph, there is a node (depicted as usual by a circle) for every variablein the distribution, as was the case for directed and undirected graphs. There are also
additional nodes (depicted by small squares) for each factor fs (xs ) in the joint distribution. Finally, there are undirected links connecting each factor node to all of thevariables nodes on which that factor depends. Consider, for example, a distributionthat is expressed in terms of the factorizationp(x) = fa (x1 , x2 )fb (x1 , x2 )fc (x2 , x3 )fd (x3 ).

(8.60)

This can be expressed by the factor graph shown in Figure 8.40. Note that there are
two factors fa (x1 , x2 ) and fb (x1 , x2 ) that are deﬁned over the same set of variables.
In an undirected graph, the product of two such factors would simply be lumped
together into the same clique potential. Similarly, fc (x2 , x3 ) and fd (x3 ) could be
combined into a single potential over x2 and x3 . The factor graph, however, keeps
such factors explicit and so is able to convey more detailed information about theunderlying factorization.
x1

x2

x1

x2
f

x1

x2
fa

fb
x3
(a)

x3
(b)

x3
(c)

Figure 8.41 (a) An undirected graph with a single clique potential ψ(x1 , x2 , x3 ). (b) A factor graph with factor
f (x1 , x2 , x3 ) = ψ(x1 , x2 , x3 ) representing the same distribution as the undirected graph. (c) A different factor
graph representing the same distribution, whose factors satisfy fa (x1 , x2 , x3 )fb (x1 , x2 ) = ψ(x1 , x2 , x3 ).

401

8.4. Inference in Graphical Models
x1

x2

x1

x2

x1

f

x2
fc

fa
x3
(a)

fb

x3
(b)

x3
(c)

Figure 8.42 (a) A directed graph with the factorization p(x1 )p(x2 )p(x3 |x1 , x2 ). (b) A factor graph representing
the same distribution as the directed graph, whose factor satisﬁes f (x1 , x2 , x3 ) = p(x1 )p(x2 )p(x3 |x1 , x2 ). (c)A different factor graph representing the same distribution with factors fa (x1 ) = p(x1 ), fb (x2 ) = p(x2 ) andfc (x1 , x2 , x3 ) = p(x3 |x1 , x2 ).

Factor graphs are said to be bipartite because they consist of two distinct kindsof nodes, and all links go between nodes of opposite type. In general, factor graphs
can therefore always be drawn as two rows of nodes (variable nodes at the top andfactor nodes at the bottom) with links between the rows, as shown in the example in
Figure 8.40. In some situations, however, other ways of laying out the graph may
be more intuitive, for example when the factor graph is derived from a directed orundirected graph, as we shall see.
If we are given a distribution that is expressed in terms of an undirected graph,then we can readily convert it to a factor graph. To do this, we create variable nodes
corresponding to the nodes in the original undirected graph, and then create additional factor nodes corresponding to the maximal cliques xs . The factors fs (xs ) arethen set equal to the clique potentials. Note that there may be several different factor
graphs that correspond to the same undirected graph. These concepts are illustrated
in Figure 8.41.
Similarly, to convert a directed graph to a factor graph, we simply create variablenodes in the factor graph corresponding to the nodes of the directed graph, and thencreate factor nodes corresponding to the conditional distributions, and then ﬁnally
add the appropriate links. Again, there can be multiple factor graphs all of whichcorrespond to the same directed graph. The conversion of a directed graph to afactor graph is illustrated in Figure 8.42.
We have already noted the importance of tree-structured graphs for performing
efﬁcient inference. If we take a directed or undirected tree and convert it into a factor
graph, then the result will again be a tree (in other words, the factor graph will have
no loops, and there will be one and only one path connecting any two nodes). In
the case of a directed polytree, conversion to an undirected graph results in loops
due to the moralization step, whereas conversion to a factor graph again results in atree, as illustrated in Figure 8.43. In fact, local cycles in a directed graph due tolinks connecting parents of a node can be removed on conversion to a factor graphby deﬁning the appropriate factor function, as shown in Figure 8.44.
We have seen that multiple different factor graphs can represent the same directed or undirected graph. This allows factor graphs to be more speciﬁc about the402

8. GRAPHICAL MODELS

(a)

(b)

(c)

Figure 8.43 (a) A directed polytree. (b) The result of converting the polytree into an undirected graph showing
the creation of loops. (c) The result of converting the polytree into a factor graph, which retains the tree structure.

precise form of the factorization. Figure 8.45 shows an example of a fully connected
undirected graph along with two different factor graphs. In (b), the joint distribution is given by a general form p(x) = f (x1 , x2 , x3 ), whereas in (c), it is givenby the more speciﬁc factorization p(x) = fa (x1 , x2 )fb (x1 , x3 )fc (x2 , x3 ). It shouldbe emphasized that the factorization in (c) does not correspond to any conditional
independence properties.

8.4.4 The sum-product algorithm

Section 13.3
Figure 8.44

We shall now make use of the factor graph framework to derive a powerful classof efﬁcient, exact inference algorithms that are applicable to tree-structured graphs.
Here we shall focus on the problem of evaluating local marginals over nodes orsubsets of nodes, which will lead us to the sum-product algorithm. Later we shallmodify the technique to allow the most probable state to be found, giving rise to the
max-sum algorithm.
Also we shall suppose that all of the variables in the model are discrete, andso marginalization corresponds to performing sums. The framework, however, isequally applicable to linear-Gaussian models in which case marginalization involves
integration, and we shall consider an example of this in detail when we discuss linear
dynamical systems.
(a) A fragment of a directed graph having a local cycle. (b) Conversion
to a fragment of a factor
graph having a tree structure, in which f (x1 , x2 , x3 ) =p(x1 )p(x2 |x1 )p(x3 |x1 , x2 ).

x1

x2

x1

x2

f (x1 , x2 , x3 )
x3
(a)

x3
(b)

403

8.4. Inference in Graphical Models
x1

x2

x1

x2

x1

f (x1 , x2 , x3 )
x3
(a)

x2

fa

fb

fc

x3

x3

(b)

(c)

Figure 8.45 (a) A fully connected undirected graph. (b) and (c) Two factor graphs each of which corresponds
to the undirected graph in (a).

There is an algorithm for exact inference on directed graphs without loops known
as belief propagation (Pearl, 1988; Lauritzen and Spiegelhalter, 1988), and is equivalent to a special case of the sum-product algorithm. Here we shall consider only thesum-product algorithm because it is simpler to derive and to apply, as well as being
more general.
We shall assume that the original graph is an undirected tree or a directed tree or
polytree, so that the corresponding factor graph has a tree structure. We ﬁrst convert
the original graph into a factor graph so that we can deal with both directed andundirected models using the same framework. Our goal is to exploit the structure of
the graph to achieve two things: (i) to obtain an efﬁcient, exact inference algorithm
for ﬁnding marginals; (ii) in situations where several marginals are required to allow
computations to be shared efﬁciently.
We begin by considering the problem of ﬁnding the marginal p(x) for particular variable node x. For the moment, we shall suppose that all of the variables
are hidden. Later we shall see how to modify the algorithm to incorporate evidence
corresponding to observed variables. By deﬁnition, the marginal is obtained by summing the joint distribution over all variables except x so thatp(x)(8.61)
p(x) =
x\x

where x \ x denotes the set of variables in x with variable x omitted. The idea isto substitute for p(x) using the factor graph expression (8.59) and then interchangesummations and products in order to obtain an efﬁcient algorithm. Consider the
fragment of graph shown in Figure 8.46 in which we see that the tree structure of
the graph allows us to partition the factors in the joint distribution into groups, with
one group associated with each of the factor nodes that is a neighbour of the variablenode x. We see that the joint distribution can be written as a product of the formFs (x, Xs )(8.62)
p(x) =s∈ne(x)ne(x) denotes the set of factor nodes that are neighbours of x, and Xs denotes theset of all variables in the subtree connected to the variable node x via the factor node

8. GRAPHICAL MODELS
Figure 8.46

A fragment of a factor graph illustrating theevaluation of the marginal p(x).

µfs →x (x)Fs (x, Xs )

404

fs

x

fs , and Fs (x, Xs ) represents the product of all the factors in the group associatedwith factor fs .
Substituting (8.62) into (8.61) and interchanging the sums and products, we obtain 
Fs (x, Xs )p(x) =s∈ne(x)=Xsµfs →x (x).

(8.63)s∈ne(x)Here we have introduced a set of functions µfs →x (x), deﬁned byµfs →x (x) ≡Fs (x, Xs )(8.64)

Xs

which can be viewed as messages from the factor nodes fs to the variable node x.
We see that the required marginal p(x) is given by the product of all the incoming
messages arriving at node x.
In order to evaluate these messages, we again turn to Figure 8.46 and note thateach factor Fs (x, Xs ) is described by a factor (sub-)graph and so can itself be factorized. In particular, we can write
Fs (x, Xs ) = fs (x, x1 , . . . , xM )G1 (x1 , Xs1 ) . . . GM (xM , XsM )(8.65)

where, for convenience, we have denoted the variables associated with factor fx , inaddition to x, by x1 , . . . , xM . This factorization is illustrated in Figure 8.47. Notethat the set of variables {x, x1 , . . . , xM } is the set of variables on which the factor
fs depends, and so it can also be denoted xs , using the notation of (8.59).
Substituting (8.65) into (8.64) we obtain


 

µfs →x (x) =...
fs (x, x1 , . . . , xM )Gm (xm , Xsm )
x1

=


x1

xM

...


xM

m∈ne(fs )\x

fs (x, x1 , . . . , xM )m∈ne(fs )\xXxmµxm →fs (xm )(8.66)

8.4. Inference in Graphical Models
Figure 8.47

405

xM

Illustration of the factorization of the subgraph associated with factor node fs .

µxM →fs (xM )fsµfs →x (x)xxmGm (xm , Xsm )where ne(fs ) denotes the set of variable nodes that are neighbours of the factor node
fs , and ne(fs ) \ x denotes the same set but with node x removed. Here we havedeﬁned the following messages from variable nodes to factor nodes
µxm →fs (xm ) ≡Gm (xm , Xsm ).
(8.67)Xsm

We have therefore introduced two distinct kinds of message, those that go from factor
nodes to variable nodes denoted µf →x (x), and those that go from variable nodes tofactor nodes denoted µx→f (x). In each case, we see that messages passed along alink are always a function of the variable associated with the variable node that linkconnects to.
The result (8.66) says that to evaluate the message sent by a factor node to a variable node along the link connecting them, take the product of the incoming messages
along all other links coming into the factor node, multiply by the factor associatedwith that node, and then marginalize over all of the variables associated with theincoming messages. This is illustrated in Figure 8.47. It is important to note thata factor node can send a message to a variable node once it has received incoming
messages from all other neighbouring variable nodes.
Finally, we derive an expression for evaluating the messages from variable nodes
to factor nodes, again by making use of the (sub-)graph factorization. From Figure 8.48, we see that term Gm (xm , Xsm ) associated with node xm is given by a
product of terms Fl (xm , Xml ) each associated with one of the factor nodes fl that is
linked to node xm (excluding node fs ), so thatGm (xm , Xsm ) =Fl (xm , Xml )(8.68)l∈ne(xm )\fs

where the product is taken over all neighbours of node xm except for node fs . Notethat each of the factors Fl (xm , Xml ) represents a subtree of the original graph ofprecisely the same kind as introduced in (8.62). Substituting (8.68) into (8.67), we406

8. GRAPHICAL MODELS
Figure 8.48

Illustration of the evaluation of the message sent by avariable node to an adjacent factor node.

fL

fs

xm
fl
Fl (xm , Xml )then obtainµxm →fs (xm ) =l∈ne(xm )\fs

=







Fl (xm , Xml )Xmlµfl →xm (xm )(8.69)l∈ne(xm )\fs

where we have used the deﬁnition (8.64) of the messages passed from factor nodes tovariable nodes. Thus to evaluate the message sent by a variable node to an adjacentfactor node along the connecting link, we simply take the product of the incoming
messages along all of the other links. Note that any variable node that has only
two neighbours performs no computation but simply passes messages through unchanged. Also, we note that a variable node can send a message to a factor node
once it has received incoming messages from all other neighbouring factor nodes.
Recall that our goal is to calculate the marginal for variable node x, and that thismarginal is given by the product of incoming messages along all of the links arriving
at that node. Each of these messages can be computed recursively in terms of othermessages. In order to start this recursion, we can view the node x as the root of thetree and begin at the leaf nodes. From the deﬁnition (8.69), we see that if a leaf node
is a variable node, then the message that it sends along its one and only link is givenby(8.70)µx→f (x) = 1
as illustrated in Figure 8.49(a). Similarly, if the leaf node is a factor node, we see
from (8.66) that the message sent should take the form
µf →x (x) = f (x)Figure 8.49

The sum-product algorithm
begins with messages sentby the leaf nodes, which depend on whether the leaf
node is (a) a variable node,or (b) a factor node.

µf →x (x) = f (x)µx→f (x) = 1
x

f
(a)

(8.71)

xf(b)

8.4. Inference in Graphical Models

Exercise 8.20

407

as illustrated in Figure 8.49(b).
At this point, it is worth pausing to summarize the particular version of the sumproduct algorithm obtained so far for evaluating the marginal p(x). We start byviewing the variable node x as the root of the factor graph and initiating messages
at the leaves of the graph using (8.70) and (8.71). The message passing steps (8.66)
and (8.69) are then applied recursively until messages have been propagated along
every link, and the root node has received messages from all of its neighbours. Eachnode can send a message towards the root once it has received messages from allof its other neighbours. Once the root node has received messages from all of its
neighbours, the required marginal can be evaluated using (8.63). We shall illustrate
this process shortly.
To see that each node will always receive enough messages to be able to send out
a message, we can use a simple inductive argument as follows. Clearly, for a graphcomprising a variable root node connected directly to several factor leaf nodes, thealgorithm trivially involves sending messages of the form (8.71) directly from theleaves to the root. Now imagine building up a general graph by adding nodes one ata time, and suppose that for some particular graph we have a valid algorithm. Whenone more (variable or factor) node is added, it can be connected only by a single
link because the overall graph must remain a tree, and so the new node will be a leaf
node. It therefore sends a message to the node to which it is linked, which in turnwill therefore receive all the messages it requires in order to send its own message
towards the root, and so again we have a valid algorithm, thereby completing theproof.
Now suppose we wish to ﬁnd the marginals for every variable node in the graph.
This could be done by simply running the above algorithm afresh for each such node.
However, this would be very wasteful as many of the required computations would
be repeated. We can obtain a much more efﬁcient procedure by ‘overlaying’ these
multiple message passing algorithms to obtain the general sum-product algorithm
as follows. Arbitrarily pick any (variable or factor) node and designate it as theroot. Propagate messages from the leaves to the root as before. At this point, theroot node will have received messages from all of its neighbours. It can therefore
send out messages to all of its neighbours. These in turn will then have received
messages from all of their neighbours and so can send out messages along the links
going away from the root, and so on. In this way, messages are passed outwards
from the root all the way to the leaves. By now, a message will have passed inboth directions across every link in the graph, and every node will have received
a message from all of its neighbours. Again a simple inductive argument can beused to verify the validity of this message passing protocol. Because every variablenode will have received messages from all of its neighbours, we can readily calculate
the marginal distribution for every variable in the graph. The number of messages
that have to be computed is given by twice the number of links in the graph andso involves only twice the computation involved in ﬁnding a single marginal. By
comparison, if we had run the sum-product algorithm separately for each node, theamount of computation would grow quadratically with the size of the graph. Notethat this algorithm is in fact independent of which node was designated as the root,408

8. GRAPHICAL MODELS
Figure 8.50

Exercise 8.21

The sum-product algorithm can be viewed
purely in terms of messages sent out by factor
nodes to other factor nodes. In this example,
the outgoing message shown by the blue arrowis obtained by taking the product of all the incoming messages shown by green arrows, multiplying by the factor fs , and marginalizing overthe variables x1 and x2 .

x1
x3
x2

fs

and indeed the notion of one node having a special status was introduced only as aconvenient way to explain the message passing protocol.
Next suppose we wish to ﬁnd the marginal distributions p(xs ) associated withthe sets of variables belonging to each of the factors. By a similar argument to thatused above, it is easy to see that the marginal associated with a factor is given by the
product of messages arriving at the factor node and the local factor at that nodeµxi →fs (xi )(8.72)p(xs ) = fs (xs )i∈ne(fs )in complete analogy with the marginals at the variable nodes. If the factors areparameterized functions and we wish to learn the values of the parameters using
the EM algorithm, then these marginals are precisely the quantities we will need to
calculate in the E step, as we shall see in detail when we discuss the hidden Markov
model in Chapter 13.
The message sent by a variable node to a factor node, as we have seen, is simply
the product of the incoming messages on other links. We can if we wish view thesum-product algorithm in a slightly different form by eliminating messages fromvariable nodes to factor nodes and simply considering messages that are sent out byfactor nodes. This is most easily seen by considering the example in Figure 8.50.
So far, we have rather neglected the issue of normalization. If the factor graph
was derived from a directed graph, then the joint distribution is already correctly normalized, and so the marginals obtained by the sum-product algorithm will similarly
be normalized correctly. However, if we started from an undirected graph, then ingeneral there will be an unknown normalization coefﬁcient 1/Z. As with the simple
chain example of Figure 8.38, this is easily handled by working with an unnormalp(x) of the joint distribution, where p(x) = 
p(x)/Z. We ﬁrst run theized version 
sum-product algorithm to ﬁnd the corresponding unnormalized marginals p(xi ). The
coefﬁcient 1/Z is then easily obtained by normalizing any one of these marginals,
and this is computationally efﬁcient because the normalization is done over a single
variable rather than over the entire set of variables as would be required to normalizep(x) directly.
At this point, it may be helpful to consider a simple example to illustrate theoperation of the sum-product algorithm. Figure 8.51 shows a simple 4-node factor

8.4. Inference in Graphical Models
Figure 8.51

x1

A simple factor graph used to illustrate thesum-product algorithm.

x2

409
x3

fa

fb
fc

x4

graph whose unnormalized joint distribution is given byp(x) = fa (x1 , x2 )fb (x2 , x3 )fc (x2 , x4 ).

(8.73)In order to apply the sum-product algorithm to this graph, let us designate node x3
as the root, in which case there are two leaf nodes x1 and x4 . Starting with the leaf
nodes, we then have the following sequence of six messagesµx1 →fa (x1 ) = 1
fa (x1 , x2 )µfa →x2 (x2 ) =(8.74)(8.75)x1µx4 →fc (x4 ) = 1
fc (x2 , x4 )µfc →x2 (x2 ) =(8.76)(8.77)x4µx2 →fb (x2 ) = µfa →x2 (x2 )µfc →x2 (x2 )fb (x2 , x3 )µx2 →fb .
µfb →x3 (x3 ) =(8.78)(8.79)

x2

The direction of ﬂow of these messages is illustrated in Figure 8.52. Once this message propagation is complete, we can then propagate messages from the root nodeout to the leaf nodes, and these are given byµx3 →fb (x3 ) = 1
fb (x2 , x3 )µfb →x2 (x2 ) =(8.80)(8.81)x3µx2 →fa (x2 ) = µfb →x2 (x2 )µfc →x2 (x2 )
fa (x1 , x2 )µx2 →fa (x2 )µfa →x1 (x1 ) =(8.82)(8.83)x2µx2 →fc (x2 ) = µfa →x2 (x2 )µfb →x2 (x2 )fc (x2 , x4 )µx2 →fc (x2 ).
µfc →x4 (x4 ) =x2(8.84)(8.85)

410

8. GRAPHICAL MODELS
x1

x2

x3

x1

x2

x4

x4

(a)

(b)

x3

Figure 8.52 Flow of messages for the sum-product algorithm applied to the example graph in Figure 8.51. (a)From the leaf nodes x1 and x4 towards the root node x3 . (b) From the root node towards the leaf nodes.

One message has now passed in each direction across each link, and we can now
evaluate the marginals. As a simple check, let us verify that the marginal p(x2 ) is
given by the correct expression. Using (8.63) and substituting for the messages using
the above results, we havep(x2 ) = µfa →x2 (x2 )µfb →x2 (x2 )µfc →x2 (x2 )






fa (x1 , x2 )fb (x2 , x3 )fc (x2 , x4 )
=
x1

=
=


x1

x2

x4

x1

x3

x4



x3

x4

fa (x1 , x2 )fb (x2 , x3 )fc (x2 , x4 )p(x)(8.86)

as required.
So far, we have assumed that all of the variables in the graph are hidden. In mostpractical applications, a subset of the variables will be observed, and we wish to calculate posterior distributions conditioned on these observations. Observed nodes areeasily handled within the sum-product algorithm as follows. Suppose we partition x
into hidden variables h and observed variables v, and that the observed
value of v
. Then we simply multiply the joint distribution p(x) by i I(vi , vi ),is denoted v
where I(v, v ) = 1 if v = 
v and I(v, v ) = 0 otherwise. This product corresponds) and hence is an unnormalized version of p(h|v = v
). By runto p(h, v = vning the sum-product algorithm, we can efﬁciently calculate the posterior marginalsp(hi |v = v) up to a normalization coefﬁcient whose value can be found efﬁciently
using a local computation. Any summations over variables in v then collapse into asingle term.
We have assumed throughout this section that we are dealing with discrete variables. However, there is nothing speciﬁc to discrete variables either in the graphicalframework or in the probabilistic construction of the sum-product algorithm. For

8.4. Inference in Graphical Models
Table 8.1

Section 13.3

Example of a joint distribution over two binary variables forwhich the maximum of the joint distribution occurs for different variable values compared to the maxima of the two
marginals.

y=0
y=1

x=0
0.3
0.3

411
x=1
0.4
0.0

continuous variables the summations are simply replaced by integrations. We shallgive an example of the sum-product algorithm applied to a graph of linear-Gaussianvariables when we consider linear dynamical systems.

8.4.5 The max-sum algorithm
The sum-product algorithm allows us to take a joint distribution p(x) expressed
as a factor graph and efﬁciently ﬁnd marginals over the component variables. Twoother common tasks are to ﬁnd a setting of the variables that has the largest probability and to ﬁnd the value of that probability. These can be addressed through aclosely related algorithm called max-sum, which can be viewed as an application of
dynamic programming in the context of graphical models (Cormen et al., 2001).
A simple approach to ﬁnding latent variable values having high probability
would be to run the sum-product algorithm to obtain the marginals p(xi ) for every variable, and then, for each marginal in turn, to ﬁnd the value xi that maximizes
that marginal. However, this would give the set of values that are individually themost probable. In practice, we typically wish to ﬁnd the set of values that jointly
have the largest probability, in other words the vector xmax that maximizes the joint
distribution, so that
(8.87)
xmax = arg max p(x)x

for which the corresponding value of the joint probability will be given byp(xmax ) = max p(x).
xExercise 8.27

(8.88)

In general, xmax is not the same as the set of xi values, as we can easily show using
a simple example. Consider the joint distribution p(x, y) over two binary variables
x, y ∈ {0, 1} given in Table 8.1. The joint distribution is maximized by setting x =1 and y = 0, corresponding the value 0.4. However, the marginal for p(x), obtainedby summing over both values of y, is given by p(x = 0) = 0.6 and p(x = 1) = 0.4,
and similarly the marginal for y is given by p(y = 0) = 0.7 and p(y = 1) = 0.3,
and so the marginals are maximized by x = 0 and y = 0, which corresponds to avalue of 0.3 for the joint distribution. In fact, it is not difﬁcult to construct examples
for which the set of individually most probable values has probability zero under thejoint distribution.
We therefore seek an efﬁcient algorithm for ﬁnding the value of x that maximizes the joint distribution p(x) and that will allow us to obtain the value of thejoint distribution at its maximum. To address the second of these problems, we shall
simply write out the max operator in terms of its components
max p(x) = max . . . max p(x)
x

x1

xM

(8.89)

412

8. GRAPHICAL MODELS
where M is the total number of variables, and then substitute for p(x) using its
expansion in terms of a product of factors. In deriving the sum-product algorithm,
we made use of the distributive law (8.53) for multiplication. Here we make use of
the analogous law for the max operator
max(ab, ac) = a max(b, c)(8.90)

which holds if a  0 (as will always be the case for the factors in a graphical model).
This allows us to exchange products with maximizations.
Consider ﬁrst the simple example of a chain of nodes described by (8.49). The
evaluation of the probability maximum can be written as
1
max · · · max [ψ1,2 (x1 , x2 ) · · · ψN −1,N (xN −1 , xN )]
xN
Z x1



1
max ψ1,2 (x1 , x2 ) · · · max ψN −1,N (xN −1 , xN ) .
xN
Z x1

max p(x) =x=

As with the calculation of marginals, we see that exchanging the max and productoperators results in a much more efﬁcient computation, and one that is easily interpreted in terms of messages passed from node xN backwards along the chain to nodex1 .
We can readily generalize this result to arbitrary tree-structured factor graphs
by substituting the expression (8.59) for the factor graph expansion into (8.89) andagain exchanging maximizations with products. The structure of this calculation is
identical to that of the sum-product algorithm, and so we can simply translate thoseresults into the present context. In particular, suppose that we designate a particularvariable node as the ‘root’ of the graph. Then we start a set of messages propagating
inwards from the leaves of the tree towards the root, with each node sending its
message towards the root once it has received all incoming messages from its otherneighbours. The ﬁnal maximization is performed over the product of all messages
arriving at the root node, and gives the maximum value for p(x). This could be called
the max-product algorithm and is identical to the sum-product algorithm except thatsummations are replaced by maximizations. Note that at this stage, messages havebeen sent from leaves to the root, but not in the other direction.
In practice, products of many small probabilities can lead to numerical underﬂow problems, and so it is convenient to work with the logarithm of the joint distribution. The logarithm is a monotonic function, so that if a > b then ln a > ln b, andhence the max operator and the logarithm function can be interchanged, so that
(8.91)
ln max p(x) = max ln p(x).
xxThe distributive property is preserved because
max(a + b, a + c) = a + max(b, c).

(8.92)

Thus taking the logarithm simply has the effect of replacing the products in the
max-product algorithm with sums, and so we obtain the max-sum algorithm. From

8.4. Inference in Graphical Models413

the results (8.66) and (8.69) derived earlier for the sum-product algorithm, we canreadily write down the max-sum algorithm in terms of message passing simply by
replacing ‘sum’ with ‘max’ and replacing products with sums of logarithms to give⎡⎤µf →x (x) =max ⎣ln f (x, x1 , . . . , xM ) +µxm →f (xm )⎦ (8.93)
x1 ,...,xMµx→f (x) =m∈ne(fs )\xµfl →x (x).

(8.94)l∈ne(x)\f

The initial messages sent by the leaf nodes are obtained by analogy with (8.70) and(8.71) and are given byµx→f (x) = 0µf →x (x) = ln f (x)(8.95)(8.96)

while at the root node the maximum probability can then be computed, by analogy
with (8.63), using⎡⎤µfs →x (x)⎦ .
(8.97)
pmax = max ⎣xs∈ne(x)

So far, we have seen how to ﬁnd the maximum of the joint distribution by propagating messages from the leaves to an arbitrarily chosen root node. The result willbe the same irrespective of which node is chosen as the root. Now we turn to thesecond problem of ﬁnding the conﬁguration of the variables for which the joint distribution attains this maximum value. So far, we have sent messages from the leaves
to the root. The process of evaluating (8.97) will also give the value xmax for themost probable value of the root node variable, deﬁned by⎡⎤µfs →x (x)⎦ .
(8.98)
xmax = arg max ⎣xs∈ne(x)At this point, we might be tempted simply to continue with the message passing algorithm and send messages from the root back out to the leaves, using (8.93) and(8.94), then apply (8.98) to all of the remaining variable nodes. However, because
we are now maximizing rather than summing, it is possible that there may be multiple conﬁgurations of x all of which give rise to the maximum value for p(x). In
such cases, this strategy can fail because it is possible for the individual variablevalues obtained by maximizing the product of messages at each node to belong todifferent maximizing conﬁgurations, giving an overall conﬁguration that no longer
corresponds to a maximum.
The problem can be resolved by adopting a rather different kind of message
passing from the root node to the leaves. To see how this works, let us return once
again to the simple chain example of N variables x1 , . . . , xN each having K states,

414

8. GRAPHICAL MODELS

Figure 8.53 A lattice, or trellis, diagram showing explicitly the K possible states (one per row
of the diagram) for each of the variables xn in the k = 1
chain model. In this illustration K = 3. The arrow shows the direction of message passing in the
max-product algorithm. For every state k of eachvariable xn (corresponding to column n of the diagram) the function φ(xn ) deﬁnes a unique state at
the previous variable, indicated by the black lines. k = 2
The two paths through the lattice correspond to
conﬁgurations that give the global maximum of thejoint probability distribution, and either of thesecan be found by tracing back along the black lines
in the opposite direction to the arrow.

k=3
n−2

n−1

n

n+1

corresponding to the graph shown in Figure 8.38. Suppose we take node xN to be
the root node. Then in the ﬁrst phase, we propagate messages from the leaf node x1
to the root node usingµxn →fn,n+1 (xn ) = µfn−1,n →xn (xn )µfn−1,n →xn (xn ) = max ln fn−1,n (xn−1 , xn ) + µxn−1 →f n−1,n (xn )xn−1which follow from applying (8.94) and (8.93) to this particular graph. The initialmessage sent from the leaf node is simplyµx1 →f1,2 (x1 ) = 0.
The most probable value for xN is then given by= arg max µfN −1,N →xN (xN ) .
xmax
N
xN

(8.99)(8.100)

Now we need to determine the states of the previous variables that correspond to thesame maximizing conﬁguration. This can be done by keeping track of which valuesof the variables gave rise to the maximum state of each variable, in other words bystoring quantities given byφ(xn ) = arg max ln fn−1,n (xn−1 , xn ) + µxn−1 →f n−1,n (xn ) .
(8.101)xn−1

To understand better what is happening, it is helpful to represent the chain of variables in terms of a lattice or trellis diagram as shown in Figure 8.53. Note that thisis not a probabilistic graphical model because the nodes represent individual statesof variables, while each variable corresponds to a column of such states in the diagram. For each state of a given variable, there is a unique state of the previous
variable that maximizes the probability (ties are broken either systematically or at
random), corresponding to the function φ(xn ) given by (8.101), and this is indicated8.4. Inference in Graphical Models

415

by the lines connecting the nodes. Once we know the most probable value of the ﬁnal node xN , we can then simply follow the link back to ﬁnd the most probable stateof node xN −1 and so on back to the initial node x1 . This corresponds to propagatinga message back down the chain usingmax
xmax
n−1 = φ(xn )Section 13.2

(8.102)and is known as back-tracking. Note that there could be several values of xn−1 allof which give the maximum value in (8.101). Provided we chose one of these values
when we do the back-tracking, we are assured of a globally consistent maximizing
conﬁguration.
In Figure 8.53, we have indicated two paths, each of which we shall suppose
corresponds to a global maximum of the joint probability distribution. If k = 2
and k = 3 each represent possible values of xmax
N , then starting from either state
and tracing back along the black lines, which corresponds to iterating (8.102), we
obtain a valid global maximum conﬁguration. Note that if we had run a forward
pass of max-sum message passing followed by a backward pass and then applied(8.98) at each node separately, we could end up selecting some states from one pathand some from the other path, giving an overall conﬁguration that is not a global
maximizer. We see that it is necessary instead to keep track of the maximizing states
during the forward pass using the functions φ(xn ) and then use back-tracking to ﬁnd
a consistent solution.
The extension to a general tree-structured factor graph should now be clear. If
a message is sent from a factor node f to a variable node x, a maximization is
performed over all other variable nodes x1 , . . . , xM that are neighbours of that factor node, using (8.93). When we perform this maximization, we keep a record ofwhich values of the variables x1 , . . . , xM gave rise to the maximum. Then in the
back-tracking step, having found xmax , we can then use these stored values to assign consistent maximizing states xmax
, . . . , xmax
1
M . The max-sum algorithm, withback-tracking, gives an exact maximizing conﬁguration for the variables provided
the factor graph is a tree. An important application of this technique is for ﬁnding
the most probable sequence of hidden states in a hidden Markov model, in whichcase it is known as the Viterbi algorithm.
As with the sum-product algorithm, the inclusion of evidence in the form ofobserved variables is straightforward. The observed variables are clamped to theirobserved values, and the maximization is performed over the remaining hidden variables. This can be shown formally by including identity functions for the observed
variables into the factor functions, as we did for the sum-product algorithm.
It is interesting to compare max-sum with the iterated conditional modes (ICM)
algorithm described on page 389. Each step in ICM is computationally simpler because the ‘messages’ that are passed from one node to the next comprise a single
value consisting of the new state of the node for which the conditional distribution
is maximized. The max-sum algorithm is more complex because the messages arefunctions of node variables x and hence comprise a set of K values for each possible state of x. Unlike max-sum, however, ICM is not guaranteed to ﬁnd a global
maximum even for tree-structured graphs.

416

8. GRAPHICAL MODELS

8.4.6 Exact inference in general graphs
The sum-product and max-sum algorithms provide efﬁcient and exact solutions
to inference problems in tree-structured graphs. For many practical applications,however, we have to deal with graphs having loops.
The message passing framework can be generalized to arbitrary graph topologies, giving an exact inference procedure known as the junction tree algorithm (Lauritzen and Spiegelhalter, 1988; Jordan, 2007). Here we give a brief outline of the
key steps involved. This is not intended to convey a detailed understanding of thealgorithm, but rather to give a ﬂavour of the various stages involved. If the starting
point is a directed graph, it is ﬁrst converted to an undirected graph by moralization, whereas if starting from an undirected graph this step is not required. Next thegraph is triangulated, which involves ﬁnding chord-less cycles containing four ormore nodes and adding extra links to eliminate such chord-less cycles. For instance,in the graph in Figure 8.36, the cycle A–C–B–D–A is chord-less a link could be
added between A and B or alternatively between C and D. Note that the joint distribution for the resulting triangulated graph is still deﬁned by a product of the samepotential functions, but these are now considered to be functions over expanded sets
of variables. Next the triangulated graph is used to construct a new tree-structured
undirected graph called a join tree, whose nodes correspond to the maximal cliquesof the triangulated graph, and whose links connect pairs of cliques that have variables in common. The selection of which pairs of cliques to connect in this way isimportant and is done so as to give a maximal spanning tree deﬁned as follows. Of
all possible trees that link up the cliques, the one that is chosen is one for which theweight of the tree is largest, where the weight for a link is the number of nodes sharedby the two cliques it connects, and the weight for the tree is the sum of the weights
for the links. If the tree is condensed, so that any clique that is a subset of anotherclique is absorbed into the larger clique, this gives a junction tree. As a consequenceof the triangulation step, the resulting tree satisﬁes the running intersection property,
which means that if a variable is contained in two cliques, then it must also be contained in every clique on the path that connects them. This ensures that inference
about variables will be consistent across the graph. Finally, a two-stage message
passing algorithm, essentially equivalent to the sum-product algorithm, can now beapplied to this junction tree in order to ﬁnd marginals and conditionals. Although
the junction tree algorithm sounds complicated, at its heart is the simple idea that
we have used already of exploiting the factorization properties of the distribution toallow sums and products to be interchanged so that partial summations can be performed, thereby avoiding having to work directly with the joint distribution. The
role of the junction tree is to provide a precise and efﬁcient way to organize thesecomputations. It is worth emphasizing that this is achieved using purely graphicaloperations!
The junction tree is exact for arbitrary graphs and is efﬁcient in the sense thatfor a given graph there does not in general exist a computationally cheaper approach.
Unfortunately, the algorithm must work with the joint distributions within each node(each of which corresponds to a clique of the triangulated graph) and so the computational cost of the algorithm is determined by the number of variables in the largest

8.4. Inference in Graphical Models

417

clique and will grow exponentially with this number in the case of discrete variables.
An important concept is the treewidth of a graph (Bodlaender, 1993), which is deﬁned in terms of the number of variables in the largest clique. In fact, it is deﬁned to
be as one less than the size of the largest clique, to ensure that a tree has a treewidthof 1. Because there in general there can be multiple different junction trees that canbe constructed from a given starting graph, the treewidth is deﬁned by the junction
tree for which the largest clique has the fewest variables. If the treewidth of theoriginal graph is high, the junction tree algorithm becomes impractical.

8.4.7 Loopy belief propagation
For many problems of practical interest, it will not be feasible to use exact inference, and so we need to exploit effective approximation methods. An importantclass of such approximations, that can broadly be called variational methods, will be
discussed in detail in Chapter 10. Complementing these deterministic approaches isa wide range of sampling methods, also called Monte Carlo methods, that are basedon stochastic numerical sampling from distributions and that will be discussed at
length in Chapter 11.
Here we consider one simple approach to approximate inference in graphs withloops, which builds directly on the previous discussion of exact inference in trees.
The idea is simply to apply the sum-product algorithm even though there is no guarantee that it will yield good results. This approach is known as loopy belief propagation (Frey and MacKay, 1998) and is possible because the message passing rules(8.66) and (8.69) for the sum-product algorithm are purely local. However, because
the graph now has cycles, information can ﬂow many times around the graph. For
some models, the algorithm will converge, whereas for others it will not.
In order to apply this approach, we need to deﬁne a message passing schedule.
Let us assume that one message is passed at a time on any given link and in any
given direction. Each message sent from a node replaces any previous message sentin the same direction across the same link and will itself be a function only of themost recent messages received by that node at previous steps of the algorithm.
We have seen that a message can only be sent across a link from a node when
all other messages have been received by that node across its other links. Because
there are loops in the graph, this raises the problem of how to initiate the message
passing algorithm. To resolve this, we suppose that an initial message given by theunit function has been passed across every link in each direction. Every node is thenin a position to send a message.
There are now many possible ways to organize the message passing schedule.
For example, the ﬂooding schedule simultaneously passes a message across every
link in both directions at each time step, whereas schedules that pass one message ata time are called serial schedules.
Following Kschischnang et al. (2001), we will say that a (variable or factor)
node a has a message pending on its link to a node b if node a has received any
message on any of its other links since the last time it send a message to b. Thus,
when a node receives a message on one of its links, this creates pending messages
on all of its other links. Only pending messages need to be transmitted because

418

8. GRAPHICAL MODELS

Exercise 8.29

other messages would simply duplicate the previous message on the same link. Forgraphs that have a tree structure, any schedule that sends only pending messages
will eventually terminate once a message has passed in each direction across every
link. At this point, there are no pending messages, and the product of the received
messages at every variable give the exact marginal. In graphs having loops, however,
the algorithm may never terminate because there might always be pending messages,
although in practice it is generally found to converge within a reasonable time formost applications. Once the algorithm has converged, or once it has been stopped
if convergence is not observed, the (approximate) local marginals can be computed
using the product of the most recently received incoming messages to each variablenode or factor node on every link.
In some applications, the loopy belief propagation algorithm can give poor results, whereas in other applications it has proven to be very effective. In particular,state-of-the-art algorithms for decoding certain kinds of error-correcting codes areequivalent to loopy belief propagation (Gallager, 1963; Berrou et al., 1993; McEliece
et al., 1998; MacKay and Neal, 1999; Frey, 1998).

8.4.8 Learning the graph structure
In our discussion of inference in graphical models, we have assumed that thestructure of the graph is known and ﬁxed. However, there is also interest in going beyond the inference problem and learning the graph structure itself from data
(Friedman and Koller, 2003). This requires that we deﬁne a space of possible structures as well as a measure that can be used to score each structure.
From a Bayesian viewpoint, we would ideally like to compute a posterior distribution over graph structures and to make predictions by averaging with respect
to this distribution. If we have a prior p(m) over graphs indexed by m, then theposterior distribution is given byp(m|D) ∝ p(m)p(D|m)(8.103)where D is the observed data set. The model evidence p(D|m) then provides thescore for each model. However, evaluation of the evidence involves marginalization
over the latent variables and presents a challenging computational problem for manymodels.
Exploring the space of structures can also be problematic. Because the numberof different graph structures grows exponentially with the number of nodes, it isoften necessary to resort to heuristics to ﬁnd good candidates.

Exercises

8.1 ( ) www By marginalizing out the variables in order, show that the representation(8.5) for the joint distribution of a directed graph is correctly normalized, provided
each of the conditional distributions is normalized.
8.2 ( ) www Show that the property of there being no directed cycles in a directed
graph follows from the statement that there exists an ordered numbering of the nodes
such that for each node there are no links going to a lower-numbered node.

Exercises
Table 8.2

The joint distribution over three binary variables.

a
0
0
0
0
1
1
1
1

b
0
0
1
1
0
0
1
1

c
0
1
0
1
0
1
0
1

419
p(a, b, c)
0.192
0.144
0.048
0.216
0.192
0.064
0.048
0.096

8.3 ( ) Consider three binary variables a, b, c ∈ {0, 1} having the joint distribution
given in Table 8.2. Show by direct evaluation that this distribution has the property
that a and b are marginally dependent, so that p(a, b) = p(a)p(b), but that they
become independent when conditioned on c, so that p(a, b|c) = p(a|c)p(b|c) for
both c = 0 and c = 1.
8.4 ( ) Evaluate the distributions p(a), p(b|c), and p(c|a) corresponding to the joint
distribution given in Table 8.2. Hence show by direct evaluation that p(a, b, c) =p(a)p(c|a)p(b|c). Draw the corresponding directed graph.
8.5 ( ) www Draw a directed probabilistic graphical model corresponding to therelevance vector machine described by (7.79) and (7.80).
8.6 ( ) For the model shown in Figure 8.13, we have seen that the number of parameters
required to specify the conditional distribution p(y|x1 , . . . , xM ), where xi ∈ {0, 1},
could be reduced from 2M to M + 1 by making use of the logistic sigmoid representation (8.10). An alternative representation (Pearl, 1988) is given byMp(y = 1|x1 , . . . , xM ) = 1 − (1 − µ0 ) (1 − µi )xi(8.104)i=1

where the parameters µi represent the probabilities p(xi = 1), and µ0 is an additionalparameters satisfying 0  µ0  1. The conditional distribution (8.104) is known as
the noisy-OR. Show that this can be interpreted as a ‘soft’ (probabilistic) form of thelogical OR function (i.e., the function that gives y = 1 whenever at least one of thexi = 1). Discuss the interpretation of µ0 .
8.7 ( ) Using the recursion relations (8.15) and (8.16), show that the mean and covariance of the joint distribution for the graph shown in Figure 8.14 are given by (8.17)
and (8.18), respectively.
8.8 ( ) www

Show that a ⊥⊥ b, c | d implies a ⊥⊥ b | d.

8.9 ( ) www Using the d-separation criterion, show that the conditional distribution
for a node x in a directed graph, conditioned on all of the nodes in the Markov
blanket, is independent of the remaining variables in the graph.

420

8. GRAPHICAL MODELS
Figure 8.54

Example of a graphical model used to explore the con- aditional independence properties of the head-to-head
path a–c–b when a descendant of c, namely the noded, is observed.

b

c

d

8.10 ( ) Consider the directed graph shown in Figure 8.54 in which none of the variables
is observed. Show that a ⊥⊥ b | ∅. Suppose we now observe the variable d. Show
that in general a ⊥⊥ b | d.
8.11 ( ) Consider the example of the car fuel system shown in Figure 8.21, and supposethat instead of observing the state of the fuel gauge G directly, the gauge is seen by
the driver D who reports to us the reading on the gauge. This report is either that thegauge shows full D = 1 or that it shows empty D = 0. Our driver is a bit unreliable,
as expressed through the following probabilitiesp(D = 1|G = 1) = 0.9
p(D = 0|G = 0) = 0.9.

(8.105)(8.106)

Suppose that the driver tells us that the fuel gauge shows empty, in other words
that we observe D = 0. Evaluate the probability that the tank is empty given only
this observation. Similarly, evaluate the corresponding probability given also theobservation that the battery is ﬂat, and note that this second probability is lower.
Discuss the intuition behind this result, and relate the result to Figure 8.54.
8.12 ( ) www Show that there are 2M (M −1)/2 distinct undirected graphs over a set of
M distinct random variables. Draw the 8 possibilities for the case of M = 3.
8.13 ( ) Consider the use of iterated conditional modes (ICM) to minimize the energyfunction given by (8.42). Write down an expression for the difference in the valuesof the energy associated with the two states of a particular variable xj , with all othervariables held ﬁxed, and show that it depends only on quantities that are local to xj
in the graph.
8.14 ( ) Consider a particular case of the energy function given by (8.42) in which the
coefﬁcients β = h = 0. Show that the most probable conﬁguration of the latentvariables is given by xi = yi for all i.
8.15 ( ) www Show that the joint distribution p(xn−1 , xn ) for two neighbouring
nodes in the graph shown in Figure 8.38 is given by an expression of the form (8.58).

Exercises

421

8.16 ( ) Consider the inference problem of evaluating p(xn |xN ) for the graph shownin Figure 8.38, for all nodes n ∈ {1, . . . , N − 1}. Show that the message passing
algorithm discussed in Section 8.4.1 can be used to solve this efﬁciently, and discuss
which messages are modiﬁed and in what way.
8.17 ( ) Consider a graph of the form shown in Figure 8.38 having N = 5 nodes, inwhich nodes x3 and x5 are observed. Use d-separation to show that x2 ⊥⊥ x5 | x3 .
Show that if the message passing algorithm of Section 8.4.1 is applied to the evaluation of p(x2 |x3 , x5 ), the result will be independent of the value of x5 .
8.18 ( ) www Show that a distribution represented by a directed tree can triviallybe written as an equivalent distribution over the corresponding undirected tree. Alsoshow that a distribution expressed as an undirected tree can, by suitable normalization of the clique potentials, be written as a directed tree. Calculate the number ofdistinct directed trees that can be constructed from a given undirected tree.
8.19 ( ) Apply the sum-product algorithm derived in Section 8.4.4 to the chain-ofnodes model discussed in Section 8.4.1 and show that the results (8.54), (8.55), and(8.57) are recovered as a special case.
8.20 ( ) www Consider the message passing protocol for the sum-product algorithm ona tree-structured factor graph in which messages are ﬁrst propagated from the leaves
to an arbitrarily chosen root node and then from the root node out to the leaves. Use
proof by induction to show that the messages can be passed in such an order thatat every step, each node that must send a message has received all of the incoming
messages necessary to construct its outgoing messages.
8.21 ( ) www Show that the marginal distributions p(xs ) over the sets of variables
xs associated with each of the factors fx (xs ) in a factor graph can be found by ﬁrst
running the sum-product message passing algorithm and then evaluating the required
marginals using (8.72).
8.22 ( ) Consider a tree-structured factor graph, in which a given subset of the variablenodes form a connected subgraph (i.e., any variable node of the subset is connected
to at least one of the other variable nodes via a single factor node). Show how thesum-product algorithm can be used to compute the marginal distribution over that
subset.
8.23 ( ) www In Section 8.4.4, we showed that the marginal distribution p(xi ) for avariable node xi in a factor graph is given by the product of the messages arriving at
this node from neighbouring factor nodes in the form (8.63). Show that the marginalp(xi ) can also be written as the product of the incoming message along any one of
the links with the outgoing message along the same link.
8.24 ( ) Show that the marginal distribution for the variables xs in a factor fs (xs ) ina tree-structured factor graph, after running the sum-product message passing algorithm, can be written as the product of the message arriving at the factor node along
all its links, times the local factor f (xs ), in the form (8.72).

422

8. GRAPHICAL MODELS
8.25 ( ) In (8.86), we veriﬁed that the sum-product algorithm run on the graph in
Figure 8.51 with node x3 designated as the root node gives the correct marginal forx2 . Show that the correct marginals are obtained also for x1 and x3 . Similarly, showthat the use of the result (8.72) after running the sum-product algorithm on this graphgives the correct joint distribution for x1 , x2 .
8.26 ( ) Consider a tree-structured factor graph over discrete variables, and suppose wewish to evaluate the joint distribution p(xa , xb ) associated with two variables xa andxb that do not belong to a common factor. Deﬁne a procedure for using the sumproduct algorithm to evaluate this joint distribution in which one of the variables is
successively clamped to each of its allowed values.
8.27 ( ) Consider two discrete variables x and y each having three possible states, forexample x, y ∈ {0, 1, 2}. Construct a joint distribution p(x, y) over these variables
x that maximizes the marginal p(x), along withhaving the property that the value 
the value y that maximizes the marginal p(y), together have probability zero under
the joint distribution, so that p(x, y ) = 0.
8.28 ( ) www The concept of a pending message in the sum-product algorithm fora factor graph was deﬁned in Section 8.4.7. Show that if the graph has one or morecycles, there will always be at least one pending message irrespective of how long
the algorithm runs.
8.29 ( ) www Show that if the sum-product algorithm is run on a factor graph with atree structure (no loops), then after a ﬁnite number of messages have been sent, there
will be no pending messages.

9
Mixture Models
and EM

Section 9.1

If we deﬁne a joint distribution over observed and latent variables, the corresponding distribution of the observed variables alone is obtained by marginalization. This
allows relatively complex marginal distributions over observed variables to be expressed in terms of more tractable joint distributions over the expanded space ofobserved and latent variables. The introduction of latent variables thereby allows
complicated distributions to be formed from simpler components. In this chapter,
we shall see that mixture distributions, such as the Gaussian mixture discussed in
Section 2.3.9, can be interpreted in terms of discrete latent variables. Continuous
latent variables will form the subject of Chapter 12.
As well as providing a framework for building more complex probability distributions, mixture models can also be used to cluster data. We therefore begin our
discussion of mixture distributions by considering the problem of ﬁnding clusters
in a set of data points, which we approach ﬁrst using a nonprobabilistic techniquecalled the K-means algorithm (Lloyd, 1982). Then we introduce the latent variable

423

424

Section 9.2

Section 9.3
Section 9.4

9. MIXTURE MODELS AND EM
view of mixture distributions in which the discrete latent variables can be interpreted
as deﬁning assignments of data points to speciﬁc components of the mixture. A general technique for ﬁnding maximum likelihood estimators in latent variable models
is the expectation-maximization (EM) algorithm. We ﬁrst of all use the Gaussian
mixture distribution to motivate the EM algorithm in a fairly informal way, and thenwe give a more careful treatment based on the latent variable viewpoint. We shallsee that the K-means algorithm corresponds to a particular nonprobabilistic limit of
EM applied to mixtures of Gaussians. Finally, we discuss EM in some generality.
Gaussian mixture models are widely used in data mining, pattern recognition,
machine learning, and statistical analysis. In many applications, their parameters are
determined by maximum likelihood, typically using the EM algorithm. However, as
we shall see there are some signiﬁcant limitations to the maximum likelihood approach, and in Chapter 10 we shall show that an elegant Bayesian treatment can begiven using the framework of variational inference. This requires little additional
computation compared with EM, and it resolves the principal difﬁculties of maximum likelihood while also allowing the number of components in the mixture to beinferred automatically from the data.

9.1. K-means Clustering
We begin by considering the problem of identifying groups, or clusters, of data points
in a multidimensional space. Suppose we have a data set {x1 , . . . , xN } consistingof N observations of a random D-dimensional Euclidean variable x. Our goal is topartition the data set into some number K of clusters, where we shall suppose forthe moment that the value of K is given. Intuitively, we might think of a cluster ascomprising a group of data points whose inter-point distances are small comparedwith the distances to points outside of the cluster. We can formalize this notion by
ﬁrst introducing a set of D-dimensional vectors µk , where k = 1, . . . , K, in whichµk is a prototype associated with the k th cluster. As we shall see shortly, we canthink of the µk as representing the centres of the clusters. Our goal is then to ﬁnd
an assignment of data points to clusters, as well as a set of vectors {µk }, such thatthe sum of the squares of the distances of each data point to its closest vector µk , isa minimum.
It is convenient at this point to deﬁne some notation to describe the assignmentof data points to clusters. For each data point xn , we introduce a corresponding setof binary indicator variables rnk ∈ {0, 1}, where k = 1, . . . , K describing which of
the K clusters the data point xn is assigned to, so that if data point xn is assigned tocluster k then rnk = 1, and rnj = 0 for j = k. This is known as the 1-of-K coding
scheme. We can then deﬁne an objective function, sometimes called a distortion
measure, given byN Krnk xn − µk 2(9.1)
J=n=1 k=1

which represents the sum of the squares of the distances of each data point to its

9.1. K-means Clustering

Section 9.4

425

assigned vector µk . Our goal is to ﬁnd values for the {rnk } and the {µk } so as tominimize J. We can do this through an iterative procedure in which each iteration
involves two successive steps corresponding to successive optimizations with respect
to the rnk and the µk . First we choose some initial values for the µk . Then in the ﬁrst
phase we minimize J with respect to the rnk , keeping the µk ﬁxed. In the second
phase we minimize J with respect to the µk , keeping rnk ﬁxed. This two-stage
optimization is then repeated until convergence. We shall see that these two stagesof updating rnk and updating µk correspond respectively to the E (expectation) andM (maximization) steps of the EM algorithm, and to emphasize this we shall use theterms E step and M step in the context of the K-means algorithm.
Consider ﬁrst the determination of the rnk . Because J in (9.1) is a linear function of rnk , this optimization can be performed easily to give a closed form solution.
The terms involving different n are independent and so we can optimize for eachn separately by choosing rnk to be 1 for whichever value of k gives the minimum
value of xn − µk 2 . In other words, we simply assign the nth data point to theclosest cluster centre. More formally, this can be expressed as
rnk =

1
0

if k = arg minj xn − µj 2otherwise.

(9.2)

Now consider the optimization of the µk with the rnk held ﬁxed. The objectivefunction J is a quadratic function of µk , and it can be minimized by setting its
derivative with respect to µk to zero giving
2N

rnk (xn − µk ) = 0

(9.3)

n=1

which we can easily solve for µk to given rnk xn.
µk = 
n rnk

Exercise 9.1

Appendix A(9.4)

The denominator in this expression is equal to the number of points assigned tocluster k, and so this result has a simple interpretation, namely set µk equal to themean of all of the data points xn assigned to cluster k. For this reason, the procedure
is known as the K-means algorithm.
The two phases of re-assigning data points to clusters and re-computing the cluster means are repeated in turn until there is no further change in the assignments (oruntil some maximum number of iterations is exceeded). Because each phase reduces
the value of the objective function J, convergence of the algorithm is assured. However, it may converge to a local rather than global minimum of J. The convergenceproperties of the K-means algorithm were studied by MacQueen (1967).
The K-means algorithm is illustrated using the Old Faithful data set in Figure 9.1. For the purposes of this example, we have made a linear re-scaling of thedata, known as standardizing, such that each of the variables has zero mean andunit standard deviation. For this example, we have chosen K = 2, and so in this

426

2

9. MIXTURE MODELS AND EM

(a)

2

(b)

2

0

0

0

−2

−2

−2

−2

2

0

2

(d)

−2

2

0

2

(e)

−2

2

0

0

0

−2

−2

−2

−2

2

0

2

(g)

−2

2

0

2

(h)

2

0

0

−2

−2

−2

0

2

−2

0

2

0

2

0

2

0

2

(f)

−2

0

−2

(c)

(i)

−2

Figure 9.1 Illustration of the K-means algorithm using the re-scaled Old Faithful data set. (a) Green points
denote the data set in a two-dimensional Euclidean space. The initial choices for centres µ1 and µ2 are shownby the red and blue crosses, respectively. (b) In the initial E step, each data point is assigned either to the redcluster or to the blue cluster, according to which cluster centre is nearer. This is equivalent to classifying thepoints according to which side of the perpendicular bisector of the two cluster centres, shown by the magentaline, they lie on. (c) In the subsequent M step, each cluster centre is re-computed to be the mean of the points
assigned to the corresponding cluster. (d)–(i) show successive E and M steps through to ﬁnal convergence of
the algorithm.

9.1. K-means Clustering
Figure 9.2

Plot of the cost function J given by(9.1) after each E step (blue points) 1000
and M step (red points) of the Kmeans algorithm for the example
shown in Figure 9.1. The algo- J
rithm has converged after the third
M step, and the ﬁnal EM cycle produces no changes in either the as- 500
signments or the prototype vectors.

0

Section 9.2.2

Section 2.3.5
Exercise 9.2

427

1

2

3

4

case, the assignment of each data point to the nearest cluster centre is equivalent to aclassiﬁcation of the data points according to which side they lie of the perpendicularbisector of the two cluster centres. A plot of the cost function J given by (9.1) for
the Old Faithful example is shown in Figure 9.2.
Note that we have deliberately chosen poor initial values for the cluster centres
so that the algorithm takes several steps before convergence. In practice, a better
initialization procedure would be to choose the cluster centres µk to be equal to arandom subset of K data points. It is also worth noting that the K-means algorithm
itself is often used to initialize the parameters in a Gaussian mixture model before
applying the EM algorithm.
A direct implementation of the K-means algorithm as discussed here can be
relatively slow, because in each E step it is necessary to compute the Euclidean distance between every prototype vector and every data point. Various schemes havebeen proposed for speeding up the K-means algorithm, some of which are based on
precomputing a data structure such as a tree such that nearby points are in the same
subtree (Ramasubramanian and Paliwal, 1990; Moore, 2000). Other approaches
make use of the triangle inequality for distances, thereby avoiding unnecessary distance calculations (Hodgson, 1998; Elkan, 2003).
So far, we have considered a batch version of K-means in which the whole data
set is used together to update the prototype vectors. We can also derive an on-line
stochastic algorithm (MacQueen, 1967) by applying the Robbins-Monro procedure
to the problem of ﬁnding the roots of the regression function given by the derivativesof J in (9.1) with respect to µk . This leads to a sequential update in which, for eachdata point xn in turn, we update the nearest prototype µk usingoldµnew= µoldkk + ηn (xn − µk )(9.5)

where ηn is the learning rate parameter, which is typically made to decrease monotonically as more data points are considered.
The K-means algorithm is based on the use of squared Euclidean distance as themeasure of dissimilarity between a data point and a prototype vector. Not only does
this limit the type of data variables that can be considered (it would be inappropriate
for cases where some or all of the variables represent categorical labels for instance),428

9. MIXTURE MODELS AND EM

Section 2.3.7

but it can also make the determination of the cluster means nonrobust to outliers. We
can generalize the K-means algorithm by introducing a more general dissimilarity
measure V(x, x ) between two vectors x and x and then minimizing the following
distortion measure
N 
K


J=
rnk V(xn , µk )(9.6)
n=1 k=1

which gives the K-medoids algorithm. The E step again involves, for given cluster
prototypes µk , assigning each data point to the cluster for which the dissimilarity to
the corresponding prototype is smallest. The computational cost of this is O(KN ),as is the case for the standard K-means algorithm. For a general choice of dissimilarity measure, the M step is potentially more complex than for K-means, and so it
is common to restrict each cluster prototype to be equal to one of the data vectors assigned to that cluster, as this allows the algorithm to be implemented for any choiceof dissimilarity measure V(·, ·) so long as it can be readily evaluated. Thus the M
step involves, for each cluster k, a discrete search over the Nk points assigned to thatcluster, which requires O(Nk2 ) evaluations of V(·, ·).
One notable feature of the K-means algorithm is that at each iteration, every
data point is assigned uniquely to one, and only one, of the clusters. Whereas somedata points will be much closer to a particular centre µk than to any other centre,
there may be other data points that lie roughly midway between cluster centres. In
the latter case, it is not clear that the hard assignment to the nearest cluster is themost appropriate. We shall see in the next section that by adopting a probabilistic
approach, we obtain ‘soft’ assignments of data points to clusters in a way that reﬂects
the level of uncertainty over the most appropriate assignment. This probabilistic
formulation brings with it numerous beneﬁts.

9.1.1 Image segmentation and compression
As an illustration of the application of the K-means algorithm, we consider
the related problems of image segmentation and image compression. The goal ofsegmentation is to partition an image into regions each of which has a reasonably
homogeneous visual appearance or which corresponds to objects or parts of objects(Forsyth and Ponce, 2003). Each pixel in an image is a point in a 3-dimensional spacecomprising the intensities of the red, blue, and green channels, and our segmentation
algorithm simply treats each pixel in the image as a separate data point. Note thatstrictly this space is not Euclidean because the channel intensities are bounded by
the interval [0, 1]. Nevertheless, we can apply the K-means algorithm without difﬁculty. We illustrate the result of running K-means to convergence, for any particularvalue of K, by re-drawing the image replacing each pixel vector with the {R, G, B}
intensity triplet given by the centre µk to which that pixel has been assigned. Resultsfor various values of K are shown in Figure 9.3. We see that for a given value of K,
the algorithm is representing the image using a palette of only K colours. It shouldbe emphasized that this use of K-means is not a particularly sophisticated approachto image segmentation, not least because it takes no account of the spatial proximity
of different pixels. The image segmentation problem is in general extremely difﬁcult

9.1. K-means Clustering

K =2

K =3

K = 10429

Original image

Figure 9.3 Two examples of the application of the K-means clustering algorithm to image segmentation showing the initial images together with their K-means segmentations obtained using various values of K. This
also illustrates of the use of vector quantization for data compression, in which smaller values of K give higher
compression at the expense of poorer image quality.

and remains the subject of active research and is introduced here simply to illustrate
the behaviour of the K-means algorithm.
We can also use the result of a clustering algorithm to perform data compression. It is important to distinguish between lossless data compression, in which
the goal is to be able to reconstruct the original data exactly from the compressed
representation, and lossy data compression, in which we accept some errors in thereconstruction in return for higher levels of compression than can be achieved in thelossless case. We can apply the K-means algorithm to the problem of lossy datacompression as follows. For each of the N data points, we store only the identity
k of the cluster to which it is assigned. We also store the values of the K cluster centres µk , which typically requires signiﬁcantly less data, provided we choose
K
N . Each data point is then approximated by its nearest centre µk . New data
points can similarly be compressed by ﬁrst ﬁnding the nearest µk and then storing
the label k instead of the original data vector. This framework is often called vector
quantization, and the vectors µk are called code-book vectors.

430

9. MIXTURE MODELS AND EM
The image segmentation problem discussed above also provides an illustrationof the use of clustering for data compression. Suppose the original image has N
pixels comprising {R, G, B} values each of which is stored with 8 bits of precision.
Then to transmit the whole image directly would cost 24N bits. Now suppose weﬁrst run K-means on the image data, and then instead of transmitting the original
pixel intensity vectors we transmit the identity of the nearest vector µk . Because
there are K such vectors, this requires log2 K bits per pixel. We must also transmit
the K code book vectors µk , which requires 24K bits, and so the total number ofbits required to transmit the image is 24K + N log2 K (rounding up to the nearest
integer). The original image shown in Figure 9.3 has 240 × 180 = 43, 200 pixels
and so requires 24 × 43, 200 = 1, 036, 800 bits to transmit directly. By comparison,
the compressed images require 43, 248 bits (K = 2), 86, 472 bits (K = 3), and173, 040 bits (K = 10), respectively, to transmit. These represent compression ratios
compared to the original image of 4.2%, 8.3%, and 16.7%, respectively. We see that
there is a trade-off between degree of compression and image quality. Note that ouraim in this example is to illustrate the K-means algorithm. If we had been aiming to
produce a good image compressor, then it would be more fruitful to consider small
blocks of adjacent pixels, for instance 5 × 5, and thereby exploit the correlations thatexist in natural images between nearby pixels.

9.2. Mixtures of Gaussians
In Section 2.3.9 we motivated the Gaussian mixture model as a simple linear superposition of Gaussian components, aimed at providing a richer class of density models than the single Gaussian. We now turn to a formulation of Gaussian mixtures interms of discrete latent variables. This will provide us with a deeper insight into thisimportant distribution, and will also serve to motivate the expectation-maximization
algorithm.
Recall from (2.188) that the Gaussian mixture distribution can be written as alinear superposition of Gaussians in the form
p(x) =K

πk N (x|µk , Σk ).

(9.7)

k=1

Let us introduce a K-dimensional binary random variable z having a 1-of-K representation in which a particular element zk is equal to 1 and allother elements areequal to 0. The values of zk therefore satisfy zk ∈ {0, 1} and k zk = 1, and we
see that there are K possible states for the vector z according to which element isnonzero. We shall deﬁne the joint distribution p(x, z) in terms of a marginal distribution p(z) and a conditional distribution p(x|z), corresponding to the graphicalmodel in Figure 9.4. The marginal distribution over z is speciﬁed in terms of themixing coefﬁcients πk , such thatp(zk = 1) = πk9.2. Mixtures of Gaussians
Figure 9.4

Graphical representation of a mixture model, in which
the joint distribution is expressed in the form p(x, z) =p(z)p(x|z).

431

z

x

where the parameters {πk } must satisfy
0  πk  1
together with

K


(9.8)

πk = 1(9.9)

k=1

in order to be valid probabilities. Because z uses a 1-of-K representation, we canalso write this distribution in the formp(z) =

K

πkzk .

(9.10)

k=1

Similarly, the conditional distribution of x given a particular value for z is a Gaussianp(x|zk = 1) = N (x|µk , Σk )which can also be written in the formp(x|z) =K

N (x|µk , Σk )zk .

(9.11)

k=1

Exercise 9.3

The joint distribution is given by p(z)p(x|z), and the marginal distribution of x isthen obtained by summing the joint distribution over all possible states of z to givep(x) =

z

p(z)p(x|z) =Kπk N (x|µk , Σk )(9.12)

k=1

where we have made use of (9.10) and (9.11). Thus the marginal distribution of x is
a Gaussian mixture of the form (9.7). If we have several observations x1 , . . . , xN ,then,
 because we have represented the marginal distribution in the form p(x) =z p(x, z), it follows that for every observed data point xn there is a corresponding
latent variable zn .
We have therefore found an equivalent formulation of the Gaussian mixture involving an explicit latent variable. It might seem that we have not gained muchby doing so. However, we are now able to work with the joint distribution p(x, z)432

9. MIXTURE MODELS AND EM
instead of the marginal distribution p(x), and this will lead to signiﬁcant simpliﬁcations, most notably through the introduction of the expectation-maximization (EM)algorithm.
Another quantity that will play an important role is the conditional probabilityof z given x. We shall use γ(zk ) to denote p(zk = 1|x), whose value can be foundusing Bayes’ theorem
γ(zk ) ≡ p(zk = 1|x) =p(zk = 1)p(x|zk = 1)Kp(zj = 1)p(x|zj = 1)

j =1

=

πk N (x|µk , Σk )K.

(9.13)

πj N (x|µj , Σj )j =1

Section 8.1.2

We shall view πk as the prior probability of zk = 1, and the quantity γ(zk ) as thecorresponding posterior probability once we have observed x. As we shall see later,γ(zk ) can also be viewed as the responsibility that component k takes for ‘explaining’ the observation x.
We can use the technique of ancestral sampling to generate random samples
distributed according to the Gaussian mixture model. To do this, we ﬁrst generate az, from the marginal distribution p(z) and then generatevalue for z, which we denote a value for x from the conditional distribution p(x|z). Techniques for sampling fromstandard distributions are discussed in Chapter 11. We can depict samples from thejoint distribution p(x, z) by plotting points at the corresponding values of x andthen colouring them according to the value of z, in other words according to which
Gaussian component was responsible for generating them, as shown in Figure 9.5(a).
Similarly samples from the marginal distribution p(x) are obtained by taking thesamples from the joint distribution and ignoring the values of z. These are illustrated
in Figure 9.5(b) by plotting the x values without any coloured labels.
We can also use this synthetic data set to illustrate the ‘responsibilities’ by evaluating, for every data point, the posterior probability for each component in the
mixture distribution from which this data set was generated. In particular, we canrepresent the value of the responsibilities γ(znk ) associated with data point xn byplotting the corresponding point using proportions of red, blue, and green ink givenby γ(znk ) for k = 1, 2, 3, respectively, as shown in Figure 9.5(c). So, for instance,a data point for which γ(zn1 ) = 1 will be coloured red, whereas one for which
γ(zn2 ) = γ(zn3 ) = 0.5 will be coloured with equal proportions of blue and green
ink and so will appear cyan. This should be compared with Figure 9.5(a) in which
the data points were labelled using the true identity of the component from which
they were generated.

9.2.1 Maximum likelihood
Suppose we have a data set of observations {x1 , . . . , xN }, and we wish to model
this data using a mixture of Gaussians. We can represent this data set as an N × D433

9.2. Mixtures of Gaussians

1

1

(a)

1

(b)

0.5

0.5

0.5

0

0

0

0

0.5

1

0

0.5

(c)

1

0

0.5

1

Figure 9.5 Example of 500 points drawn from the mixture of 3 Gaussians shown in Figure 2.23. (a) Samples
from the joint distribution p(z)p(x|z) in which the three states of z, corresponding to the three components of the
mixture, are depicted in red, green, and blue, and (b) the corresponding samples from the marginal distribution
p(x), which is obtained by simply ignoring the values of z and just plotting the x values. The data set in (a) is
said to be complete, whereas that in (b) is incomplete. (c) The same samples in which the colours represent thevalue of the responsibilities γ(znk ) associated with data point xn , obtained by plotting the corresponding point
using proportions of red, blue, and green ink given by γ(znk ) for k = 1, 2, 3, respectively

matrix X in which the nth row is given by xT
n . Similarly, the corresponding latentvariables will be denoted by an N × K matrix Z with rows zT
n . If we assume thatthe data points are drawn independently from the distribution, then we can express
the Gaussian mixture model for this i.i.d. data set using the graphical representation
shown in Figure 9.6. From (9.7) the log of the likelihood function is given by
K

N


ln
πk N (xn |µk , Σk ) .
(9.14)
ln p(X|π, µ, Σ) =
n=1

k=1

Before discussing how to maximize this function, it is worth emphasizing that
there is a signiﬁcant problem associated with the maximum likelihood framework
applied to Gaussian mixture models, due to the presence of singularities. For simplicity, consider a Gaussian mixture whose components have covariance matrices
given by Σk = σk2 I, where I is the unit matrix, although the conclusions will holdfor general covariance matrices. Suppose that one of the components of the mixture
model, let us say the j th component, has its mean µj exactly equal to one of the data
Figure 9.6

Graphical representation of a Gaussian mixture model
for a set of N i.i.d. data points {xn }, with correspondingπlatent points {zn }, where n = 1, . . . , N .

zn

xn
µ

Σ
N

434

9. MIXTURE MODELS AND EM
Figure 9.7

Illustration of how singularities in thelikelihood function arise with mixturesof Gaussians. This should be com- p(x)pared with the case of a single Gaussian shown in Figure 1.14 for which no
singularities arise.

x

points so that µj = xn for some value of n. This data point will then contribute aterm in the likelihood function of the form
1
1.
(9.15)N (xn |xn , σj2 I) =(2π)1/2 σj

Section 10.1

If we consider the limit σj → 0, then we see that this term goes to inﬁnity andso the log likelihood function will also go to inﬁnity. Thus the maximization of
the log likelihood function is not a well posed problem because such singularities
will always be present and will occur whenever one of the Gaussian components‘collapses’ onto a speciﬁc data point. Recall that this problem did not arise in thecase of a single Gaussian distribution. To understand the difference, note that if asingle Gaussian collapses onto a data point it will contribute multiplicative factors
to the likelihood function arising from the other data points and these factors will goto zero exponentially fast, giving an overall likelihood that goes to zero rather thaninﬁnity. However, once we have (at least) two components in the mixture, one of
the components can have a ﬁnite variance and therefore assign ﬁnite probability toall of the data points while the other component can shrink onto one speciﬁc data
point and thereby contribute an ever increasing additive value to the log likelihood.
This is illustrated in Figure 9.7. These singularities provide another example of thesevere over-ﬁtting that can occur in a maximum likelihood approach. We shall seethat this difﬁculty does not occur if we adopt a Bayesian approach. For the moment,
however, we simply note that in applying maximum likelihood to Gaussian mixture
models we must take steps to avoid ﬁnding such pathological solutions and instead
seek local maxima of the likelihood function that are well behaved. We can hope to
avoid the singularities by using suitable heuristics, for instance by detecting when aGaussian component is collapsing and resetting its mean to a randomly chosen value
while also resetting its covariance to some large value, and then continuing with the
optimization.
A further issue in ﬁnding maximum likelihood solutions arises from the factthat for any given maximum likelihood solution, a K-component mixture will havea total of K! equivalent solutions corresponding to the K! ways of assigning K
sets of parameters to K components. In other words, for any given (nondegenerate)
point in the space of parameter values there will be a further K! − 1 additional points
all of which give rise to exactly the same distribution. This problem is known as

9.2. Mixtures of Gaussians

435

identiﬁability (Casella and Berger, 2002) and is an important issue when we wish tointerpret the parameter values discovered by a model. Identiﬁability will also arise
when we discuss models having continuous latent variables in Chapter 12. However,for the purposes of ﬁnding a good density model, it is irrelevant because any of theequivalent solutions is as good as any other.
Maximizing the log likelihood function (9.14) for a Gaussian mixture modelturns out to be a more complex problem than for the case of a single Gaussian. The
difﬁculty arises from the presence of the summation over k that appears inside thelogarithm in (9.14), so that the logarithm function no longer acts directly on theGaussian. If we set the derivatives of the log likelihood to zero, we will no longer
obtain a closed form solution, as we shall see shortly.
One approach is to apply gradient-based optimization techniques (Fletcher, 1987;
Nocedal and Wright, 1999; Bishop and Nabney, 2008). Although gradient-based
techniques are feasible, and indeed will play an important role when we discuss
mixture density networks in Chapter 5, we now consider an alternative approach
known as the EM algorithm which has broad applicability and which will lay thefoundations for a discussion of variational inference techniques in Chapter 10.

9.2.2 EM for Gaussian mixtures

Section 10.1

An elegant and powerful method for ﬁnding maximum likelihood solutions formodels with latent variables is called the expectation-maximization algorithm, or EM
algorithm (Dempster et al., 1977; McLachlan and Krishnan, 1997). Later we shallgive a general treatment of EM, and we shall also show how EM can be generalized
to obtain the variational inference framework. Initially, we shall motivate the EM
algorithm by giving a relatively informal treatment in the context of the Gaussian
mixture model. We emphasize, however, that EM has broad applicability, and indeed
it will be encountered in the context of a variety of different models in this book.
Let us begin by writing down the conditions that must be satisﬁed at a maximumof the likelihood function. Setting the derivatives of ln p(X|π, µ, Σ) in (9.14) withrespect to the means µk of the Gaussian components to zero, we obtainN
π N (xn |µk , Σk )kΣk (xn − µk )0=−
πj N (xn |µj , Σj )n=1 ( j)*+γ(znk )

(9.16)

where we have made use of the form (2.43) for the Gaussian distribution. Note thatthe posterior probabilities, or responsibilities, given by (9.13) appear naturally on
1
the right-hand side. Multiplying by Σ−k (which we assume to be nonsingular) andrearranging we obtainN
1 γ(znk )xn
(9.17)µk =Nkn=1

where we have deﬁned
Nk =

N

n=1

γ(znk ).

(9.18)

436

9. MIXTURE MODELS AND EM

Section 2.3.4

We can interpret Nk as the effective number of points assigned to cluster k. Notecarefully the form of this solution. We see that the mean µk for the k th Gaussiancomponent is obtained by taking a weighted mean of all of the points in the data set,in which the weighting factor for data point xn is given by the posterior probabilityγ(znk ) that component k was responsible for generating xn .
If we set the derivative of ln p(X|π, µ, Σ) with respect to Σk to zero, and follow
a similar line of reasoning, making use of the result for the maximum likelihood
solution for the covariance matrix of a single Gaussian, we obtain1 Σk =γ(znk )(xn − µk )(xn − µk )TNkN(9.19)

n=1

Appendix E

which has the same form as the corresponding result for a single Gaussian ﬁtted tothe data set, but again with each data point weighted by the corresponding posterior probability and with the denominator given by the effective number of points
associated with the corresponding component.
Finally, we maximize ln p(X|π, µ, Σ) with respect to the mixing coefﬁcientsπk . Here we must take account of the constraint (9.9), which requires the mixing
coefﬁcients to sum to one. This can be achieved using a Lagrange multiplier andmaximizing the following quantityK
πk − 1(9.20)
ln p(X|π, µ, Σ) + λ
k=1

which gives
0=

N

n=1

N (xn |µk , Σk )+λj πj N (xn |µj , Σj )



(9.21)

where again we see the appearance of the responsibilities. If we now multiply bothsides by πk and sum over k making use of the constraint (9.9), we ﬁnd λ = −N .
Using this to eliminate λ and rearranging we obtain
πk =

Nk
N

(9.22)

so that the mixing coefﬁcient for the k th component is given by the average responsibility which that component takes for explaining the data points.
It is worth emphasizing that the results (9.17), (9.19), and (9.22) do not constitute a closed-form solution for the parameters of the mixture model because theresponsibilities γ(znk ) depend on those parameters in a complex way through (9.13).
However, these results do suggest a simple iterative scheme for ﬁnding a solution to
the maximum likelihood problem, which as we shall see turns out to be an instanceof the EM algorithm for the particular case of the Gaussian mixture model. We
ﬁrst choose some initial values for the means, covariances, and mixing coefﬁcients.
Then we alternate between the following two updates that we shall call the E step

437

9.2. Mixtures of Gaussians

2

2

2
L=1

0

0

0

−2

−2

−2

−2

0

(a)

2

2

−2

0

(b)

2

2
L=2

−2

0

−2

−2

−2

(d)

2

2

0

(f)

2

L = 20

0

0

(c)

2
L=5

0

−2

0

−2

0

(e)

2

−2

Figure 9.8 Illustration of the EM algorithm using the Old Faithful set as used for the illustration of the K-means
algorithm in Figure 9.1. See the text for details.

Section 9.4

and the M step, for reasons that will become apparent shortly. In the expectation
step, or E step, we use the current values for the parameters to evaluate the posterior
probabilities, or responsibilities, given by (9.13). We then use these probabilities in
the maximization step, or M step, to re-estimate the means, covariances, and mixing coefﬁcients using the results (9.17), (9.19), and (9.22). Note that in so doing
we ﬁrst evaluate the new means using (9.17) and then use these new values to ﬁnd
the covariances using (9.19), in keeping with the corresponding result for a single
Gaussian distribution. We shall show that each update to the parameters resulting
from an E step followed by an M step is guaranteed to increase the log likelihood
function. In practice, the algorithm is deemed to have converged when the change
in the log likelihood function, or alternatively in the parameters, falls below somethreshold. We illustrate the EM algorithm for a mixture of two Gaussians applied to
the rescaled Old Faithful data set in Figure 9.8. Here a mixture of two Gaussians
is used, with centres initialized using the same values as for the K-means algorithm
in Figure 9.1, and with precision matrices initialized to be proportional to the unit
matrix. Plot (a) shows the data points in green, together with the initial conﬁguration of the mixture model in which the one standard-deviation contours for the two438

9. MIXTURE MODELS AND EM
Gaussian components are shown as blue and red circles. Plot (b) shows the resultof the initial E step, in which each data point is depicted using a proportion of blue
ink equal to the posterior probability of having been generated from the blue component, and a corresponding proportion of red ink given by the posterior probabilityof having been generated by the red component. Thus, points that have a signiﬁcantprobability for belonging to either cluster appear purple. The situation after the ﬁrst
M step is shown in plot (c), in which the mean of the blue Gaussian has moved tothe mean of the data set, weighted by the probabilities of each data point belonging
to the blue cluster, in other words it has moved to the centre of mass of the blue ink.
Similarly, the covariance of the blue Gaussian is set equal to the covariance of theblue ink. Analogous results hold for the red component. Plots (d), (e), and (f) show
the results after 2, 5, and 20 complete cycles of EM, respectively. In plot (f) thealgorithm is close to convergence.
Note that the EM algorithm takes many more iterations to reach (approximate)convergence compared with the K-means algorithm, and that each cycle requires
signiﬁcantly more computation. It is therefore common to run the K-means algorithm in order to ﬁnd a suitable initialization for a Gaussian mixture model that is
subsequently adapted using EM. The covariance matrices can conveniently be initialized to the sample covariances of the clusters found by the K-means algorithm,
and the mixing coefﬁcients can be set to the fractions of data points assigned to therespective clusters. As with gradient-based approaches for maximizing the log likelihood, techniques must be employed to avoid singularities of the likelihood functionin which a Gaussian component collapses onto a particular data point. It should beemphasized that there will generally be multiple local maxima of the log likelihood
function, and that EM is not guaranteed to ﬁnd the largest of these maxima. Because
the EM algorithm for Gaussian mixtures plays such an important role, we summarize
it below.
EM for Gaussian Mixtures
Given a Gaussian mixture model, the goal is to maximize the likelihood function
with respect to the parameters (comprising the means and covariances of thecomponents and the mixing coefﬁcients).
1. Initialize the means µk , covariances Σk and mixing coefﬁcients πk , andevaluate the initial value of the log likelihood.
2. E step. Evaluate the responsibilities using the current parameter values
γ(znk ) =πk N (xn |µk , Σk ).
K
πj N (xn |µj , Σj )j =1

(9.23)

439

9.3. An Alternative View of EM
3. M step. Re-estimate the parameters using the current responsibilities
1 γ(znk )xn
Nk

(9.24)1 
new Tγ(znk ) (xn − µnewk ) (xn − µk )Nk(9.25)

Nk
N(9.26)

N

µnew
k

=

n=1
N

Σnew
k

=

n=1

πknew

=

where
Nk =

N


γ(znk ).

(9.27)

n=1

4. Evaluate the log likelihoodln p(X|µ, Σ, π) =

N


ln

K


n=1


πk N (xn |µk , Σk )(9.28)

k=1

and check for convergence of either the parameters or the log likelihood. If
the convergence criterion is not satisﬁed return to step 2.

9.3. An Alternative View of EM
In this section, we present a complementary view of the EM algorithm that recognizes the key role played by latent variables. We discuss this approach ﬁrst of all
in an abstract setting, and then for illustration we consider once again the case ofGaussian mixtures.
The goal of the EM algorithm is to ﬁnd maximum likelihood solutions for models having latent variables. We denote the set of all observed data by X, in which thenth row represents xTn , and similarly we denote the set of all latent variables by Z,
with a corresponding row zTn . The set of all model parameters is denoted by θ, andso the log likelihood function is given by
p(X, Z|θ) .
(9.29)
ln p(X|θ) = lnZ

Note that our discussion will apply equally well to continuous latent variables simply
by replacing the sum over Z with an integral.
A key observation is that the summation over the latent variables appears inside
the logarithm. Even if the joint distribution p(X, Z|θ) belongs to the exponential

440

9. MIXTURE MODELS AND EM
family, the marginal distribution p(X|θ) typically does not as a result of this summation. The presence of the sum prevents the logarithm from acting directly on the
joint distribution, resulting in complicated expressions for the maximum likelihood
solution.
Now suppose that, for each observation in X, we were told the corresponding
value of the latent variable Z. We shall call {X, Z} the complete data set, and weshall refer to the actual observed data X as incomplete, as illustrated in Figure 9.5.
The likelihood function for the complete data set simply takes the form ln p(X, Z|θ),and we shall suppose that maximization of this complete-data log likelihood function
is straightforward.
In practice, however, we are not given the complete data set {X, Z}, but only
the incomplete data X. Our state of knowledge of the values of the latent variables
in Z is given only by the posterior distribution p(Z|X, θ). Because we cannot use
the complete-data log likelihood, we consider instead its expected value under theposterior distribution of the latent variable, which corresponds (as we shall see) to theE step of the EM algorithm. In the subsequent M step, we maximize this expectation.
If the current estimate for the parameters is denoted θ old , then a pair of successive
E and M steps gives rise to a revised estimate θ new . The algorithm is initialized bychoosing some starting value for the parameters θ 0 . The use of the expectation may
seem somewhat arbitrary. However, we shall see the motivation for this choice when
we give a deeper treatment of EM in Section 9.4.
In the E step, we use the current parameter values θ old to ﬁnd the posterior
distribution of the latent variables given by p(Z|X, θ old ). We then use this posterior
distribution to ﬁnd the expectation of the complete-data log likelihood evaluated forsome general parameter value θ. This expectation, denoted Q(θ, θ old ), is given byp(Z|X, θ old ) ln p(X, Z|θ).
(9.30)
Q(θ, θ old ) =Z

In the M step, we determine the revised parameter estimate θ new by maximizing thisfunction(9.31)
θ new = arg max Q(θ, θ old ).
θNote that in the deﬁnition of Q(θ, θ ), the logarithm acts directly on the joint
distribution p(X, Z|θ), and so the corresponding M-step maximization will, by supposition, be tractable.
The general EM algorithm is summarized below. It has the property, as we shallshow later, that each cycle of EM will increase the incomplete-data log likelihood(unless it is already at a local maximum).
old

Section 9.4

The General EM Algorithm
Given a joint distribution p(X, Z|θ) over observed variables X and latent variables Z, governed by parameters θ, the goal is to maximize the likelihood function p(X|θ) with respect to θ.
1. Choose an initial setting for the parameters θ old .

9.3. An Alternative View of EM

441

2. E step Evaluate p(Z|X, θ old ).
3. M step Evaluate θ new given byθ new = arg max Q(θ, θ old )
θ

where

Q(θ, θ old ) =p(Z|X, θ old ) ln p(X, Z|θ).

(9.32)(9.33)

Z

4. Check for convergence of either the log likelihood or the parameter values.
If the convergence criterion is not satisﬁed, then let
θ old ← θ new(9.34)

and return to step 2.

Exercise 9.4

The EM algorithm can also be used to ﬁnd MAP (maximum posterior) solutions
for models in which a prior p(θ) is deﬁned over the parameters. In this case the E
step remains the same as in the maximum likelihood case, whereas in the M step thequantity to be maximized is given by Q(θ, θ old ) + ln p(θ). Suitable choices for theprior will remove the singularities of the kind illustrated in Figure 9.7.
Here we have considered the use of the EM algorithm to maximize a likelihood
function when there are discrete latent variables. However, it can also be applied
when the unobserved variables correspond to missing values in the data set. The
distribution of the observed values is obtained by taking the joint distribution of all
the variables and then marginalizing over the missing ones. EM can then be used
to maximize the corresponding likelihood function. We shall show an example of
the application of this technique in the context of principal component analysis in
Figure 12.11. This will be a valid procedure if the data values are missing at random,
meaning that the mechanism causing values to be missing does not depend on theunobserved values. In many situations this will not be the case, for instance if asensor fails to return a value whenever the quantity it is measuring exceeds somethreshold.

9.3.1 Gaussian mixtures revisited
We now consider the application of this latent variable view of EM to the speciﬁc case of a Gaussian mixture model. Recall that our goal is to maximize the log
likelihood function (9.14), which is computed using the observed data set X, and we
saw that this was more difﬁcult than for the case of a single Gaussian distribution
due to the presence of the summation over k that occurs inside the logarithm. Suppose then that in addition to the observed data set X, we were also given the valuesof the corresponding discrete variables Z. Recall that Figure 9.5(a) shows a ‘complete’ data set (i.e., one that includes labels showing which component generated
each data point) while Figure 9.5(b) shows the corresponding ‘incomplete’ data set.
The graphical model for the complete data is shown in Figure 9.9.

442

9. MIXTURE MODELS AND EM
Figure 9.9This shows the same graph as in Figure 9.6 except that
we now suppose that the discrete variables zn are obπ
served, as well as the data variables xn .

zn

xn
µ

Σ
N

Now consider the problem of maximizing the likelihood for the complete data
set {X, Z}. From (9.10) and (9.11), this likelihood function takes the formp(X, Z|µ, Σ, π) =N K
n=1 k=1

πkznk N (xn |µk , Σk )znk

(9.35)

where znk denotes the k th component of zn . Taking the logarithm, we obtainln p(X, Z|µ, Σ, π) =

N 
K


znk {ln πk + ln N (xn |µk , Σk )} .

(9.36)

n=1 k=1

Comparison with the log likelihood function (9.14) for the incomplete data showsthat the summation over k and the logarithm have been interchanged. The logarithm now acts directly on the Gaussian distribution, which itself is a member ofthe exponential family. Not surprisingly, this leads to a much simpler solution to
the maximum likelihood problem, as we now show. Consider ﬁrst the maximization
with respect to the means and covariances. Because zn is a K-dimensional vector with all elements equal to 0 except for a single element having the value 1, thecomplete-data log likelihood function is simply a sum of K independent contributions, one for each mixture component. Thus the maximization with respect to amean or a covariance is exactly as for a single Gaussian, except that it involves only
the subset of data points that are ‘assigned’ to that component. For the maximization
with respect to the mixing coefﬁcients, we note that these are coupled for different
values of k by virtue of the summation constraint (9.9). Again, this can be enforced
using a Lagrange multiplier as before, and leads to the result
1 
znk
πk =
N
N

(9.37)

n=1

so that the mixing coefﬁcients are equal to the fractions of data points assigned tothe corresponding components.
Thus we see that the complete-data log likelihood function can be maximized
trivially in closed form. In practice, however, we do not have values for the latentvariables so, as discussed earlier, we consider the expectation, with respect to theposterior distribution of the latent variables, of the complete-data log likelihood.

9.3. An Alternative View of EM

443

Using (9.10) and (9.11) together with Bayes’ theorem, we see that this posterior
distribution takes the form
N K
p(Z|X, µ, Σ, π) ∝z

[πk N (xn |µk , Σk )] nk .

(9.38)

n=1 k=1

Exercise 9.5
Section 8.2

and hence factorizes over n so that under the posterior distribution the {zn } areindependent. This is easily veriﬁed by inspection of the directed graph in Figure 9.6
and making use of the d-separation criterion. The expected value of the indicatorvariable znk under this posterior distribution is then given byzznk [πk N (xn |µk , Σk )] nk
E[znk ] =

znk


znj
πj N (xn |µj , Σj )znj=

πk N (xn |µk , Σk )K= γ(znk )(9.39)

πj N (xn |µj , Σj )j =1

which is just the responsibility of component k for data point xn . The expected valueof the complete-data log likelihood function is therefore given by
EZ [ln p(X, Z|µ, Σ, π)] =

N 
K


γ(znk ) {ln πk + ln N (xn |µk , Σk )} .

(9.40)

n=1 k=1

Exercise 9.8

We can now proceed as follows. First we choose some initial values for the parameters µold , Σold and π old , and use these to evaluate the responsibilities (the E step).
We then keep the responsibilities ﬁxed and maximize (9.40) with respect to µk , Σkand πk (the M step). This leads to closed form solutions for µnew , Σnew and π new
given by (9.17), (9.19), and (9.22) as before. This is precisely the EM algorithm for
Gaussian mixtures as derived earlier. We shall gain more insight into the role of the
expected complete-data log likelihood function when we give a proof of convergenceof the EM algorithm in Section 9.4.

9.3.2 Relation to K-means
Comparison of the K-means algorithm with the EM algorithm for Gaussian
mixtures shows that there is a close similarity. Whereas the K-means algorithm
performs a hard assignment of data points to clusters, in which each data point isassociated uniquely with one cluster, the EM algorithm makes a soft assignment
based on the posterior probabilities. In fact, we can derive the K-means algorithm
as a particular limit of EM for Gaussian mixtures as follows.
Consider a Gaussian mixture model in which the covariance matrices of the
mixture components are given by I, where  is a variance parameter that is shared

444

9. MIXTURE MODELS AND EM
by all of the components, and I is the identity matrix, so thatp(x|µk , Σk ) =1
1
exp − x − µk 2 .
1
/
2
2
(2π)

(9.41)

We now consider the EM algorithm for a mixture of K Gaussians of this form inwhich we treat  as a ﬁxed constant, instead of a parameter to be re-estimated. From
(9.13) the posterior probabilities, or responsibilities, for a particular data point xn ,are given by
πk exp {−xn − µk 2 /2}
.
(9.42)γ(znk ) = 2j πj exp −xn − µj  /2

Exercise 9.11

If we consider the limit  → 0, we see that in the denominator the term for whichxn − µj 2 is smallest will go to zero most slowly, and hence the responsibilities
γ(znk ) for the data point xn all go to zero except for term j, for which the responsibility γ(znj ) will go to unity. Note that this holds independently of the values of theπk so long as none of the πk is zero. Thus, in this limit, we obtain a hard assignmentof data points to clusters, just as in the K-means algorithm, so that γ(znk ) → rnk
where rnk is deﬁned by (9.2). Each data point is thereby assigned to the cluster
having the closest mean.
The EM re-estimation equation for the µk , given by (9.17), then reduces to theK-means result (9.4). Note that the re-estimation formula for the mixing coefﬁcients(9.22) simply re-sets the value of πk to be equal to the fraction of data points assigned
to cluster k, although these parameters no longer play an active role in the algorithm.
Finally, in the limit  → 0 the expected complete-data log likelihood, given by(9.40), becomes1 
rnk xn − µk 2 + const.
2N

EZ [ln p(X, Z|µ, Σ, π)] → −K(9.43)

n=1 k=1

Thus we see that in this limit, maximizing the expected complete-data log likelihood
is equivalent to minimizing the distortion measure J for the K-means algorithm
given by (9.1).
Note that the K-means algorithm does not estimate the covariances of the clusters but only the cluster means. A hard-assignment version of the Gaussian mixture
model with general covariance matrices, known as the elliptical K-means algorithm,
has been considered by Sung and Poggio (1994).

9.3.3 Mixtures of Bernoulli distributions

Section 13.2

So far in this chapter, we have focussed on distributions over continuous variables described by mixtures of Gaussians. As a further example of mixture modelling, and to illustrate the EM algorithm in a different context, we now discuss mixtures of discrete binary variables described by Bernoulli distributions. This modelis also known as latent class analysis (Lazarsfeld and Henry, 1968; McLachlan andPeel, 2000). As well as being of practical importance in its own right, our discussion of Bernoulli mixtures will also lay the foundation for a consideration of hidden
Markov models over discrete variables.

9.3. An Alternative View of EM

445

Consider a set of D binary variables xi , where i = 1, . . . , D, each of which isgoverned by a Bernoulli distribution with parameter µi , so thatp(x|µ) =Dµxi i (1 − µi )(1−xi )(9.44)

i=1

where x = (x1 , . . . , xD )T and µ = (µ1 , . . . , µD )T . We see that the individual
variables xi are independent, given µ. The mean and covariance of this distribution
are easily seen to be
E[x] = µcov[x] = diag{µi (1 − µi )}.

(9.45)(9.46)

Now let us consider a ﬁnite mixture of these distributions given byp(x|µ, π) =Kπk p(x|µk )(9.47)

k=1

where µ = {µ1 , . . . , µK }, π = {π1 , . . . , πK }, andp(x|µk ) =Dµxkii (1 − µki )(1−xi ) .

(9.48)

i=1

Exercise 9.12

The mean and covariance of this mixture distribution are given byE[x] =

K

πk µk(9.49)Tπk Σk + µk µT
k − E[x]E[x](9.50)

k=1

cov[x] =K
k=1

where Σk = diag {µki (1 − µki )}. Because the covariance matrix cov[x] is nolonger diagonal, the mixture distribution can capture correlations between the variables, unlike a single Bernoulli distribution.
If we are given a data set X = {x1 , . . . , xN } then the log likelihood function
for this model is given by
K

N


ln
πk p(xn |µk ) .
(9.51)
ln p(X|µ, π) =
n=1

k=1

Again we see the appearance of the summation inside the logarithm, so that the
maximum likelihood solution no longer has closed form.
We now derive the EM algorithm for maximizing the likelihood function forthe mixture of Bernoulli distributions. To do this, we ﬁrst introduce an explicit latent

446

9. MIXTURE MODELS AND EM
variable z associated with each instance of x. As in the case of the Gaussian mixture,z = (z1 , . . . , zK )T is a binary K-dimensional variable having a single component
equal to 1, with all other components equal to 0. We can then write the conditional
distribution of x, given the latent variable, asp(x|z, µ) =Kp(x|µk )zk(9.52)

k=1

while the prior distribution for the latent variables is the same as for the mixture ofGaussians model, so thatK
πkzk .
(9.53)
p(z|π) =k=1

Exercise 9.14

If we form the product of p(x|z, µ) and p(z|π) and then marginalize over z, then werecover (9.47).
In order to derive the EM algorithm, we ﬁrst write down the complete-data log
likelihood function, which is given by

N 
K

znk ln πkln p(X, Z|µ, π) =n=1 k=1+

D[xni ln µki + (1 − xni ) ln(1 − µki )](9.54)

i=1

where X = {xn } and Z = {zn }. Next we take the expectation of the complete-data
log likelihood with respect to the posterior distribution of the latent variables to give

N 
K

γ(znk ) ln πk
EZ [ln p(X, Z|µ, π)] =n=1 k=1+

D[xni ln µki + (1 − xni ) ln(1 − µki )](9.55)

i=1

where γ(znk ) = E[znk ] is the posterior probability, or responsibility, of component
k given data point xn . In the E step, these responsibilities are evaluated using Bayes’
theorem, which takes the formzznk [πk p(xn |µk )] nk
γ(znk ) = E[znk ] =

znk


znj
πj p(xn |µj )znj=

πk p(xn |µk ).
Kπj p(xn |µj )j =1(9.56)

9.3. An Alternative View of EM447

If we consider the sum over n in (9.55), we see that the responsibilities enter
only through two terms, which can be written as
Nk

=

N


γ(znk )

(9.57)1 γ(znk )xn
Nk(9.58)

n=1
N

xk

=

n=1

Exercise 9.15

where Nk is the effective number of data points associated with component k. In the
M step, we maximize the expected complete-data log likelihood with respect to theparameters µk and π. If we set the derivative of (9.55) with respect to µk equal tozero and rearrange the terms, we obtainµk = xk .

Exercise 9.16

We see that this sets the mean of component k equal to a weighted mean of thedata, with weighting coefﬁcients given by the responsibilities that component k takesfor data points. For the maximization with respect to πk , we need to introduce a
Lagrange multiplier to enforce the constraint k πk = 1. Following analogous
steps to those used for the mixture of Gaussians, we then obtainπk =Exercise 9.17

Section 9.4(9.59)

Nk
N(9.60)

which represents the intuitively reasonable result that the mixing coefﬁcient for component k is given by the effective fraction of points in the data set explained by thatcomponent.
Note that in contrast to the mixture of Gaussians, there are no singularities inwhich the likelihood function goes to inﬁnity. This can be seen by noting that thelikelihood function is bounded above because 0  p(xn |µk )  1. There exist
singularities at which the likelihood function goes to zero, but these will not befound by EM provided it is not initialized to a pathological starting point, because
the EM algorithm always increases the value of the likelihood function, until a local
maximum is found. We illustrate the Bernoulli mixture model in Figure 9.10 byusing it to model handwritten digits. Here the digit images have been turned into
binary vectors by setting all elements whose values exceed 0.5 to 1 and setting theremaining elements to 0. We now ﬁt a data set of N = 600 such digits, comprising
the digits ‘2’, ‘3’, and ‘4’, with a mixture of K = 3 Bernoulli distributions byrunning 10 iterations of the EM algorithm. The mixing coefﬁcients were initialized
uniformly into πk = 1/K, and the parameters µkj were set to random values chosenthe range (0.25, 0.75) and then normalized to satisfy the constraint that j µkj = 1.
We see that a mixture of 3 Bernoulli distributions is able to ﬁnd the three clusters in
the data set corresponding to the different digits.
The conjugate prior for the parameters of a Bernoulli distribution is given by
the beta distribution, and we have seen that a beta prior is equivalent to introducing

448

9. MIXTURE MODELS AND EM

Figure 9.10 Illustration of the Bernoulli mixture model in which the top row shows examples from the digits data
set after converting the pixel values from grey scale to binary using a threshold of 0.5. On the bottom row the ﬁrst
three images show the parameters µki for each of the three components in the mixture model. As a comparison,
we also ﬁt the same data set using a single multivariate Bernoulli distribution, again using maximum likelihood.
This amounts to simply averaging the counts in each pixel and is shown by the right-most image on the bottomrow.

Section 2.1.1
Exercise 9.18
Exercise 9.19

additional effective observations of x. We can similarly introduce priors into the
Bernoulli mixture model, and use EM to maximize the posterior probability distributions.
It is straightforward to extend the analysis of Bernoulli mixtures to the case ofmultinomial binary variables having M > 2 states by making use of the discrete distribution (2.26). Again, we can introduce Dirichlet priors over the model parameters
if desired.

9.3.4 EM for Bayesian linear regression
As a third example of the application of EM, we return to the evidence approximation for Bayesian linear regression. In Section 3.5.2, we obtained the reestimation equations for the hyperparameters α and β by evaluation of the evidence
and then setting the derivatives of the resulting expression to zero. We now turn toan alternative approach for ﬁnding α and β based on the EM algorithm. Recall thatour goal is to maximize the evidence function p(t|α, β) given by (3.77) with respect
to α and β. Because the parameter vector w is marginalized out, we can regard it asa latent variable, and hence we can optimize this marginal likelihood function using
EM. In the E step, we compute the posterior distribution of w given the current setting of the parameters α and β and then use this to ﬁnd the expected complete-data
log likelihood. In the M step, we maximize this quantity with respect to α and β. We
have already derived the posterior distribution of w because this is given by (3.49).
The complete-data log likelihood function is then given byln p(t, w|α, β) = ln p(t|w, β) + ln p(w|α)(9.61)

9.3. An Alternative View of EM

449

where the likelihood p(t|w, β) and the prior p(w|α) are given by (3.10) and (3.52),
respectively, and y(x, w) is given by (3.3). Taking the expectation with respect tothe posterior distribution of w then gives M α α  TN
βE [ln p(t, w|α, β)] =− E w w +
ln
ln
2
2π
2
2
2π
−

N
β 
E (tn − wT φn )2 .
2

(9.62)

n=1

Exercise 9.20

Exercise 9.21

Setting the derivatives with respect to α to zero, we obtain the M step re-estimation
equation
M
M
α=
= T
.
(9.63)
T
E [w w]
mN mN + Tr(SN )An analogous result holds for β.
Note that this re-estimation equation takes a slightly different form from thecorresponding result (3.92) derived by direct evaluation of the evidence function.
However, they each involve computation and inversion (or eigen decomposition) ofan M × M matrix and hence will have comparable computational cost per iteration.
These two approaches to determining α should of course converge to the same
result (assuming they ﬁnd the same local maximum of the evidence function). Thiscan be veriﬁed by ﬁrst noting that the quantity γ is deﬁned by
γ =M −αMi=11= M − αTr(SN ).
λi + α(9.64)

At a stationary point of the evidence function, the re-estimation equation (3.92) will
be self-consistently satisﬁed, and hence we can substitute for γ to give
αmT
N mN = γ = M − αTr(SN )(9.65)and solving for α we obtain (9.63), which is precisely the EM re-estimation equation.
As a ﬁnal example, we consider a closely related model, namely the relevance
vector machine for regression discussed in Section 7.2.1. There we used direct maximization of the marginal likelihood to derive re-estimation equations for the hyperparameters α and β. Here we consider an alternative approach in which we view theweight vector w as a latent variable and apply the EM algorithm. The E step involves
ﬁnding the posterior distribution over the weights, and this is given by (7.81). In the
M step we maximize the expected complete-data log likelihood, which is deﬁned by
Ew [ln p(t|X, w, β)p(w|α)]

Exercise 9.22

(9.66)

where the expectation is taken with respect to the posterior distribution computed
using the ‘old’ parameter values. To compute the new parameter values we maximize
with respect to α and β to give450

9. MIXTURE MODELS AND EM

Exercise 9.23

αinew

=

(β new )−1=1
m2i + Σiit − ΦmN 2 + β −1 i γiN

(9.67)(9.68)

These re-estimation equations are formally equivalent to those obtained by direct
maxmization.

9.4. The EM Algorithm in General

Section 10.1

The expectation maximization algorithm, or EM algorithm, is a general technique forﬁnding maximum likelihood solutions for probabilistic models having latent variables (Dempster et al., 1977; McLachlan and Krishnan, 1997). Here we give a very
general treatment of the EM algorithm and in the process provide a proof that the
EM algorithm derived heuristically in Sections 9.2 and 9.3 for Gaussian mixtures
does indeed maximize the likelihood function (Csiszàr and Tusnàdy, 1984; Hathaway, 1986; Neal and Hinton, 1999). Our discussion will also form the basis for thederivation of the variational inference framework.
Consider a probabilistic model in which we collectively denote all of the observed variables by X and all of the hidden variables by Z. The joint distributionp(X, Z|θ) is governed by a set of parameters denoted θ. Our goal is to maximize
the likelihood function that is given byp(X, Z|θ).
(9.69)
p(X|θ) =Z

Here we are assuming Z is discrete, although the discussion is identical if Z comprises continuous variables or a combination of discrete and continuous variables,with summation replaced by integration as appropriate.
We shall suppose that direct optimization of p(X|θ) is difﬁcult, but that optimization of the complete-data likelihood function p(X, Z|θ) is signiﬁcantly easier.
Next we introduce a distribution q(Z) deﬁned over the latent variables, and we observe that, for any choice of q(Z), the following decomposition holds
ln p(X|θ) = L(q, θ) + KL(qp)(9.70)

where we have deﬁned
L(q, θ) =

q(Z) lnZ

KL(qp) = −Zq(Z) lnp(X, Z|θ)q(Z)p(Z|X, θ)q(Z)(9.71).

(9.72)

Note that L(q, θ) is a functional (see Appendix D for a discussion of functionals)of the distribution q(Z), and a function of the parameters θ. It is worth studying

9.4. The EM Algorithm in General
Figure 9.11

Illustration of the decomposition givenby (9.70), which holds for any choiceof distribution q(Z). Because theKullback-Leibler divergence satisﬁes KL(q||p)KL(qp)  0, we see that the quantity L(q, θ) is a lower bound on the log
likelihood function ln p(X|θ).

L(q, θ)Exercise 9.24

ln p(X|θ)carefully the forms of the expressions (9.71) and (9.72), and in particular noting thatthey differ in sign and also that L(q, θ) contains the joint distribution of X and Z
while KL(qp) contains the conditional distribution of Z given X. To verify thedecomposition (9.70), we ﬁrst make use of the product rule of probability to giveln p(X, Z|θ) = ln p(Z|X, θ) + ln p(X|θ)

Section 1.6.1

451(9.73)

which we then substitute into the expression for L(q, θ). This gives rise to two terms,
one of which cancels KL(qp) while the other gives the required log likelihoodln p(X|θ) after noting that q(Z) is a normalized distribution that sums to 1.
From (9.72), we see that KL(qp) is the Kullback-Leibler divergence between
q(Z) and the posterior distribution p(Z|X, θ). Recall that the Kullback-Leibler divergence satisﬁes KL(qp)  0, with equality if, and only if, q(Z) = p(Z|X, θ). It
therefore follows from (9.70) that L(q, θ)  ln p(X|θ), in other words that L(q, θ)
is a lower bound on ln p(X|θ). The decomposition (9.70) is illustrated in Figure 9.11.
The EM algorithm is a two-stage iterative optimization technique for ﬁnding
maximum likelihood solutions. We can use the decomposition (9.70) to deﬁne the
EM algorithm and to demonstrate that it does indeed maximize the log likelihood.
Suppose that the current value of the parameter vector is θ old . In the E step, thelower bound L(q, θ old ) is maximized with respect to q(Z) while holding θ old ﬁxed.
The solution to this maximization problem is easily seen by noting that the valueof ln p(X|θ old ) does not depend on q(Z) and so the largest value of L(q, θ old ) will
occur when the Kullback-Leibler divergence vanishes, in other words when q(Z) isequal to the posterior distribution p(Z|X, θ old ). In this case, the lower bound will
equal the log likelihood, as illustrated in Figure 9.12.
In the subsequent M step, the distribution q(Z) is held ﬁxed and the lower boundL(q, θ) is maximized with respect to θ to give some new value θ new . This willcause the lower bound L to increase (unless it is already at a maximum), which will
necessarily cause the corresponding log likelihood function to increase. Because the
distribution q is determined using the old parameter values rather than the new values
and is held ﬁxed during the M step, it will not equal the new posterior distributionp(Z|X, θ new ), and hence there will be a nonzero KL divergence. The increase in thelog likelihood function is therefore greater than the increase in the lower bound, as

452

9. MIXTURE MODELS AND EM
Figure 9.12

Illustration of the E step of KL(q||p) = 0
the EM algorithm. The q
distribution is set equal tothe posterior distribution forthe current parameter values θ old , causing the lower
bound to move up to thesame value as the log likelihood function, with the KL
L(q, θ old )divergence vanishing.

ln p(X|θ old )shown in Figure 9.13. If we substitute q(Z) = p(Z|X, θ old ) into (9.71), we see that,after the E step, the lower bound takes the form
L(q, θ) =p(Z|X, θ old ) ln p(X, Z|θ) −p(Z|X, θ old ) ln p(Z|X, θ old )Z= Q(θ, θZ
old) + const(9.74)

where the constant is simply the negative entropy of the q distribution and is therefore independent of θ. Thus in the M step, the quantity that is being maximized is the
expectation of the complete-data log likelihood, as we saw earlier in the case of mixtures of Gaussians. Note that the variable θ over which we are optimizing appears
only inside the logarithm. If the joint distribution p(Z, X|θ) comprises a member ofthe exponential family, or a product of such members, then we see that the logarithm
will cancel the exponential and lead to an M step that will be typically much simpler
than the maximization of the corresponding incomplete-data log likelihood functionp(X|θ).
The operation of the EM algorithm can also be viewed in the space of parameters, as illustrated schematically in Figure 9.14. Here the red curve depicts the (inFigure 9.13

Illustration of the M step of the EM
algorithm.
The distribution q(Z) KL(q||p)is held ﬁxed and the lower boundL(q, θ) is maximized with respect
to the parameter vector θ to givea revised value θ new . Because the
KL divergence is nonnegative, thiscauses the log likelihood ln p(X|θ)
to increase by at least as much as
the lower bound does.

L(q, θ new )ln p(X|θ new )9.4. The EM Algorithm in General
Figure 9.14

The EM algorithm involves alternately computing a lower boundon the log likelihood for the current parameter values and thenmaximizing this bound to obtainthe new parameter values. See
the text for a full discussion.

453

ln p(X|θ)L (q, θ)θ old θExercise 9.25

newcomplete data) log likelihood function whose value we wish to maximize. We startwith some initial parameter value θ old , and in the ﬁrst E step we evaluate the posterior distribution over latent variables, which gives rise to a lower bound L(θ, θ (old) )whose value equals the log likelihood at θ (old) , as shown by the blue curve. Note thatthe bound makes a tangential contact with the log likelihood at θ (old) , so that bothcurves have the same gradient. This bound is a convex function having a uniquemaximum (for mixture components from the exponential family). In the M step, thebound is maximized giving the value θ (new) , which gives a larger value of log likelihood than θ (old) . The subsequent E step then constructs a bound that is tangential atθ (new) as shown by the green curve.
For the particular case of an independent, identically distributed data set, X
will comprise N data points {xn } while Z will comprise N corresponding latentvariables {zn}, where n = 1, . . . , N . From the independence assumption, we have
p(X,
Z)
=
n p(xn , zn ) and, by marginalizing over the {zn } we have p(X) =

p(x
).
Using
the sum and product rules, we see that the posterior probabilitynnthat is evaluated in the E step takes the formNp(xn , zn |θ)Np(X, Z|θ)n=1
p(Z|X, θ) = =p(zn |xn , θ)=Np(X, Z|θ)n=1
p(xn , zn |θ)Z(9.75)

Z n=1

and so the posterior distribution also factorizes with respect to n. In the case of
the Gaussian mixture model this simply says that the responsibility that each of the
mixture components takes for a particular data point xn depends only on the valueof xn and on the parameters θ of the mixture components, not on the values of theother data points.
We have seen that both the E and the M steps of the EM algorithm are increasing the value of a well-deﬁned bound on the log likelihood function and that the454

9. MIXTURE MODELS AND EM
complete EM cycle will change the model parameters in such a way as to cause
the log likelihood to increase (unless it is already at a maximum, in which case theparameters remain unchanged).
We can also use the EM algorithm to maximize the posterior distribution p(θ|X)
for models in which we have introduced a prior p(θ) over the parameters. To see this,
we note that as a function of θ, we have p(θ|X) = p(θ, X)/p(X) and soln p(θ|X) = ln p(θ, X) − ln p(X).

(9.76)

Making use of the decomposition (9.70), we haveln p(θ|X) = L(q, θ) + KL(qp) + ln p(θ) − ln p(X) L(q, θ) + ln p(θ) − ln p(X).

(9.77)

where ln p(X) is a constant. We can again optimize the right-hand side alternately
with respect to q and θ. The optimization with respect to q gives rise to the same Estep equations as for the standard EM algorithm, because q only appears in L(q, θ).
The M-step equations are modiﬁed through the introduction of the prior term ln p(θ),
which typically requires only a small modiﬁcation to the standard maximum likelihood M-step equations.
The EM algorithm breaks down the potentially difﬁcult problem of maximizing
the likelihood function into two stages, the E step and the M step, each of which willoften prove simpler to implement. Nevertheless, for complex models it may be thecase that either the E step or the M step, or indeed both, remain intractable. This
leads to two possible extensions of the EM algorithm, as follows.
The generalized EM, or GEM, algorithm addresses the problem of an intractable
M step. Instead of aiming to maximize L(q, θ) with respect to θ, it seeks instead
to change the parameters in such a way as to increase its value. Again, because
L(q, θ) is a lower bound on the log likelihood function, each complete EM cycle of
the GEM algorithm is guaranteed to increase the value of the log likelihood (unlessthe parameters already correspond to a local maximum). One way to exploit the
GEM approach would be to use one of the nonlinear optimization strategies, suchas the conjugate gradients algorithm, during the M step. Another form of GEM
algorithm, known as the expectation conditional maximization, or ECM, algorithm,
involves making several constrained optimizations within each M step (Meng andRubin, 1993). For instance, the parameters might be partitioned into groups, and the
M step is broken down into multiple steps each of which involves optimizing one of
the subset with the remainder held ﬁxed.
We can similarly generalize the E step of the EM algorithm by performing apartial, rather than complete, optimization of L(q, θ) with respect to q(Z) (Neal and
Hinton, 1999). As we have seen, for any given value of θ there is a unique maximumof L(q, θ) with respect to q(Z) that corresponds to the posterior distribution qθ (Z) =p(Z|X, θ) and that for this choice of q(Z) the bound L(q, θ) is equal to the log
likelihood function ln p(X|θ). It follows that any algorithm that converges to theglobal maximum of L(q, θ) will ﬁnd a value of θ that is also a global maximumof the log likelihood ln p(X|θ). Provided p(X, Z|θ) is a continuous function of θ

Exercises

Exercise 9.26

455

then, by continuity, any local maximum of L(q, θ) will also be a local maximum ofln p(X|θ).
Consider the case of N independent data points x1 , . . . , xN with corresponding
latent variables z1 , . . . , zN . The joint distribution p(X, Z|θ) factorizes over the data
points, and this structure can be exploited in an incremental form of EM in which
at each EM cycle only one data point is processed at a time. In the E step, insteadof recomputing the responsibilities for all of the data points, we just re-evaluate theresponsibilities for one data point. It might appear that the subsequent M step would
require computation involving the responsibilities for all of the data points. However, if the mixture components are members of the exponential family, then the
responsibilities enter only through simple sufﬁcient statistics, and these can be updated efﬁciently. Consider, for instance, the case of a Gaussian mixture, and suppose
we perform an update for data point m in which the corresponding old and newvalues of the responsibilities are denoted γ old (zmk ) and γ new (zmk ). In the M step,
the required sufﬁcient statistics can be updated incrementally. For instance, for themeans the sufﬁcient statistics are deﬁned by (9.17) and (9.18) from which we obtain new
γ (zmk ) − γ old (zmk )newoldxm − µold(9.78)µ k = µk +kNknew
together with

Nknew = Nkold + γ new (zmk ) − γ old (zmk ).

(9.79)

The corresponding results for the covariances and the mixing coefﬁcients are analogous.
Thus both the E step and the M step take ﬁxed time that is independent of thetotal number of data points. Because the parameters are revised after each data point,
rather than waiting until after the whole data set is processed, this incremental version can converge faster than the batch version. Each E or M step in this incremental
algorithm is increasing the value of L(q, θ) and, as we have shown above, if thealgorithm converges to a local (or global) maximum of L(q, θ), this will correspondto a local (or global) maximum of the log likelihood function ln p(X|θ).

Exercises

9.1 ( ) www Consider the K-means algorithm discussed in Section 9.1. Show that asa consequence of there being a ﬁnite number of possible assignments for the set of
discrete indicator variables rnk , and that for each such assignment there is a uniqueoptimum for the {µk }, the K-means algorithm must converge after a ﬁnite numberof iterations.
9.2 ( ) Apply the Robbins-Monro sequential estimation procedure described in Section 2.3.5 to the problem of ﬁnding the roots of the regression function given by
the derivatives of J in (9.1) with respect to µk . Show that this leads to a stochastic
K-means algorithm in which, for each data point xn , the nearest prototype µk is
updated using (9.5).

456

9. MIXTURE MODELS AND EM
9.3 ( ) www Consider a Gaussian mixture model in which the marginal distributionp(z) for the latent variable is given by (9.10), and the conditional distribution p(x|z)
for the observed variable is given by (9.11). Show that the marginal distribution
p(x), obtained by summing p(z)p(x|z) over all possible values of z, is a Gaussian
mixture of the form (9.7).
9.4 ( ) Suppose we wish to use the EM algorithm to maximize the posterior distribution over parameters p(θ|X) for a model containing latent variables, where X isthe observed data set. Show that the E step remains the same as in the maximum
likelihood case, whereas in the M step the quantity to be maximized is given byQ(θ, θ old ) + ln p(θ) where Q(θ, θ old ) is deﬁned by (9.30).
9.5 ( ) Consider the directed graph for a Gaussian mixture model shown in Figure 9.6.
By making use of the d-separation criterion discussed in Section 8.2, show that theposterior distribution of the latent variables factorizes with respect to the different
data points so thatp(Z|X, µ, Σ, π) =Np(zn |xn , µ, Σ, π).

(9.80)

n=1

9.6 ( ) Consider a special case of a Gaussian mixture model in which the covariance matrices Σk of the components are all constrained to have a common valueΣ. Derive the EM equations for maximizing the likelihood function under such amodel.
9.7 ( ) www Verify that maximization of the complete-data log likelihood (9.36) fora Gaussian mixture model leads to the result that the means and covariances of eachcomponent are ﬁtted independently to the corresponding group of data points, andthe mixing coefﬁcients are given by the fractions of points in each group.
9.8 ( ) www Show that if we maximize (9.40) with respect to µk while keeping the
responsibilities γ(znk ) ﬁxed, we obtain the closed form solution given by (9.17).
9.9 ( ) Show that if we maximize (9.40) with respect to Σk and πk while keeping the
responsibilities γ(znk ) ﬁxed, we obtain the closed form solutions given by (9.19)
and (9.22).
9.10 () Consider a density model given by a mixture distribution
p(x) =K

πk p(x|k)(9.81)

k=1

and suppose that we partition the vector x into two parts so that x = (xa , xb ).
Show that the conditional density p(xb |xa ) is itself a mixture distribution and ﬁnd
expressions for the mixing coefﬁcients and for the component densities.

Exercises

457

9.11 ( ) In Section 9.3.2, we obtained a relationship between K means and EM for
Gaussian mixtures by considering a mixture model in which all components havecovariance I. Show that in the limit  → 0, maximizing the expected completedata log likelihood for this model, given by (9.40), is equivalent to minimizing thedistortion measure J for the K-means algorithm given by (9.1).
9.12 ( ) www

Consider a mixture distribution of the form
p(x) =K

πk p(x|k)(9.82)

k=1

where the elements of x could be discrete or continuous or a combination of these.
Denote the mean and covariance of p(x|k) by µk and Σk , respectively. Show thatthe mean and covariance of the mixture distribution are given by (9.49) and (9.50).
9.13 ( ) Using the re-estimation equations for the EM algorithm, show that a mixture of Bernoulli distributions, with its parameters set to values corresponding to a
maximum of the likelihood function, has the property that1 
xn ≡ x.
N
N

E[x] =(9.83)

n=1

Hence show that if the parameters of this model are initialized such that all components have the same mean µk = µ for k = 1, . . . , K, then the EM algorithm will
converge after one iteration, for any choice of the initial mixing coefﬁcients, and that
this solution has the property µk = x. Note that this represents a degenerate case of
the mixture model in which all of the components are identical, and in practice wetry to avoid such solutions by using an appropriate initialization.
9.14 ( ) Consider the joint distribution of latent and observed variables for the Bernoulli
distribution obtained by forming the product of p(x|z, µ) given by (9.52) and p(z|π)
given by (9.53). Show that if we marginalize this joint distribution with respect to z,
then we obtain (9.47).
9.15 ( ) www Show that if we maximize the expected complete-data log likelihood
function (9.55) for a mixture of Bernoulli distributions with respect to µk , we obtainthe M step equation (9.59).
9.16 ( ) Show that if we maximize the expected complete-data log likelihood function(9.55) for a mixture of Bernoulli distributions with respect to the mixing coefﬁcients
πk , using a Lagrange multiplier to enforce the summation constraint, we obtain the
M step equation (9.60).
9.17 ( ) www Show that as a consequence of the constraint 0  p(xn |µk )  1 forthe discrete variable xn , the incomplete-data log likelihood function for a mixtureof Bernoulli distributions is bounded above, and hence that there are no singularities
for which the likelihood goes to inﬁnity.

458

9. MIXTURE MODELS AND EM
9.18 ( ) Consider a Bernoulli mixture model as discussed in Section 9.3.3, together
with a prior distribution p(µk |ak , bk ) over each of the parameter vectors µk givenby the beta distribution (2.13), and a Dirichlet prior p(π|α) given by (2.38). Derive
the EM algorithm for maximizing the posterior probability p(µ, π|X).
9.19 ( ) Consider a D-dimensional variable x each of whose components i is itself amultinomial variable of degree M so that x is a binary vector with components
xij
where i = 1, . . . , D and j = 1, . . . , M , subject to the constraint that j xij = 1 for
all i. Suppose that the distribution of these variables is described by a mixture of thediscrete multinomial distributions considered in Section 2.2 so thatp(x) =K

πk p(x|µk )(9.84)

k=1

wherep(x|µk ) =

D 
M


x

ij
µkij
.

(9.85)

i=1 j =1

The parameters µkij represent the probabilities p(xij = 1|µk ) and must satisfy
0  µkij  1 together with the constraint j µkij = 1 for all values of k and i.
Given an observed data set {xn }, where n = 1, . . . , N , derive the E and M stepequations of the EM algorithm for optimizing the mixing coefﬁcients πk and thecomponent parameters µkij of this distribution by maximum likelihood.
9.20 ( ) www Show that maximization of the expected complete-data log likelihood
function (9.62) for the Bayesian linear regression model leads to the M step reestimation result (9.63) for α.
9.21 ( ) Using the evidence framework of Section 3.5, derive the M-step re-estimation
equations for the parameter β in the Bayesian linear regression model, analogous to
the result (9.63) for α.
9.22 ( ) By maximization of the expected complete-data log likelihood deﬁned by(9.66), derive the M step equations (9.67) and (9.68) for re-estimating the hyperparameters of the relevance vector machine for regression.
9.23 ( ) www In Section 7.2.1 we used direct maximization of the marginal likelihood to derive the re-estimation equations (7.87) and (7.88) for ﬁnding values of
the hyperparameters α and β for the regression RVM. Similarly, in Section 9.3.4
we used the EM algorithm to maximize the same marginal likelihood, giving there-estimation equations (9.67) and (9.68). Show that these two sets of re-estimation
equations are formally equivalent.
9.24 ( ) Verify the relation (9.70) in which L(q, θ) and KL(qp) are deﬁned by (9.71)
and (9.72), respectively.

Exercises

459

Show that the lower bound L(q, θ) given by (9.71), with q(Z) =9.25 ( ) wwwp(Z|X, θ (old) ), has the same gradient with respect to θ as the log likelihood functionln p(X|θ) at the point θ = θ (old) .
9.26 ( ) www Consider the incremental form of the EM algorithm for a mixture ofGaussians, in which the responsibilities are recomputed only for a speciﬁc data point
xm . Starting from the M-step formulae (9.17) and (9.18), derive the results (9.78)
and (9.79) for updating the component means.
9.27 ( ) Derive M-step formulae for updating the covariance matrices and mixing
coefﬁcients in a Gaussian mixture model when the responsibilities are updated incrementally, analogous to the result (9.78) for updating the means.

10Approximate
Inference

A central task in the application of probabilistic models is the evaluation of the posterior distribution p(Z|X) of the latent variables Z given the observed (visible) data
variables X, and the evaluation of expectations computed with respect to this distribution. The model might also contain some deterministic parameters, which wewill leave implicit for the moment, or it may be a fully Bayesian model in which any
unknown parameters are given prior distributions and are absorbed into the set oflatent variables denoted by the vector Z. For instance, in the EM algorithm we needto evaluate the expectation of the complete-data log likelihood with respect to theposterior distribution of the latent variables. For many models of practical interest, it
will be infeasible to evaluate the posterior distribution or indeed to compute expectations with respect to this distribution. This could be because the dimensionality of
the latent space is too high to work with directly or because the posterior distribution
has a highly complex form for which expectations are not analytically tractable. In
the case of continuous variables, the required integrations may not have closed-form

461

462

10. APPROXIMATE INFERENCE
analytical solutions, while the dimensionality of the space and the complexity of theintegrand may prohibit numerical integration. For discrete variables, the marginalizations involve summing over all possible conﬁgurations of the hidden variables,
and though this is always possible in principle, we often ﬁnd in practice that there
may be exponentially many hidden states so that exact calculation is prohibitively
expensive.
In such situations, we need to resort to approximation schemes, and these fallbroadly into two classes, according to whether they rely on stochastic or deterministic approximations. Stochastic techniques such as Markov chain Monte Carlo, described in Chapter 11, have enabled the widespread use of Bayesian methods across
many domains. They generally have the property that given inﬁnite computational
resource, they can generate exact results, and the approximation arises from the useof a ﬁnite amount of processor time. In practice, sampling methods can be computationally demanding, often limiting their use to small-scale problems. Also, it canbe difﬁcult to know whether a sampling scheme is generating independent samples
from the required distribution.
In this chapter, we introduce a range of deterministic approximation schemes,
some of which scale well to large applications. These are based on analytical approximations to the posterior distribution, for example by assuming that it factorizes
in a particular way or that it has a speciﬁc parametric form such as a Gaussian. As
such, they can never generate exact results, and so their strengths and weaknesses
are complementary to those of sampling methods.
In Section 4.4, we discussed the Laplace approximation, which is based on a
local Gaussian approximation to a mode (i.e., a maximum) of the distribution. Here
we turn to a family of approximation techniques called variational inference or variational Bayes, which use more global criteria and which have been widely applied.
We conclude with a brief introduction to an alternative variational framework known
as expectation propagation.

10.1. Variational Inference
Variational methods have their origins in the 18th century with the work of Euler,
Lagrange, and others on the calculus of variations. Standard calculus is concernedwith ﬁnding derivatives of functions. We can think of a function as a mapping thattakes the value of a variable as the input and returns the value of the function as theoutput. The derivative of the function then describes how the output value varies
as we make inﬁnitesimal changes to the input value. Similarly, we can deﬁne afunctional as a mapping that takes a function as the input and that returns the valueof the functional as the output. An example would be the entropy H[p], which takes
a probability distribution p(x) as the input and returns the quantity
H[p] = p(x) ln p(x) dx(10.1)

10.1. Variational Inference463

as the output. We can the introduce the concept of a functional derivative, which expresses how the value of the functional changes in response to inﬁnitesimal changes
to the input function (Feynman et al., 1964). The rules for the calculus of variations
mirror those of standard calculus and are discussed in Appendix D. Many problems
can be expressed in terms of an optimization problem in which the quantity being
optimized is a functional. The solution is obtained by exploring all possible input
functions to ﬁnd the one that maximizes, or minimizes, the functional. Variational
methods have broad applicability and include such areas as ﬁnite element methods(Kapur, 1989) and maximum entropy (Schwarz, 1988).
Although there is nothing intrinsically approximate about variational methods,
they do naturally lend themselves to ﬁnding approximate solutions. This is doneby restricting the range of functions over which the optimization is performed, forinstance by considering only quadratic functions or by considering functions composed of a linear combination of ﬁxed basis functions in which only the coefﬁcientsof the linear combination can vary. In the case of applications to probabilistic inference, the restriction may for example take the form of factorization assumptions(Jordan et al., 1999; Jaakkola, 2001).
Now let us consider in more detail how the concept of variational optimization
can be applied to the inference problem. Suppose we have a fully Bayesian model inwhich all parameters are given prior distributions. The model may also have latentvariables as well as parameters, and we shall denote the set of all latent variables
and parameters by Z. Similarly, we denote the set of all observed variables by X.
For example, we might have a set of N independent, identically distributed data,
for which X = {x1 , . . . , xN } and Z = {z1 , . . . , zN }. Our probabilistic model
speciﬁes the joint distribution p(X, Z), and our goal is to ﬁnd an approximation forthe posterior distribution p(Z|X) as well as for the model evidence p(X). As in our
discussion of EM, we can decompose the log marginal probability using
ln p(X) = L(q) + KL(qp)where we have deﬁned(10.2)

p(X, Z)q(Z) lndZq(Z)p(Z|X)dZ.
KL(qp) = − q(Z) lnq(Z)L(q) =(10.3)(10.4)

This differs from our discussion of EM only in that the parameter vector θ no longer
appears, because the parameters are now stochastic variables and are absorbed into
Z. Since in this chapter we will mainly be interested in continuous variables we haveused integrations rather than summations in formulating this decomposition. However, the analysis goes through unchanged if some or all of the variables are discrete
simply by replacing the integrations with summations as required. As before, we
can maximize the lower bound L(q) by optimization with respect to the distribution
q(Z), which is equivalent to minimizing the KL divergence. If we allow any possible
choice for q(Z), then the maximum of the lower bound occurs when the KL divergence vanishes, which occurs when q(Z) equals the posterior distribution p(Z|X).

464

10. APPROXIMATE INFERENCE

1

40

0.8

30

0.6
20
0.4
10

0.2
0
−2

−1

0

1

2

3

4

0
−2

−1

0

1

2

3

4

Figure 10.1 Illustration of the variational approximation for the example considered earlier in Figure 4.14. The
left-hand plot shows the original distribution (yellow) along with the Laplace (red) and variational (green) approximations, and the right-hand plot shows the negative logarithms of the corresponding curves.

However, we shall suppose the model is such that working with the true posterior
distribution is intractable.
We therefore consider instead a restricted family of distributions q(Z) and thenseek the member of this family for which the KL divergence is minimized. Our goal
is to restrict the family sufﬁciently that they comprise only tractable distributions,while at the same time allowing the family to be sufﬁciently rich and ﬂexible that it
can provide a good approximation to the true posterior distribution. It is important toemphasize that the restriction is imposed purely to achieve tractability, and that subject to this requirement we should use as rich a family of approximating distributions
as possible. In particular, there is no ‘over-ﬁtting’ associated with highly ﬂexible distributions. Using more ﬂexible approximations simply allows us to approach the true
posterior distribution more closely.
One way to restrict the family of approximating distributions is to use a parametric distribution q(Z|ω) governed by a set of parameters ω. The lower bound L(q)then becomes a function of ω, and we can exploit standard nonlinear optimization
techniques to determine the optimal values for the parameters. An example of thisapproach, in which the variational distribution is a Gaussian and we have optimized
with respect to its mean and variance, is shown in Figure 10.1.

10.1.1 Factorized distributions
Here we consider an alternative way in which to restrict the family of distributions q(Z). Suppose we partition the elements of Z into disjoint groups that wedenote by Zi where i = 1, . . . , M . We then assume that the q distribution factorizes
with respect to these groups, so thatq(Z) =

M
i=1

qi (Zi ).

(10.5)

10.1. Variational Inference465

It should be emphasized that we are making no further assumptions about the distribution. In particular, we place no restriction on the functional forms of the individual
factors qi (Zi ). This factorized form of variational inference corresponds to an approximation framework developed in physics called mean ﬁeld theory (Parisi, 1988).
Amongst all distributions q(Z) having the form (10.5), we now seek that distribution for which the lower bound L(q) is largest. We therefore wish to make a freeform (variational) optimization of L(q) with respect to all of the distributions qi (Zi ),which we do by optimizing with respect to each of the factors in turn. To achievethis, we ﬁrst substitute (10.5) into (10.3) and then dissect out the dependence on oneof the factors qj (Zj ). Denoting qj (Zj ) by simply qj to keep the notation uncluttered,
we then obtain

  

L(q) =
qi ln p(X, Z) −ln qi dZ



i

qj

=

i


ln p(X, Z)

i= j


=



qj ln p(X, Zj ) dZj −qi dZidZj −qj ln qj dZj + const
qj ln qj dZj + const(10.6)

where we have deﬁned a new distribution p(X, Zj ) by the relationln p(X, Zj ) = Ei=j [ln p(X, Z)] + const.

(10.7)

Here the notation Ei=j [· · · ] denotes an expectation with respect to the q distributions
over all variables zi for i = j, so that
Ei=j [ln p(X, Z)] = ln p(X, Z)
qi dZi .
(10.8)
i= j

Now suppose we keep the {qi=j } ﬁxed and maximize L(q) in (10.6) with respect to all possible forms for the distribution qj (Zj ). This is easily done by recognizing that (10.6) is a negative Kullback-Leibler divergence between qj (Zj ) and
p(X, Zj ). Thus maximizing (10.6) is equivalent to minimizing the Kullback-Leibler

Leonhard Euler
1707–1783
Euler was a Swiss mathematician
and physicist who worked in St.
Petersburg and Berlin and who iswidely considered to be one of thegreatest mathematicians of all time.
He is certainly the most proliﬁc, andhis collected works ﬁll 75 volumes. Amongst his many

contributions, he formulated the modern theory of thefunction, he developed (together with Lagrange) the
calculus of variations, and he discovered the formulaeiπ = −1, which relates four of the most importantnumbers in mathematics. During the last 17 years of
his life, he was almost totally blind, and yet he produced nearly half of his results during this period.

466

10. APPROXIMATE INFERENCE
p(X, Zj ). Thus we obtain adivergence, and the minimum occurs when qj (Zj ) = 
general expression for the optimal solution qj (Zj ) given byln qj (Zj ) = Ei=j [ln p(X, Z)] + const.

(10.9)

It is worth taking a few moments to study the form of this solution as it provides thebasis for applications of variational methods. It says that the log of the optimal solution for factor qj is obtained simply by considering the log of the joint distribution
over all hidden and visible variables and then taking the expectation with respect toall of the other factors {qi } for i = j.
The additive constant in (10.9) is set by normalizing the distribution qj (Zj ).
Thus if we take the exponential of both sides and normalize, we haveqj (Zj ) = 

exp (Ei=j [ln p(X, Z)]).

exp (Ei=j [ln p(X, Z)]) dZj

In practice, we shall ﬁnd it more convenient to work with the form (10.9) and then reinstate the normalization constant (where required) by inspection. This will become
clear from subsequent examples.
The set of equations given by (10.9) for j = 1, . . . , M represent a set of consistency conditions for the maximum of the lower bound subject to the factorization
constraint. However, they do not represent an explicit solution because the expression on the right-hand side of (10.9) for the optimum qj (Zj ) depends on expectations
computed with respect to the other factors qi (Zi ) for i = j. We will therefore seeka consistent solution by ﬁrst initializing all of the factors qi (Zi ) appropriately andthen cycling through the factors and replacing each in turn with a revised estimate
given by the right-hand side of (10.9) evaluated using the current estimates for all of
the other factors. Convergence is guaranteed because bound is convex with respect
to each of the factors qi (Zi ) (Boyd and Vandenberghe, 2004).

10.1.2 Properties of factorized approximations
Our approach to variational inference is based on a factorized approximation to
the true posterior distribution. Let us consider for a moment the problem of approximating a general distribution by a factorized distribution. To begin with, we discuss
the problem of approximating a Gaussian distribution using a factorized Gaussian,which will provide useful insight into the types of inaccuracy introduced in using
factorized approximations. Consider a Gaussian distribution p(z) = N (z|µ, Λ−1 )over two correlated variables z = (z1 , z2 ) in which the mean and precision haveelements µ1
Λ11 Λ12µ=,Λ=(10.10)µ2
Λ21 Λ22and Λ21 = Λ12 due to the symmetry of the precision matrix. Now suppose wewish to approximate this distribution using a factorized Gaussian of the form q(z) =q1 (z1 )q2 (z2 ). We ﬁrst apply the general result (10.9) to ﬁnd an expression for the10.1. Variational Inference

467

optimal factor q1 (z1 ). In doing so it is useful to note that on the right-hand side weonly need to retain those terms that have some functional dependence on z1 because
all other terms can be absorbed into the normalization constant. Thus we haveln q1 (z1 ) = Ez2 [ln p(z)] + const1
2= Ez2 − (z1 − µ1 ) Λ11 − (z1 − µ1 )Λ12 (z2 − µ2 ) + const2
1 2= − z1 Λ11 + z1 µ1 Λ11 − z1 Λ12 (E[z2 ] − µ2 ) + const. (10.11)
2

Section 2.3.1

Next we observe that the right-hand side of this expression is a quadratic function of
z1 , and so we can identify q  (z1 ) as a Gaussian distribution. It is worth emphasizingthat we did not assume that q(zi ) is Gaussian, but rather we derived this result by
variational optimization of the KL divergence over all possible distributions q(zi ).
Note also that we do not need to consider the additive constant in (10.9) explicitly
because it represents the normalization constant that can be found at the end byinspection if required. Using the technique of completing the square, we can identify
the mean and precision of this Gaussian, giving

where

1
q  (z1 ) = N (z1 |m1 , Λ−
11 )(10.12)1
m1 = µ1 − Λ−
11 Λ12 (E[z2 ] − µ2 ) .

(10.13)By symmetry, q2 (z2 ) is also Gaussian and can be written as

in which

Exercise 10.2

1
q2 (z2 ) = N (z2 |m2 , Λ−
22 )(10.14)1
m2 = µ2 − Λ−22 Λ21 (E[z1 ] − µ1 ) .

(10.15)Note that these solutions are coupled, so that q  (z1 ) depends on expectations computed with respect to q  (z2 ) and vice versa. In general, we address this by treating
the variational solutions as re-estimation equations and cycling through the variables
in turn updating them until some convergence criterion is satisﬁed. We shall see
an example of this shortly. Here, however, we note that the problem is sufﬁciently
simple that a closed form solution can be found. In particular, because E[z1 ] = m1
and E[z2 ] = m2 , we see that the two equations are satisﬁed if we take E[z1 ] = µ1
and E[z2 ] = µ2 , and it is easily shown that this is the only solution provided the distribution is nonsingular. This result is illustrated in Figure 10.2(a). We see that themean is correctly captured but that the variance of q(z) is controlled by the directionof smallest variance of p(z), and that the variance along the orthogonal direction issigniﬁcantly under-estimated. It is a general result that a factorized variational approximation tends to give approximations to the posterior distribution that are toocompact.
By way of comparison, suppose instead that we had been minimizing the reverse
Kullback-Leibler divergence KL(pq). As we shall see, this form of KL divergence468

10. APPROXIMATE INFERENCE

Figure 10.2 Comparisonofthe two alternative forms for the
Kullback-Leibler divergence. The
green contours corresponding to
1, 2, and 3 standard deviations fora correlated Gaussian distributionp(z) over two variables z1 and z2 ,and the red contours represent
the corresponding levels for an
approximating
distributionq(z)over the same variables given by
the product of two independent
univariate Gaussian distributions
whose parameters are obtained byminimization of (a) the KullbackLeibler divergence KL(qp), and(b) the reverse Kullback-Leibler
divergence KL(pq).

Section 10.7

1

1

z2

z2

0.5

0.5

0

0

0.5
(a)

1

z1

0

0

0.5
(b)

z1

1

is used in an alternative approximate inference framework called expectation propagation. We therefore consider the general problem of minimizing KL(pq) whenq(Z) is a factorized approximation of the form (10.5). The KL divergence can thenbe written in the formMln qi (Zi ) dZ + const(10.16)KL(pq) = − p(Z)i=1

Exercise 10.3

where the constant term is simply the entropy of p(Z) and so does not depend on
q(Z). We can now optimize with respect to each of the factors qj (Zj ), which iseasily done using a Lagrange multiplier to givedZi = p(Zj ).
(10.17)qj (Zj ) = p(Z)i= jIn this case, we ﬁnd that the optimal solution for qj (Zj ) is just given by the corresponding marginal distribution of p(Z). Note that this is a closed-form solution andso does not require iteration.
To apply this result to the illustrative example of a Gaussian distribution p(z)over a vector z we can use (2.98), which gives the result shown in Figure 10.2(b).
We see that once again the mean of the approximation is correct, but that it places
signiﬁcant probability mass in regions of variable space that have very low probability.
The difference between these two results can be understood by noting that there
is a large positive contribution to the Kullback-Leibler divergence

p(Z)
dZ
(10.18)KL(qp) = − q(Z) lnq(Z)10.1. Variational Inference

(a)

(b)

469

(c)

Figure 10.3 Another comparison of the two alternative forms for the Kullback-Leibler divergence. (a) The blue
contours show a bimodal distribution p(Z) given by a mixture of two Gaussians, and the red contours correspondto the single Gaussian distribution q(Z) that best approximates p(Z) in the sense of minimizing the KullbackLeibler divergence KL(pq). (b) As in (a) but now the red contours correspond to a Gaussian distribution q(Z)found by numerical minimization of the Kullback-Leibler divergence KL(qp). (c) As in (b) but showing a different
local minimum of the Kullback-Leibler divergence.

Section 10.7

from regions of Z space in which p(Z) is near zero unless q(Z) is also close to
zero. Thus minimizing this form of KL divergence leads to distributions q(Z) thatavoid regions in which p(Z) is small. Conversely, the Kullback-Leibler divergence
KL(pq) is minimized by distributions q(Z) that are nonzero in regions where p(Z)is nonzero.
We can gain further insight into the different behaviour of the two KL divergences if we consider approximating a multimodal distribution by a unimodal one,as illustrated in Figure 10.3. In practical applications, the true posterior distribution will often be multimodal, with most of the posterior mass concentrated insome number of relatively small regions of parameter space. These multiple modes
may arise through nonidentiﬁability in the latent space or through complex nonlinear dependence on the parameters. Both types of multimodality were encountered in
Chapter 9 in the context of Gaussian mixtures, where they manifested themselves asmultiple maxima in the likelihood function, and a variational treatment based on theminimization of KL(qp) will tend to ﬁnd one of these modes. By contrast, if wewere to minimize KL(pq), the resulting approximations would average across allof the modes and, in the context of the mixture model, would lead to poor predictive
distributions (because the average of two good parameter values is typically itself
not a good parameter value). It is possible to make use of KL(pq) to deﬁne a usefulinference procedure, but this requires a rather different approach to the one discussed
here, and will be considered in detail when we discuss expectation propagation.
The two forms of Kullback-Leibler divergence are members of the alpha family

470

10. APPROXIMATE INFERENCE
of divergences (Ali and Silvey, 1966; Amari, 1985; Minka, 2005) deﬁned by4(1+α)/2(1−α)/2
1 − p(x)q(x)
dx(10.19)Dα (pq) =1 − α2

Exercise 10.6

where −∞ < α < ∞ is a continuous parameter. The Kullback-Leibler divergence
KL(pq) corresponds to the limit α → 1, whereas KL(qp) corresponds to the limit
α → −1. For all values of α we have Dα (pq)  0, with equality if, and only if,p(x) = q(x). Suppose p(x) is a ﬁxed distribution, and we minimize Dα (pq) withrespect to some set of distributions q(x). Then for α  −1 the divergence is zero
forcing, so that any values of x for which p(x) = 0 will have q(x) = 0, and typicallyq(x) will under-estimate the support of p(x) and will tend to seek the mode with thelargest mass. Conversely for α  1 the divergence is zero-avoiding, so that valuesof x for which p(x) > 0 will have q(x) > 0, and typically q(x) will stretch to cover
all of p(x), and will over-estimate the support of p(x). When α = 0 we obtain asymmetric divergence that is linearly related to the Hellinger distance given byp(x)1/2 − q(x)1/2 dx.
(10.20)DH (pq) =The square root of the Hellinger distance is a valid distance metric.

10.1.3 Example: The univariate Gaussian
We now illustrate the factorized variational approximation using a Gaussian distribution over a single variable x (MacKay, 2003). Our goal is to infer the posterior
distribution for the mean µ and precision τ , given a data set D = {x1 , . . . , xN } ofobserved values of x which are assumed to be drawn independently from the Gaussian. The likelihood function is given byN τ N/2
τ2
exp −(xn − µ) .
(10.21)p(D|µ, τ ) =
2π
2
n=1

We now introduce conjugate prior distributions for µ and τ given byp(µ|τ ) = N µ|µ0 , (λ0 τ )−1
p(τ ) = Gam(τ |a0 , b0 )Section 2.3.6
Exercise 2.44

(10.22)(10.23)where Gam(τ |a0 , b0 ) is the gamma distribution deﬁned by (2.146). Together these
distributions constitute a Gaussian-Gamma conjugate prior distribution.
For this simple problem the posterior distribution can be found exactly, and again
takes the form of a Gaussian-gamma distribution. However, for tutorial purposes
we will consider a factorized variational approximation to the posterior distribution
given by(10.24)q(µ, τ ) = qµ (µ)qτ (τ ).

10.1. Variational Inference

471

Note that the true posterior distribution does not factorize in this way. The optimum
factors qµ (µ) and qτ (τ ) can be obtained from the general result (10.9) as follows.
For qµ (µ) we haveln qµ (µ) = Eτ [ln p(D|µ, τ ) + ln p(µ|τ )] + const


N

E[τ ]
2
2
λ0 (µ − µ0 ) +(xn − µ) + const. (10.25)
= −
2
n=1

Exercise 10.7

1Completing the square over µ we see that qµ (µ) is a Gaussian N µ|µN , λ−withNmean and precision given by

λ0 µ0 + N x
λ0 + N= (λ0 + N )E[τ ].

µN

=

λN

(10.26)(10.27)Note that for N → ∞ this gives the maximum likelihood result in which µN = x
and the precision is inﬁnite.
Similarly, the optimal solution for the factor qτ (τ ) is given byln qτ (τ ) = Eµ [ln p(D|µ, τ ) + ln p(µ|τ )] + ln p(τ ) + constNln τ= (a0 − 1) ln τ − b0 τ +
2
N

τ
− Eµ
(xn − µ)2 + λ0 (µ − µ0 )2 + const
2(10.28)n=1

and hence qτ (τ ) is a gamma distribution Gam(τ |aN , bN ) with parameters

Exercise 10.8

Section 10.4.1

N
2

aN

= a0 +bN

1= b0 + Eµ2

N(10.29)(xn − µ)2 + λ0 (µ − µ0 )2 .

(10.30)n=1

Again this exhibits the expected behaviour when N → ∞.
It should be emphasized that we did not assume these speciﬁc functional forms
for the optimal distributions qµ (µ) and qτ (τ ). They arose naturally from the structureof the likelihood function and the corresponding conjugate priors.
Thus we have expressions for the optimal distributions qµ (µ) and qτ (τ ) each ofwhich depends on moments evaluated with respect to the other distribution. One approach to ﬁnding a solution is therefore to make an initial guess for, say, the moment
E[τ ] and use this to re-compute the distribution qµ (µ). Given this revised distribution we can then extract the required moments E[µ] and E[µ2 ], and use these torecompute the distribution qτ (τ ), and so on. Since the space of hidden variables forthis example is only two dimensional, we can illustrate the variational approximation to the posterior distribution by plotting contours of both the true posterior andthe factorized approximation, as illustrated in Figure 10.4.

472

10. APPROXIMATE INFERENCE

2

2
(a)

(b)

τ

τ

1

1

0
−1

0

µ

0
−1

1

2

0

µ

1

0

µ

1

2
(c)

(d)

τ

τ

1

0
−1

1

0

µ

1

0
−1

Figure 10.4 Illustration of variational inference for the mean µ and precision τ of a univariate Gaussian distribution. Contours of the true posterior distribution p(µ, τ |D) are shown in green. (a) Contours of the initial factorized
approximation qµ (µ)qτ (τ ) are shown in blue. (b) After re-estimating the factor qµ (µ). (c) After re-estimating thefactor qτ (τ ). (d) Contours of the optimal factorized approximation, to which the iterative scheme converges, areshown in red.

Appendix B

In general, we will need to use an iterative approach such as this in order tosolve for the optimal factorized posterior distribution. For the very simple example
we are considering here, however, we can ﬁnd an explicit solution by solving thesimultaneous equations for the optimal factors qµ (µ) and qτ (τ ). Before doing this,we can simplify these expressions by considering broad, noninformative priors inwhich µ0 = a0 = b0 = λ0 = 0. Although these parameter settings correspond to
improper priors, we see that the posterior distribution is still well deﬁned. Using thestandard result E[τ ] = aN /bN for the mean of a gamma distribution, together with
(10.29) and (10.30), we have

N
1
1 
(xn − µ)2 = x2 − 2xE[µ] + E[µ2 ].
(10.31)
=E
E[τ ]
N
n=1

Then, using (10.26) and (10.27), we obtain the ﬁrst and second order moments of10.1. Variational Inference473

qµ (µ) in the form
E[µ2 ] = x2 +E[µ] = x,
Exercise 10.9

1.
N E[τ ]

(10.32)

We can now substitute these moments into (10.31) and then solve for E[τ ] to give
1
E[τ ]

=

1
(x2 − x2 )
N −1

=

1 
(xn − x)2 .
N −1
N(10.33)n=1

Section 1.2.4

We recognize the right-hand side as the familiar unbiased estimator for the varianceof a univariate Gaussian distribution, and so we see that the use of a Bayesian approach has avoided the bias of the maximum likelihood solution.

10.1.4 Model comparison

Exercise 10.10

As well as performing inference over the hidden variables Z, we may also
wish to compare a set of candidate models, labelled by the index m, and having
prior probabilities p(m). Our goal is then to approximate the posterior probabilities
p(m|X), where X is the observed data. This is a slightly more complex situation
than that considered so far because different models may have different structure
and indeed different dimensionality for the hidden variables Z. We cannot therefore simply consider a factorized approximation q(Z)q(m), but must instead recognize that the posterior over Z must be conditioned on m, and so we must considerq(Z, m) = q(Z|m)q(m). We can readily verify the following decomposition basedon this variational distributionln p(X) = Lm −mq(Z|m)q(m) ln

Zp(Z, m|X)
q(Z|m)q(m)(10.34)

where the Lm is a lower bound on ln p(X) and is given by
Lm =


m

Exercise 10.11

Z

q(Z|m)q(m) lnp(Z, X, m)
q(Z|m)q(m).

(10.35)Here we are assuming discrete Z, but the same analysis applies to continuous latentvariables provided the summations are replaced with integrations. We can maximize
Lm with respect to the distribution q(m) using a Lagrange multiplier, with the resultq(m) ∝ p(m) exp{Lm }.

(10.36)

However, if we maximize Lm with respect to the q(Z|m), we ﬁnd that the solutions
for different m are coupled, as we expect because they are conditioned on m. We
proceed instead by ﬁrst optimizing each of the q(Z|m) individually by optimization

474

10. APPROXIMATE INFERENCE
of (10.35), and then subsequently determining the q(m) using (10.36). After normalization the resulting values for q(m) can be used for model selection or model
averaging in the usual way.

10.2. Illustration: Variational Mixture of Gaussians
We now return to our discussion of the Gaussian mixture model and apply the variational inference machinery developed in the previous section. This will provide agood illustration of the application of variational methods and will also demonstrate
how a Bayesian treatment elegantly resolves many of the difﬁculties associated withthe maximum likelihood approach (Attias, 1999b). The reader is encouraged to workthrough this example in detail as it provides many insights into the practical application of variational methods. Many Bayesian models, corresponding to much moresophisticated distributions, can be solved by straightforward extensions and generalizations of this analysis.
Our starting point is the likelihood function for the Gaussian mixture model, illustrated by the graphical model in Figure 9.6. For each observation xn we havea corresponding latent variable zn comprising a 1-of-K binary vector with elements znk for k = 1, . . . , K. As before we denote the observed data set by X ={x1 , . . . , xN }, and similarly we denote the latent variables by Z = {z1 , . . . , zN }.
From (9.10) we can write down the conditional distribution of Z, given the mixing
coefﬁcients π, in the formp(Z|π) =N K
n=1 k=1

πkznk .

(10.37)

Similarly, from (9.11), we can write down the conditional distribution of the observed data vectors, given the latent variables and the component parametersp(X|Z, µ, Λ) =

N 
K


1
N xn |µk , Λ−kznk

(10.38)n=1 k=1

Section 10.4.1

where µ = {µk } and Λ = {Λk }. Note that we are working in terms of precision
matrices rather than covariance matrices as this somewhat simpliﬁes the mathematics.
Next we introduce priors over the parameters µ, Λ and π. The analysis is considerably simpliﬁed if we use conjugate prior distributions. We therefore choose a
Dirichlet distribution over the mixing coefﬁcients πp(π) = Dir(π|α0 ) = C(α0 )Kπkα0 −1(10.39)k=1

where by symmetry we have chosen the same parameter α0 for each of the components, and C(α0 ) is the normalization constant for the Dirichlet distribution deﬁned

10.2. Illustration: Variational Mixture of Gaussians
Figure 10.5

Directed acyclic graph representing the Bayesian mix- πture of Gaussians model, in which the box (plate) denotes a set of N i.i.d. observations. Here µ denotes{µk } and Λ denotes {Λk }.

475

zn

Λ

xn

µ
N

Section 2.2.1

by (B.23). As we have seen, the parameter α0 can be interpreted as the effective
prior number of observations associated with each component of the mixture. If thevalue of α0 is small, then the posterior distribution will be inﬂuenced primarily by
the data rather than by the prior.
Similarly, we introduce an independent Gaussian-Wishart prior governing themean and precision of each Gaussian component, given byp(µ, Λ) = p(µ|Λ)p(Λ)=

KN µk |m0 , (β0 Λk )−1 W(Λk |W0 , ν0 )(10.40)k=1

Section 2.3.6

because this represents the conjugate prior distribution when both the mean and precision are unknown. Typically we would choose m0 = 0 by symmetry.
The resulting model can be represented as a directed graph as shown in Figure 10.5. Note that there is a link from Λ to µ since the variance of the distribution
over µ in (10.40) is a function of Λ.
This example provides a nice illustration of the distinction between latent variables and parameters. Variables such as zn that appear inside the plate are regarded
as latent variables because the number of such variables grows with the size of thedata set. By contrast, variables such as µ that are outside the plate are ﬁxed in
number independently of the size of the data set, and so are regarded as parameters.
From the perspective of graphical models, however, there is really no fundamental
difference between them.

10.2.1 Variational distribution
In order to formulate a variational treatment of this model, we next write down
the joint distribution of all of the random variables, which is given byp(X, Z, π, µ, Λ) = p(X|Z, µ, Λ)p(Z|π)p(π)p(µ|Λ)p(Λ)(10.41)

in which the various factors are deﬁned above. The reader should take a moment toverify that this decomposition does indeed correspond to the probabilistic graphicalmodel shown in Figure 10.5. Note that only the variables X = {x1 , . . . , xN } are
observed.

476

10. APPROXIMATE INFERENCE
We now consider a variational distribution which factorizes between the latentvariables and the parameters so thatq(Z, π, µ, Λ) = q(Z)q(π, µ, Λ).

(10.42)

It is remarkable that this is the only assumption that we need to make in order toobtain a tractable practical solution to our Bayesian mixture model. In particular, thefunctional form of the factors q(Z) and q(π, µ, Λ) will be determined automaticallyby optimization of the variational distribution. Note that we are omitting the subscripts on the q distributions, much as we do with the p distributions in (10.41), andare relying on the arguments to distinguish the different distributions.
The corresponding sequential update equations for these factors can be easily
derived by making use of the general result (10.9). Let us consider the derivation of
the update equation for the factor q(Z). The log of the optimized factor is given byln q  (Z) = Eπ,µ,Λ [ln p(X, Z, π, µ, Λ)] + const.

(10.43)

We now make use of the decomposition (10.41). Note that we are only interested in
the functional dependence of the right-hand side on the variable Z. Thus any terms
that do not depend on Z can be absorbed into the additive normalization constant,givingln q  (Z) = Eπ [ln p(Z|π)] + Eµ,Λ [ln p(X|Z, µ, Λ)] + const.

(10.44)

Substituting for the two conditional distributions on the right-hand side, and again
absorbing any terms that are independent of Z into the additive constant, we haveln q (Z) =

N 
K


znk ln ρnk + const(10.45)n=1 k=1

where we have deﬁned
ln ρnk1
D= E[ln πk ] + E [ln |Λk | ] − ln(2π)
2
2

1
T
− Eµk ,Λk (xn − µk ) Λk (xn − µk )2

(10.46)

where D is the dimensionality of the data variable x. Taking the exponential of bothsides of (10.45) we obtainq (Z) ∝N K

n=1 k=1

Exercise 10.12

nk
ρznk
.

(10.47)

Requiring that this distribution be normalized, and noting that for each value of n
the quantities znk are binary and sum to 1 over all values of k, we obtainq  (Z) =N K
n=1 k=1

znk
rnk(10.48)

10.2. Illustration: Variational Mixture of Gaussians

477

ρnk
K


(10.49)

where
rnk =

.

ρnj

j =1

We see that the optimal solution for the factor q(Z) takes the same functional form
as the prior p(Z|π). Note that because ρnk is given by the exponential of a realquantity, the quantities rnk will be nonnegative and will sum to one, as required.
For the discrete distribution q  (Z) we have the standard result
E[znk ] = rnk(10.50)from which we see that the quantities rnk are playing the role of responsibilities.
Note that the optimal solution for q  (Z) depends on moments evaluated with respect
to the distributions of other variables, and so again the variational update equations
are coupled and must be solved iteratively.
At this point, we shall ﬁnd it convenient to deﬁne three statistics of the observed
data set evaluated with respect to the responsibilities, given by
Nk

=

N


rnk

(10.51)1 
rnk xnNk(10.52)

n=1
N

xk

=

n=1

Sk

=

N
1 

Nk

rnk (xn − xk )(xn − xk )T .

(10.53)n=1

Note that these are analogous to quantities evaluated in the maximum likelihood EM
algorithm for the Gaussian mixture model.
Now let us consider the factor q(π, µ, Λ) in the variational posterior distribution. Again using the general result (10.9) we haveln q (π, µ, Λ) = ln p(π) +Kln p(µk , Λk ) + EZ [ln p(Z|π)]

k=1

+

K 
N


1
E[znk ] ln N xn |µk , Λ−
+ const.
k(10.54)k=1 n=1

We observe that the right-hand side of this expression decomposes into a sum ofterms involving only π together with terms only involving µ and Λ, which implies
that the variational posterior q(π, µ, Λ) factorizes to give q(π)q(µ, Λ). Furthermore, the terms involving µ and Λ themselves comprise a sum over k of terms
involving µk and Λk leading to the further factorization
q(π, µ, Λ) = q(π)

K

k=1

q(µk , Λk ).

(10.55)

478

10. APPROXIMATE INFERENCE
Identifying the terms on the right-hand side of (10.54) that depend on π, we haveln q  (π) = (α0 − 1)Kln πk +K N

rnk ln πk + const(10.56)k=1 n=1

k=1

where we have used (10.50). Taking the exponential of both sides, we recognizeq  (π) as a Dirichlet distributionq  (π) = Dir(π|α)(10.57)

where α has components αk given by
α k = α0 + N k .

Exercise 10.13(10.58)

Finally, the variational posterior distribution q  (µk , Λk ) does not factorize into
the product of the marginals, but we can always use the product rule to write it in theform q  (µk , Λk ) = q  (µk |Λk )q  (Λk ). The two factors can be found by inspecting(10.54) and reading off those terms that involve µk and Λk . The result, as expected,
is a Gaussian-Wishart distribution and is given byq  (µk , Λk ) = N µk |mk , (βk Λk )−1 W(Λk |Wk , νk )(10.59)where we have deﬁned
βk
mk
Wk−1
νk

Exercise 10.14= β0 + Nk
1=(β0 m0 + Nk xk )βkβ0 Nk
= W0−1 + Nk Sk +(xk − m0 )(xk − m0 )T
β0 + Nk= ν0 + Nk .

(10.60)(10.61)(10.62)(10.63)

These update equations are analogous to the M-step equations of the EM algorithm
for the maximum likelihood solution of the mixture of Gaussians. We see that thecomputations that must be performed in order to update the variational posterior
distribution over the model parameters involve evaluation of the same sums over thedata set, as arose in the maximum likelihood treatment.
In order to perform this variational M step, we need the expectations E[znk ] =rnk representing the responsibilities. These are obtained by normalizing the ρnk thatare given by (10.46). We see that this expression involves expectations with respect
to the variational distributions of the parameters, and these are easily evaluated togiveEµk ,Λk (xn − µk )T Λk (xn − µk )= Dβk−1 + νk (xn − mk )T Wk (xn − mk ) (10.64)Dνk + 1 − iψ+ D ln 2 + ln |Wk | (10.65)
ln Λk ≡ E [ln |Λk | ] =2
i=1

k ≡ E [ln πk ] = ψ(αk ) − ψ(α)ln π(10.66)

10.2. Illustration: Variational Mixture of Gaussians

Appendix B479

 k and π
where we have introduced deﬁnitionsof Λk , and ψ(·) is the digamma function
deﬁned by (B.25), with α = k αk . The results (10.65) and (10.66) follow fromthe standard properties of the Wishart and Dirichlet distributions.
If we substitute (10.64), (10.65), and (10.66) into (10.46) and make use of(10.49), we obtain the following result for the responsibilities 1/2 exp − D − νk (xn − mk )T Wk (xn − mk ) .
k Λ
rnk ∝ πk2βk2(10.67)

Notice the similarity to the corresponding result for the responsibilities in maximum
likelihood EM, which from (9.13) can be written in the form1
rnk ∝ πk |Λk |1/2 exp − (xn − µk )T Λk (xn − µk )2Section 10.4.1

Section 3.4

Exercise 10.15(10.68)where we have used the precision in place of the covariance to highlight the similarity
to (10.67).
Thus the optimization of the variational posterior distribution involves cycling
between two stages analogous to the E and M steps of the maximum likelihood EM
algorithm. In the variational equivalent of the E step, we use the current distributions
over the model parameters to evaluate the moments in (10.64), (10.65), and (10.66)
and hence evaluate E[znk ] = rnk . Then in the subsequent variational equivalentof the M step, we keep these responsibilities ﬁxed and use them to re-compute thevariational distribution over the parameters using (10.57) and (10.59). In each case,
we see that the variational posterior distribution has the same functional form as thecorresponding factor in the joint distribution (10.41). This is a general result and isa consequence of the choice of conjugate distributions.
Figure 10.6 shows the results of applying this approach to the rescaled Old Faithful data set for a Gaussian mixture model having K = 6 components. We see thatafter convergence, there are only two components for which the expected valuesof the mixing coefﬁcients are numerically distinguishable from their prior values.
This effect can be understood qualitatively in terms of the automatic trade-off in a
Bayesian model between ﬁtting the data and the complexity of the model, in which
the complexity penalty arises from components whose parameters are pushed awayfrom their prior values. Components that take essentially no responsibility for explaining the data points have rnk  0 and hence Nk  0. From (10.58), we seethat αk  α0 and from (10.60)–(10.63) we see that the other parameters revert totheir prior values. In principle such components are ﬁtted slightly to the data points,
but for broad priors this effect is too small to be seen numerically. For the variational Gaussian mixture model the expected values of the mixing coefﬁcients in theposterior distribution are given by
E[πk ] =αk + N k.
Kα0 + N

(10.69)Consider a component for which Nk  0 and αk  α0 . If the prior is broad so thatα0 → 0, then E[πk ] → 0 and the component plays no role in the model, whereas if

480

10. APPROXIMATE INFERENCE

Figure 10.6 Variational Bayesian
mixture of K = 6 Gaussians applied to the Old Faithful data set, inwhich the ellipses denote the one
standard-deviation density contours
for each of the components, and thedensity of red ink inside each ellipse
corresponds to the mean value ofthe mixing coefﬁcient for each component. The number in the top leftof each diagram shows the number of iterations of variational inference. Components whose expected
mixing coefﬁcient are numerically indistinguishable from zero are notplotted.

0

15

60

120

the prior tightly constrains the mixing coefﬁcients so that α0 → ∞, then E[πk ] →1/K.
In Figure 10.6, the prior over the mixing coefﬁcients is a Dirichlet of the form
(10.39). Recall from Figure 2.5 that for α0 < 1 the prior favours solutions in whichsome of the mixing coefﬁcients are zero. Figure 10.6 was obtained using α0 = 10−3 ,and resulted in two components having nonzero mixing coefﬁcients. If instead wechoose α0 = 1 we obtain three components with nonzero mixing coefﬁcients, andfor α = 10 all six components have nonzero mixing coefﬁcients.
As we have seen there is a close similarity between the variational solution for
the Bayesian mixture of Gaussians and the EM algorithm for maximum likelihood.
In fact if we consider the limit N → ∞ then the Bayesian treatment converges to the
maximum likelihood EM algorithm. For anything other than very small data sets,
the dominant computational cost of the variational algorithm for Gaussian mixturesarises from the evaluation of the responsibilities, together with the evaluation andinversion of the weighted data covariance matrices. These computations mirror precisely those that arise in the maximum likelihood EM algorithm, and so there is little
computational overhead in using this Bayesian approach as compared to the traditional maximum likelihood one. There are, however, some substantial advantages.
First of all, the singularities that arise in maximum likelihood when a Gaussian component ‘collapses’ onto a speciﬁc data point are absent in the Bayesian treatment.

10.2. Illustration: Variational Mixture of Gaussians

Section 10.2.4481

Indeed, these singularities are removed if we simply introduce a prior and then use a
MAP estimate instead of maximum likelihood. Furthermore, there is no over-ﬁtting
if we choose a large number K of components in the mixture, as we saw in Figure 10.6. Finally, the variational treatment opens up the possibility of determining
the optimal number of components in the mixture without resorting to techniquessuch as cross validation.

10.2.2 Variational lower boundWe can also straightforwardly evaluate the lower bound (10.3) for this model.
In practice, it is useful to be able to monitor the bound during the re-estimation inorder to test for convergence. It can also provide a valuable check on both the mathematical expressions for the solutions and their software implementation, because ateach step of the iterative re-estimation procedure the value of this bound should notdecrease. We can take this a stage further to provide a deeper test of the correctnessof both the mathematical derivation of the update equations and of their software implementation by using ﬁnite differences to check that each update does indeed givea (constrained) maximum of the bound (Svensén and Bishop, 2004).
For the variational mixture of Gaussians, the lower bound (10.3) is given byp(X, Z, π, µ, Λ)q(Z, π, µ, Λ) ln
dπ dµ dΛL =q(Z, π, µ, Λ)Z= E[ln p(X, Z, π, µ, Λ)] − E[ln q(Z, π, µ, Λ)]= E[ln p(X|Z, µ, Λ)] + E[ln p(Z|π)] + E[ln p(π)] + E[ln p(µ, Λ)]−E[ln q(Z)] − E[ln q(π)] − E[ln q(µ, Λ)](10.70)Exercise 10.16

where, to keep the notation uncluttered, we have omitted the superscript on theq distributions, along with the subscripts on the expectation operators because each
expectation is taken with respect to all of the random variables in its argument. The
various terms in the bound are easily evaluated to give the following results1E[ln p(X|Z, µ, Λ)] =Nk2K k − Dβ −1 − νk Tr(Sk Wk )
ln Λ
k

k=1

−νk (xk − mk )T Wk (xk − mk ) − D ln(2π)
E[ln p(Z|π)] =

N 
K


rnk ln π
k(10.71)(10.72)n=1 k=1

E[ln p(π)] = ln C(α0 ) + (α0 − 1)

K

k=1

ln π
k

(10.73)

482

10. APPROXIMATE INFERENCE
1
 k − Dβ0
D ln(β0 /2π) + ln Λ2βkKE[ln p(µ, Λ)] =k=1−β0 νk (mk − m0 )T Wk (mk − m0 )++ K ln B(W0 , ν0 )1
(ν0 − D − 1)  ln Λk −
νk Tr(W0−1 Wk )2
2

E[ln q(Z)] =

K

K

k=1

k=1

N 
K


rnk ln rnk(10.74)(10.75)n=1 k=1

E[ln q(π)] =K(αk − 1) ln π
k + ln C(α)

k=1E[ln q(µ, Λ)] =

K

k=1

Exercise 10.18

1 D
ln Λk + ln
2
2



βk
2π

(10.76)

−D− H [q(Λk )]2(10.77)

where D is the dimensionality of x, H[q(Λk )] is the entropy of the Wishart distribution given by (B.82), and the coefﬁcients C(α) and B(W, ν) are deﬁned by (B.23)
and (B.79), respectively. Note that the terms involving expectations of the logs of theq distributions simply represent the negative entropies of those distributions. Some
simpliﬁcations and combination of terms can be performed when these expressions
are summed to give the lower bound. However, we have kept the expressions separate for ease of understanding.
Finally, it is worth noting that the lower bound provides an alternative approachfor deriving the variational re-estimation equations obtained in Section 10.2.1. To dothis we use the fact that, since the model has conjugate priors, the functional form of
the factors in the variational posterior distribution is known, namely discrete for Z,
Dirichlet for π, and Gaussian-Wishart for (µk , Λk ). By taking general parametricforms for these distributions we can derive the form of the lower bound as a functionof the parameters of the distributions. Maximizing the bound with respect to theseparameters then gives the required re-estimation equations.

10.2.3 Predictive density
In applications of the Bayesian mixture of Gaussians model we will often be of the observed variable. Asinterested in the predictive density for a new value x
sociated with this observation will be a corresponding latent variable z, and the predictive density is then given by|X) =p(x|
z, µ, Λ)p(
z|π)p(π, µ, Λ|X) dπ dµ dΛ(10.78)p(xbz10.2. Illustration: Variational Mixture of Gaussians

483

where p(π, µ, Λ|X) is the (unknown) true posterior distribution of the parameters.
Using (10.37) and (10.38) we can ﬁrst perform the summation over z to givep(x|X) =K 1
πk N xp(π, µ, Λ|X) dπ dµ dΛ.
|µk , Λ−k(10.79)k=1

Because the remaining integrations are intractable, we approximate the predictive
density by replacing the true posterior distribution p(π, µ, Λ|X) with its variational
approximation q(π)q(µ, Λ) to give|X) =
p(x

K 


1
|µk , Λ−
πk N xq(π)q(µk , Λk ) dπ dµk dΛkk(10.80)k=1

Exercise 10.19

where we have made use of the factorization (10.55) and in each term we have implicitly integrated out all variables {µj , Λj } for j = k The remaining integrations
can now be evaluated analytically giving a mixture of Student’s t-distributions
1|X) =
αk St(x|mk , Lk , νk + 1 − D)
p(x
α

K

(10.81)k=1

in which the k th component has mean mk , and the precision is given by
Lk =Exercise 10.20(νk + 1 − D)βk
Wk
(1 + βk )(10.82)in which νk is given by (10.63). When the size N of the data set is large the predictive
distribution (10.81) reduces to a mixture of Gaussians.

10.2.4 Determining the number of components
Section 10.1.4

Exercise 10.21

We have seen that the variational lower bound can be used to determine a posterior distribution over the number K of components in the mixture model. Thereis, however, one subtlety that needs to be addressed. For any given setting of theparameters in a Gaussian mixture model (except for speciﬁc degenerate settings),
there will exist other parameter settings for which the density over the observed variables will be identical. These parameter values differ only through a re-labelling of
the components. For instance, consider a mixture of two Gaussians and a single observed variable x, in which the parameters have the values π1 = a, π2 = b, µ1 = c,µ2 = d, σ1 = e, σ2 = f . Then the parameter values π1 = b, π2 = a, µ1 = d,µ2 = c, σ1 = f , σ2 = e, in which the two components have been exchanged, willby symmetry give rise to the same value of p(x). If we have a mixture model comprising K components, then each parameter setting will be a member of a family ofK! equivalent settings.
In the context of maximum likelihood, this redundancy is irrelevant because theparameter optimization algorithm (for example EM) will, depending on the initialization of the parameters, ﬁnd one speciﬁc solution, and the other equivalent solutions play no role. In a Bayesian setting, however, we marginalize over all possible

484

10. APPROXIMATE INFERENCE
Figure 10.7

Plot of the variational lower bound
L versus the number K of components in the Gaussian mixture
model, for the Old Faithful data,
showing a distinct peak at K =2 components. For each valueof K, the model is trained from100 different random starts, andthe results shown as ‘+’ symbols
p(D|K)
plotted with small random horizontal perturbations so that they
can be distinguished. Note thatsome solutions ﬁnd suboptimal
local maxima, but that this happens infrequently.

1

2

3

4

5

6

K

Exercise 10.22

Section 3.4

Exercise 10.23

parameter values. We have seen in Figure 10.2 that if the true posterior distribution
is multimodal, variational inference based on the minimization of KL(qp) will tend
to approximate the distribution in the neighbourhood of one of the modes and ignore
the others. Again, because equivalent modes have equivalent predictive densities,
this is of no concern provided we are considering a model having a speciﬁc number
K of components. If, however, we wish to compare different values of K, then weneed to take account of this multimodality. A simple approximate solution is to add
a term ln K! onto the lower bound when used for model comparison and averaging.
Figure 10.7 shows a plot of the lower bound, including the multimodality factor, versus the number K of components for the Old Faithful data set. It is worthemphasizing once again that maximum likelihood would lead to values of the likelihood function that increase monotonically with K (assuming the singular solutions
have been avoided, and discounting the effects of local maxima) and so cannot be
used to determine an appropriate model complexity. By contrast, Bayesian inference
automatically makes the trade-off between model complexity and ﬁtting the data.
This approach to the determination of K requires that a range of models having
different K values be trained and compared. An alternative approach to determininga suitable value for K is to treat the mixing coefﬁcients π as parameters and makepoint estimates of their values by maximizing the lower bound (Corduneanu andBishop, 2001) with respect to π instead of maintaining a probability distribution
over them as in the fully Bayesian approach. This leads to the re-estimation equation
1 
rnk
πk =
N
N

(10.83)n=1

and this maximization is interleaved with the variational updates for the q distribution
over the remaining parameters. Components that provide insufﬁcient contribution

10.2. Illustration: Variational Mixture of Gaussians

Section 7.2.2

485

to explaining the data will have their mixing coefﬁcients driven to zero during the
optimization, and so they are effectively removed from the model through automatic
relevance determination. This allows us to make a single training run in which we
start with a relatively large initial value of K, and allow surplus components to bepruned out of the model. The origins of the sparsity when optimizing with respect tohyperparameters is discussed in detail in the context of the relevance vector machine.

10.2.5 Induced factorizations
In deriving these variational update equations for the Gaussian mixture model,
we assumed a particular factorization of the variational posterior distribution givenby (10.42). However, the optimal solutions for the various factors exhibit additional
factorizations. In particular, the solution for q  (µ, Λ) is given by the product of an
independent distribution q  (µk , Λk ) over each of the components k of the mixture,
whereas the variational posterior distribution q  (Z) over the latent variables, givenby (10.48), factorizes into an independent distribution q  (zn ) for each observation n(note that it does not further factorize with respect to k because, for each value of n,
the znk are constrained to sum to one over k). These additional factorizations are aconsequence of the interaction between the assumed factorization and the conditional
independence properties of the true distribution, as characterized by the directed
graph in Figure 10.5.
We shall refer to these additional factorizations as induced factorizations because they arise from an interaction between the factorization assumed in the variational posterior distribution and the conditional independence properties of the true
joint distribution. In a numerical implementation of the variational approach it isimportant to take account of such additional factorizations. For instance, it would
be very inefﬁcient to maintain a full precision matrix for the Gaussian distribution
over a set of variables if the optimal form for that distribution always had a diagonal precision matrix (corresponding to a factorization with respect to the individual
variables described by that Gaussian).
Such induced factorizations can easily be detected using a simple graphical test
based on d-separation as follows. We partition the latent variables into three disjoint
groups A, B, C and then let us suppose that we are assuming a factorization between
C and the remaining latent variables, so thatq(A, B, C) = q(A, B)q(C).

(10.84)Using the general result (10.9), together with the product rule for probabilities, we
see that the optimal solution for q(A, B) is given byln q  (A, B) = EC [ln p(X, A, B, C)] + const= EC [ln p(A, B|X, C)] + const.

(10.85)

We now ask whether this resulting solution will factorize between A and B, inother words whether q  (A, B) = q  (A)q  (B). This will happen if, and only if,
ln p(A, B|X, C) = ln p(A|X, C) + ln p(B|X, C), that is, if the conditional independence relation
A ⊥⊥ B | X, C
(10.86)

486

10. APPROXIMATE INFERENCE
is satisﬁed. We can test to see if this relation does hold, for any choice of A and B
by making use of the d-separation criterion.
To illustrate this, consider again the Bayesian mixture of Gaussians represented
by the directed graph in Figure 10.5, in which we are assuming a variational factorization given by (10.42). We can see immediately that the variational posterior
distribution over the parameters must factorize between π and the remaining parameters µ and Λ because all paths connecting π to either µ or Λ must pass throughone of the nodes zn all of which are in the conditioning set for our conditional independence test and all of which are head-to-tail with respect to such paths.

10.3. Variational Linear Regression

Exercise 10.26

As a second illustration of variational inference, we return to the Bayesian linear
regression model of Section 3.3. In the evidence framework, we approximated theintegration over α and β by making point estimates obtained by maximizing the log
marginal likelihood. A fully Bayesian approach would integrate over the hyperparameters as well as over the parameters. Although exact integration is intractable,
we can use variational methods to ﬁnd a tractable approximation. In order to simplify the discussion, we shall suppose that the noise precision parameter β is known,
and is ﬁxed to its true value, although the framework is easily extended to include
the distribution over β. For the linear regression model, the variational treatment
will turn out to be equivalent to the evidence framework. Nevertheless, it provides agood exercise in the use of variational methods and will also lay the foundation forvariational treatment of Bayesian logistic regression in Section 10.6.
Recall that the likelihood function for w, and the prior over w, are given byp(t|w) =N

N (tn |wT φn , β −1 )(10.87)n=1p(w|α) = N (w|0, α−1 I)(10.88)where φn = φ(xn ). We now introduce a prior distribution over α. From our discussion in Section 2.3.6, we know that the conjugate prior for the precision of a
Gaussian is given by a gamma distribution, and so we choosep(α) = Gam(α|a0 , b0 )(10.89)where Gam(·|·, ·) is deﬁned by (B.26). Thus the joint distribution of all the variables
is given byp(t, w, α) = p(t|w)p(w|α)p(α).
(10.90)This can be represented as a directed graphical model as shown in Figure 10.8.

10.3.1 Variational distribution
Our ﬁrst goal is to ﬁnd an approximation to the posterior distribution p(w, α|t).
To do this, we employ the variational framework of Section 10.1, with a variational

10.3. Variational Linear Regression
Figure 10.8487
αProbabilistic graphical model representing the joint distribution (10.90) for the Bayesian linear regression
model.

φn
w

β
tn

N

posterior distribution given by the factorized expressionq(w, α) = q(w)q(α).

(10.91)

We can ﬁnd re-estimation equations for the factors in this distribution by making useof the general result (10.9). Recall that for each factor, we take the log of the joint
distribution over all variables and then average with respect to those variables not inthat factor. Consider ﬁrst the distribution over α. Keeping only terms that have afunctional dependence on α, we haveln q  (α) = ln p(α) + Ew [ln p(w|α)] + const
Mα(10.92)ln α − E[wT w] + const.
= (a0 − 1) ln α − b0 α +2
2
We recognize this as the log of a gamma distribution, and so identifying the coefﬁcients of α and ln α we obtainq  (α) = Gam(α|aN , bN )(10.93)

whereM(10.94)2
1(10.95)bN = b0 + E[wT w].
2
Similarly, we can ﬁnd the variational re-estimation equation for the posterior
distribution over w. Again, using the general result (10.9), and keeping only thoseterms that have a functional dependence on w, we haveaN

= a0 +

ln q  (w) = ln p(t|w) + Eα [ln p(w|α)] + const
β T
1{w φn − tn }2 − E[α]wT w + const
2
2(10.96)N= −(10.97)n=1

1= − wT E[α]I + βΦT Φ w + βwT ΦT t + const.
2(10.98)Because this is a quadratic form, the distribution q  (w) is Gaussian, and so we cancomplete the square in the usual way to identify the mean and covariance, givingq  (w) = N (w|mN , SN )(10.99)488

10. APPROXIMATE INFERENCE
where
mN
SN

= βSN ΦT t=

(10.100)

E[α]I + βΦ Φ
T

−1

.

(10.101)

Note the close similarity to the posterior distribution (3.52) obtained when α was
treated as a ﬁxed parameter. The difference is that here α is replaced by its expectation E[α] under the variational distribution. Indeed, we have chosen to use the same
notation for the covariance matrix SN in both cases.
Using the standard results (B.27), (B.38), and (B.39), we can obtain the required
moments as follows
E[α] = aN /bN
E[wwT ] = mN mT
N + SN .

(10.102)(10.103)

The evaluation of the variational posterior distribution begins by initializing the parameters of one of the distributions q(w) or q(α), and then alternately re-estimatesthese factors in turn until a suitable convergence criterion is satisﬁed (usually speciﬁed in terms of the lower bound to be discussed shortly).
It is instructive to relate the variational solution to that found using the evidence
framework in Section 3.5. To do this consider the case a0 = b0 = 0, corresponding
to the limit of an inﬁnitely broad prior over α. The mean of the variational posterior
distribution q(α) is then given byE[α] =

M
M/2
aN
= T
=
.
bN
E[wT w]/2
mN mN + Tr(SN )(10.104)

Comparison with (9.63) shows that in the case of this particularly simple model,
the variational approach gives precisely the same expression as that obtained by
maximizing the evidence function using EM except that the point estimate for α
is replaced by its expected value. Because the distribution q(w) depends on q(α)
only through the expectation E[α], we see that the two approaches will give identicalresults for the case of an inﬁnitely broad prior.

10.3.2 Predictive distribution
The predictive distribution over t, given a new input x, is easily evaluated forthis model using the Gaussian variational posterior for the parametersp(t|x, w)p(w|t) dw
p(t|x, t) =
p(t|x, w)q(w) dw
N (t|wT φ(x), β −1 )N (w|mN , SN ) dw=
2
= N (t|mT
N φ(x), σ (x))(10.105)

10.3. Variational Linear Regression

489

where we have evaluated the integral by making use of the result (2.115) for the
linear-Gaussian model. Here the input-dependent variance is given byσ 2 (x) =1+ φ(x)T SN φ(x).
β(10.106)

Note that this takes the same form as the result (3.59) obtained with ﬁxed α exceptthat now the expected value E[α] appears in the deﬁnition of SN .

10.3.3 Lower boundAnother quantity of importance is the lower bound L deﬁned byL(q) = E[ln p(w, α, t)] − E[ln q(w, α)]= Ew [ln p(t|w)] + Ew,α [ln p(w|α)] + Eα [ln p(α)]−Eα [ln q(w)]w − E[ln q(α)].
Exercise 10.27(10.107)

Evaluation of the various terms is straightforward, making use of results obtained inprevious chapters, and gives
 
β
N
β
T
− tT t + βmTln
E[ln p(t|w)]w =
NΦ t
2
2π
2
β 
− Tr ΦT Φ(mN mT(10.108)
N + SN )2
M
M(ψ(aN ) − ln bN )E[ln p(w|α)]w,α = − ln(2π) +
2
2

aN
−
mT
(10.109)
N mN + Tr(SN )2bN
E[ln p(α)]α = a0 ln b0 + (a0 − 1) [ψ(aN ) − ln bN ]aN
−b0
− ln Γ(aN )(10.110)bN
1
M
ln |SN | +[1 + ln(2π)](10.111)−E[ln q(w)]w =22−E[ln q(α)]α = ln Γ(aN ) − (aN − 1)ψ(aN ) − ln bN + aN . (10.112)
Figure 10.9 shows a plot of the lower bound L(q) versus the degree of a polynomial
model for a synthetic data set generated from a degree three polynomial. Here theprior parameters have been set to a0 = b0 = 0, corresponding to the noninformative
prior p(α) ∝ 1/α, which is uniform over ln α as discussed in Section 2.3.6. As
we saw in Section 10.1, the quantity L represents lower bound on the log marginal
likelihood p(t|M ) for the model. If we assign equal prior probabilities p(M ) to thedifferent values of M , then we can interpret L as an approximation to the posterior model probability p(M |t). Thus the variational framework assigns the highest
probability to the model with M = 3. This should be contrasted with the maximum
likelihood result, which assigns ever smaller residual error to models of increasing
complexity until the residual error is driven to zero, causing maximum likelihood tofavour severely over-ﬁtted models.

490

10. APPROXIMATE INFERENCE
Figure 10.9

Plot of the lower bound L versus the order M of the polynomial, for a polynomial model, in
which a set of 10 data points isgenerated from a polynomial with
M = 3 sampled over the interval (−5, 5) with additive Gaussian
noise of variance 0.09. The valueof the bound gives the log probability of the model, and we seethat the value of the bound peaks
at M = 3, corresponding to thetrue model from which the data
set was generated.

1

3

5

7

9

10.4. Exponential Family Distributions
In Chapter 2, we discussed the important role played by the exponential family of
distributions and their conjugate priors. For many of the models discussed in thisbook, the complete-data likelihood is drawn from the exponential family. However,in general this will not be the case for the marginal likelihood function for the observed data. For example, in a mixture of Gaussians, the joint distribution of observations xn and corresponding hidden variables zn is a member of the exponential
family, whereas the marginal distribution of xn is a mixture of Gaussians and hence
is not.
Up to now we have grouped the variables in the model into observed variables
and hidden variables. We now make a further distinction between latent variables,
denoted Z, and parameters, denoted θ, where parameters are intensive (ﬁxed in number independent of the size of the data set), whereas latent variables are extensive(scale in number with the size of the data set). For example, in a Gaussian mixture
model, the indicator variables zkn (which specify which component k is responsible
for generating data point xn ) represent the latent variables, whereas the means µk ,precisions Λk and mixing proportions πk represent the parameters.
Consider the case of independent identically distributed data. We denote thedata values by X = {xn }, where n = 1, . . . N , with corresponding latent variables
Z = {zn }. Now suppose that the joint distribution of observed and latent variables
is a member of the exponential family, parameterized by natural parameters η so thatNh(xn , zn )g(η) exp η T u(xn , zn ) .

(10.113)

We shall also use a conjugate prior for η, which can be written asp(η|ν0 , v0 ) = f (ν0 , χ0 )g(η)ν0 exp νo η T χ0 .

(10.114)p(X, Z|η) =n=1

Recall that the conjugate prior distribution can be interpreted as a prior number ν0of observations all having the value χ0 for the u vector. Now consider a variational

10.4. Exponential Family Distributions

491

distribution that factorizes between the latent variables and the parameters, so thatq(Z, η) = q(Z)q(η). Using the general result (10.9), we can solve for the two
factors as follows
ln q  (Z) = Eη [ln p(X, Z|η)] + const
=

N




ln h(xn , zn ) + E[η T ]u(xn , zn ) + const. (10.115)

n=1Section 10.2.5

Thus we see that this decomposes into a sum of independent terms, one for each


value
 of n, and hence the solution for q (Z) will factorize over n so that q (Z) =n q (zn ). This is an example of an induced factorization. Taking the exponentialof both sides, we have
(10.116)q  (zn ) = h(xn , zn )g (E[η]) exp E[η T ]u(xn , zn )where the normalization coefﬁcient has been re-instated by comparison with thestandard form for the exponential family.
Similarly, for the variational distribution over the parameters, we haveln q  (η) = ln p(η|ν0 , χ0 ) + EZ [ln p(X, Z|η)] + const= ν0 ln g(η) + η T χ0 +N



(10.117)ln g(η) + η T Ezn [u(xn , zn )] + const. (10.118)

n=1

Again, taking the exponential of both sides, and re-instating the normalization coefﬁcient by inspection, we have
(10.119)q  (η) = f (νN , χN )g(η)νN exp η T χN
where we have deﬁnedνNχN= ν0 + NN= χ0 +
Ezn [u(xn , zn )].

(10.120)(10.121)n=1

Note that the solutions for q  (zn ) and q  (η) are coupled, and so we solve them iteratively in a two-stage procedure. In the variational E step, we evaluate the expected
sufﬁcient statistics E[u(xn , zn )] using the current posterior distribution q(zn ) overthe latent variables and use this to compute a revised posterior distribution q(η) overthe parameters. Then in the subsequent variational M step, we use this revised parameter posterior distribution to ﬁnd the expected natural parameters E[η T ], which
gives rise to a revised variational distribution over the latent variables.

10.4.1 Variational message passing
We have illustrated the application of variational methods by considering a speciﬁc model, the Bayesian mixture of Gaussians, in some detail. This model can be

492

10. APPROXIMATE INFERENCE
described by the directed graph shown in Figure 10.5. Here we consider more generally the use of variational methods for models described by directed graphs andderive a number of widely applicable results.
The joint distribution corresponding to a directed graph can be written using thedecompositionp(x) =p(xi |pai )(10.122)iwhere xi denotes the variable(s) associated with node i, and pai denotes the parent
set corresponding to node i. Note that xi may be a latent variable or it may belong
to the set of observed variables. Now consider a variational approximation in which
the distribution q(x) is assumed to factorize with respect to the xi so thatq(x) =qi (xi ).
(10.123)i

Note that for observed nodes, there is no factor q(xi ) in the variational distribution.
We now substitute (10.122) into our general result (10.9) to give
ln qj (xj ) = Ei=jln p(xi |pai ) + const.
(10.124)i

Any terms on the right-hand side that do not depend on xj can be absorbed into
the additive constant. In fact, the only terms that do depend on xj are the conditional distribution for xj given by p(xj |paj ) together with any other conditional
distributions that have xj in the conditioning set. By deﬁnition, these conditional
distributions correspond to the children of node j, and they therefore also depend on
the co-parents of the child nodes, i.e., the other parents of the child nodes besides
node xj itself. We see that the set of all nodes on which q  (xj ) depends corresponds
to the Markov blanket of node xj , as illustrated in Figure 8.26. Thus the updateof the factors in the variational posterior distribution represents a local calculationon the graph. This makes possible the construction of general purpose software forvariational inference in which the form of the model does not need to be speciﬁed inadvance (Bishop et al., 2003).
If we now specialize to the case of a model in which all of the conditional distributions have a conjugate-exponential structure, then the variational update procedure can be cast in terms of a local message passing algorithm (Winn and Bishop,
2005). In particular, the distribution associated with a particular node can be updated
once that node has received messages from all of its parents and all of its children.
This in turn requires that the children have already received messages from their coparents. The evaluation of the lower bound can also be simpliﬁed because many of
the required quantities are already evaluated as part of the message passing scheme.
This distributed message passing formulation has good scaling properties and is well
suited to large networks.

493

10.5. Local Variational Methods

10.5. Local Variational Methods

Section 1.6.1

The variational framework discussed in Sections 10.1 and 10.2 can be considered a‘global’ method in the sense that it directly seeks an approximation to the full posterior distribution over all random variables. An alternative ‘local’ approach involves
ﬁnding bounds on functions over individual variables or groups of variables withina model. For instance, we might seek a bound on a conditional distribution p(y|x),which is itself just one factor in a much larger probabilistic model speciﬁed by adirected graph. The purpose of introducing the bound of course is to simplify the
resulting distribution. This local approximation can be applied to multiple variables
in turn until a tractable approximation is obtained, and in Section 10.6.1 we shallgive a practical example of this approach in the context of logistic regression. Here
we focus on developing the bounds themselves.
We have already seen in our discussion of the Kullback-Leibler divergence thatthe convexity of the logarithm function played a key role in developing the lower
bound in the global variational approach. We have deﬁned a (strictly) convex function as one for which every chord lies above the function. Convexity also plays acentral role in the local variational framework. Note that our discussion will apply equally to concave functions with ‘min’ and ‘max’ interchanged and with lower
bounds replaced by upper bounds.
Let us begin by considering a simple example, namely the function f (x) =exp(−x), which is a convex function of x, and which is shown in the left-hand plotof Figure 10.10. Our goal is to approximate f (x) by a simpler function, in particulara linear function of x. From Figure 10.10, we see that this linear function will be alower bound on f (x) if it corresponds to a tangent. We can obtain the tangent line
y(x) at a speciﬁc value of x, say x = ξ, by making a ﬁrst order Taylor expansion
y(x) = f (ξ) + f  (ξ)(x − ξ)(10.125)so that y(x)  f (x) with equality when x = ξ. For our example function f (x) =Figure 10.10 In the left-hand ﬁgure the red curve shows the function
exp(−x), and the blue line shows
the tangent at x = ξ deﬁned by(10.125) with ξ = 1. This line hasslope λ = f  (ξ) = − exp(−ξ). Notethat any other tangent line, for example the ones shown in green, will
have a smaller value of y at x =ξ. The right-hand ﬁgure shows thecorresponding plot of the function
λξ − g(λ), where g(λ) is given by(10.131), versus λ for ξ = 1, in
which the maximum corresponds to
λ = − exp(−ξ) = −1/e.

1

0.4

0.5

0.2

0

0

ξ

1.5

x

3

0
−1

λξ − g(λ)

−0.5

λ

0

494

10. APPROXIMATE INFERENCE
y

y
f (x)

f (x)
−g(λ)
x

x

λx
λx − g(λ)
Figure 10.11 In the left-hand plot the red curve shows a convex function f (x), and the blue line represents thelinear function λx, which is a lower bound on f (x) because f (x) > λx for all x. For the given value of slope λ thecontact point of the tangent line having the same slope is found by minimizing with respect to x the discrepancy(shown by the green dashed lines) given by f (x) − λx. This deﬁnes the dual function g(λ), which corresponds
to the (negative of the) intercept of the tangent line having slope λ.

exp(−x), we therefore obtain the tangent line in the formy(x) = exp(−ξ) − exp(−ξ)(x − ξ)(10.126)

which is a linear function parameterized by ξ. For consistency with subsequentdiscussion, let us deﬁne λ = − exp(−ξ) so thaty(x, λ) = λx − λ + λ ln(−λ).

(10.127)

Different values of λ correspond to different tangent lines, and because all such lines
are lower bounds on the function, we have f (x)  y(x, λ). Thus we can write thefunction in the formf (x) = max {λx − λ + λ ln(−λ)} .
λ(10.128)

We have succeeded in approximating the convex function f (x) by a simpler, linear function y(x, λ). The price we have paid is that we have introduced a variationalparameter λ, and to obtain the tightest bound we must optimize with respect to λ.
We can formulate this approach more generally using the framework of convex
duality (Rockafellar, 1972; Jordan et al., 1999). Consider the illustration of a convex
function f (x) shown in the left-hand plot in Figure 10.11. In this example, thefunction λx is a lower bound on f (x) but it is not the best lower bound that canbe achieved by a linear function having slope λ, because the tightest bound is givenby the tangent line. Let us write the equation of the tangent line, having slope λ as
λx − g(λ) where the (negative) intercept g(λ) clearly depends on the slope λ of thetangent. To determine the intercept, we note that the line must be moved vertically byan amount equal to the smallest vertical distance between the line and the function,
as shown in Figure 10.11. Thus
g(λ) = − min {f (x) − λx}x= max {λx − f (x)} .
x(10.129)

10.5. Local Variational Methods

495

Now, instead of ﬁxing λ and varying x, we can consider a particular x and thenadjust λ until the tangent plane is tangent at that particular x. Because the y valueof the tangent line at a particular x is maximized when that value coincides with its
contact point, we havef (x) = max {λx − g(λ)} .
(10.130)
λWe see that the functions f (x) and g(λ) play a dual role, and are related through(10.129) and (10.130).
Let us apply these duality relations to our simple example f (x) = exp(−x).
From (10.129) we see that the maximizing value of x is given by ξ = − ln(−λ), andback-substituting we obtain the conjugate function g(λ) in the formg(λ) = λ − λ ln(−λ)(10.131)

as obtained previously. The function λξ − g(λ) is shown, for ξ = 1 in the right-handplot in Figure 10.10. As a check, we can substitute (10.131) into (10.130), which
gives the maximizing value of λ = − exp(−x), and back-substituting then recovers
the original function f (x) = exp(−x).
For concave functions, we can follow a similar argument to obtain upper bounds,
in which max’ is replaced with ‘min’, so thatf (x) = min {λx − g(λ)}(10.132)g(λ) = min {λx − f (x)} .

(10.133)

λx

If the function of interest is not convex (or concave), then we cannot directly
apply the method above to obtain a bound. However, we can ﬁrst seek invertible
transformations either of the function or of its argument which change it into a convex form. We then calculate the conjugate function and then transform back to theoriginal variables.
An important example, which arises frequently in pattern recognition, is thelogistic sigmoid function deﬁned by
σ(x) =

Exercise 10.30

1.
1 + e−x

As it stands this function is neither convex nor concave. However, if we take thelogarithm we obtain a function which is concave, as is easily veriﬁed by ﬁnding thesecond derivative. From (10.133) the corresponding conjugate function then takes
the formg(λ) = min {λx − f (x)} = −λ ln λ − (1 − λ) ln(1 − λ)xAppendix B(10.134)(10.135)

which we recognize as the binary entropy function for a variable whose probabilityof having the value 1 is λ. Using (10.132), we then obtain an upper bound on the log
sigmoidln σ(x)  λx − g(λ)(10.136)496

10. APPROXIMATE INFERENCE
1

1

ξ = 2.5

λ = 0.2

0.5

0.5
λ = 0.7

0
−6

0

0
−6

6

−ξ

0

ξ

6

Figure 10.12 The left-hand plot shows the logistic sigmoid function σ(x) deﬁned by (10.134) in red, together
with two examples of the exponential upper bound (10.137) shown in blue. The right-hand plot shows the logistic
sigmoid again in red together with the Gaussian lower bound (10.144) shown in blue. Here the parameterξ = 2.5, and the bound is exact at x = ξ and x = −ξ, denoted by the dashed green lines.

and taking the exponential, we obtain an upper bound on the logistic sigmoid itselfof the form
σ(x)  exp(λx − g(λ))(10.137)
which is plotted for two values of λ on the left-hand plot in Figure 10.12.
We can also obtain a lower bound on the sigmoid having the functional form ofa Gaussian. To do this, we follow Jaakkola and Jordan (2000) and make transformations both of the input variable and of the function itself. First we take the log of thelogistic function and then decompose it so thatln σ(x) = − ln(1 + e−x ) = − ln e−x/2 (ex/2 + e−x/2 )= x/2 − ln(ex/2 + e−x/2 ).
Exercise 10.31(10.138)

We now note that the function f (x) = − ln(ex/2 + e−x/2 ) is a convex function of
the variable x2 , as can again be veriﬁed by ﬁnding the second derivative. This leads
to a lower bound on f (x), which is a linear function of x2 whose conjugate function
is given by

√ 
2
g(λ) = max
λx
−
f
x2
.
(10.139)2x

The stationarity condition leads to
0=λ−

x
1
dx d
.
f
(x)
=
λ
+
tanh
dx2 dx4x2(10.140)

If we denote this value of x, corresponding to the contact point of the tangent line
for this particular value of λ, by ξ, then we have
 


1
ξ
1
1
λ(ξ) = − tanh=−σ(ξ) −.
(10.141)4ξ
22ξ
2

10.5. Local Variational Methods

497

Instead of thinking of λ as the variational parameter, we can let ξ play this role as
this leads to simpler expressions for the conjugate function, which is then given byg(λ) = λ(ξ)ξ 2 − f (ξ) = λ(ξ)ξ 2 + ln(eξ/2 + e−ξ/2 ).

(10.142)Hence the bound on f (x) can be written as

Section 4.5

Section 4.3

f (x)  λx2 − g(λ) = λx2 − λξ 2 − ln(eξ/2 + e−ξ/2 ).

(10.143)

The bound on the sigmoid then becomesσ(x)  σ(ξ) exp (x − ξ)/2 − λ(ξ)(x2 − ξ 2 )(10.144)

where λ(ξ) is deﬁned by (10.141). This bound is illustrated in the right-hand plot of
Figure 10.12. We see that the bound has the form of the exponential of a quadratic
function of x, which will prove useful when we seek Gaussian representations ofposterior distributions deﬁned through logistic sigmoid functions.
The logistic sigmoid arises frequently in probabilistic models over binary variables because it is the function that transforms a log odds ratio into a posterior probability. The corresponding transformation for a multiclass distribution is given by
the softmax function. Unfortunately, the lower bound derived here for the logistic
sigmoid does not directly extend to the softmax. Gibbs (1997) proposes a method
for constructing a Gaussian distribution that is conjectured to be a bound (although
no rigorous proof is given), which may be used to apply local variational methods tomulticlass problems.
We shall see an example of the use of local variational bounds in Sections 10.6.1.
For the moment, however, it is instructive to consider in general terms how thesebounds can be used. Suppose we wish to evaluate an integral of the form
(10.145)
I = σ(a)p(a) da
where σ(a) is the logistic sigmoid, and p(a) is a Gaussian probability density. Suchintegrals arise in Bayesian models when, for instance, we wish to evaluate the predictive distribution, in which case p(a) represents a posterior parameter distribution.
Because the integral is intractable, we employ the variational bound (10.144), which
we write in the form σ(a)  f (a, ξ) where ξ is a variational parameter. The integral now becomes the product of two exponential-quadratic functions and so can be
integrated analytically to give a bound on I
(10.146)
I  f (a, ξ)p(a) da = F (ξ).
We now have the freedom to choose the variational parameter ξ, which we do byﬁnding the value ξ  that maximizes the function F (ξ). The resulting value F (ξ  )represents the tightest bound within this family of bounds and can be used as an
approximation to I. This optimized bound, however, will in general not be exact.

498

10. APPROXIMATE INFERENCE
Although the bound σ(a)  f (a, ξ) on the logistic sigmoid can be optimized exactly,
the required choice for ξ depends on the value of a, so that the bound is exact for one
value of a only. Because the quantity F (ξ) is obtained by integrating over all valuesof a, the value of ξ  represents a compromise, weighted by the distribution p(a).

10.6. Variational Logistic Regression
We now illustrate the use of local variational methods by returning to the Bayesianlogistic regression model studied in Section 4.5. There we focussed on the use of
the Laplace approximation, while here we consider a variational treatment based on
the approach of Jaakkola and Jordan (2000). Like the Laplace method, this also
leads to a Gaussian approximation to the posterior distribution. However, the greater
ﬂexibility of the variational approximation leads to improved accuracy comparedto the Laplace method. Furthermore (unlike the Laplace method), the variational
approach is optimizing a well deﬁned objective function given by a rigourous boundon the model evidence. Logistic regression has also been treated by Dybowski andRoberts (2005) from a Bayesian perspective using Monte Carlo sampling techniques.

10.6.1 Variational posterior distribution
Here we shall make use of a variational approximation based on the local bounds
introduced in Section 10.5. This allows the likelihood function for logistic regression, which is governed by the logistic sigmoid, to be approximated by the exponential of a quadratic form. It is therefore again convenient to choose a conjugate
Gaussian prior of the form (4.140). For the moment, we shall treat the hyperparameters m0 and S0 as ﬁxed constants. In Section 10.6.3, we shall demonstrate how thevariational formalism can be extended to the case where there are unknown hyperparameters whose values are to be inferred from the data.
In the variational framework, we seek to maximize a lower bound on the marginal
likelihood. For the Bayesian logistic regression model, the marginal likelihood takes
the form 
Np(tn |w) p(w) dw.
(10.147)p(t) = p(t|w)p(w) dw =n=1

We ﬁrst note that the conditional distribution for t can be written as1−tp(t|w) = σ(a)t {1 − σ(a)}

t 
1−t
1
1
=
1−
1 + e−a1 + e−a−ae= eat= eat σ(−a)
1 + e−a(10.148)

where a = wT φ. In order to obtain a lower bound on p(t), we make use of thevariational lower bound on the logistic sigmoid function given by (10.144), which10.6. Variational Logistic Regression499

we reproduce here for convenience
σ(z)  σ(ξ) exp (z − ξ)/2 − λ(ξ)(z 2 − ξ 2 )11
σ(ξ) −.
λ(ξ) =2ξ
2

where(10.149)(10.150)

We can therefore write
p(t|w) = eat σ(−a)  eat σ(ξ) exp −(a + ξ)/2 − λ(ξ)(a2 − ξ 2 ) .

(10.151)

Note that because this bound is applied to each of the terms in the likelihood function
separately, there is a variational parameter ξn corresponding to each training set
observation (φn , tn ). Using a = wT φ, and multiplying by the prior distribution, we
obtain the following bound on the joint distribution of t and wp(t, w) = p(t|w)p(w)  h(w, ξ)p(w)(10.152)

where ξ denotes the set {ξn } of variational parameters, andh(w, ξ) =Nσ(ξn ) exp wT φn tn − (wT φn + ξn )/2

n=1
− λ(ξn )([wT φn ]2 − ξn2 ) .

(10.153)

Evaluation of the exact posterior distribution would require normalization of the lefthand side of this inequality. Because this is intractable, we work instead with theright-hand side. Note that the function on the right-hand side cannot be interpreted
as a probability density because it is not normalized. Once it is normalized to give avariational posterior distribution q(w), however, it no longer represents a bound.
Because the logarithm function is monotonically increasing, the inequality A 
B implies ln A  ln B. This gives a lower bound on the log of the joint distributionof t and w of the form
ln {p(t|w)p(w)}  ln p(w) +Nln σ(ξn ) + wT φn tn

n=1− (w φn + ξn )/2 − λ(ξn )([wT φn ]2 − ξn2 ) .
T(10.154)

Substituting for the prior p(w), the right-hand side of this inequality becomes, as afunction of w
11
− (w − m0 )T S−
0 (w − m0 )
2
N

 T

w φn (tn − 1/2) − λ(ξn )wT (φn φT+n )w + const.
n=1(10.155)

500

10. APPROXIMATE INFERENCE
This is a quadratic function of w, and so we can obtain the corresponding variational
approximation to the posterior distribution by identifying the linear and quadratic
terms in w, giving a Gaussian variational posterior of the formq(w) = N (w|mN , SN )
where


mN

= SN

1
S−
0 m0 +

N

(10.156)(tn − 1/2)φn(10.157)

n=1
1
S−
N

Exercise 10.32

1
= S−
0 +2

N


λ(ξn )φn φTn.

(10.158)

n=1

As with the Laplace framework, we have again obtained a Gaussian approximation
to the posterior distribution. However, the additional ﬂexibility provided by the variational parameters {ξn } leads to improved accuracy in the approximation (Jaakkola
and Jordan, 2000).
Here we have considered a batch learning context in which all of the training
data is available at once. However, Bayesian methods are intrinsically well suited
to sequential learning in which the data points are processed one at a time and thendiscarded. The formulation of this variational approach for the sequential case isstraightforward.
Note that the bound given by (10.149) applies only to the two-class problem andso this approach does not directly generalize to classiﬁcation problems with K > 2classes. An alternative bound for the multiclass case has been explored by Gibbs(1997).

10.6.2 Optimizing the variational parameters
We now have a normalized Gaussian approximation to the posterior distribution,
which we shall use shortly to evaluate the predictive distribution for new data points.
First, however, we need to determine the variational parameters {ξn } by maximizing
the lower bound on the marginal likelihood.
To do this, we substitute the inequality (10.152) back into the marginal likelihood to giveln p(t) = ln p(t|w)p(w) dw  ln h(w, ξ)p(w) dw = L(ξ).
(10.159)
As with the optimization of the hyperparameter α in the linear regression model of
Section 3.5, there are two approaches to determining the ξn . In the ﬁrst approach, we
recognize that the function L(ξ) is deﬁned by an integration over w and so we canview w as a latent variable and invoke the EM algorithm. In the second approach,
we integrate over w analytically and then perform a direct maximization over ξ. Let
us begin by considering the EM approach.
The EM algorithm starts by choosing some initial values for the parameters{ξn }, which we denote collectively by ξ old . In the E step of the EM algorithm,

10.6. Variational Logistic Regression

501

we then use these parameter values to ﬁnd the posterior distribution over w, which
is given by (10.156). In the M step, we then maximize the expected complete-data
log likelihood which is given byQ(ξ, ξ old ) = E [ln h(w, ξ)p(w)](10.160)

where the expectation is taken with respect to the posterior distribution q(w) evaluated using ξ old . Noting that p(w) does not depend on ξ, and substituting for h(w, ξ)
we obtain
Q(ξ, ξ

old

)=

N




T
2
ln σ(ξn ) − ξn /2 − λ(ξn )(φTn E[ww ]φn − ξn ) + const

n=1(10.161)
where ‘const’ denotes terms that are independent of ξ. We now set the derivative withrespect to ξn equal to zero. A few lines of algebra, making use of the deﬁnitions ofσ(ξ) and λ(ξ), then givesT2
0 = λ (ξn )(φTn E[ww ]φn − ξn ).

Exercise 10.33

We now note that λ (ξ) is a monotonic function of ξ for ξ  0, and that we canrestrict attention to nonnegative values of ξ without loss of generality due to thesymmetry of the bound around ξ = 0. Thus λ (ξ) = 0, and hence we obtain thefollowing re-estimation equations
TT
T(ξnnew )2 = φTn E[ww ]φn = φn SN + mN mN φnExercise 10.34

Exercise 10.35(10.162)(10.163)

where we have used (10.156).
Let us summarize the EM algorithm for ﬁnding the variational posterior distribution. We ﬁrst initialize the variational parameters ξ old . In the E step, we evaluate
the posterior distribution over w given by (10.156), in which the mean and covariance are deﬁned by (10.157) and (10.158). In the M step, we then use this variational
posterior to compute a new value for ξ given by (10.163). The E and M steps arerepeated until a suitable convergence criterion is satisﬁed, which in practice typicallyrequires only a few iterations.
An alternative approach to obtaining re-estimation equations for ξ is to notethat in the integral over w in the deﬁnition (10.159) of the lower bound L(ξ), theintegrand has a Gaussian-like form and so the integral can be evaluated analytically.
Having evaluated the integral, we can then differentiate with respect to ξn . It turns
out that this gives rise to exactly the same re-estimation equations as does the EM
approach given by (10.163).
As we have emphasized already, in the application of variational methods it isuseful to be able to evaluate the lower bound L(ξ) given by (10.159). The integration
over w can be performed analytically by noting that p(w) is Gaussian and h(w, ξ)is the exponential of a quadratic function of w. Thus, by completing the squareand making use of the standard result for the normalization coefﬁcient of a Gaussian
distribution, we can obtain a closed form solution which takes the form

502

10. APPROXIMATE INFERENCE

4

4

2

2

0

0
0.99

−2

0.25
0.75

6

0.0
1

6

−2

−4
−6
−4

−4
−2

0

2

4

−6
−4

−2

0

2

4

Figure 10.13 Illustration of the Bayesian approach to logistic regression for a simple linearly separable data
set. The plot on the left shows the predictive distribution obtained using variational inference. We see thatthe decision boundary lies roughly mid way between the clusters of data points, and that the contours of thepredictive distribution splay out away from the data reﬂecting the greater uncertainty in the classiﬁcation of suchregions. The plot on the right shows the decision boundaries corresponding to ﬁve samples of the parameter
vector w drawn from the posterior distribution p(w|t).

L(ξ) =

1 |SN | 1 T −1
1
S−1 m0ln− m N SN m N + m T
2
|S0 |
2
2 0 0
+

N

n=1

1
ln σ(ξn ) − ξn − λ(ξn )ξn2 .
2(10.164)

This variational framework can also be applied to situations in which the datais arriving sequentially (Jaakkola and Jordan, 2000). In this case we maintain a
Gaussian posterior distribution over w, which is initialized using the prior p(w). As
each data point arrives, the posterior is updated by making use of the bound (10.151)
and then normalized to give an updated posterior distribution.
The predictive distribution is obtained by marginalizing over the posterior distribution, and takes the same form as for the Laplace approximation discussed in
Section 4.5.2. Figure 10.13 shows the variational predictive distributions for a synthetic data set. This example provides interesting insights into the concept of ‘large
margin’, which was discussed in Section 7.1 and which has qualitatively similar behaviour to the Bayesian solution.

10.6.3 Inference of hyperparameters
So far, we have treated the hyperparameter α in the prior distribution as a known
constant. We now extend the Bayesian logistic regression model to allow the value of
this parameter to be inferred from the data set. This can be achieved by combining
the global and local variational approximations into a single framework, so as tomaintain a lower bound on the marginal likelihood at each stage. Such a combined
approach was adopted by Bishop and Svensén (2003) in the context of a Bayesiantreatment of the hierarchical mixture of experts model.

10.6. Variational Logistic Regression

503

Speciﬁcally, we consider once again a simple isotropic Gaussian prior distribution of the formp(w|α) = N (w|0, α−1 I).
(10.165)
Our analysis is readily extended to more general Gaussian priors, for instance if wewish to associate a different hyperparameter with different subsets of the parameters wj . As usual, we consider a conjugate hyperprior over α given by a gamma
distributionp(α) = Gam(α|a0 , b0 )(10.166)
governed by the constants a0 and b0 .
The marginal likelihood for this model now takes the formp(t) =
p(w, α, t) dw dα(10.167)

where the joint distribution is given byp(w, α, t) = p(t|w)p(w|α)p(α).

(10.168)

We are now faced with an analytically intractable integration over w and α, which
we shall tackle by using both the local and global variational approaches in the samemodel
To begin with, we introduce a variational distribution q(w, α), and then apply
the decomposition (10.2), which in this instance takes the formln p(t) = L(q) + KL(qp)(10.169)

where the lower bound L(q) and the Kullback-Leibler divergence KL(qp) are deﬁned byp(w, α, t)q(w, α) ln
dw dα(10.170)
L(q) =
q(w, α)p(w, α|t))KL(qp) = −q(w, α) ln
dw dα.
(10.171)q(w, α)At this point, the lower bound L(q) is still intractable due to the form of thelikelihood factor p(t|w). We therefore apply the local variational bound to each of
the logistic sigmoid factors as before. This allows us to use the inequality (10.152)
and place a lower bound on L(q), which will therefore also be a lower bound on thelog marginal likelihood ξ)
ln p(t)  L(q)  L(q,h(w, ξ)p(w|α)p(α)=
q(w, α) lnq(w, α)dw dα. (10.172)

Next we assume that the variational distribution factorizes between parameters andhyperparameters so thatq(w, α) = q(w)q(α).
(10.173)

504

10. APPROXIMATE INFERENCE
With this factorization we can appeal to the general result (10.9) to ﬁnd expressions
for the optimal factors. Consider ﬁrst the distribution q(w). Discarding terms thatare independent of w, we haveln q(w) = Eα [ln {h(w, ξ)p(w|α)p(α)}] + const= ln h(w, ξ) + Eα [ln p(w|α)] + const.
We now substitute for ln h(w, ξ) using (10.153), and for ln p(w|α) using (10.165),
givingE[α] Tln q(w) = −(tn − 1/2)wT φn − λ(ξn )wT φn φTw w+
n w + const.
2
N

n=1

We see that this is a quadratic function of w and so the solution for q(w) will be
Gaussian. Completing the square in the usual way, we obtainq(w) = N (w|µN , ΣN )(10.174)

where we have deﬁned
1
Σ−
N µN

=

N


(tn − 1/2)φn(10.175)

n=1
1
Σ−
N

= E[α]I + 2Nλ(ξn )φn φTn.

(10.176)n=1

Similarly, the optimal solution for the factor q(α) is obtained fromln q(α) = Ew [ln p(w|α)] + ln p(α) + const.
Substituting for ln p(w|α) using (10.165), and for ln p(α) using (10.166), we obtainln q(α) =α Mln α − E wT w + (a0 − 1) ln α − b0 α + const.
2
2

We recognize this as the log of a gamma distribution, and so we obtainq(α) = Gam(α|aN , bN ) =1
ab0 αa0 −1 e−b0 αΓ(a0 ) 0(10.177)

where
aN
bN

M
2

1
= b0 + Ew wT w .
2= a0 +(10.178)(10.179)

10.7. Expectation Propagation

505

We also need to optimize the variational parameters ξn , and this is also done by ξ). Omitting terms that are independent of ξ, andmaximizing the lower bound L(q,integrating over α, we haveL(q, ξ) = q(w) ln h(w, ξ) dw + const.
(10.180)
Note that this has precisely the same form as (10.159), and so we can again appeal
to our earlier result (10.163), which can be obtained by direct optimization of themarginal likelihood function, leading to re-estimation equations of the formT(ξnnew )2 = φTn ΣN + µN µN φn .

Appendix B(10.181)

We have obtained re-estimation equations for the three quantities q(w), q(α),
and ξ, and so after making suitable initializations, we can cycle through these quantities, updating each in turn. The required moments are given byaN(10.182)
E [α] =bN(10.183)
E w T w = ΣN + µT
N µN .

10.7. Expectation Propagation
We conclude this chapter by discussing an alternative form of deterministic approximate inference, known as expectation propagation or EP (Minka, 2001a; Minka,
2001b). As with the variational Bayes methods discussed so far, this too is basedon the minimization of a Kullback-Leibler divergence but now of the reverse form,
which gives the approximation rather different properties.
Consider for a moment the problem of minimizing KL(pq) with respect to q(z)when p(z) is a ﬁxed distribution and q(z) is a member of the exponential family andso, from (2.194), can be written in the form
(10.184)q(z) = h(z)g(η) exp η T u(z) .
As a function of η, the Kullback-Leibler divergence then becomesKL(pq) = − ln g(η) − η T Ep(z) [u(z)] + const(10.185)

where the constant terms are independent of the natural parameters η. We can minimize KL(pq) within this family of distributions by setting the gradient with respectto η to zero, giving(10.186)−∇ ln g(η) = Ep(z) [u(z)].
However, we have already seen in (2.226) that the negative gradient of ln g(η) is
given by the expectation of u(z) under the distribution q(z). Equating these two
results, we obtain
Eq(z) [u(z)] = Ep(z) [u(z)].
(10.187)

506

10. APPROXIMATE INFERENCE
We see that the optimum solution simply corresponds to matching the expected sufﬁcient statistics. So, for instance, if q(z) is a Gaussian N (z|µ, Σ) then we minimize
the Kullback-Leibler divergence by setting the mean µ of q(z) equal to the mean ofthe distribution p(z) and the covariance Σ equal to the covariance of p(z). This issometimes called moment matching. An example of this was seen in Figure 10.3(a).
Now let us exploit this result to obtain a practical algorithm for approximate
inference. For many probabilistic models, the joint distribution of data D and hidden
variables (including parameters) θ comprises a product of factors in the formp(D, θ) =fi (θ).
(10.188)iThis would arise, for example, in a model for independent, identically distributed
data in which there is one factor fn (θ) = p(xn |θ) for each data point xn , alongwith a factor f0 (θ) = p(θ) corresponding to the prior. More generally, it would also
apply to any model deﬁned by a directed probabilistic graph in which each factor is aconditional distribution corresponding to one of the nodes, or an undirected graph inwhich each factor is a clique potential. We are interested in evaluating the posterior
distribution p(θ|D) for the purpose of making predictions, as well as the model
evidence p(D) for the purpose of model comparison. From (10.188) the posterior isgiven by1 fi (θ)(10.189)p(θ|D) =p(D)iand the model evidence is given byp(D) = fi (θ) dθ.

(10.190)iHere we are considering continuous variables, but the following discussion applies
equally to discrete variables with integrals replaced by summations. We shall suppose that the marginalization over θ, along with the marginalizations with respect tothe posterior distribution required to make predictions, are intractable so that someform of approximation is required.
Expectation propagation is based on an approximation to the posterior distribution which is also given by a product of factors
q(θ) =1 
fi (θ)
Z(10.191)iin which each factor fi (θ) in the approximation corresponds to one of the factorsfi (θ) in the true posterior (10.189), and the factor 1/Z is the normalizing constantneeded to ensure that the left-hand side of (10.191) integrates to unity. In order toobtain a practical algorithm, we need to constrain the factors fi (θ) in some way,
and in particular we shall assume that they come from the exponential family. The
product of the factors will therefore also be from the exponential family and so can10.7. Expectation Propagation507

be described by a ﬁnite set of sufﬁcient statistics. For example, if each of the fi (θ)
is a Gaussian, then the overall approximation q(θ) will also be Gaussian.
Ideally we would like to determine the fi (θ) by minimizing the Kullback-Leibler
divergence between the true posterior and the approximation given by
'


'1
1 
'
fi (θ)'fi (θ) .
(10.192)
KL (pq) = KL
p(D)
'Z
i

i

Note that this is the reverse form of KL divergence compared with that used in variational inference. In general, this minimization will be intractable because the KL divergence involves averaging with respect to the true distribution. As a rough approximation, we could instead minimize the KL divergences between the corresponding
pairs fi (θ) and fi (θ) of factors. This represents a much simpler problem to solve,
and has the advantage that the algorithm is noniterative. However, because each factor is individually approximated, the product of the factors could well give a poor
approximation.
Expectation propagation makes a much better approximation by optimizing eachfactor in turn in the context of all of the remaining factors. It starts by initializing
the factors fi (θ), and then cycles through the factors reﬁning them one at a time.
This is similar in spirit to the update of factors in the variational Bayes framework
considered earlier. Suppose we wish to reﬁne factor fj (θ). We ﬁrst remove this
factor from the product to give i=j fi (θ). Conceptually, we will now determine arevised form of the factor fj (θ) by ensuring that the productq new (θ) ∝ fj (θ)

fi (θ)(10.193)

i= j

is as close as possible tofj (θ)

fi (θ)(10.194)i= jSection 10.7.1

in which we keep ﬁxed all of the factors fi (θ) for i = j. This ensures that theapproximation is most accurate in the regions of high posterior probability as deﬁned
by the remaining factors. We shall see an example of this effect when we apply EP
to the ‘clutter problem’. To achieve this, we ﬁrst remove the factor fj (θ) from thecurrent approximation to the posterior by deﬁning the unnormalized distribution
q \j (θ) =q(θ).
fj (θ)(10.195)

Note that we could instead ﬁnd q \j (θ) from the product of factors i = j, although
in practice division is usually easier. This is now combined with the factor fj (θ) togive a distribution1
fj (θ)q \j (θ)(10.196)Zj

508

10. APPROXIMATE INFERENCE
1

40

0.8

30

0.6
20
0.4
10

0.2
0
−2

−1

0

1

2

3

4

0
−2

−1

0

1

2

3

4

Figure 10.14 Illustration of the expectation propagation approximation using a Gaussian distribution for theexample considered earlier in Figures 4.14 and 10.1. The left-hand plot shows the original distribution (yellow)
along with the Laplace (red), global variational (green), and EP (blue) approximations, and the right-hand plotshows the corresponding negative logarithms of the distributions. Note that the EP distribution is broader thanthat variational inference, as a consequence of the different form of KL divergence.

where Zj is the normalization constant given by
Zj = fj (θ)q \j (θ) dθ.

(10.197)

We now determine a revised factor fj (θ) by minimizing the Kullback-Leibler divergence'
fj (θ)q \j (θ) '' q new (θ) .
(10.198)
KL'
Zj
This is easily solved because the approximating distribution q new (θ) is from the exponential family, and so we can appeal to the result (10.187), which tells us that theparameters of q new (θ) are obtained by matching its expected sufﬁcient statistics tothe corresponding moments of (10.196). We shall assume that this is a tractable operation. For example, if we choose q(θ) to be a Gaussian distribution N (θ|µ, Σ), thenµ is set equal to the mean of the (unnormalized) distribution fj (θ)q \j (θ), and Σ is
set to its covariance. More generally, it is straightforward to obtain the required expectations for any member of the exponential family, provided it can be normalized,
because the expected statistics can be related to the derivatives of the normalization
coefﬁcient, as given by (2.226). The EP approximation is illustrated in Figure 10.14.
From (10.193), we see that the revised factor fj (θ) can be found by taking
q new (θ) and dividing out the remaining factors so that
new

q (θ)
fj (θ) = K \jq (θ)(10.199)

where we have used (10.195). The coefﬁcient K is determined by multiplying both

10.7. Expectation Propagation
sides of (10.199) by q \i (θ) and integrating to giveK = fj (θ)q \j (θ) dθ

509(10.200)

where we have used the fact that q new (θ) is normalized. The value of K can therefore
be found by matching zeroth-order moments\j
fj (θ)q (θ) dθ = fj (θ)q \j (θ) dθ.
(10.201)
Combining this with (10.197), we then see that K = Zj and so can be found by
evaluating the integral in (10.197).
In practice, several passes are made through the set of factors, revising eachfactor in turn. The posterior distribution p(θ|D) is then approximated using (10.191),
and the model evidence p(D) can be approximated by using (10.190) with the factorsfi (θ) replaced by their approximations fi (θ).
Expectation Propagation
We are given a joint distribution over observed data D and stochastic variables
θ in the form of a product of factorsfi (θ)(10.202)p(D, θ) =i

and we wish to approximate the posterior distribution p(θ|D) by a distributionof the form
1 
fi (θ).
(10.203)q(θ) =Zi

We also wish to approximate the model evidence p(D).
1. Initialize all of the approximating factors fi (θ).
2. Initialize the posterior approximation by settingq(θ) ∝fi (θ).

(10.204)i

3. Until convergence:(a) Choose a factor fj (θ) to reﬁne.
(b) Remove fj (θ) from the posterior by divisionq \j (θ) =q(θ).
fj (θ)(10.205)

510

10. APPROXIMATE INFERENCE(c) Evaluate the new posterior by setting the sufﬁcient statistics (moments)of q new (θ) equal to those of q \j (θ)fj (θ), including evaluation of thenormalization constant(10.206)
Zj = q \j (θ)fj (θ) dθ.
(d) Evaluate and store the new factor
new

q (θ)
.
fj (θ) = Zj \jq (θ)
4. Evaluate the approximation to the model evidence p(D) 
fi (θ) dθ.

(10.207)(10.208)i

A special case of EP, known as assumed density ﬁltering (ADF) or moment
matching (Maybeck, 1982; Lauritzen, 1992; Boyen and Koller, 1998; Opper andWinther, 1999), is obtained by initializing all of the approximating factors except
the ﬁrst to unity and then making one pass through the factors updating each of themonce. Assumed density ﬁltering can be appropriate for on-line learning in which data
points are arriving in a sequence and we need to learn from each data point and thendiscard it before considering the next point. However, in a batch setting we have theopportunity to re-use the data points many times in order to achieve improved accuracy, and it is this idea that is exploited in expectation propagation. Furthermore,
if we apply ADF to batch data, the results will have an undesirable dependence onthe (arbitrary) order in which the data points are considered, which again EP canovercome.
One disadvantage of expectation propagation is that there is no guarantee thatthe iterations will converge. However, for approximations q(θ) in the exponential
family, if the iterations do converge, the resulting solution will be a stationary pointof a particular energy function (Minka, 2001a), although each iteration of EP does
not necessarily decrease the value of this energy function. This is in contrast to
variational Bayes, which iteratively maximizes a lower bound on the log marginal
likelihood, in which each iteration is guaranteed not to decrease the bound. It is
possible to optimize the EP cost function directly, in which case it is guaranteed
to converge, although the resulting algorithms can be slower and more complex to
implement.
Another difference between variational Bayes and EP arises from the form of
KL divergence that is minimized by the two algorithms, because the former minimizes KL(qp) whereas the latter minimizes KL(pq). As we saw in Figure 10.3,
for distributions p(θ) which are multimodal, minimizing KL(pq) can lead to poor
approximations. In particular, if EP is applied to mixtures the results are not sensible because the approximation tries to capture all of the modes of the posterior
distribution. Conversely, in logistic-type models, EP often out-performs both local
variational methods and the Laplace approximation (Kuss and Rasmussen, 2006).

511

10.7. Expectation Propagation
Figure 10.15

Illustration of the clutter problem
for a data space dimensionality of
D = 1. Training data points, denoted by the crosses, are drawn
from a mixture of two Gaussians
with components shown in red
and green. The goal is to infer themean of the green Gaussian fromthe observed data.

−5

0

θ

5

x

10

10.7.1 Example: The clutter problem
Following Minka (2001b), we illustrate the EP algorithm using a simple example in which the goal is to infer the mean θ of a multivariate Gaussian distribution
over a variable x given a set of observations drawn from that distribution. To makethe problem more interesting, the observations are embedded in background clutter,
which itself is also Gaussian distributed, as illustrated in Figure 10.15. The distribution of observed values x is therefore a mixture of Gaussians, which we take to beof the formp(x|θ) = (1 − w)N (x|θ, I) + wN (x|0, aI)(10.209)
where w is the proportion of background clutter and is assumed to be known. The
prior over θ is taken to be Gaussianp(θ) = N (θ|0, bI)(10.210)and Minka (2001a) chooses the parameter values a = 10, b = 100 and w = 0.5.
The joint distribution of N observations D = {x1 , . . . , xN } and θ is given byp(D, θ) = p(θ)

N

p(xn |θ)(10.211)n=1

and so the posterior distribution comprises a mixture of 2N Gaussians. Thus thecomputational cost of solving this problem exactly would grow exponentially withthe size of the data set, and so an exact solution is intractable for moderately largeN.
To apply EP to the clutter problem, we ﬁrst identify the factors f0 (θ) = p(θ)
and fn (θ) = p(xn |θ). Next we select an approximating distribution from the exponential family, and for this example it is convenient to choose a spherical Gaussianq(θ) = N (θ|m, vI).

(10.212)512

10. APPROXIMATE INFERENCE
The factor approximations will therefore take the form of exponential-quadratic
functions of the formfn (θ) = sn N (θ|mn , vn I)(10.213)

Exercise 10.37

Exercise 10.38

where n = 1, . . . , N , and we set f0 (θ) equal to the prior p(θ). Note that the use of
N (θ|·, ·) does not imply that the right-hand side is a well-deﬁned Gaussian density(in fact, as we shall see, the variance parameter vn can be negative) but is simply aconvenient shorthand notation. The approximations fn (θ), for n = 1, . . . , N , canbe initialized to unity, corresponding to sn = (2πvn )D/2 , vn → ∞ and mn = 0,
where D is the dimensionality of x and hence of θ. The initial q(θ), deﬁned by(10.191), is therefore equal to the prior.
We then iteratively reﬁne the factors by taking one factor fn (θ) at a time andapplying (10.205), (10.206), and (10.207). Note that we do not need to revise theterm f0 (θ) because an EP update will leave this term unchanged. Here we state theresults and leave the reader to ﬁll in the details.
First we remove the current estimate fn (θ) from q(θ) by division using (10.205)
to give q \n (θ), which has mean and inverse variance given by
m\n
\n −1

(v )

= m + v \n vn−1 (m − mn )(10.214)= v(10.215)−1− vn−1 .

Next we evaluate the normalization constant Zn using (10.206) to giveZn = (1 − w)N (xn |m\n , (v \n + 1)I) + wN (xn |0, aI).
Exercise 10.39(10.216)

Similarly, we compute the mean and variance of q new (θ) by ﬁnding the mean andvariance of q \n (θ)fn (θ) to givev \n(xn − m\n )(10.217)m = m\n + ρn \n
v +1(v \n )2
(v \n )2 xn − m\n 2+ ρn (1 − ρn )v = v \n − ρn \n(10.218)
v +1
D(v \n + 1)2
where the quantityw
N (xn |0, aI)(10.219)
Zn
has a simple interpretation as the probability of the point xn not being clutter. Then
we use (10.207) to compute the reﬁned factor fn (θ) whose parameters are given byρn = 1 −vn−1= (v new )−1 − (v \n )−1

mn

= m

sn

=

\n

\n

(10.220)\n −1\n+ (vn + v )(v ) (m
−m )
Zn
.
(2πvn )D/2 N (mn |m\n , (vn + v \n )I)
new(10.221)(10.222)

This reﬁnement process is repeated until a suitable termination criterion is satisﬁed,
for instance that the maximum change in parameter values resulting from a complete

513

10.7. Expectation Propagation

−5

0

5

θ

10

−5

0

5

θ

10

Figure 10.16 Examples of the approximation of speciﬁc factors for a one-dimensional version of the clutter
problem, showing fn (θ) in blue, fen (θ) in red, and q \n (θ) in green. Notice that the current form for q \n (θ) controls
the range of θ over which fen (θ) will be a good approximation to fn (θ).

pass through all factors is less than some threshold. Finally, we use (10.208) toevaluate the approximation to the model evidence, given byp(D)  (2πv new )D/2 exp(B/2)

Nsn (2πvn )−D/2

(10.223)

n=1

where

(mnew )T mnew  mT
n mn
.
−
v
vn
N

B=

(10.224)

n=1

Examples factor approximations for the clutter problem with a one-dimensional parameter space θ are shown in Figure 10.16. Note that the factor approximations canhave inﬁnite or even negative values for the ‘variance’ parameter vn . This simply
corresponds to approximations that curve upwards instead of downwards and are notnecessarily problematic provided the overall approximate posterior q(θ) has positive variance. Figure 10.17 compares the performance of EP with variational Bayes(mean ﬁeld theory) and the Laplace approximation on the clutter problem.

10.7.2 Expectation propagation on graphs
So far in our general discussion of EP, we have allowed the factors fi (θ) in the
distribution p(θ) to be functions of all of the components of θ, and similarly for theapproximating factors f(θ) in the approximating distribution q(θ). We now consider
situations in which the factors depend only on subsets of the variables. Such restrictions can be conveniently expressed using the framework of probabilistic graphicalmodels, as discussed in Chapter 8. Here we use a factor graph representation because
this encompasses both directed and undirected graphs.

514

10. APPROXIMATE INFERENCE
Evidence

Posterior mean
0

10

vb
−200

Error

Error

10

laplace vb

−5

10

−202

laplace

10

ep

ep
4

−204

6

10

10
FLOPS

10

4

6

10

10
FLOPS

Figure 10.17 Comparison of expectation propagation, variational inference, and the Laplace approximation on
the clutter problem. The left-hand plot shows the error in the predicted posterior mean versus the number of
ﬂoating point operations, and the right-hand plot shows the corresponding results for the model evidence.

Section 8.4.4

We shall focus on the case in which the approximating distribution is fully factorized, and we shall show that in this case expectation propagation reduces to loopy
belief propagation (Minka, 2001a). To start with, we show this in the context of asimple example, and then we shall explore the general case.
First of all, recall from (10.17) that if we minimize the Kullback-Leibler divergence KL(pq) with respect to a factorized distribution q, then the optimal solution
for each factor is simply the corresponding marginal of p.
Now consider the factor graph shown on the left in Figure 10.18, which wasintroduced earlier in the context of the sum-product algorithm. The joint distribution
is given by(10.225)p(x) = fa (x1 , x2 )fb (x2 , x3 )fc (x2 , x4 ).
We seek an approximation q(x) that has the same factorization, so thatq(x) ∝ fa (x1 , x2 )fb (x2 , x3 )fc (x2 , x4 ).

(10.226)

Note that normalization constants have been omitted, and these can be re-instated at
the end by local normalization, as is generally done in belief propagation. Now suppose we restrict attention to approximations in which the factors themselves factorize
with respect to the individual variables so thatq(x) ∝ fa1 (x1 )fa2 (x2 )fb2 (x2 )fb3 (x3 )fc2 (x2 )fc4 (x4 )(10.227)

which corresponds to the factor graph shown on the right in Figure 10.18. Because
the individual factors are factorized, the overall distribution q(x) is itself fully factorized.
Now we apply the EP algorithm using the fully factorized approximation. Suppose that we have initialized all of the factors and that we choose to reﬁne factor

515

10.7. Expectation Propagation
x1

x2

x3

fa

x1

x2
f˜a1

fb

f˜a2

x3

f˜c2

f˜b2

f˜b3

fc
f˜c4

x4

x4

Figure 10.18 On the left is a simple factor graph from Figure 8.51 and reproduced here for convenience. On
the right is the corresponding factorized approximation.

fb (x2 , x3 ) = fb2 (x2 )fb3 (x3 ). We ﬁrst remove this factor from the approximating
distribution to giveq \b (x) = fa1 (x1 )fa2 (x2 )fc2 (x2 )fc4 (x4 )(10.228)

and we then multiply this by the exact factor fb (x2 , x3 ) to givep(x) = q \b (x)fb (x2 , x3 ) = fa1 (x1 )fa2 (x2 )fc2 (x2 )fc4 (x4 )fb (x2 , x3 ). (10.229)
pq new ).
We now ﬁnd q new (x) by minimizing the Kullback-Leibler divergence KL(new
The result, as noted above, is that q (z) comprises the product of factors, one foreach variable xi , in which each factor is given by the corresponding marginal ofp(x). These four marginals are given byp(x1 ) ∝ fa1 (x1 )p(x2 ) ∝ fa2 (x2 )fc2 (x2 )fb (x2 , x3 )(10.230)(10.231)x3

p(x3 ) ∝fb (x2 , x3 )fa2 (x2 )fc2 (x2 )(10.232)x2

p(x4 ) ∝ fc4 (x4 )(10.233)

and q new (x) is obtained by multiplying these marginals together. We see that the
only factors in q(x) that change when we update fb (x2 , x3 ) are those that involve
the variables in fb namely x2 and x3 . To obtain the reﬁned factor fb (x2 , x3 ) =fb2 (x2 )fb3 (x3 ) we simply divide q new (x) by q \b (x), which givesfb2 (x2 ) ∝fb (x2 , x3 )(10.234)x3

fb3 (x3 ) ∝fb (x2 , x3 )fa2 (x2 )fc2 (x2 ) .
x2(10.235)

516

10. APPROXIMATE INFERENCE

Section 8.4.4

These are precisely the messages obtained using belief propagation in which messages from variable nodes to factor nodes have been folded into the messages fromfactor nodes to variable nodes. In particular, fb2 (x2 ) corresponds to the messageµfb →x2 (x2 ) sent by factor node fb to variable node x2 and is given by (8.81). Similarly, if we substitute (8.78) into (8.79), we obtain (10.235) in which fa2 (x2 ) corresponds to µfa →x2 (x2 ) and fc2 (x2 ) corresponds to µfc →x2 (x2 ), giving the message
fb3 (x3 ) which corresponds to µfb →x3 (x3 ).
This result differs slightly from standard belief propagation in that messages are
passed in both directions at the same time. We can easily modify the EP procedure
to give the standard form of the sum-product algorithm by updating just one of thefactors at a time, for instance if we reﬁne only fb3 (x3 ), then fb2 (x2 ) is unchanged
by deﬁnition, while the reﬁned version of fb3 (x3 ) is again given by (10.235). If
we are reﬁning only one term at a time, then we can choose the order in which thereﬁnements are done as we wish. In particular, for a tree-structured graph we canfollow a two-pass update scheme, corresponding to the standard belief propagation
schedule, which will result in exact inference of the variable and factor marginals.
The initialization of the approximation factors in this case is unimportant.
Now let us consider a general factor graph corresponding to the distributionfi (θ i )(10.236)p(θ) =iwhere θ i represents the subset of variables associated with factor fi . We approximate
this using a fully factorized distribution of the formfik (θk )(10.237)q(θ) ∝ikwhere θk corresponds to an individual variable node. Suppose that we wish to reﬁnethe particular term fjl (θl ) keeping all other terms ﬁxed. We ﬁrst remove the termfj (θ j ) from q(θ) to givefik (θk )(10.238)
q \j (θ) ∝i= j kand then multiply by the exact factor fj (θ j ). To determine the reﬁned term fjl (θl ),we need only consider the functional dependence on θl , and so we simply ﬁnd thecorresponding marginal ofq \j (θ)fj (θ j ).
(10.239)
Up to a multiplicative constant, this involves taking the marginal of fj (θ j ) multipliedby any terms from q \j (θ) that are functions of any of the variables in θ j . Terms that
correspond to other factors fi (θ i ) for i = j will cancel between numerator anddenominator when we subsequently divide by q \j (θ). We therefore obtainfjl (θl ) ∝fj (θ j )fkm (θm ).
(10.240)θm=l ∈θ jk m=l

Exercises

517

We recognize this as the sum-product rule in the form in which messages from variable nodes to factor nodes have been eliminated, as illustrated by the example shownin Figure 8.50. The quantity fjm (θm ) corresponds to the message µfj →θm (θm ),which factor node j sends to variable node m, and the product over k in (10.240)
is over all factors that depend on the variables θm that have variables (other thanvariable θl ) in common with factor fj (θ j ). In other words, to compute the outgoingmessage from a factor node, we take the product of all the incoming messages fromother factor nodes, multiply by the local factor, and then marginalize.
Thus, the sum-product algorithm arises as a special case of expectation propagation if we use an approximating distribution that is fully factorized. This suggeststhat more ﬂexible approximating distributions, corresponding to partially disconnected graphs, could be used to achieve higher accuracy. Another generalization is
to group factors fi (θ i ) together into sets and to reﬁne all the factors in a set together
at each iteration. Both of these approaches can lead to improvements in accuracy(Minka, 2001b). In general, the problem of choosing the best combination of grouping and disconnection is an open research issue.
We have seen that variational message passing and expectation propagation optimize two different forms of the Kullback-Leibler divergence. Minka (2005) has
shown that a broad range of message passing algorithms can be derived from a common framework involving minimization of members of the alpha family of divergences, given by (10.19). These include variational message passing, loopy belief
propagation, and expectation propagation, as well as a range of other algorithms,
which we do not have space to discuss here, such as tree-reweighted message passing (Wainwright et al., 2005), fractional belief propagation (Wiegerinck and Heskes,
2003), and power EP (Minka, 2004).

Exercises

10.1 ( ) www Verify that the log marginal distribution of the observed data ln p(X)can be decomposed into two terms in the form (10.2) where L(q) is given by (10.3)
and KL(qp) is given by (10.4).
10.2 ( ) Use the properties E[z1 ] = m1 and E[z2 ] = m2 to solve the simultaneous equations (10.13) and (10.15), and hence show that, provided the original distributionp(z) is nonsingular, the unique solution for the means of the factors in the approximation distribution is given by E[z1 ] = µ1 and E[z2 ] = µ2 .
10.3 ( ) www Consider a factorized variational distribution q(Z) of the form (10.5).
By using the technique of Lagrange multipliers, verify that minimization of the
Kullback-Leibler divergence KL(pq) with respect to one of the factors qi (Zi ),keeping all other factors ﬁxed, leads to the solution (10.17).
10.4 ( ) Suppose that p(x) is some ﬁxed distribution and that we wish to approximate
it using a Gaussian distribution q(x) = N (x|µ, Σ). By writing down the form of
the KL divergence KL(pq) for a Gaussian q(x) and then differentiating, show that

518

10. APPROXIMATE INFERENCE
minimization of KL(pq) with respect to µ and Σ leads to the result that µ is givenby the expectation of x under p(x) and that Σ is given by the covariance.
10.5 ( ) www Consider a model in which the set of all hidden stochastic variables, denoted collectively by Z, comprises some latent variables z together with some model
parameters θ. Suppose we use a variational distribution that factorizes between latent variables and parameters so that q(z, θ) = qz (z)qθ (θ), in which the distributionqθ (θ) is approximated by a point estimate of the form qθ (θ) = δ(θ − θ 0 ) where θ 0
is a vector of free parameters. Show that variational optimization of this factorized
distribution is equivalent to an EM algorithm, in which the E step optimizes qz (z),and the M step maximizes the expected complete-data log posterior distribution of θwith respect to θ 0 .
10.6 ( ) The alpha family of divergences is deﬁned by (10.19). Show that the KullbackLeibler divergence KL(pq) corresponds to α → 1. This can be done by writingp = exp( ln p) = 1 +  ln p + O(2 ) and then taking  → 0. Similarly show that
KL(qp) corresponds to α → −1.
10.7 ( ) Consider the problem of inferring the mean and precision of a univariate Gaussian using a factorized variational approximation, as considered in Section 10.1.3.
1Show that the factor qµ (µ) is a Gaussian of the form N (µ|µN , λ−N ) with mean andprecision given by (10.26) and (10.27), respectively. Similarly show that the factorqτ (τ ) is a gamma distribution of the form Gam(τ |aN , bN ) with parameters given by(10.29) and (10.30).
10.8 ( ) Consider the variational posterior distribution for the precision of a univariate
Gaussian whose parameters are given by (10.29) and (10.30). By using the standard
results for the mean and variance of the gamma distribution given by (B.27) and(B.28), show that if we let N → ∞, this variational posterior distribution has amean given by the inverse of the maximum likelihood estimator for the variance ofthe data, and a variance that goes to zero.
10.9 ( ) By making use of the standard result E[τ ] = aN /bN for the mean of a gamma
distribution, together with (10.26), (10.27), (10.29), and (10.30), derive the result(10.33) for the reciprocal of the expected precision in the factorized variational treatment of a univariate Gaussian.
10.10 ( ) www Derive the decomposition given by (10.34) that is used to ﬁnd approximate posterior distributions over models using variational inference.
10.11 ( ) www By using a Lagrange multiplier to enforce the normalization constraint
on the distribution q(m), show that the maximum of the lower bound (10.35) is givenby (10.36).
10.12 ( ) Starting from the joint distribution (10.41), and applying the general result(10.9), show that the optimal variational distribution q  (Z) over the latent variables
for the Bayesian mixture of Gaussians is given by (10.48) by verifying the steps
given in the text.

Exercises

519

10.13 ( ) www Starting from (10.54), derive the result (10.59) for the optimum variational posterior distribution over µk and Λk in the Bayesian mixture of Gaussians,
and hence verify the expressions for the parameters of this distribution given by(10.60)–(10.63).
10.14 () Using the distribution (10.59), verify the result (10.64).

10.15 ( ) Using the result (B.17), show that the expected value of the mixing coefﬁcientsin the variational mixture of Gaussians is given by (10.69).
10.16 ( ) www Verify the results (10.71) and (10.72) for the ﬁrst two terms in thelower bound for the variational Gaussian mixture model given by (10.70).
10.17 () Verify the results (10.73)–(10.77) for the remaining terms in the lower boundfor the variational Gaussian mixture model given by (10.70).
10.18 () In this exercise, we shall derive the variational re-estimation equations for
the Gaussian mixture model by direct differentiation of the lower bound. To do this
we assume that the variational distribution has the factorization deﬁned by (10.42)
and (10.55) with factors given by (10.48), (10.57), and (10.59). Substitute these into
(10.70) and hence obtain the lower bound as a function of the parameters of the variational distribution. Then, by maximizing the bound with respect to these parameters,
derive the re-estimation equations for the factors in the variational distribution, andshow that these are the same as those obtained in Section 10.2.1.
10.19 ( ) Derive the result (10.81) for the predictive distribution in the variational treatment of the Bayesian mixture of Gaussians model.
10.20 ( ) www This exercise explores the variational Bayes solution for the mixture ofGaussians model when the size N of the data set is large and shows that it reduces (aswe would expect) to the maximum likelihood solution based on EM derived in Chapter 9. Note that results from Appendix B may be used to help answer this exercise.
First show that the posterior distribution q  (Λk ) of the precisions becomes sharply
peaked around the maximum likelihood solution. Do the same for the posterior distribution of the means q  (µk |Λk ). Next consider the posterior distribution q  (π)
for the mixing coefﬁcients and show that this too becomes sharply peaked aroundthe maximum likelihood solution. Similarly, show that the responsibilities become
equal to the corresponding maximum likelihood values for large N , by making useof the following asymptotic result for the digamma function for large x
ψ(x) = ln x + O (1/x) .

(10.241)

Finally, by making use of (10.80), show that for large N , the predictive distribution
becomes a mixture of Gaussians.
10.21 ( ) Show that the number of equivalent parameter settings due to interchange symmetries in a mixture model with K components is K!.

520

10. APPROXIMATE INFERENCE
10.22 ( ) We have seen that each mode of the posterior distribution in a Gaussian mixture model is a member of a family of K! equivalent modes. Suppose that the resultof running the variational inference algorithm is an approximate posterior distribution q that is localized in the neighbourhood of one of the modes. We can thenapproximate the full posterior distribution as a mixture of K! such q distributions,once centred on each mode and having equal mixing coefﬁcients. Show that if we
assume negligible overlap between the components of the q mixture, the resulting
lower bound differs from that for a single component q distribution through the addition of an extra term ln K!.
10.23 ( ) www Consider a variational Gaussian mixture model in which there is no
prior distribution over mixing coefﬁcients {πk }. Instead, the mixing coefﬁcients aretreated as parameters, whose values are to be found by maximizing the variational
lower bound on the log marginal likelihood. Show that maximizing this lower boundwith respect to the mixing coefﬁcients, using a Lagrange multiplier to enforce theconstraint that the mixing coefﬁcients sum to one, leads to the re-estimation result(10.83). Note that there is no need to consider all of the terms in the lower bound but
only the dependence of the bound on the {πk }.
10.24 ( ) www We have seen in Section 10.2 that the singularities arising in the maximum likelihood treatment of Gaussian mixture models do not arise in a Bayesiantreatment. Discuss whether such singularities would arise if the Bayesian model
were solved using maximum posterior (MAP) estimation.
10.25 ( ) The variational treatment of the Bayesian mixture of Gaussians, discussed in
Section 10.2, made use of a factorized approximation (10.5) to the posterior distribution. As we saw in Figure 10.2, the factorized assumption causes the variance of theposterior distribution to be under-estimated for certain directions in parameter space.
Discuss qualitatively the effect this will have on the variational approximation to themodel evidence, and how this effect will vary with the number of components inthe mixture. Hence explain whether the variational Gaussian mixture will tend tounder-estimate or over-estimate the optimal number of components.
10.26 (
) Extend the variational treatment of Bayesian linear regression to include
a gamma hyperprior Gam(β|c0 , d0 ) over β and solve variationally, by assuming afactorized variational distribution of the form q(w)q(α)q(β). Derive the variational
update equations for the three factors in the variational distribution and also obtainan expression for the lower bound and for the predictive distribution.
10.27 ( ) By making use of the formulae given in Appendix B show that the variational
lower bound for the linear basis function regression model, deﬁned by (10.107), canbe written in the form (10.107) with the various terms deﬁned by (10.108)–(10.112).
10.28 (
) Rewrite the model for the Bayesian mixture of Gaussians, introduced in
Section 10.2, as a conjugate model from the exponential family, as discussed in
Section 10.4. Hence use the general results (10.115) and (10.119) to derive thespeciﬁc results (10.48), (10.57), and (10.59).

Exercises

521

10.29 ( ) www Show that the function f (x) = ln(x) is concave for 0 < x < ∞by computing its second derivative. Determine the form of the dual function g(λ)
deﬁned by (10.133), and verify that minimization of λx − g(λ) with respect to λaccording to (10.132) indeed recovers the function ln(x).
10.30 ( ) By evaluating the second derivative, show that the log logistic function f (x) =− ln(1 + e−x ) is concave. Derive the variational upper bound (10.137) directly bymaking a second order Taylor expansion of the log logistic function around a pointx = ξ.
10.31 ( ) By ﬁnding the second derivative with respect to x, show that the function
f (x) = − ln(ex/2 + e−x/2 ) is a concave function of x. Now consider the second
derivatives with respect to the variable x2 and hence show that it is a convex functionof x2 . Plot graphs of f (x) against x and against x2 . Derive the lower bound (10.144)
on the logistic sigmoid function directly by making a ﬁrst order Taylor series expansion of the function f (x) in the variable x2 centred on the value ξ 2 .
10.32 ( ) www Consider the variational treatment of logistic regression with sequential learning in which data points are arriving one at a time and each must be processed and discarded before the next data point arrives. Show that a Gaussian approximation to the posterior distribution can be maintained through the use of thelower bound (10.151), in which the distribution is initialized using the prior, and as
each data point is absorbed its corresponding variational parameter ξn is optimized.
10.33 ( ) By differentiating the quantity Q(ξ, ξ old ) deﬁned by (10.161) with respect tothe variational parameter ξn show that the update equation for ξn for the Bayesianlogistic regression model is given by (10.163).
10.34 ( ) In this exercise we derive re-estimation equations for the variational parameters ξ in the Bayesian logistic regression model of Section 4.5 by direct maximizationof the lower bound given by (10.164). To do this set the derivative of L(ξ) with respect to ξn equal to zero, making use of the result (3.117) for the derivative of the logof a determinant, together with the expressions (10.157) and (10.158) which deﬁne
the mean and covariance of the variational posterior distribution q(w).
10.35 ( ) Derive the result (10.164) for the lower bound L(ξ) in the variational logistic
regression model. This is most easily done by substituting the expressions for theGaussian prior q(w) = N (w|m0 , S0 ), together with the lower bound h(w, ξ) on
the likelihood function, into the integral (10.159) which deﬁnes L(ξ). Next gather
together the terms which depend on w in the exponential and complete the squareto give a Gaussian integral, which can then be evaluated by invoking the standard
result for the normalization coefﬁcient of a multivariate Gaussian. Finally take thelogarithm to obtain (10.164).
10.36 ( ) Consider the ADF approximation scheme discussed in Section 10.7, and showthat inclusion of the factor fj (θ) leads to an update of the model evidence of theform(10.242)pj (D)  pj−1 (D)Zj

522

10. APPROXIMATE INFERENCE
where Zj is the normalization constant deﬁned by (10.197). By applying this result
recursively, and initializing with p0 (D) = 1, derive the resultp(D) 
Zj .
(10.243)
j10.37 ( ) www Consider the expectation propagation algorithm from Section 10.7, andsuppose that one of the factors f0 (θ) in the deﬁnition (10.188) has the same exponential family functional form as the approximating distribution q(θ). Show that if
the factor f0 (θ) is initialized to be f0 (θ), then an EP update to reﬁne f0 (θ) leavesf0 (θ) unchanged. This situation typically arises when one of the factors is the priorp(θ), and so we see that the prior factor can be incorporated once exactly and does
not need to be reﬁned.
10.38 () In this exercise and the next, we shall verify the results (10.214)–(10.224)
for the expectation propagation algorithm applied to the clutter problem. Begin byusing the division formula (10.205) to derive the expressions (10.214) and (10.215)
by completing the square inside the exponential to identify the mean and variance.
Also, show that the normalization constant Zn , deﬁned by (10.206), is given for the
clutter problem by (10.216). This can be done by making use of the general result(2.115).
10.39 (
) Show that the mean and variance of q new (θ) for EP applied to the clutter
problem are given by (10.217) and (10.218). To do this, ﬁrst prove the following
results for the expectations of θ and θθ T under q new (θ)
E[θ] = m\n + v \n ∇m\n ln Zn\n 2(10.244)E[θ θ] = 2(v ) ∇v\n ln Zn + 2E[θ] m
T

T

\n

\n 2

− m 

(10.245)and then make use of the result (10.216) for Zn . Next, prove the results (10.220)–(10.222) by using (10.207) and completing the square in the exponential. Finally,use (10.208) to derive the result (10.223).

11
Sampling
Methods

For most probabilistic models of practical interest, exact inference is intractable, andso we have to resort to some form of approximation. In Chapter 10, we discussed
inference algorithms based on deterministic approximations, which include methods
such as variational Bayes and expectation propagation. Here we consider approximate inference methods based on numerical sampling, also known as Monte Carlotechniques.
Although for some applications the posterior distribution over unobserved variables will be of direct interest in itself, for most situations the posterior distribution
is required primarily for the purpose of evaluating expectations, for example in orderto make predictions. The fundamental problem that we therefore wish to address inthis chapter involves ﬁnding the expectation of some function f (z) with respect to aprobability distribution p(z). Here, the components of z might comprise discrete orcontinuous variables or some combination of the two. Thus in the case of continuous

523

524

11. SAMPLING METHODS
Figure 11.1

Schematic illustration of a function f (z)whose expectation is to be evaluated withrespect to a distribution p(z).

p(z)

f (z)

z

variables, we wish to evaluate the expectation
E[f ] = f (z)p(z) dz(11.1)

where the integral is replaced by summation in the case of discrete variables. Thisis illustrated schematically for a single continuous variable in Figure 11.1. We shallsuppose that such expectations are too complex to be evaluated exactly using analytical techniques.
The general idea behind sampling methods is to obtain a set of samples z(l)(where l = 1, . . . , L) drawn independently from the distribution p(z). This allows
the expectation (11.1) to be approximated by a ﬁnite sum
1
f =
L

Exercise 11.1

Lf (z(l) ).

(11.2)

l=1

As long as the samples z(l) are drawn from the distribution p(z), then E[f] = E[f ]and so the estimator f has the correct mean. The variance of the estimator is givenby1 (11.3)
var[f] = E (f − E[f ])2
L
is the variance of the function f (z) under the distribution p(z). It is worth emphasizing that the accuracy of the estimator therefore does not depend on the dimensionality of z, and that, in principle, high accuracy may be achievable with a relatively
small number of samples z(l) . In practice, ten or twenty independent samples may
sufﬁce to estimate an expectation to sufﬁcient accuracy.
The problem, however, is that the samples {z(l) } might not be independent, andso the effective sample size might be much smaller than the apparent sample size.
Also, referring back to Figure 11.1, we note that if f (z) is small in regions wherep(z) is large, and vice versa, then the expectation may be dominated by regionsof small probability, implying that relatively large sample sizes will be required toachieve sufﬁcient accuracy.
For many models, the joint distribution p(z) is conveniently speciﬁed in termsof a graphical model. In the case of a directed graph with no observed variables, it is11. SAMPLING METHODS

525

straightforward to sample from the joint distribution (assuming that it is possible tosample from the conditional distributions at each node) using the following ancestral sampling approach, discussed brieﬂy in Section 8.1.2. The joint distribution is
speciﬁed byM
p(z) =p(zi |pai )(11.4)
i=1

where zi are the set of variables associated with node i, and pai denotes the set ofvariables associated with the parents of node i. To obtain a sample from the joint
distribution, we make one pass through the set of variables in the order z1 , . . . , zM
sampling from the conditional distributions p(zi |pai ). This is always possible because at each step all of the parent values will have been instantiated. After one passthrough the graph, we will have obtained a sample from the joint distribution.
Now consider the case of a directed graph in which some of the nodes are instantiated with observed values. We can in principle extend the above procedure, atleast in the case of nodes representing discrete variables, to give the following logic
sampling approach (Henrion, 1988), which can be seen as a special case of importance sampling discussed in Section 11.1.4. At each step, when a sampled value isobtained for a variable zi whose value is observed, the sampled value is comparedto the observed value, and if they agree then the sample value is retained and the algorithm proceeds to the next variable in turn. However, if the sampled value and theobserved value disagree, then the whole sample so far is discarded and the algorithm
starts again with the ﬁrst node in the graph. This algorithm samples correctly fromthe posterior distribution because it corresponds simply to drawing samples from thejoint distribution of hidden variables and data variables and then discarding thosesamples that disagree with the observed data (with the slight saving of not continuing with the sampling from the joint distribution as soon as one contradictory value isobserved). However, the overall probability of accepting a sample from the posterior
decreases rapidly as the number of observed variables increases and as the numberof states that those variables can take increases, and so this approach is rarely usedin practice.
In the case of probability distributions deﬁned by an undirected graph, there isno one-pass sampling strategy that will sample even from the prior distribution withno observed variables. Instead, computationally more expensive techniques must be
employed, such as Gibbs sampling, which is discussed in Section 11.3.
As well as sampling from conditional distributions, we may also require samples
from a marginal distribution. If we already have a strategy for sampling from a joint
distribution p(u, v), then it is straightforward to obtain samples from the marginal
distribution p(u) simply by ignoring the values for v in each sample.
There are numerous texts dealing with Monte Carlo methods. Those of particular interest from the statistical inference perspective include Chen et al. (2001),
Gamerman (1997), Gilks et al. (1996), Liu (2001), Neal (1996), and Robert andCasella (1999). Also there are review articles by Besag et al. (1995), Brooks (1998),
Diaconis and Saloff-Coste (1998), Jerrum and Sinclair (1996), Neal (1993), Tierney(1994), and Andrieu et al. (2003) that provide additional information on sampling526

11. SAMPLING METHODS
methods for statistical inference.
Diagnostic tests for convergence of Markov chain Monte Carlo algorithms are
summarized in Robert and Casella (1999), and some practical guidance on the use ofsampling methods in the context of machine learning is given in Bishop and Nabney(2008).

11.1. Basic Sampling Algorithms
In this section, we consider some simple strategies for generating random samples
from a given distribution. Because the samples will be generated by a computer
algorithm they will in fact be pseudo-random numbers, that is, they will be deterministically calculated, but must nevertheless pass appropriate tests for randomness.
Generating such numbers raises several subtleties (Press et al., 1992) that lie outside
the scope of this book. Here we shall assume that an algorithm has been provided
that generates pseudo-random numbers distributed uniformly over (0, 1), and indeed
most software environments have such a facility built in.

11.1.1 Standard distributions
We ﬁrst consider how to generate random numbers from simple nonuniform distributions, assuming that we already have available a source of uniformly distributed
random numbers. Suppose that z is uniformly distributed over the interval (0, 1),
and that we transform the values of z using some function f (·) so that y = f (z).
The distribution of y will be governed by  dz 
(11.5)
p(y) = p(z)  dywhere, in this case, p(z) = 1. Our goal is to choose the function f (z) such that theresulting values of y have some speciﬁc desired distribution p(y). Integrating (11.5)
we obtain yz = h(y) ≡p(y ) d
y(11.6)−∞

Exercise 11.2

which is the indeﬁnite integral of p(y). Thus, y = h−1 (z), and so we have totransform the uniformly distributed random numbers using a function which is theinverse of the indeﬁnite integral of the desired distribution. This is illustrated in
Figure 11.2.
Consider for example the exponential distributionp(y) = λ exp(−λy)(11.7)

where 0  y < ∞. In this case the lower limit of the integral in (11.6) is 0, and soh(y) = 1 − exp(−λy). Thus, if we transform our uniformly distributed variable z
using y = −λ−1 ln(1 − z), then y will have an exponential distribution.

11.1. Basic Sampling Algorithms
Figure 11.2

Geometrical interpretation of the transformation method for generating nonuni- 1
formly distributed random numbers. h(y)is the indeﬁnite integral of the desired distribution p(y). If a uniformly distributed
random variable z is transformed usingy = h−1 (z), then y will be distributed according to p(y).

527

h(y)

p(y)

0

y

Another example of a distribution to which the transformation method can beapplied is given by the Cauchy distributionp(y) =

Exercise 11.3

1 1.
π 1 + y2

(11.8)In this case, the inverse of the indeﬁnite integral can be expressed in terms of the‘tan’ function.
The generalization to multiple variables is straightforward and involves the Jacobian of the change of variables, so that ∂(z1 , . . . , zM ) .
(11.9)p(y1 , . . . , yM ) = p(z1 , . . . , zM ) ∂(y1 , . . . , yM ) 
As a ﬁnal example of the transformation method we consider the Box-Mullermethod for generating samples from a Gaussian distribution. First, suppose we generate pairs of uniformly distributed random numbers z1 , z2 ∈ (−1, 1), which we cando by transforming a variable distributed uniformly over (0, 1) using z → 2z − 1.
Next we discard each pair unless it satisﬁes z12 + z22  1. This leads to a uniform
distribution of points inside the unit circle with p(z1 , z2 ) = 1/π, as illustrated in
Figure 11.3. Then, for each pair z1 , z2 we evaluate the quantities

Figure 11.3

The Box-Muller method for generating Gaussian distributed random numbers starts by generating samples
from a uniform distribution inside the unit circle.

1

z2

−1
−1

z1

1

528

11. SAMPLING METHODS


y1

= z1


y2
Exercise 11.4

Exercise 11.5

= z2

−2 ln z1r2
−2 ln z2
r2

1/2
(11.10)1/2

where r2 = z12 + z22 . Then the joint distribution of y1 and y2 is given by ∂(z1 , z2 ) p(y1 , y2 ) = p(z1 , z2 ) ∂(y1 , y2 ) 



1
1
2
2
√ exp(−y1 /2) √ exp(−y2 /2)=
2π
2π(11.11)(11.12)and so y1 and y2 are independent and each has a Gaussian distribution with zeromean and unit variance.
If y has a Gaussian distribution with zero mean and unit variance, then σy + µ
will have a Gaussian distribution with mean µ and variance σ 2 . To generate vectorvalued variables having a multivariate Gaussian distribution with mean µ and covariance Σ, we can make use of the Cholesky decomposition, which takes the form
Σ = LLT (Press et al., 1992). Then, if z is a vector valued random variable whose
components are independent and Gaussian distributed with zero mean and unit variance, then y = µ + Lz will have mean µ and covariance Σ.
Obviously, the transformation technique depends for its success on the ability
to calculate and then invert the indeﬁnite integral of the required distribution. Such
operations will only be feasible for a limited number of simple distributions, and so
we must turn to alternative approaches in search of a more general strategy. Herewe consider two techniques called rejection sampling and importance sampling. Although mainly limited to univariate distributions and thus not directly applicable tocomplex problems in many dimensions, they do form important components in moregeneral strategies.

11.1.2 Rejection sampling
The rejection sampling framework allows us to sample from relatively complex
distributions, subject to certain constraints. We begin by considering univariate distributions and discuss the extension to multiple dimensions subsequently.
Suppose we wish to sample from a distribution p(z) that is not one of the simple,
standard distributions considered so far, and that sampling directly from p(z) is difﬁcult. Furthermore suppose, as is often the case, that we are easily able to evaluatep(z) for any given value of z, up to some normalizing constant Z, so thatp(z) =

1

p(z)
Zp

(11.13)p(z) can readily be evaluated, but Zp is unknown.
where In order to apply rejection sampling, we need some simpler distribution q(z),sometimes called a proposal distribution, from which we can readily draw samples.

529

11.1. Basic Sampling Algorithms
Figure 11.4

Exercise 11.6

In the rejection sampling method,
samples are drawn from a simple distribution q(z) and rejected
if they fall in the grey area between the unnormalized distribution ep(z) and the scaled distribution kq(z). The resulting samples
are distributed according to p(z),which is the normalized version of
e
p(z).

kq(z)

kq(z0 )

u0
z0

p̃(z)
z

We next introduce a constant k whose value is chosen such that kq(z)  
p(z) forall values of z. The function kq(z) is called the comparison function and is illustrated for a univariate distribution in Figure 11.4. Each step of the rejection sampler
involves generating two random numbers. First, we generate a number z0 from the
distribution q(z). Next, we generate a number u0 from the uniform distribution over[0, kq(z0 )]. This pair of random numbers has uniform distribution under the curvep(z0 ) then the sample is rejected, otherwiseof the function kq(z). Finally, if u0 > 
u0 is retained. Thus the pair is rejected if it lies in the grey shaded region in Figp(z),ure 11.4. The remaining pairs then have uniform distribution under the curve of and hence the corresponding z values are distributed according to p(z), as desired.
The original values of z are generated from the distribution q(z), and these samples are then accepted with probability p(z)/kq(z), and so the probability that asample will be accepted is given by{
p(z)/kq(z)} q(z) dz
p(accept) =1
p(z) dz.
(11.14)=kThus the fraction of points that are rejected by this method depends on the ratio ofp(z) to the area under the curve kq(z).
the area under the unnormalized distribution 
We therefore see that the constant k should be as small as possible subject to thelimitation that kq(z) must be nowhere less than p(z).
As an illustration of the use of rejection sampling, consider the task of samplingfrom the gamma distribution
Gam(z|a, b) =Exercise 11.7

ba z a−1 exp(−bz)Γ(a)(11.15)which, for a > 1, has a bell-shaped form, as shown in Figure 11.5. A suitableproposal distribution is therefore the Cauchy (11.8) because this too is bell-shaped
and because we can use the transformation method, discussed earlier, to sample fromit. We need to generalize the Cauchy slightly to ensure that it nowhere has a smaller
value than the gamma distribution. This can be achieved by transforming a uniform
random variable y using z = b tan y + c, which gives random numbers distributed
according to.

530

11. SAMPLING METHODS
Figure 11.5

Plot showing the gamma distribution given by (11.15) as the green
curve, with a scaled Cauchy proposal distribution shown by the red
curve. Samples from the gamma
distribution can be obtained bysampling from the Cauchy andthen applying the rejection sampling criterion.

0.15

0.1
p(z)
0.05

0

0

10

20

30

z

q(z) =k.
1 + (z − c)2 /b2(11.16)The minimum reject rate is obtained by setting c = a − 1, b2 = 2a − 1 and choosing the constant k to be as small as possible while still satisfying the requirement
kq(z)  p(z). The resulting comparison function is also illustrated in Figure 11.5.

11.1.3 Adaptive rejection sampling

Exercise 11.9
Figure 11.6

In many instances where we might wish to apply rejection sampling, it proves
difﬁcult to determine a suitable analytic form for the envelope distribution q(z). An
alternative approach is to construct the envelope function on the ﬂy based on measured values of the distribution p(z) (Gilks and Wild, 1992). Construction of anenvelope function is particularly straightforward for cases in which p(z) is log concave, in other words when ln p(z) has derivatives that are nonincreasing functionsof z. The construction of a suitable envelope function is illustrated graphically in
Figure 11.6.
The function ln p(z) and its gradient are evaluated at some initial set of grid
points, and the intersections of the resulting tangent lines are used to construct theenvelope function. Next a sample value is drawn from the envelope distribution.
This is straightforward because the log of the envelope distribution is a succession
In the case of distributions that arelog concave, an envelope function ln p(z)for use in rejection sampling can beconstructed using the tangent lines
computed at a set of grid points. If asample point is rejected, it is added
to the set of grid points and used toreﬁne the envelope distribution.

z1

z2

z3

z

531

11.1. Basic Sampling Algorithms
Figure 11.7

Illustrative example of rejection
sampling involving sampling from a
Gaussian distribution p(z) shown by
the green curve, by using rejection
sampling from a proposal distribution q(z) that is also Gaussian
and whose scaled version kq(z) isshown by the red curve.

0.5
p(z)

0.25

0
−5

0

z

5

of linear functions, and hence the envelope distribution itself comprises a piecewise
exponential distribution of the form
q(z) = ki λi exp {−λi (z − zi−1 )}zi−1 < z  zi .

(11.17)

Once a sample has been drawn, the usual rejection criterion can be applied. If thesample is accepted, then it will be a draw from the desired distribution. If, however,the sample is rejected, then it is incorporated into the set of grid points, a new tangent
line is computed, and the envelope function is thereby reﬁned. As the number ofgrid points increases, so the envelope function becomes a better approximation of
the desired distribution p(z) and the probability of rejection decreases.
A variant of the algorithm exists that avoids the evaluation of derivatives (Gilks,
1992). The adaptive rejection sampling framework can also be extended to distributions that are not log concave, simply by following each rejection sampling stepwith a Metropolis-Hastings step (to be discussed in Section 11.2.2), giving rise toadaptive rejection Metropolis sampling (Gilks et al., 1995).
Clearly for rejection sampling to be of practical value, we require that the comparison function be close to the required distribution so that the rate of rejection is
kept to a minimum. Now let us examine what happens when we try to use rejection
sampling in spaces of high dimensionality. Consider, for the sake of illustration,a somewhat artiﬁcial problem in which we wish to sample from a zero-mean multivariate Gaussian distribution with covariance σp2 I, where I is the unit matrix, by
rejection sampling from a proposal distribution that is itself a zero-mean Gaussian
distribution having covariance σq2 I. Obviously, we must have σq2  σp2 in order that
there exists a k such that kq(z)  p(z). In D-dimensions the optimum value of kis given by k = (σq /σp )D , as illustrated for D = 1 in Figure 11.7. The acceptance
rate will be the ratio of volumes under p(z) and kq(z), which, because both distributions are normalized, is just 1/k. Thus the acceptance rate diminishes exponentiallywith dimensionality. Even if σq exceeds σp by just one percent, for D = 1, 000 theacceptance ratio will be approximately 1/20, 000. In this illustrative example thecomparison function is close to the required distribution. For more practical examples, where the desired distribution may be multimodal and sharply peaked, it will
be extremely difﬁcult to ﬁnd a good proposal distribution and comparison function.

532

11. SAMPLING METHODS
Figure 11.8

Importance sampling addresses the problem of evaluating the expectation of a function f (z) with respect to a distribution p(z)from which it is difﬁcult to draw samples directly. Instead, samples {z (l) } are drawn
from a simpler distribution q(z), and thecorresponding terms in the summation are
weighted by the ratios p(z (l) )/q(z (l) ).

p(z)

q(z)

f (z)

z

Furthermore, the exponential decrease of acceptance rate with dimensionality is a
generic feature of rejection sampling. Although rejection can be a useful techniquein one or two dimensions it is unsuited to problems of high dimensionality. It can,however, play a role as a subroutine in more sophisticated algorithms for samplingin high dimensional spaces.

11.1.4 Importance sampling
One of the principal reasons for wishing to sample from complicated probability
distributions is to be able to evaluate expectations of the form (11.1). The techniqueof importance sampling provides a framework for approximating expectations directly but does not itself provide a mechanism for drawing samples from distributionp(z).
The ﬁnite sum approximation to the expectation, given by (11.2), depends on
being able to draw samples from the distribution p(z). Suppose, however, that it isimpractical to sample directly from p(z) but that we can evaluate p(z) easily for any
given value of z. One simplistic strategy for evaluating expectations would be todiscretize z-space into a uniform grid and to evaluate the integrand as a sum of theformLp(z(l) )f (z(l) ).
(11.18)E[f ] l=1

An obvious problem with this approach is that the number of terms in the summation
grows exponentially with the dimensionality of z. Furthermore, as we have alreadynoted, the kinds of probability distributions of interest will often have much of theirmass conﬁned to relatively small regions of z space and so uniform sampling will bevery inefﬁcient because in high-dimensional problems, only a very small proportionof the samples will make a signiﬁcant contribution to the sum. We would really liketo choose the sample points to fall in regions where p(z) is large, or ideally where
the product p(z)f (z) is large.
As in the case of rejection sampling, importance sampling is based on the useof a proposal distribution q(z) from which it is easy to draw samples, as illustrated
in Figure 11.8. We can then express the expectation in the form of a ﬁnite sum over

11.1. Basic Sampling Algorithms
samples {z(l) } drawn from q(z)533E[f ] =

f (z)p(z) dz



=

f (z)

p(z)
q(z) dzq(z)1  p(z(l) )f (z(l) ).
Lq(z(l) )L(11.19)l=1

The quantities rl = p(z(l) )/q(z(l) ) are known as importance weights, and they correct the bias introduced by sampling from the wrong distribution. Note that, unlike
rejection sampling, all of the samples generated are retained.
It will often be the case that the distribution p(z) can only be evaluated up to anormalization constant, so that p(z) = 
p(z)/Zp where p(z) can be evaluated easily,whereas Zp is unknown. Similarly, we may wish to use an importance samplingq (z)/Zq , which has the same property. We then have
distribution q(z) = f (z)p(z) dz
E[f ] =

Zq

p(z)
f (z)
q(z) dz
=
Zp

q (z)
Zq 1 
rl f (z(l) ).
Zp L
L



(11.20)l=1

rl = 
p(z(l) )/
q (z(l) ). We can use the same sample set to evaluate the ratio
where Zp /Zq with the result


Zp
1
p(z)

q(z) dz=p(z) dz =

Zq
Zq

q (z)
1

rl
L
L



(11.21)l=1

and henceE[f ] L


wl f (z(l) )(11.22)p(z(l) )/q(z(l) ).
p(z(m) )/q(z(m) )m

(11.23)l=1

where we have deﬁned
wl = 


rl

rm
m

=

As with rejection sampling, the success of the importance sampling approach
depends crucially on how well the sampling distribution q(z) matches the desired

534

11. SAMPLING METHODS
distribution p(z). If, as is often the case, p(z)f (z) is strongly varying and has a signiﬁcant proportion of its mass concentrated over relatively small regions of z space,then the set of importance weights {rl } may be dominated by a few weights having large values, with the remaining weights being relatively insigniﬁcant. Thus theeffective sample size can be much smaller than the apparent sample size L. The problem is even more severe if none of the samples falls in the regions where p(z)f (z)is large. In that case, the apparent variances of rl and rl f (z(l) ) may be small eventhough the estimate of the expectation may be severely wrong. Hence a major drawback of the importance sampling method is the potential to produce results that arearbitrarily in error and with no diagnostic indication. This also highlights a key requirement for the sampling distribution q(z), namely that it should not be small orzero in regions where p(z) may be signiﬁcant.
For distributions deﬁned in terms of a graphical model, we can apply the importance sampling technique in various ways. For discrete variables, a simple approachis called uniform sampling. The joint distribution for a directed graph is deﬁnedby (11.4). Each sample from the joint distribution is obtained by ﬁrst setting thosevariables zi that are in the evidence set equal to their observed values. Each of theremaining variables is then sampled independently from a uniform distribution overthe space of possible instantiations. To determine the corresponding weight associq (z) is uniform overated with a sample z(l) , we note that the sampling distribution 
the possible choices for z, and that p(z|x) = p(z), where x denotes the subset ofvariables that are observed, and the equality follows from the fact that every sample
z that is generated is necessarily consistent with the evidence. Thus the weights rl
are simply proportional to p(z). Note that the variables can be sampled in any order.
This approach can yield poor results if the posterior distribution is far from uniform,
as is often the case in practice.
An improvement on this approach is called likelihood weighted sampling (Fungand Chang, 1990; Shachter and Peot, 1990) and is based on ancestral sampling of
the variables. For each variable in turn, if that variable is in the evidence set, then it
is just set to its instantiated value. If it is not in the evidence set, then it is sampledfrom the conditional distribution p(zi |pai ) in which the conditioning variables are
set to their currently sampled values. The weighting associated with the resulting
sample z is then given byr(z) =

 p(zi |pa )  p(zi |pa )
i

zi ∈e

p(zi |pai )

i

zi ∈e

1

=



p(zi |pai ).

(11.24)zi ∈eThis method can be further extended using self-importance sampling (Shachter andPeot, 1990) in which the importance sampling distribution is continually updated toreﬂect the current estimated posterior distribution.

11.1.5 Sampling-importance-resampling
The rejection sampling method discussed in Section 11.1.2 depends in part for
its success on the determination of a suitable value for the constant k. For many
pairs of distributions p(z) and q(z), it will be impractical to determine a suitable11.1. Basic Sampling Algorithms

535

value for k in that any value that is sufﬁciently large to guarantee a bound on thedesired distribution will lead to impractically small acceptance rates.
As in the case of rejection sampling, the sampling-importance-resampling (SIR)
approach also makes use of a sampling distribution q(z) but avoids having to determine the constant k. There are two stages to the scheme. In the ﬁrst stage,
L samples z(1) , . . . , z(L) are drawn from q(z). Then in the second stage, weights
w1 , . . . , wL are constructed using (11.23). Finally, a second set of L samples isdrawn from the discrete distribution (z(1) , . . . , z(L) ) with probabilities given by theweights (w1 , . . . , wL ).
The resulting L samples are only approximately distributed according to p(z),but the distribution becomes correct in the limit L → ∞. To see this, consider theunivariate case, and note that the cumulative distribution of the resampled values isgiven byp(z  a) =wll : z (l )  a



=

(l )
 a)
p(z (l) )/q(z (l) )l I(zp(z (l) )/q(z (l) )l

(11.25)where I(.) is the indicator function (which equals 1 if its argument is true and 0
otherwise). Taking the limit L → ∞, and assuming suitable regularity of the distributions, we can replace the sums by integrals weighted according to the original
sampling distribution q(z)I(z  a) {p(z)/q(z)} q(z) dzp(z  a) ={
p(z)/q(z)} q(z) dzI(z  a)p(z) dz=p(z) dzI(z  a)p(z) dz(11.26)=
which is the cumulative distribution function of p(z). Again, we see that the normalization of p(z) is not required.
For a ﬁnite value of L, and a given initial sample set, the resampled values will
only approximately be drawn from the desired distribution. As with rejection sampling, the approximation improves as the sampling distribution q(z) gets closer tothe desired distribution p(z). When q(z) = p(z), the initial samples (z(1) , . . . , z(L) )have the desired distribution, and the weights wn = 1/L so that the resampled values
also have the desired distribution.
If moments with respect to the distribution p(z) are required, then they can be

536

11. SAMPLING METHODS
evaluated directly using the original samples together with the weights, becausef (z)p(z) dz
E[f (z)] =
f (z)[p(z)/q(z)]q(z) dz=[
p(z)/q(z)]q(z) dz
L

wl f (zl ).

(11.27)l=1

11.1.6 Sampling and the EM algorithm
In addition to providing a mechanism for direct implementation of the Bayesianframework, Monte Carlo methods can also play a role in the frequentist paradigm,
for example to ﬁnd maximum likelihood solutions. In particular, sampling methods
can be used to approximate the E step of the EM algorithm for models in which the
E step cannot be performed analytically. Consider a model with hidden variablesZ, visible (observed) variables X, and parameters θ. The function that is optimized
with respect to θ in the M step is the expected complete-data log likelihood, givenby(11.28)Q(θ, θ old ) = p(Z|X, θ old ) ln p(Z, X|θ) dZ.
We can use sampling methods to approximate this integral by a ﬁnite sum over samples {Z(l) }, which are drawn from the current estimate for the posterior distributionp(Z|X, θ old ), so that1
ln p(Z(l) , X|θ).
L
L

Q(θ, θ old ) (11.29)l=1

The Q function is then optimized in the usual way in the M step. This procedure iscalled the Monte Carlo EM algorithm.
It is straightforward to extend this to the problem of ﬁnding the mode of theposterior distribution over θ (the MAP estimate) when a prior distribution p(θ) has
been deﬁned, simply by adding ln p(θ) to the function Q(θ, θ old ) before performing
the M step.
A particular instance of the Monte Carlo EM algorithm, called stochastic EM,
arises if we consider a ﬁnite mixture model, and draw just one sample at each E step.
Here the latent variable Z characterizes which of the K components of the mixture
is responsible for generating each data point. In the E step, a sample of Z is takenfrom the posterior distribution p(Z|X, θ old ) where X is the data set. This effectively
makes a hard assignment of each data point to one of the components in the mixture.
In the M step, this sampled approximation to the posterior distribution is used to
update the model parameters in the usual way.

11.2. Markov Chain Monte Carlo537

Now suppose we move from a maximum likelihood approach to a full Bayesiantreatment in which we wish to sample from the posterior distribution over the parameter vector θ. In principle, we would like to draw samples from the joint posteriorp(θ, Z|X), but we shall suppose that this is computationally difﬁcult. Suppose further that it is relatively straightforward to sample from the complete-data parameter
posterior p(θ|Z, X). This inspires the data augmentation algorithm, which alternates between two steps known as the I-step (imputation step, analogous to an Estep) and the P-step (posterior step, analogous to an M step).
IP Algorithm
I-step. We wish to sample from p(Z|X) but we cannot do this directly. We
therefore note the relation
p(Z|X) = p(Z|θ, X)p(θ|X) dθ
(11.30)and hence for l = 1, . . . , L we ﬁrst draw a sample θ (l) from the current estimate for p(θ|X), and then use this to draw a sample Z(l) from p(Z|θ (l) , X).
P-step. Given the relationp(θ|X) =p(θ|Z, X)p(Z|X) dZ(11.31)

we use the samples {Z(l) } obtained from the I-step to compute a revised
estimate of the posterior distribution over θ given by1
p(θ|Z(l) , X).
L
Lp(θ|X) (11.32)

l=1

By assumption, it will be feasible to sample from this approximation in the
I-step.
Note that we are making a (somewhat artiﬁcial) distinction between parameters θand hidden variables Z. From now on, we blur this distinction and focus simply on
the problem of drawing samples from a given posterior distribution.

11.2. Markov Chain Monte CarloIn the previous section, we discussed the rejection sampling and importance sampling strategies for evaluating expectations of functions, and we saw that they sufferfrom severe limitations particularly in spaces of high dimensionality. We therefore
turn in this section to a very general and powerful framework called Markov chain
Monte Carlo (MCMC), which allows sampling from a large class of distributions,538

11. SAMPLING METHODS

Section 11.2.1

and which scales well with the dimensionality of the sample space. Markov chain
Monte Carlo methods have their origins in physics (Metropolis and Ulam, 1949),
and it was only towards the end of the 1980s that they started to have a signiﬁcant
impact in the ﬁeld of statistics.
As with rejection and importance sampling, we again sample from a proposal
distribution. This time, however, we maintain a record of the current state z(τ ) , andthe proposal distribution q(z|z(τ ) ) depends on this current state, and so the sequenceof samples z(1) , z(2) , . . . forms a Markov chain. Again, if we write p(z) = 
p(z)/Zp ,p(z) can readily be evaluated for any given value of z, although
we will assume that 
the value of Zp may be unknown. The proposal distribution itself is chosen to be
sufﬁciently simple that it is straightforward to draw samples from it directly. At
each cycle of the algorithm, we generate a candidate sample z from the proposal
distribution and then accept the sample according to an appropriate criterion.
In the basic Metropolis algorithm (Metropolis et al., 1953), we assume that the
proposal distribution is symmetric, that is q(zA |zB ) = q(zB |zA ) for all values ofzA and zB . The candidate sample is then accepted with probabilityp(z )
 (τ ).
(11.33)A(z , z ) = min 1,p(z(τ ) )This can be achieved by choosing a random number u with uniform distribution overthe unit interval (0, 1) and then accepting the sample if A(z , z(τ ) ) > u. Note that
if the step from z(τ ) to z causes an increase in the value of p(z), then the candidate
point is certain to be kept.
If the candidate sample is accepted, then z(τ +1) = z , otherwise the candidate
point z is discarded, z(τ +1) is set to z(τ ) and another candidate sample is drawn
from the distribution q(z|z(τ +1) ). This is in contrast to rejection sampling, where rejected samples are simply discarded. In the Metropolis algorithm when a candidate
point is rejected, the previous sample is included instead in the ﬁnal list of samples,leading to multiple copies of samples. Of course, in a practical implementation,
only a single copy of each retained sample would be kept, along with an integer
weighting factor recording how many times that state appears. As we shall see, aslong as q(zA |zB ) is positive for any values of zA and zB (this is a sufﬁcient but
not necessary condition), the distribution of z(τ ) tends to p(z) as τ → ∞. It shouldbe emphasized, however, that the sequence z(1) , z(2) , . . . is not a set of independent
samples from p(z) because successive samples are highly correlated. If we wish toobtain independent samples, then we can discard most of the sequence and just retain every M th sample. For M sufﬁciently large, the retained samples will for allpractical purposes be independent. Figure 11.9 shows a simple illustrative example of sampling from a two-dimensional Gaussian distribution using the Metropolis
algorithm in which the proposal distribution is an isotropic Gaussian.
Further insight into the nature of Markov chain Monte Carlo algorithms can begleaned by looking at the properties of a speciﬁc example, namely a simple random

539

11.2. Markov Chain Monte Carlo
Figure 11.9

A simple illustration using Metropo3
lis algorithm to sample from a
Gaussian distribution whose one
standard-deviation contour is shown 2.5
by the ellipse. The proposal distribution is an isotropic Gaussian distribution whose standard deviation is2
0.2. Steps that are accepted areshown as green lines, and rejected
steps are shown in red. A total of 1.5
150 candidate samples are generated, of which 43 are rejected.
1

0.5
0

0

0.5

1

1.5

2

2.5

3

walk. Consider a state space z consisting of the integers, with probabilitiesp(z (τ +1) = z (τ ) ) = 0.5p(zp(zExercise 11.10(τ +1)(τ +1)

=z
=z

(τ )
(τ )

(11.34)+ 1) = 0.25

(11.35)− 1) = 0.25

(11.36)

where z (τ ) denotes the state at step τ . If the initial state is z (1) = 0, then by symmetry the expected state at time τ will also be zero E[z (τ ) ] = 0, and similarly it iseasily seen that E[(z (τ ) )2 ] = τ /2. Thus after τ steps, the random walk has only travelled a distance that on average is proportional to the square root of τ . This squareroot dependence is typical of random walk behaviour and shows that random walks
are very inefﬁcient in exploring the state space. As we shall see, a central goal indesigning Markov chain Monte Carlo methods is to avoid random walk behaviour.

11.2.1 Markov chains
Before discussing Markov chain Monte Carlo methods in more detail, it is useful to study some general properties of Markov chains in more detail. In particular,
we ask under what circumstances will a Markov chain converge to the desired distribution. A ﬁrst-order Markov chain is deﬁned to be a series of random variables
z(1) , . . . , z(M ) such that the following conditional independence property holds form ∈ {1, . . . , M − 1}p(z(m+1) |z(1) , . . . , z(m) ) = p(z(m+1) |z(m) ).

(11.37)This of course can be represented as a directed graph in the form of a chain, an example of which is shown in Figure 8.38. We can then specify the Markov chain bygiving the probability distribution for the initial variable p(z(0) ) together with the

540

11. SAMPLING METHODS
conditional probabilities for subsequent variables in the form of transition probabilities Tm (z(m) , z(m+1) ) ≡ p(z(m+1) |z(m) ). A Markov chain is called homogeneous
if the transition probabilities are the same for all m.
The marginal probability for a particular variable can be expressed in terms ofthe marginal probability for the previous variable in the chain in the formp(z(m+1) |z(m) )p(z(m) ).
(11.38)p(z(m+1) ) =z(m)A distribution is said to be invariant, or stationary, with respect to a Markov chain
if each step in the chain leaves that distribution invariant. Thus, for a homogeneous
Markov chain with transition probabilities T (z , z), the distribution p (z) is invariantif
T (z , z)p (z ).
(11.39)p (z) =z

Note that a given Markov chain may have more than one invariant distribution. For
instance, if the transition probabilities are given by the identity transformation, thenany distribution will be invariant.
A sufﬁcient (but not necessary) condition for ensuring that the required distribution p(z) is invariant is to choose the transition probabilities to satisfy the property
of detailed balance, deﬁned byp (z)T (z, z ) = p (z )T (z , z)(11.40)for the particular distribution p (z). It is easily seen that a transition probability
that satisﬁes detailed balance with respect to a particular distribution will leave that
distribution invariant, becausep (z )T (z , z) =p (z)T (z, z ) = p (z)p(z |z) = p (z). (11.41)
z

z

z

A Markov chain that respects detailed balance is said to be reversible.
Our goal is to use Markov chains to sample from a given distribution. We canachieve this if we set up a Markov chain such that the desired distribution is invariant.
However, we must also require that for m → ∞, the distribution p(z(m) ) converges
to the required invariant distribution p (z), irrespective of the choice of initial distribution p(z(0) ). This property is called ergodicity, and the invariant distribution
is then called the equilibrium distribution. Clearly, an ergodic Markov chain canhave only one equilibrium distribution. It can be shown that a homogeneous Markov
chain will be ergodic, subject only to weak restrictions on the invariant distribution
and the transition probabilities (Neal, 1993).
In practice we often construct the transition probabilities from a set of ‘base’transitions B1 , . . . , BK . This can be achieved through a mixture distribution of theformKαk Bk (z , z)(11.42)T (z , z) =k=1

11.2. Markov Chain Monte Carlo541

for some set of mixing coefﬁcients α1 , . . . , αK satisfying αk  0 and k αk = 1.
Alternatively, the base transitions may be combined through successive application,
so thatT (z , z) =...
B1 (z , z1 ) . . . BK−1 (zK−2 , zK−1 )BK (zK−1 , z). (11.43)
z1

zn−1

If a distribution is invariant with respect to each of the base transitions, then obviously it will also be invariant with respect to either of the T (z , z) given by (11.42)
or (11.43). For the case of the mixture (11.42), if each of the base transitions satisﬁes detailed balance, then the mixture transition T will also satisfy detailed balance. This does not hold for the transition probability constructed using (11.43), although by symmetrizing the order of application of the base transitions, in the form
B1 , B2 , . . . , BK , BK , . . . , B2 , B1 , detailed balance can be restored. A common example of the use of composite transition probabilities is where each base transitionchanges only a subset of the variables.

11.2.2 The Metropolis-Hastings algorithm
Earlier we introduced the basic Metropolis algorithm, without actually demonstrating that it samples from the required distribution. Before giving a proof, we
ﬁrst discuss a generalization, known as the Metropolis-Hastings algorithm (Hastings, 1970), to the case where the proposal distribution is no longer a symmetricfunction of its arguments. In particular at step τ of the algorithm, in which the current state is z(τ ) , we draw a sample z from the distribution qk (z|z(τ ) ) and thenaccept it with probability Ak (z , zτ ) where
p(z )qk (z(τ ) |z ) (τ ).
(11.44)Ak (z , z ) = min 1,p(z(τ ) )qk (z |z(τ ) )Here k labels the members of the set of possible transitions being considered. Again,
the evaluation of the acceptance criterion does not require knowledge of the normalp(z)/Zp . For a symmetricizing constant Zp in the probability distribution p(z) = 
proposal distribution the Metropolis-Hastings criterion (11.44) reduces to the standard Metropolis criterion given by (11.33).
We can show that p(z) is an invariant distribution of the Markov chain deﬁned
by the Metropolis-Hastings algorithm by showing that detailed balance, deﬁned by(11.40), is satisﬁed. Using (11.44) we havep(z)qk (z|z )Ak (z , z) = min (p(z)qk (z|z ), p(z )qk (z |z))= min (p(z )qk (z |z), p(z)qk (z|z ))= p(z )qk (z |z)Ak (z, z )(11.45)as required.
The speciﬁc choice of proposal distribution can have a marked effect on theperformance of the algorithm. For continuous state spaces, a common choice is a
Gaussian centred on the current state, leading to an important trade-off in determining the variance parameter of this distribution. If the variance is small, then the

542

11. SAMPLING METHODS
Figure 11.10

Schematic illustration of the use of an isotropic
Gaussian proposal distribution (blue circle) tosample from a correlated multivariate Gaussian
distribution (red ellipse) having very different standard deviations in different directions, using theMetropolis-Hastings algorithm. In order to keep
the rejection rate low, the scale ρ of the proposal
distribution should be on the order of the smallest
standard deviation σmin , which leads to random
walk behaviour in which the number of steps sep- σminarating states that are approximately independentis of order (σmax /σmin )2 where σmax is the largest
standard deviation.

σmax
ρ

proportion of accepted transitions will be high, but progress through the state spacetakes the form of a slow random walk leading to long correlation times. However,if the variance parameter is large, then the rejection rate will be high because, in thekind of complex problems we are considering, many of the proposed steps will beto states for which the probability p(z) is low. Consider a multivariate distributionp(z) having strong correlations between the components of z, as illustrated in Figure 11.10. The scale ρ of the proposal distribution should be as large as possible
without incurring high rejection rates. This suggests that ρ should be of the sameorder as the smallest length scale σmin . The system then explores the distribution
along the more extended direction by means of a random walk, and so the numberof steps to arrive at a state that is more or less independent of the original state isof order (σmax /σmin )2 . In fact in two dimensions, the increase in rejection rate as ρ
increases is offset by the larger steps sizes of those transitions that are accepted, andmore generally for a multivariate Gaussian the number of steps required to obtain
independent samples scales like (σmax /σ2 )2 where σ2 is the second-smallest standard deviation (Neal, 1993). These details aside, it remains the case that if the lengthscales over which the distributions vary are very different in different directions, thenthe Metropolis Hastings algorithm can have very slow convergence.

11.3. Gibbs Sampling
Gibbs sampling (Geman and Geman, 1984) is a simple and widely applicable Markov
chain Monte Carlo algorithm and can be seen as a special case of the MetropolisHastings algorithm.
Consider the distribution p(z) = p(z1 , . . . , zM ) from which we wish to sample,
and suppose that we have chosen some initial state for the Markov chain. Each stepof the Gibbs sampling procedure involves replacing the value of one of the variables
by a value drawn from the distribution of that variable conditioned on the values of
the remaining variables. Thus we replace zi by a value drawn from the distributionp(zi |z\i ), where zi denotes the ith component of z, and z\i denotes z1 , . . . , zM but
with zi omitted. This procedure is repeated either by cycling through the variables

11.3. Gibbs Sampling543

in some particular order or by choosing the variable to be updated at each step at
random from some distribution.
For example, suppose we have a distribution p(z1 , z2 , z3 ) over three variables,
(τ )
(τ )
(τ )
and at step τ of the algorithm we have selected values z1 , z2 and z3 . We ﬁrst
(τ )(τ +1)
replace z1 by a new value z1
obtained by sampling from the conditional distribution
(τ )
(τ )
p(z1 |z2 , z3 ).
(11.46)(τ )(τ +1)

by a value z2

Next we replace z2
distribution

obtained by sampling from the conditional(τ +1)p(z2 |z1(τ ), z3 )(11.47)so that the new value for z1 is used straight away in subsequent sampling steps. Then(τ +1)
drawn from
we update z3 with a sample z3(τ +1)p(z3 |z1(τ +1), z2)(11.48)and so on, cycling through the three variables in turn.
Gibbs Sampling
1. Initialize {zi : i = 1, . . . , M }2. For τ = 1, . . . , T :(τ +1)

(τ )

(τ )

(τ )

– Sample z1∼ p(z1 |z2 , z3 , . . . , zM ).

– Sample z2..
.

∼ p(z2 |z1(τ +1)(τ +1)(τ +1)(τ +1)(τ )(τ ), z3 , . . . , zM ).
(τ +1)(τ )(τ )– Sample zj∼ p(zj |z1, . . . , zj−1 , zj +1 , . . . , zM ).
..
.
(τ +1)(τ +1)(τ +1)(τ +1)– Sample zM∼ p(zM |z1, z2, . . . , zM −1 ).

Josiah Willard Gibbs
1839–1903
Gibbs spent almost his entire life living in a house built by his father in
New Haven, Connecticut. In 1863,
Gibbs was granted the ﬁrst PhD inengineering in the United States,
and in 1871 he was appointed tothe ﬁrst chair of mathematical physics in the UnitedStates at Yale, a post for which he received no salarybecause at the time he had no publications. He developed the ﬁeld of vector analysis and made contributions to crystallography and planetary orbits. His
most famous work, entitled On the Equilibrium of Heterogeneous Substances, laid the foundations for thescience of physical chemistry.

544

11. SAMPLING METHODSTo show that this procedure samples from the required distribution, we ﬁrst of
all note that the distribution p(z) is an invariant of each of the Gibbs sampling steps
individually and hence of the whole Markov chain. This follows from the fact thatwhen we sample from p(zi |{z\i ), the marginal distribution p(z\i ) is clearly invariant
because the value of z\i is unchanged. Also, each step by deﬁnition samples from thecorrect conditional distribution p(zi |z\i ). Because these conditional and marginal
distributions together specify the joint distribution, we see that the joint distribution
is itself invariant.
The second requirement to be satisﬁed in order that the Gibbs sampling procedure samples from the correct distribution is that it be ergodic. A sufﬁcient condition
for ergodicity is that none of the conditional distributions be anywhere zero. If thisis the case, then any point in z space can be reached from any other point in a ﬁnite
number of steps involving one update of each of the component variables. If thisrequirement is not satisﬁed, so that some of the conditional distributions have zeros,then ergodicity, if it applies, must be proven explicitly.
The distribution of initial states must also be speciﬁed in order to complete thealgorithm, although samples drawn after many iterations will effectively become
independent of this distribution. Of course, successive samples from the Markov
chain will be highly correlated, and so to obtain samples that are nearly independent
it will be necessary to subsample the sequence.
We can obtain the Gibbs sampling procedure as a particular instance of theMetropolis-Hastings algorithm as follows. Consider a Metropolis-Hastings sampling
step involving the variable zk in which the remaining variables z\k remain ﬁxed, andfor which the transition probability from z to z is given by qk (z |z) = p(zk |z\k ).
We note that z\k = z\k because these components are unchanged by the samplingstep. Also, p(z) = p(zk |z\k )p(z\k ). Thus the factor that determines the acceptanceprobability in the Metropolis-Hastings (11.44) is given byp(zk |z\k )p(z\k )p(zk |z\k )p(z )qk (z|z )A(z , z) ===1p(z)qk (z |z)p(zk |z\k )p(z\k )p(zk |z\k )(11.49)where we have used z\k = z\k . Thus the Metropolis-Hastings steps are always
accepted.
As with the Metropolis algorithm, we can gain some insight into the behaviour ofGibbs sampling by investigating its application to a Gaussian distribution. Consider
a correlated Gaussian in two variables, as illustrated in Figure 11.11, having conditional distributions of width l and marginal distributions of width L. The typicalstep size is governed by the conditional distributions and will be of order l. Because
the state evolves according to a random walk, the number of steps needed to obtain
independent samples from the distribution will be of order (L/l)2 . Of course if theGaussian distribution were uncorrelated, then the Gibbs sampling procedure would
be optimally efﬁcient. For this simple problem, we could rotate the coordinate system in order to decorrelate the variables. However, in practical applications it will
generally be infeasible to ﬁnd such transformations.
One approach to reducing random walk behaviour in Gibbs sampling is called
over-relaxation (Adler, 1981). In its original form, this applies to problems for which11.3. Gibbs Sampling
Figure 11.11

Illustration of Gibbs sampling by alter- z2nate updates of two variables whose
distribution is a correlated Gaussian.
The step size is governed by the standard deviation of the conditional distribution (green curve), and is O(l), leading to slow progress in the direction ofelongation of the joint distribution (redellipse). The number of steps needed
to obtain an independent sample fromthe distribution is O((L/l)2 ).

545

L

l

z1

the conditional distributions are Gaussian, which represents a more general class of
distributions than the multivariate Gaussian because, for example, the non-Gaussian
distribution p(z, y) ∝ exp(−z 2 y 2 ) has Gaussian conditional distributions. At eachstep of the Gibbs sampling algorithm, the conditional distribution for a particularcomponent zi has some mean µi and some variance σi2 . In the over-relaxation framework, the value of zi is replaced withzi = µi + α(zi − µi ) + σi (1 − αi2 )1/2 ν(11.50)where ν is a Gaussian random variable with zero mean and unit variance, and α
is a parameter such that −1 < α < 1. For α = 0, the method is equivalent tostandard Gibbs sampling, and for α < 0 the step is biased to the opposite side of themean. This step leaves the desired distribution invariant because if zi has mean µi
and variance σi2 , then so too does zi . The effect of over-relaxation is to encourage
directed motion through state space when the variables are highly correlated. The
framework of ordered over-relaxation (Neal, 1999) generalizes this approach to nonGaussian distributions.
The practical applicability of Gibbs sampling depends on the ease with whichsamples can be drawn from the conditional distributions p(zk |z\k ). In the case ofprobability distributions speciﬁed using graphical models, the conditional distributions for individual nodes depend only on the variables in the corresponding Markov
blankets, as illustrated in Figure 11.12. For directed graphs, a wide choice of conditional distributions for the individual nodes conditioned on their parents will lead toconditional distributions for Gibbs sampling that are log concave. The adaptive rejection sampling methods discussed in Section 11.1.3 therefore provide a frameworkfor Monte Carlo sampling from directed graphs with broad applicability.
If the graph is constructed using distributions from the exponential family, andif the parent-child relationships preserve conjugacy, then the full conditional distributions arising in Gibbs sampling will have the same functional form as the orig-

546

11. SAMPLING METHODS

Figure 11.12 The Gibbs sampling method requires samples
to be drawn from the conditional distribution of a variable conditioned on the remaining variables. For graphical models, thisconditional distribution is a function only of the states of thenodes in the Markov blanket. For an undirected graph this comprises the set of neighbours, as shown on the left, while for a
directed graph the Markov blanket comprises the parents, thechildren, and the co-parents, as shown on the right.

inal conditional distributions (conditioned on the parents) deﬁning each node, andso standard sampling techniques can be employed. In general, the full conditional
distributions will be of a complex form that does not permit the use of standard sampling algorithms. However, if these conditionals are log concave, then sampling canbe done efﬁciently using adaptive rejection sampling (assuming the corresponding
variable is a scalar).
If, at each stage of the Gibbs sampling algorithm, instead of drawing a sample
from the corresponding conditional distribution, we make a point estimate of thevariable given by the maximum of the conditional distribution, then we obtain theiterated conditional modes (ICM) algorithm discussed in Section 8.3.3. Thus ICM
can be seen as a greedy approximation to Gibbs sampling.
Because the basic Gibbs sampling technique considers one variable at a time,
there are strong dependencies between successive samples. At the opposite extreme,
if we could draw samples directly from the joint distribution (an operation that weare supposing is intractable), then successive samples would be independent. We canhope to improve on the simple Gibbs sampler by adopting an intermediate strategy inwhich we sample successively from groups of variables rather than individual variables. This is achieved in the blocking Gibbs sampling algorithm by choosing blocksof variables, not necessarily disjoint, and then sampling jointly from the variables ineach block in turn, conditioned on the remaining variables (Jensen et al., 1995).

11.4. Slice Sampling
We have seen that one of the difﬁculties with the Metropolis algorithm is the sensitivity to step size. If this is too small, the result is slow decorrelation due to random
walk behaviour, whereas if it is too large the result is inefﬁciency due to a high rejection rate. The technique of slice sampling (Neal, 2003) provides an adaptive step sizethat is automatically adjusted to match the characteristics of the distribution. Againp(z).
it requires that we are able to evaluate the unnormalized distribution 
Consider ﬁrst the univariate case. Slice sampling involves augmenting z withan additional variable u and then drawing samples from the joint (z, u) space. We
shall see another example of this approach when we discuss hybrid Monte Carlo in
Section 11.5. The goal is to sample uniformly from the area under the distribution

547

11.4. Slice Sampling
p̃(z)

p̃(z)
zmin

u

z

z (τ )uzmaxzz (τ )(a)(b)

Figure 11.13 Illustration of slice sampling. (a) For a given value z (τ ) , a value of u is chosen uniformly inthe region 0  u  ep(z (τ ) ), which then deﬁnes a ‘slice’ through the distribution, shown by the solid horizontal
lines. (b) Because it is infeasible to sample directly from a slice, a new sample of z is drawn from a region
zmin  z  zmax , which contains the previous value z (τ ) .

given byp(z, u) =where Zp =



1/Zp
0

if 0  u  p(z)otherwise(11.51)p(z) dz. The marginal distribution over z is given by
p(z, u) du = ep(z)
0

1

p(z)
du =
= p(z)
Zp
Zp(11.52)and so we can sample from p(z) by sampling from p(z, u) and then ignoring the uvalues. This can be achieved by alternately sampling z and u. Given the value of zp(z) and then sample u uniformly in the range 0  u  p(z), which is
we evaluate straightforward. Then we ﬁx u and sample z uniformly from the ‘slice’ through thep(z) > u}. This is illustrated in Figure 11.13(a).
distribution deﬁned by {z : In practice, it can be difﬁcult to sample directly from a slice through the distribution and so instead we deﬁne a sampling scheme that leaves the uniform distributionp(z, u) invariant, which can be achieved by ensuring that detailed balance isunder satisﬁed. Suppose the current value of z is denoted z (τ ) and that we have obtained
a corresponding sample u. The next value of z is obtained by considering a region
zmin  z  zmax that contains z (τ ) . It is in the choice of this region that the adaptation to the characteristic length scales of the distribution takes place. We want theregion to encompass as much of the slice as possible so as to allow large moves in z
space while having as little as possible of this region lying outside the slice, because
this makes the sampling less efﬁcient.
One approach to the choice of region involves starting with a region containingz (τ ) having some width w and then testing each of the end points to see if they lie
within the slice. If either end point does not, then the region is extended in that
direction by increments of value w until the end point lies outside the region. A
candidate value z  is then chosen uniformly from this region, and if it lies within theslice, then it forms z (τ +1) . If it lies outside the slice, then the region is shrunk suchthat z  forms an end point and such that the region still contains z (τ ) . Then another548

11. SAMPLING METHODS
candidate point is drawn uniformly from this reduced region and so on, until a valueof z is found that lies within the slice.
Slice sampling can be applied to multivariate distributions by repeatedly sampling each variable in turn, in the manner of Gibbs sampling. This requires that
we are able to compute, for each component zi , a function that is proportional top(zi |z\i ).

11.5. The Hybrid Monte Carlo Algorithm
As we have already noted, one of the major limitations of the Metropolis algorithm
is that it can exhibit random walk behaviour whereby the distance traversed throughthe state space grows only as the square root of the number of steps. The problem
cannot be resolved simply by taking bigger steps as this leads to a high rejection rate.
In this section, we introduce a more sophisticated class of transitions based on an
analogy with physical systems and that has the property of being able to make largechanges to the system state while keeping the rejection probability small. It is applicable to distributions over continuous variables for which we can readily evaluate
the gradient of the log probability with respect to the state variables. We will discuss
the dynamical systems framework in Section 11.5.1, and then in Section 11.5.2 we
explain how this may be combined with the Metropolis algorithm to yield the powerful hybrid Monte Carlo algorithm. A background in physics is not required as thissection is self-contained and the key results are all derived from ﬁrst principles.

11.5.1 Dynamical systems
The dynamical approach to stochastic sampling has its origins in algorithms for
simulating the behaviour of physical systems evolving under Hamiltonian dynamics. In a Markov chain Monte Carlo simulation, the goal is to sample from a given
probability distribution p(z). The framework of Hamiltonian dynamics is exploitedby casting the probabilistic simulation in the form of a Hamiltonian system. In orderto remain in keeping with the literature in this area, we make use of the relevant
dynamical systems terminology where appropriate, which will be deﬁned as we goalong.
The dynamics that we consider corresponds to the evolution of the state variablez = {zi } under continuous time, which we denote by τ . Classical dynamics is described by Newton’s second law of motion in which the acceleration of an object is
proportional to the applied force, corresponding to a second-order differential equation over time. We can decompose a second-order equation into two coupled ﬁrstorder equations by introducing intermediate momentum variables r, corresponding
to the rate of change of the state variables z, having components
ri =

dz i
dτ

(11.53)where the zi can be regarded as position variables in this dynamics perspective. Thus

11.5. The Hybrid Monte Carlo Algorithm549

for each position variable there is a corresponding momentum variable, and the joint
space of position and momentum variables is called phase space.
Without loss of generality, we can write the probability distribution p(z) in theform
1p(z) =
exp (−E(z))(11.54)
Zp
where E(z) is interpreted as the potential energy of the system when in state z. The
system acceleration is the rate of change of momentum and is given by the applied
force, which itself is the negative gradient of the potential energy
dri
∂E(z)
=−
.
dτ
∂zi

(11.55)

It is convenient to reformulate this dynamical system using the Hamiltonian
framework. To do this, we ﬁrst deﬁne the kinetic energy byK(r) =

1 2
1
ri .
r2 =
2
2(11.56)i

The total energy of the system is then the sum of its potential and kinetic energies
H(z, r) = E(z) + K(r)Exercise 11.15(11.57)

where H is the Hamiltonian function. Using (11.53), (11.55), (11.56), and (11.57),
we can now express the dynamics of the system in terms of the Hamiltonian equations given by
dz i
dτ
dri
dτ

William Hamilton
1805–1865
William Rowan Hamilton was an
Irish mathematician and physicist,and child prodigy, who was appointed Professor of Astronomy at
Trinity College, Dublin, in 1827, before he had even graduated. One
of Hamilton’s most important contributions was a new
formulation of dynamics, which played a signiﬁcantrole in the later development of quantum mechanics.

∂H
∂ri
∂H
.
= −
∂zi

=

(11.58)(11.59)His other great achievement was the development ofquaternions, which generalize the concept of complex
numbers by introducing three distinct square roots ofminus one, which satisfy i2 = j 2 = k2 = ijk = −1.
It is said that these equations occurred to him while
walking along the Royal Canal in Dublin with his wife,on 16 October 1843, and he promptly carved theequations into the side of Broome bridge. Although
there is no longer any evidence of the carving, there is
now a stone plaque on the bridge commemorating thediscovery and displaying the quaternion equations.

550

11. SAMPLING METHODS
During the evolution of this dynamical system, the value of the Hamiltonian H isconstant, as is easily seen by differentiation
dH
dτ

=


i

=


i

∂H dz i ∂H dri+∂zi dτ
∂ri dτ
∂H ∂H
∂H ∂H
−
∂zi ∂ri∂ri ∂zi= 0.

(11.60)A second important property of Hamiltonian dynamical systems, known as Liouville’s Theorem, is that they preserve volume in phase space. In other words, if
we consider a region within the space of variables (z, r), then as this region evolves
under the equations of Hamiltonian dynamics, its shape may change but its volume
will not. This can be seen by noting that the ﬂow ﬁeld (rate of change of location inphase space) is given by


dz dr
(11.61),V=dτ dτ
and that the divergence of this ﬁeld vanishes
div V =


i

=


i

∂ dz i∂ dri+∂zi dτ
∂ri dτ
−

∂ ∂H
∂ ∂H
+
∂zi ∂ri∂ri ∂zi= 0.

(11.62)

Now consider the joint distribution over phase space whose total energy is the
Hamiltonian, i.e., the distribution given byp(z, r) =1
exp(−H(z, r)).
ZH

(11.63)Using the two results of conservation of volume and conservation of H, it follows
that the Hamiltonian dynamics will leave p(z, r) invariant. This can be seen byconsidering a small region of phase space over which H is approximately constant.
If we follow the evolution of the Hamiltonian equations for a ﬁnite time, then thevolume of this region will remain unchanged as will the value of H in this region, andhence the probability density, which is a function only of H, will also be unchanged.
Although H is invariant, the values of z and r will vary, and so by integrating
the Hamiltonian dynamics over a ﬁnite time duration it becomes possible to makelarge changes to z in a systematic way that avoids random walk behaviour.
Evolution under the Hamiltonian dynamics will not, however, sample ergodically from p(z, r) because the value of H is constant. In order to arrive at an ergodic
sampling scheme, we can introduce additional moves in phase space that change
the value of H while also leaving the distribution p(z, r) invariant. The simplest
way to achieve this is to replace the value of r with one drawn from its distribution
conditioned on z. This can be regarded as a Gibbs sampling step, and hence from

11.5. The Hybrid Monte Carlo Algorithm

Exercise 11.16

551

Section 11.3 we see that this also leaves the desired distribution invariant. Notingthat z and r are independent in the distribution p(z, r), we see that the conditional
distribution p(r|z) is a Gaussian from which it is straightforward to sample.
In a practical application of this approach, we have to address the problem of
performing a numerical integration of the Hamiltonian equations. This will necessarily introduce numerical errors and so we should devise a scheme that minimizes
the impact of such errors. In fact, it turns out that integration schemes can be devisedfor which Liouville’s theorem still holds exactly. This property will be important in
the hybrid Monte Carlo algorithm, which is discussed in Section 11.5.2. One scheme
for achieving this is called the leapfrog discretization and involves alternately updating discrete-time approximations z and r to the position and momentum variables
using
 ∂E
(
z(τ ))2 ∂zizi (τ + ) = zi (τ ) + ri (τ + /2) ∂E
ri (τ + ) = ri (τ + /2) −(z(τ + )).
2 ∂zi

ri (τ + /2) = ri (τ ) −(11.64)(11.65)(11.66)

We see that this takes the form of a half-step update of the momentum variables withstep size /2, followed by a full-step update of the position variables with step size ,followed by a second half-step update of the momentum variables. If several leapfrog
steps are applied in succession, it can be seen that half-step updates to the momentum
variables can be combined into full-step updates with step size . The successive
updates to position and momentum variables then leapfrog over each other. In orderto advance the dynamics by a time interval τ , we need to take τ / steps. The error
involved in the discretized approximation to the continuous time dynamics will go tozero, assuming a smooth function E(z), in the limit  → 0. However, for a nonzero
 as used in practice, some residual error will remain. We shall see in Section 11.5.2
how the effects of such errors can be eliminated in the hybrid Monte Carlo algorithm.
In summary then, the Hamiltonian dynamical approach involves alternating between a series of leapfrog updates and a resampling of the momentum variables fromtheir marginal distribution.
Note that the Hamiltonian dynamics method, unlike the basic Metropolis algorithm, is able to make use of information about the gradient of the log probability
distribution as well as about the distribution itself. An analogous situation is familiarfrom the domain of function optimization. In most cases where gradient information is available, it is highly advantageous to make use of it. Informally, this follows
from the fact that in a space of dimension D, the additional computational cost ofevaluating a gradient compared with evaluating the function itself will typically be aﬁxed factor independent of D, whereas the D-dimensional gradient vector conveys
D pieces of information compared with the one piece of information given by thefunction itself.

552

11. SAMPLING METHODS

11.5.2 Hybrid Monte Carlo
As we discussed in the previous section, for a nonzero step size , the discretization of the leapfrog algorithm will introduce errors into the integration of the Hamiltonian dynamical equations. Hybrid Monte Carlo (Duane et al., 1987; Neal, 1996)
combines Hamiltonian dynamics with the Metropolis algorithm and thereby removes
any bias associated with the discretization.
Speciﬁcally, the algorithm uses a Markov chain consisting of alternate stochastic
updates of the momentum variable r and Hamiltonian dynamical updates using theleapfrog algorithm. After each application of the leapfrog algorithm, the resulting
candidate state is accepted or rejected according to the Metropolis criterion basedon the value of the Hamiltonian H. Thus if (z, r) is the initial state and (z , r )is the state after the leapfrog integration, then this candidate state is accepted withprobabilitymin (1, exp{H(z, r) − H(z , r )}) .
(11.67)
If the leapfrog integration were to simulate the Hamiltonian dynamics perfectly,
then every such candidate step would automatically be accepted because the valueof H would be unchanged. Due to numerical errors, the value of H may sometimes
decrease, and we would like the Metropolis criterion to remove any bias due to thiseffect and ensure that the resulting samples are indeed drawn from the required distribution. In order for this to be the case, we need to ensure that the update equations
corresponding to the leapfrog integration satisfy detailed balance (11.40). This iseasily achieved by modifying the leapfrog scheme as follows.
Before the start of each leapfrog integration sequence, we choose at random,with equal probability, whether to integrate forwards in time (using step size ) or
backwards in time (using step size −). We ﬁrst note that the leapfrog integration
scheme (11.64), (11.65), and (11.66) is time-reversible, so that integration for L steps
using step size − will exactly undo the effect of integration for L steps using step
size . Next we show that the leapfrog integration preserves phase-space volume
exactly. This follows from the fact that each step in the leapfrog scheme updates
either a zi variable or an ri variable by an amount that is a function only of the othervariable. As shown in Figure 11.14, this has the effect of shearing a region of phasespace while not altering its volume.
Finally, we use these results to show that detailed balance holds. Consider asmall region R of phase space that, under a sequence of L leapfrog iterations ofstep size , maps to a region R . Using conservation of volume under the leapfrog
iteration, we see that if R has volume δV then so too will R . If we choose an initialpoint from the distribution (11.63) and then update it using L leapfrog interactions,the probability of the transition going from R to R is given by1
1exp(−H(R))δV min {1, exp(−H(R) + H(R ))} .
ZH
2(11.68)where the factor of 1/2 arises from the probability of choosing to integrate with apositive step size rather than a negative one. Similarly, the probability of starting in

553

11.5. The Hybrid Monte Carlo Algorithm

ri

ri

zi

zi

Figure 11.14 Each step of the leapfrog algorithm (11.64)–(11.66) modiﬁes either a position variable zi or a
momentum variable ri . Because the change to one variable is a function only of the other, any region in phasespace will be sheared without change of volume.

region R and integrating backwards in time to end up in region R is given by1
1exp(−H(R ))δV min {1, exp(−H(R ) + H(R))} .
ZH
2
Exercise 11.17(11.69)It is easily seen that the two probabilities (11.68) and (11.69) are equal, and hence
detailed balance holds. Note that this proof ignores any overlap between the regions
R and R but is easily generalized to allow for such overlap.
It is not difﬁcult to construct examples for which the leapfrog algorithm returns
to its starting position after a ﬁnite number of iterations. In such cases, the random
replacement of the momentum values before each leapfrog integration will not be
sufﬁcient to ensure ergodicity because the position variables will never be updated.
Such phenomena are easily avoided by choosing the magnitude of the step size at
random from some small interval, before each leapfrog integration.
We can gain some insight into the behaviour of the hybrid Monte Carlo algorithm by considering its application to a multivariate Gaussian. For convenience,
consider a Gaussian distribution p(z) with independent components, for which the
Hamiltonian is given by1 1 2 1 2
z +ri .
(11.70)H(z, r) =
2
σi2 i
2
i

i

Our conclusions will be equally valid for a Gaussian distribution having correlated
components because the hybrid Monte Carlo algorithm exhibits rotational isotropy.
During the leapfrog integration, each pair of phase-space variables zi , ri evolves independently. However, the acceptance or rejection of the candidate point is basedon the value of H, which depends on the values of all of the variables. Thus, asigniﬁcant integration error in any one of the variables could lead to a high probability of rejection. In order that the discrete leapfrog integration be a reasonably

554

11. SAMPLING METHODS
good approximation to the true continuous-time dynamics, it is necessary for theleapfrog integration scale  to be smaller than the shortest length-scale over which
the potential is varying signiﬁcantly. This is governed by the smallest value of σi ,which we denote by σmin . Recall that the goal of the leapfrog integration in hybridMonte Carlo is to move a substantial distance through phase space to a new statethat is relatively independent of the initial state and still achieve a high probability ofacceptance. In order to achieve this, the leapfrog integration must be continued for anumber of iterations of order σmax /σmin .
By contrast, consider the behaviour of a simple Metropolis algorithm with anisotropic Gaussian proposal distribution of variance s2 , considered earlier. In order
to avoid high rejection rates, the value of s must be of order σmin . The exploration ofstate space then proceeds by a random walk and takes of order (σmax /σmin )2 steps
to arrive at a roughly independent state.

11.6. Estimating the Partition Function
As we have seen, most of the sampling algorithms considered in this chapter require only the functional form of the probability distribution up to a multiplicative
constant. Thus if we writepE (z) =1exp(−E(z))
ZE(11.71)then the value of the normalization constant ZE , also known as the partition function, is not needed in order to draw samples from p(z). However, knowledge of thevalue of ZE can be useful for Bayesian model comparison since it represents themodel evidence (i.e., the probability of the observed data given the model), and so
it is of interest to consider how its value might be obtained. We assume that direct
evaluation by summing, or integrating, the function exp(−E(z)) over the state spaceof z is intractable.
For model comparison, it is actually the ratio of the partition functions for twomodels that is required. Multiplication of this ratio by the ratio of prior probabilities
gives the ratio of posterior probabilities, which can then be used for model selection
or model averaging.
One way to estimate a ratio of partition functions is to use importance samplingfrom a distribution with energy function G(z)
exp(−E(z))
ZE
= z
ZG
exp(−G(z))
zexp(−E(z)+ G(z)) exp(−G(z))
z=z exp(−G(z))= EG(z) [exp(−E + G)]exp(−E(z(l) ) + G(z(l) ))(11.72)l11.6. Estimating the Partition Function

555

where {z(l) } are samples drawn from the distribution deﬁned by pG (z). If the distribution pG is one for which the partition function can be evaluated analytically, forexample a Gaussian, then the absolute value of ZE can be obtained.
This approach will only yield accurate results if the importance sampling distribution pG is closely matched to the distribution pE , so that the ratio pE /pG does nothave wide variations. In practice, suitable analytically speciﬁed importance sampling
distributions cannot readily be found for the kinds of complex models considered inthis book.
An alternative approach is therefore to use the samples obtained from a Markov
chain to deﬁne the importance-sampling distribution. If the transition probability for
the Markov chain is given by T (z, z ), and the sample set is given by z(1) , . . . , z(L) ,then the sampling distribution can be written as
1
exp (−G(z)) =T (z(l) , z)ZG
L

(11.73)l=1

which can be used directly in (11.72).
Methods for estimating the ratio of two partition functions require for their success that the two corresponding distributions be reasonably closely matched. This isespecially problematic if we wish to ﬁnd the absolute value of the partition function
for a complex distribution because it is only for relatively simple distributions thatthe partition function can be evaluated directly, and so attempting to estimate theratio of partition functions directly is unlikely to be successful. This problem can betackled using a technique known as chaining (Neal, 1993; Barber and Bishop, 1997),
which involves introducing a succession of intermediate distributions p2 , . . . , pM −1
that interpolate between a simple distribution p1 (z) for which we can evaluate thenormalization coefﬁcient Z1 and the desired complex distribution pM (z). We then
have
Z2 Z3
ZM
ZM
=
···
(11.74)
Z1
Z1 Z2
ZM −1
in which the intermediate ratios can be determined using Monte Carlo methods as
discussed above. One way to construct such a sequence of intermediate systems
is to use an energy function containing a continuous parameter 0  α  1 thatinterpolates between the two distributions
Eα (z) = (1 − α)E1 (z) + αEM (z).

(11.75)

If the intermediate ratios in (11.74) are to be found using Monte Carlo, it may be
more efﬁcient to use a single Markov chain run than to restart the Markov chain foreach ratio. In this case, the Markov chain is run initially for the system p1 and thenafter some suitable number of steps moves on to the next distribution in the sequence.
Note, however, that the system must remain close to the equilibrium distribution ateach stage.

556

11. SAMPLING METHODS

Exercises

11.1 ( ) www Show that the ﬁnite sample estimator f deﬁned by (11.2) has meanequal to E[f ] and variance given by (11.3).
11.2 ( ) Suppose that z is a random variable with uniform distribution over (0, 1) andthat we transform z using y = h−1 (z) where h(y) is given by (11.6). Show that y
has the distribution p(y).
11.3 ( ) Given a random variable z that is uniformly distributed over (0, 1), ﬁnd a transformation y = f (z) such that y has a Cauchy distribution given by (11.8).
11.4 ( ) Suppose that z1 and z2 are uniformly distributed over the unit circle, as
shown in Figure 11.3, and that we make the change of variables given by (11.10)
and (11.11). Show that (y1 , y2 ) will be distributed according to (11.12).
11.5 ( ) www Let z be a D-dimensional random variable having a Gaussian distribution with zero mean and unit covariance matrix, and suppose that the positive deﬁnite
symmetric matrix Σ has the Cholesky decomposition Σ = LLT where L is a lowertriangular matrix (i.e., one with zeros above the leading diagonal). Show that thevariable y = µ + Lz has a Gaussian distribution with mean µ and covariance Σ.
This provides a technique for generating samples from a general multivariate Gaussian using samples from a univariate Gaussian having zero mean and unit variance.
11.6 ( ) www In this exercise, we show more carefully that rejection sampling does
indeed draw samples from the desired distribution p(z). Suppose the proposal distribution is q(z) and show that the probability of a sample value z being accepted isgiven by p(z)/kq(z) where 
p is any unnormalized distribution that is proportional top(z), and the constant k is set to the smallest value that ensures kq(z)  
p(z) for all
values of z. Note that the probability of drawing a value z is given by the probabilityof drawing that value from q(z) times the probability of accepting that value giventhat it has been drawn. Make use of this, along with the sum and product rules ofprobability, to write down the normalized form for the distribution over z, and showthat it equals p(z).
11.7 ( ) Suppose that z has a uniform distribution over the interval [0, 1]. Show that thevariable y = b tan z + c has a Cauchy distribution given by (11.16).
11.8 ( ) Determine expressions for the coefﬁcients ki in the envelope distribution(11.17) for adaptive rejection sampling using the requirements of continuity and normalization.
11.9 ( ) By making use of the technique discussed in Section 11.1.1 for samplingfrom a single exponential distribution, devise an algorithm for sampling from thepiecewise exponential distribution deﬁned by (11.17).

11.10 ( ) Show that the simple random walk over the integers deﬁned by (11.34), (11.35),
and (11.36) has the property that E[(z (τ ) )2 ] = E[(z (τ −1) )2 ] + 1/2 and hence by
induction that E[(z (τ ) )2 ] = τ /2.

557

Exercises
Figure 11.15

A probability distribution over two variables z1 z2and z2 that is uniform over the shaded regions
and that is zero everywhere else.

z1

11.11 ( ) www Show that the Gibbs sampling algorithm, discussed in Section 11.3,
satisﬁes detailed balance as deﬁned by (11.40).
11.12 ( ) Consider the distribution shown in Figure 11.15. Discuss whether the standard
Gibbs sampling procedure for this distribution is ergodic, and therefore whether it
would sample correctly from this distribution
11.13 ( ) Consider the simple 3-node graph shown in Figure 11.16 in which the observed
node x is given by a Gaussian distribution N (x|µ, τ −1 ) with mean µ and precision
τ . Suppose that the marginal distributions over the mean and precision are given
by N (µ|µ0 , s0 ) and Gam(τ |a, b), where Gam(·|·, ·) denotes a gamma distribution.
Write down expressions for the conditional distributions p(µ|x, τ ) and p(τ |x, µ) thatwould be required in order to apply Gibbs sampling to the posterior distributionp(µ, τ |x).
11.14 ( ) Verify that the over-relaxation update (11.50), in which zi has mean µi andvariance σi , and where ν has zero mean and unit variance, gives a value zi withmean µi and variance σi2 .
11.15 ( ) www Using (11.56) and (11.57), show that the Hamiltonian equation (11.58)
is equivalent to (11.53). Similarly, using (11.57) show that (11.59) is equivalent to(11.55).
11.16 ( ) By making use of (11.56), (11.57), and (11.63), show that the conditional distribution p(r|z) is a Gaussian.
Figure 11.16

A graph involving an observed Gaussian variable x withprior distributions over its mean µ and precision τ .

µ

τ

x

558

11. SAMPLING METHODS
11.17 ( ) www Verify that the two probabilities (11.68) and (11.69) are equal, and hence
that detailed balance holds for the hybrid Monte Carlo algorithm.

Appendix A

In Chapter 9, we discussed probabilistic models having discrete latent variables, suchas the mixture of Gaussians. We now explore models in which some, or all, of thelatent variables are continuous. An important motivation for such models is thatmany data sets have the property that the data points all lie close to a manifold ofmuch lower dimensionality than that of the original data space. To see why thismight arise, consider an artificial data set constructed by taking one of the off-line
digits, represented by a 64 x 64 pixel grey-level image, and embedding it in a larger
image of size 100 x 100 by padding with pixels having the value zero (corresponding
to white pixels) in which the location and orientation of the digit is varied at random,
as illustrated in Figure 12.1. Each of the resulting images is represented by a point in
the 100 x 100 = 10, OOO-dimensional data space. However, across a data set of such
images, there are only three degrees offreedom of variability, corresponding to thevertical and horizontal translations and the rotations. The data points will therefore
live on a subspace of the data space whose intrinsic dimensionality is three. Note

559

560

12. CONTINUOUS LATENT VARIABLES

Figure 12.1

A synthetic data sel obtained by taking one of the off-line digit images and creating multiple copies in each of which the digit has undergone a random displacement and rotation
within some larger image field. The resulting images each have 100 )( 100 = 10.000

pixels.

that the manifold will be nonlinear because. for instance. if we translate the digit
past a particular pixel, that pixel value will go from zero (white) 10 one (black) andback to zero again. which is clearly a nonlinear function of the digit position. In

this example. !.he lranslation and rotation parameters are latent variables because weAppendiX A

SeCTion 8.1..JSection 12.1

observe only the image vectors and are not told which values of the translation or
rotation variables were used to create them.
For real digit image data, there will be a funher degree of freedom arising from
scaling. Moreover there will be multiple addilional degrees of freedom associaled
wilh more complex deformations due to the variability in an individual's wriling
3S well as lhe differences in writing slyles between individuals. evenheless. thenumber of such degrees of freedom will be small compared to the dimensionality ofIhe data set.
Another example is provided by the oil flow data set. in which (for a given geometrical configuration of the gas, WOller, and oil phases) there are only two degrees
of freedom of variability corresponding to the fraction of oil in the pipe and the fraction of water (the fraction of gas Ihen being determined). Ahhough the data spacecomprises 12 measuremenlS, a data set of points will lie close to a Iwo-dimensional
manifold embedded within this space. In this case, the manifold comprises scveral
distinct segments corresponding to different flow regimes. each such segment beinga (noisy) continuous two-dimensional manifold. If our goal is data compression. or
density modelling, then there can be benefits in exploiling this manifold struclUre.
In praclice. the data points will not be confined precisely to a smooth lowdimensional manifold, and we can interpret the departures of data points from the
manifold as ·noise'. This leads naturally to a generative view of such models inwhich we first select a poinl within the manifold according to some latent variable
distribution and then generate an observed data point by :ldding noise, drawn fromsome conditional distribution of the data varillbles given the latent varillbles.
Thc simplest continuous latent variable model assumes Gaussian distributions
for both thc latent and observed variables and makes use of a linear,Gaussian dependence of the observed variables on Ihe slate of the latent variables. This leads
to a probabilislic fonnulation of the well-known technique of principal component
analysis (PeA), as well as 10 a related model called factor analysis.
In this chapter w will begin wilh a slandard, nonprobabilistic treatment of PeA.
and thcn we show how PeA arises naturally as the maximum likelihood solution 10

12.1. Principal C01n[>OM"1 AnalJsis
Flgu,e12.2

561

P'if>cipal compooont a",,~" seeks" $pace

01 !owe, dimensionality. kt"(>WIl as !he P<lno>
pal subSpace "nd denoted I:Jy the magenta "1line. SUCh Itlet the Grthogonet [jiojectiOh 01!he data points ('ed doIsl onto tP'Ns ~"""'imizes the varia,..,., of !he proja<:ted points(green doIs). An "It",nati"" ....finilion 01 PCA
is based on m..mizing the """,-<>I·squaresof !he projection errors. ind'cated by the bfi.>elines.

S'crio" 12.2

S'di"" 12.4

a panlcula, fonn of linear-Gau"ian latem "ariable model. This probabilistic reformulation bring~ many ad\'imlag~s, su~h as tl>l: use I)f EM for parameter eslimalion,
rrinciple<J c~tensioos 10 Oli~turc, of PeA model" and Ba)'~sian formulat;ons thatallow tbe number of rrincipal com[>Oncnts to be detennined aUlOmatically from !bedata. Finally'. "c diSl;us< briefly ""'eral gencrali,ation, of the latent Yariable conceptthat g<l be~ood tbe linear-Gaussian assumption including non·Gau"i"n I.tcnt yariabies .....hich lea'" to tbe fr.me....ork of indrl"'mJ.m compon.nl anal,-.;., as .... ell a,models ha"ing a nonlinear rclationship bet ....een latent and oose",e<J ,'lUiable,.

____'c2=.~1. Principal Component Analysis
Principal compooem analy,;" or rcA.;s a technique tha! is "'idely u<ed for appli.
cations such as dimensionality .-eduction, lossy data comprc"ion, feature e>tracti"".
and data v;,ualizatiOll (Jolliffe, 2(02). It;s also kno.... " as tile Karoan.n·I..,;"" tran,·f~.

lbcrc an: t....o commonly used definitions of PeA that giye rise to the >arne
algorithm. PeA can be defined as the unhog<lnal projtttion of the data O/1tO a lo....er
dimensionallincar space. kno....n as the pri/lcip.al $uh.•p.aa. soch that the \'ariance ofthe projttted data i' ma~imi,e<J (1I",.lIing. 1933). Equi"alemly,;t can be defined as
tbe linear projection that minimi"'. the average projttlion cost. defined as t~ meansqua.-ed distance !letween the data [>Oint< and tbeir p<ojtttioo, (Pearson, 19(1). The
l"J'"OC"s< of onhogonal projection i' illustraled in FiguTe 12.2. We con,ider each ofthese definitions in tum.

12,1.1

Mllximllm variance lormulation

Con,ider a dala set <If obser"\lations {x,,} where" = 1..... S, and x" i, a
Euclidean variable "'ilh dimen,ionality D. Our goal is to project If>/:: data onto a'pace ha"ing dimen,ionality M < D" hile Ill3Jli",i,illg the "ariallCe of the projttted
data. For the !noll..nl. we 'hall assume that tbe "alue of M is g;\·en. Latcr in this562

12. CONTINUOUS LATENT VARIABLES
chapter, we shall consider techniques to determine an appropriate value of IV! fromthe data.
To begin with, consider the projection onto a one-dimensional space (M = 1).
We can define the direction of this space using a D-dimensional vector Ul, which
for convenience (and without loss of generality) we shall choose to be a unit vector
so that uf Ul = 1 (note that we are only interested in the direction defined by Ul,
not in the magnitude of Ul itself). Each data point X n is then projected onto a scalarvalue uf X n . The mean of the projected data is ufx where x is the sample set mean
given by(12.1)and the variance of the projected data is given by(12.2)

where S is the data covariance matrix defined by

1 N
S = -NLJ"(x n - x)(x n - x)T .

(12.3)

n=l

Appendix E

We now maximize the projected variance UfSUl with respect to Ul. Clearly, this has
to be a constrained maximization to prevent Ilulll ..... 00. The appropriate constraint
comes from the normalization condition uf Ul = 1. To enforce this constraint,
we introduce a Lagrange multiplier that we shall denote by AI, and then make anunconstrained maximization of(12.4)

By setting the derivative with respect to Ul equal to zero, we see that this quantity
will have a stationary point when
(12.5)

which says that Ul must be an eigenvector of S. If we left-multiply by uf and makeuse of uf Ul = 1, we see that the variance is given by(12.6)and so the variance will be a maximum when we set Ul equal to the eigenvector
having the largest eigenvalue AI. This eigenvector is known as the first principal
component.
We can define additional principal components in an incremental fashion bychoosing each new direction to be that which maximizes the projected variance12.1. Principal Component Analysis

Exercise 12.1

Section 12.2.2

563

amongst all possible directions orthogonal to those already considered. If we consider the general case of an M -dimensional projection space, the optimal linear projection for which the variance of the projected data is maximized is now defined by
the M eigenvectors U 1, ... , U M of the data covariance matrix S corresponding to the
M largest eigenvalues >'1, ... , AM. This is easily shown using proof by induction.
To summarize, principal component analysis involves evaluating the mean x
and the covariance matrix S of the data set and then finding the M eigenvectors of S
corresponding to the M largest eigenvalues. Algorithms for finding eigenvectors andeigenvalues, as well as additional theorems related to eigenvector decomposition,
can be found in Golub and Van Loan (1996). Note that the computational cost ofcomputing the full eigenvector decomposition for a matrix of size D x Dis O(D3).
If we plan to project our data onto the first M principal components, then we only
need to find the first M eigenvalues and eigenvectors. This can be done with more
efficient techniques, such as the power method (Golub and Van Loan, 1996), thatscale like O(M D 2 ), or alternatively we can make use of the EM algorithm.

12.1.2 Minimum-error formulation
Appendix C

We now discuss an alternative formulation of peA based on projection error
minimization. To do this, we introduce a complete orthonormal set of D-dimensional
basis vectors {Ui} where i = 1, ... , D that satisfy(12.7)
Because this basis is complete, each data point can be represented exactly by a linear
combination of the basis vectors
D(12.8)

X n = Laniui
i=lwhere the coefficients ani will be different for different data points. This simply
corresponds to a rotation of the coordinate system to a new system defined by the{Ui}, and the original D components {Xnl' ... , XnD} are replaced by an equivalent
set {anl' ... ,anD}. Taking the inner product with Uj, and making use of the orthonormality property, we obtain anj = x; Uj, and so without loss of generality wecan writeD=Xn

L (X~Ui) Ui·(12.9)

i=l

Our goal, however, is to approximate this data point using a representation involving a restricted number M < D of variables corresponding to a projection onto
a lower-dimensional subspace. The M -dimensional linear subspace can be represented, without loss of generality, by the first M of the basis vectors, and so weapproximate each data point X n byM

xn = L
i=l

D
ZniUi

+ L
i=M+l

biUi

(12.10)564

12. CONTINUOUS LATENT VARIABLES
where the {Zni} depend on the particular data point, whereas the {bd are constants
that are the same for all data points. We are free to choose the {Ui}, the {Zni}, andthe {bd so as to minimize the distortion introduced by the reduction in dimensionality. As our distortion measure, we shall use the squared distance between the original
data point X n and its approximation Xn , averaged over the data set, so that our goal
is to minimizeN~ L IlxJ =n -xn 11 2 .

(12.11)n=lConsider first of all the minimization with respect to the quantities {Zni}. Substituting for Xn , setting the derivative with respect to Znj to zero, and making use of
the orthonormality conditions, we obtain(12.12)where j = 1, ... ,M. Similarly, setting the derivative of J with respect to bi to zero,and again making use of the orthonormality relations, gives

bj = -T
X Uj(12.13)where j = M + 1, ... ,D. If we substitute for Zni and bi , and make use of the generalexpansion (12.9), we obtainD

Xn - Xn

=

L {(

Xn -

ud Ui

x) T

(12.14)i=M+lxfrom which we see that the displacement vector from X n to n lies in the spaceorthogonal to the principal subspace, because it is a linear combination of {ud for
i = M + 1, ... , D, as illustrated in Figure 12.2. This is to be expected because theprojected points n must lie within the principal subspace, but we can move them
freely within that subspace, and so the minimum error is given by the orthogonal
projection.
We therefore obtain an expression for the distortion measure J as a function
purely of the {ud in the form

x

1 ~ ~
J = N LL
n=l i=M+l(T_T)2 =X Ui - X Ui
n

L
D

T

U i SUi.

(12.15)i=M+lThere remains the task of minimizing J with respect to the {Ui}, which must
be a constrained minimization otherwise we will obtain the vacuous result Ui = O.
The constraints arise from the orthonormality conditions and, as we shall see, thesolution will be expressed in terms of the eigenvector expansion of the covariancematrix. Before considering a formal solution, let us try to obtain some intuition about
the result by considering the case of a two-dimensional data space D = 2 and a onedimensional principal subspace M = 1. We have to choose a direction U2 so as to12.1. Principal Component Analysis

565

minimize J = UISU2' subject to the normalization constraint uI U2 = 1. Using a
Lagrange multiplier A2 to enforce the constraint, we consider the minimization of(12.16)

Exercise 12.2

Setting the derivative with respect to U2 to zero, we obtain SU2 = A2U2 so that U2
is an eigenvector of S with eigenvalue A2. Thus any eigenvector will define a stationary point of the distortion measure. To find the value of J at the minimum, weback-substitute the solution for U2 into the distortion measure to give J = A2. We
therefore obtain the minimum value of J by choosing U2 to be the eigenvector corresponding to the smaller of the two eigenvalues. Thus we should choose the principal
subspace to be aligned with the eigenvector having the larger eigenvalue. This result
accords with our intuition that, in order to minimize the average squared projection
distance, we should choose the principal component subspace to pass through themean of the data points and to be aligned with the directions of maximum variance.
For the case when the eigenvalues are equal, any choice of principal direction will
give rise to the same value of J.
The general solution to the minimization of J for arbitrary D and arbitrary M <D is obtained by choosing the {Ui} to be eigenvectors of the covariance matrix givenbySUi = AiUi
(12.17)
where i = 1, ... ,D, and as usual the eigenvectors {Ui} are chosen to be orthonormal. The corresponding value of the distortion measure is then given by
D

L

J=

Ai

(12.18)i=M+lwhich is simply the sum of the eigenvalues of those eigenvectors that are orthogonal
to the principal subspace. We therefore obtain the minimum value of J by selecting
these eigenvectors to be those having the D - M smallest eigenvalues, and hence
the eigenvectors defining the principal subspace are those corresponding to the Mlargest eigenvalues.
Although we have considered M < D, the PCA analysis still holds if M =D, in which case there is no dimensionality reduction but simply a rotation of thecoordinate axes to align with principal components.
Finally, it is worth noting that there exists a closely related linear dimensionality
reduction technique called canonical correlation analysis, or CCA (Hotelling, 1936;
Bach and Jordan, 2002). Whereas PCA works with a single random variable, CCA
considers two (or more) variables and tries to find a corresponding pair of linear
subspaces that have high cross-correlation, so that each component within one of thesubspaces is correlated with a single component from the other subspace. Its solution
can be expressed in terms of a generalized eigenvector problem.

12.1.3 Applications of peA
Appendix A

We can illustrate the use of PCA for data compression by considering the offline digits data set. Because each eigenvector of the covariance matrix is a vector

12. COl\'TINUOliS LATf;I\'T \'ARIAIILES

566

Figure 12.3 The mean ~'" x aklog with !he II"'t lou' PCA e;gerrvecl<)rll Ul,. .. '" lor the 011-.....
cligits data set. t<>getl'ler with !he correspondi~ ~.
;n the OIigi",,1 D-<limensional space. we can represent tho eigenw:cto<s as imago< oftho same silO as ,1>0 data poi",,_ 11,. first Ih'e .ig.n,'occOfS. along wich tl>o corresponding .igen,'slue,. are <IIo"'n in Figure 12,3, A plO! ofll>o complete spect"'m ufoigo",·alue,. sone<! into decreasing order. is shown in Figure 12.4{ai. The di'tortion
measure J aSSQCiated wilh choo<ing a particular value of M is gi.'en by tho sumof the eig.n",lues from M + I up to 0 and is ptO!ted for different ,'aluo< of .\1 inFigure 12,4(b).
If "'e <utlslitut. (12, 12) and (12.13) into (12.10). we can write the I'CA appro~­imation to a data "eel'" x~ i" the fonn
M

'-

~

,
",

.-.

",

,.,

~

~

;

" (xl'u,)u,
I:
._M+l

(12.19)M

x+L(X~U,-XTU,)U;•(12.20)

,

,
,
,

, to'

,
"

L{x~",)u,+

, 10'

""
0

",

,.,

~

~

"

FIIIUre 12,4 (a) PIol at !he eJoI;nv.loo .".,etrum lor the off·1ine digits data set (b) P10t 01 !he sum at the<:liscarded ."".Ioos, which "'l'fesoots!he s.um-ol·SQ"",es distortlon J i<*~ by projecti<Xl the data onto
a p<incipal componenl slll>spaee '" dimensionalitv M.

11. 1. I'rindpall.:"l11pon~nt Anal~·.i.

FIIIUr. 1:1:.5

567

An ",>gi",,1 ~mpIe Irom lI>e 011·_ digils data ...ttOll"1her with its PeA re<:onstnxlions
oblair...:! by 'e1aio"li!Xl ,If j)<incipal ~n1S 10< various val,," 01 ,If. As ,II increason!tie re<:onst,uctiOfI ~s more ao::urate and woukl ~ portee! when .-If K D ~28 x 28 ~ ."-1.

where we ha"e made moe of the relation" (x'",) u;
x= L

,-,

AI''''''''/;'\' A
Seer/on 9.1(12.21)

which follow. from the completene" of the {u, I, Thi. represent. a contpre"ioo"f the data >ct. Ilttau>e for each data poim we ha,.. repla«d the V·dimensiooal"o<:lor x" Wilh an ,I[.din>en,ional "o<:tor having componem, (x~ '" _ X'",). 11Ie
'mailer the "alue of M. the greater the degree of comp.-e",ion. Example. of PeA,""on't""tioos of data points for the digits data set are shown in Figure 12.5
Anolher application of priocipal compcmenl analy,i. i' to data pre-processing.
In thi' case, lhe goal is nO! dimensionality redUC1ion but rather the tmn,formmion ofa data sel in or<k' to standa'lli'.e eenain of ilS pmpenies. This can be in'portanl inallowing .ubsequent pallem ,""ognition algorithm. 10 be applied successfully 10 thedata >ct. Typically. il is done wilen the original "ariable. are mea,ured in "arioos dif.
ferent unil' or !la"e significantly difTerent ,'ariabilil}'. For instance in the Old Faithfuldata sel. the time betv.-een eruption. i. typicany an order of magni1Ude greater thanlhe dUrali"" of.n erupt;,,". When W'e applied the ".nlCans algorill"" 10 thi< dataset, ".-e first made a separ.te linear re-sealing of the individual "anable' socb thm
each "ariable had zero mean and unit "ariance. llUs is known as slllNlardiv·.,g thedota. and the cO\'anance matrix for lhe 'lando,di/,ed dala has components(12,22)where <1, is the ,'anaoce of :c,. This i< known as the (",,,el,,,;,,,, matri.' of the original
dota and ha' the propeny thai if t""o rompooent, X; and x, of the data are perfee1ly
correl.ted. then Ai _ I.•nd if they a.-e uocorrelated. then Ai _ O.
11",,'1""', using PeA we can make a It>Of'e subst.mial nonnalizat;oo of the data
to gi\'C it zero mean and unit co'·ariance. so that different "anables become derorrelate<l To do this. we first ""rile the ei8Cn"cclor equation (12, 17) in the form

su= UL

(12.23)

568

12. CONTINUOUS LATENT VARIABLES

100

2

90

70
60
50

B

00'

80

2
000

0

,=~o

0

40
4

0

0

-2
2

O

6

0

2

0

0

0

tj

O~

~

~OOID

-2
-2

08
cPO 0

-2

0

2

Figure 12.6 Illustration of the effects of linear pre-processing applied to the Old Faithful data set. The plot on
the left shows the original data. The centre plot shows the result of standardizing the individual variables to zeromean and unit variance. Also shown are the principal axes of this normalized data set, plotted over the range±A~/2. The plot on the right shows the result of whitening of the data to give it zero mean and unit covariance.

where L is a D x D diagonal matrix with elements Ai, and U is a D x D orthogonal matrix with columns given by Ui. Then we define, for each data point X n , atransformed value given by(12.24)
where x is the sample mean defined by (12.1). Clearly, the set {Yn} has zero mean,
and its covariance is given by the identity matrix because
N

1~ L L -1/2UT (Xn - x)(x n - x)TUL -1/2
n=l

L ~1/2UTSUL -1/2 = L -1/2LL-1/2 = I.

Appendix A

Appendix A(12.25)This operation is known as whitening or sphereing the data and is illustrated for the
Old Faithful data set in Figure 12.6.
It is interesting to compare PCA with the Fisher linear discriminant which was
discussed in Section 4.1.4. Both methods can be viewed as techniques for linear
dimensionality reduction. However, PCA is unsupervised and depends only on thevalues X n whereas Fisher linear discriminant also uses class-label information. Thisdifference is highlighted by the example in Figure 12.7.
Another common application of principal component analysis is to data visualization. Here each data point is projected onto a two-dimensional (M = 2) principal
subspace, so that a data point X n is plotted at Cartesian coordinates given by x'J. U1
and x'J. U2, where Ul and U2 are the eigenvectors corresponding to the largest andsecond largest eigenvalues. An example of such a plot, for the oil flow data set, isshown in Figure 12.8.

12.1. I'<incipal Cl)m ..."n~nt Anal}'s;,
Fig"", 12.7

A comparison 01 pro:ipal compoMnt analysis .... 111 Fisha(s linaar
discriminant 101 """", <*man""'"ality r&duclion. Here too data in
two dimansions, belonging to twoclasses sIIOWI1 in red and blue. is
to be PfOI"Cled onto a s.ingle di·mension. PCA c/>xlsas the direc·

"

.

-

569

'~'" ~ .•••._',.. , . ...' ' ~ .• ''''".,

. -:- '~':r-'---~+·_ ~-Jtion 01 maximum varia""e. sIIOWI1
try
ma9""ta Co""'.
.,
to
~,

tha
wt11ch leads
strong class overlap. whereas!he Fisl>ef IiMar diSCfOrnillant takes
accoun1 <:A too class labels andleads to a projection onto the g<ean
CUM! giving much t>etler class.". , ' : - - - - ; ! - - - ; - "
_.S

0

3

separation
Fig"", 12.8

Visualilatlon 01 !he oill'low <lata lIet obtained
try projoecting the <lata onto the lirst two prin.
cipal compone<1ts. The <ed, blue, and 9r&en
points corre-spond to !he 'IamiNI(, 't>omogenoous', and '8nnula~ flow oonligurations",specriveIy.

12.1.4peA for high-dimensional data

In some application. of pliTlCipal component analysis. the number of data points
is smaller than t!>c dimensionality of troe data 'pace. FOI" example. ",e might want toapply PeA to a data <el of a few hundred images, each of ,,'hich rorrespoOOs to a"eetor in a 'pace of poIentially .....ml million dimensiOlls (COITesponding tn thfl'e
enlour "alues for each of the pi.",ls in troe image), NOIe that in a D-<limen,ional space
a set of jY points. ",'here N < D. defines a linear subspa::e ",hose dimensi"nality
is at ""'st N - 1, and SO there is linle point in applying PeA for ,'alue< of M
thaI"'" greater than N - I, Indeed, if "'e pelf"",, PeA we will find that at least
D - N + I of the eigen".lues art lero. eorrespnnding tQ eigenvectors aloog ",hosedireclioos the data <el has 10m varianee. Funhem>ore. typical algOl"ithm, for finding
the eigen,'eet"" of a D x D matrix ha"e a computatiooal eosl thm scales like O( D~ J.
aOO so for appliealions such as the image e,ample. a direc' application of PeA will
be computatiooally infe,,-sibJe.
W. can resoh'e this problem as foIl",",'" Fir;l. let us define X to be the (N " DJ·570

12. CONTINUOUS LATENT VARIABLES
dimensional centred data matrix, whose nth row is given by (x n - X)T. The covariance matrix (12.3) can then be written as S = N- 1 XTX, and the corresponding
eigenvector equation becomes
1 T-X
N XUi = AiUi .

(12.26)

Now pre-multiply both sides by X to give1
T
NXX (XUi) = Ai(XUi)'(12.27)

If we now define Vi = XUi, we obtain

1
T-XX Vi = AiVi

N

(12.28)which is an eigenvector equation for the N x N matrix N- 1 XX T . We see that thishas the same N -1 eigenvalues as the original covariance matrix (which itself has anadditional D - N + 1 eigenvalues of value zero). Thus we can solve the eigenvector
problem in spaces of lower dimensionality with computational cost O(N 3 ) insteadof O(D 3 ). In order to determine the eigenvectors, we multiply both sides of (12.28)
by X T to giveT
T
1 T)
( NX X (X Vi) = Ai(X Vi)(12.29)from which we see that (XTVi) is an eigenvector of S with eigenvalue Ai. Note,however, that these eigenvectors need not be normalized. To determine the appropriate normalization, we re-scale Ui ex: X TVi by a constant such that Ilui II = 1, which,assuming Vi has been normalized to unit length, gives
1

Ui =( NA i)1/2 X

TVi·(12.30)In summary, to apply this approach we first evaluate XX T and then find its eigenvectors and eigenvalues and then compute the eigenvectors in the original data space
using (12.30).

12.2. Probabilistic peA
The formulation of PCA discussed in the previous section was based on a linear
projection of the data onto a subspace of lower dimensionality than the original data
space. We now show that PCA can also be expressed as the maximum likelihood
solution of a probabilistic latent variable model. This reformulation of PCA, known
as probabilistic peA, brings several advantages compared with conventional PCA:
• Probabilistic PCA represents a constrained form of the Gaussian distributionin which the number of free parameters can be restricted while still allowing
the model to capture the dominant correlations in a data set.

12.2. Probabilistic peA

Section 12.2.2

571• We can derive an EM algorithm for PCA that is computationally efficient insituations where only a few leading eigenvectors are required and that avoids
having to evaluate the data covariance matrix as an intermediate step.
• The combination of a probabilistic model and EM allows us to deal with missing values in the data set.
• Mixtures of probabilistic PCA models can be formulated in a principled way
and trained using the EM algorithm.

Section 12.2.3• Probabilistic PCA forms the basis for a Bayesian treatment of PCA in which
the dimensionality of the principal subspace can be found automatically fromthe data.
• The existence of a likelihood function allows direct comparison with otherprobabilistic density models. By contrast, conventional PCA will assign a low
reconstruction cost to data points that are close to the principal subspace even
if they lie arbitrarily far from the training data.
• Probabilistic PCA can be used to model class-conditional densities and hence
be applied to classification problems.
• The probabilistic PCA model can be run generatively to provide samples fromthe distribution.

Section 8.1.4

This formulation of PCA as a probabilistic model was proposed independently by
Tipping and Bishop (1997, 1999b) and by Roweis (1998). As we shall see later, it isclosely related to factor analysis (Basilevsky, 1994).
Probabilistic PCA is a simple example of the linear-Gaussian framework, inwhich all of the marginal and conditional distributions are Gaussian. We can formulate probabilistic PCA by first introducing an explicit latent variable z corresponding
to the principal-component subspace. Next we define a Gaussian prior distributionp( z) over the latent variable, together with a Gaussian conditional distribution p( xl z)
for the observed variable x conditioned on the value of the latent variable. Specifically, the prior distribution over z is given by a zero-mean unit-covariance Gaussianp(z) = N(zIO, I).

(12.31)

Similarly, the conditional distribution of the observed variable x, conditioned on thevalue of the latent variable z, is again Gaussian, of the formp(xlz) = N(xlWz + J-L, a 2 I)Section 8.2.2(12.32)in which the mean of x is a general linear function of z governed by the D x M
matrix Wand the D-dimensional vector J-L. Note that this factorizes with respect tothe elements of x, in other words this is an example of the naive Bayes model. As
we shall see shortly, the columns of W span a linear subspace within the data spacethat corresponds to the principal subspace. The other parameter in this model is thescalar a 2 governing the variance of the conditional distribution. Note that there is no572

11. CONTINUOUS LAT!::NT VANIM1LI::S

,
,,

,

,,

,,

.- /
,

Flgu .. 12.9 I\n ~I"'tfat"" oIlt>e II"""fative vi&w oI1t>e p<ot>abi!;st", PeA modeIfof" two-dimensiooal <!ala
space and a on&-<lirnent.ionallat/l<1t space, An Ob&erved <!ala point x Is generated by first drawing a value i
fof 1t>e Iat&n1 vafiatlle /f(lm ~s prior dist,~t"" P(~) and Itlen drawing a val"" fof x lrom an iSO/fopK: Gaussian
distr~t"" (iijust,al&(l by the red cir<:ie's) having mean wi +" and coY8r1.once ,,'1 The l/f&er\ ellips.&$ show l!ledensity """toors!of the marg'''''1 dis1r1bulion PIx).
loss of ge""raJity in assuming a zero mean. unit co\'ariance Gau"ian for the latent
distributi"n II{Z) because a more gcneral Gau"i3n di"ributi"n would gi"e rise to an
equivalent probabili"ic n>odel.
We can view the probabilistic PeA model from a geoerati"e \'iew""int in "hicha sampled '-alue of the ob""Yed ,..riable is obIained by first choo,ing a ,..Iue forthe latent ,'ariahle aod then >ampling the OO",,,'e;j ,-ariable cooditioned on this laotent \'alue, Specifically, the V-dimen'ional OO"''''ed '-ariable x is defined by a lin·ea, tran,formati,," of the '\/·dimen,i"nal latcnt '-ariable z plu, additi'-e Gaussian'noise' , <0 that,,=\VZ+,,+~(12.33)w!>ere z is an M-di""'nsional Gaussian lalent variable. and .. is a V·dimensi"nal,ero-mean Gau.. ian-distributed noi.., "ariable witb co'-ariance ,,21. This generative
process is illustrated in Figure 12.9. NOIe that this frame".-orl< is based on a mappingfrom latent ,pace 10 data space. in contrast 10 the nl()l'(: C(""'cnti,,,,"1 "iew "f I'CA
dis.cus"'d alx",e, 11Ie ",,'e= mapping, from data space to the latent space. ,,-ill he
OOlained ,honly using Ha ycs· lhwn:m.
SUf!ll'OSC we wish 10 deten"ine the "alues ofll>o parameters \V. I' and ,,' uSIng
maximum likelihuo<l, To write """"n lhe likeliltood function, we need an ""pression
for tl>o marginal distributioo p{") of tl>o ~,,'ed ...riahle_ This is exprt__ sed. fmn'
the sum aod p,oduct rules "fprobability, in the form

(11,34)E,e,,-ise 12,7

ll""auS(: this corresponds to a linear·Gau"i,n lT1(llIcL thi< marginal di,tribulion isagain Gaussian. atld is given by,,(,,) _ N{xllf,C)(IUS)

12.2. Probabilistic peA

573

where the D x D covariance matrix C is defined by

C = WW T + 0- 2 1.

(12.36)This result can also be derived more directly by noting that the predictive distribution
will be Gaussian and then evaluating its mean and covariance using (12.33). Thisgives

IE [x]
cov[x]

IE[Wz + JL + E] = JL
IE [(Wz + E)(WZ + E)T]IE [WZZTW T] + IE[EE T] = WW T + 0- 2 1(12.37)(12.38)where we have used the fact that z and E are independent random variables and hence
are uncorrelated.
Intuitively, we can think of the distribution p(x) as being defined by taking an
isotropic Gaussian 'spray can' and moving it across the principal subspace spraying
Gaussian ink with density determined by 0- 2 and weighted by the prior distribution.
The accumulated ink density gives rise to a 'pancake' shaped distribution representing the marginal density p(x).
The predictive distribution p( x) is governed by the parameters JL, W, and 0- 2 •However, there is redundancy in this parameterization corresponding to rotations of
the latent space coordinates. To see this, consider a matrix W = WR where R isan orthogonal matrix. Using the orthogonality property RRT = I, we see that thequantity WW T that appears in the covariance matrix C takes the form
(12.39)and hence is independent of R. Thus there is a whole family of matrices W all ofwhich give rise to the same predictive distribution. This invariance can be understoodin terms of rotations within the latent space. We shall return to a discussion of thenumber of independent parameters in this model later.
When we evaluate the predictive distribution, we require C- 1 , which involves
the inversion of a D x D matrix. The computation required to do this can be reducedby making use of the matrix inversion identity (C.7) to giveC- 1 = 0-- 1 1 - 0--2WM- 1 W T(12.40)where the M x M matrix M is defined by

M = WTW + 0- 2 1.

Exercise 12.8

(12.41)

Because we invert M rather than inverting C directly, the cost of evaluating C- 1 isreduced from O(D 3 ) to O(M 3 ).
As well as the predictive distribution p(x), we will also require the posterior
distributionp(zlx), which can again be written down directly using the result (2.116)
for linear-Gaussian models to give(12.42)
Note that the posterior mean depends on x, whereas the posterior covariance is independent of x.

574

12. CONTINUOUS LATENT VARIABLES
Figure 12.10 The probabilistic peA model for a data set of N observations of x can be expressed as a directed graph inwhich each observation X n is associated with a value
Zn of the latent variable.

..-+--w
N

12.2.1

Maximum likelihood peA

We next consider the determination of the model parameters using maximum
likelihood. Given a data set X = {x n } of observed data points, the probabilisticpeA model can be expressed as a directed graph, as shown in Figure 12.10. The
corresponding log likelihood function is given, from (12.35), byN

Inp(XIJL, W,O' 2 ) =L ln p(x IW,JL,O'
n

2

)

n=l

1""
Ie[ - 2
L,..(x- 2
N

ND ln(2n)

--2-

N ln

n -

JL) T c- 1 (x n - JL). (12.43)

n=lSetting the derivative of the log likelihood with respect to JL equal to zero gives the
expected result JL = x where x is the data mean defined by (12.1). Back-substituting
we can then write the log likelihood function in the form

N
Inp(XIW, JL, 0'2) = -2 {D In(2n) + In Ie[ + Tr (C-1S)}(12.44)

where S is the data covariance matrix defined by (12.3). Because the log likelihood
is a quadratic function of JL, this solution represents the unique maximum, as can be
confirmed by computing second derivatives.
Maximization with respect to W and 0'2 is more complex but nonetheless has
an exact closed-form solution. It was shown by Tipping and Bishop (1999b) that allof the stationary points of the log likelihood function can be written as(12.45)where U M is a D x M matrix whose columns are given by any subset (of size M)of the eigenvectors of the data covariance matrix S, the M x M diagonal matrix
L M has elements given by the corresponding eigenvalues .\ , and R is an arbitrary
M x M orthogonal matrix.
Furthermore, Tipping and Bishop (1999b) showed that the maximum of the likelihood function is obtained when the M eigenvectors are chosen to be those whose
eigenvalues are the M largest (all other solutions being saddle points). A similar result was conjectured independently by Roweis (1998), although no proof was given.

12.2. Probabilistic peA

575

Again, we shall assume that the eigenvectors have been arranged in order of decreasing values of the corresponding eigenvalues, so that the M principal eigenvectors are
Ul,"" UM. In this case, the columns of W define the principal subspace of standard PCA. The corresponding maximum likelihood solution for (J'2 is then given by

1(J'~L = D-M

D

L Ai

(12.46)i=M+lSection 12.2.2

so that (J'~L is the average variance associated with the discarded dimensions.
Because R is orthogonal, it can be interpreted as a rotation matrix in the M x M
latent space. If we substitute the solution for W into the expression for C, and makeuse of the orthogonality property RR T = I, we see that C is independent of R.
This simply says that the predictive density is unchanged by rotations in the latent
space as discussed earlier. For the particular case of R = I, we see that the columnsof W are the principal component eigenvectors scaled by the variance parameters
Ai - (J'2. The interpretation of these scaling factors is clear once we recognize thatfor a convolution of independent Gaussian distributions (in this case the latent space
distribution and the noise model) the variances are additive. Thus the variance Ai
in the direction of an eigenvector Ui is composed of the sum of a contribution Ai (J'2 from the projection of the unit-variance latent space distribution into data space
through the corresponding column of W, plus an isotropic contribution of variance(J'2 which is added in all directions by the noise model.
It is worth taking a moment to study the form of the covariance matrix givenby (12.36). Consider the variance of the predictive distribution along some direction
specified by the unit vector v, where vTv = 1, which is given by vTCv. First
suppose that v is orthogonal to the principal subspace, in other words it is given bysome linear combination of the discarded eigenvectors. Then v TV = 0 and hencev TCv = (J'2. Thus the model predicts a noise variance orthogonal to the principal
subspace, which, from (12.46), is just the average of the discarded eigenvalues. Now
suppose that v = Ui where Ui is one of the retained eigenvectors defining the principal subspace. Then vTCv = (Ai - (J'2) + (J'2 = Ai. In other words, this model
correctly captures the variance of the data along the principal axes, and approximates
the variance in all remaining directions with a single average value (J'2.
One way to construct the maximum likelihood density model would simply be
to find the eigenvectors and eigenvalues of the data covariance matrix and then toevaluate Wand (J'2 using the results given above. In this case, we would choose
R = I for convenience. However, if the maximum likelihood solution is found bynumerical optimization of the likelihood function, for instance using an algorithm
such as conjugate gradients (Fletcher, 1987; Nocedal and Wright, 1999; Bishop andNabney, 2008) or through the EM algorithm, then the resulting value of R is essentially arbitrary. This implies that the columns of W need not be orthogonal. If
an orthogonal basis is required, the matrix W can be post-processed appropriately(Golub and Van Loan, 1996). Alternatively, the EM algorithm can be modified insuch a way as to yield orthonormal principal directions, sorted in descending orderof the corresponding eigenvalues, directly (Ahn and Oh, 2003).

576

12. CONTINUOUS LATENT VARIABLES
The rotational invariance in latent space represents a form of statistical nonidentifiability, analogous to that encountered for mixture models in the case of discretelatent variables. Here there is a continuum of parameters all of which lead to thesame predictive density, in contrast to the discrete nonidentifiability associated withcomponent re-labelling in the mixture setting.
If we consider the case of M = D, so that there is no reduction of dimensionality, then U M = U and L M = L. Making use of the orthogonality propertiesUU T = I and RRT = I, we see that the covariance C of the marginal distribution
for x becomes(12.47)and so we obtain the standard maximum likelihood solution for an unconstrained
Gaussian distribution in which the covariance matrix is given by the sample covariance.
Conventional PCA is generally formulated as a projection of points from the Ddimensional data space onto an M -dimensional linear subspace. Probabilistic PCA,
however, is most naturally expressed as a mapping from the latent space into the data
space via (12.33). For applications such as visualization and data compression, we
can reverse this mapping using Bayes' theorem. Any point x in data space can thenbe summarized by its posterior mean and covariance in latent space. From (12.42)
the mean is given by(12.48)where M is given by (12.41). This projects to a point in data space given by

WlE[zlx] + J-L.
Section 3.3.1(12.49)Note that this takes the same form as the equations for regularized linear regression
and is a consequence of maximizing the likelihood function for a linear Gaussianmodel. Similarly, the posterior covariance is given from (12.42) by 0-2M- 1 and isindependent of x.
If we take the limit 0- 2 ----t 0, then the posterior mean reduces to(12.50)Exercise 12.11
Exercise 12.12

Section 2.3

which represents an orthogonal projection of the data point onto the latent space,and so we recover the standard PCA model. The posterior covariance in this limit iszero, however, and the density becomes singular. For 0- 2 > 0, the latent projection
is shifted towards the origin, relative to the orthogonal projection.
Finally, we note that an important role for the probabilistic PCA model is in
defining a multivariate Gaussian distribution in which the number of degrees of freedom, in other words the number of independent parameters, can be controlled whilst
still allowing the model to capture the dominant correlations in the data. Recall
that a general Gaussian distribution has D(D + 1)/2 independent parameters in its
covariance matrix (plus another D parameters in its mean). Thus the number ofparameters scales quadratically with D and can become excessive in spaces of high12.2. Probabilistic peA

577

dimensionality. If we restrict the covariance matrix to be diagonal, then it has only D
independent parameters, and so the number of parameters now grows linearly withdimensionality. However, it now treats the variables as if they were independent andhence can no longer express any correlations between them. Probabilistic PeA provides an elegant compromise in which the M most significant correlations can be
captured while still ensuring that the total number of parameters grows only linearlywith D. We can see this by evaluating the number of degrees of freedom in the
PPCA model as follows. The covariance matrix C depends on the parameters W,
which has size D x M, and a 2 , giving a total parameter count of DM + 1. However,
we have seen that there is some redundancy in this parameterization associated withrotations of the coordinate system in the latent space. The orthogonal matrix R that
expresses these rotations has size M x M. In the first column of this matrix there are
M - 1 independent parameters, because the column vector must be normalized tounit length. In the second column there are M - 2 independent parameters, because
the column must be normalized and also must be orthogonal to the previous column,
and so on. Summing this arithmetic series, we see that R has a total of M(M -1)/2
independent parameters. Thus the number of degrees of freedom in the covariancematrix C is given by(12.51)
DM + 1 - M(M - 1)/2.

Exercise 12.14

The number of independent parameters in this model therefore only grows linearlywith D, for fixed M. If we take M = D - 1, then we recover the standard result
for a full covariance Gaussian. In this case, the variance along D - 1 linearly independent directions is controlled by the columns of W, and the variance along theremaining direction is given by a 2 . If M = 0, the model is equivalent to the isotropic
covariance case.

12.2.2

Section 12.2.4

Section 9.4

EM algorithm for peA

As we have seen, the probabilistic PCA model can be expressed in terms of amarginalization over a continuous latent space z in which for each data point X n ,there is a corresponding latent variable Zn. We can therefore make use of the EM
algorithm to find maximum likelihood estimates of the model parameters. This may
seem rather pointless because we have already obtained an exact closed-form solution for the maximum likelihood parameter values. However, in spaces of highdimensionality, there may be computational advantages in using an iterative EM
procedure rather than working directly with the sample covariance matrix. This EM
procedure can also be extended to the factor analysis model, for which there is no
closed-form solution. Finally, it allows missing data to be handled in a principledway.
We can derive the EM algorithm for probabilistic PCA by following the generalframework for EM. Thus we write down the complete-data log likelihood and takeits expectation with respect to the posterior distribution of the latent distribution
evaluated using 'old' parameter values. Maximization of this expected completedata log likelihood then yields the 'new' parameter values. Because the data points

578

12. CONTINUOUS LATENT VARIABLES
are assumed independent, the complete-data log likelihood function takes the formN

Inp (X, ZIJL, W, (J2) =L {lnp(xnlzn ) + lnp(zn)}(12.52)n=lwhere the nth row of the matrix Z is given by Zn. We already know that the exact
maximum likelihood solution for JL is given by the sample mean x defined by (12.1),
and it is convenient to substitute for JL at this stage. Making use of the expressions(12.31) and (12.32) for the latent and conditional distributions, respectively, and taking the expectation with respect to the posterior distribution over the latent variables,
we obtain

Note that this depends on the posterior distribution only through the sufficient statistics of the Gaussian. Thus in the E step, we use the old parameter values to evaluate

M-1WT(Xn - x)
(J2M-

Exercise 12.15

1+ lE[zn]lE[zn]T(12.54)(12.55)which follow directly from the posterior distribution (12.42) together with the standard result lE[znz~] = cov[zn] + JE[zn]JE[zn]T. Here M is defined by (12.41).
In the M step, we maximize with respect to Wand (J2, keeping the posterior
statistics fixed. Maximization with respect to (T2 is straightforward. For the maximization with respect to W we make use of (C.24), and obtain the M-step equations[t,exn -X)IlIZn]T] [t,Il[ZnZ~]]-'

W new

2

(Jnew

=

(12.56)1 N
2ND{llxn - xl1 - 2lE[zn]TW~ew(xn - x)Ln=l

+Tr (JE[znzJ]W~ewW new )}.

(12.57)The EM algorithm for probabilistic PCA proceeds by initializing the parameters
and then alternately computing the sufficient statistics of the latent space posterior
distribution using (12.54) and (12.55) in the E step and revising the parameter values
using (12.56) and (12.57) in the M step.
One of the benefits of the EM algorithm for PCA is computational efficiency
for large-scale applications (Roweis, 1998). Unlike conventional PCA based on an12.2. Probabilistic peA

579

eigenvector decomposition of the sample covariance matrix, the EM approach isiterative and so might appear to be less attractive. However, each cycle of the EM
algorithm can be computationally much more efficient than conventional PCA inspaces of high dimensionality. To see this, we note that the eigendecomposition of
the covariance matrix requires O(D 3 ) computation. Often we are interested onlyin the first M eigenvectors and their corresponding eigenvalues, in which case we
can use algorithms that are 0 (M D 2 ). However, the evaluation of the covariancematrix itself takes 0 (ND 2 ) computations, where N is the number of data points.
Algorithms such as the snapshot method (Sirovich, 1987), which assume that the
eigenvectors are linear combinations of the data vectors, avoid direct evaluation of
the covariance matrix but are O(N 3 ) and hence unsuited to large data sets. The EM
algorithm described here also does not construct the covariance matrix explicitly.
Instead, the most computationally demanding steps are those involving sums overthe data set that are 0 (N D M). For large D, and M « D, this can be a significant
saving compared to 0 (N D 2 ) and can offset the iterative nature of the EM algorithm.
Note that this EM algorithm can be implemented in an on-line form in which
each D-dimensional data point is read in and processed and then discarded before
the next data point is considered. To see this, note that the quantities evaluated in
the E step (an M-dimensional vector and an M x M matrix) can be computed foreach data point separately, and in the M step we need to accumulate sums over data
points, which we can do incrementally. This approach can be advantageous if both
Nand D are large.
Because we now have a fully probabilistic model for PCA, we can deal withmissing data, provided that it is missing at random, by marginalizing over the distribution of the unobserved variables. Again these missing values can be treatedusing the EM algorithm. We give an example of the use of this approach for datavisualization in Figure 12.11.
Another elegant feature ofthe EM approach is that we can take the limit a 2 ----t 0,
corresponding to standard PCA, and still obtain a valid EM-like algorithm (Roweis,1998). From (12.55), we see that the only quantity we need to compute in the Estep
is JE[zn]. Furthermore, the M step is simplifie~ because M = WTW. To emphasize
the simplicity of the algorithm, let us define X to be a matrix of size N x D whose
nth row is given by the vector X n - x and similarly define 0 to be a matrix of size
D x M whose nth row is given by the vector JE[zn]. The Estep (12.54) of the EM
algorithm for PCA then becomeso = (W~d Wold)-lW~dX(12.58)and the M step (12.56) takes the form

W new = XTOT(OOT)-l.

(12.59)Again these can be implemented in an on-line form. These equations have a simple
interpretation as follows. From our earlier discussion, we see that the E step involves
an orthogonal projection of the data points onto the current estimate for the principal
subspace. Correspondingly, the M step represents a re-estimation of the principal

580

12. CONTlNlJOlJS I"Ht;I'IT Vi\ RIARlES

Fig".. 12.11 Probabilistic PCA visoo,zsbon 01 a portion 0I1he "" !low data setlo< Ihe !irsl 100 (lata »einls, The
left..,...nd plot oIIOWS Ihe I'O"leoo< mean proj9c1ions oIlhfI (lata poims on lhe principal subspace. The ,;gtrI·hi\ndplot is obtained by firsl ran<lomly omitting 30% 0I1he variable .aloo. and lhen us>rlg EM 10 MndIe I"" mi......
values. Note I!IaI eac/1 data poinl1hen NoS allea., one missing mea.u,ement but lhoallhe plot i. ""ry ..mia, tolhe ona obtained wit"""l miss.... valL>ll$subspace to minimize !he squared reoonslruCtioo error in 'oIhich the proje<:tion, areEwrrise /2,/7

C.,N.
We ean gh'e a ,imple physical analogy for this EM algorithm. which is easily
visualized for D = 2 and M = 1. Coo,ider a collectioo nf data point' ,n tWI)
dimension', aod let tile u""'-dimensiunal principal subspace be represented by a <ohdrod. Now atlaCh each data point to the nxI via a ,pring oo<:)"ing HooI;:e', law ("umJ
energy i, propol1ional 10 ,lie square of lile spring". length). In ll1e E 'tel', we keep
the nxI hed and allow the attachment point' tn ,Iide up and <I<,wn ll1e nxI '" a, tominimize ll1e e",,'llY, This cau",. each attachment point (independently) 10 position
Itself at the orthogonal pmjeclion of the c~sponding data point onto the nxI. In
the M 'tel'. we keep the attachment poiOl' fil<ed and then release tile nxI and allow it
to m'>,'e 10 tile minimum energy posilion. 11Ie E and M 'teps are then repeated until
a ,uitable c""vergence cri.eri"" is ..,isfled. a. is illuSlrated in Figure 12.12.

12.2.3 Bayesian peA
S<J far in OIlr di",""ioo of PeA. we have ",'.nled Ihal tile '"Ine ,II for ,liedl,nen,ionalit)" of tile principal .ubspace is gi"en, In praclice. ".-e nlmt cOOose asuilable ,..I"" according 10 the application. For ,isuali,a,ion. we ge""",ny choose.\1 = 2. whereas for OIher application, the approrrialC choice for ,1/ ma)" be lessdea,. One appmao:h i. 10 pi", the eigen"alue 'peclrum for lhe data set. analog,•."10 the example in Figure 12.4 for the off_line digits dala SCI, and look to see if lite
eige",,,I .... nmurally form two groups comprising a set of ,mall ,'alues separated by
a ,ign;flcant gap from a ",t of relativel)" large ,'alues, indicating a natural cholcc f<>rAI, In practice. such a gap i, oflen ''''' seen

, ",

•,

•
....
-,

•

~,

0

,
, ",

0

,

,

, ,<,

0

0

-,

-,
-,

0

,

-,

0

,

-,

o

,

,

o

-,
-,
Flgu.. 12.12

o

,

-,

o

,

Synt"'elic <lata illustrating too EM algorithm !of PCA defined by (12.58) and (1259). (8) A data

set X with the data points shown in 1JI'e«l, t"ll"tM' W'i1!1l!>e t'IM pMdpal """"""",IS (shown as eigenveclor1
scaled by It>e squafll 'OOIS 04 the eigeJ'l\lllluel). (b) Initial configurat"'" 01 too principalsul>sl>a<:<t defined by W,
shown in md, tOO"lhfIr with the fK'(Ijeclions 01 the latll<11 points Z inlo too <lata space, giItoo by ZW T , shown incyan, (oj Alter""" M step, too laten! SI'B«l P>as been update<! wiIh Z r>el(l nxed. (d) Me' tt>e success.... E Slep,
It>e ""'-'eo 01 Z havu been up<!atll<:1. ~ '" Ihogoooal r>rojecliQn$, with W h&k! fixed. (e) Aft.... tile se<:o<l<l M
S!flp.
After l!Ie MC()<>;l E st"l'<')S,uion I.J

Be<:au,", th~ pm/xlhi li>lic PeA modd has a well·defined likelillood f"flCtion, we<wId employ cros,-,-.1idation to delermine the \"aJue of di"",nsiooa!ity by "'Iecting
tit<: large,t log likelihood t>I1 a '-alidation data set Such an opprooch. hov.·~\-er. canbecome computationally ro<lly. p3rticularl)' if we CQnsid<:, • probabilistic miXlUreof PeA modds (Tipping and Bishop. 1999a) in "hich we seek 10 <!etermi'" theappropriate dimen,ionalily ",paraltly for toch componenl in lt1e mixm""
Gi'-en thaiha,-e a probabilislic formulalion of PeA, il s«ms natural 10 s«ku Buye,ian approach 10 model seleclion. To do thi,. ,,"'e nee<! 10 marginalize 001
the model paramele" /'. \V. und ,,' wilh ""peel to appropriate prior distribution'.
This Can be done by u,ing a ,-ariation.l framework to .pproxim'le the allulylic.lly
intractable murginaliUOi;oo, (Bi,hop. 1mb). 1lIc marginal likelihood v.lues. given
by ttle ,'ari.,ionallower bour.d, cun lhen be c<>mpun:d for a r.nge of different '"Tue'"f;\I ar.d Itie '"Iue giving Iht largest marginal likelihood ",Iecloo_l1ere we consider. simpler approach introducoo by b.ased on the rddmu "p-

w.

582

12. CONTINUOUS LATENT VARIABLES
Figure 12.13

Probabilistic graphical model for Bayesian peA in
which the distribution over the parameter matrix W
is governed by a vector a of hyperparameters.

w
N

proximation, which is appropriate when the number of data points is relatively large
and the corresponding posterior distribution is tightly peaked (Bishop, 1999a). It
involves a specific choice of prior over W that allows surplus dimensions in theprincipal subspace to be pruned out of the model. This corresponds to an example ofautomatic relevance determination, or ARD, discussed in Section 7.2.2. Specifically,
we define an independent Gaussian prior over each column of W, which represent
the vectors defining the principal subspace. Each such Gaussian has an independent
variance governed by a precision hyperparameter O:i so that
(12.60)Section 7.2

where Wi is the i th column of W. The resulting model can be represented using thedirected graph shown in Figure 12.13.
The values for O:i will be found iteratively by maximizing the marginallikelihood function in which W has been integrated out. As a result of this optimization,some of the O:i may be driven to infinity, with the corresponding parameters vector Wi being driven to zero (the posterior distribution becomes a delta function at
the origin) giving a sparse solution. The effective dimensionality of the principal
subspace is then determined by the number of finite O:i values, and the corresponding vectors Wi can be thought of as 'relevant' for modelling the data distribution.
In this way, the Bayesian approach is automatically making the trade-off between
improving the fit to the data, by using a larger number of vectors Wi with their corresponding eigenvalues Ai each tuned to the data, and reducing the complexity ofthe model by suppressing some of the Wi vectors. The origins of this sparsity werediscussed earlier in the context of relevance vector machines.
The values of O:i are re-estimated during training by maximizing the log marginal
likelihood given byp(Xla, J-L, 0'2) =J

p(XIW, J-L, O'2)p(Wla) dW(12.61)

where the log of p(XIW, J-L, 0'2) is given by (12.43). Note that for simplicity we also
treat J-L and 0'2 as parameters to be estimated, rather than defining priors over theseparameters.

12.2. Probabilistic peA

Section 4.4
Section 3.5.3

583

Because this integration is intractable, we make use of the Laplace approximation. If we assume that the posterior distribution is sharply peaked, as will occur forsufficiently large data sets, then the re-estimation equations obtained by maximizing
the marginal likelihood with respect to ai take the simple form
(12.62)
which follows from (3.98), noting that the dimensionality of Wi is D. These reestimations are interleaved with the EM algorithm updates for determining Wanda 2 • The E-step equations are again given by (12.54) and (12.55). Similarly, the Mstep equation for a 2 is again given by (12.57). The only change is to the M-stepequation for W, which is modified to give(12.63)where A = diag(ai)' The value of I-" is given by the sample mean, as before.
If we choose M = D - 1 then, if all ai values are finite, the model represents
a full-covariance Gaussian, while if all the ai go to infinity the model is equivalent
to an isotropic Gaussian, and so the model can encompass all pennissible values forthe effective dimensionality of the principal subspace. It is also possible to consider
smaller values of M, which will save on computational cost but which will limit
the maximum dimensionality of the subspace. A comparison of the results of thisalgorithm with standard probabilistic PCA is shown in Figure 12.14.
Bayesian PCA provides an opportunity to illustrate the Gibbs sampling algorithm discussed in Section 11.3. Figure 12.15 shows an example of the samples
from the hyperparameters In ai for a data set in D = 4 dimensions in which the dimensionality of the latent space is M = 3 but in which the data set is generated froma probabilistic PCA model having one direction of high variance, with the remaining
directions comprising low variance noise. This result shows clearly the presence of
three distinct modes in the posterior distribution. At each step of the iteration, one of
the hyperparameters has a small value and the remaining two have large values, sothat two of the three latent variables are suppressed. During the course of the Gibbs
sampling, the solution makes sharp transitions between the three modes.
The model described here involves a prior only over the matrix W. A fully
Bayesian treatment of PCA, including priors over 1-", a 2 , and n, and solved using variational methods, is described in Bishop (1999b). For a discussion of various Bayesian approaches to detennining the appropriate dimensionality for a PCA
model, see Minka (2001c).

12.2.4

Factor analysis

Factor analysis is a linear-Gaussian latent variable model that is closely related
to probabilistic PCA. Its definition differs from that of probabilistic PCA only in thatthe conditional distribution of the observed variable x given the latent variable z is584

12. CONTINUOUS LATENT VARIABLES

•••
• • •

•
•

•

•

•

•

••

•

•• ••

•

•• •• •••••••• • ••••••
Figure 12.14

•

•
•
• •

•

•• •
•

• • ••••'Hinloo' diagrams of the matrix W in which each element 01 the matrix is depicted asa square (white lor positive and black lor negative values) whose area is proportional
to the magnitude of that element. The synthetic data sel comprises 300 data points in
D = 10 dimensions sampled from a Gaussian distribution having standard deviation 1.0
in 3 directions and standard deviation 0.5 in the remaining 7 directions for a data set in
D = 10 dimensions having AT = 3 directions with larger variance than the remaining 7
directions. The left-hand plol shows the result Irom maximum likelihood probabilistic PCA,
and the left·hand plot shows the corresponding resuft from Bayesian peA. We see how
the Bayesian model is able to discover the appropriate dimensionality by suppressing the
6 surplus degrees of freedom.

taken to have a diagonal rather than an isotropic covariance so thatp(xlz) = N(xlWz + 1'. \II)(12.64)

where ill is a D x D diagonal matrix. Note that the factor analysis model, in common
with probabilistic PCA. assumes that the observed variables Xl, ... ,Xo are independent. given the latent variable z. In essence. the factor analysis model is explaining
the observed covariance structure of the data by representing the independent variance associated with each coordinate in the matrix 1J.' and capturing the covariance
between variables in the matrix W. In the factor analysis literature. the columnsof W. which capture the correlations between observed variables. are calledfaclOrloadings. and the diagonal elements of 1J.'. which represent the independent noise
variances for each of the variables, are called llniqllenesses.
The origins of factor analysis are as old as those of PCA. and discussions offactor analysis can be found in the books by Everitt (1984). Bartholomew (1987),
and Basilevsky (1994). Links between factor analysis and PCA were investigatedby Lilwley (1953) and Anderson (1963) who showed that at stationary points of
the likelihood function. for a faclOr analysis model with 1J.' = (121, the columns of
W are scaled eigenvectors of the sample covariance matrix. and (12 is the averageof the discarded eigenvalues. Later. Tipping and Bishop (1999b) showed that the
maximum of the log likelihood function occurs when the eigenvectors comprising
Ware chosen to be the principal eigenvectors.
Making use of (2.115). we see that the marginal distribution for the observed

12.2.l'ru":ohilislk I'CA

585

FllIure12.15 Gillbs .,,,,>p!j"lllo< Bay<lslan
PCA sh<Ming plots oj Ino,
versus ~eralion number br
three " values. showing
tr"""tions
betw.....
tbeth"'" moOts <A !he posterior
distribution.

,-"riabl, i' gi,-,n by 1'( x) ~ N( Xlj', C) whe ... now

C=WWT+'i'.
Eurr"e 12,19

S"na" 12.4

(12,6~)

As with probabilistic PC A, thi, moMI is im-"ri.rrl to 'Olalions in 11><0 latent 'pace.
Histoocally, factor anal)',;s has been lhe ,ubjerl of cOl1tro,-ersy wroe" a!tempt<h",-e bttn "'a<k: to place an intc'P"'t"lioo on the ind;vidual faclon (the cOOfdinatesin z_space). which h3.\ pm"en problematic due to lr.e """i<lcmifiabilily of factOf
analysis associmed with Mation' in this 'pace. From oor perspeoh-e, howe,-er. weshall .iew factor analysis as a form of lalent "ariable densily model. in which theform of tl>c lalent 'pace i' of interest but nO! the particular choicc of coordinates
used to descrit>c il. If we wish to remove the degeneracy a'sociated with latent 'paceroIations. ""e mu,t con'ider non-Gaussian latent ,-"riable di'tribution,. gi"irrg rise 10
independent component .n.lysi, (lCA) models.
We can detenni"e the parameters I'. \V. "nd .... in the fac!Of an.ly,i, model by
muimum likelihood. 11.. solution for I' i' ag"in given by the ",,,,pie "'ean. How·
eyC'. "nli~e probabili,tic l'CA.lllcre i' no longer a closed-form maximum likelihood
solution for \V. ",'hich mu.\ltherdorc be found i'er.li,'c1)'_ Because faclor anal)',i. isa latent variable modeL thi' can be don. using an EM algorilhm (R.bin and Thayer.
1982) !h"t is "nalogou, to the one used (Of pml>;lbili.tie PeA. Specihcally. lhe E-'lep
eqnJtioo, are g;'-en by

E[zoJ =
E[z"z~J _GW T",-'(x n - xl
G + E[zo]E[z,,]T(12,66)(1267)

where ""e h",'e defi""d(l2,6H)

NOie th"t thi' i' e.pre,<ed in a for'" thai in,-oh'es inycrsi"n of mal rices "f SilO ,\ I x ,Ifrathe'lhan D x D (ex"",,,, for tbe D x D diagooal matrix oJ' "'hose in,-erse i.' 'ri"ial586

12. CONTINUOUS LATENT VARIABLES

Exercise 12.22

to compute in O(D) steps), which is convenient because often M « D. Similarly,
the M-step equations take the formw new[~(x" -XllllIZn]"] [~Ill[Znz~I]-'diag

Exercise 12.25

{s-

W.'w

~ ~1ll[Zn](Xn _ xl" }(12.69)(12.70)where the 'diag' operator sets all of the nondiagonal elements of a matrix to zero. A
Bayesian treatment of the factor analysis model can be obtained by a straightforward
application of the techniques discussed in this book.
Another difference between probabilistic PCA and factor analysis concerns theirdifferent behaviour under transformations of the data set. For PCA and probabilistic PCA, if we rotate the coordinate system in data space, then we obtain exactly
the same fit to the data but with the W matrix transformed by the corresponding
rotation matrix. However, for factor analysis, the analogous property is that if wemake a component-wise re-scaling of the data vectors, then this is absorbed into a
corresponding re-scaling of the elements of \)i.

12.3. Kernel peA
In Chapter 6, we saw how the technique of kernel substitution allows us to take analgorithm expressed in terms of scalar products of the form x T x' and generalizethat algorithm by replacing the scalar products with a nonlinear kernel. Here weapply this technique of kernel substitution to principal component analysis, thereby
obtaining a nonlinear generalization called kernel peA (Scholkopf et al., 1998).
Consider a data set {x n } of observations, where n = 1, ... , N, in a space ofdimensionality D. In order to keep the notation uncluttered, we shall assume that
we have already subtracted the sample mean from each of the vectors X n , so that
Ln X n = O. The first step is to express conventional PCA in such a form that thedata vectors {x n } appear only in the form of the scalar products x~ X m . Recall thatthe principal components are defined by the eigenvectors Ui of the covariance matrix
SUi = AiUi

(12.71)

where i = 1, ... ,D. Here the D x D sample covariance matrix S is defined by(12.72)uT

and the eigenvectors are normalized such thatUi = 1.
Now consider a nonlinear transformation ¢(x) into an M -dimensional feature
space, so that each data point X n is thereby projected onto a point ¢(xn ). We can12.3. K~mci l'Co'.

587

'\

...

Figu'.12.16 SctIematic _,.lion 01 kernel PeA. A <Utll HI In lhe Oflglnal <Uta space l~'_ plot} ..

PfOlEled by' """'*- tranllklfmalion ~,,} 1nIo. fa.tur. space (fIght_ plot). By I*b'~ PCA in the!Hue 111**. we oblaO'Ilha pmeiIlaI ~"llS. "'who<:tllha tnt Ie ......... in blUe.,.., .. oJano4ed by lha_v, Tha gr-. .... In IMIuN apam indicMa Iha _plOlKlio". onIO lhe Iirsl poiridl* '""" ......... 11.
_od 110 ""'*- poOf&Cllu. in .... <lfisIonaI Oillll 111**. Hole IIuIIIn gMefM' la nol pox ' .. 110'. . . . . . 1ha,... iIio_ poi ........ 00i' ........ by._1n "apam.

"""'*'__ ptrform )l-.bnl PeA ill
fnlllK lopICe. ..-Iudl,mpIiclIly «lIi'Il'S • - ' _....-.. model ill
onpnll cbuo ~ as ,1lU>tr*d in FllIft 12-16.
"IO..It. . lei '" lOS!oUlllt 1Nl Illt ~ diu. ~ lObo halnro mean,
4>("'. J .. O. We dWl rctlll'1l 10 Itl,~ pol'" .Ihonly. 1llt.1f " .If
CO\-.ullCC mMfU ,n (~ .If*'e ,~ l " " byprinClpaIFujI) !hal"'"""*L.

,

C-.~. L <>(x.j4>(X.)T..,(12.73)and ,l~ ",,,,n'"MOl" opan,ion i' «lined by
Cv, = A,v,

(12,74); = 1... ,. M. Our goal is 10 soh'" lhis eigen"lIlue problem WilhoUl ha"inlllO work."plici,ly in ,he f.lIture 'pace. From !he definilion of C. lhe .ill"""""l"'" equal ionslell' U, thaI Y, !-ali,fies...

s L" <bC".) {<b(x.lT v,} - )"Y,,(12.7~)........ ..-" . - lN1 (proo.idcd A, > 0) tilt "CC'lor v, is li''n by • I i _ rombllla"onof Illt d>(

J..... JO <;.- he "-"llm iIllhc (orm,v, '"'

L 11,.4>(".).

...

(12.76)

588

12. CONTINUOUS LATENT VARIABLES
Substituting this expansion back into the eigenvector equation, we obtain
1 N
NN
N L ¢(xn)¢(xn)T L aim¢(X m ) = Ai L ain¢(Xn ),
n=l
m=l
n=l

(12.77)The key step is now to express this in terms of the kernel function k (x n , x m ) =¢(Xn)T ¢(xm ), which we do by multiplying both sides by ¢(xZ)T to give
1 NmN
N Lk(XI'Xn ) L aimk(Xn,xm ) = Ai Laink(XI'Xn ),
n=l
m=l
n=l

(12.78)This can be written in matrix notation as(12.79)where ai is an N-dimensional column vector with elements ani for n = 1, ... ,N.
We can find solutions for ai by solving the following eigenvalue problem(12.80)Exercise 12.26

in which we have removed a factor of K from both sides of (12.79). Note thatthe solutions of (12.79) and (12.80) differ only by eigenvectors of K having zero
eigenvalues that do not affect the principal components projection.
The normalization condition for the coefficients ai is obtained by requiring thatthe eigenvectors in feature space be normalized. Using (12.76) and (12.80), we haveN

1 = V;Vi =N

L L ainaim¢(xn)T¢(xm ) = a;K~ = AiNa;ai'(12.81)n=l m=lHaving solved the eigenvector problem, the resulting principal component projections can then also be cast in terms of the kernel function so that, using (12.76),
the projection of a point x onto eigenvector i is given by
Yi(X) = ¢(x)TVi =

N

N

n=l

n=l

L ain¢(x)T ¢(xn ) = L aink(X, x n )(12.82)and so again is expressed in terms of the kernel function.
In the original D-dimensional x space there are D orthogonal eigenvectors andhence we can find at most D linear principal components. The dimensionality Mof the feature space, however, can be much larger than D (even infinite), and thus
we can find a number of nonlinear principal components that can exceed D. Note,however, that the number of nonzero eigenvalues cannot exceed the number N ofdata points, because (even if M > N) the covariance matrix in feature space has
rank at most equal to N. This is reflected in the fact that kernel PCA involves the
eigenvector expansion of the N x N matrix K.

12.3. Kernel PCA

589

So far we have assumed that the projected data set given by ¢(xn ) has zeromean, which in general will not be the case. We cannot simply compute and thensubtract off the mean, since we wish to avoid working directly in feature space, andso again, we formulate the algorithm purely in-!erms of the kernel function. The
projected data points after centralizing, denoted ¢(xn ), are given by(12.83)and the corresponding elements of the Gram matrix are given by

K nm

=

¢(xn)T¢(xm )1 N¢(xn)T¢(xm ) - N L¢(xn)T¢(xZ)

Z=l
1

1

N

N

N

- N L¢(XZ)T¢(Xm ) + N2 LL¢(Xj)T¢(xZ)
Z=l
j=l Z=l
1 N
k(x n , x m ) - N L k(xz, x m )
Z=l
1 N
1 N N
(12.84)- N Lk(xn,xz) + N2 LLk(Xj,Xl)'
Z=l
j=l 1=1
This can be expressed in matrix notation as(12.85)where IN denotes the N x N matrix in which every element takes the value l/N.
Thus we can evaluate K using only the kernel function and then use K to determine
the eigenvalues and eigenvectors. Note that the standard PCA algorithm is recovered
as a special case if we use a linear kernel k(x, x') = xTx/. Figure 12.17 shows anexample of kernel PCA applied to a synthetic data set (Scholkopf et al., 1998). Herea 'Gaussian' kernel of the form~

Exercise 12.27~

k(x, x') = exp( -llx - x/11 2 /0.1)(12.86)

is applied to a synthetic data set. The lines correspond to contours along which the
projection onto the corresponding principal component, defined by
N¢(X?Vi = L
n=lis constant.

aink(X, x n )(12.87)590_.

12. CONTINUOUS LATENT VAlUABLES

Figure 12.11 E"llmple 01 kernel PCA, with a Gaussian kernel awIiOO 10 a synthetic <lata sat in two <:Iirnensions,
showing !he firsl flight eigenfunclions along w~h l!>eir e9tnvailNls. The oootours am lines along which !heprojoc1ion onlo
COffaspMding principal componen1ls co<>stam, Nola haw Ihe firsl two ~ ....,..rat.
!he th"'" dusters. !he ""'"
~ spIiI
oIlhe eluste, into haMoS. and t"" lolIowing Ihree~ again spI~!he duste," into halves along directions orthogonal 10 tho prEMouS splils,t""Ill"'"""'*'

One obvioo' dls.aJmota~eof I:emel !'CA Is thaf if invoh'es finding lhe elgen"e<tors of the N x N malri>: K raW. Ihan lhe D x D malri, S of cor,..emionallinear!'CA. and!iO In prac1lce for large data "'1' appro,lmation< are often uS(:dFinally. ""e OOIe that i" <tandard linear I'CA, we often retain some redoce<l num·ber L < Dof eigenvectors and then appro,lmale 0 data vttl<:>r Xn b}' its projection
i~ 0,,1" lhe L-dimensional principal subspace, defined by,

i~-L:«",)""(12.88)I" kernell'CA. this will in gencr~1 not be flO'slble, To see thl', OOIe Ihat the mapping 4'(x) maps the D-dimensional x space i"t" 0 D-dimensioo.l manijQiII in lhe
M-dimemioo.l femure space <1>. TlIe: .'ector x i' koown a< lhe f'",.imagr of lhec","""ponding poi"l 4'(x). However, fhe projec1ioo of poinl> in feature <J'3C" ""tothe linear rcA ,ub,p""" in that 'pace will typically"''' lie On fhe nonlinear Ddimensional manifold and !iO will nul
a c"""",pondlng p",.lmo~e in dOlO spa<."C,Technlque< ho.-e lherefore bttn proposed for finding approximale pre-image< iB""lrha.,.

Nat.. 2(04).

12.4. Nonlinear Latent Variable Models

591

12.4. Nonlinear Latent Variable Models

Exercise 12.28

In this chapter, we have focussed on the simplest class of models having continuous
latent variables, namely those based on linear-Gaussian distributions. As well as
having great practical importance, these models are relatively easy to analyse andto fit to data and can also be used as components in more complex models. Herewe consider briefly some generalizations of this framework to models that are either
nonlinear or non-Gaussian, or both.
In fact, the issues of nonlinearity and non-Gaussianity are related because ageneral probability density can be obtained from a simple fixed reference density,
such as a Gaussian, by making a nonlinear change of variables. This idea forms thebasis of several practical latent variable models as we shall see shortly.

12.4.1

Independent component analysis

We begin by considering models in which the observed variables are related
linearly to the latent variables, but for which the latent distribution is non-Gaussian.
An important class of such models, known as independent component analysis, orleA, arises when we consider a distribution over the latent variables that factorizes,
so thatMp(z) = IIp(Zj).

(12.89)j=l

To understand the role of such models, consider a situation in which two people
are talking at the same time, and we record their voices using two microphones.
If we ignore effects such as time delay and echoes, then the signals received by
the microphones at any point in time will be given by linear combinations of theamplitudes of the two voices. The coefficients of this linear combination will be
constant, and if we can infer their values from sample data, then we can invert the
mixing process (assuming it is nonsingular) and thereby obtain two clean signals
each of which contains the voice of just one person. This is an example of a problem
called blind source separation in which 'blind' refers to the fact that we are given
only the mixed data, and neither the original sources nor the mixing coefficients areobserved (Cardoso, 1998).
This type of problem is sometimes addressed using the following approach(MacKay, 2003) in which we ignore the temporal nature of the signals and treat thesuccessive samples as i.i.d. We consider a generative model in which there are twolatent variables corresponding to the unobserved speech signal amplitudes, and there
are two observed variables given by the signal values at the microphones. The latentvariables have a joint distribution that factorizes as above, and the observed variables
are given by a linear combination of the latent variables. There is no need to include
a noise distribution because the number of latent variables equals the number of observed variables, and therefore the marginal distribution of the observed variables
will not in general be singular, so the observed variables are simply deterministic
linear combinations of the latent variables. Given a data set of observations, the592

12. CONTINUOUS LATENT VARIABLES

Exercise 12.29

likelihood function for this model is a function of the coefficients in the linear combination. The log likelihood can be maximized using gradient-based optimization
giving rise to a particular version of independent component analysis.
The success of this approach requires that the latent variables have non-Gaussian
distributions. To see this, recall that in probabilistic PCA (and in factor analysis) the
latent-space distribution is given by a zero-mean isotropic Gaussian. The model
therefore cannot distinguish between two different choices for the latent variables
where these differ simply by a rotation in latent space. This can be verified directly
by noting that the marginal density (12.35), and hence the likelihood function, is
unchanged if we make the transformation W -) WR where R is an orthogonal
matrix satisfying RRT = I, because the matrix C given by (12.36) is itself invariant.
Extending the model to allow more general Gaussian latent distributions does notchange this conclusion because, as we have seen, such a model is equivalent to thezero-mean isotropic Gaussian latent variable model.
Another way to see why a Gaussian latent variable distribution in a linear modelis insufficient to find independent components is to note that the principal components represent a rotation of the coordinate system in data space such as to diagonalize the covariance matrix, so that the data distribution in the new coordinates is thenuncorrelated. Although zero correlation is a necessary condition for independence
it is not, however, sufficient. In practice, a common choice for the latent-variable
distribution is given by

1
7fcosh(zj)p(z) = --,.-----,J1(12.90)which has heavy tails compared to a Gaussian, reflecting the observation that many
real-world distributions also exhibit this property.
The original ICA model (Bell and Sejnowski, 1995) was based on the optimization of an objective function defined by information maximization. One advantage
of a probabilistic latent variable formulation is that it helps to motivate and formulate generalizations of basic ICA. For instance, independent factor analysis (Attias,
1999a) considers a model in which the number of latent and observed variables candiffer, the observed variables are noisy, and the individual latent variables have flexible distributions modelled by mixtures of Gaussians. The log likelihood for thismodel is maximized using EM, and the reconstruction of the latent variables is approximated using a variational approach. Many other types of model have been
considered, and there is now a huge literature on ICA and its applications (Jutten
and Herault, 1991; Comon et at., 1991; Amari et at., 1996; Pearlmutter and Parra,
1997; Hyvarinen and Oja, 1997; Hinton et at., 2001; Miskin and MacKay, 2001;
Hojen-Sorensen et at., 2002; Choudrey and Roberts, 2003; Chan et at., 2003; Stone,2004).

12.4.2

Autoassociative neural networks

In Chapter 5 we considered neural networks in the context of supervised learning, where the role of the network is to predict the output variables given values12.4. Nonlinear Latent Variable Models
Figure 12.18 An autoassociative mUltilayer perceptron having
two layers of weights. Such a network is trained tomap input vectors onto themselves by minimization ot a sum-ot-squares error. Even with nonlinear units in the hidden layer, such a network
is equivalent to linear principal component anal- inputs
ysis. Links representing bias parameters havebeen omitted for clarity.

593

outputs

for the input variables. However, neural networks have also been applied to unsupervised learning where they have been used for dimensionality reduction. Thisis achieved by using a network having the same number of outputs as inputs, andoptimizing the weights so as to minimize some measure of the reconstruction error
between inputs and outputs with respect to a set of training data.
Consider first a multilayer perceptron of the form shown in Figure 12.18, having D inputs, D output units and M hidden units, with M < D. The targets used
to train the network are simply the input vectors themselves, so that the network
is attempting to map each input vector onto itself. Such a network is said to form
an autoassociative mapping. Since the number of hidden units is smaller than thenumber of inputs, a perfect reconstruction of all input vectors is not in general possible. We therefore determine the network parameters w by minimizing an error
function which captures the degree of mismatch between the input vectors and theirreconstructions. In particular, we shall choose a sum-of-squares error of the form
1 N

E(w) = "2

L Ily(x w) - x 11
n ,

n

2

•

(12.91)n=l

If the hidden units have linear activations functions, then it can be shown that theerror function has a unique global minimum, and that at this minimum the network
performs a projection onto the M -dimensional subspace which is spanned by the first
M principal components of the data (Bourlard and Kamp, 1988; Baldi and Hornik,
1989). Thus, the vectors of weights which lead into the hidden units in Figure 12.18
form a basis set which spans the principal subspace. Note, however, that these vectors need not be orthogonal or normalized. This result is unsurprising, since bothprincipal component analysis and the neural network are using linear dimensionality
reduction and are minimizing the same sum-of-squares error function.
It might be thought that the limitations of a linear dimensionality reduction could
be overcome by using nonlinear (sigmoidal) activation functions for the hidden units
in the network in Figure 12.18. However, even with nonlinear hidden units, the minimum error solution is again given by the projection onto the principal component
subspace (Bourlard and Kamp, 1988). There is therefore no advantage in using twolayer neural networks to perform dimensionality reduction. Standard techniques forprincipal component analysis (based on singular value decomposition) are guaranteed to give the correct solution in finite time, and they also generate an ordered setof eigenvalues with corresponding orthonormal eigenvectors.

594

12. CONTINUOUS LATENT VARIABLES
Figure 12.19 Addition of extra hidden layers of noolinear units gives an
auloassocialive network which
can perform a noolinear dimensiooality reduction.

F,

•

F,

•
outputs

inputsx,x,

The situation is different, however. if additional hidden layers are pcrmillcd inthe network. Consider the four-layer autoassociativc network shown in Figure 12.19.

Again the output units are linear, and the M units in the second hidden layer can also

be linear. however, the first and third hidden layers have sigmoidal nonlinear activation functions. The network is again trained by minimization of the error function(12.91). We can view this network as two successive functional mappings F] andF 2 , as indicated in Figure 12.19. The first mapping F] projects the original Ddimensional data onto an AI-dimensional subspace S defined by the activations of
the units in the second hidden layer. Because of the presence of the first hidden layerof nonlinear units. this mapping is very general. and in particular is not restricted tobeing linear. Similarly. the second half of the network defines an arbitrary functional
mapping from the M -dimensional space back into the original D-dimensional input
space. This has a simple geometrical interpretation. as indicated for the case D = 3
and M = 2 in Figure 12.20.
Such a network effectively perfonns a nonlinear principal component analysis.
X3

F, "

•
x,

"

Figure 12.20 Geometrical interpretation of the mappings performed by the network in Figure 12.1 g for the caseof 0 = 3 inputs and AI = 2 units in the middle hidden layer. The function F, maps from an M-dimensional
space S into a D-dimensiooal space and therefore defines the way in which the space S is embedded within theoriginal x-space. Since the mapping F, can be r"I()(llinear, the embedding 01 S can be nonplanar, as indicatedin the figure. The mapping F. then defines a projectiorl of points in the original D-dimensional space into the
M -dimensional subspace S.

12.4. Nonlinear Latent Variable Models

595

It has the advantage of not being limited to linear transformations, although it contains standard principal component analysis as a special case. However, training
the network now involves a nonlinear optimization problem, since the error function(12.91) is no longer a quadratic function of the network parameters. Computationally intensive nonlinear optimization techniques must be used, and there is the risk offinding a suboptimal local minimum of the error function. Also, the dimensionalityof the subspace must be specified before training the network.

12.4.3Modelling nonlinear manifolds

As we have already noted, many natural sources of data correspond to lowdimensional, possibly noisy, nonlinear manifolds embedded within the higher dimensional observed data space. Capturing this property explicitly can lead to improved density modelling compared with more general methods. Here we consider
briefly a range of techniques that attempt to do this.
One way to model the nonlinear structure is through a combination of linear
models, so that we make a piece-wise linear approximation to the manifold. This canbe obtained, for instance, by using a clustering technique such as K -means based on
Euclidean distance to partition the data set into local groups with standard PCA applied to each group. A better approach is to use the reconstruction error for cluster
assignment (Kambhatla and Leen, 1997; Hinton et al., 1997) as then a common costfunction is being optimized in each stage. However, these approaches still sufferfrom limitations due to the absence of an overall density model. By using probabilistic PCA it is straightforward to define a fully probabilistic model simply byconsidering a mixture distribution in which the components are probabilistic PCA
models (Tipping and Bishop, 1999a). Such a model has both discrete latent variables, corresponding to the discrete mixture, as well as continuous latent variables,
and the likelihood function can be maximized using the EM algorithm. A fully
Bayesian treatment, based on variational inference (Bishop and Winn, 2000), allows
the number of components in the mixture, as well as the effective dimensionalitiesof the individual models, to be inferred from the data. There are many variants ofthis model in which parameters such as the W matrix or the noise variances are tied
across components in the mixture, or in which the isotropic noise distributions are
replaced by diagonal ones, giving rise to a mixture of factor analysers (Ghahramaniand Hinton, 1996a; Ghahramani and Beal, 2000). The mixture of probabilistic PCA
models can also be extended hierarchically to produce an interactive data visualization algorithm (Bishop and Tipping, 1998).
An alternative to considering a mixture of linear models is to consider a single
nonlinear model. Recall that conventional PCA finds a linear subspace that passes
close to the data in a least-squares sense. This concept can be extended to onedimensional nonlinear surfaces in the form of principal curves (Hastie and Stuetzle,1989). We can describe a curve in a D-dimensional data space using a vector-valued
function f ().), which is a vector each of whose elements is a function of the scalar )..
There are many possible ways to parameterize the curve, of which a natural choiceis the arc length along the curve. For any given point in data space, we can findthe point on the curve that is closest in Euclidean distance. We denote this point by

x

596

12. CONTINUOUS LATENT VARIABLES>.. = gf(X) because it depends on the particular curve f(>"). For a continuous data
density p(x), a principal curve is defined as one for which every point on the curveis the mean of all those points in data space that project to it, so thatJE [Xlgf(X) = >..] = f(>").

(12.92)For a given continuous density, there can be many principal curves. In practice, we
are interested in finite data sets, and we also wish to restrict attention to smooth
curves. Hastie and Stuetzle (1989) propose a two-stage iterative procedure for finding such principal curves, somewhat reminiscent of the EM algorithm for PCA. The
curve is initialized using the first principal component, and then the algorithm alternates between a data projection step and curve re-estimation step. In the projection
step, each data point is assigned to a value of >.. corresponding to the closest pointon the curve. Then in the re-estimation step, each point on the curve is given bya weighted average of those points that project to nearby points on the curve, withpoints closest on the curve given the greatest weight. In the case where the subspaceis constrained to be linear, the procedure converges to the first principal component
and is equivalent to the power method for finding the largest eigenvector of the covariance matrix. Principal curves can be generalized to multidimensional manifolds
called principal surfaces although these have found limited use due to the difficultyof data smoothing in higher dimensions even for two-dimensional manifolds.
PCA is often used to project a data set onto a lower-dimensional space, for example two dimensional, for the purposes of visualization. Another linear techniquewith a similar aim is multidimensional scaling, or MDS (Cox and Cox, 2000). It finds
a low-dimensional projection of the data such as to preserve, as closely as possible,
the pairwise distances between data points, and involves finding the eigenvectors ofthe distance matrix. In the case where the distances are Euclidean, it gives equivalent
results to PCA. The MDS concept can be extended to a wide variety of data types
specified in terms of a similarity matrix, giving nonmetric MDS.
Two other nonprobabilistic methods for dimensionality reduction and data visualization are worthy of mention. Locally linear embedding, or LLE (Roweis andSaul, 2000) first computes the set of coefficients that best reconstructs each data
point from its neighbours. These coefficients are arranged to be invariant to rotations, translations, and scalings of that data point and its neighbours, and hence they
characterize the local geometrical properties of the neighbourhood. LLE then maps
the high-dimensional data points down to a lower dimensional space while preserving these neighbourhood coefficients. If the local neighbourhood for a particular
data point can be considered linear, then the transformation can be achieved using
a combination of translation, rotation, and scaling, such as to preserve the angles
formed between the data points and their neighbours. Because the weights are invariant to these transformations, we expect the same weight values to reconstruct thedata points in the low-dimensional space as in the high-dimensional data space. In
spite of the nonlinearity, the optimization for LLE does not exhibit local minima.
In isometric feature mapping, or isomap (Tenenbaum et ai., 2000), the goal isto project the data to a lower-dimensional space using MDS, but where the dissimilarities are defined in terms of the geodesic distances measured along the mani-12.4. Nonlinear Latent Variable Models

Chapter 5

Chapter JJ

597

fold. For instance, if two points lie on a circle, then the geodesic is the arc-length
distance measured around the circumference of the circle not the straight line distance measured along the chord connecting them. The algorithm first defines theneighbourhood for each data point, either by finding the K nearest neighbours or byfinding all points within a sphere of radius E. A graph is then constructed by linking all neighbouring points and labelling them with their Euclidean distance. The
geodesic distance between any pair of points is then approximated by the sum of
the arc lengths along the shortest path connecting them (which itself is found using
standard algorithms). Finally, metric MDS is applied to the geodesic distance matrix
to find the low-dimensional projection.
Our focus in this chapter has been on models for which the observed variables are continuous. We can also consider models having continuous latent variables together with discrete observed variables, giving rise to latent trait models(Bartholomew, 1987). In this case, the marginalization over the continuous latentvariables, even for a linear relationship between latent and observed variables, cannot be performed analytically, and so more sophisticated techniques are required.
Tipping (1999) uses variational inference in a model with a two-dimensional latent
space, allowing a binary data set to be visualized analogously to the use of PCA tovisualize continuous data. Note that this model is the dual of the Bayesian logistic
regression problem discussed in Section 4.5. In the case of logistic regression we
have N observations of the feature vector <l>n which are parameterized by a single
parameter vector w, whereas in the latent space visualization model there is a single
latent space variable x (analogous to <1» and N copies of the latent variable W n . A
generalization of probabilistic latent variable models to general exponential family
distributions is described in Collins et al. (2002).
We have already noted that an arbitrary distribution can be formed by taking a
Gaussian random variable and transforming it through a suitable nonlinearity. Thisis exploited in a general latent variable model called a density network (MacKay,
1995; MacKay and Gibbs, 1999) in which the nonlinear function is governed by amultilayered neural network. If the network has enough hidden units, it can approximate a given nonlinear function to any desired accuracy. The downside of havingsuch a flexible model is that the marginalization over the latent variables, required inorder to obtain the likelihood function, is no longer analytically tractable. Instead,the likelihood is approximated using Monte Carlo techniques by drawing samples
from the Gaussian prior. The marginalization over the latent variables then becomesa simple sum with one term for each sample. However, because a large numberof sample points may be required in order to give an accurate representation of themarginal, this procedure can be computationally costly.
If we consider more restricted forms for the nonlinear function, and make an appropriate choice of the latent variable distribution, then we can construct a latent variable model that is both nonlinear and efficient to train. The generative topographic
mapping, or GTM (Bishop et aI., 1996; Bishop et aI., 1997a; Bishop et aI., 1998b)
uses a latent distribution that is defined by a finite regular grid of delta functions overthe (typically two-dimensional) latent space. Marginalization over the latent spacethen simply involves summing over the contributions from each of the grid locations.

598

12. CONTINUOUS LATENT VA K1AHU':S

.•.

'"

FllIu.e 12.21 ?lot ot trle oillkYw <:lata Wllisualiz.ed using PeA on the left and GTM on Itle ngr,t FOf tile GTM
model. each <lata poinIls plollfld at tile mean ot its posM'k>< dislribution in ..tent s;>ace, Tile "","ineanty mlhe
GTM 1TlOd8I._lha sepamlion betwoon the groups of data points to be .....n """. ckl.arfy,
Ch"l'l~f j
S~etioo/.4

The no"liotar mapping is gi,'en by a linear regression model thaI allow, for general
IIO/llinearily while being a linear fuoction of tile adapli'-e parameler<, NOIe thaI tilt
usual limitation of linear regression models arising from the en"", of dimen,iooalily
does 1101 arise in the Contr~1 of lhe GT~I si""'e the "\3nifold generall)' ha< t,,'o di"ltn·sions irrespecti'-e of the dimensionality of the data space, A coo""!",,nce of Illese
11"0 cooices is that the likelihood funclion can be e~pressed analytically in dosed
form and can be optimilC<.! efficiently o,ing the EM algorithm_ The resolting GTM
model hIs a lwo-dimensional nonlinear manifold 10 tile dala sel. and by e"alualing
the posterior distrilJ",lion (Wer latent space for the data poi"", they can he projectt<Jback to the lalent 'JI'K'" for .'isualilalion purposes, Figure 12,21 sl"""s a comparisonof the oil data..,1 "isualired wilh lincar PeA and wilh lhe IIO/lhnear GT~I,
TIlt GTM can be seen as a probabilistic "'rsion of an earlier nlOd<l callM theorg"nidng ""'p. or SOM (Kohonen. 1982: Kobonen. (995). which also representsa Iwo-dimensiooal IIO/llincar manifoid as a regular array of disc"'le points. The
SOM i' somewhat remin;""'nt of the K·trlCan, algorithm in that data points area.,igr.ed to nearby ProlOl)'j>C '-eclon thaI are lhen subsc<juenlly updale<!. Initially.
lhe proIOI)'jl('S are distribuled at random, and during the training process they 'selrorganize' so as to aPl'ro~imale a smoolh manifold. Unlike K -mean'. how'e"e.. theSOM is TIOI optimizing any well.ddine<! cost function (Erwin .. al.. 1992) making"difficult to s." the parameters of the model and 10 assess con'-ergence. There i' also
no guarantee that the '",If-<>rganilalion' will take place .. this is depen""nl 00 the
choice of appropriate paranlttcr "aloC' f,,, any particular data sel.
By OOfItrast, GTM optimize, the log likelihood functioo, and the resolting model
define' a probabilily den,ity in dma ,pace, In faeL il corre,ponds to a con,m,incd
mi,ture of Gaussian, in which the component.' ,h.re a COnlnlOn ".riance.•nd themean, are con'trained to lie on a 'mooIh tw-o-diITlCn,iooal n1anifold. This proba-'''IfExercises

bilistic foundation also makes it very straightforward to define generalizations of
GTM (Bishop et al., 1998a) such as a Bayesian treatment, dealing with missing values, a principled extension to discrete variables, the use of Gaussian processes todefine the manifold, or a hierarchical GTM model (Tino and Nabney, 2002).
Because the manifold in GTM is defined as a continuous surface, not just at the
prototype vectors as in the SOM, it is possible to compute the magnification factors
corresponding to the local expansions and compressions of the manifold needed to
fit the data set (Bishop et al., 1997b) as well as the directional curvatures of the
manifold (Tino et al., 2001). These can be visualized along with the projected dataand provide additional insight into the model.

Section 6.4

Exercises
12.1

Appendix E

599

lIB

(* *)
In this exercise, we use proof by induction to show that the linear
projection onto an M -dimensional subspace that maximizes the variance of the projected data is defined by the M eigenvectors of the data covariance matrix S, givenby (12.3), corresponding to the M largest eigenvalues. In Section 12.1, this result
was proven for the case of M = 1. Now suppose the result holds for some generalvalue of M and show that it consequently holds for dimensionality M + 1. To dothis, first set the derivative of the variance of the projected data with respect to avector UM+1 defining the new direction in data space equal to zero. This should
be done subject to the constraints that UM +l be orthogonal to the existing vectors
U1,"" UM, and also that it be normalized to unit length. Use Lagrange multipliers to enforce these constraints. Then make use of the orthonormality properties ofthe vectors U1,"" UM to show that the new vector UM+1 is an eigenvector of S.
Finally, show that the variance is maximized if the eigenvector is chosen to be theone corresponding to eigenvector AM+1 where the eigenvalues have been ordered indecreasing value.

12.2 (**) Show that the minimum value of the PCA distortion measure J given by(12.15) with respect to the Ui, subject to the orthonormality constraints (12.7), is
obtained when the Ui are eigenvectors of the data covariance matrix S. To do this,introduce a matrix H of Lagrange multipliers, one for each constraint, so that themodified distortion measure, in matrix notation reads] = Tr {UTSU} + Tr { H(I - UTU) }(12.93)where U is a m~trix of dimensio~D x (D - M) whose columns are gi:::..en b~ Ui.
Now minimize J with respect to U and show that the s~ution satisfies SU = UH.
Clearly, one possible solution is that the columns of U are eigenvectors of S, inwhich case H is a diagonal matrix containing the corresponding eigenvalues. To
obtain the general solution, show that H can be assumed to be a symmetr~ ma~ix,
and by using its eigenvect£r expansion show that the general solution to SU =~UH
gives the same value for J as the specific solution in which the columns of U are600

12. CONTINUOUS LATENT VARIABLES
the eigenvectors of S. Because these solutions are all equivalent, it is convenient tochoose the eigenvector solution.

12.3 (*)

Verify that the eigenvectors defined by (12.30) are normalized to unit length,
assuming that the eigenvectors Vi have unit length.

12.4 (*) Imm Suppose we replace the zero-mean, unit-covariance latent space distribution (12.31) in the probabilistic PCA model by a general Gaussian distribution of
the formN(zlm, ~). By redefining the parameters of the model, show that this leads
to an identical model for the marginal distribution p(x) over the observed variables
for any valid choice of m and ~.

12.5 (* *)Let x be a D-dimensional random variable having a Gaussian distribution
given by N(xIJL, ~), and consider the M-dimensional random variable given by
y = Ax + b where A is an M x D matrix. Show that y also has a Gaussian
distribution, and find expressions for its mean and covariance. Discuss the form of
this Gaussian distribution for M < D, for M = D, and for M > D.

12.6 (*) Imm Draw a directed probabilistic graph for the probabilistic PCA model
described in Section 12.2 in which the components of the observed variable x are
shown explicitly as separate nodes. Hence verify that the probabilistic PCA modelhas the same independence structure as the naive Bayes model discussed in Section 8.2.2.

12.7 (* *) By making use of the results (2.270) and (2.271) for the mean and covarianceof a general distribution, derive the result (12.35) for the marginal distribution p(x)in the probabilistic PCA model.

12.8 (* *) Imm By making use of the result (2.116), show that the posterior distributionp(zlx) for the probabilistic PCA model is given by (12.42).
12.9 (*)

Verify that maximizing the log likelihood (12.43) for the probabilistic PCA
model with respect to the parameter JL gives the result JLML = x where x is themean of the data vectors.

12.10 (**) By evaluating the second derivatives of the log likelihood function (12.43) forthe probabilistic PCA model with respect to the parameter JL, show that the stationary
point JLML = x represents the unique maximum.

12.11(* *) Imm Show that in the limit (Y2 - . 0, the posterior mean for the probabilistic
PCA model becomes an orthogonal projection onto the principal subspace, as inconventional PCA.

12.12 (* *) For (Y2 > 0 show that the posterior mean in the probabilistic PCA model isshifted towards the origin relative to the orthogonal projection.

12.13 (* *) Show that the optimal reconstruction of a data point under probabilistic PCA,
according to the least squares projection cost of conventional PCA, is given by(12.94)

Exercises

601

12.14 (*) The number of independent parameters in the covariance matrix for the probabilistic PCA model with an M -dimensional latent space and a D-dimensional data
space is given by (12.51). Verify that in the case of M = D - 1, the number of
independent parameters is the same as in a general covariance Gaussian, whereas for
M = it is the same as for a Gaussian with an isotropic covariance.

°

12.15 (**) IIiI!I Derive the M-step equations (12.56) and (12.57) for the probabilistic
PCA model by maximization of the expected complete-data log likelihood function
given by (12.53).
12.16 (* * *) In Figure 12.11, we showed an application of probabilistic PCA to a data set
in which some of the data values were missing at random. Derive the EM algorithm
for maximizing the likelihood function for the probabilistic PCA model in this situation. Note that the {zn}, as well as the missing data values that are components ofthe vectors {x n }, are now latent variables. Show that in the special case in which allof the data values are observed, this reduces to the EM algorithm for probabilistic
PCA derived in Section 12.2.2.
12.17 (**) IIiI!I Let W be a D x M matrix whose columns define a linear subspaceof dimensionality M embedded within a data space of dimensionality D, and let J1
be a D-dimensional vector. Given a data set {x n } where n = 1, ... , N, we canapproximate the data points using a linear mapping from a set of M -dimensional
vectors {zn}, so that Xn is approximated by W Zn + J1. The associated sum-ofsquares reconstruction cost is given by
N

J =

L Ilx

n -

2

J1- Wz n 11 .

(12.95)n=lFirst show that minimizing J with respect to J1leads to an analogous expression with
X n and Zn replaced by zero-mean variables X n - x and Zn - Z, respectively, where x
and Z denote sample means. Then show that minimizing J with respect to Zn, where
W is kept fixed, gives rise to the PCA Estep (12.58), and that minimizing J withrespect to W, where {zn} is kept fixed, gives rise to the PCA M step (12.59).
12.18 (*) Derive an expression for the number of independent parameters in the factor
analysis model described in Section 12.2.4.
12.19 (**) IIiI!I Show that the factor analysis model described in Section 12.2.4 isinvariant under rotations of the latent space coordinates.
12.20 (**) By considering second derivatives, show that the only stationary point of
the log likelihood function for the factor analysis model discussed in Section 12.2.4
with respect to the parameter J1 is given by the sample mean defined by (12.1).
Furthermore, show that this stationary point is a maximum.
12.21(**) Derive the formulae (12.66) and (12.67) for the E step of the EM algorithm
for factor analysis. Note that from the result of Exercise 12.20, the parameter J1 canbe replaced by the sample mean x.

602

12. CONTINUOUS LATENT VARIABLES
12.22 (* *) Write down an expression for the expected complete-data log likelihood function for the factor analysis model, and hence derive the corresponding M step equations (12.69) and (12.70).
12.23 (*) III!I Draw a directed probabilistic graphical model representing a discrete
mixture of probabilistic PCA models in which each PCA model has its own valuesof W, JL, and 0- 2 • Now draw a modified graph in which these parameter values areshared between the components of the mixture.
12.24 (***) We saw in Section 2.3.7 that Student's t-distribution can be viewed as an
infinite mixture of Gaussians in which we marginalize with respect to a continuous latent variable. By exploiting this representation, formulate an EM algorithm
for maximizing the log likelihood function for a multivariate Student's t-distribution
given an observed set of data points, and derive the forms of the E and M step equations.
12.25 (**) III!I Consider a linear-Gaussian latent-variable model having a latent space
distribution p(z) = N(xIO, I) and a conditional distribution for the observed variable p(xlz) = N(xlWz + IL, <p) where <P is an arbitrary symmetric, positivedefinite noise covariance matrix. Now suppose that we make a nonsingular linear
transformation of the data variables x ---t Ax, where A is a D x D matrix. If
JLML' W ML and <PML represent the maximum likelihood solution corresponding tothe original untransformed data, show that AJLML' AWML, and A <PMLA T will represent the corresponding maximum likelihood solution for the transformed data set.
Finally, show that the form of the model is preserved in two cases: (i) A is a diagonal
matrix and <P is a diagonal matrix. This corresponds to the case of factor analysis.
The transformed <P remains diagonal, and hence factor analysis is covariant undercomponent-wise re-scaling of the data variables; (ii) A is orthogonal and <P is proportional to the unit matrix so that <P = 0- 2 1. This corresponds to probabilistic PCA.
The transformed <P matrix remains proportional to the unit matrix, and hence probabilistic PCA is covariant under a rotation of the axes of data space, as is the case forconventional PCA.
\

12.26 (**) Show that any vector ai that satisfies (12.80) will also satisfy (12.79). Also,show that for any solution of (12.80) having eigenvalue A, we can add any multipleof an eigenvector of K having zero eigenvalue, and obtain a solution to (12.79)
that also has eigenvalue A. Finally, show that such modifications do not affect the
principal-component projection given by (12.82).
12.27 (* *) Show that the conventional linear PCA algorithm is recovered as a special caseof kernel PCA if we choose the linear kernel function given by k(x, x') = x T x'.
12.28 (* *) III!I Use the transformation property (1.27) of a probability density undera change of variable to show that any density p(y) can be obtained from a fixeddensity q(x) that is everywhere nonzero by making a nonlinear change of variabley = f(x) in which f(x) is a monotonic function so that 0 :::; j'(x) < 00. Write
down the differential equation satisfied by f (x) and draw a diagram illustrating thetransformation of the density.

Exercises

Em

603

12.29 (**)
Suppose that two variables Zl and Z2 are independent so thatp(zl' Z2) =P(Zl)P(Z2)' Show that the covariance matrix between these variables is diagonal.
This shows that independence is a sufficient condition for two variables to be uncorrelated. Now consider two variables Yl and Y2 in which -1 :0;; Yl :0;; 1 andY2 = yg. Write down the conditional distribution p(Y2IYl) and observe that this isdependent on Yb showing that the two variables are not independent. Now showthat the covariance matrix between these two variables is again diagonal. To do this,use the relation P(Yl, Y2) = P(YI )p(Y2IYl) to show that the off-diagonal terms are
zero. This counter-example shows that zero correlation is not a sufficient condition
for independence.

13
Sequential
Data

So far in this book, we have focussed primarily on sets of data points that were assumed to be independent and identically distributed (i.i.d.). This assumption allowed
us to express the likelihood function as the product over all data points of the probability distribution evaluated at each data point. For many applications, however,
the i.i.d. assumption will be a poor one. Here we consider a particularly importantclass of such data sets, namely those that describe sequential data. These often arise
through measurement of time series, for example the rainfall measurements on successive days at a particular location, or the daily values of a currency exchange rate,or the acoustic features at successive time frames used for speech recognition. An
example involving speech data is shown in Figure 13.1. Sequential data can also
arise in contexts other than time series, for example the sequence of nucleotide basepairs along a strand of DNA or the sequence of characters in an English sentence.
For convenience, we shall sometimes refer to ‘past’ and ‘future’ observations in asequence. However, the models explored in this chapter are equally applicable to all

605

606

13. SEQUENTIAL DATA

Figure 13.1 Example of a spectrogram of the spoken words “Bayes’ theorem” showing a plot of the intensity of thespectral coefﬁcients versus time index.

forms of sequential data, not just temporal sequences.
It is useful to distinguish between stationary and nonstationary sequential distributions. In the stationary case, the data evolves in time, but the distribution fromwhich it is generated remains the same. For the more complex nonstationary situation, the generative distribution itself is evolving with time. Here we shall focus on
the stationary case.
For many applications, such as ﬁnancial forecasting, we wish to be able to predict the next value in a time series given observations of the previous values. Intuitively, we expect that recent observations are likely to be more informative thanmore historical observations in predicting future values. The example in Figure 13.1
shows that successive observations of the speech spectrum are indeed highly correlated. Furthermore, it would be impractical to consider a general dependence of
future observations on all previous observations because the complexity of such amodel would grow without limit as the number of observations increases. This leads
us to consider Markov models in which we assume that future predictions are inde-13.1. Markov Models
Figure 13.2

The simplest approach tomodelling a sequence of observations is to treat them x1
as independent, corresponding to a graph without links.

x2

x3

607

x4

pendent of all but the most recent observations.
Although such models are tractable, they are also severely limited. We can obtain a more general framework, while still retaining tractability, by the introductionof latent variables, leading to state space models. As in Chapters 9 and 12, we shallsee that complex models can thereby be constructed from simpler components (inparticular, from distributions belonging to the exponential family) and can be readily characterized using the framework of probabilistic graphical models. Here wefocus on the two most important examples of state space models, namely the hidden Markov model, in which the latent variables are discrete, and linear dynamicalsystems, in which the latent variables are Gaussian. Both models are described by directed graphs having a tree structure (no loops) for which inference can be performed
efﬁciently using the sum-product algorithm.

13.1. Markov Models
The easiest way to treat sequential data would be simply to ignore the sequential
aspects and treat the observations as i.i.d., corresponding to the graph in Figure 13.2.
Such an approach, however, would fail to exploit the sequential patterns in the data,such as correlations between observations that are close in the sequence. Suppose,
for instance, that we observe a binary variable denoting whether on a particular day
it rained or not. Given a time series of recent observations of this variable, we wish
to predict whether it will rain on the next day. If we treat the data as i.i.d., then the
only information we can glean from the data is the relative frequency of rainy days.
However, we know in practice that the weather often exhibits trends that may last forseveral days. Observing whether or not it rains today is therefore of signiﬁcant help
in predicting if it will rain tomorrow.
To express such effects in a probabilistic model, we need to relax the i.i.d. assumption, and one of the simplest ways to do this is to consider a Markov model.
First of all we note that, without loss of generality, we can use the product rule toexpress the joint distribution for a sequence of observations in the formp(x1 , . . . , xN ) =N

p(xn |x1 , . . . , xn−1 ).

(13.1)

n=1

If we now assume that each of the conditional distributions on the right-hand side
is independent of all previous observations except the most recent, we obtain theﬁrst-order Markov chain, which is depicted as a graphical model in Figure 13.3. The

608

13. SEQUENTIAL DATA
Figure 13.3

A ﬁrst-order Markov chain of observations {xn } in which the distribution p(xn |xn−1 ) of a particu- x1
lar observation xn is conditionedon the value of the previous observation xn−1 .

x2

x3

x4

joint distribution for a sequence of N observations under this model is given byp(x1 , . . . , xN ) = p(x1 )Np(xn |xn−1 ).

(13.2)

n=2

Section 8.2

From the d-separation property, we see that the conditional distribution for observation xn , given all of the observations up to time n, is given byp(xn |x1 , . . . , xn−1 ) = p(xn |xn−1 )Exercise 13.1

(13.3)

which is easily veriﬁed by direct evaluation starting from (13.2) and using the product rule of probability. Thus if we use such a model to predict the next observation
in a sequence, the distribution of predictions will depend only on the value of the immediately preceding observation and will be independent of all earlier observations.
In most applications of such models, the conditional distributions p(xn |xn−1 )that deﬁne the model will be constrained to be equal, corresponding to the assumption of a stationary time series. The model is then known as a homogeneous Markov
chain. For instance, if the conditional distributions depend on adjustable parameters(whose values might be inferred from a set of training data), then all of the conditional distributions in the chain will share the same values of those parameters.
Although this is more general than the independence model, it is still very restrictive. For many sequential observations, we anticipate that the trends in the dataover several successive observations will provide important information in predicting the next value. One way to allow earlier observations to have an inﬂuence is to
move to higher-order Markov chains. If we allow the predictions to depend also on
the previous-but-one value, we obtain a second-order Markov chain, represented by
the graph in Figure 13.4. The joint distribution is now given byp(x1 , . . . , xN ) = p(x1 )p(x2 |x1 )N

p(xn |xn−1 , xn−2 ).

(13.4)

n=3

Again, using d-separation or by direct evaluation, we see that the conditional distribution of xn given xn−1 and xn−2 is independent of all observations x1 , . . . xn−3 .
Figure 13.4

A second-order Markov chain, in
which the conditional distributionof a particular observation xn
depends on the values of the two
previous observations xn−1 and x1
xn−2 .

x2

x3

x4

13.1. Markov Models
Figure 13.5 We can represent sequential data using a Markov chain of latentvariables, with each observation conditioned on the state of the corresponding
latent variable. This important graphicalstructure forms the foundation both for thehidden Markov model and for linear dynamical systems.

609

z1

z2

zn−1

zn

zn+1

x1

x2

xn−1

xn

xn+1

Each observation is now inﬂuenced by two previous observations. We can similarly
consider extensions to an M th order Markov chain in which the conditional distribution for a particular variable depends on the previous M variables. However, wehave paid a price for this increased ﬂexibility because the number of parameters in
the model is now much larger. Suppose the observations are discrete variables having K states. Then the conditional distribution p(xn |xn−1 ) in a ﬁrst-order Markov
chain will be speciﬁed by a set of K − 1 parameters for each of the K states of xn−1
giving a total of K(K − 1) parameters. Now suppose we extend the model to an
M th order Markov chain, so that the joint distribution is built up from conditionals
p(xn |xn−M , . . . , xn−1 ). If the variables are discrete, and if the conditional distributions are represented by general conditional probability tables, then the numberof parameters in such a model will have K M −1 (K − 1) parameters. Because thisgrows exponentially with M , it will often render this approach impractical for larger
values of M .
For continuous variables, we can use linear-Gaussian conditional distributions
in which each node has a Gaussian distribution whose mean is a linear functionof its parents. This is known as an autoregressive or AR model (Box et al., 1994;
Thiesson et al., 2004). An alternative approach is to use a parametric model forp(xn |xn−M , . . . , xn−1 ) such as a neural network. This technique is sometimes
called a tapped delay line because it corresponds to storing (delaying) the previous
M values of the observed variable in order to predict the next value. The numberof parameters can then be much smaller than in a completely general model (for example it may grow linearly with M ), although this is achieved at the expense of a
restricted family of conditional distributions.
Suppose we wish to build a model for sequences that is not limited by theMarkov assumption to any order and yet that can be speciﬁed using a limited numberof free parameters. We can achieve this by introducing additional latent variables topermit a rich class of models to be constructed out of simple components, as we didwith mixture distributions in Chapter 9 and with continuous latent variable models in
Chapter 12. For each observation xn , we introduce a corresponding latent variable
zn (which may be of different type or dimensionality to the observed variable). We
now assume that it is the latent variables that form a Markov chain, giving rise to thegraphical structure known as a state space model, which is shown in Figure 13.5. It
satisﬁes the key conditional independence property that zn−1 and zn+1 are independent given zn , so thatzn+1 ⊥⊥ zn−1 | zn .
(13.5)

610

13. SEQUENTIAL DATA
The joint distribution for this model is given byNNp(x1 , . . . , xN , z1 , . . . , zN ) = p(z1 )p(zn |zn−1 )p(xn |zn ).
n=2

Section 13.2

Section 13.3

(13.6)

n=1

Using the d-separation criterion, we see that there is always a path connecting any
two observed variables xn and xm via the latent variables, and that this path is neverblocked. Thus the predictive distribution p(xn+1 |x1 , . . . , xn ) for observation xn+1
given all previous observations does not exhibit any conditional independence properties, and so our predictions for xn+1 depends on all previous observations. The
observed variables, however, do not satisfy the Markov property at any order. We
shall discuss how to evaluate the predictive distribution in later sections of this chapter.
There are two important models for sequential data that are described by thisgraph. If the latent variables are discrete, then we obtain the hidden Markov model,
or HMM (Elliott et al., 1995). Note that the observed variables in an HMM may
be discrete or continuous, and a variety of different conditional distributions can beused to model them. If both the latent and the observed variables are Gaussian (witha linear-Gaussian dependence of the conditional distributions on their parents), thenwe obtain the linear dynamical system.

13.2. Hidden Markov Models
The hidden Markov model can be viewed as a speciﬁc instance of the state spacemodel of Figure 13.5 in which the latent variables are discrete. However, if we
examine a single time slice of the model, we see that it corresponds to a mixture
distribution, with component densities given by p(x|z). It can therefore also beinterpreted as an extension of a mixture model in which the choice of mixture component for each observation is not selected independently but depends on the choiceof component for the previous observation. The HMM is widely used in speech
recognition (Jelinek, 1997; Rabiner and Juang, 1993), natural language modelling(Manning and Schütze, 1999), on-line handwriting recognition (Nag et al., 1986),
and for the analysis of biological sequences such as proteins and DNA (Krogh et al.,
1994; Durbin et al., 1998; Baldi and Brunak, 2001).
As in the case of a standard mixture model, the latent variables are the discrete
multinomial variables zn describing which component of the mixture is responsible
for generating the corresponding observation xn . Again, it is convenient to use a1-of-K coding scheme, as used for mixture models in Chapter 9. We now allow the
probability distribution of zn to depend on the state of the previous latent variablezn−1 through a conditional distribution p(zn |zn−1 ). Because the latent variables are
K-dimensional binary variables, this conditional distribution corresponds to a tableof numbers that we denote by A, the elements of which are known as transition
probabilities. They are given by Ajk ≡ p(znk =1|zn−1,j = 1), and because they
are probabilities, they satisfy 0  Ajk  1 with k Ajk = 1, so that the matrix A

13.2. Hidden Markov Models
Figure 13.6

611

A22

Transition diagram showing a model whose latent variables have three possible states corresponding to the three boxes. The black lines
denote the elements of the transition matrix
Ajk .

A21
k=2
A32

A23
k=3

A12
k=1

A11

A31
A13

A33

has K(K−1) independent parameters. We can then write the conditional distribution
explicitly in the formp(zn |zn−1,A ) =

K 
K


z

z

Ajkn−1,j nk .

(13.7)

k=1 j =1

The initial latent node z1 is special in that it does not have a parent node, and soit has a marginal distribution p(z1 ) represented by a vector of probabilities π withelements πk ≡ p(z1k = 1), so thatp(z1 |π) =Section 8.4.5

K


πkz1k

(13.8)

k=1

where k πk = 1.
The transition matrix is sometimes illustrated diagrammatically by drawing thestates as nodes in a state transition diagram as shown in Figure 13.6 for the case of
K = 3. Note that this does not represent a probabilistic graphical model, because
the nodes are not separate variables but rather states of a single variable, and so wehave shown the states as boxes rather than circles.
It is sometimes useful to take a state transition diagram, of the kind shown in
Figure 13.6, and unfold it over time. This gives an alternative representation of thetransitions between latent states, known as a lattice or trellis diagram, and which isshown for the case of the hidden Markov model in Figure 13.7.
The speciﬁcation of the probabilistic model is completed by deﬁning the conditional distributions of the observed variables p(xn |zn , φ), where φ is a set of parameters governing the distribution. These are known as emission probabilities, andmight for example be given by Gaussians of the form (9.11) if the elements of x arecontinuous variables, or by conditional probability tables if x is discrete. Because
xn is observed, the distribution p(xn |zn , φ) consists, for a given value of φ, of avector of K numbers corresponding to the K possible states of the binary vector zn .

612

13. SEQUENTIAL DATA

Figure 13.7 If we unfold the state transition diagram of Figure 13.6 over time, we obtain a lattice,or trellis, representation of the latent states. Each k = 1
column of this diagram corresponds to one of thelatent variables zn .

A11

A11

A11

k=2

k=3
n−2

A33

n−1

A33

n

A33

n+1

We can represent the emission probabilities in the formp(xn |zn , φ) =Kp(xn |φk )znk .

(13.9)

k=1

We shall focuss attention on homogeneous models for which all of the conditional distributions governing the latent variables share the same parameters A, andsimilarly all of the emission distributions share the same parameters φ (the extension
to more general cases is straightforward). Note that a mixture model for an i.i.d. data
set corresponds to the special case in which the parameters Ajk are the same for all
values of j, so that the conditional distribution p(zn |zn−1 ) is independent of zn−1 .
This corresponds to deleting the horizontal links in the graphical model shown in
Figure 13.5.
The joint probability distribution over both latent and observed variables is thengiven byNNp(zn |zn−1 , A)p(xm |zm , φ)(13.10)p(X, Z|θ) = p(z1 |π)n=2Exercise 13.4

m=1where X = {x1 , . . . , xN }, Z = {z1 , . . . , zN }, and θ = {π, A, φ} denotes the setof parameters governing the model. Most of our discussion of the hidden Markov
model will be independent of the particular choice of the emission probabilities.
Indeed, the model is tractable for a wide range of emission distributions including
discrete tables, Gaussians, and mixtures of Gaussians. It is also possible to exploit
discriminative models such as neural networks. These can be used to model theemission density p(x|z) directly, or to provide a representation for p(z|x) that canbe converted into the required emission density p(x|z) using Bayes’ theorem (Bishop
et al., 2004).
We can gain a better understanding of the hidden Markov model by considering
it from a generative point of view. Recall that to generate samples from a mixture of13.2. Hidden Markov Models

1

613

1

0.5

0.5

k=1

k=3
k=2

0

0

0.5

1

0

0

0.5

1

Figure 13.8 Illustration of sampling from a hidden Markov model having a 3-state latent variable z and a
Gaussian emission model p(x|z) where x is 2-dimensional. (a) Contours of constant probability density for theemission distributions corresponding to each of the three states of the latent variable. (b) A sample of 50 points
drawn from the hidden Markov model, colour coded according to the component that generated them and withlines connecting the successive observations. Here the transition matrix was ﬁxed so that in any state there is a
5% probability of making a transition to each of the other states, and consequently a 90% probability of remaining
in the same state.

Section 8.1.2

Gaussians, we ﬁrst chose one of the components at random with probability given by
the mixing coefﬁcients πk and then generate a sample vector x from the corresponding Gaussian component. This process is repeated N times to generate a data set of
N independent samples. In the case of the hidden Markov model, this procedure is
modiﬁed as follows. We ﬁrst choose the initial latent variable z1 with probabilities
governed by the parameters πk and then sample the corresponding observation x1 .
Now we choose the state of the variable z2 according to the transition probabilitiesp(z2 |z1 ) using the already instantiated value of z1 . Thus suppose that the sample forz1 corresponds to state j. Then we choose the state k of z2 with probabilities Ajk
for k = 1, . . . , K. Once we know z2 we can draw a sample for x2 and also sample
the next latent variable z3 and so on. This is an example of ancestral sampling fora directed graphical model. If, for instance, we have a model in which the diagonal transition elements Akk are much larger than the off-diagonal elements, then atypical data sequence will have long runs of points generated from a single component, with infrequent transitions from one component to another. The generation ofsamples from a hidden Markov model is illustrated in Figure 13.8.
There are many variants of the standard HMM model, obtained for instance by
imposing constraints on the form of the transition matrix A (Rabiner, 1989). Here wemention one of particular practical importance called the left-to-right HMM, which
is obtained by setting the elements Ajk of A to zero if k < j, as illustrated in the

614

13. SEQUENTIAL DATA
Figure 13.9

Example of the state transition diagram for a 3-stateleft-to-right hidden Markov model. Note that once astate has been vacated, it cannot later be re-entered.

A11

A22
A12

k=1

A33
A23

k=2

k=3

A13

state transition diagram for a 3-state HMM in Figure 13.9. Typically for such models
the initial state probabilities for p(z1 ) are modiﬁed so that p(z11 ) = 1 and p(z1j ) = 0
for j = 1, in other words every sequence is constrained to start in state j = 1. The
transition matrix may be further constrained to ensure that large changes in the stateindex do not occur, so that Ajk = 0 if k > j + ∆. This type of model is illustratedusing a lattice diagram in Figure 13.10.
Many applications of hidden Markov models, for example speech recognition,or on-line character recognition, make use of left-to-right architectures. As an illustration of the left-to-right hidden Markov model, we consider an example involving
handwritten digits. This uses on-line data, meaning that each digit is represented
by the trajectory of the pen as a function of time in the form of a sequence of pen
coordinates, in contrast to the off-line digits data, discussed in Appendix A, which
comprises static two-dimensional pixellated images of the ink. Examples of the online digits are shown in Figure 13.11. Here we train a hidden Markov model on asubset of data comprising 45 examples of the digit ‘2’. There are K = 16 states,
each of which can generate a line segment of ﬁxed length having one of 16 possible
angles, and so the emission distribution is simply a 16 × 16 table of probabilities
associated with the allowed angle values for each state index value. Transition probabilities are all set to zero except for those that keep the state index k the same orthat increment it by 1, and the model parameters are optimized using 25 iterations of
EM. We can gain some insight into the resulting model by running it generatively, as
shown in Figure 13.11.
Figure 13.10 Lattice diagram for a 3-state leftto-right HMM in which the state index k is allowed
to increase by at most 1 at each transition.
k=1

A11

A11

A11

k=2

k=3
n−2

A33

n−1

A33

n

A33

n+1

13.2. Hidden Markov Models
Figure 13.11

615

Top row: examples of on-line handwritten
digits. Bottom row: synthetic digits sampled generatively from a left-to-right hidden Markov model that has been trainedon a data set of 45 handwritten digits.

One of the most powerful properties of hidden Markov models is their ability to
exhibit some degree of invariance to local warping (compression and stretching) of
the time axis. To understand this, consider the way in which the digit ‘2’ is written
in the on-line handwritten digits example. A typical digit comprises two distinct
sections joined at a cusp. The ﬁrst part of the digit, which starts at the top left, has asweeping arc down to the cusp or loop at the bottom left, followed by a second moreor-less straight sweep ending at the bottom right. Natural variations in writing style
will cause the relative sizes of the two sections to vary, and hence the location of thecusp or loop within the temporal sequence will vary. From a generative perspective
such variations can be accommodated by the hidden Markov model through changes
in the number of transitions to the same state versus the number of transitions to thesuccessive state. Note, however, that if a digit ‘2’ is written in the reverse order, thatis, starting at the bottom right and ending at the top left, then even though the pen tip
coordinates may be identical to an example from the training set, the probability ofthe observations under the model will be extremely small. In the speech recognition
context, warping of the time axis is associated with natural variations in the speed ofspeech, and again the hidden Markov model can accommodate such a distortion andnot penalize it too heavily.

13.2.1 Maximum likelihood for the HMM
If we have observed a data set X = {x1 , . . . , xN }, we can determine the parameters of an HMM using maximum likelihood. The likelihood function is obtained
from the joint distribution (13.10) by marginalizing over the latent variablesp(X|θ) =p(X, Z|θ).
(13.11)Z

Because the joint distribution p(X, Z|θ) does not factorize over n (in contrast to the
mixture distribution considered in Chapter 9), we cannot simply treat each of thesummations over zn independently. Nor can we perform the summations explicitly
because there are N variables to be summed over, each of which has K states, resulting in a total of K N terms. Thus the number of terms in the summation grows

616

Section 9.2

13. SEQUENTIAL DATA
exponentially with the length of the chain. In fact, the summation in (13.11) corresponds to summing over exponentially many paths through the lattice diagram in
Figure 13.7.
We have already encountered a similar difﬁculty when we considered the inference problem for the simple chain of variables in Figure 8.32. There we were able
to make use of the conditional independence properties of the graph to re-order thesummations in order to obtain an algorithm whose cost scales linearly, instead of
exponentially, with the length of the chain. We shall apply a similar technique to thehidden Markov model.
A further difﬁculty with the expression (13.11) for the likelihood function is that,
because it corresponds to a generalization of a mixture distribution, it represents asummation over the emission models for different settings of the latent variables.
Direct maximization of the likelihood function will therefore lead to complex expressions with no closed-form solutions, as was the case for simple mixture models(recall that a mixture model for i.i.d. data is a special case of the HMM).
We therefore turn to the expectation maximization algorithm to ﬁnd an efﬁcient
framework for maximizing the likelihood function in hidden Markov models. The
EM algorithm starts with some initial selection for the model parameters, which wedenote by θ old . In the E step, we take these parameter values and ﬁnd the posterior
distribution of the latent variables p(Z|X, θ old ). We then use this posterior distribution to evaluate the expectation of the logarithm of the complete-data likelihood
function, as a function of the parameters θ, to give the function Q(θ, θ old ) deﬁnedbyp(Z|X, θold ) ln p(X, Z|θ).
(13.12)Q(θ, θ old ) =Z

At this point, it is convenient to introduce some notation. We shall use γ(zn ) todenote the marginal posterior distribution of a latent variable zn , and ξ(zn−1 , zn ) todenote the joint posterior distribution of two successive latent variables, so thatγ(zn ) = p(zn |X, θ old )(13.13)

ξ(zn−1 , zn ) = p(zn−1 , zn |X, θ

old

).

(13.14)For each value of n, we can store γ(zn ) using a set of K nonnegative numbersthat sum to unity, and similarly we can store ξ(zn−1 , zn ) using a K × K matrix ofnonnegative numbers that again sum to unity. We shall also use γ(znk ) to denote theconditional probability of znk = 1, with a similar use of notation for ξ(zn−1,j , znk )and for other probabilistic variables introduced later. Because the expectation of abinary random variable is just the probability that it takes the value 1, we haveγ(z)znk(13.15)γ(znk ) = E[znk ] =zξ(zn−1,j , znk ) = E[zn−1,j znk ] =γ(z)zn−1,j znk .

(13.16)z

If we substitute the joint distribution p(X, Z|θ) given by (13.10) into (13.12),

617

13.2. Hidden Markov Models
and make use of the deﬁnitions of γ and ξ , we obtain
Q(θ, θ

old

) =

K


γ(z1k ) ln πk +ξ(zn−1,j , znk ) ln Ajk

n=2 j =1 k=1

k=1

+

N 
K 
K


N 
K


γ(znk ) ln p(xn |φk ).

(13.17)n=1 k=1

Exercise 13.5

The goal of the E step will be to evaluate the quantities γ(zn ) and ξ(zn−1 , zn ) efﬁciently, and we shall discuss this in detail shortly.
In the M step, we maximize Q(θ, θ old ) with respect to the parameters θ ={π, A, φ} in which we treat γ(zn ) and ξ(zn−1 , zn ) as constant. Maximization withrespect to π and A is easily achieved using appropriate Lagrange multipliers withthe resultsπk=

γ(z1k )
K


(13.18)γ(z1j )

j =1
N


Ajk

=

ξ(zn−1,j , znk )

n=2
K 
N


.

(13.19)ξ(zn−1,j , znl )l=1 n=2

Exercise 13.6

The EM algorithm must be initialized by choosing starting values for π and A, whichshould of course respect the summation constraints associated with their probabilistic interpretation. Note that any elements of π or A that are set to zero initially willremain zero in subsequent EM updates. A typical initialization procedure would
involve selecting random starting values for these parameters subject to the summation and non-negativity constraints. Note that no particular modiﬁcation to the EM
results are required for the case of left-to-right models beyond choosing initial values
for the elements Ajk in which the appropriate elements are set to zero, because thesewill remain zero throughout.
To maximize Q(θ, θ old ) with respect to φk , we notice that only the ﬁnal termin (13.17) depends on φk , and furthermore this term has exactly the same form as
the data-dependent term in the corresponding function for a standard mixture distribution for i.i.d. data, as can be seen by comparison with (9.40) for the case of a
Gaussian mixture. Here the quantities γ(znk ) are playing the role of the responsibilities. If the parameters φk are independent for the different components, then thisterm decouples into a sum of terms one for each value of k, each of which can be
maximized independently. We are then simply maximizing the weighted log likelihood function for the emission density p(x|φk ) with weights γ(znk ). Here we shallsuppose that this maximization can be done efﬁciently. For instance, in the case of

618

13. SEQUENTIAL DATA
Gaussian emission densities we have p(x|φk ) = N (x|µk , Σk ), and maximizationof the function Q(θ, θ old ) then gives
N


µk

=

γ(znk )xnn=1
N

(13.20)γ(znk )

n=1
N


Σk

=

γ(znk )(xn − µk )(xn − µk )T

n=1
N


.

(13.21)γ(znk )

n=1

For the case of discrete multinomial observed variables, the conditional distributionof the observations takes the formp(x|z) =D Kµxiki zk(13.22)i=1 k=1

Exercise 13.8

and the corresponding M-step equations are given byNγ(znk )xni

n=1
µik = N
.

(13.23)γ(znk )

n=1

An analogous result holds for Bernoulli observed variables.
The EM algorithm requires initial values for the parameters of the emission distribution. One way to set these is ﬁrst to treat the data initially as i.i.d. and ﬁt theemission density by maximum likelihood, and then use the resulting values to initialize the parameters for EM.

13.2.2 The forward-backward algorithm

Section 8.4

Next we seek an efﬁcient procedure for evaluating the quantities γ(znk ) andξ(zn−1,j , znk ), corresponding to the E step of the EM algorithm. The graph for thehidden Markov model, shown in Figure 13.5, is a tree, and so we know that theposterior distribution of the latent variables can be obtained efﬁciently using a twostage message passing algorithm. In the particular context of the hidden Markov
model, this is known as the forward-backward algorithm (Rabiner, 1989), or the
Baum-Welch algorithm (Baum, 1972). There are in fact several variants of the basic
algorithm, all of which lead to the exact marginals, according to the precise form of

13.2. Hidden Markov Models619

the messages that are propagated along the chain (Jordan, 2007). We shall focus on
the most widely used of these, known as the alpha-beta algorithm.
As well as being of great practical importance in its own right, the forwardbackward algorithm provides us with a nice illustration of many of the concepts
introduced in earlier chapters. We shall therefore begin in this section with a ‘conventional’ derivation of the forward-backward equations, making use of the sum
and product rules of probability, and exploiting conditional independence properties
which we shall obtain from the corresponding graphical model using d-separation.
Then in Section 13.2.3, we shall see how the forward-backward algorithm can beobtained very simply as a speciﬁc example of the sum-product algorithm introducedin Section 8.4.4.
It is worth emphasizing that evaluation of the posterior distributions of the latentvariables is independent of the form of the emission density p(x|z) or indeed ofwhether the observed variables are continuous or discrete. All we require is thevalues of the quantities p(xn |zn ) for each value of zn for every n. Also, in thissection and the next we shall omit the explicit dependence on the model parameters
θ old because these ﬁxed throughout.
We therefore begin by writing down the following conditional independence
properties (Jordan, 2007)
p(X|zn ) = p(x1 , . . . , xn |zn )(13.24)p(xn+1 , . . . , xN |zn )p(x1 , . . . , xn−1 |xn , zn ) = p(x1 , . . . , xn−1 |zn )(13.25)(13.26)p(x1 , . . . , xn−1 |zn−1 , zn ) = p(x1 , . . . , xn−1 |zn−1 )(13.27)p(xn+1 , . . . , xN |zn , zn+1 ) = p(xn+1 , . . . , xN |zn+1 )(13.28)p(xn+2 , . . . , xN |zn+1 , xn+1 ) = p(xn+2 , . . . , xN |zn+1 )p(X|zn−1 , zn ) = p(x1 , . . . , xn−1 |zn−1 )p(xn |zn )p(xn+1 , . . . , xN |zn ) (13.29)(13.30)p(xN +1 |X, zN +1 ) = p(xN +1 |zN +1 )p(zN +1 |zN , X) = p(zN +1 |zN )(13.31)

Exercise 13.10

where X = {x1 , . . . , xN }. These relations are most easily proved using d-separation.
For instance in the ﬁrst of these results, we note that every path from any one of thenodes x1 , . . . , xn−1 to the node xn passes through the node zn , which is observed.
Because all such paths are head-to-tail, it follows that the conditional independence
property must hold. The reader should take a few moments to verify each of theseproperties in turn, as an exercise in the application of d-separation. These relations
can also be proved directly, though with signiﬁcantly greater effort, from the joint
distribution for the hidden Markov model using the sum and product rules of probability.
Let us begin by evaluating γ(znk ). Recall that for a discrete multinomial random variable the expected value of one of its components is just the probability ofthat component having the value 1. Thus we are interested in ﬁnding the posterior
distribution p(zn |x1 , . . . , xN ) of zn given the observed data set x1 , . . . , xN . This

620

13. SEQUENTIAL DATA
represents a vector of length K whose entries correspond to the expected values ofznk . Using Bayes’ theorem, we haveγ(zn ) = p(zn |X) =p(X|zn )p(zn ).
p(X)(13.32)

Note that the denominator p(X) is implicitly conditioned on the parameters θ oldof the HMM and hence represents the likelihood function. Using the conditional
independence property (13.24), together with the product rule of probability, we
obtainp(x1 , . . . , xn , zn )p(xn+1 , . . . , xN |zn )α(zn )β(zn )γ(zn ) ==(13.33)p(X)p(X)
where we have deﬁnedα(zn ) ≡ p(x1 , . . . , xn , zn )β(zn ) ≡ p(xn+1 , . . . , xN |zn ).

(13.34)(13.35)The quantity α(zn ) represents the joint probability of observing all of the given
data up to time n and the value of zn , whereas β(zn ) represents the conditional
probability of all future data from time n + 1 up to N given the value of zn . Again,α(zn ) and β(zn ) each represent set of K numbers, one for each of the possible
settings of the 1-of-K coded binary vector zn . We shall use the notation α(znk ) todenote the value of α(zn ) when znk = 1, with an analogous interpretation of β(znk ).
We now derive recursion relations that allow α(zn ) and β(zn ) to be evaluatedefﬁciently. Again, we shall make use of conditional independence properties, inparticular (13.25) and (13.26), together with the sum and product rules, allowing us
to express α(zn ) in terms of α(zn−1 ) as follows
p(x1 , . . . , xn , zn )p(x1 , . . . , xn |zn )p(zn )p(xn |zn )p(x1 , . . . , xn−1 |zn )p(zn )p(xn |zn )p(x1 , . . . , xn−1 , zn )p(x1 , . . . , xn−1 , zn−1 , zn )= p(xn |zn )α(zn ) =
=
=
=

zn−1

= p(xn |zn )p(x1 , . . . , xn−1 , zn |zn−1 )p(zn−1 )zn−1= p(xn |zn )p(x1 , . . . , xn−1 |zn−1 )p(zn |zn−1 )p(zn−1 )zn−1= p(xn |zn )p(x1 , . . . , xn−1 , zn−1 )p(zn |zn−1 )zn−1Making use of the deﬁnition (13.34) for α(zn ), we then obtainα(zn ) = p(xn |zn )α(zn−1 )p(zn |zn−1 ).
zn−1(13.36)

13.2. Hidden Markov Models
Figure 13.12

621

Illustration of the forward recursion (13.36) forα(zn,1 )α(zn−1,1 )evaluation of the α variables. In this fragment
A11of the lattice, we see that the quantity α(zn1 ) k = 1
is obtained by taking the elements α(zn−1,j ) ofα(zn−1 ) at step n−1 and summing them up with
A21
p(xn |zn,1 )weights given by Aj1 , corresponding to the val)
α(z
n−1,2
ues of p(zn |zn−1 ), and then multiplying by thedata contribution p(xn |zn1 ).

k=2

A31

α(zn−1,3 )
k=3
n−1

n

It is worth taking a moment to study this recursion relation in some detail. Notethat there are K terms in the summation, and the right-hand side has to be evaluatedfor each of the K values of zn so each step of the α recursion has computational
cost that scaled like O(K 2 ). The forward recursion equation for α(zn ) is illustratedusing a lattice diagram in Figure 13.12.
In order to start this recursion, we need an initial condition that is given byα(z1 ) = p(x1 , z1 ) = p(z1 )p(x1 |z1 ) =Kz{πk p(x1 |φk )} 1k(13.37)k=1

which tells us that α(z1k ), for k = 1, . . . , K, takes the value πk p(x1 |φk ). Startingat the ﬁrst node of the chain, we can then work along the chain and evaluate α(zn )for every latent node. Because each step of the recursion involves multiplying by a
K × K matrix, the overall cost of evaluating these quantities for the whole chain isof O(K 2 N ).
We can similarly ﬁnd a recursion relation for the quantities β(zn ) by making
use of the conditional independence properties (13.27) and (13.28) giving
β(zn ) = p(xn+1 , . . . , xN |zn )=p(xn+1 , . . . , xN , zn+1 |zn )
zn+1

=



p(xn+1 , . . . , xN |zn , zn+1 )p(zn+1 |zn )

zn+1

=



p(xn+1 , . . . , xN |zn+1 )p(zn+1 |zn )

zn+1

=



zn+1

p(xn+2 , . . . , xN |zn+1 )p(xn+1 |zn+1 )p(zn+1 |zn ).

622

13. SEQUENTIAL DATA
Figure 13.13

Illustration of the backward recursion
β(zn+1,1 )β(zn,1 )(13.38) for evaluation of the β variables. In
A11this fragment of the lattice, we see that the k = 1
quantity β(zn1 ) is obtained by taking thecomponents β(zn+1,k ) of β(zn+1 ) at stepp(xn |zn+1,1 )A12n + 1 and summing them up with weights
β(zn+1,2 )given by the products of A1k , corresponding to the values of p(zn+1 |zn ) and the corresponding values of the emission density k = 2
A13p(xn |zn+1,k ).

p(xn |zn+1,2 )β(zn+1,3 )
k=3
n

n+1

Making use of the deﬁnition (13.35) for β(zn ), we then obtainβ(zn+1 )p(xn+1 |zn+1 )p(zn+1 |zn ).
β(zn ) =p(xn |zn+1,3 )(13.38)zn+1

Note that in this case we have a backward message passing algorithm that evaluates
β(zn ) in terms of β(zn+1 ). At each step, we absorb the effect of observation xn+1
through the emission probability p(xn+1 |zn+1 ), multiply by the transition matrixp(zn+1 |zn ), and then marginalize out zn+1 . This is illustrated in Figure 13.13.
Again we need a starting condition for the recursion, namely a value for β(zN ).
This can be obtained by setting n = N in (13.33) and replacing α(zN ) with its
deﬁnition (13.34) to givep(zN |X) =p(X, zN )β(zN )p(X)(13.39)which we see will be correct provided we take β(zN ) = 1 for all settings of zN .
In the M step equations, the quantity p(X) will cancel out, as can be seen, forinstance, in the M-step equation for µk given by (13.20), which takes the form
n


n


γ(znk )xnn=1
µk = nγ(znk )α(znk )β(znk )xn

n=1
= n


n=1

.

(13.40)α(znk )β(znk )n=1

However, the quantity p(X) represents the likelihood function whose value we typically wish to monitor during the EM optimization, and so it is useful to be able toevaluate it. If we sum both sides of (13.33) over zn , and use the fact that the left-handside is a normalized distribution, we obtainα(zn )β(zn ).
(13.41)
p(X) =zn

13.2. Hidden Markov Models

623

Thus we can evaluate the likelihood function by computing this sum, for any convenient choice of n. For instance, if we only want to evaluate the likelihood function,then we can do this by running the α recursion from the start to the end of the chain,and then use this result for n = N , making use of the fact that β(zN ) is a vector of
1s. In this case no β recursion is required, and we simply havep(X) =α(zN ).
(13.42)zN

Let us take a moment to interpret this result for p(X). Recall that to compute thelikelihood we should take the joint distribution p(X, Z) and sum over all possible
values of Z. Each such value represents a particular choice of hidden state for every
time step, in other words every term in the summation is a path through the lattice
diagram, and recall that there are exponentially many such paths. By expressing
the likelihood function in the form (13.42), we have reduced the computational costfrom being exponential in the length of the chain to being linear by swapping theorder of the summation and multiplications, so that at each time step n we sum
the contributions from all paths passing through each of the states znk to give theintermediate quantities α(zn ).
Next we consider the evaluation of the quantities ξ(zn−1 , zn ), which correspondto the values of the conditional probabilities p(zn−1 , zn |X) for each of the K × K
settings for (zn−1 , zn ). Using the deﬁnition of ξ(zn−1 , zn ), and applying Bayes’
theorem, we haveξ(zn−1 , zn ) = p(zn−1 , zn |X)
p(X|zn−1 , zn )p(zn−1 , zn )=p(X)p(x1 , . . . , xn−1 |zn−1 )p(xn |zn )p(xn+1 , . . . , xN |zn )p(zn |zn−1 )p(zn−1 )=p(X)α(zn−1 )p(xn |zn )p(zn |zn−1 )β(zn )=(13.43)p(X)where we have made use of the conditional independence property (13.29) together
with the deﬁnitions of α(zn ) and β(zn ) given by (13.34) and (13.35). Thus we cancalculate the ξ(zn−1 , zn ) directly by using the results of the α and β recursions.
Let us summarize the steps required to train a hidden Markov model using
the EM algorithm. We ﬁrst make an initial selection of the parameters θ old whereθ ≡ (π, A, φ). The A and π parameters are often initialized either uniformly or
randomly from a uniform distribution (respecting their non-negativity and summation constraints). Initialization of the parameters φ will depend on the form of the
distribution. For instance in the case of Gaussians, the parameters µk might be initialized by applying the K-means algorithm to the data, and Σk might be initialized
to the covariance matrix of the corresponding K means cluster. Then we run boththe forward α recursion and the backward β recursion and use the results to evaluateγ(zn ) and ξ(zn−1 , zn ). At this stage, we can also evaluate the likelihood function.

624

13. SEQUENTIAL DATA

Exercise 13.12

This completes the E step, and we use the results to ﬁnd a revised set of parameters
θ new using the M-step equations from Section 13.2.1. We then continue to alternate
between E and M steps until some convergence criterion is satisﬁed, for instancewhen the change in the likelihood function is below some threshold.
Note that in these recursion relations the observations enter through conditional
distributions of the form p(xn |zn ). The recursions are therefore independent of
the type or dimensionality of the observed variables or the form of this conditional
distribution, so long as its value can be computed for each of the K possible statesof zn . Since the observed variables {xn } are ﬁxed, the quantities p(xn |zn ) can bepre-computed as functions of zn at the start of the EM algorithm, and remain ﬁxed
throughout.
We have seen in earlier chapters that the maximum likelihood approach is mosteffective when the number of data points is large in relation to the number of parameters. Here we note that a hidden Markov model can be trained effectively, using maximum likelihood, provided the training sequence is sufﬁciently long. Alternatively,
we can make use of multiple shorter sequences, which requires a straightforward
modiﬁcation of the hidden Markov model EM algorithm. In the case of left-to-rightmodels, this is particularly important because, in a given observation sequence, agiven state transition corresponding to a nondiagonal element of A will seen at mostonce.
Another quantity of interest is the predictive distribution, in which the observed
data is X = {x1 , . . . , xN } and we wish to predict xN +1 , which would be important
for real-time applications such as ﬁnancial forecasting. Again we make use of thesum and product rules together with the conditional independence properties (13.29)
and (13.31) givingp(xN +1 , zN +1 |X)
p(xN +1 |X) =
zN +1

=



p(xN +1 |zN +1 )p(zN +1 |X)

zN +1

=



p(xN +1 |zN +1 )

zN +1

=




p(xN +1 |zN +1 )p(zN +1 |zN )p(zN |X)

zN

p(xN +1 |zN +1 )

zN +1

=

p(zN +1 , zN |X)

zN

zN +1

=





p(zN +1 |zN )zN

p(zN , X)
p(X)


1 
p(xN +1 |zN +1 )p(zN +1 |zN )α(zN ) (13.44)
p(X)
zN +1

zN

which can be evaluated by ﬁrst running a forward α recursion and then computing
the ﬁnal summations over zN and zN +1 . The result of the ﬁrst summation over zN
can be stored and used once the value of xN +1 is observed in order to run the αrecursion forward to the next step in order to predict the subsequent value xN +2 .

13.2. Hidden Markov Models
Figure 13.14 A fragment of the factor graph representation for the hidden
Markov model.

χ

zn−1

z1

g1

x1

Section 10.1

ψn

zn

gn−1

xn−1

625

gn

xn

Note that in (13.44), the inﬂuence of all data from x1 to xN is summarized in the Kvalues of α(zN ). Thus the predictive distribution can be carried forward indeﬁnitely
using a ﬁxed amount of storage, as may be required for real-time applications.
Here we have discussed the estimation of the parameters of an HMM using maximum likelihood. This framework is easily extended to regularized maximum likelihood by introducing priors over the model parameters π, A and φ whose values arethen estimated by maximizing their posterior probability. This can again be done using the EM algorithm in which the E step is the same as discussed above, and the M
step involves adding the log of the prior distribution p(θ) to the function Q(θ, θ old )before maximization and represents a straightforward application of the techniquesdeveloped at various points in this book. Furthermore, we can use variational methods to give a fully Bayesian treatment of the HMM in which we marginalize over theparameter distributions (MacKay, 1997). As with maximum likelihood, this leads toa two-pass forward-backward recursion to compute posterior probabilities.

13.2.3 The sum-product algorithm for the HMM

Section 8.4.4

The directed graph that represents the hidden Markov model, shown in Figure 13.5, is a tree and so we can solve the problem of ﬁnding local marginals for thehidden variables using the sum-product algorithm. Not surprisingly, this turns out tobe equivalent to the forward-backward algorithm considered in the previous section,and so the sum-product algorithm therefore provides us with a simple way to derive
the alpha-beta recursion formulae.
We begin by transforming the directed graph of Figure 13.5 into a factor graph,of which a representative fragment is shown in Figure 13.14. This form of the factor graph shows all variables, both latent and observed, explicitly. However, forthe purpose of solving the inference problem, we shall always be conditioning on
the variables x1 , . . . , xN , and so we can simplify the factor graph by absorbing theemission probabilities into the transition probability factors. This leads to the simpliﬁed factor graph representation in Figure 13.15, in which the factors are givenbyh(z1 ) = p(z1 )p(x1 |z1 )fn (zn−1 , zn ) = p(zn |zn−1 )p(xn |zn ).

(13.45)(13.46)

626

13. SEQUENTIAL DATA

Figure 13.15 A simpliﬁed form of factor graph to describe the hidden Markov
model.

fn

h
z1

zn−1

zn

To derive the alpha-beta algorithm, we denote the ﬁnal hidden variable zN as
the root node, and ﬁrst pass messages from the leaf node h to the root. From thegeneral results (8.66) and (8.69) for message propagation, we see that the messages
which are propagated in the hidden Markov model take the form
µzn−1 →fn (zn−1 ) = µfn−1 →zn−1 (zn−1 )µfn →zn (zn ) =fn (zn−1 , zn )µzn−1 →fn (zn−1 )(13.47)(13.48)zn−1These equations represent the propagation of messages forward along the chain andare equivalent to the alpha recursions derived in the previous section, as we shallnow show. Note that because the variable nodes zn have only two neighbours, they
perform no computation.
We can eliminate µzn−1 →fn (zn−1 ) from (13.48) using (13.47) to give a recursion for the f → z messages of the formµfn →zn (zn ) =fn (zn−1 , zn )µfn−1 →zn−1 (zn−1 ).
(13.49)zn−1

If we now recall the deﬁnition (13.46), and if we deﬁneα(zn ) = µfn →zn (zn )(13.50)then we obtain the alpha recursion given by (13.36). We also need to verify thatthe quantities α(zn ) are themselves equivalent to those deﬁned previously. Thisis easily done by using the initial condition (8.71) and noting that α(z1 ) is givenby h(z1 ) = p(z1 )p(x1 |z1 ) which is identical to (13.37). Because the initial α isthe same, and because they are iteratively computed using the same equation, allsubsequent α quantities must be the same.
Next we consider the messages that are propagated from the root node back to
the leaf node. These take the form
fn+1 (zn , zn+1 )µfn+2 →fn+1 (zn+1 )(13.51)µfn+1 →fn (zn ) =zn+1

where, as before, we have eliminated the messages of the type z → f since thevariable nodes perform no computation. Using the deﬁnition (13.46) to substitutefor fn+1 (zn , zn+1 ), and deﬁning
β(zn ) = µfn+1 →zn (zn )(13.52)

13.2. Hidden Markov Models

627

we obtain the beta recursion given by (13.38). Again, we can verify that the beta
variables themselves are equivalent by noting that (8.70) implies that the initial message send by the root variable node is µzN →fN (zN ) = 1, which is identical to theinitialization of β(zN ) given in Section 13.2.2.
The sum-product algorithm also speciﬁes how to evaluate the marginals once all
the messages have been evaluated. In particular, the result (8.63) shows that the local
marginal at the node zn is given by the product of the incoming messages. Because
we have conditioned on the variables X = {x1 , . . . , xN }, we are computing thejoint distributionp(zn , X) = µfn →zn (zn )µfn+1 →zn (zn ) = α(zn )β(zn ).

(13.53)

Dividing both sides by p(X), we then obtain
γ(zn ) =Exercise 13.11

p(zn , X)α(zn )β(zn )
=
p(X)
p(X)

(13.54)in agreement with (13.33). The result (13.43) can similarly be derived from (8.72).

13.2.4 Scaling factors
There is an important issue that must be addressed before we can make use of theforward backward algorithm in practice. From the recursion relation (13.36), we notethat at each step the new value α(zn ) is obtained from the previous value α(zn−1 )by multiplying by quantities p(zn |zn−1 ) and p(xn |zn ). Because these probabilities
are often signiﬁcantly less than unity, as we work our way forward along the chain,
the values of α(zn ) can go to zero exponentially quickly. For moderate lengths ofchain (say 100 or so), the calculation of the α(zn ) will soon exceed the dynamic
range of the computer, even if double precision ﬂoating point is used.
In the case of i.i.d. data, we implicitly circumvented this problem with the evaluation of likelihood functions by taking logarithms. Unfortunately, this will not help
here because we are forming sums of products of small numbers (we are in fact implicitly summing over all possible paths through the lattice diagram of Figure 13.7).
We therefore work with re-scaled versions of α(zn ) and β(zn ) whose values remainof order unity. As we shall see, the corresponding scaling factors cancel out when
we use these re-scaled quantities in the EM algorithm.
In (13.34), we deﬁned α(zn ) = p(x1 , . . . , xn , zn ) representing the joint distribution of all the observations up to xn and the latent variable zn . Now we deﬁne anormalized version of α given byα (zn ) = p(zn |x1 , . . . , xn ) =α(zn )p(x1 , . . . , xn )(13.55)which we expect to be well behaved numerically because it is a probability distribution over K variables for any value of n. In order to relate the scaled and original alpha variables, we introduce scaling factors deﬁned by conditional distributions over
the observed variables
cn = p(xn |x1 , . . . , xn−1 ).
(13.56)

628

13. SEQUENTIAL DATA
From the product rule, we then havep(x1 , . . . , xn ) =

n


cm

(13.57)m=1

and soα(zn ) = p(zn |x1 , . . . , xn )p(x1 , . . . , xn ) =

 n



cm

α
 (zn ).

(13.58)m=1

 given by
We can then turn the recursion equation (13.36) for α into one for α(zn ) = p(xn |zn )α(zn−1 )p(zn |zn−1 ).
(13.59)cn αzn−1

(zn ),
Note that at each stage of the forward message passing phase, used to evaluate α
we have to evaluate and store cn , which is easily done because it is the coefﬁcientthat normalizes the right-hand side of (13.59) to give α (zn ).
 n ) using
We can similarly deﬁne re-scaled variables β(z
 N


 n)
cm β(z(13.60)β(zn ) =m=n+1

which will again remain within machine precision because, from (13.35), the quan n ) are simply the ratio of two conditional probabilities
tities β(z n) =β(zp(xn+1 , . . . , xN |zn ).
p(xn+1 , . . . , xN |x1 , . . . , xn )(13.61)

The recursion result (13.38) for β then gives the following recursion for the re-scaled
variables n) =
 n+1 )p(xn+1 |zn+1 )p(zn+1 |zn ).
β(z(13.62)cn+1 β(zzn+1

In applying this recursion relation, we make use of the scaling factors cn that werepreviously computed in the α phase.
From (13.57), we see that the likelihood function can be found usingp(X) =

Ncn .

(13.63)n=1

Exercise 13.15

Similarly, using (13.33) and (13.43), together with (13.63), we see that the required
marginals are given by n) (zn )β(zγ(zn ) = α

 n ).
ξ(zn−1 , zn ) = cn α (zn−1 )p(xn |zn )p(zn |z−1 )β(z(13.64)(13.65)

13.2. Hidden Markov Models

Section 13.3

629

Finally, we note that there is an alternative formulation of the forward-backward
algorithm (Jordan, 2007) in which the backward pass is deﬁned by a recursion based n ) instead of using β(z n ). This α–γ recursion
the quantities γ(zn ) = α (zn )β(z (zn )
requires that the forward pass be completed ﬁrst so that all the quantities α
are available for the backward pass, whereas the forward and backward passes ofthe α–β algorithm can be done independently. Although these two algorithms havecomparable computational cost, the α–β version is the most commonly encountered
one in the case of hidden Markov models, whereas for linear dynamical systems arecursion analogous to the α–γ form is more usual.

13.2.5 The Viterbi algorithm
In many applications of hidden Markov models, the latent variables have somemeaningful interpretation, and so it is often of interest to ﬁnd the most probablesequence of hidden states for a given observation sequence. For instance in speech
recognition, we might wish to ﬁnd the most probable phoneme sequence for a given
series of acoustic observations. Because the graph for the hidden Markov model isa directed tree, this problem can be solved exactly using the max-sum algorithm.
We recall from our discussion in Section 8.4.5 that the problem of ﬁnding the mostprobable sequence of latent states is not the same as that of ﬁnding the set of statesthat are individually the most probable. The latter problem can be solved by ﬁrst
running the forward-backward (sum-product) algorithm to ﬁnd the latent variablemarginals γ(zn ) and then maximizing each of these individually (Duda et al., 2001).
However, the set of such states will not, in general, correspond to the most probablesequence of states. In fact, this set of states might even represent a sequence having
zero probability, if it so happens that two successive states, which in isolation are
individually the most probable, are such that the transition matrix element connecting
them is zero.
In practice, we are usually interested in ﬁnding the most probable sequence ofstates, and this can be solved efﬁciently using the max-sum algorithm, which in thecontext of hidden Markov models is known as the Viterbi algorithm (Viterbi, 1967).
Note that the max-sum algorithm works with log probabilities and so there is noneed to use re-scaled variables as was done with the forward-backward algorithm.
Figure 13.16 shows a fragment of the hidden Markov model expanded as lattice
diagram. As we have already noted, the number of possible paths through the lattice
grows exponentially with the length of the chain. The Viterbi algorithm searches thisspace of paths efﬁciently to ﬁnd the most probable path with a computational costthat grows only linearly with the length of the chain.
As with the sum-product algorithm, we ﬁrst represent the hidden Markov model
as a factor graph, as shown in Figure 13.15. Again, we treat the variable node zN
as the root, and pass messages to the root starting with the leaf nodes. Using theresults (8.93) and (8.94), we see that the messages passed in the max-sum algorithm
are given by(13.66)µzn →fn+1 (zn ) = µfn →zn (zn )µfn+1 →zn+1 (zn+1 ) = max ln fn+1 (zn , zn+1 ) + µzn →fn+1 (zn ) . (13.67)
zn

630

13. SEQUENTIAL DATA

Figure 13.16 A fragment of the HMM lattice
showing two possible paths. The Viterbi algorithm
efﬁciently determines the most probable path from k = 1
amongst the exponentially many possibilities. For
any given path, the corresponding probability isgiven by the product of the elements of the transition matrix Ajk , corresponding to the probabilities p(zn+1 |zn ) for each segment of the path,along with the emission densities p(xn |k) asso- k = 2
ciated with each node on the path.

k=3
n−2

n−1

n

n+1

If we eliminate µzn →fn+1 (zn ) between these two equations, and make use of (13.46),
we obtain a recursion for the f → z messages of the form
ω(zn+1 ) = ln p(xn+1 |zn+1 ) + max {ln p(x+1 |zn ) + ω(zn )}zn(13.68)where we have introduced the notation ω(zn ) ≡ µfn →zn (zn ).
From (8.95) and (8.96), these messages are initialized using
ω(z1 ) = ln p(z1 ) + ln p(x1 |z1 ).

Exercise 13.16(13.69)where we have used (13.45). Note that to keep the notation uncluttered, we omit
the dependence on the model parameters θ that are held ﬁxed when ﬁnding the mostprobable sequence.
The Viterbi algorithm can also be derived directly from the deﬁnition (13.6) of
the joint distribution by taking the logarithm and then exchanging maximizations
and summations. It is easily seen that the quantities ω(zn ) have the probabilistic
interpretation(13.70)ω(zn ) = max p(x1 , . . . , xn , z1 , . . . , zn ).
z1 ,...,zn−1

Once we have completed the ﬁnal maximization over zN , we will obtain thevalue of the joint distribution p(X, Z) corresponding to the most probable path. We
also wish to ﬁnd the sequence of latent variable values that corresponds to this path.
To do this, we simply make use of the back-tracking procedure discussed in Section 8.4.5. Speciﬁcally, we note that the maximization over zn must be performed
for each of the K possible values of zn+1 . Suppose we keep a record of the valuesof zn that correspond to the maxima for each value of the K values of zn+1 . Let us
denote this function by ψ(kn ) where k ∈ {1, . . . , K}. Once we have passed messages to the end of the chain and found the most probable state of zN , we can thenuse this function to backtrack along the chain by applying it recursively
knmax = ψ(knmax+1 ).

(13.71)

13.2. Hidden Markov Models

631

Intuitively, we can understand the Viterbi algorithm as follows. Naively, we
could consider explicitly all of the exponentially many paths through the lattice,evaluate the probability for each, and then select the path having the highest probability. However, we notice that we can make a dramatic saving in computational cost
as follows. Suppose that for each path we evaluate its probability by summing up
products of transition and emission probabilities as we work our way forward alongeach path through the lattice. Consider a particular time step n and a particular state
k at that time step. There will be many possible paths converging on the corresponding node in the lattice diagram. However, we need only retain that particular paththat so far has the highest probability. Because there are K states at time step n, we
need to keep track of K such paths. At time step n + 1, there will be K 2 possible
paths to consider, comprising K possible paths leading out of each of the K current
states, but again we need only retain K of these corresponding to the best path foreach state at time n+1. When we reach the ﬁnal time step N we will discover whichstate corresponds to the overall most probable path. Because there is a unique pathcoming into that state we can trace the path back to step N − 1 to see what state it
occupied at that time, and so on back through the lattice to the state n = 1.

13.2.6 Extensions of the hidden Markov model
The basic hidden Markov model, along with the standard training algorithm
based on maximum likelihood, has been extended in numerous ways to meet therequirements of particular applications. Here we discuss a few of the more important
examples.
We see from the digits example in Figure 13.11 that hidden Markov models canbe quite poor generative models for the data, because many of the synthetic digits
look quite unrepresentative of the training data. If the goal is sequence classiﬁcation, there can be signiﬁcant beneﬁt in determining the parameters of hidden Markov
models using discriminative rather than maximum likelihood techniques. Suppose
we have a training set of R observation sequences Xr , where r = 1, . . . , R, each ofwhich is labelled according to its class m, where m = 1, . . . , M . For each class, wehave a separate hidden Markov model with its own parameters θ m , and we treat the
problem of determining the parameter values as a standard classiﬁcation problem inwhich we optimize the cross-entropyRln p(mr |Xr ).

(13.72)

r =1

Using Bayes’ theorem this can be expressed in terms of the sequence probabilities
associated with the hidden Markov modelsRp(Xr |θ r )p(mr )ln M(13.73)
p(X
|θ
)p(l
)
r
l
r
r =1
l=1
where p(m) is the prior probability of class m. Optimization of this cost function
is more complex than for maximum likelihood (Kapadia, 1998), and in particular

632

13. SEQUENTIAL DATA
Figure 13.17

Section of an autoregressive hidden
Markov model, in which the distributionof the observation xn depends on asubset of the previous observations aswell as on the hidden state zn . In thisexample, the distribution of xn depends
on the two previous observations xn−1
and xn−2 .

zn−1

zn

zn+1

xn−1

xn

xn+1

requires that every training sequence be evaluated under each of the models in order to compute the denominator in (13.73). Hidden Markov models, coupled withdiscriminative training methods, are widely used in speech recognition (Kapadia,
1998).
A signiﬁcant weakness of the hidden Markov model is the way in which it represents the distribution of times for which the system remains in a given state. To see
the problem, note that the probability that a sequence sampled from a given hidden
Markov model will spend precisely T steps in state k and then make a transition to adifferent state is given byp(T ) = (Akk )T (1 − Akk ) ∝ exp (−T ln Akk )(13.74)and so is an exponentially decaying function of T . For many applications, this will
be a very unrealistic model of state duration. The problem can be resolved by modelling state duration directly in which the diagonal coefﬁcients Akk are all set to zero,and each state k is explicitly associated with a probability distribution p(T |k) of possible duration times. From a generative point of view, when a state k is entered, avalue T representing the number of time steps that the system will remain in state kis then drawn from p(T |k). The model then emits T values of the observed variable
xt , which are generally assumed to be independent so that the corresponding emisTsion density is simply t=1 p(xt |k). This approach requires some straightforward
modiﬁcations to the EM optimization procedure (Rabiner, 1989).
Another limitation of the standard HMM is that it is poor at capturing longrange correlations between the observed variables (i.e., between variables that are
separated by many time steps) because these must be mediated via the ﬁrst-orderMarkov chain of hidden states. Longer-range effects could in principle be included
by adding extra links to the graphical model of Figure 13.5. One way to address thisis to generalize the HMM to give the autoregressive hidden Markov model (Ephraim
et al., 1989), an example of which is shown in Figure 13.17. For discrete observations, this corresponds to expanded tables of conditional probabilities for the emission distributions. In the case of a Gaussian emission density, we can use the linearGaussian framework in which the conditional distribution for xn given the valuesof the previous observations, and the value of zn , is a Gaussian whose mean is alinear combination of the values of the conditioning variables. Clearly the numberof additional links in the graph must be limited to avoid an excessive the number offree parameters. In the example shown in Figure 13.17, each observation depends on

13.2. Hidden Markov Models
Figure 13.18

Exercise 13.18

Example of an input-output hidden
Markov model. In this case, both theemission probabilities and the transition
probabilities depend on the values of asequence of observations u1 , . . . , uN .

633

un−1

un

un+1

zn−1

zn

zn+1

xn−1

xn

xn+1

the two preceding observed variables as well as on the hidden state. Although thisgraph looks messy, we can again appeal to d-separation to see that in fact it still has
a simple probabilistic structure. In particular, if we imagine conditioning on zn we
see that, as with the standard HMM, the values of zn−1 and zn+1 are independent,
corresponding to the conditional independence property (13.5). This is easily veriﬁed by noting that every path from node zn−1 to node zn+1 passes through at least
one observed node that is head-to-tail with respect to that path. As a consequence,
we can again use a forward-backward recursion in the E step of the EM algorithm todetermine the posterior distributions of the latent variables in a computational timethat is linear in the length of the chain. Similarly, the M step involves only a minor
modiﬁcation of the standard M-step equations. In the case of Gaussian emission
densities this involves estimating the parameters using the standard linear regression
equations, discussed in Chapter 3.
We have seen that the autoregressive HMM appears as a natural extension of thestandard HMM when viewed as a graphical model. In fact the probabilistic graphicalmodelling viewpoint motivates a plethora of different graphical structures based on
the HMM. Another example is the input-output hidden Markov model (Bengio andFrasconi, 1995), in which we have a sequence of observed variables u1 , . . . , uN , inaddition to the output variables x1 , . . . , xN , whose values inﬂuence either the distribution of latent variables or output variables, or both. An example is shown in
Figure 13.18. This extends the HMM framework to the domain of supervised learning for sequential data. It is again easy to show, through the use of the d-separation
criterion, that the Markov property (13.5) for the chain of latent variables still holds.
To verify this, simply note that there is only one path from node zn−1 to node zn+1
and this is head-to-tail with respect to the observed node zn . This conditional independence property again allows the formulation of a computationally efﬁcient learning algorithm. In particular, we can determine the parameters θ of the model by
maximizing the likelihood function L(θ) = p(X|U, θ) where U is a matrix whoserows are given by uT
n . As a consequence of the conditional independence property(13.5) this likelihood function can be maximized efﬁciently using an EM algorithm
in which the E step involves forward and backward recursions.
Another variant of the HMM worthy of mention is the factorial hidden Markov
model (Ghahramani and Jordan, 1997), in which there are multiple independent

634

13. SEQUENTIAL DATA
Figure 13.19

A factorial hidden Markov model comprising two Markov chains of latent variables. For continuous observed variables
x, one possible choice of emission model
is a linear-Gaussian density in which themean of the Gaussian is a linear combination of the states of the corresponding
latent variables.

zn−1

(2)

zn

zn−1

(1)

xn−1

(2)

(2)

zn+1

zn

(1)

zn+1

xn

xn+1

(1)

Markov chains of latent variables, and the distribution of the observed variable ata given time step is conditional on the states of all of the corresponding latent variables at that same time step. Figure 13.19 shows the corresponding graphical model.
The motivation for considering factorial HMM can be seen by noting that in order torepresent, say, 10 bits of information at a given time step, a standard HMM would
need K = 210 = 1024 latent states, whereas a factorial HMM could make use of 10
binary latent chains. The primary disadvantage of factorial HMMs, however, lies in
the additional complexity of training them. The M step for the factorial HMM modelis straightforward. However, observation of the x variables introduces dependencies
between the latent chains, leading to difﬁculties with the E step. This can be seen
(1)
(2)
by noting that in Figure 13.19, the variables zn and zn are connected by a path
which is head-to-head at node xn and hence they are not d-separated. The exact E
step for this model does not correspond to running forward and backward recursions
along the M Markov chains independently. This is conﬁrmed by noting that the key
conditional independence property (13.5) is not satisﬁed for the individual Markov
chains in the factorial HMM model, as is shown using d-separation in Figure 13.20.
Now suppose that there are M chains of hidden nodes and for simplicity supposethat all latent variables have the same number K of states. Then one approach would
be to note that there are K M combinations of latent variables at a given time step
Figure 13.20

Example of a path, highlighted in green,
which is head-to-head at the observed
nodes xn−1 and xn+1 , and head-to-tail(2)(2)
at the unobserved nodes zn−1 , zn and(2)
zn+1 . Thus the path is not blocked andso the conditional independence property(13.5) does not hold for the individual latent chains of the factorial HMM model.
As a consequence, there is no efﬁcient
exact E step for this model.

zn−1

(2)

zn

zn−1

(1)

xn−1

(2)

(2)

zn+1

zn

(1)

zn+1

xn

xn+1

(1)

13.3. Linear Dynamical Systems

Section 10.1

635

and so we can transform the model into an equivalent standard HMM having a single
chain of latent variables each of which has K M latent states. We can then run thestandard forward-backward recursions in the E step. This has computational complexity O(N K 2M ) that is exponential in the number M of latent chains and so will
be intractable for anything other than small values of M . One solution would be
to use sampling methods (discussed in Chapter 11). As an elegant deterministic alternative, Ghahramani and Jordan (1997) exploited variational inference techniques
to obtain a tractable algorithm for approximate inference. This can be done using
a simple variational posterior distribution that is fully factorized with respect to thelatent variables, or alternatively by using a more powerful approach in which thevariational distribution is described by independent Markov chains corresponding to
the chains of latent variables in the original model. In the latter case, the variational
inference algorithms involves running independent forward and backward recursions
along each chain, which is computationally efﬁcient and yet is also able to capture
correlations between variables within the same chain.
Clearly, there are many possible probabilistic structures that can be constructedaccording to the needs of particular applications. Graphical models provide a generaltechnique for motivating, describing, and analysing such structures, and variational
methods provide a powerful framework for performing inference in those models forwhich exact solution is intractable.

13.3. Linear Dynamical Systems
In order to motivate the concept of linear dynamical systems, let us consider thefollowing simple problem, which often arises in practical settings. Suppose we wish
to measure the value of an unknown quantity z using a noisy sensor that returns aobservation x representing the value of z plus zero-mean Gaussian noise. Given asingle measurement, our best guess for z is to assume that z = x. However, we
can improve our estimate for z by taking lots of measurements and averaging them,because the random noise terms will tend to cancel each other. Now let’s make thesituation more complicated by assuming that we wish to measure a quantity z thatis changing over time. We can take regular measurements of x so that at some point
in time we have obtained x1 , . . . , xN and we wish to ﬁnd the corresponding values
z1 , . . . , xN . If we simply average the measurements, the error due to random noise
will be reduced, but unfortunately we will just obtain a single averaged estimate, inwhich we have averaged over the changing value of z, thereby introducing a newsource of error.
Intuitively, we could imagine doing a bit better as follows. To estimate the valueof zN , we take only the most recent few measurements, say xN −L , . . . , xN and justaverage these. If z is changing slowly, and the random noise level in the sensor ishigh, it would make sense to choose a relatively long window of observations toaverage. Conversely, if the signal is changing quickly, and the noise levels are small,
we might be better just to use xN directly as our estimate of zN . Perhaps we could
do even better if we take a weighted average, in which more recent measurements

636

13. SEQUENTIAL DATA
make a greater contribution than less recent ones.
Although this sort of intuitive argument seems plausible, it does not tell us how
to form a weighted average, and any sort of hand-crafted weighing is hardly likely
to be optimal. Fortunately, we can address problems such as this much more systematically by deﬁning a probabilistic model that captures the time evolution andmeasurement processes and then applying the inference and learning methods developed in earlier chapters. Here we shall focus on a widely used model known as alinear dynamical system.
As we have seen, the HMM corresponds to the state space model shown in
Figure 13.5 in which the latent variables are discrete but with arbitrary emission
probability distributions. This graph of course describes a much broader class ofprobability distributions, all of which factorize according to (13.6). We now consider
extensions to other distributions for the latent variables. In particular, we consider
continuous latent variables in which the summations of the sum-product algorithm
become integrals. The general form of the inference algorithms will, however, bethe same as for the hidden Markov model. It is interesting to note that, historically,
hidden Markov models and linear dynamical systems were developed independently.
Once they are both expressed as graphical models, however, the deep relationshipbetween them immediately becomes apparent.
One key requirement is that we retain an efﬁcient algorithm for inference which
is linear in the length of the chain. This requires that, for instance, when we takea quantity α (zn−1 ), representing the posterior probability of zn given observations
x1 , . . . , xn , and multiply by the transition probability p(zn |zn−1 ) and the emission
probability p(xn |zn ) and then marginalize over zn−1 , we obtain a distribution over
 (zn−1 ). That is to say, thezn that is of the same functional form as that over α
distribution must not become more complex at each stage, but must only change inits parameter values. Not surprisingly, the only distributions that have this property
of being closed under multiplication are those belonging to the exponential family.
Here we consider the most important example from a practical perspective,
which is the Gaussian. In particular, we consider a linear-Gaussian state space modelso that the latent variables {zn }, as well as the observed variables {xn }, are multivariate Gaussian distributions whose means are linear functions of the states of theirparents in the graph. We have seen that a directed graph of linear-Gaussian units
is equivalent to a joint Gaussian distribution over all of the variables. Furthermore, (zn ) are also Gaussian, so that the functional form of the mesmarginals such as αsages is preserved and we will obtain an efﬁcient inference algorithm. By contrast,
suppose that the emission densities p(xn |zn ) comprise a mixture of K Gaussians
each of which has a mean that is linear in zn . Then even if α(z1 ) is Gaussian, thequantity α(z2 ) will be a mixture of K Gaussians, α (z3 ) will be a mixture of K 2
Gaussians, and so on, and exact inference will not be of practical value.
We have seen that the hidden Markov model can be viewed as an extension of
the mixture models of Chapter 9 to allow for sequential correlations in the data.
In a similar way, we can view the linear dynamical system as a generalization of thecontinuous latent variable models of Chapter 12 such as probabilistic PCA and factor
analysis. Each pair of nodes {zn , xn } represents a linear-Gaussian latent variable13.3. Linear Dynamical Systems

Exercise 13.19

637

model for that particular observation. However, the latent variables {zn } are nolonger treated as independent but now form a Markov chain.
Because the model is represented by a tree-structured directed graph, inference
problems can be solved efﬁciently using the sum-product algorithm. The forward recursions, analogous to the α messages of the hidden Markov model, are known as the
Kalman ﬁlter equations (Kalman, 1960; Zarchan and Musoff, 2005), and the backward recursions, analogous to the β messages, are known as the Kalman smootherequations, or the Rauch-Tung-Striebel (RTS) equations (Rauch et al., 1965). The
Kalman ﬁlter is widely used in many real-time tracking applications.
Because the linear dynamical system is a linear-Gaussian model, the joint distribution over all variables, as well as all marginals and conditionals, will be Gaussian.
It follows that the sequence of individually most probable latent variable values is
the same as the most probable latent sequence. There is thus no need to consider the
analogue of the Viterbi algorithm for the linear dynamical system.
Because the model has linear-Gaussian conditional distributions, we can write
the transition and emission distributions in the general formp(zn |zn−1 ) = N (zn |Azn−1 , Γ)p(xn |zn ) = N (xn |Czn , Σ).

(13.75)(13.76)The initial latent variable also has a Gaussian distribution which we write asp(z1 ) = N (z1 |µ0 , V0 ).

Exercise 13.24(13.77)Note that in order to simplify the notation, we have omitted additive constant terms
from the means of the Gaussians. In fact, it is straightforward to include them if
desired. Traditionally, these distributions are more commonly expressed in an equivalent form in terms of noisy linear equations given byzn = Azn−1 + wn
xn = Czn + vn
z 1 = µ0 + u(13.78)(13.79)(13.80)where the noise terms have the distributionsw ∼ N (w|0, Γ)v ∼ N (v|0, Σ)u ∼ N (u|0, V0 ).

(13.81)(13.82)(13.83)The parameters of the model, denoted by θ = {A, Γ, C, Σ, µ0 , V0 }, can be
determined using maximum likelihood through the EM algorithm. In the E step, we
need to solve the inference problem of determining the local posterior marginals forthe latent variables, which can be solved efﬁciently using the sum-product algorithm,
as we discuss in the next section.

638

13. SEQUENTIAL DATA

13.3.1 Inference in LDS
We now turn to the problem of ﬁnding the marginal distributions for the latentvariables conditional on the observation sequence. For given parameter settings, we
also wish to make predictions of the next latent state zn and of the next observation
xn conditioned on the observed data x1 , . . . , xn−1 for use in real-time applications.
These inference problems can be solved efﬁciently using the sum-product algorithm,
which in the context of the linear dynamical system gives rise to the Kalman ﬁlter
and Kalman smoother equations.
It is worth emphasizing that because the linear dynamical system is a linearGaussian model, the joint distribution over all latent and observed variables is simply
a Gaussian, and so in principle we could solve inference problems by using thestandard results derived in previous chapters for the marginals and conditionals of amultivariate Gaussian. The role of the sum-product algorithm is to provide a more
efﬁcient way to perform such computations.
Linear dynamical systems have the identical factorization, given by (13.6), to
hidden Markov models, and are again described by the factor graphs in Figures 13.14
and 13.15. Inference algorithms therefore take precisely the same form except thatsummations over latent variables are replaced by integrations. We begin by considering the forward equations in which we treat zN as the root node, and propagate
messages from the leaf node h(z1 ) to the root. From (13.77), the initial message will
be Gaussian, and because each of the factors is Gaussian, all subsequent messages
will also be Gaussian. By convention, we shall propagate messages that are normalized marginal distributions corresponding to p(zn |x1 , . . . , xn ), which we denotebyα (zn ) = N (zn |µn , Vn ).
(13.84)(zn ) given byThis is precisely analogous to the propagation of scaled variables α(13.59) in the discrete case of the hidden Markov model, and so the recursion equation now takes the form
 (zn ) = p(xn |zn ) α(13.85)cn α
 (zn−1 )p(zn |zn−1 ) dzn−1 .
Substituting for the conditionals p(zn |zn−1 ) and p(xn |zn ), using (13.75) and (13.76),
respectively, and making use of (13.84), we see that (13.85) becomescn N (zn |µn , Vn ) = N (xn |Czn , Σ)N (zn |Azn−1 , Γ)N (zn−1 |µn−1 , Vn−1 ) dzn−1 .

(13.86)

Here we are supposing that µn−1 and Vn−1 are known, and by evaluating the integral in (13.86), we wish to determine values for µn and Vn . The integral is easily
evaluated by making use of the result (2.115), from which it follows thatN (zn |Azn−1 , Γ)N (zn−1 |µn−1 , Vn−1 ) dzn−1= N (zn |Aµn−1 , Pn−1 )(13.87)

13.3. Linear Dynamical Systems
where we have deﬁned

Pn−1 = AVn−1 AT + Γ.

639

(13.88)

We can now combine this result with the ﬁrst factor on the right-hand side of (13.86)
by making use of (2.115) and (2.116) to giveµn = Aµn−1 + Kn (xn − CAµn−1 )Vn = (I − Kn C)Pn−1
cn = N (xn |CAµn−1 , CPn−1 CT + Σ).

(13.89)(13.90)(13.91)

Here we have made use of the matrix inverse identities (C.5) and (C.7) and alsodeﬁned the Kalman gain matrix
Kn = Pn−1 CT CPn−1 CT + Σ−1.

(13.92)Thus, given the values of µn−1 and Vn−1 , together with the new observation xn ,we can evaluate the Gaussian marginal for zn having mean µn and covariance Vn ,as well as the normalization coefﬁcient cn .
The initial conditions for these recursion equations are obtained from (z1 ) = p(z1 )p(x1 |z1 ).
c1 α(13.93)

Because p(z1 ) is given by (13.77), and p(x1 |z1 ) is given by (13.76), we can again
make use of (2.115) to calculate c1 and (2.116) to calculate µ1 and V1 givingµ1 = µ0 + K1 (x1 − Cµ0 )V1 = (I − K1 C)V0
c1 = N (x1 |Cµ0 , CV0 CT + Σ)
whereK1 = V0 CT CV0 CT + Σ−1.

(13.94)(13.95)(13.96)(13.97)

Similarly, the likelihood function for the linear dynamical system is given by (13.63)
in which the factors cn are found using the Kalman ﬁltering equations.
We can interpret the steps involved in going from the posterior marginal overzn−1 to the posterior marginal over zn as follows. In (13.89), we can view thequantity Aµn−1 as the prediction of the mean over zn obtained by simply taking themean over zn−1 and projecting it forward one step using the transition probability
matrix A. This predicted mean would give a predicted observation for xn given by
CAzn−1 obtained by applying the emission probability matrix C to the predicted
hidden state mean. We can view the update equation (13.89) for the mean of thehidden variable distribution as taking the predicted mean Aµn−1 and then adding
a correction that is proportional to the error xn − CAzn−1 between the predicted
observation and the actual observation. The coefﬁcient of this correction is given by
the Kalman gain matrix. Thus we can view the Kalman ﬁlter as a process of making
successive predictions and then correcting these predictions in the light of the newobservations. This is illustrated graphically in Figure 13.21.

640

13. SEQUENTIAL DATA

zn−1

zn

zn

Figure 13.21 The linear dynamical system can be viewed as a sequence of steps in which increasing uncertainty in the state variable due to diffusion is compensated by the arrival of new data. In the left-hand plot,
the blue curve shows the distribution p(zn−1 |x1 , . . . , xn−1 ), which incorporates all the data up to step n − 1.
The diffusion arising from the nonzero variance of the transition probability p(zn |zn−1 ) gives the distributionp(zn |x1 , . . . , xn−1 ), shown in red in the centre plot. Note that this is broader and shifted relative to the blue curve(which is shown dashed in the centre plot for comparison). The next data observation xn contributes through theemission density p(xn |zn ), which is shown as a function of zn in green on the right-hand plot. Note that this is nota density with respect to zn and so is not normalized to one. Inclusion of this new data point leads to a revised
distribution p(zn |x1 , . . . , xn ) for the state density shown in blue. We see that observation of the data has shifted
and narrowed the distribution compared to p(zn |x1 , . . . , xn−1 ) (which is shown in dashed in the right-hand plot
for comparison).

Exercise 13.27

Exercise 13.28

If we consider a situation in which the measurement noise is small comparedto the rate at which the latent variable is evolving, then we ﬁnd that the posterior
distribution for zn depends only on the current measurement xn , in accordance with
the intuition from our simple example at the start of the section. Similarly, if thelatent variable is evolving slowly relative to the observation noise level, we ﬁnd thatthe posterior mean for zn is obtained by averaging all of the measurements obtained
up to that time.
One of the most important applications of the Kalman ﬁlter is to tracking, andthis is illustrated using a simple example of an object moving in two dimensions inFigure 13.22.
So far, we have solved the inference problem of ﬁnding the posterior marginal
for a node zn given observations from x1 up to xn . Next we turn to the problem ofﬁnding the marginal for a node zn given all observations x1 to xN . For temporal
data, this corresponds to the inclusion of future as well as past observations. Although this cannot be used for real-time prediction, it plays a key role in learning theparameters of the model. By analogy with the hidden Markov model, this problem
can be solved by propagating messages from node xN back to node x1 and combining this information with that obtained during the forward message passing stage
 (zn ).
used to compute the α
In the LDS literature, it is usual to formulate this backward recursion in terms
 n ) rather than in terms of β(z n ). Because γ(zn ) must also be(zn )β(zof γ(zn ) = α
Gaussian, we write it in the form n ) = N (zn |µ n ).
(zn )β(z n, Vγ(zn ) = α(13.98)

To derive the required recursion, we start from the backward recursion (13.62) forFigure 13.22

Exercise 13.29

13.3. Linear Dynamical Systems

641

 n ), which, for continuous latent variables, can be written in the formβ(z n+1 )p(xn+1 |zn+1 )p(zn+1 |zn ) dzn+1 .
cn+1 β(zn ) = β(z(13.99)An illustration of a linear dynamical system being used to
track a moving object. The blue
points indicate the true positionsof the object in a two-dimensional
space at successive time steps,
the green points denote noisy
measurements of the positions,and the red crosses indicate themeans of the inferred posterior
distributions of the positions obtained by running the Kalman ﬁltering equations. The covariances of the inferred positions
are indicated by the red ellipses,
which correspond to contours
having one standard deviation.

(zn ) and substitute for p(xn+1 |zn+1 )We now multiply both sides of (13.99) by α
and p(zn+1 |zn ) using (13.75) and (13.76). Then we make use of (13.89), (13.90)
and (13.91), together with (13.98), and after some manipulation we obtain
µ
n

n
V
where we have deﬁned

 n+1 − AµN= µn + Jn µ
 n+1 − Pn JT= V n + Jn Vn−1

Jn = Vn AT (Pn )(13.100)(13.101)(13.102)

and we have made use of AVn = Pn Jn . Note that these recursions require that theforward pass be completed ﬁrst so that the quantities µn and Vn will be available
for the backward pass.
For the EM algorithm, we also require the pairwise posterior marginals, which
can be obtained from (13.65) in the formT−1

ξ(zn−1 , zn ) = (cn )=Exercise 13.31

 n)α
 (zn−1 )p(xn |zn )p(zn |z−1 )β(z n)N (zn−1 |µn−1 , Vn−1 )N (zn |Azn−1 , Γ)N (xn |Czn , Σ)N (zn |µ n, V.
cn α
 (zn )
(13.103) (zn ) using (13.84) and rearranging, we see that ξ(zn−1 , zn ) is a
Substituting for αGaussian with mean given with components γ(zn−1 ) and γ(zn ), and a covariance
between zn and zn−1 given by n.
cov[zn , zn−1 ] = Jn−1 V(13.104)642

13. SEQUENTIAL DATA

13.3.2 Learning in LDS
So far, we have considered the inference problem for linear dynamical systems,
assuming that the model parameters θ = {A, Γ, C, Σ, µ0 , V0 } are known. Next, weconsider the determination of these parameters using maximum likelihood (Ghahramani and Hinton, 1996b). Because the model has latent variables, this can be addressed using the EM algorithm, which was discussed in general terms in Chapter 9.
We can derive the EM algorithm for the linear dynamical system as follows. Let
us denote the estimated parameter values at some particular cycle of the algorithmby θ old . For these parameter values, we can run the inference algorithm to determine
the posterior distribution of the latent variables p(Z|X, θ old ), or more precisely those
local posterior marginals that are required in the M step. In particular, we shallrequire the following expectations

n
E [zn ] = µ Tn + µE zn zn−1 = Jn−1 V
 nµ
T
n−1
 T
T

 nµ
n
E zn zn = V n + µ(13.105)(13.106)(13.107)

where we have used (13.104).
Now we consider the complete-data log likelihood function, which is obtainedby taking the logarithm of (13.6) and is therefore given byln p(X, Z|θ) = ln p(z1 |µ0 , V0 ) +N

ln p(zn |zn−1 , A, Γ)

n=2

+

N


ln p(xn |zn , C, Σ)(13.108)

n=1

in which we have made the dependence on the parameters explicit. We now take theexpectation of the complete-data log likelihood with respect to the posterior distribution p(Z|X, θ old ) which deﬁnes the function
Q(θ, θ old ) = EZ|θold [ln p(X, Z|θ)] .

(13.109)

In the M step, this function is maximized with respect to the components of θ.
Consider ﬁrst the parameters µ0 and V0 . If we substitute for p(z1 |µ0 , V0 ) in
(13.108) using (13.77), and then take the expectation with respect to Z, we obtain11(z1 − µ0 )T V0−1 (z1 − µ0 ) + const
Q(θ, θ old ) = − ln |V0 | − EZ|θold
2
2

Exercise 13.32

where all terms not dependent on µ0 or V0 have been absorbed into the additive
constant. Maximization with respect to µ0 and V0 is easily performed by making
use of the maximum likelihood solution for a Gaussian distribution discussed in
Section 2.3.4, giving

13.3. Linear Dynamical Systems= E[z1 ]T= E[z1 zT
1 ] − E[z1 ]E[z1 ].

µnew
0
V0new

643
(13.110)(13.111)

Similarly, to optimize A and Γ, we substitute for p(zn |zn−1 , A, Γ) in (13.108)
using (13.75) giving
N −1
ln |Γ|
Q(θ, θ old ) = −2
 N1
−EZ|θold(zn − Azn−1 )T Γ−1 (zn − Azn−1 ) + const
2(13.112)

n=2

Exercise 13.33

in which the constant comprises terms that are independent of A and Γ. Maximizing
with respect to these parameters then gives

 N
−1
N
 
 
=
E zn zT
E zn−1 zT
n−1
n−1

Anew

n=2

Γnew

(13.113)n=2N1  
new
E zn zT
E zn−1 zT
n −A
n
N −1
n=2



new
new
T
new T
−E zn zT
A
+
A
E
z
z
(A
)
.
n−
1
n−1
n−1

=

(13.114)

Note that Anew must be evaluated ﬁrst, and the result can then be used to determine
Γnew .
Finally, in order to determine the new values of C and Σ, we substitute forp(xn |zn , C, Σ) in (13.108) using (13.76) giving
Q(θ, θ old ) = −N
ln |Σ|
2


−EZ|θold1
(xn − Czn )T Σ−1 (xn − Czn ) + const.
2
N

n=1

Exercise 13.34

Maximizing with respect to C and Σ then gives
Cnew

=

N

n=1


xn E zT
n

 N
−1
 
E zn zTn(13.115)

n=1

1 
new
xn xT
E [zn ] xT
n −C
n
N
n=1
 T new
new−xn E zn C
+ Cnew E zn zT
.
n C
N

Σnew

=

(13.116)644

13. SEQUENTIAL DATA
We have approached parameter learning in the linear dynamical system using
maximum likelihood. Inclusion of priors to give a MAP estimate is straightforward,
and a fully Bayesian treatment can be found by applying the analytical approximation techniques discussed in Chapter 10, though a detailed treatment is precluded
here due to lack of space.

13.3.3 Extensions of LDS

Chapter 10

As with the hidden Markov model, there is considerable interest in extending
the basic linear dynamical system in order to increase its capabilities. Although theassumption of a linear-Gaussian model leads to efﬁcient algorithms for inference
and learning, it also implies that the marginal distribution of the observed variables
is simply a Gaussian, which represents a signiﬁcant limitation. One simple extensionof the linear dynamical system is to use a Gaussian mixture as the initial distribution
for z1 . If this mixture has K components, then the forward recursion equations(13.85) will lead to a mixture of K Gaussians over each hidden variable zn , and so
the model is again tractable.
For many applications, the Gaussian emission density is a poor approximation.
If instead we try to use a mixture of K Gaussians as the emission density, then the (z1 ) will also be a mixture of K Gaussians. However, from (13.85) the
posterior α(z2 ) will comprise a mixture of K 2 Gaussians, and so on, with α (zn )
posterior α
being given by a mixture of K n Gaussians. Thus the number of components grows
exponentially with the length of the chain, and so this model is impractical.
More generally, introducing transition or emission models that depart from thelinear-Gaussian (or other exponential family) model leads to an intractable inference problem. We can make deterministic approximations such as assumed density ﬁltering or expectation propagation, or we can make use of sampling methods,
as discussed in Section 13.3.4. One widely used approach is to make a Gaussian
approximation by linearizing around the mean of the predicted distribution, which
gives rise to the extended Kalman ﬁlter (Zarchan and Musoff, 2005).
As with hidden Markov models, we can develop interesting extensions of the basic linear dynamical system by expanding its graphical representation. For example,
the switching state space model (Ghahramani and Hinton, 1998) can be viewed as
a combination of the hidden Markov model with a set of linear dynamical systems.
The model has multiple Markov chains of continuous linear-Gaussian latent variables, each of which is analogous to the latent chain of the linear dynamical system
discussed earlier, together with a Markov chain of discrete variables of the form used
in a hidden Markov model. The output at each time step is determined by stochastically choosing one of the continuous latent chains, using the state of the discretelatent variable as a switch, and then emitting an observation from the corresponding
conditional output distribution. Exact inference in this model is intractable, but variational methods lead to an efﬁcient inference scheme involving forward-backward
recursions along each of the continuous and discrete Markov chains independently.
Note that, if we consider multiple chains of discrete latent variables, and use one as
the switch to select from the remainder, we obtain an analogous model having only
discrete latent variables known as the switching hidden Markov model.

13.3. Linear Dynamical Systems

645

13.3.4 Particle ﬁlters
Chapter 11

For dynamical systems which do not have a linear-Gaussian, for example, ifthey use a non-Gaussian emission density, we can turn to sampling methods in order
to ﬁnd a tractable inference algorithm. In particular, we can apply the samplingimportance-resampling formalism of Section 11.1.5 to obtain a sequential Monte
Carlo algorithm known as the particle ﬁlter.
Consider the class of distributions represented by the graphical model in Figure 13.5, and suppose we are given the observed values Xn = (x1 , . . . , xn ) and
we wish to draw L samples from the posterior distribution p(zn |Xn ). Using Bayes’
theorem, we haveE[f (zn )] =f (zn )p(zn |Xn ) dzn
=f (zn )p(zn |xn , Xn−1 ) dzn
f (zn )p(xn |zn )p(zn |Xn−1 ) dzn
=p(xn |zn )p(zn |Xn−1 ) dznLwn(l) f (z(nl) )(13.117)l=1(l )where {zn } is a set of samples drawn from p(zn |Xn−1 ) and we have made use of
the conditional independence property p(xn |zn , Xn−1 ) = p(xn |zn ), which follows(l )from the graph in Figure 13.5. The sampling weights {wn } are deﬁned bywn(l) = L(l )p(xn |zn )(m)
m=1 p(xn |zn )(13.118)

where the same samples are used in the numerator as in the denominator. Thus the(l )posterior distribution p(zn |xn ) is represented by the set of samples {zn } together
(l )
(l )
with the corresponding weights {wn }. Note that these weights satisfy 0  wn 1
 (l )and l wn = 1.
Because we wish to ﬁnd a sequential sampling scheme, we shall suppose thata set of samples and weights have been obtained at time step n, and that we havesubsequently observed the value of xn+1 , and we wish to ﬁnd the weights and samples at time step n + 1. We ﬁrst sample from the distribution p(zn+1 |Xn ). This is646

13. SEQUENTIAL DATA
straightforward since, again using Bayes’ theoremp(zn+1 |zn , Xn )p(zn |Xn ) dzn
p(zn+1 |Xn ) ==p(zn+1 |zn )p(zn |Xn ) dzn
=p(zn+1 |zn )p(zn |xn , Xn−1 ) dzn
p(zn+1 |zn )p(xn |zn )p(zn |Xn−1 ) dzn
=p(xn |zn )p(zn |Xn−1 ) dzn
=wn(l) p(zn+1 |z(nl) )(13.119)lwhere we have made use of the conditional independence propertiesp(zn+1 |zn , Xn ) = p(zn+1 |zn )p(xn |zn , Xn−1 ) = p(xn |zn )(13.120)(13.121)

which follow from the application of the d-separation criterion to the graph in Figure 13.5. The distribution given by (13.119) is a mixture distribution, and samples
can be drawn by choosing a component l with probability given by the mixing coefﬁcients w(l) and then drawing a sample from the corresponding component.
In summary, we can view each step of the particle ﬁlter algorithm as comprising
two stages. At time step n, we have a sample representation of the posterior dis(l )(l )tribution p(zn |Xn ) expressed as samples {zn } with corresponding weights {wn }.
This can be viewed as a mixture representation of the form (13.119). To obtain thecorresponding representation for the next time step, we ﬁrst draw L samples fromthe mixture distribution (13.119), and then for each sample we use the new obser(l )(l )vation xn+1 to evaluate the corresponding weights wn+1 ∝ p(xn+1 |zn+1 ). This is
illustrated, for the case of a single variable z, in Figure 13.23.
The particle ﬁltering, or sequential Monte Carlo, approach has appeared in theliterature under various names including the bootstrap ﬁlter (Gordon et al., 1993),
survival of the ﬁttest (Kanazawa et al., 1995), and the condensation algorithm (Isard
and Blake, 1998).

Exercises

13.1 ( ) www Use the technique of d-separation, discussed in Section 8.2, to verify
that the Markov model shown in Figure 13.3 having N nodes in total satisﬁes theconditional independence properties (13.3) for n = 2, . . . , N . Similarly, show thata model described by the graph in Figure 13.4 in which there are N nodes in total

Exercises

647

p(zn |Xn )p(zn+1 |Xn )p(xn+1 |zn+1 )p(zn+1 |Xn+1 )Figure 13.23

z

Schematic illustration of the operation of the particle ﬁlter for a one-dimensional latent
space. At time step n, the posterior p(zn |xn ) is represented as a mixture distribution,(l)shown schematically as circles whose sizes are proportional to the weights wn . A set of(l)
L samples is then drawn from this distribution and the new weights wn+1 evaluated using(l)p(xn+1 |zn+1 ).

satisﬁes the conditional independence properties
p(xn |x1 , . . . , xn−1 ) = p(xn |xn−1 , xn−2 )(13.122)for n = 3, . . . , N .
13.2 ( ) Consider the joint probability distribution (13.2) corresponding to the directed
graph of Figure 13.3. Using the sum and product rules of probability, verify that
this joint distribution satisﬁes the conditional independence property (13.3) for n =2, . . . , N . Similarly, show that the second-order Markov model described by thejoint distribution (13.4) satisﬁes the conditional independence propertyp(xn |x1 , . . . , xn−1 ) = p(xn |xn−1 , xn−2 )(13.123)

for n = 3, . . . , N .
13.3 ( ) By using d-separation, show that the distribution p(x1 , . . . , xN ) of the observed
data for the state space model represented by the directed graph in Figure 13.5 does
not satisfy any conditional independence properties and hence does not exhibit theMarkov property at any ﬁnite order.
13.4 ( ) www Consider a hidden Markov model in which the emission densities are
represented by a parametric model p(x|z, w), such as a linear regression model ora neural network, in which w is a vector of adaptive parameters. Describe how theparameters w can be learned from data using maximum likelihood.

648

13. SEQUENTIAL DATA
13.5 ( ) Verify the M-step equations (13.18) and (13.19) for the initial state probabilities and transition probability parameters of the hidden Markov model by maximization of the expected complete-data log likelihood function (13.17), using appropriate
Lagrange multipliers to enforce the summation constraints on the components of πand A.
13.6 ( ) Show that if any elements of the parameters π or A for a hidden Markov
model are initially set to zero, then those elements will remain zero in all subsequent
updates of the EM algorithm.
13.7 ( ) Consider a hidden Markov model with Gaussian emission densities. Show thatmaximization of the function Q(θ, θ old ) with respect to the mean and covarianceparameters of the Gaussians gives rise to the M-step equations (13.20) and (13.21).
13.8 ( ) www For a hidden Markov model having discrete observations governed bya multinomial distribution, show that the conditional distribution of the observations
given the hidden variables is given by (13.22) and the corresponding M step equations are given by (13.23). Write down the analogous equations for the conditional
distribution and the M step equations for the case of a hidden Markov with multiple
binary output variables each of which is governed by a Bernoulli conditional distribution. Hint: refer to Sections 2.1 and 2.2 for a discussion of the corresponding
maximum likelihood solutions for i.i.d. data if required.
13.9 ( ) www Use the d-separation criterion to verify that the conditional independence properties (13.24)–(13.31) are satisﬁed by the joint distribution for the hidden
Markov model deﬁned by (13.6).
13.10 () By applying the sum and product rules of probability, verify that the conditional independence properties (13.24)–(13.31) are satisﬁed by the joint distribution
for the hidden Markov model deﬁned by (13.6).
13.11 ( ) Starting from the expression (8.72) for the marginal distribution over the variables of a factor in a factor graph, together with the results for the messages in thesum-product algorithm obtained in Section 13.2.3, derive the result (13.43) for thejoint posterior distribution over two successive latent variables in a hidden Markov
model.
13.12 ( ) Suppose we wish to train a hidden Markov model by maximum likelihoodusing data that comprises R independent sequences of observations, which we denote by X(r) where r = 1, . . . , R. Show that in the E step of the EM algorithm,
we simply evaluate posterior probabilities for the latent variables by running the αand β recursions independently for each of the sequences. Also show that in the
M step, the initial probability and transition probability parameters are re-estimatedExercises

649

using modiﬁed forms of (13.18 ) and (13.19) given by
R


πk

=

r =1

(r )

γ(z1k )

R 
K


(13.124)(r )γ(z1j )

r =1 j =1
R 
N


Ajk

=

r =1 n=2

(r )

(r )

ξ(zn−1,j , zn,k )

R 
K 
N

r =1 l=1 n=2(13.125)
(r )

(r )

ξ(zn−1,j , zn,l )where, for notational convenience, we have assumed that the sequences are of thesame length (the generalization to sequences of different lengths is straightforward).
Similarly, show that the M-step equation for re-estimation of the means of Gaussian
emission models is given by
R 
N


(r )

γ(znk )x(nr)µk = r=1Rn=1N
r =1 n=1.

(13.126)(r )γ(znk )

Note that the M-step equations for other emission model parameters and distributions
take an analogous form.
13.13 ( ) www Use the deﬁnition (8.64) of the messages passed from a factor node
to a variable node in a factor graph, together with the expression (13.6) for the joint
distribution in a hidden Markov model, to show that the deﬁnition (13.50) of thealpha message is the same as the deﬁnition (13.34).
13.14 ( ) Use the deﬁnition (8.67) of the messages passed from a factor node to avariable node in a factor graph, together with the expression (13.6) for the joint
distribution in a hidden Markov model, to show that the deﬁnition (13.52) of thebeta message is the same as the deﬁnition (13.35).
13.15 ( ) Use the expressions (13.33) and (13.43) for the marginals in a hidden Markov
model to derive the corresponding results (13.64) and (13.65) expressed in terms ofre-scaled variables.
13.16 () In this exercise, we derive the forward message passing equation for theViterbi algorithm directly from the expression (13.6) for the joint distribution. This
involves maximizing over all of the hidden variables z1 , . . . , zN . By taking the logarithm and then exchanging maximizations and summations, derive the recursion

650

13. SEQUENTIAL DATA
(13.68) where the quantities ω(zn ) are deﬁned by (13.70). Show that the initialcondition for this recursion is given by (13.69).
13.17 ( ) www Show that the directed graph for the input-output hidden Markov model,
given in Figure 13.18, can be expressed as a tree-structured factor graph of the formshown in Figure 13.15 and write down expressions for the initial factor h(z1 ) andfor the general factor fn (zn−1 , zn ) where 2  n  N .
13.18 () Using the result of Exercise 13.17, derive the recursion equations, including the initial conditions, for the forward-backward algorithm for the input-output
hidden Markov model shown in Figure 13.18.
13.19 ( ) www The Kalman ﬁlter and smoother equations allow the posterior distributions over individual latent variables, conditioned on all of the observed variables,
to be found efﬁciently for linear dynamical systems. Show that the sequence oflatent variable values obtained by maximizing each of these posterior distributionsindividually is the same as the most probable sequence of latent values. To do this,simply note that the joint distribution of all latent and observed variables in a linear
dynamical system is Gaussian, and hence all conditionals and marginals will also be
Gaussian, and then make use of the result (2.98).
13.20 (

) www

Use the result (2.115) to prove (13.87).

13.21 ( ) Use the results (2.115) and (2.116), together with the matrix identities (C.5)
and (C.7), to derive the results (13.89), (13.90), and (13.91), where the Kalman gain
matrix Kn is deﬁned by (13.92).
13.22 ( ) www Using (13.93), together with the deﬁnitions (13.76) and (13.77) andthe result (2.115), derive (13.96).
13.23 ( ) Using (13.93), together with the deﬁnitions (13.76) and (13.77) and the result(2.116), derive (13.94), (13.95) and (13.97).
13.24 ( ) www Consider a generalization of (13.75) and (13.76) in which we include
constant terms a and c in the Gaussian means, so thatp(zn |zn−1 ) = N (zn |Azn−1 + a, Γ)p(xn |zn ) = N (xn |Czn + c, Σ).

(13.127)(13.128)Show that this extension can be re-case in the framework discussed in this chapter by
deﬁning a state vector z with an additional component ﬁxed at unity, and then augmenting the matrices A and C using extra columns corresponding to the parameters
a and c.
13.25 ( ) In this exercise, we show that when the Kalman ﬁlter equations are applied
to independent observations, they reduce to the results given in Section 2.3 for the
maximum likelihood solution for a single Gaussian distribution. Consider the problem of ﬁnding the mean µ of a single Gaussian random variable x, in which we are
given a set of independent observations {x1 , . . . , xN }. To model this we can use

Exercises

651

a linear dynamical system governed by (13.75) and (13.76), with latent variables{z1 , . . . , zN } in which C becomes the identity matrix and where the transition probability A = 0 because the observations are independent. Let the parameters m0
and V0 of the initial state be denoted by µ0 and σ02 , respectively, and suppose thatΣ becomes σ 2 . Write down the corresponding Kalman ﬁlter equations starting fromthe general results (13.89) and (13.90), together with (13.94) and (13.95). Show thatthese are equivalent to the results (2.141) and (2.142) obtained directly by considering independent data.
13.26 () Consider a special case of the linear dynamical system of Section 13.3 that isequivalent to probabilistic PCA, so that the transition matrix A = 0, the covarianceΓ = I, and the noise covariance Σ = σ 2 I. By making use of the matrix inversion
identity (C.7) show that, if the emission density matrix C is denoted W, then theposterior distribution over the hidden states deﬁned by (13.89) and (13.90) reduces
to the result (12.42) for probabilistic PCA.
Consider a linear dynamical system of the form discussed in Sec13.27 ( ) wwwtion 13.3 in which the amplitude of the observation noise goes to zero, so that Σ = 0.
Show that the posterior distribution for zn has mean xn and zero variance. Thisaccords with our intuition that if there is no noise, we should just use the current
observation xn to estimate the state variable zn and ignore all previous observations.
13.28 () Consider a special case of the linear dynamical system of Section 13.3 inwhich the state variable zn is constrained to be equal to the previous state variable,
which corresponds to A = I and Γ = 0. For simplicity, assume also that V0 → ∞so that the initial conditions for z are unimportant, and the predictions are determined
purely by the data. Use proof by induction to show that the posterior mean for statezn is determined by the average of x1 , . . . , xn . This corresponds to the intuitive
result that if the state variable is constant, our best estimate is obtained by averaging
the observations.
13.29 (
) Starting from the backwards recursion equation (13.99), derive the RTS
smoothing equations (13.100) and (13.101) for the Gaussian linear dynamical system.
13.30 ( ) Starting from the result (13.65) for the pairwise posterior marginal in a state
space model, derive the speciﬁc form (13.103) for the case of the Gaussian linear
dynamical system.

 (zn ) using (13.84),
13.31 ( ) Starting from the result (13.103) and by substituting for α
verify the result (13.104) for the covariance between zn and zn−1 .
13.32 ( ) www Verify the results (13.110) and (13.111) for the M-step equations forµ0 and V0 in the linear dynamical system.
13.33 ( ) Verify the results (13.113) and (13.114) for the M-step equations for A and Γin the linear dynamical system.

652

13. SEQUENTIAL DATA
13.34 ( ) Verify the results (13.115) and (13.116) for the M-step equations for C and Σin the linear dynamical system.

14
Combining
ModelsIn earlier chapters, we have explored a range of different models for solving classiﬁcation and regression problems. It is often found that improved performance can beobtained by combining multiple models together in some way, instead of just using
a single model in isolation. For instance, we might train L different models and thenmake predictions using the average of the predictions made by each model. Suchcombinations of models are sometimes called committees. In Section 14.2, we discuss ways to apply the committee concept in practice, and we also give some insight
into why it can sometimes be an effective procedure.
One important variant of the committee method, known as boosting, involves
training multiple models in sequence in which the error function used to train a particular model depends on the performance of the previous models. This can produce
substantial improvements in performance compared to the use of a single model andis discussed in Section 14.3.
Instead of averaging the predictions of a set of models, an alternative form of

653

654

14. COMBINING MODELS
model combination is to select one of the models to make the prediction, in which
the choice of model is a function of the input variables. Thus different models become responsible for making predictions in different regions of input space. One
widely used framework of this kind is known as a decision tree in which the selection process can be described as a sequence of binary selections corresponding to
the traversal of a tree structure and is discussed in Section 14.4. In this case, theindividual models are generally chosen to be very simple, and the overall ﬂexibilityof the model arises from the input-dependent selection process. Decision trees canbe applied to both classiﬁcation and regression problems.
One limitation of decision trees is that the division of input space is based on
hard splits in which only one model is responsible for making predictions for any
given value of the input variables. The decision process can be softened by moving
to a probabilistic framework for combining models, as discussed in Section 14.5. For
example, if we have a set of K models for a conditional distribution p(t|x, k) where
x is the input variable, t is the target variable, and k = 1, . . . , K indexes the model,
then we can form a probabilistic mixture of the formp(t|x) =Kπk (x)p(t|x, k)(14.1)

k=1

in which πk (x) = p(k|x) represent the input-dependent mixing coefﬁcients. Suchmodels can be viewed as mixture distributions in which the component densities, aswell as the mixing coefﬁcients, are conditioned on the input variables and are known
as mixtures of experts. They are closely related to the mixture density network model
discussed in Section 5.6.

14.1. Bayesian Model Averaging

Section 9.2

It is important to distinguish between model combination methods and Bayesianmodel averaging, as the two are often confused. To understand the difference, consider the example of density estimation using a mixture of Gaussians in which several
Gaussian components are combined probabilistically. The model contains a binary
latent variable z that indicates which component of the mixture is responsible forgenerating the corresponding data point. Thus the model is speciﬁed in terms of a
joint distributionp(x, z)(14.2)and the corresponding density over the observed variable x is obtained by marginalizing over the latent variable
p(x, z).
(14.3)
p(x) =z14.2. Committees

655

In the case of our Gaussian mixture example, this leads to a distribution of the form
p(x) =K

πk N (x|µk , Σk )(14.4)

k=1

with the usual interpretation of the symbols. This is an example of model combination. For independent, identically distributed data, we can use (14.3) to write themarginal probability of a data set X = {x1 , . . . , xN } in the form

N
N



p(X) =p(xn ) =p(xn , zn ) .
(14.5)
n=1

n=1

zn

Thus we see that each observed data point xn has a corresponding latent variable zn .
Now suppose we have several different models indexed by h = 1, . . . , H withprior probabilities p(h). For instance one model might be a mixture of Gaussians andanother model might be a mixture of Cauchy distributions. The marginal distribution
over the data set is given byp(X) =Hp(X|h)p(h).

(14.6)

h=1

Exercise 14.1

This is an example of Bayesian model averaging. The interpretation of this summation over h is that just one model is responsible for generating the whole data set,
and the probability distribution over h simply reﬂects our uncertainty as to whichmodel that is. As the size of the data set increases, this uncertainty reduces, andthe posterior probabilities p(h|X) become increasingly focussed on just one of themodels.
This highlights the key difference between Bayesian model averaging and model
combination, because in Bayesian model averaging the whole data set is generated
by a single model. By contrast, when we combine multiple models, as in (14.5), we
see that different data points within the data set can potentially be generated fromdifferent values of the latent variable z and hence by different components.
Although we have considered the marginal probability p(X), the same considerations apply for the predictive density p(x|X) or for conditional distributions suchas p(t|x, X, T).

14.2. Committees

Section 3.2

The simplest way to construct a committee is to average the predictions of a set ofindividual models. Such a procedure can be motivated from a frequentist perspective
by considering the trade-off between bias and variance, which decomposes the error due to a model into the bias component that arises from differences between themodel and the true function to be predicted, and the variance component that represents the sensitivity of the model to the individual data points. Recall from Figure 3.5

656

14. COMBINING MODELS
that when we trained multiple polynomials using the sinusoidal data, and then averaged the resulting functions, the contribution arising from the variance term tended tocancel, leading to improved predictions. When we averaged a set of low-bias models (corresponding to higher order polynomials), we obtained accurate predictions
for the underlying sinusoidal function from which the data were generated.
In practice, of course, we have only a single data set, and so we have to ﬁnd
a way to introduce variability between the different models within the committee.
One approach is to use bootstrap data sets, discussed in Section 1.2.3. Consider aregression problem in which we are trying to predict the value of a single continuous
variable, and suppose we generate M bootstrap data sets and then use each to train
a separate copy ym (x) of a predictive model where m = 1, . . . , M . The committee
prediction is given byM
1 yCOM (x) =ym (x).
(14.7)
M
m=1

This procedure is known as bootstrap aggregation or bagging (Breiman, 1996).
Suppose the true regression function that we are trying to predict is given byh(x), so that the output of each of the models can be written as the true value plus
an error in the form
(14.8)ym (x) = h(x) + m (x).
The average sum-of-squares error then takes the form
2
Ex {ym (x) − h(x)} = Ex m (x)2

(14.9)

where Ex [·] denotes a frequentist expectation with respect to the distribution of the
input vector x. The average error made by the models acting individually is therefore
1 Ex m (x)2 .
M
M

EAV =

(14.10)m=1Similarly, the expected error from the committee (14.7) is given by
⎡
2 ⎤
M

1
ym (x) − h(x) ⎦ECOM = Ex ⎣
M
m=1
⎡
2 ⎤
M

1
= Ex ⎣
m (x) ⎦M(14.11)m=1

If we assume that the errors have zero mean and are uncorrelated, so that
Ex [m (x)] = 0
Ex [m (x)l (x)] = 0,m=l(14.12)(14.13)

14.3. Boosting
Exercise 14.2

657

then we obtain

1
EAV .
(14.14)
M
This apparently dramatic result suggests that the average error of a model can be
reduced by a factor of M simply by averaging M versions of the model. Unfortunately, it depends on the key assumption that the errors due to the individual models
are uncorrelated. In practice, the errors are typically highly correlated, and the reduction in overall error is generally small. It can, however, be shown that the expectedcommittee error will not exceed the expected error of the constituent models, so that
ECOM  EAV . In order to achieve more signiﬁcant improvements, we turn to a moresophisticated technique for building committees, known as boosting.
ECOM =

Exercise 14.3

14.3. Boosting
Boosting is a powerful technique for combining multiple ‘base’ classiﬁers to produce
a form of committee whose performance can be signiﬁcantly better than that of anyof the base classiﬁers. Here we describe the most widely used form of boostingalgorithm called AdaBoost, short for ‘adaptive boosting’, developed by Freund andSchapire (1996). Boosting can give good results even if the base classiﬁers have a
performance that is only slightly better than random, and hence sometimes the baseclassiﬁers are known as weak learners. Originally designed for solving classiﬁcation
problems, boosting can also be extended to regression (Friedman, 2001).
The principal difference between boosting and the committee methods such as
bagging discussed above, is that the base classiﬁers are trained in sequence, andeach base classiﬁer is trained using a weighted form of the data set in which theweighting coefﬁcient associated with each data point depends on the performanceof the previous classiﬁers. In particular, points that are misclassiﬁed by one of thebase classiﬁers are given greater weight when used to train the next classiﬁer inthe sequence. Once all the classiﬁers have been trained, their predictions are then
combined through a weighted majority voting scheme, as illustrated schematicallyin Figure 14.1.
Consider a two-class classiﬁcation problem, in which the training data comprises
input vectors x1 , . . . , xN along with corresponding binary target variables t1 , . . . , tN
where tn ∈ {−1, 1}. Each data point is given an associated weighting parameter
wn , which is initially set 1/N for all data points. We shall suppose that we havea procedure available for training a base classiﬁer using weighted data to give afunction y(x) ∈ {−1, 1}. At each stage of the algorithm, AdaBoost trains a newclassiﬁer using a data set in which the weighting coefﬁcients are adjusted according
to the performance of the previously trained classiﬁer so as to give greater weightto the misclassiﬁed data points. Finally, when the desired number of base classiﬁers
have been trained, they are combined to form a committee using coefﬁcients thatgive different weight to different base classiﬁers. The precise form of the AdaBoost
algorithm is given below.

658

14. COMBINING MODELS
Figure 14.1

Schematic illustration of theboosting framework.
Eachbase classiﬁer ym (x) is trainedon a weighted form of the training set (blue arrows) in which(m)the weights wn depend on
the performance of the previous base classiﬁer ym−1 (x)(green arrows). Once all baseclassiﬁers have been trained,
they are combined to givethe ﬁnal classiﬁer YM (x) (redarrows).

{wn }

(1)

{wn }

(2)

{wn

(M )

y1 (x)y2 (x)yM (x)}M
YM (x) = sign
αm ym (x)m

AdaBoost(1)

1. Initialize the data weighting coefﬁcients {wn } by setting wn = 1/N forn = 1, . . . , N .
2. For m = 1, . . . , M :(a) Fit a classiﬁer ym (x) to the training data by minimizing the weighted
error function
Jm =

N


wn(m) I(ym (xn ) = tn )(14.15)n=1

where I(ym (xn ) = tn ) is the indicator function and equals 1 whenym (xn ) = tn and 0 otherwise.
(b) Evaluate the quantities
N


m =

wn(m) I(ym (xn ) = tn )

n=1
N


(14.16)wn(m)n=1

and then use these to evaluate
αm = ln1 − mm.

(14.17)(c) Update the data weighting coefﬁcientswn(m+1) = wn(m) exp {αm I(ym (xn ) = tn )}(14.18)

14.3. Boosting
3. Make predictions using the ﬁnal model, which is given by MYM (x) = sign
αm ym (x) .

659(14.19)m=1Section 14.4

We see that the ﬁrst base classiﬁer y1 (x) is trained using weighting coefﬁ(1)cients wn that are all equal, which therefore corresponds to the usual procedure
for training a single classiﬁer. From (14.18), we see that in subsequent iterations(m)the weighting coefﬁcients wn are increased for data points that are misclassiﬁed
and decreased for data points that are correctly classiﬁed. Successive classiﬁers are
therefore forced to place greater emphasis on points that have been misclassiﬁed byprevious classiﬁers, and data points that continue to be misclassiﬁed by successiveclassiﬁers receive ever greater weight. The quantities m represent weighted measures of the error rates of each of the base classiﬁers on the data set. We therefore
see that the weighting coefﬁcients αm deﬁned by (14.17) give greater weight to themore accurate classiﬁers when computing the overall output given by (14.19).
The AdaBoost algorithm is illustrated in Figure 14.2, using a subset of 30 data
points taken from the toy classiﬁcation data set shown in Figure A.7. Here each baselearners consists of a threshold on one of the input variables. This simple classiﬁer
corresponds to a form of decision tree known as a ‘decision stumps’, i.e., a decision tree with a single node. Thus each base learner classiﬁes an input according towhether one of the input features exceeds some threshold and therefore simply partitions the space into two regions separated by a linear decision surface that is parallel
to one of the axes.

14.3.1 Minimizing exponential error
Boosting was originally motivated using statistical learning theory, leading toupper bounds on the generalization error. However, these bounds turn out to be tooloose to have practical value, and the actual performance of boosting is much better
than the bounds alone would suggest. Friedman et al. (2000) gave a different andvery simple interpretation of boosting in terms of the sequential minimization of an
exponential error function.
Consider the exponential error function deﬁned byE=N

exp {−tn fm (xn )}(14.20)n=1

where fm (x) is a classiﬁer deﬁned in terms of a linear combination of base classiﬁers
yl (x) of the formm1
fm (x) =αl yl (x)(14.21)2
l=1

and tn ∈ {−1, 1} are the training set target values. Our goal is to minimize E withrespect to both the weighting coefﬁcients αl and the parameters of the base classiﬁers
yl (x).

660

14. COMBINING MODELS

2

m =1

2

m =2

2

0

0

0

−2

−2

−2

−1

0

1

2

2
m =6

−1

0

1

2

2
m = 10

−1

0

0

−2

−2

−2

0

1

2

−1

0

1

2

0

1

2

0

−1

m =3

2
m = 150

−1

0

1

2

Figure 14.2 Illustration of boosting in which the base learners consist of simple thresholds applied to one orother of the axes. Each ﬁgure shows the number m of base learners trained so far, along with the decisionboundary of the most recent base learner (dashed black line) and the combined decision boundary of the ensemble (solid green line). Each data point is depicted by a circle whose radius indicates the weight assigned tothat data point when training the most recently added base learner. Thus, for instance, we see that points thatare misclassiﬁed by the m = 1 base learner are given greater weight when training the m = 2 base learner.

Instead of doing a global error function minimization, however, we shall suppose that the base classiﬁers y1 (x), . . . , ym−1 (x) are ﬁxed, as are their coefﬁcients
α1 , . . . , αm−1 , and so we are minimizing only with respect to αm and ym (x). Separating off the contribution from base classiﬁer ym (x), we can then write the error
function in the form
E

=

N

n=1

=

N

n=1

1
exp −tn fm−1 (xn ) − tn αm ym (xn )21wn(m) exp − tn αm ym (xn )2(m)(14.22)where the coefﬁcients wn = exp{−tn fm−1 (xn )} can be viewed as constantsbecause we are optimizing only αm and ym (x). If we denote by Tm the set ofdata points that are correctly classiﬁed by ym (x), and if we denote the remaining
misclassiﬁed points by Mm , then we can in turn rewrite the error function in the14.3. BoostingformE= e−αm /2wn(m) + eαm /2

n∈Tm
αm /2

= (e

wn(m)

n∈Mm

−αm /2

−e



661

)

N


(m)

−αm /2wn I(ym (xn ) = tn ) + e

n=1

N


wn(m) .

n=1

(14.23)

Exercise 14.6

When we minimize this with respect to ym (x), we see that the second term is constant, and so this is equivalent to minimizing (14.15) because the overall multiplicative factor in front of the summation does not affect the location of the minimum.
Similarly, minimizing with respect to αm , we obtain (14.17) in which m is deﬁnedby (14.16).
From (14.22) we see that, having found αm and ym (x), the weights on the data
points are updated using1wn(m+1) = wn(m) exp − tn αm ym (xn ) .
2(14.24)Making use of the fact thattn ym (xn ) = 1 − 2I(ym (xn ) = tn )(m)we see that the weights wn(14.25)are updated at the next iteration using

wn(m+1) = wn(m) exp(−αm /2) exp {αm I(ym (xn ) = tn )} .

(14.26)

Because the term exp(−αm /2) is independent of n, we see that it weights all data
points by the same factor and so can be discarded. Thus we obtain (14.18).
Finally, once all the base classiﬁers are trained, new data points are classiﬁed byevaluating the sign of the combined function deﬁned according to (14.21). Because
the factor of 1/2 does not affect the sign it can be omitted, giving (14.19).

14.3.2 Error functions for boosting
The exponential error function that is minimized by the AdaBoost algorithm
differs from those considered in previous chapters. To gain some insight into thenature of the exponential error function, we ﬁrst consider the expected error givenby
exp{−ty(x)}p(t|x)p(x) dx.
(14.27)
Ex,t [exp{−ty(x)}] =t

Exercise 14.7

If we perform a variational minimization with respect to all possible functions y(x),we obtainp(t = 1|x)1
(14.28)y(x) = ln2p(t = −1|x)662

14. COMBINING MODELS
Figure 14.3

Plot of the exponential (green) andrescaled cross-entropy (red) error
functions along with the hinge error (blue) used in support vector
machines, and the misclassiﬁcation
error (black). Note that for large
negative values of z = ty(x), thecross-entropy gives a linearly increasing penalty, whereas the exponential loss gives an exponentially increasing penalty.

E(z)

−2

Section 7.1.2

Exercise 14.8
Section 4.3.4

Exercise 14.9

−1

0

1

2

z

which is half the log-odds. Thus the AdaBoost algorithm is seeking the best approximation to the log odds ratio, within the space of functions represented by the linear
combination of base classiﬁers, subject to the constrained minimization resulting
from the sequential optimization strategy. This result motivates the use of the signfunction in (14.19) to arrive at the ﬁnal classiﬁcation decision.
We have already seen that the minimizer y(x) of the cross-entropy error (4.90)
for two-class classiﬁcation is given by the posterior class probability. In the caseof a target variable t ∈ {−1, 1}, we have seen that the error function is given byln(1 + exp(−yt)). This is compared with the exponential error function in Figure 14.3, where we have divided the cross-entropy error by a constant factor ln(2)so that it passes through the point (0, 1) for ease of comparison. We see that bothcan be seen as continuous approximations to the ideal misclassiﬁcation error function. An advantage of the exponential error is that its sequential minimization leads
to the simple AdaBoost scheme. One drawback, however, is that it penalizes large
negative values of ty(x) much more strongly than cross-entropy. In particular, we
see that for large negative values of ty, the cross-entropy grows linearly with |ty|,whereas the exponential error function grows exponentially with |ty|. Thus the exponential error function will be much less robust to outliers or misclassiﬁed data
points. Another important difference between cross-entropy and the exponential error function is that the latter cannot be interpreted as the log likelihood function of
any well-deﬁned probabilistic model. Furthermore, the exponential error does not
generalize to classiﬁcation problems having K > 2 classes, again in contrast to thecross-entropy for a probabilistic model, which is easily generalized to give (4.108).
The interpretation of boosting as the sequential optimization of an additive model
under an exponential error (Friedman et al., 2000) opens the door to a wide rangeof boosting-like algorithms, including multiclass extensions, by altering the choiceof error function. It also motivates the extension to regression problems (Friedman,
2001). If we consider a sum-of-squares error function for regression, then sequential
minimization of an additive model of the form (14.21) simply involves ﬁtting eachnew base classiﬁer to the residual errors tn −fm−1 (xn ) from the previous model. As
we have noted, however, the sum-of-squares error is not robust to outliers, and this663

14.4. Tree-based Models
Figure 14.4

Comparison of the squared error(green) with the absolute error (red)showing how the latter places muchless emphasis on large errors andhence is more robust to outliers andmislabelled data points.

E(z)

−1

0

z

1

can be addressed by basing the boosting algorithm on the absolute deviation |y − t|instead. These two error functions are compared in Figure 14.4.

14.4. Tree-based Models
There are various simple, but widely used, models that work by partitioning the
input space into cuboid regions, whose edges are aligned with the axes, and thenassigning a simple model (for example, a constant) to each region. They can beviewed as a model combination method in which only one model is responsible
for making predictions at any given point in input space. The process of selecting
a speciﬁc model, given a new input x, can be described by a sequential decision
making process corresponding to the traversal of a binary tree (one that splits into
two branches at each node). Here we focus on a particular tree-based framework
called classiﬁcation and regression trees, or CART (Breiman et al., 1984), although
there are many other variants going by such names as ID3 and C4.5 (Quinlan, 1986;
Quinlan, 1993).
Figure 14.5 shows an illustration of a recursive binary partitioning of the input
space, along with the corresponding tree structure. In this example, the ﬁrst step
Figure 14.5

Illustration of a two-dimensional in- x2
put space that has been partitioned
into ﬁve regions using axis-aligned
boundaries.

θ3

E
B

C

θ2

D

A
θ1

θ4

x1

664

14. COMBINING MODELS
Figure 14.6

Binary tree corresponding to the partitioning of input space shown in Figure 14.5.

x1 > θ1x2  θ2x2 > θ3x1  θ4

A

Exercise 14.10

B

C

D

E

divides the whole of the input space into two regions according to whether x1  θ1
or x1 > θ1 where θ1 is a parameter of the model. This creates two subregions, eachof which can then be subdivided independently. For instance, the region x1  θ1
is further subdivided according to whether x2  θ2 or x2 > θ2 , giving rise to theregions denoted A and B. The recursive subdivision can be described by the traversalof the binary tree shown in Figure 14.6. For any new input x, we determine which
region it falls into by starting at the top of the tree at the root node and following
a path down to a speciﬁc leaf node according to the decision criteria at each node.
Note that such decision trees are not probabilistic graphical models.
Within each region, there is a separate model to predict the target variable. For
instance, in regression we might simply predict a constant over each region, or inclassiﬁcation we might assign each region to a speciﬁc class. A key property of treebased models, which makes them popular in ﬁelds such as medical diagnosis, forexample, is that they are readily interpretable by humans because they correspondto a sequence of binary decisions applied to the individual input variables. For instance, to predict a patient’s disease, we might ﬁrst ask “is their temperature greaterthan some threshold?”. If the answer is yes, then we might next ask “is their bloodpressure less than some threshold?”. Each leaf of the tree is then associated with a
speciﬁc diagnosis.
In order to learn such a model from a training set, we have to determine thestructure of the tree, including which input variable is chosen at each node to form
the split criterion as well as the value of the threshold parameter θi for the split. We
also have to determine the values of the predictive variable within each region.
Consider ﬁrst a regression problem in which the goal is to predict a single target
variable t from a D-dimensional vector x = (x1 , . . . , xD )T of input variables. The
training data consists of input vectors {x1 , . . . , xN } along with the corresponding
continuous labels {t1 , . . . , tN }. If the partitioning of the input space is given, and we
minimize the sum-of-squares error function, then the optimal value of the predictive
variable within any given region is just given by the average of the values of tn for
those data points that fall in that region.
Now consider how to determine the structure of the decision tree. Even for aﬁxed number of nodes in the tree, the problem of determining the optimal structure(including choice of input variable for each split as well as the corresponding thresh-14.4. Tree-based Models

665olds) to minimize the sum-of-squares error is usually computationally infeasible dueto the combinatorially large number of possible solutions. Instead, a greedy optimization is generally done by starting with a single root node, corresponding to thewhole input space, and then growing the tree by adding nodes one at a time. At eachstep there will be some number of candidate regions in input space that can be split,
corresponding to the addition of a pair of leaf nodes to the existing tree. For eachof these, there is a choice of which of the D input variables to split, as well as thevalue of the threshold. The joint optimization of the choice of region to split, and the
choice of input variable and threshold, can be done efﬁciently by exhaustive searchnoting that, for a given choice of split variable and threshold, the optimal choice ofpredictive variable is given by the local average of the data, as noted earlier. Thisis repeated for all possible choices of variable to be split, and the one that gives thesmallest residual sum-of-squares error is retained.
Given a greedy strategy for growing the tree, there remains the issue of whento stop adding nodes. A simple approach would be to stop when the reduction inresidual error falls below some threshold. However, it is found empirically that often
none of the available splits produces a signiﬁcant reduction in error, and yet after
several more splits a substantial error reduction is found. For this reason, it is common practice to grow a large tree, using a stopping criterion based on the numberof data points associated with the leaf nodes, and then prune back the resulting tree.
The pruning is based on a criterion that balances residual error against a measure ofmodel complexity. If we denote the starting tree for pruning by T0 , then we deﬁne
T ⊂ T0 to be a subtree of T0 if it can be obtained by pruning nodes from T0 (inother words, by collapsing internal nodes by combining the corresponding regions).
Suppose the leaf nodes are indexed by τ = 1, . . . , |T |, with leaf node τ representinga region Rτ of input space having Nτ data points, and |T | denoting the total numberof leaf nodes. The optimal prediction for region Rτ is then given by
1 
yτ =
tn
(14.29)Nτ
xn ∈Rτ

and the corresponding contribution to the residual sum-of-squares is then2Qτ (T ) ={tn − yτ } .

(14.30)xn ∈Rτ

The pruning criterion is then given byC(T ) =|T |
Qτ (T ) + λ|T |(14.31)

τ =1

The regularization parameter λ determines the trade-off between the overall residual
sum-of-squares error and the complexity of the model as measured by the number|T | of leaf nodes, and its value is chosen by cross-validation.
For classiﬁcation problems, the process of growing and pruning the tree is similar, except that the sum-of-squares error is replaced by a more appropriate measure

666

14. COMBINING MODELS
of performance. If we deﬁne pτ k to be the proportion of data points in region Rτ
assigned to class k, where k = 1, . . . , K, then two commonly used choices are thecross-entropyKpτ k ln pτ k
(14.32)Qτ (T ) =k=1

and the Gini index
Qτ (T ) =Kpτ k (1 − pτ k ) .

(14.33)

k=1

Exercise 14.11

These both vanish for pτ k = 0 and pτ k = 1 and have a maximum at pτ k = 0.5. They
encourage the formation of regions in which a high proportion of the data points are
assigned to one class. The cross entropy and the Gini index are better measures thanthe misclassiﬁcation rate for growing the tree because they are more sensitive to thenode probabilities. Also, unlike misclassiﬁcation rate, they are differentiable andhence better suited to gradient based optimization methods. For subsequent pruningof the tree, the misclassiﬁcation rate is generally used.
The human interpretability of a tree model such as CART is often seen as its
major strength. However, in practice it is found that the particular tree structure thatis learned is very sensitive to the details of the data set, so that a small change to thetraining data can result in a very different set of splits (Hastie et al., 2001).
There are other problems with tree-based methods of the kind considered inthis section. One is that the splits are aligned with the axes of the feature space,which may be very suboptimal. For instance, to separate two classes whose optimal
decision boundary runs at 45 degrees to the axes would need a large number ofaxis-parallel splits of the input space as compared to a single non-axis-aligned split.
Furthermore, the splits in a decision tree are hard, so that each region of input spaceis associated with one, and only one, leaf node model. The last issue is particularly
problematic in regression where we are typically aiming to model smooth functions,and yet the tree model produces piecewise-constant predictions with discontinuities
at the split boundaries.

14.5. Conditional Mixture Models

Chapter 9

We have seen that standard decision trees are restricted by hard, axis-aligned splits of
the input space. These constraints can be relaxed, at the expense of interpretability,
by allowing soft, probabilistic splits that can be functions of all of the input variables,
not just one of them at a time. If we also give the leaf models a probabilistic interpretation, we arrive at a fully probabilistic tree-based model called the hierarchicalmixture of experts, which we consider in Section 14.5.3.
An alternative way to motivate the hierarchical mixture of experts model is tostart with a standard probabilistic mixtures of unconditional density models such as
Gaussians and replace the component densities with conditional distributions. Herewe consider mixtures of linear regression models (Section 14.5.1) and mixtures of14.5. Conditional Mixture Models

667

logistic regression models (Section 14.5.2). In the simplest case, the mixing coefﬁcients are independent of the input variables. If we make a further generalization toallow the mixing coefﬁcients also to depend on the inputs then we obtain a mixtureof experts model. Finally, if we allow each component in the mixture model to beitself a mixture of experts model, then we obtain a hierarchical mixture of experts.

14.5.1 Mixtures of linear regression models

Exercise 14.12

One of the many advantages of giving a probabilistic interpretation to the linear regression model is that it can then be used as a component in more complex
probabilistic models. This can be done, for instance, by viewing the conditional
distribution representing the linear regression model as a node in a directed probabilistic graph. Here we consider a simple example corresponding to a mixture oflinear regression models, which represents a straightforward extension of the Gaussian mixture model discussed in Section 9.2 to the case of conditional Gaussian
distributions.
We therefore consider K linear regression models, each governed by its own
weight parameter wk . In many applications, it will be appropriate to use a common
noise variance, governed by a precision parameter β, for all K components, and thisis the case we consider here. We will once again restrict attention to a single target
variable t, though the extension to multiple outputs is straightforward. If we denotethe mixing coefﬁcients by πk , then the mixture distribution can be writtenp(t|θ) =Kπk N (t|wkT φ, β −1 )(14.34)k=1

where θ denotes the set of all adaptive parameters in the model, namely W = {wk },π = {πk }, and β. The log likelihood function for this model, given a data set ofobservations {φn , tn }, then takes the form
K

N


ln
πk N (tn |wkT φn , β −1 )(14.35)ln p(t|θ) =
n=1

k=1

T

Exercise 14.13

where t = (t1 , . . . , tN ) denotes the vector of target variables.
In order to maximize this likelihood function, we can once again appeal to the
EM algorithm, which will turn out to be a simple extension of the EM algorithm forunconditional Gaussian mixtures of Section 9.2. We can therefore build on our experience with the unconditional mixture and introduce a set Z = {zn } of binary latentvariables where znk ∈ {0, 1} in which, for each data point n, all of the elements
k = 1, . . . , K are zero except for a single value of 1 indicating which componentof the mixture was responsible for generating that data point. The joint distribution
over latent and observed variables can be represented by the graphical model shownin Figure 14.7.
The complete-data log likelihood function then takes the formln p(t, Z|θ) =N K
n=1 k=1

znk ln πk N (tn |wkT φn , β −1 ) .

(14.36)

668

14. COMBINING MODELS
Figure 14.7

Probabilistic directed graph representing a mixture oflinear regression models, deﬁned by (14.35).

zn

π

φn

β
tn

W

N

The EM algorithm begins by ﬁrst choosing an initial value θ old for the model parameters. In the E step, these parameter values are then used to evaluate the posterior
probabilities, or responsibilities, of each component k for every data point n givenbyπk N (tn |wkT φn , β −1 )γnk = E[znk ] = p(k|φn , θ old ) = .
(14.37)T−1 )j πj N (tn |wj φn , βThe responsibilities are then used to determine the expectation, with respect to theposterior distribution p(Z|t, θ old ), of the complete-data log likelihood, which takes
the form
Q(θ, θ

old) = EZ [ln p(t, Z|θ)] =

N 
K




γnk ln πk + ln N (tn |wkT φn , β −1 ) .

n=1 k=1

Exercise 14.14

In the M step, we maximize the function Q(θ, θ old ) with respect to θ, keeping theγnk ﬁxed. For the optimization with respect to the mixing coefﬁcients πk we needto take account of the constraint k πk = 1, which can be done with the aid of a
Lagrange multiplier, leading to an M-step re-estimation equation for πk in the form
1 
γnk .
N
N

πk =

(14.38)n=1Note that this has exactly the same form as the corresponding result for a simple
mixture of unconditional Gaussians given by (9.22).
Next consider the maximization with respect to the parameter vector wk of thek th linear regression model. Substituting for the Gaussian distribution, we see thatthe function Q(θ, θ old ), as a function of the parameter vector wk , takes the form
Q(θ, θ

old

)=

N

n=1

γnk

−

β
2
tn − wkT φn
2+ const(14.39)where the constant term includes the contributions from other weight vectors wj for
j = k. Note that the quantity we are maximizing is similar to the (negative of the)standard sum-of-squares error (3.12) for a single linear regression model, but with
the inclusion of the responsibilities γnk . This represents a weighted least squares

14.5. Conditional Mixture Models

669

problem, in which the term corresponding to the nth data point carries a weighting
coefﬁcient given by βγnk , which could be interpreted as an effective precision foreach data point. We see that each component linear regression model in the mixture,
governed by its own parameter vector wk , is ﬁtted separately to the whole data set in
the M step, but with each data point n weighted by the responsibility γnk that modelk takes for that data point. Setting the derivative of (14.39) with respect to wk equal
to zero givesN0=
γnk tn − wkT φn φn(14.40)n=1which we can write in matrix notation as
0 = ΦT Rk (t − Φwk )(14.41)

where Rk = diag(γnk ) is a diagonal matrix of size N × N . Solving for wk , weobtain−1 T
wk = ΦT Rk Φ
Φ Rk t.
(14.42)This represents a set of modiﬁed normal equations corresponding to the weighted
least squares problem, of the same form as (4.99) found in the context of logistic
regression. Note that after each E step, the matrix Rk will change and so we will
have to solve the normal equations afresh in the subsequent M step.
Finally, we maximize Q(θ, θ old ) with respect to β. Keeping only terms that
depend on β, the function Q(θ, θ old ) can be writtenQ(θ, θ old ) =

N 
K


γnk

n=1 k=11β2tn − wkT φn.
ln β −
2
2(14.43)

Setting the derivative with respect to β equal to zero, and rearranging, we obtain the
M-step equation for β in the form
1
1 
2
=
γnk tn − wkT φn .
β
N
N

K

(14.44)n=1 k=1

In Figure 14.8, we illustrate this EM algorithm using the simple example ofﬁtting a mixture of two straight lines to a data set having one input variable x andone target variable t. The predictive density (14.34) is plotted in Figure 14.9 using
the converged parameter values obtained from the EM algorithm, corresponding to
the right-hand plot in Figure 14.8. Also shown in this ﬁgure is the result of ﬁtting
a single linear regression model, which gives a unimodal predictive density. We seethat the mixture model gives a much better representation of the data distribution,and this is reﬂected in the higher likelihood value. However, the mixture model
also assigns signiﬁcant probability mass to regions where there is no data because its
predictive distribution is bimodal for all values of x. This problem can be resolved byextending the model to allow the mixture coefﬁcients themselves to be functions ofx, leading to models such as the mixture density networks discussed in Section 5.6,
and hierarchical mixture of experts discussed in Section 14.5.3.

670

14. COMBINING MODELS

1.5
1
0.5
0
−0.5
−1
−1.5

1.5
1
0.5
0
−0.5
−1
−1.5
−1

−0.5

0

0.5

1

1.5
1
0.5
0
−0.5
−1
−1.5
−1

−0.5

0

0.5

1

1

1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

−1

−0.5

0

0.5

1

0

−1

−0.5

0

0.5

1

0

−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

1

Figure 14.8 Example of a synthetic data set, shown by the green points, having one input variable x and one
target variable t, together with a mixture of two linear regression models whose mean functions y(x, wk ), wherek ∈ {1, 2}, are shown by the blue and red lines. The upper three plots show the initial conﬁguration (left), theresult of running 30 iterations of EM (centre), and the result after 50 iterations of EM (right). Here β was initialized
to the reciprocal of the true variance of the set of target values. The lower three plots show the corresponding
responsibilities plotted as a vertical line for each data point in which the length of the blue segment gives theposterior probability of the blue line for that data point (and similarly for the red segment).

14.5.2 Mixtures of logistic models
Because the logistic regression model deﬁnes a conditional distribution for thetarget variable, given the input vector, it is straightforward to use it as the component
distribution in a mixture model, thereby giving rise to a richer family of conditional
distributions compared to a single logistic regression model. This example involves
a straightforward combination of ideas encountered in earlier sections of the bookand will help consolidate these for the reader.
The conditional distribution of the target variable, for a probabilistic mixture of
K logistic regression models, is given byp(t|φ, θ) =

K


1−t

πk ykt [1 − yk ](14.45)k=1

where φ is the feature vector, yk = σ wkT φ is the output of component k, and θ
denotes the adjustable parameters namely {πk } and {wk }.
Now suppose we are given a data set {φn , tn }. The corresponding likelihood

14.5. Conditional Mixture Models

671

Figure 14.9 The left plot shows the predictive conditional density corresponding to the converged solution in
Figure 14.8. This gives a log likelihood value of −3.0. A vertical slice through one of these plots at a particularvalue of x represents the corresponding conditional distribution p(t|x), which we see is bimodal. The plot on theright shows the predictive density for a single linear regression model ﬁtted to the same data set using maximum
likelihood. This model has a smaller log likelihood of −27.6.

function is then given byp(t|θ) =

N


K


n=1

k=1


1−t
tn
πk ynk[1 − ynk ] n(14.46)

where ynk = σ(wkT φn ) and t = (t1 , . . . , tN )T . We can maximize this likelihood
function iteratively by making use of the EM algorithm. This involves introducing
latent variables znk that correspond to a 1-of-K coded binary indicator variable foreach data point n. The complete-data likelihood function is then given byp(t, Z|θ) =

N 
K


n=1 k=1

1−tn znk

tn
πk ynk[1 − ynk ](14.47)

where Z is the matrix of latent variables with elements znk . We initialize the EM
algorithm by choosing an initial value θ old for the model parameters. In the E step,
we then use these parameter values to evaluate the posterior probabilities of the components k for each data point n, which are given by
1−t

πk y tn [1 − ynk ] n
γnk = E[znk ] = p(k|φn , θ old ) =  nktn
1−tn .
j πj ynj [1 − ynj ](14.48)

These responsibilities are then used to ﬁnd the expected complete-data log likelihood
as a function of θ, given byQ(θ, θ old ) = EZ [ln p(t, Z|θ)]
=

N 
K

n=1 k=1

γnk {ln πk + tn ln ynk + (1 − tn ) ln (1 − ynk )} . (14.49)

672

14. COMBINING MODELSThe M step involves maximization of this function with respect to θ, keeping θ old ,and hence γnk , ﬁxed. Maximization with respect to πk can be done
 in the usual way,with a Lagrange multiplier to enforce the summation constraint k πk = 1, giving
the familiar result
N
1 
γnk .
(14.50)πk =N
n=1Section 4.3.3

To determine the {wk }, we note that the Q(θ, θ old ) function comprises a sum
over terms indexed by k each of which depends only on one of the vectors wk , sothat the different vectors are decoupled in the M step of the EM algorithm. In otherwords, the different components interact only via the responsibilities, which are ﬁxed
during the M step. Note that the M step does not have a closed-form solution andmust be solved iteratively using, for instance, the iterative reweighted least squares(IRLS) algorithm. The gradient and the Hessian for the vector wk are given by∇k Q =N

γnk (tn − ynk )φn(14.51)n=1

Hk= −∇k ∇k Q =Nγnk ynk (1 − ynk )φn φTn(14.52)n=1Section 4.3.3

Exercise 14.16

where ∇k denotes the gradient with respect to wk . For ﬁxed γnk , these are independent of {wj } for j = k and so we can solve for each wk separately using the IRLS
algorithm. Thus the M-step equations for component k correspond simply to ﬁtting
a single logistic regression model to a weighted data set in which data point n carries
a weight γnk . Figure 14.10 shows an example of the mixture of logistic regression
models applied to a simple classiﬁcation problem. The extension of this model to a
mixture of softmax models for more than two classes is straightforward.

14.5.3 Mixtures of experts
In Section 14.5.1, we considered a mixture of linear regression models, and in
Section 14.5.2 we discussed the analogous mixture of linear classiﬁers. Althoughthese simple mixtures extend the ﬂexibility of linear models to include more complex (e.g., multimodal) predictive distributions, they are still very limited. We canfurther increase the capability of such models by allowing the mixing coefﬁcientsthemselves to be functions of the input variable, so thatp(t|x) =Kπk (x)pk (t|x).

(14.53)k=1

This is known as a mixture of experts model (Jacobs et al., 1991) in which the mixing coefﬁcients πk (x) are known as gating functions and the individual component
densities pk (t|x) are called experts. The notion behind the terminology is that different components can model the distribution in different regions of input space (they14.5. Conditional Mixture Models

673

Figure 14.10 Illustration of a mixture of logistic regression models. The left plot shows data points drawn
from two classes denoted red and blue, in which the background colour (which varies from pure red to pure blue)
denotes the true probability of the class label. The centre plot shows the result of ﬁtting a single logistic regression
model using maximum likelihood, in which the background colour denotes the corresponding probability of theclass label. Because the colour is a near-uniform purple, we see that the model assigns a probability of around
0.5 to each of the classes over most of input space. The right plot shows the result of ﬁtting a mixture of two
logistic regression models, which now gives much higher probability to the correct labels for many of the points
in the blue class.

Exercise 14.17

Section 4.3.3

are ‘experts’ at making predictions in their own regions), and the gating functions
determine which components are dominant in which region.
The gating functions πk (x) must satisfy
 the usual constraints for mixing coefﬁcients, namely 0  πk (x)  1 and k πk (x) = 1. They can therefore be
represented, for example, by linear softmax models of the form (4.104) and (4.105).
If the experts are also linear (regression or classiﬁcation) models, then the wholemodel can be ﬁtted efﬁciently using the EM algorithm, with iterative reweighted
least squares being employed in the M step (Jordan and Jacobs, 1994).
Such a model still has signiﬁcant limitations due to the use of linear models
for the gating and expert functions. A much more ﬂexible model is obtained byusing a multilevel gating function to give the hierarchical mixture of experts, or
HME model (Jordan and Jacobs, 1994). To understand the structure of this model,
imagine a mixture distribution in which each component in the mixture is itself amixture distribution. For simple unconditional mixtures, this hierarchical mixture istrivially equivalent to a single ﬂat mixture distribution. However, when the mixing
coefﬁcients are input dependent, this hierarchical model becomes nontrivial. The
HME model can also be viewed as a probabilistic version of decision trees discussed
in Section 14.4 and can again be trained efﬁciently by maximum likelihood using an
EM algorithm with IRLS in the M step. A Bayesian treatment of the HME has been
given by Bishop and Svensén (2003) based on variational inference.
We shall not discuss the HME in detail here. However, it is worth pointing out
the close connection with the mixture density network discussed in Section 5.6. The
principal advantage of the mixtures of experts model is that it can be optimized by
EM in which the M step for each mixture component and gating model involves
a convex optimization (although the overall optimization is nonconvex). By contrast, the advantage of the mixture density network approach is that the component

674

14. COMBINING MODELS
densities and the mixing coefﬁcients share the hidden units of the neural network.
Furthermore, in the mixture density network, the splits of the input space are further
relaxed compared to the hierarchical mixture of experts in that they are not only soft,and not constrained to be axis aligned, but they can also be nonlinear.

Exercises

14.1 ( ) www Consider a set models of the form p(t|x, zh , θ h , h) in which x is the
input vector, t is the target vector, h indexes the different models, zh is a latent variable for model h, and θ h is the set of parameters for model h. Suppose the models
have prior probabilities p(h) and that we are given a training set X = {x1 , . . . , xN }and T = {t1 , . . . , tN }. Write down the formulae needed to evaluate the predictive distribution p(t|x, X, T) in which the latent variables and the model index are
marginalized out. Use these formulae to highlight the difference between Bayesian
averaging of different models and the use of latent variables within a single model.
14.2 ( ) The expected sum-of-squares error EAV for a simple committee model canbe deﬁned by (14.10), and the expected error of the committee itself is given by(14.11). Assuming that the individual errors satisfy (14.12) and (14.13), derive theresult (14.14).
14.3 ( ) www By making use of Jensen’s inequality (1.115), for the special case of
the convex function f (x) = x2 , show that the average expected sum-of-squares
error EAV of the members of a simple committee model, given by (14.10), and the
expected error ECOM of the committee itself, given by (14.11), satisfy
ECOM  EAV .

(14.54)14.4 ( ) By making use of Jensen’s in equality (1.115), show that the result (14.54)
derived in the previous exercise hods for any error function E(y), not just sum-ofsquares, provided it is a convex function of y.
14.5 ( ) www Consider a committee in which we allow unequal weighting of theconstituent models, so thatyCOM (x) =Mαm ym (x).

(14.55)m=1

In order to ensure that the predictions yCOM (x) remain within sensible limits, suppose that we require that they be bounded at each value of x by the minimum andmaximum values given by any of the members of the committee, so thatymin (x)  yCOM (x)  ymax (x).

(14.56)Show that a necessary and sufﬁcient condition for this constraint is that the coefﬁcients αm satisfyMαm  0,αm = 1.
(14.57)m=1

Exercises

675

14.6 ( ) www By differentiating the error function (14.23) with respect to αm , showthat the parameters αm in the AdaBoost algorithm are updated using (14.17) inwhich m is deﬁned by (14.16).
14.7 ( ) By making a variational minimization of the expected exponential error function
given by (14.27) with respect to all possible functions y(x), show that the minimizing
function is given by (14.28).
14.8 ( ) Show that the exponential error function (14.20), which is minimized by the
AdaBoost algorithm, does not correspond to the log likelihood of any well-behaved
probabilistic model. This can be done by showing that the corresponding conditional
distribution p(t|x) cannot be correctly normalized.
14.9 ( ) www Show that the sequential minimization of the sum-of-squares error function for an additive model of the form (14.21) in the style of boosting simply involves
ﬁtting each new base classiﬁer to the residual errors tn −fm−1 (xn ) from the previous
model.
14.10 ( ) Verify that if we minimize the sum-of-squares error between a set of training
values {tn } and a single predictive value t, then the optimal solution for t is givenby the mean of the {tn }.
14.11 ( ) Consider a data set comprising 400 data points from class C1 and 400 data
points from class C2 . Suppose that a tree model A splits these into (300, 100) at
the ﬁrst leaf node and (100, 300) at the second leaf node, where (n, m) denotes thatn points are assigned to C1 and m points are assigned to C2 . Similarly, supposethat a second tree model B splits them into (200, 400) and (200, 0). Evaluate themisclassiﬁcation rates for the two trees and hence show that they are equal. Similarly,
evaluate the cross-entropy (14.32) and Gini index (14.33) for the two trees and showthat they are both lower for tree B than for tree A.
14.12 ( ) Extend the results of Section 14.5.1 for a mixture of linear regression models
to the case of multiple target values described by a vector t. To do this, make use of
the results of Section 3.1.5.
14.13 ( ) www Verify that the complete-data log likelihood function for the mixture oflinear regression models is given by (14.36).
14.14 ( ) Use the technique of Lagrange multipliers (Appendix E) to show that the M-step
re-estimation equation for the mixing coefﬁcients in the mixture of linear regression
models trained by maximum likelihood EM is given by (14.38).
14.15 ( ) www We have already noted that if we use a squared loss function in a regression problem, the corresponding optimal prediction of the target variable for a new
input vector is given by the conditional mean of the predictive distribution. Show
that the conditional mean for the mixture of linear regression models discussed in
Section 14.5.1 is given by a linear combination of the means of each component distribution. Note that if the conditional distribution of the target data is multimodal,
the conditional mean can give poor predictions.

676

14. COMBINING MODELS
14.16 (
) Extend the logistic regression mixture model of Section 14.5.2 to a mixtureof softmax classiﬁers representing C  2 classes. Write down the EM algorithm fordetermining the parameters of this model through maximum likelihood.
14.17 ( ) www
form

Consider a mixture model for a conditional distribution p(t|x) of thep(t|x) =Kπk ψk (t|x)(14.58)k=1

in which each mixture component ψk (t|x) is itself a mixture model. Show that thistwo-level hierarchical mixture is equivalent to a conventional single-level mixture
model. Now suppose that the mixing coefﬁcients in both levels of such a hierarchical model are arbitrary functions of x. Again, show that this hierarchical model
is again equivalent to a single-level model with x-dependent mixing coefﬁcients.
Finally, consider the case in which the mixing coefﬁcients at both levels of the hierarchical mixture are constrained to be linear classiﬁcation (logistic or softmax)
models. Show that the hierarchical mixture cannot in general be represented by asingle-level mixture having linear classiﬁcation models for the mixing coefﬁcients.
Hint: to do this it is sufﬁcient to construct a single counter-example, so consider amixture of two components in which one of those components is itself a mixture oftwo components, with mixing coefﬁcients given by linear-logistic models. Show that
this cannot be represented by a single-level mixture of 3 components having mixing
coefﬁcients determined by a linear-softmax model.

Appendix A. Data Sets
In this appendix, we give a brief introduction to the data sets used to illustrate someof the algorithms described in this book. Detailed information on ﬁle formats forthese data sets, as well as the data ﬁles themselves, can be obtained from the book
web site:

http://research.microsoft.com/∼cmbishop/PRML

Handwritten Digits
The digits data used in this book is taken from the MNIST data set (LeCun et al.,1998), which itself was constructed by modifying a subset of the much larger data
set produced by NIST (the National Institute of Standards and Technology). It comprises a training set of 60, 000 examples and a test set of 10, 000 examples. Someof the data was collected from Census Bureau employees and the rest was collectedfrom high-school children, and care was taken to ensure that the test examples were
written by different individuals to the training examples.
The original NIST data had binary (black or white) pixels. To create MNIST,
these images were size normalized to ﬁt in a 20 × 20 pixel box while preserving theiraspect ratio. As a consequence of the anti-aliasing used to change the resolution of
the images, the resulting MNIST digits are grey scale. These images were thencentred in a 28 × 28 box. Examples of the MNIST digits are shown in Figure A.1.
Error rates for classifying the digits range from 12% for a simple linear classiﬁer, through 0.56% for a carefully designed support vector machine, to 0.4% for a
convolutional neural network (LeCun et al., 1998).

677

678

A. DATA SETS
Figure A.1

One hundred examples of the
MNIST digits chosen at random from the training set.

Oil FlowThis is a synthetic data set that arose out of a project aimed at measuring noninvasively the proportions of oil, water, and gas in North Sea oil transfer pipelines(Bishop and James, 1993). It is based on the principle of dual-energy gamma densitometry. The ideas is that if a narrow beam of gamma rays is passed through the pipe,
the attenuation in the intensity of the beam provides information about the density ofmaterial along its path. Thus, for instance, the beam will be attenuated more strongly
by oil than by gas.
A single attenuation measurement alone is not sufﬁcient because there are twodegrees of freedom corresponding to the fraction of oil and the fraction of water (thefraction of gas is redundant because the three fractions must add to one). To address
this, two gamma beams of different energies (in other words different frequencies or
wavelengths) are passed through the pipe along the same path, and the attenuation ofeach is measured. Because the absorbtion properties of different materials vary differently as a function of energy, measurement of the attenuations at the two energies
provides two independent pieces of information. Given the known absorbtion properties of oil, water, and gas at the two energies, it is then a simple matter to calculate
the average fractions of oil and water (and hence of gas) measured along the path of
the gamma beams.
There is a further complication, however, associated with the motion of the materials along the pipe. If the ﬂow velocity is small, then the oil ﬂoats on top of thewater with the gas sitting above the oil. This is known as a laminar or stratiﬁed

679

A. DATA SETS
Figure A.2

The three geometrical conﬁgurations of the oil,water, and gas phases used to generate the oilﬂow data set. For each conﬁguration, the proportions of the three phases can vary.

Stratiﬁed

Annular
Oil
Water
Gas
Mix

Homogeneous

ﬂow conﬁguration and is illustrated in Figure A.2. As the ﬂow velocity is increased,
more complex geometrical conﬁgurations of the oil, water, and gas can arise. For the
purposes of this data set, two speciﬁc idealizations are considered. In the annularconﬁguration the oil, water, and gas form concentric cylinders with the water aroundthe outside and the gas in the centre, whereas in the homogeneous conﬁguration theoil, water and gas are assumed to be intimately mixed as might occur at high ﬂow
velocities under turbulent conditions. These conﬁgurations are also illustrated in
Figure A.2.
We have seen that a single dual-energy beam gives the oil and water fractions
measured along the path length, whereas we are interested in the volume fractions ofoil and water. This can be addressed by using multiple dual-energy gamma densitometers whose beams pass through different regions of the pipe. For this particular
data set, there are six such beams, and their spatial arrangement is shown in Figure A.3. A single observation is therefore represented by a 12-dimensional vector
comprising the fractions of oil and water measured along the paths of each of thebeams. We are, however, interested in obtaining the overall volume fractions of thethree phases in the pipe. This is much like the classical problem of tomographic reconstruction, used in medical imaging for example, in which a two-dimensional disFigure A.3

Cross section of the pipe showing the arrangement of thesix beam lines, each of which comprises a single dualenergy gamma densitometer. Note that the vertical beams
are asymmetrically arranged relative to the central axis
(shown by the dotted line).

680

A. DATA SETS
tribution is to be reconstructed from an number of one-dimensional averages. Here
there are far fewer line measurements than in a typical tomography application. On
the other hand the range of geometrical conﬁgurations is much more limited, and so
the conﬁguration, as well as the phase fractions, can be predicted with reasonableaccuracy from the densitometer data.
For safety reasons, the intensity of the gamma beams is kept relatively weak andso to obtain an accurate measurement of the attenuation, the measured beam intensity
is integrated over a speciﬁc time interval. For a ﬁnite integration time, there are
random ﬂuctuations in the measured intensity due to the fact that the gamma beams
comprise discrete packets of energy called photons. In practice, the integration time
is chosen as a compromise between reducing the noise level (which requires a longintegration time) and detecting temporal variations in the ﬂow (which requires a short
integration time). The oil ﬂow data set is generated using realistic known values forthe absorption properties of oil, water, and gas at the two gamma energies used, andwith a speciﬁc choice of integration time (10 seconds) chosen as characteristic of atypical practical setup.
Each point in the data set is generated independently using the following steps:
1. Choose one of the three phase conﬁgurations at random with equal probability.
2. Choose three random numbers f1 , f2 and f3 from the uniform distribution over(0, 1) and deﬁne
foil =

f1,
f1 + f2 + f3

fwater =f2.
f1 + f2 + f3

(A.1)

This treats the three phases on an equal footing and ensures that the volume
fractions add to one.
3. For each of the six beam lines, calculate the effective path lengths through oil
and water for the given phase conﬁguration.
4. Perturb the path lengths using the Poisson distribution based on the known
beam intensities and integration time to allow for the effect of photon statistics.
Each point in the data set comprises the 12 path length measurements, together
with the fractions of oil and water and a binary label describing the phase conﬁguration. The data set is divided into training, validation, and test sets, each of which
comprises 1, 000 independent data points. Details of the data format are available
from the book web site.
In Bishop and James (1993), statistical machine learning techniques were used
to predict the volume fractions and also the geometrical conﬁguration of the phases
shown in Figure A.2, from the 12-dimensional vector of measurements. The 12dimensional observation vectors can also be used to test data visualization algorithms.
This data set has a rich and interesting structure, as follows. For any given
conﬁguration there are two degrees of freedom corresponding to the fractions of

A. DATA SETS

681

oil and water, and so for inﬁnite integration time the data will locally live on a twodimensional manifold. For a ﬁnite integration time, the individual data points will be
perturbed away from the manifold by the photon noise. In the homogeneous phase
conﬁguration, the path lengths in oil and water are linearly related to the fractions ofoil and water, and so the data points lie close to a linear manifold. For the annularconﬁguration, the relationship between phase fraction and path length is nonlinear
and so the manifold will be nonlinear. In the case of the laminar conﬁguration thesituation is even more complex because small variations in the phase fractions cancause one of the horizontal phase boundaries to move across one of the horizontal
beam lines leading to a discontinuous jump in the 12-dimensional observation space.
In this way, the two-dimensional nonlinear manifold for the laminar conﬁguration isbroken into six distinct segments. Note also that some of the manifolds for different
phase conﬁgurations meet at speciﬁc points, for example if the pipe is ﬁlled entirely
with oil, it corresponds to speciﬁc instances of the laminar, annular, and homogeneous conﬁgurations.

Old Faithful
Old Faithful, shown in Figure A.4, is a hydrothermal geyser in Yellowstone National
Park in the state of Wyoming, U.S.A., and is a popular tourist attraction. Its namestems from the supposed regularity of its eruptions.
The data set comprises 272 observations, each of which represents a single eruption and contains two variables corresponding to the duration in minutes of the eruption, and the time until the next eruption, also in minutes. Figure A.5 shows a plot of
the time to the next eruption versus the duration of the eruptions. It can be seen thatthe time to the next eruption varies considerably, although knowledge of the durationof the current eruption allows it to be predicted more accurately. Note that there exist
several other data sets relating to the eruptions of Old Faithful.
Figure A.4

The Old Faithful geyser
in Yellowstone National
Park. c Bruce T. Gourleywww.brucegourley.com.

682

A. DATA SETS
Figure A.5

Plot of the time to the next eruption
100in minutes (vertical axis) versus theduration of the eruption in minutes(horizontal axis) for the Old Faithful 90
data set.

80
70
60
50
40

1

2

3

4

5

6

Synthetic Data
Throughout the book, we use two simple synthetic data sets to illustrate many of thealgorithms. The ﬁrst of these is a regression problem, based on the sinusoidal function, shown in Figure A.6. The input values {xn } are generated uniformly in range(0, 1), and the corresponding target values {tn } are obtained by ﬁrst computing thecorresponding values of the function sin(2πx), and then adding random noise witha Gaussian distribution having standard deviation 0.3. Various forms of this data set,
having different numbers of data points, are used in the book.
The second data set is a classiﬁcation problem having two classes, with equal
prior probabilities, and is shown in Figure A.7. The blue class is generated from asingle Gaussian while the red class comes from a mixture of two Gaussians. Because we know the class priors and the class-conditional densities, it is straightforward to evaluate and plot the true posterior probabilities as well as the minimum
misclassiﬁcation-rate decision boundary, as shown in Figure A.7.

683

A. DATA SETS

1

1

t

t
0

0

−1

−1

0

1

x

0

x

1

Figure A.6 The left-hand plot shows the synthetic regression data set along with the underlying sinusoidal
function from which the data points were generated. The right-hand plot shows the true conditional distributionp(t|x) from which the labels are generated, in which the green curve denotes the mean, and the shaded regionspans one standard deviation on each side of the mean.

2

0

−2
−2

0

2

Figure A.7 The left plot shows the synthetic classiﬁcation data set with data from the two classes shown in
red and blue. On the right is a plot of the true posterior probabilities, shown on a colour scale going from pure
red denoting probability of the red class is 1 to pure blue denoting probability of the red class is 0. Because
these probabilities are known, the optimal decision boundary for minimizing the misclassiﬁcation rate (whichcorresponds to the contour along which the posterior probabilities for each class equal 0.5) can be evaluatedand is shown by the green curve. This decision boundary is also plotted on the left-hand ﬁgure.

Appendix B. Probability Distributions
In this appendix, we summarize the main properties of some of the most widely usedprobability distributions, and for each distribution we list some key statistics such as
the expectation E[x], the variance (or covariance), the mode, and the entropy H[x].
All of these distributions are members of the exponential family and are widely usedas building blocks for more sophisticated probabilistic models.

BernoulliThis is the distribution for a single binary variable x ∈ {0, 1} representing, forexample, the result of ﬂipping a coin. It is governed by a single continuous parameterµ ∈ [0, 1] that represents the probability of x = 1.
Bern(x|µ) = µx (1 − µ)1−x
E[x] = µvar[x] = µ(1 − µ)

(B.1)
(B.2)
(B.3)

1 if µ  0.5,
0 otherwise(B.4)

H[x] = −µ ln µ − (1 − µ) ln(1 − µ).

(B.5)

mode[x] =The Bernoulli is a special case of the binomial distribution for the case of a single
observation. Its conjugate prior for µ is the beta distribution.

685

686

B. PROBABILITY DISTRIBUTIONS

BetaThis is a distribution over a continuous variable µ ∈ [0, 1], which is often used torepresent the probability for some binary event. It is governed by two parameters aand b that are constrained by a > 0 and b > 0 to ensure that the distribution can benormalized.
Beta(µ|a, b) =E[µ] =
var[µ] =mode[µ] =Γ(a + b) a−1µ (1 − µ)b−1
Γ(a)Γ(b)aa+bab2(a + b) (a + b + 1)
a−1
.
a+b−2

(B.6)
(B.7)
(B.8)
(B.9)

The beta is the conjugate prior for the Bernoulli distribution, for which a and b can
be interpreted as the effective prior number of observations of x = 1 and x = 0,
respectively. Its density is ﬁnite if a  1 and b  1, otherwise there is a singularity
at µ = 0 and/or µ = 1. For a = b = 1, it reduces to a uniform distribution. The beta
distribution is a special case of the K-state Dirichlet distribution for K = 2.

Binomial
The binomial distribution gives the probability of observing m occurrences of x = 1
in a set of N samples from a Bernoulli distribution, where the probability of observing x = 1 is µ ∈ [0, 1].
 N mµ (1 − µ)N −m(B.10)
Bin(m|N, µ) =m
E[m] = N µ(B.11)
var[m] = N µ(1 − µ)(B.12)
mode[m] = (N + 1)µ
(B.13)
where (N + 1)µ denotes the largest integer that is less than or equal to (N + 1)µ,and the quantity
 
N!
N(B.14)=
m!(N − m)!
m
denotes the number of ways of choosing m objects out of a total of N identicalobjects. Here m!, pronounced ‘factorial m’, denotes the product m × (m − 1) ×
. . . , ×2 × 1. The particular case of the binomial distribution for N = 1 is known as
the Bernoulli distribution, and for large N the binomial distribution is approximately
Gaussian. The conjugate prior for µ is the beta distribution.

B. PROBABILITY DISTRIBUTIONS

687

Dirichlet
The Dirichlet is a multivariate distribution over K random variables 0  µ k  1,
where k = 1, . . . , K, subject to the constraintsK0  µk  1,

µk = 1.

(B.15)

k=1

Denoting µ = (µ1 , . . . , µK )T and α = (α1 , . . . , αK )T , we haveK

Dir(µ|α) = C(α)

k −1
µα
k

(B.16)k=1

E[µk ] =var[µk ] =cov[µj µk ] =mode[µk ] =E[ln µk ] =

αk
α

αk (α
 − αk )2α (α + 1)
αj αk
− 2
α
 (α
 + 1)
αk − 1α−Kψ(αk ) − ψ(α)H[µ] = −(B.17)(B.18)(B.19)(B.20)(B.21)

K(αk − 1) {ψ(αk ) − ψ(α)} − ln C(α)(B.22)

k=1

whereC(α) =Γ(α)Γ(α1 ) · · · Γ(αK )

and
α
=

K


αk .

(B.23)(B.24)

k=1

Here

d
ln Γ(a)(B.25)
dais known as the digamma function (Abramowitz and Stegun, 1965). The parameters
αk are subject to the constraint αk > 0 in order to ensure that the distribution can benormalized.
The Dirichlet forms the conjugate prior for the multinomial distribution and represents a generalization of the beta distribution. In this case, the parameters α k can
be interpreted as effective numbers of observations of the corresponding values of
the K-dimensional binary observation vector x. As with the beta distribution, theDirichlet has ﬁnite density everywhere provided αk  1 for all k.
ψ(a) ≡

688

B. PROBABILITY DISTRIBUTIONS

Gamma
The Gamma is a probability distribution over a positive random variable τ > 0
governed by parameters a and b that are subject to the constraints a > 0 and b > 0
to ensure that the distribution can be normalized.
Gam(τ |a, b) =E[τ ] =
var[τ ] =mode[τ ] =E[ln τ ] =H[τ ] =

1 a a−1 −bτ
e
b τ
Γ(a)
a
b
a
b2
a−1
for α  1b
ψ(a) − ln bln Γ(a) − (a − 1)ψ(a) − ln b + a(B.26)(B.27)(B.28)(B.29)(B.30)(B.31)

where ψ(·) is the digamma function deﬁned by (B.25). The gamma distribution isthe conjugate prior for the precision (inverse variance) of a univariate Gaussian. Fora  1 the density is everywhere ﬁnite, and the special case of a = 1 is known as the
exponential distribution.

Gaussian
The Gaussian is the most widely used distribution for continuous variables. It is also
known as the normal distribution. In the case of a single variable x ∈ (−∞, ∞) it isgoverned by two parameters, the mean µ ∈ (−∞, ∞) and the variance σ 2 > 0.
N (x|µ, σ 2 ) =1
1 /2(2πσ 2 )exp −1(x − µ)22σ 2

E[x] = µvar[x] = σ 2
mode[x] = µ11ln σ 2 + (1 + ln(2π)) .
H[x] =
2
2(B.32)(B.33)(B.34)(B.35)(B.36)The inverse of the variance τ = 1/σ 2 is called the precision, and the square rootof the variance σ is called the standard deviation. The conjugate prior for µ is theGaussian, and the conjugate prior for τ is the gamma distribution. If both µ and τ
are unknown, their joint conjugate prior is the Gaussian-gamma distribution.
For a D-dimensional vector x, the Gaussian is governed by a D-dimensionalmean vector µ and a D × D covariance matrix Σ that must be symmetric and689

B. PROBABILITY DISTRIBUTIONS
positive-deﬁnite.
N (x|µ, Σ) =E[x] =
cov[x] =mode[x] =H[x] =

1

1

D/2 |Σ|1/2

1
exp − (x − µ)T Σ−1 (x − µ)
2

(2π)
µ
Σ
µ
D
1
ln |Σ| + (1 + ln(2π)) .
22(B.37)(B.38)(B.39)(B.40)(B.41)The inverse of the covariance matrix Λ = Σ−1 is the precision matrix, which is also
symmetric and positive deﬁnite. Averages of random variables tend to a Gaussian, by
the central limit theorem, and the sum of two Gaussian variables is again Gaussian.
The Gaussian is the distribution that maximizes the entropy for a given variance(or covariance). Any linear transformation of a Gaussian random variable is again
Gaussian. The marginal distribution of a multivariate Gaussian with respect to a
subset of the variables is itself Gaussian, and similarly the conditional distribution is
also Gaussian. The conjugate prior for µ is the Gaussian, the conjugate prior for Λis the Wishart, and the conjugate prior for (µ, Λ) is the Gaussian-Wishart.
If we have a marginal Gaussian distribution for x and a conditional Gaussian
distribution for y given x in the form
p(x) = N (x|µ, Λ−1 )p(y|x) = N (y|Ax + b, L−1 )(B.42)(B.43)

then the marginal distribution of y, and the conditional distribution of x given y, aregiven byp(y) = N (y|Aµ + b, L−1 + AΛ−1 AT )p(x|y) = N (x|Σ{AT L(y − b) + Λµ}, Σ)
where(B.44)(B.45)

Σ = (Λ + AT LA)−1 .

(B.46)−1

If we have a joint Gaussian distribution N (x|µ, Σ) with Λ ≡ Σ
deﬁne the following partitions
 
 
xa
µa
,
µ=
x=
xb
µb




Σaa ΣabΛaa Λab,Λ=Σ=Σba ΣbbΛba Λbb

and we(B.47)(B.48)

then the conditional distribution p(xa |xb ) is given by1
p(xa |xb ) = N (x|µa|b , Λ−
aa )

µa|b

=

1
µa − Λ−aa Λab (xb − µb )(B.49)(B.50)

690

B. PROBABILITY DISTRIBUTIONS
and the marginal distribution p(xa ) is given byp(xa ) = N (xa |µa , Σaa ).

(B.51)

Gaussian-Gamma
This is the conjugate prior distribution for a univariate Gaussian N (x|µ, λ−1 ) in
which the mean µ and the precision λ are both unknown and is also called thenormal-gamma distribution. It comprises the product of a Gaussian distribution forµ, whose precision is proportional to λ, and a gamma distribution over λ.
p(µ, λ|µ0 , β, a, b) = N µ|µo , (βλ)−1 Gam(λ|a, b).

(B.52)

Gaussian-Wishart
This is the conjugate prior distribution for a multivariate Gaussian N (x|µ, Λ) in
which both the mean µ and the precision Λ are unknown, and is also called thenormal-Wishart distribution. It comprises the product of a Gaussian distribution forµ, whose precision is proportional to Λ, and a Wishart distribution over Λ.
p(µ, Λ|µ0 , β, W, ν) = N µ|µ0 , (βΛ)−1 W(Λ|W, ν).

(B.53)For the particular case of a scalar x, this is equivalent to the Gaussian-gamma distribution.

Multinomial
If we generalize the Bernoulli distribution
 to an K-dimensional binary variable x
with components xk ∈ {0, 1} such that k xk = 1, then we obtain the following
discrete distribution
p(x) =

Kµxkk(B.54)

k=1

E[xk ] = µk
var[xk ] = µk (1 − µk )cov[xj xk ] = Ijk µk
H[x] = −M
k=1µk ln µk

(B.55)(B.56)(B.57)(B.58)

B. PROBABILITY DISTRIBUTIONS

691

where Ijk is the j, k element of the identity
 matrix. Because p(xk = 1) = µk , theparameters must satisfy 0  µk  1 and k µk = 1.
The multinomial distribution is a multivariate generalization of the binomial andgives the distribution over counts mk for a K-state discrete variable to be in state kgiven a total number of observations N .

Mult(m1 , m2 , . . . , mK |µ, N ) =N
m1 m2 . . . mM

E[mk ] = N µk
var[mk ] = N µk (1 − µk )cov[mj mk ] = −N µj µk
where µ = (µ1 , . . . , µK )T , and the quantity


N!
N
=
m1 m2 . . . mK
m1 ! . . . mK !


M

k
µm
k

(B.59)

k=1(B.60)(B.61)(B.62)(B.63)

gives the number of ways of taking N identical objects and assigning mk of them tobin k for k = 1, . . . , K. The value of µk gives the probability of the random variabletakingstate k, and so these parameters are subject to the constraints 0  µk  1
and k µk = 1. The conjugate prior distribution for the parameters {µk } is the
Dirichlet.

NormalThe normal distribution is simply another name for the Gaussian. In this book, we
use the term Gaussian throughout, although we retain the conventional use of thesymbol N to denote this distribution. For consistency, we shall refer to the normalgamma distribution as the Gaussian-gamma distribution, and similarly the normalWishart is called the Gaussian-Wishart.

Student’s tThis distribution was published by William Gosset in 1908, but his employer, Guiness Breweries, required him to publish under a pseudonym, so he chose ‘Student’.
In the univariate form, Student’s t-distribution is obtained by placing a conjugate
gamma prior over the precision of a univariate Gaussian distribution and then integrating out the precision variable. It can therefore be viewed as an inﬁnite mixture

692

B. PROBABILITY DISTRIBUTIONS
of Gaussians having the same mean but different variances.

 1/2 −ν/2−1/2
Γ(ν/2 + 1/2) λλ(x − µ)2
1+(B.64)
Γ(ν/2)πννE[x] = µ for ν > 1(B.65)1 ν
for ν > 2
(B.66)
var[x] =λν−2mode[x] = µ.
(B.67)

St(x|µ, λ, ν) =Here ν > 0 is called the number of degrees of freedom of the distribution. The
particular case of ν = 1 is called the Cauchy distribution.
For a D-dimensional variable x, Student’s t-distribution corresponds to marginalizing the precision matrix of a multivariate Gaussian with respect to a conjugate
Wishart prior and takes the form
St(x|µ, Λ, ν) =E[x] =
cov[x] =mode[x] =−ν/2−D/2Γ(ν/2 + D/2) |Λ|1/2∆21+
Γ(ν/2)
ν(νπ)D/2µ for ν > 1ν
Λ−1 for ν > 2
ν−2µ(B.68)(B.69)(B.70)(B.71)

where ∆2 is the squared Mahalanobis distance deﬁned by∆2 = (x − µ)T Λ(x − µ).

(B.72)In the limit ν → ∞, the t-distribution reduces to a Gaussian with mean µ and precision Λ. Student’s t-distribution provides a generalization of the Gaussian whose
maximum likelihood parameter values are robust to outliers.

Uniform
This is a simple distribution for a continuous variable x deﬁned over a ﬁnite intervalx ∈ [a, b] where b > a.
1b−a(b + a)E[x] =
2(b − a)2
var[x] =12
H[x] = ln(b − a).

U(x|a, b) =(B.73)(B.74)(B.75)(B.76)

If x has distribution U(x|0, 1), then a + (b − a)x will have distribution U(x|a, b).

B. PROBABILITY DISTRIBUTIONS

693

Von Mises
The von Mises distribution, also known as the circular normal or the circular Gaussian, is a univariate Gaussian-like periodic distribution for a variable θ ∈ [0, 2π).
1exp {m cos(θ − θ0 )}2πI0 (m)p(θ|θ0 , m) =(B.77)

where I0 (m) is the zeroth-order Bessel function of the ﬁrst kind. The distribution
has period 2π so that p(θ + 2π) = p(θ) for all θ. Care must be taken in interpreting this distribution because simple expectations will be dependent on the (arbitrary)choice of origin for the variable θ. The parameter θ0 is analogous to the mean of a
univariate Gaussian, and the parameter m > 0, known as the concentration parameter, is analogous to the precision (inverse variance). For large m, the von Mises
distribution is approximately a Gaussian centred on θ0 .

Wishart
The Wishart distribution is the conjugate prior for the precision matrix of a multivariate Gaussian.
1(ν−D−1)/2−1
W(Λ|W, ν) = B(W, ν)|Λ|
exp − Tr(W Λ)(B.78)
2
where


−1
D

ν
+
1
−
i
2νD/2 π D(D−1)/4
Γ(B.79)
2


B(W, ν) ≡ |W|−ν/2i=1E[Λ] = νW
E [ln |Λ|] =

D

i=1

ψ



ν+1−i
2

H[Λ] = − ln B(W, ν) −(B.80)

+ D ln 2 + ln |W|(ν − D − 1)νDE [ln |Λ|] +2
2(B.81)(B.82)

where W is a D × D symmetric, positive deﬁnite matrix, and ψ(·) is the digammafunction deﬁned by (B.25). The parameter ν is called the number of degrees of
freedom of the distribution and is restricted to ν > D − 1 to ensure that the Gamma
function in the normalization factor is well-deﬁned. In one dimension, the Wishart
reduces to the gamma distribution Gam(λ|a, b) given by (B.26) with parametersa = ν/2 and b = 1/2W .

Appendix C. Properties of Matrices
In this appendix, we gather together some useful properties and identities involving
matrices and determinants. This is not intended to be an introductory tutorial, and
it is assumed that the reader is already familiar with basic linear algebra. For someresults, we indicate how to prove them, whereas in more complex cases we leave
the interested reader to refer to standard textbooks on the subject. In all cases, we
assume that inverses exist and that matrix dimensions are such that the formulae
are correctly deﬁned. A comprehensive discussion of linear algebra can be found in
Golub and Van Loan (1996), and an extensive collection of matrix properties is given
by Lütkepohl (1996). Matrix derivatives are discussed in Magnus and Neudecker(1999).

Basic Matrix Identities
A matrix A has elements Aij where i indexes the rows, and j indexes the columns.
We use IN to denote the N × N identity matrix (also called the unit matrix), andwhere there is no ambiguity over dimensionality we simply use I. The transpose
matrix AT has elements (AT )ij = Aji . From the deﬁnition of transpose, we have(AB)T = BT AT(C.1)

which can be veriﬁed by writing out the indices. The inverse of A, denoted A−1 ,satisﬁes(C.2)
AA−1 = A−1 A = I.
Because ABB−1 A−1 = I, we have(AB)−1 = B−1 A−1 .
Also we have

AT

−1

= A−1

T

(C.3)
(C.4)
695

696

C. PROPERTIES OF MATRICES
which is easily proven by taking the transpose of (C.2) and applying (C.1).
A useful identity involving matrix inverses is the following(P−1 + BT R−1 B)−1 BT R−1 = PBT (BPBT + R)−1 .

(C.5)

which is easily veriﬁed by right multiplying both sides by (BPBT + R). Supposethat P has dimensionality N × N while R has dimensionality M × M , so that B is
M × N . Then if M  N , it will be much cheaper to evaluate the right-hand side of(C.5) than the left-hand side. A special case that sometimes arises is(I + AB)−1 A = A(I + BA)−1 .

(C.6)

Another useful identity involving inverses is the following:(A + BD−1 C)−1 = A−1 − A−1 B(D + CA−1 B)−1 CA−1

(C.7)

which is known as the Woodbury identity and which can be veriﬁed by multiplying
both sides by (A + BD−1 C). This is useful, for instance, when A is large anddiagonal, and hence easy to invert, while B has many rows but few columns (andconversely for C) so that the right-hand side is much cheaper to evaluate than theleft-hand side.
 A set of vectors {a1 , . . . , aN } is said to be linearly independent if the relationn αn an = 0 holds only if all αn = 0. This implies that none of the vectors
can be expressed as a linear combination of the remainder. The rank of a matrix isthe maximum number of linearly independent rows (or equivalently the maximum
number of linearly independent columns).

Traces and Determinants
Trace and determinant apply to square matrices. The trace Tr(A) of a matrix A
is deﬁned as the sum of the elements on the leading diagonal. By writing out theindices, we see that
Tr(AB) = Tr(BA).
(C.8)
By applying this formula multiple times to the product of three matrices, we see that
Tr(ABC) = Tr(CAB) = Tr(BCA)(C.9)

which is known as the cyclic property of the trace operator and which clearly extends
to the product of any number of matrices. The determinant |A| of an N × N matrix
A is deﬁned by(±1)A1i1 A2i2 · · · AN iN(C.10)|A| =in which the sum is taken over all products consisting of precisely one element fromeach row and one element from each column, with a coefﬁcient +1 or −1 according

C. PROPERTIES OF MATRICES

697

to whether the permutation i1 i2 . . . iN is even or odd, respectively. Note that |I| = 1.
Thus, for a 2 × 2 matrix, the determinant takes the form
 a11 a12  = a11 a22 − a12 a21 .
|A| = (C.11)
a21 a22 
The determinant of a product of two matrices is given by|AB| = |A||B|(C.12)

as can be shown from (C.10). Also, the determinant of an inverse matrix is given by −1 
A  = 1|A|(C.13)

which can be shown by taking the determinant of (C.2) and applying (C.12).
If A and B are matrices of size N × M , then IN + ABT  = IM + AT B .
(C.14)
A useful special case isIN + abT  = 1 + aT b

(C.15)

where a and b are N -dimensional column vectors.

Matrix Derivatives
Sometimes we need to consider derivatives of vectors and matrices with respect toscalars. The derivative of a vector a with respect to a scalar x is itself a vector whose
components are given by
 
∂a
∂ai
=
(C.16)∂x i∂xwith an analogous deﬁnition for the derivative of a matrix. Derivatives with respect
to vectors and matrices can also be deﬁned, for instance
 
∂x
∂x
=
(C.17)∂a i ∂ai
and similarly



∂a
∂b


=
ij

∂ai
.
∂bj

(C.18)

The following is easily proven by writing out the components∂∂
xT a =
aT x = a.
∂x∂x(C.19)

698

C. PROPERTIES OF MATRICES
Similarly∂A∂B∂(AB) =
B+A
.
∂x
∂x
∂x
The derivative of the inverse of a matrix can be expressed as∂∂A −1
A−1 = −A−1
A∂x∂x

(C.20)(C.21)

as can be shown by differentiating the equation A−1 A = I using (C.20) and thenright multiplying by A−1 . Also


∂
−1 ∂A
ln |A| = Tr A
(C.22)∂x∂x
which we shall prove later. If we choose x to be one of the elements of A, we have∂
Tr (AB) = Bji∂Aij(C.23)

as can be seen by writing out the matrices using index notation. We can write thisresult more compactly in the form∂
Tr (AB) = BT .
∂A

(C.24)

With this notation, we have the following properties∂
Tr AT B = B∂A∂
Tr(A) = I
∂A∂
Tr(ABAT ) = A(B + BT )∂A

(C.25)(C.26)(C.27)

which can again be proven by writing out the matrix indices. We also have∂T
ln |A| = A−1∂A(C.28)

which follows from (C.22) and (C.26).

Eigenvector Equation
For a square matrix A of size M × M , the eigenvector equation is deﬁned by
Aui = λi ui(C.29)

699

C. PROPERTIES OF MATRICES

for i = 1, . . . , M , where ui is an eigenvector and λi is the corresponding eigenvalue.
This can be viewed as a set of M simultaneous homogeneous linear equations, andthe condition for a solution is that|A − λi I| = 0(C.30)

which is known as the characteristic equation. Because this is a polynomial of order
M in λi , it must have M solutions (though these need not all be distinct). The rankof A is equal to the number of nonzero eigenvalues.
Of particular interest are symmetric matrices, which arise as covariance matrices, kernel matrices, and Hessians. Symmetric matrices have the property that
Aij = Aji , or equivalently AT = A. The inverse of a symmetric matrix is also symmetric, as can be seen by taking the transpose of A−1 A = I and using AA−1 = I
together with the symmetry of I.
In general, the eigenvalues of a matrix are complex numbers, but for symmetricmatrices the eigenvalues λi are real. This can be seen by ﬁrst left multiplying (C.29)
by (ui )T , where denotes the complex conjugate, to give(ui ) Aui = λi (ui ) ui .
T

T

(C.31)

Next we take the complex conjugate of (C.29) and left multiply by uT
i to give

 T 
uT
i Aui = λi ui ui .

(C.32)

where we have used A = A because we consider only real matrices A. Taking
the transpose of the second of these equations, and using AT = A, we see that theleft-hand sides of the two equations are equal, and hence that λi = λi and so λimust be real.
The eigenvectors ui of a real symmetric matrix can be chosen to be orthonormal(i.e., orthogonal and of unit length) so thatuT
i uj = Iij(C.33)

where Iij are the elements of the identity matrix I. To show this, we ﬁrst left multiply(C.29) by uT
j to giveTuT(C.34)
j Aui = λi uj ui
and hence, by exchange of indices, we haveTuT
i Auj = λj ui uj .

(C.35)

We now take the transpose of the second equation and make use of the symmetry
property AT = A, and then subtract the two equations to give
(λi − λj ) uT
i uj = 0.

(C.36)

Hence, for λi = λj , we have uT
i uj = 0, and hence ui and uj are orthogonal. If thetwo eigenvalues are equal, then any linear combination αui + βuj is also an eigenvector with the same eigenvalue, so we can select one linear combination arbitrarily,

700

C. PROPERTIES OF MATRICES
and then choose the second to be orthogonal to the ﬁrst (it can be shown that the degenerate eigenvectors are never linearly dependent). Hence the eigenvectors can bechosen to be orthogonal, and by normalizing can be set to unit length. Because there
are M eigenvalues, the corresponding M orthogonal eigenvectors form a complete
set and so any M -dimensional vector can be expressed as a linear combination of
the eigenvectors.
We can take the eigenvectors ui to be the columns of an M × M matrix U,
which from orthonormality satisﬁes
UT U = I.

(C.37)Such a matrix is said to be orthogonal. Interestingly, the rows of this matrix are also
orthogonal, so that UUT = I. To show this, note that (C.37) implies UT UU−1 =
U−1 = UT and so UU−1 = UUT = I. Using (C.12), it also follows that |U| = 1.
The eigenvector equation (C.29) can be expressed in terms of U in the form
AU = UΛ

(C.38)

where Λ is an M × M diagonal matrix whose diagonal elements are given by the
eigenvalues λi .
If we consider a column vector x that is transformed by an orthogonal matrix U
to give a new vectorx = Ux(C.39)
then the length of the vector is preserved becausex
T x
 = xT UT Ux = xT x

(C.40)and similarly the angle between any two such vectors is preserved becausex
T y = xT UT Uy = xT y.

(C.41)

Thus, multiplication by U can be interpreted as a rigid rotation of the coordinate
system.
From (C.38), it follows that
UT AU = Λ(C.42)

and because Λ is a diagonal matrix, we say that the matrix A is diagonalized by thematrix U. If we left multiply by U and right multiply by UT , we obtain
A = UΛUT(C.43)Taking the inverse of this equation, and using (C.3) together with U−1 = UT , wehave
(C.44)
A−1 = UΛ−1 UT .

C. PROPERTIES OF MATRICES

701

These last two equations can also be written in the form
A =

M


λ i ui uT
i(C.45)

i=1

A−1

=

M

1
i=1

λi

ui uT
i .

(C.46)

If we take the determinant of (C.43), and use (C.12), we obtain
|A| =

M


λi .

(C.47)

i=1

Similarly, taking the trace of (C.43), and using the cyclic property (C.8) of the trace
operator together with UT U = I, we have
Tr(A) =M


λi .

(C.48)

i=1

We leave it as an exercise for the reader to verify (C.22) by making use of the results(C.33), (C.45), (C.46), and (C.47).
A matrix A is said to be positive deﬁnite, denoted by A  0, if wT Aw > 0 for
all values of the vector w. Equivalently, a positive deﬁnite matrix has λi > 0 for allof its eigenvalues (as can be seen by setting w to each of the eigenvectors in turn,and by noting that an arbitrary vector can be expanded as a linear combination of theeigenvectors). Note that positive deﬁnite is not the same as all the elements being
positive. For example, the matrix
1 2(C.49)
3 4
has eigenvalues λ1  5.37 and λ2  −0.37. A matrix is said to be positive semidefinite if wT Aw  0 holds for all values of w, which is denoted A
0, and isequivalent to λi  0.

Appendix D. Calculus of Variations
We can think of a function y(x) as being an operator that, for any input value x,
returns an output value y. In the same way, we can deﬁne a functional F [y] to be
an operator that takes a function y(x) and returns an output value F . An example ofa functional is the length of a curve drawn in a two-dimensional plane in which thepath of the curve is deﬁned in terms of a function. In the context of machine learning,a widely used functional is the entropy H[x] for a continuous variable x because, for
any choice of probability density function p(x), it returns a scalar value representing
the entropy of x under that density. Thus the entropy of p(x) could equally well havebeen written as H[p].
A common problem in conventional calculus is to ﬁnd a value of x that maximizes (or minimizes) a function y(x). Similarly, in the calculus of variations weseek a function y(x) that maximizes (or minimizes) a functional F [y]. That is, of all
possible functions y(x), we wish to ﬁnd the particular function for which the functional F [y] is a maximum (or minimum). The calculus of variations can be used, forinstance, to show that the shortest path between two points is a straight line or thatthe maximum entropy distribution is a Gaussian.
If we weren’t familiar with the rules of ordinary calculus, we could evaluate aconventional derivative dy/ dx by making a small change to the variable x andthen expanding in powers of , so thatdy
+ O( 2 )(D.1)
dx
and ﬁnally taking the limit → 0. Similarly, for a function of several variables
y(x1 , . . . , xD ), the corresponding partial derivatives are deﬁned byy(x + ) = y(x) +y(x1 + 1 , . . . , xD + D ) = y(x1 , . . . , xD ) +

D

∂y
i=1

∂xi

i + O(

2

).

(D.2)

The analogous deﬁnition of a functional derivative arises when we consider how
much a functional F [y] changes when we make a small change η(x) to the function
703

704

D. CALCULUS OF VARIATIONS
Figure D.1

A functional derivative can be deﬁned byconsidering how the value of a functional
F [y] changes when the function y(x) ischanged to y(x) + η(x) where η(x) is anarbitrary function of x.

y(x)y(x) + η(x)xy(x), where η(x) is an arbitrary function of x, as illustrated in Figure D.1. We denotethe functional derivative of E[f ] with respect to f (x) by δF/δf (x), and deﬁne it by
the following relation:

δF
(D.3)
F [y(x) + η(x)] = F [y(x)] +η(x) dx + O( 2 ).
δy(x)This can be seen as a natural extension of (D.2) in which F [y] now depends on acontinuous set of variables, namely the values of y at all points x. Requiring that thefunctional be stationary with respect to small variations in the function y(x) givesδE
η(x) dx = 0.
(D.4)
δy(x)
Because this must hold for an arbitrary choice of η(x), it follows that the functional
derivative must vanish. To see this, imagine choosing a perturbation η(x) that is zerox, in which case the functional
everywhere except in the neighbourhood of a point derivative must be zero at x = 
x. However, because this must be true for every
choice of x, the functional derivative must vanish for all values of x.
Consider a functional that is deﬁned by an integral over a function G(y, y  , x)that depends on both y(x) and its derivative y  (x) as well as having a direct dependence on x(D.5)
F [y] = G (y(x), y  (x), x) dx
where the value of y(x) is assumed to be ﬁxed at the boundary of the region of
integration (which might be at inﬁnity). If we now consider variations in the function
y(x), we obtain
∂G∂G
η(x) +  η  (x) dx + O( 2 ). (D.6)
F [y(x) + η(x)] = F [y(x)] +∂y∂y
We now have to cast this in the form (D.3). To do so, we integrate the second term byparts and make use of the fact that η(x) must vanish at the boundary of the integral(because y(x) is ﬁxed at the boundary). This gives



∂G
d ∂G
−
η(x) dx + O( 2 ) (D.7)
F [y(x) + η(x)] = F [y(x)] +∂y
dx ∂y 

D. CALCULUS OF VARIATIONS

705

from which we can read off the functional derivative by comparison with (D.3).
Requiring that the functional derivative vanishes then gives


∂G
d ∂G
−
=0
(D.8)
∂y
dx ∂y which are known as the Euler-Lagrange equations. For example, if
G = y(x)2 + (y  (x))

2

(D.9)

then the Euler-Lagrange equations take the formy(x) −

Appendix E

d2 y
= 0.
dx2

(D.10)

This second order differential equation can be solved for y(x) by making use of theboundary conditions on y(x).
Often, we consider functionals deﬁned by integrals whose integrands take theform G(y, x) and that do not depend on the derivatives of y(x). In this case, stationarity simply requires that ∂G/∂y(x) = 0 for all values of x.
If we are optimizing a functional with respect to a probability distribution, thenwe need to maintain the normalization constraint on the probabilities. This is often
most conveniently done using a Lagrange multiplier, which then allows an unconstrained optimization to be performed.
The extension of the above results to a multidimensional variable x is straightforward. For a more comprehensive discussion of the calculus of variations, see
Sagan (1969).

Appendix E. Lagrange Multipliers
Lagrange multipliers, also sometimes called undetermined multipliers, are used toﬁnd the stationary points of a function of several variables subject to one or moreconstraints.
Consider the problem of ﬁnding the maximum of a function f (x1 , x2 ) subject toa constraint relating x1 and x2 , which we write in the formg(x1 , x2 ) = 0.

(E.1)

One approach would be to solve the constraint equation (E.1) and thus express x2 asa function of x1 in the form x2 = h(x1 ). This can then be substituted into f (x1 , x2 )to give a function of x1 alone of the form f (x1 , h(x1 )). The maximum with respect
to x1 could then be found by differentiation in the usual way, to give the stationary
value x1 , with the corresponding value of x2 given by x2 = h(x1 ).
One problem with this approach is that it may be difﬁcult to ﬁnd an analytic
solution of the constraint equation that allows x2 to be expressed as an explicit function of x1 . Also, this approach treats x1 and x2 differently and so spoils the naturalsymmetry between these variables.
A more elegant, and often simpler, approach is based on the introduction of aparameter λ called a Lagrange multiplier. We shall motivate this technique froma geometrical perspective. Consider a D-dimensional variable x with components
x1 , . . . , xD . The constraint equation g(x) = 0 then represents a (D−1)-dimensionalsurface in x-space as indicated in Figure E.1.
We ﬁrst note that at any point on the constraint surface the gradient ∇g(x) of
the constraint function will be orthogonal to the surface. To see this, consider a point
x that lies on the constraint surface, and consider a nearby point x +  that also lieson the surface. If we make a Taylor expansion around x, we haveg(x + )  g(x) + T ∇g(x).

(E.2)

Because both x and x +  lie on the constraint surface, we have g(x) = g(x + ) andhence T ∇g(x)  0. In the limit  → 0 we have T ∇g(x) = 0, and because  is
707

708

E. LAGRANGE MULTIPLIERS
Figure E.1

A geometrical picture of the technique of Lagrange multipliers in which we seek to maximize afunction f (x), subject to the constraint g(x) = 0.
If x is D dimensional, the constraint g(x) = 0 corresponds to a subspace of dimensionality D − 1,indicated by the red curve. The problem canbe solved by optimizing the Lagrangian function
L(x, λ) = f (x) + λg(x).

∇f (x)xA∇g(x)g(x) = 0then parallel to the constraint surface g(x) = 0, we see that the vector ∇g is normal
to the surface.
Next we seek a point x on the constraint surface such that f (x) is maximized.
Such a point must have the property that the vector ∇f (x) is also orthogonal to theconstraint surface, as illustrated in Figure E.1, because otherwise we could increase
the value of f (x) by moving a short distance along the constraint surface. Thus ∇fand ∇g are parallel (or anti-parallel) vectors, and so there must exist a parameter λsuch that∇f + λ∇g = 0(E.3)
where λ = 0 is known as a Lagrange multiplier. Note that λ can have either sign.
At this point, it is convenient to introduce the Lagrangian function deﬁned byL(x, λ) ≡ f (x) + λg(x).

(E.4)

The constrained stationarity condition (E.3) is obtained by setting ∇x L = 0. Furthermore, the condition ∂L/∂λ = 0 leads to the constraint equation g(x) = 0.
Thus to ﬁnd the maximum of a function f (x) subject to the constraint g(x) = 0,
we deﬁne the Lagrangian function given by (E.4) and we then ﬁnd the stationary
point of L(x, λ) with respect to both x and λ. For a D-dimensional vector x, thisgives D + 1 equations that determine both the stationary point x and the value of λ.
If we are only interested in x , then we can eliminate λ from the stationarity equations without needing to ﬁnd its value (hence the term ‘undetermined multiplier’).
As a simple example, suppose we wish to ﬁnd the stationary point of the function
f (x1 , x2 ) = 1 − x21 − x22 subject to the constraint g(x1 , x2 ) = x1 + x2 − 1 = 0, as
illustrated in Figure E.2. The corresponding Lagrangian function is given byL(x, λ) = 1 − x21 − x22 + λ(x1 + x2 − 1).

(E.5)

The conditions for this Lagrangian to be stationary with respect to x1 , x2 , and λ givethe following coupled equations:−2x1 + λ = 0−2x2 + λ = 0x1 + x2 − 1 = 0.

(E.6)
(E.7)
(E.8)

709

E. LAGRANGE MULTIPLIERS
Figure E.2

A simple example of the use of Lagrange multipliers in which the aim is to maximize f (x1 , x2 ) =1 − x21 − x22 subject to the constraint g(x1 , x2 ) = 0
where g(x1 , x2 ) = x1 + x2 − 1. The circles show
contours of the function f (x1 , x2 ), and the diagonalline shows the constraint surface g(x1 , x2 ) = 0.

x2(x1 , x2 )x1
g(x1 , x2 ) = 0

Solution of these equations then gives the stationary point as (x1 , x2 ) = ( 12 , 12 ), andthe corresponding value for the Lagrange multiplier is λ = 1.
So far, we have considered the problem of maximizing a function subject to an
equality constraint of the form g(x) = 0. We now consider the problem of maximizing f (x) subject to an inequality constraint of the form g(x)  0, as illustrated
in Figure E.3.
There are now two kinds of solution possible, according to whether the constrained stationary point lies in the region where g(x) > 0, in which case the constraint is inactive, or whether it lies on the boundary g(x) = 0, in which case theconstraint is said to be active. In the former case, the function g(x) plays no roleand so the stationary condition is simply ∇f (x) = 0. This again corresponds toa stationary point of the Lagrange function (E.4) but this time with λ = 0. The
latter case, where the solution lies on the boundary, is analogous to the equality constraint discussed previously and corresponds to a stationary point of the Lagrangefunction (E.4) with λ = 0. Now, however, the sign of the Lagrange multiplier iscrucial, because the function f (x) will only be at a maximum if its gradient is oriented away from the region g(x) > 0, as illustrated in Figure E.3. We therefore have∇f (x) = −λ∇g(x) for some value of λ > 0.
For either of these two cases, the product λg(x) = 0. Thus the solution to the
Figure E.3∇f (x)

Illustration of the problem of maximizing
f (x) subject to the inequality constraint
g(x)  0.

xA
∇g(x)
xB

g(x) > 0g(x) = 0710

E. LAGRANGE MULTIPLIERS
problem of maximizing f (x) subject to g(x)  0 is obtained by optimizing the
Lagrange function (E.4) with respect to x and λ subject to the conditions
g(x)  0λ  0λg(x) = 0(E.9)
(E.10)(E.11)

These are known as the Karush-Kuhn-Tucker (KKT) conditions (Karush, 1939; Kuhn
and Tucker, 1951).
Note that if we wish to minimize (rather than maximize) the function f (x) subject to an inequality constraint g(x)  0, then we minimize the Lagrangian function
L(x, λ) = f (x) − λg(x) with respect to x, again subject to λ  0.
Finally, it is straightforward to extend the technique of Lagrange multipliers to
the case of multiple equality and inequality constraints. Suppose we wish to maximize f (x) subject to gj (x) = 0 for j = 1, . . . , J, and hk (x)  0 for k = 1, . . . , K.
We then introduce Lagrange multipliers {λj } and {µk }, and then optimize the Lagrangian function given byL(x, {λj }, {µk }) = f (x) +

J

j =1

Appendix D

λj gj (x) +Kµk hk (x)(E.12)

k=1

subject to µk  0 and µk hk (x) = 0 for k = 1, . . . , K. Extensions to constrained
functional derivatives are similarly straightforward. For a more detailed discussionof the technique of Lagrange multipliers, see Nocedal and Wright (1999).

REFERENCES

711

References
Abramowitz, M. and I. A. Stegun (1965). Handbook Amari, S., A. Cichocki, and H. H. Yang (1996). Aof Mathematical Functions. Dover.
new learning algorithm for blind signal separation. In D. S. Touretzky, M. C. Mozer, and M. E.
Adler, S. L. (1981). Over-relaxation method for the
Hasselmo (Eds.), Advances in Neural InformaMonte Carlo evaluation of the partition function Processing Systems, Volume 8, pp. 757–763.
tion for multiquadratic actions. Physical Review
MIT Press.
D 23, 2901–2904.
Ahn, J. H. and J. H. Oh (2003). A constrained EM Amari, S. I. (1998). Natural gradient works efﬁciently in learning. Neural Computation 10,
algorithm for principal component analysis. Neu251–276.
ral Computation 15(1), 57–65.
Aizerman, M. A., E. M. Braverman, and L. I. Rozo- Anderson, J. A. and E. Rosenfeld (Eds.) (1988).
Neurocomputing: Foundations of Research. MIT
noer (1964). The probability problem of pattern
Press.
recognition learning and the method of potentialfunctions. Automation and Remote Control 25, Anderson, T. W. (1963). Asymptotic theory for prin1175–1190.
cipal component analysis. Annals of MathematiAkaike, H. (1974). A new look at statistical modelcal Statistics 34, 122–148.
identiﬁcation. IEEE Transactions on Automatic
Andrieu, C., N. de Freitas, A. Doucet, and M. I. JorControl 19, 716–723.
dan (2003). An introduction to MCMC for maAli, S. M. and S. D. Silvey (1966). A general classchine learning. Machine Learning 50, 5–43.
of coefﬁcients of divergence of one distribution
from another. Journal of the Royal Statistical So- Anthony, M. and N. Biggs (1992). An Introduction
to Computational Learning Theory. Cambridge
ciety, B 28(1), 131–142.
University Press.
Allwein, E. L., R. E. Schapire, and Y. Singer (2000).
Reducing multiclass to binary: a unifying ap- Attias, H. (1999a). Independent factor analysis. Neural Computation 11(4), 803–851.
proach for margin classiﬁers. Journal of Machine
Learning Research 1, 113–141.
Attias, H. (1999b). Inferring parameters and strucAmari, S. (1985). Differential-Geometrical Methods
ture of latent variable models by variationalin Statistics. Springer.
Bayes. In K. B. Laskey and H. Prade (Eds.),

712

REFERENCES

Uncertainty in Artiﬁcial Intelligence: Proceed- Baum, L. E. (1972). An inequality and associatedings of the Fifth Conference, pp. 21–30. Morgan
maximization technique in statistical estimation
Kaufmann.
of probabilistic functions of Markov processes.
Inequalities 3, 1–8.
Bach, F. R. and M. I. Jordan (2002). Kernel independent component analysis. Journal of Machine Becker, S. and Y. Le Cun (1989). Improving the conLearning Research 3, 1–48.
vergence of back-propagation learning with second order methods. In D. Touretzky, G. E. HinBakir, G. H., J. Weston, and B. Schölkopf (2004).
ton, and T. J. Sejnowski (Eds.), Proceedings ofLearning to ﬁnd pre-images. In S. Thrun, L. K.
the 1988 Connectionist Models Summer School,
Saul, and B. Schölkopf (Eds.), Advances in Neupp. 29–37. Morgan Kaufmann.
ral Information Processing Systems, Volume 16,
Bell, A. J. and T. J. Sejnowski (1995). An inforpp. 449–456. MIT Press.
mation maximization approach to blind separaBaldi, P. and S. Brunak (2001). Bioinformatics: Thetion and blind deconvolution. Neural ComputaMachine Learning Approach (Second ed.). MITtion 7(6), 1129–1159.
Press.
Baldi, P. and K. Hornik (1989). Neural networks Bellman, R. (1961). Adaptive Control Processes: A
Guided Tour. Princeton University Press.
and principal component analysis: learning fromexamples without local minima. Neural Net- Bengio, Y. and P. Frasconi (1995). An input output
HMM architecture. In G. Tesauro, D. S. Touretworks 2(1), 53–58.
zky, and T. K. Leen (Eds.), Advances in Neural
Barber, D. and C. M. Bishop (1997). BayesianInformation Processing Systems, Volume 7, pp.
model comparison by Monte Carlo chaining. In
427–434.
MIT Press.
M. Mozer, M. Jordan, and T. Petsche (Eds.), Advances in Neural Information Processing Sys- Bennett, K. P. (1992). Robust linear programming
discrimination of two linearly separable sets. Optems, Volume 9, pp. 333–339. MIT Press.
timization Methods and Software 1, 23–34.
Barber, D. and C. M. Bishop (1998a). Ensemble
learning for multi-layer networks. In M. I. Jor- Berger, J. O. (1985). Statistical Decision Theory andBayesian Analysis (Second ed.). Springer.
dan, K. J. Kearns, and S. A. Solla (Eds.), Advances in Neural Information Processing Sys- Bernardo, J. M. and A. F. M. Smith (1994). Bayesiantems, Volume 10, pp. 395–401.
Theory. Wiley.
Barber, D. and C. M. Bishop (1998b). Ensemble Berrou, C., A. Glavieux, and P. Thitimajshimalearning in Bayesian neural networks. In C. M.
(1993). Near Shannon limit error-correcting codBishop (Ed.), Generalization in Neural Networks
ing and decoding: Turbo-codes (1). In Proceedand Machine Learning, pp. 215–237. Springer.
ings ICC’93, pp. 1064–1070.
Bartholomew, D. J. (1987). Latent Variable Models Besag, J. (1974). On spatio-temporal models andand Factor Analysis. Charles Grifﬁn.
Markov ﬁelds. In Transactions of the 7th Prague
Basilevsky, A. (1994). Statistical Factor Analysis
Conference on Information Theory, Statisticaland Related Methods: Theory and Applications.
Decision Functions and Random Processes, pp.
Wiley.
47–75. Academia.
Bather, J. (2000). Decision Theory: An Introduction Besag, J. (1986). On the statistical analysis of dirty
to Dynamic Programming and Sequential Decipictures. Journal of the Royal Statistical Socisions. Wiley.
ety B-48, 259–302.
Baudat, G. and F. Anouar (2000). Generalized dis- Besag, J., P. J. Green, D. Hidgon, and K. Megersen
criminant analysis using a kernel approach. Neu(1995). Bayesian computation and stochastic
ral Computation 12(10), 2385–2404.
systems. Statistical Science 10(1), 3–66.

REFERENCES
Bishop, C. M. (1991). A fast procedure for retraining
the multilayer perceptron. International Journalof Neural Systems 2(3), 229–236.

713

Information Processing Systems, Volume 15, pp.
793–800. MIT Press.

Bishop, C. M. and M. Svensén (2003). Bayesian hierarchical mixtures of experts. In U. Kjaerulff
and C. Meek (Eds.), Proceedings Nineteenth
Conference on Uncertainty in Artiﬁcial Intelligence, pp. 57–64. Morgan Kaufmann.
Bishop, C. M. (1993). Curvature-driven smoothing:a learning algorithm for feedforward networks. Bishop, C. M., M. Svensén, and G. E. Hinton
IEEE Transactions on Neural Networks 4(5),(2004). Distinguishing text from graphics in on882–884.
line handwritten ink. In F. Kimura and H. Fujisawa (Eds.), Proceedings Ninth International
Bishop, C. M. (1994). Novelty detection and neuWorkshop on Frontiers in Handwriting Recogniral network validation. IEE Proceedings: Vision,tion, IWFHR-9, Tokyo, Japan, pp. 142–147.
Image and Signal Processing 141(4), 217–222.
Special issue on applications of neural networks. Bishop, C. M., M. Svensén, and C. K. I. Williams
Bishop, C. M. (1992). Exact calculation of the Hessian matrix for the multilayer perceptron. NeuralComputation 4(4), 494–501.

Bishop, C. M. (1995a). Neural Networks for Pattern
Recognition. Oxford University Press.
Bishop, C. M. (1995b). Training with noise is equivalent to Tikhonov regularization. Neural Computation 7(1), 108–116.

(1996). EM optimization of latent variable density models. In D. S. Touretzky, M. C. Mozer,
and M. E. Hasselmo (Eds.), Advances in Neural
Information Processing Systems, Volume 8, pp.
465–471. MIT Press.

Bishop, C. M. (1999a). Bayesian PCA. In M. S. Bishop, C. M., M. Svensén, and C. K. I. Williams(1997a). GTM: a principled alternative to the
Kearns, S. A. Solla, and D. A. Cohn (Eds.), AdSelf-Organizing Map. In M. C. Mozer, M. I. Jorvances in Neural Information Processing Sysdan, and T. Petche (Eds.), Advances in Neuraltems, Volume 11, pp. 382–388. MIT Press.
Information Processing Systems, Volume 9, pp.
Bishop, C. M. (1999b). Variational principal
354–360. MIT Press.
components. In Proceedings Ninth International Conference on Artiﬁcial Neural Networks, Bishop, C. M., M. Svensén, and C. K. I. Williams(1997b). Magniﬁcation factors for the GTM alICANN’99, Volume 1, pp. 509–514. IEE.
gorithm. In Proceedings IEE Fifth International
Bishop, C. M. and G. D. James (1993). Analysis ofConference on Artiﬁcial Neural Networks, Cammultiphase ﬂows using dual-energy gamma denbridge, U.K., pp. 64–69. Institute of Electricalsitometry and neural networks. Nuclear InstruEngineers.
ments and Methods in Physics Research A327,
580–593.

Bishop, C. M., M. Svensén, and C. K. I. Williams(1998a). Developments of the Generative ToBishop, C. M. and I. T. Nabney (1996). Modelling
pographic Mapping. Neurocomputing 21, 203–
conditional probability distributions for periodic
224.
variables. Neural Computation 8(5), 1123–1133.
Bishop, C. M. and I. T. Nabney (2008). Pattern Bishop, C. M., M. Svensén, and C. K. I. Williams(1998b). GTM: the Generative Topographic
Recognition and Machine Learning: A Matlab
Mapping. Neural Computation 10(1), 215–234.
Companion. Springer. In preparation.
Bishop, C. M., D. Spiegelhalter, and J. Winn Bishop, C. M. and M. E. Tipping (1998). A hierarchical latent variable model for data visualiza(2003). VIBES: A variational inference engine
tion. IEEE Transactions on Pattern Analysis andfor Bayesian networks. In S. Becker, S. Thrun,
Machine Intelligence 20(3), 281–293.
and K. Obermeyer (Eds.), Advances in Neural

714

REFERENCES

Bishop, C. M. and J. Winn (2000). Non-linear Breiman, L. (1996). Bagging predictors. Machine
Bayesian image modelling. In Proceedings Sixth
Learning 26, 123–140.
European Conference on Computer Vision, Breiman, L., J. H. Friedman, R. A. Olshen, andDublin, Volume 1, pp. 3–17. Springer.
P. J. Stone (1984). Classiﬁcation and Regression
Trees. Wadsworth.
Blei, D. M., M. I. Jordan, and A. Y. Ng (2003). Hierarchical Bayesian models for applications in Brooks, S. P. (1998). Markov chain Monteinformation retrieval. In J. M. B. et al. (Ed.),
Carlo method and its application. The StatistiBayesian Statistics, 7, pp. 25–43. Oxford Unician 47(1), 69–100.
versity Press.
Broomhead, D. S. and D. Lowe (1988). MultivariBlock, H. D. (1962). The perceptron: a model
able functional interpolation and adaptive netfor brain functioning. Reviews of Modern
works. Complex Systems 2, 321–355.
Physics 34(1), 123–135. Reprinted in Anderson
Buntine, W. and A. Weigend (1991). Bayesian backand Rosenfeld (1988).
propagation. Complex Systems 5, 603–643.
Blum, J. A. (1965). Multidimensional stochastic ap- Buntine, W. L. and A. S. Weigend (1993). Comproximation methods. Annals of Mathematicalputing second derivatives in feed-forward netStatistics 25, 737–744.
works: a review. IEEE Transactions on Neural
Bodlaender, H. (1993). A tourist guide throughNetworks 5(3), 480–488.
treewidth. Acta Cybernetica 11, 1–21.
Burges, C. J. C. (1998). A tutorial on support vecBoser, B. E., I. M. Guyon, and V. N. Vapnik (1992).
tor machines for pattern recognition. Knowledge
A training algorithm for optimal margin classiDiscovery and Data Mining 2(2), 121–167.
ﬁers. In D. Haussler (Ed.), Proceedings Fifth An- Cardoso, J.-F. (1998). Blind signal separation: statisnual Workshop on Computational Learning Thetical principles. Proceedings of the IEEE 9(10),ory (COLT), pp. 144–152. ACM.
2009–2025.
Bourlard, H. and Y. Kamp (1988). Auto-association Casella, G. and R. L. Berger (2002). Statistical Inby multilayer perceptrons and singular value deference (Second ed.). Duxbury.
composition. Biological Cybernetics 59, 291–
Castillo, E., J. M. Gutiérrez, and A. S. Hadi (1997).
294.
Expert Systems and Probabilistic Network ModBox, G. E. P., G. M. Jenkins, and G. C. Reinsel
els. Springer.
(1994). Time Series Analysis. Prentice Hall.
Chan, K., T. Lee, and T. J. Sejnowski (2003). VariBox, G. E. P. and G. C. Tao (1973). Bayesian Inferational Bayesian learning of ICA with missing
ence in Statistical Analysis. Wiley.
data. Neural Computation 15(8), 1991–2011.
Boyd, S. and L. Vandenberghe (2004). Convex Opti- Chen, A. M., H. Lu, and R. Hecht-Nielsen (1993).
mization. Cambridge University Press.
On the geometry of feedforward neural network
error surfaces. Neural Computation 5(6), 910–
Boyen, X. and D. Koller (1998). Tractable inference927.
for complex stochastic processes. In G. F. Cooper
and S. Moral (Eds.), Proceedings 14th Annual Chen, M. H., Q. M. Shao, and J. G. Ibrahim (Eds.)
Conference on Uncertainty in Artiﬁcial Intelli(2001). Monte Carlo Methods for Bayesian Comgence (UAI), pp. 33–42. Morgan Kaufmann.
putation. Springer.
Boykov, Y., O. Veksler, and R. Zabih (2001). Fast Chen, S., C. F. N. Cowan, and P. M. Grant (1991).
Orthogonal least squares learning algorithm forapproximate energy minimization via graph cuts.
radial basis function networks. IEEE TransacIEEE Transactions on Pattern Analysis and Mations on Neural Networks 2(2), 302–309.
chine Intelligence 23(11), 1222–1239.

REFERENCES

715

Choudrey, R. A. and S. J. Roberts (2003). Variational Cox, T. F. and M. A. A. Cox (2000). Multidimenmixture of Bayesian independent component ansional Scaling (Second ed.). Chapman and Hall.
alyzers. Neural Computation 15(1), 213–252.
Cressie, N. (1993). Statistics for Spatial Data. Wiley.
Clifford, P. (1990). Markov random ﬁelds in statis- Cristianini, N. and J. Shawe-Taylor (2000). Support
tics. In G. R. Grimmett and D. J. A. Welsh (Eds.),
vector machines and other kernel-based learning
Disorder in Physical Systems. A Volume in Honmethods. Cambridge University Press.
our of John M. Hammersley, pp. 19–32. Oxford
Csató, L. and M. Opper (2002). Sparse on-line GausUniversity Press.
sian processes. Neural Computation 14(3), 641–
Collins, M., S. Dasgupta, and R. E. Schapire (2002).
668.
A generalization of principal component analy- Csiszàr, I. and G. Tusnàdy (1984). Information gesis to the exponential family. In T. G. Dietterich,ometry and alternating minimization procedures.
S. Becker, and Z. Ghahramani (Eds.), Advances
Statistics and Decisions 1(1), 205–237.
in Neural Information Processing Systems, VolCybenko, G. (1989). Approximation by superposiume 14, pp. 617–624. MIT Press.
tions of a sigmoidal function. Mathematics ofComon, P., C. Jutten, and J. Herault (1991). Blind
Control, Signals and Systems 2, 304–314.
source separation, 2: problems statement. Signal
Dawid,A. P. (1979). Conditional independence inProcessing 24(1), 11–20.
statistical theory (with discussion). Journal of the
Corduneanu, A. and C. M. Bishop (2001). VariRoyal Statistical Society, Series B 4, 1–31.
ational Bayesian model selection for mixture
Dawid, A. P. (1980). Conditional independence for
distributions. In T. Richardson and T. Jaakkola
statistical operations. Annals of Statistics 8, 598–
(Eds.), Proceedings Eighth International Confer617.
ence on Artiﬁcial Intelligence and Statistics, pp.
deFinetti, B. (1970). Theory of Probability. Wiley
27–34. Morgan Kaufmann.
and Sons.
Cormen, T. H., C. E. Leiserson, R. L. Rivest, andC. Stein (2001). Introduction to Algorithms (Sec- Dempster, A. P., N. M. Laird, and D. B. Rubin(1977). Maximum likelihood from incomplete
ond ed.). MIT Press.
data via the EM algorithm. Journal of the Royal
Cortes, C. and V. N. Vapnik (1995). Support vector
Statistical Society, B 39(1), 1–38.
networks. Machine Learning 20, 273–297.
Denison, D. G. T., C. C. Holmes, B. K. Mallick,
Cotter, N. E. (1990). The Stone-Weierstrass theoand A. F. M. Smith (2002). Bayesian Methods forrem and its application to neural networks. IEEE
Nonlinear Classiﬁcation and Regression. Wiley.
Transactions on Neural Networks 1(4), 290–295.
Diaconis, P. and L. Saloff-Coste (1998). What do we
Cover, T. and P. Hart (1967). Nearest neighbor patknow about the Metropolis algorithm? Journal
tern classiﬁcation. IEEE Transactions on Inforof Computer and System Sciences 57, 20–36.
mation Theory IT-11, 21–27.
Dietterich, T. G. and G. Bakiri (1995). Solving
Cover, T. M. and J. A. Thomas (1991). Elements ofmulticlass learning problems via error-correcting
Information Theory. Wiley.
output codes. Journal of Artiﬁcial Intelligence
Research 2, 263–286.
Cowell, R. G., A. P. Dawid, S. L. Lauritzen, and D. J.
Spiegelhalter (1999). Probabilistic Networks and Duane, S., A. D. Kennedy, B. J. Pendleton, andExpert Systems. Springer.
D. Roweth (1987). Hybrid Monte Carlo. Physics
Letters B 195(2), 216–222.
Cox, R. T. (1946). Probability, frequency andreasonable expectation. American Journal of Duda, R. O. and P. E. Hart (1973). Pattern Classiﬁcation and Scene Analysis. Wiley.
Physics 14(1), 1–13.

716

REFERENCES

Duda, R. O., P. E. Hart, and D. G. Stork (2001). Pat- Fletcher, R. (1987). Practical Methods of Optimizatern Classiﬁcation (Second ed.). Wiley.
tion (Second ed.). Wiley.
Durbin, R., S. Eddy, A. Krogh, and G. Mitchi- Forsyth, D. A. and J. Ponce (2003). Computer Vision: A Modern Approach. Prentice Hall.
son (1998). Biological Sequence Analysis. Cambridge University Press.
Freund, Y. and R. E. Schapire (1996). Experimentswith a new boosting algorithm. In L. Saitta (Ed.),
Dybowski, R. and S. Roberts (2005). An anthology
Thirteenth International Conference on Machineof probabilistic models for medical informatics.
Learning, pp. 148–156. Morgan Kaufmann.
In D. Husmeier, R. Dybowski, and S. Roberts(Eds.), Probabilistic Modeling in Bioinformatics Frey, B. J. (1998). Graphical Models for Maand Medical Informatics, pp. 297–349. Springer.
chine Learning and Digital Communication.
MIT Press.
Efron, B. (1979). Bootstrap methods: another look
Frey, B. J. and D. J. C. MacKay (1998). A revolution: Belief propagation in graphs with cycles. In
Elkan, C. (2003). Using the triangle inequality to acM. I. Jordan, M. J. Kearns, and S. A. Solla (Eds.),
celerate k-means. In Proceedings of the Twelfth
Advances in Neural Information Processing SysInternational Conference on Machine Learning,tems, Volume 10. MIT Press.
pp. 147–153. AAAI.
Friedman, J. H. (2001). Greedy function approxiElliott, R. J., L. Aggoun, and J. B. Moore (1995).
mation: a gradient boosting machine. Annals of
Hidden Markov Models: Estimation and ConStatistics 29(5), 1189–1232.
trol. Springer.
Friedman, J. H., T. Hastie, and R. Tibshirani (2000).
Ephraim, Y., D. Malah, and B. H. Juang (1989).
Additive logistic regression: a statistical view ofOn the application of hidden Markov models forboosting. Annals of Statistics 28, 337–407.
enhancing noisy speech. IEEE Transactions on
Acoustics, Speech and Signal Processing 37(12), Friedman, N. and D. Koller (2003). Being Bayesian
about network structure: A Bayesian approach
1846–1856.
to structure discovery in Bayesian networks. MaErwin, E., K. Obermayer, and K. Schulten (1992).
chine Learning 50, 95–126.
Self-organizing maps: ordering, convergence
Frydenberg, M. (1990). The chain graph Markov
properties and energy functions. Biological Cyproperty. Scandinavian Journal of Statistics 17,
bernetics 67, 47–55.
333–353.
Everitt, B. S. (1984). An Introduction to Latent Vari- Fukunaga, K. (1990). Introduction to Statistical Patable Models. Chapman and Hall.
tern Recognition (Second ed.). Academic Press.
at the jackknife. Annals of Statistics 7, 1–26.

Faul, A. C. and M. E. Tipping (2002). Analysis of Funahashi, K. (1989). On the approximate realizasparse Bayesian learning. In T. G. Dietterich,tion of continuous mappings by neural networks.
S. Becker, and Z. Ghahramani (Eds.), Advances
Neural Networks 2(3), 183–192.
in Neural Information Processing Systems, VolFung, R. and K. C. Chang (1990). Weighting andume 14, pp. 383–389. MIT Press.
integrating evidence for stochastic simulation inFeller, W. (1966). An Introduction to Probability
Bayesian networks. In P. P. Bonissone, M. HenTheory and its Applications (Second ed.), Volrion, L. N. Kanal, and J. F. Lemmer (Eds.), Unume 2. Wiley.
certainty in Artiﬁcial Intelligence, Volume 5, pp.
208–219. Elsevier.
Feynman, R. P., R. B. Leighton, and M. Sands(1964). The Feynman Lectures of Physics, Vol- Gallager, R. G. (1963). Low-Density Parity-Check
Codes. MIT Press.
ume Two. Addison-Wesley. Chapter 19.

REFERENCES

717

Gamerman, D. (1997). Markov Chain Monte Carlo: Gilks, W. R. (1992). Derivative-free adaptive
Stochastic Simulation for Bayesian Inference.
rejection sampling for Gibbs sampling. In
Chapman and Hall.
J. Bernardo, J. Berger, A. P. Dawid, and A. F. M.
Smith (Eds.), Bayesian Statistics, Volume 4. OxGelman, A., J. B. Carlin, H. S. Stern, and D. B. Ruford University Press.
bin (2004). Bayesian Data Analysis (Second ed.).
Gilks, W. R., N. G. Best, and K. K. C. Tan (1995).
Chapman and Hall.
Adaptive rejection Metropolis sampling. Applied
Geman, S. and D. Geman (1984). Stochastic reStatistics 44, 455–472.
laxation, Gibbs distributions, and the Bayesian
restoration of images. IEEE Transactions on Pat- Gilks, W. R., S. Richardson, and D. J. Spiegelhalter (Eds.) (1996). Markov Chain Monte Carlo intern Analysis and Machine Intelligence 6(1),Practice. Chapman and Hall.
721–741.
Ghahramani, Z. and M. J. Beal (2000). Variational Gilks, W. R. and P. Wild (1992). Adaptive rejection
sampling for Gibbs sampling. Applied Statisinference for Bayesian mixtures of factor anatics 41, 337–348.
lyzers. In S. A. Solla, T. K. Leen, and K. R.
Müller (Eds.), Advances in Neural Information Gill, P. E., W. Murray, and M. H. Wright (1981).
Processing Systems, Volume 12, pp. 449–455.
Practical Optimization. Academic Press.
MIT Press.
Goldberg, P. W., C. K. I. Williams, and C. M.
Ghahramani, Z. and G. E. Hinton (1996a). The
Bishop (1998). Regression with input-dependent
EM algorithm for mixtures of factor analyzers.
noise: A Gaussian process treatment. In AdTechnical Report CRG-TR-96-1, University ofvances in Neural Information Processing SysToronto.
tems, Volume 10, pp. 493–499. MIT Press.
Ghahramani, Z. and G. E. Hinton (1996b). Param- Golub, G. H. and C. F. Van Loan (1996). Matrixeter estimation for linear dynamical systems.
Computations (Third ed.). John Hopkins UniverTechnical Report CRG-TR-96-2, University ofsity Press.
Toronto.
Good, I. (1950). Probability and the Weighing of EvGhahramani, Z. and G. E. Hinton (1998). Variational
idence. Hafners.
learning for switching state-space models. Neu- Gordon, N. J., D. J. Salmond, and A. F. M. Smith
ral Computation 12(4), 963–996.
(1993). Novel approach to nonlinear/nonGhahramani, Z. and M. I. Jordan (1994). SuperGaussian Bayesian state estimation. IEE
vised learning from incomplete data via an EM
Proceedings-F 140(2), 107–113.
appproach. In J. D. Cowan, G. T. Tesauro, and Graepel, T. (2003). Solving noisy linear operator
J. Alspector (Eds.), Advances in Neural Informaequations by Gaussian processes: Application
tion Processing Systems, Volume 6, pp. 120–127.
to ordinary and partial differential equations. In
Morgan Kaufmann.
Proceedings of the Twentieth International ConGhahramani, Z. and M. I. Jordan (1997). Factorial
ference on Machine Learning, pp. 234–241.
hidden Markov models. Machine Learning 29, Greig, D., B. Porteous, and A. Seheult (1989). Ex245–275.
act maximum a-posteriori estimation for binary
Gibbs, M. N. (1997). Bayesian Gaussian processes
images. Journal of the Royal Statistical Society,for regression and classiﬁcation. Phd thesis, UniSeries B 51(2), 271–279.
versity of Cambridge.
Gull, S. F. (1989). Developments in maximum entropy data analysis. In J. Skilling (Ed.), MaxiGibbs, M. N. and D. J. C. MacKay (2000). Variamum Entropy and Bayesian Methods, pp. 53–71.
tional Gaussian process classiﬁers. IEEE TransKluwer.
actions on Neural Networks 11, 1458–1464.

718

REFERENCES

Hassibi, B. and D. G. Stork (1993). Second orderof the International Conference on Independent
derivatives for network pruning: optimal brain
Component Analysis and Blind Signal Separasurgeon. In S. J. Hanson, J. D. Cowan, andtion, Volume 3.
C. L. Giles (Eds.), Advances in Neural Informa- Hodgson, M. E. (1998). Reducing computational retion Processing Systems, Volume 5, pp. 164–171.
quirements of the minimum-distance classiﬁer.
Morgan Kaufmann.
Remote Sensing of Environments 25, 117–128.
Hastie, T. and W. Stuetzle (1989). Principal curves.
Hoerl, A. E. and R. Kennard (1970). Ridge regresJournal of the American Statistical Associasion: biased estimation for nonorthogonal probtion 84(106), 502–516.
lems. Technometrics 12, 55–67.
Hastie, T., R. Tibshirani, and J. Friedman (2001).
The Elements of Statistical Learning. Springer. Hofmann, T. (2000). Learning the similarity of documents: an information-geometric approach to
Hastings, W. K. (1970). Monte Carlo sampling
document retrieval and classiﬁcation. In S. A.
methods using Markov chains and their applicaSolla, T. K. Leen, and K. R. Müller (Eds.), Adtions. Biometrika 57, 97–109.
vances in Neural Information Processing Systems, Volume 12, pp. 914–920. MIT Press.
Hathaway, R. J. (1986). Another interpretation of the
EM algorithm for mixture distributions. Statistics Hojen-Sorensen, P. A., O. Winther, and L. K. Hansen
and Probability Letters 4, 53–56.
(2002). Mean ﬁeld approaches to independent
Haussler, D. (1999). Convolution kernels on discrete
component analysis. Neural Computation 14(4),structures. Technical Report UCSC-CRL-99-10,
889–918.
University of California, Santa Cruz, Computer
Hornik, K. (1991). Approximation capabilities of
Science Department.
multilayer feedforward networks. Neural NetHenrion, M. (1988). Propagation of uncertainty byworks 4(2), 251–257.
logic sampling in Bayes’ networks. In J. F. LemHornik, K., M. Stinchcombe, and H. White (1989).
mer and L. N. Kanal (Eds.), Uncertainty in ArtiMultilayer feedforward networks are universal
ﬁcial Intelligence, Volume 2, pp. 149–164. North
approximators. Neural Networks 2(5), 359–366.
Holland.
Herbrich, R. (2002). Learning Kernel Classiﬁers. Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. JourMIT Press.
nal of Educational Psychology 24, 417–441.
Hertz, J., A. Krogh, and R. G. Palmer (1991). Introduction to the Theory of Neural Computation. Hotelling, H. (1936). Relations between two sets ofvariables. Biometrika 28, 321–377.
Addison Wesley.
Hinton, G. E., P. Dayan, and M. Revow (1997). Hyvärinen, A. and E. Oja (1997). A fast ﬁxed-point
algorithm for independent component analysis.
Modelling the manifolds of images of handwritNeural Computation 9(7), 1483–1492.
ten digits. IEEE Transactions on Neural Networks 8(1), 65–74.
Isard, M. and A. Blake (1998). CONDENSATION– conditional density propagation for visual
Hinton, G. E. and D. van Camp (1993). Keeping
tracking. International Journal of Computer Vineural networks simple by minimizing the desion 29(1), 5–18.
scription length of the weights. In Proceedings of
the Sixth Annual Conference on Computational Ito, Y. (1991). Representation of functions by suLearning Theory, pp. 5–13. ACM.
perpositions of a step or sigmoid function and
Hinton, G. E., M. Welling, Y. W. Teh, and S. Osindero (2001). A new view of ICA. In Proceedings

their applications to neural network theory. Neural Networks 4(3), 385–394.

REFERENCES

719

Jaakkola, T. and M. I. Jordan (2000). Bayesian Jordan, M. I. (2007). An Introduction to Probabilisparameter estimation via variational methods.
tic Graphical Models. In preparation.
Statistics and Computing 10, 25–37.
Jordan, M. I., Z. Ghahramani, T. S. Jaakkola, andJaakkola, T. S. (2001). Tutorial on variational approximation methods. In M. Opper and D. Saad
(Eds.), Advances in Mean Field Methods, pp.
129–159. MIT Press.

L. K. Saul (1999). An introduction to variational
methods for graphical models. In M. I. Jordan(Ed.), Learning in Graphical Models, pp. 105–
162. MIT Press.

Jaakkola, T. S. and D. Haussler (1999). Exploiting Jordan, M. I. and R. A. Jacobs (1994). Hierarchicalgenerative models in discriminative classiﬁers. In
mixtures of experts and the EM algorithm. NeuM. S. Kearns, S. A. Solla, and D. A. Cohn (Eds.),
ral Computation 6(2), 181–214.
Advances in Neural Information Processing SysJutten, C. and J. Herault (1991). Blind separation oftems, Volume 11. MIT Press.
sources, 1: An adaptive algorithm based on neuJacobs, R. A., M. I. Jordan, S. J. Nowlan, and G. E.
romimetic architecture. Signal Processing 24(1),Hinton (1991). Adaptive mixtures of local ex1–10.
perts. Neural Computation 3(1), 79–87.
Kalman, R. E. (1960). A new approach to linear ﬁlJaynes, E. T. (2003). Probability Theory: The Logic
tering and prediction problems. Transactions ofof Science. Cambridge University Press.
the American Society for Mechanical Engineering, Series D, Journal of Basic Engineering 82,
Jebara, T. (2004). Machine Learning: Discrimina35–45.
tive and Generative. Kluwer.
Jeffries, H. (1946). An invariant form for the prior Kambhatla, N. and T. K. Leen (1997). Dimension
reduction by local principal component analysis.
probability in estimation problems. Pro. Roy.
Neural Computation 9(7), 1493–1516.
Soc. AA 186, 453–461.
Jelinek, F. (1997). Statistical Methods for Speech Kanazawa, K., D. Koller, and S. Russel (1995).
Stochastic simulation algorithms for dynamic
Recognition. MIT Press.
probabilistic networks. In Uncertainty in ArtiﬁJensen, C., A. Kong, and U. Kjaerulff (1995). Blockcial Intelligence, Volume 11. Morgan Kaufmann.
ing gibbs sampling in very large probabilistic
expert systems. International Journal of Human Kapadia, S. (1998). Discriminative Training of Hidden Markov Models. Phd thesis, University ofComputer Studies. Special Issue on Real-World
Cambridge, U.K.
Applications of Uncertain Reasoning. 42, 647–
666.
Kapur, J. (1989). Maximum entropy methods in science and engineering. Wiley.
Jensen, F. V. (1996). An Introduction to Bayesian
Networks. UCL Press.
Karush, W. (1939). Minima of functions of several
variables with inequalities as side constraints.
Jerrum, M. and A. Sinclair (1996). The Markov
Master’s thesis, Department of Mathematics,
chain Monte Carlo method: an approach to apUniversity of Chicago.
proximate counting and integration. In D. S.
Hochbaum (Ed.), Approximation Algorithms for Kass, R. E. and A. E. Raftery (1995). Bayes facNP-Hard Problems. PWS Publishing.
tors. Journal of the American Statistical Association 90, 377–395.
Jolliffe, I. T. (2002). Principal Component Analysis(Second ed.). Springer.
Kearns, M. J. and U. V. Vazirani (1994). An IntroJordan, M. I. (1999). Learning in Graphical Models.
MIT Press.

duction to Computational Learning Theory. MIT
Press.

720

REFERENCES

Kindermann, R. and J. L. Snell (1980). Markov RanIn Advances in Neural Information Processing
dom Fields and Their Applications. American
Systems, Number 18. MIT Press. in press.
Mathematical Society.
Lasserre, J., C. M. Bishop, and T. Minka (2006).
Principled hybrids of generative and discriminaKittler, J. and J. Föglein (1984). Contextual classiﬁtive models. In Proceedings 2006 IEEE Confercation of multispectral pixel data. Image and Vienceon Computer Vision and Pattern Recognision Computing 2, 13–29.
tion, New York.
Kohonen, T. (1982). Self-organized formation of
topologically correct feature maps. Biological Lauritzen, S. and N. Wermuth (1989). Graphicalmodels for association between variables, someCybernetics 43, 59–69.
of which are qualitative some quantitative. AnKohonen, T. (1995). Self-Organizing Maps.
nals of Statistics 17, 31–57.
Springer.
Lauritzen, S. L. (1992). Propagation of probabilities,
Kolmogorov, V. and R. Zabih (2004). What enmeans and variances in mixed graphical associaergy functions can be minimized via graph cuts?
tion models. Journal of the American StatisticalIEEE Transactions on Pattern Analysis and MaAssociation 87, 1098–1108.
chine Intelligence 26(2), 147–159.
Lauritzen, S. L. (1996). Graphical Models. Oxford
University Press.
Kreinovich, V. Y. (1991). Arbitrary nonlinearity is
sufﬁcient to represent all functions by neural net- Lauritzen, S. L. and D. J. Spiegelhalter (1988). Loworks: a theorem. Neural Networks 4(3), 381–
cal computations with probabailities on graphical383.
structures and their application to expert systems.
Journal of the Royal Statistical Society 50, 157–
Krogh, A., M. Brown, I. S. Mian, K. Sjölander, and224.
D. Haussler (1994). Hidden Markov models incomputational biology: Applications to protein Lawley, D. N. (1953). A modiﬁed method of estimodelling. Journal of Molecular Biology 235,
mation in factor analysis and some large sam1501–1531.
ple results. In Uppsala Symposium on Psychological Factor Analysis, Number 3 in Nordisk
Kschischnang, F. R., B. J. Frey, and H. A. Loeliger
Psykologi Monograph Series, pp. 35–42. Upp(2001). Factor graphs and the sum-product algosala: Almqvist and Wiksell.
rithm. IEEE Transactions on Information TheLawrence, N. D., A. I. T. Rowstron, C. M. Bishop,
ory 47(2), 498–519.
and M. J. Taylor (2002). Optimising synchroKuhn, H. W. and A. W. Tucker (1951). Nonlinear
nisation times for mobile devices. In T. G. Diprogramming. In Proceedings of the 2nd Berkeetterich, S. Becker, and Z. Ghahramani (Eds.),
ley Symposium on Mathematical Statistics andAdvances in Neural Information Processing SysProbabilities, pp. 481–492. University of Calitems, Volume 14, pp. 1401–1408. MIT Press.
fornia Press.
Lazarsfeld, P. F. and N. W. Henry (1968). Latent
Kullback, S. and R. A. Leibler (1951). On inforStructure Analysis. Houghton Mifﬂin.
mation and sufﬁciency. Annals of Mathematical
Le Cun, Y., B. Boser, J. S. Denker, D. Henderson,
Statistics 22(1), 79–86.
R. E. Howard, W. Hubbard, and L. D. JackelKu̇rková, V. and P. C. Kainen (1994). Functionally(1989). Backpropagation applied to handwritten
equivalent feed-forward neural networks. Neural
zip code recognition. Neural Computation 1(4),Computation 6(3), 543–558.
541–551.
Kuss, M. and C. Rasmussen (2006). Assessing ap- Le Cun, Y., J. S. Denker, and S. A. Solla (1990).
Optimal brain damage. In D. S. Touretzky (Ed.),
proximations for Gaussian process classiﬁcation.

REFERENCES

721

Advances in Neural Information Processing SysDepartment of Physics, University of Camtems, Volume 2, pp. 598–605. Morgan Kaufbridge.
mann.
MacKay, D. J. C. (1998). Introduction to GausLeCun, Y., L. Bottou, Y. Bengio, and P. Haffner
sian processes. In C. M. Bishop (Ed.), Neural(1998). Gradient-based learning applied to docNetworks and Machine Learning, pp. 133–166.
ument recognition. Proceedings of the IEEE 86,
Springer.
2278–2324.
MacKay, D. J. C. (1999). Comparison of approxLee, Y., Y. Lin, and G. Wahba (2001). Multicategory
imate methods for handling hyperparameters.
support vector machines. Technical Report 1040,
Neural Computation 11(5), 1035–1068.
Department of Statistics, University of Madison,
MacKay, D. J. C. (2003). Information Theory, InferWisconsin.
ence and Learning Algorithms. Cambridge UniLeen, T. K. (1995). From data distributions to reguversity Press.
larization in invariant learning. Neural ComputaMacKay, D. J. C. and M. N. Gibbs (1999). Dention 7, 974–981.
sity networks. In J. W. Kay and D. M. TitteringLindley, D. V. (1982). Scoring rules and the inton (Eds.), Statistics and Neural Networks: Adevitability of probability. International Statistivances at the Interface, Chapter 5, pp. 129–145.
cal Review 50, 1–26.
Oxford University Press.
Liu, J. S. (Ed.) (2001). Monte Carlo Strategies in MacKay, D. J. C. and R. M. Neal (1999). Good errorScientiﬁc Computing. Springer.
correcting codes based on very sparse matrices.
IEEE Transactions on Information Theory 45,
Lloyd, S. P. (1982). Least squares quantization in
399–431.
PCM. IEEE Transactions on Information Theory 28(2), 129–137.
MacQueen, J. (1967). Some methods for classiﬁcation and analysis of multivariate observations. In
Lütkepohl, H. (1996). Handbook of Matrices. Wiley.
L. M. LeCam and J. Neyman (Eds.), ProceedMacKay, D. J. C. (1992a). Bayesian interpolation.
ings of the Fifth Berkeley Symposium on MatheNeural Computation 4(3), 415–447.
matical Statistics and Probability, Volume I, pp.
MacKay, D. J. C. (1992b). The evidence framework
281–297. University of California Press.
applied to classiﬁcation networks. Neural ComMagnus, J. R. and H. Neudecker (1999). Matrix Difputation 4(5), 720–736.
ferential Calculus with Applications in Statistics
MacKay, D. J. C. (1992c). A practical Bayesianand Econometrics. Wiley.
framework for back-propagation networks. NeuMallat, S. (1999). A Wavelet Tour of Signal Processral Computation 4(3), 448–472.
ing (Second ed.). Academic Press.
MacKay, D. J. C. (1994). Bayesian methods for
Manning, C. D. and H. Schütze (1999). Foundations
backprop networks. In E. Domany, J. L. vanof Statistical Natural Language Processing. MIT
Hemmen, and K. Schulten (Eds.), Models of
Press.
Neural Networks, III, Chapter 6, pp. 211–254.
Mardia, K. V. and P. E. Jupp (2000). Directional
Springer.
Statistics. Wiley.
MacKay, D. J. C. (1995). Bayesian neural networks
and density networks. Nuclear Instruments and Maybeck, P. S. (1982). Stochastic models, estimation and control. Academic Press.
Methods in Physics Research, A 354(1), 73–80.
MacKay, D. J. C. (1997). Ensemble learning for hid- McAllester, D. A. (2003). PAC-Bayesian stochastic
den Markov models. Unpublished manuscript,
model selection. Machine Learning 51(1), 5–21.

722

REFERENCES

McCullagh, P. and J. A. Nelder (1989). Generalized Minka, T. (2004). Power EP. Technical ReportLinear Models (Second ed.). Chapman and Hall.
MSR-TR-2004-149, Microsoft Research Cambridge.
McCulloch, W. S. and W. Pitts (1943). A logicalcalculus of the ideas immanent in nervous ac- Minka, T. (2005). Divergence measures and mestivity. Bulletin of Mathematical Biophysics 5,sage passing. Technical Report MSR-TR-2005115–133. Reprinted in Anderson and Rosenfeld
173, Microsoft Research Cambridge.
(1988).
Minka, T. P. (2001c). Automatic choice of dimenMcEliece, R. J., D. J. C. MacKay, and J. F. Cheng
sionality for PCA. In T. K. Leen, T. G. Diet(1998). Turbo decoding as an instance of Pearl’s
terich, and V. Tresp (Eds.), Advances in Neural‘Belief Ppropagation’ algorithm. IEEE Journal
Information Processing Systems, Volume 13, pp.
on Selected Areas in Communications 16, 140–
598–604. MIT Press.
152.
Minsky, M. L. and S. A. Papert (1969). Perceptrons.
McLachlan, G. J. and K. E. Basford (1988). Mixture
MIT Press. Expanded edition 1990.
Models: Inference and Applications to ClusterMiskin, J. W. and D. J. C. MacKay (2001). Enseming. Marcel Dekker.
ble learning for blind source separation. In S. J.
McLachlan, G. J. and T. Krishnan (1997). The EM
Roberts and R. M. Everson (Eds.), Independent
Algorithm and its Extensions. Wiley.
Component Analysis: Principles and Practice.
McLachlan, G. J. and D. Peel (2000). Finite Mixture
Cambridge University Press.
Models. Wiley.
Meng, X. L. and D. B. Rubin (1993). Maximum like- Møller, M. (1993). Efﬁcient Training of FeedForward Neural Networks. Ph. D. thesis, Aarhus
lihood estimation via the ECM algorithm: a genUniversity, Denmark.
eral framework. Biometrika 80, 267–278.
Metropolis, N., A. W. Rosenbluth, M. N. Rosen- Moody, J. and C. J. Darken (1989). Fast learning in
networks of locally-tuned processing units. Neubluth, A. H. Teller, and E. Teller (1953). Equaral Computation 1(2), 281–294.
tion of state calculations by fast computing
machines. Journal of Chemical Physics 21(6), Moore, A. W. (2000). The anchors hierarch: us1087–1092.
ing the triangle inequality to survive high dimenMetropolis, N. and S. Ulam (1949). The Montesional data. In Proceedings of the Twelfth ConCarlo method. Journal of the American Statistiference on Uncertainty in Artiﬁcial Intelligence,cal Association 44(247), 335–341.
pp. 397–405.
Mika, S., G. Rätsch, J. Weston, and B. Schölkopf Müller, K. R., S. Mika, G. Rätsch, K. Tsuda, and(1999). Fisher discriminant analysis with kerB. Schölkopf (2001). An introduction to kernelnels. In Y. H. Hu, J. Larsen, E. Wilson, andbased learning algorithms. IEEE Transactions on
S. Douglas (Eds.), Neural Networks for Signal
Neural Networks 12(2), 181–202.
Processing IX, pp. 41–48. IEEE.
Müller, P. and F. A. Quintana (2004). Nonparametric
Minka, T. (2001a). Expectation propagation for apBayesian data analysis. Statistical Science 19(1),proximate Bayesian inference. In J. Breese and
95–110.
D. Koller (Eds.), Proceedings of the Seventeenth
Conference on Uncertainty in Artiﬁcial Intelli- Nabney, I. T. (2002). Netlab: Algorithms for Pattern
Recognition. Springer.
gence, pp. 362–369. Morgan Kaufmann.
Minka, T. (2001b). A family of approximate al- Nadaraya, É. A. (1964). On estimating regression.
gorithms for Bayesian inference. Ph. D. thesis,
Theory of Probability and its Applications 9(1),MIT.
141–142.

REFERENCES

723

Nag, R., K. Wong, and F. Fallside (1986). Script Ogden, R. T. (1997). Essential Wavelets for Statistirecognition using hidden markov models. In
cal Applications and Data Analysis. Birkhäuser.
ICASSP86, pp. 2071–2074. IEEE.
Opper, M. and O. Winther (1999). A Bayesian approach to on-line learning. In D. Saad (Ed.), OnNeal, R. M. (1993). Probabilistic inference usingLine Learning in Neural Networks, pp. 363–378.
Markov chain Monte Carlo methods. TechnicalCambridge University Press.
Report CRG-TR-93-1, Department of Computer
Science, University of Toronto, Canada.
Opper, M. and O. Winther (2000a). Gaussian
processes and SVM: mean ﬁeld theory and
Neal, R. M. (1996). Bayesian Learning for Neuralleave-one-out. In A. J. Smola, P. L. Bartlett,Networks. Springer. Lecture Notes in Statistics
B. Schölkopf, and D. Shuurmans (Eds.), Ad118.
vances in Large Margin Classiﬁers, pp. 311–326.
Neal, R. M. (1997). Monte Carlo implementation of
MIT Press.
Gaussian process models for Bayesian regression
Opper, M. and O. Winther (2000b). Gaussianand classiﬁcation. Technical Report 9702, Deprocesses for classiﬁcation. Neural Computapartment of Computer Statistics, University oftion 12(11), 2655–2684.
Toronto.
Osuna, E., R. Freund, and F. Girosi (1996). Support
Neal, R. M. (1999). Suppressing random walks invector machines: training and applications. A.I.
Markov chain Monte Carlo using ordered overMemo AIM-1602, MIT.
relaxation. In M. I. Jordan (Ed.), Learning inPapoulis, A. (1984). Probability, Random Variables,
Graphical Models, pp. 205–228. MIT Press.
and Stochastic Processes (Second ed.). McGrawNeal, R. M. (2000). Markov chain sampling for
Hill.
Dirichlet process mixture models. Journal of Parisi, G. (1988). Statistical Field Theory. AddisonComputational and Graphical Statistics 9, 249–
Wesley.
265.
Pearl, J. (1988). Probabilistic Reasoning in IntelliNeal, R. M. (2003). Slice sampling. Annals of Statisgent Systems. Morgan Kaufmann.
tics 31, 705–767.
Pearlmutter, B. A. (1994). Fast exact multiplicationby the Hessian. Neural Computation 6(1), 147–
Neal, R. M. and G. E. Hinton (1999). A new view of160.
the EM algorithm that justiﬁes incremental andother variants. In M. I. Jordan (Ed.), Learning in Pearlmutter, B. A. and L. C. Parra (1997). Maximum
Graphical Models, pp. 355–368. MIT Press.
likelihood source separation: a context-sensitive
generalization of ICA. In M. C. Mozer, M. I. JorNelder, J. A. and R. W. M. Wedderburn (1972). Gendan, and T. Petsche (Eds.), Advances in Neuraleralized linear models. Journal of the Royal StaInformation Processing Systems, Volume 9, pp.
tistical Society, A 135, 370–384.
613–619. MIT Press.
Nilsson, N. J. (1965). Learning Machines. McGrawHill. Reprinted as The Mathematical Founda- Pearson, K. (1901). On lines and planes of closest ﬁt
to systems of points in space. The London, Edintions of Learning Machines, Morgan Kaufmann,
burgh and Dublin Philosophical Magazine and(1990).
Journal of Science, Sixth Series 2, 559–572.
Nocedal, J. and S. J. Wright (1999). Numerical Op- Platt, J. C. (1999). Fast training of support vector
timization. Springer.
machines using sequential minimal optimization.
In B. Schölkopf, C. J. C. Burges, and A. J. Smola
Nowlan, S. J. and G. E. Hinton (1992). Simplifying(Eds.), Advances in Kernel Methods – Support
neural networks by soft weight sharing. Neural
Vector Learning, pp. 185–208. MIT Press.
Computation 4(4), 473–493.

724

REFERENCES

Platt, J. C. (2000). Probabilities for SV machines. Ramsey, F. (1931). Truth and probability. In
In A. J. Smola, P. L. Bartlett, B. Schölkopf, and
R. Braithwaite (Ed.), The Foundations of MathD. Shuurmans (Eds.), Advances in Large Margin
ematics and other Logical Essays. Humanities
Classiﬁers, pp. 61–73. MIT Press.
Press.
Platt, J. C., N. Cristianini, and J. Shawe-Taylor Rao, C. R. and S. K. Mitra (1971). Generalized In(2000). Large margin DAGs for multiclass clasverse of Matrices and Its Applications. Wiley.
siﬁcation. In S. A. Solla, T. K. Leen, and K. R.
Müller (Eds.), Advances in Neural Information Rasmussen, C. E. (1996). Evaluation of GaussianProcesses and Other Methods for Non-Linear
Processing Systems, Volume 12, pp. 547–553.
Regression. Ph. D. thesis, University of Toronto.
MIT Press.
Poggio, T. and F. Girosi (1990). Networks for ap- Rasmussen, C. E. and J. Quiñonero-Candela (2005).
Healing the relevance vector machine by augproximation and learning. Proceedings of thementation. In L. D. Raedt and S. Wrobel (Eds.),
IEEE 78(9), 1481–1497.
Proceedings of the 22nd International ConferPowell, M. J. D. (1987). Radial basis functions forence on Machine Learning, pp. 689–696.
multivariable interpolation: a review. In J. C.
Mason and M. G. Cox (Eds.), Algorithms for Rasmussen, C. E. and C. K. I. Williams (2006).
Approximation, pp. 143–167. Oxford University
Gaussian Processes for Machine Learning. MIT
Press.
Press.
Press, W. H., S. A. Teukolsky, W. T. Vetterling, and Rauch, H. E., F. Tung, and C. T. Striebel (1965).
B. P. Flannery (1992). Numerical Recipes in C:
Maximum likelihood estimates of linear dynamThe Art of Scientiﬁc Computing (Second ed.).
ical systems. AIAA Journal 3, 1445–1450.
Cambridge University Press.
Ricotti, L. P., S. Ragazzini, and G. Martinelli (1988).
Qazaz, C. S., C. K. I. Williams, and C. M. Bishop
Learning of word stress in a sub-optimal second(1997). An upper bound on the Bayesian error
order backpropagation neural network. In Probars for generalized linear regression. In S. W.
ceedings of the IEEE International Conference
Ellacott, J. C. Mason, and I. J. Anderson (Eds.),
on Neural Networks, Volume 1, pp. 355–361.
Mathematics of Neural Networks: Models, AlgoIEEE.
rithms and Applications, pp. 295–299. Kluwer.
Quinlan, J. R. (1986). Induction of decision trees. Ripley, B. D. (1996). Pattern Recognition and Neural Networks. Cambridge University Press.
Machine Learning 1(1), 81–106.
Quinlan, J. R. (1993). C4.5: Programs for Machine Robbins, H. and S. Monro (1951). A stochastic
approximation method. Annals of Mathematical
Learning. Morgan Kaufmann.
Statistics 22, 400–407.
Rabiner, L. and B. H. Juang (1993). Fundamentalsof Speech Recognition. Prentice Hall.
Robert, C. P. and G. Casella (1999). Monte Carlo
Statistical Methods. Springer.
Rabiner, L. R. (1989). A tutorial on hidden Markov
models and selected applications in speech Rockafellar, R. (1972). Convex Analysis. Princetonrecognition. Proceedings of the IEEE 77(2),University Press.
257–285.
Rosenblatt, F. (1962). Principles of NeurodynamRamasubramanian, V. and K. K. Paliwal (1990). A
ics: Perceptrons and the Theory of Brain Mechgeneralized optimization of the k-d tree for fast
anisms. Spartan.
nearest-neighbour search. In Proceedings FourthIEEE Region 10 International Conference (TEN- Roth, V. and V. Steinhage (2000). Nonlinear discrimCON’89), pp. 565–568.
inant analysis using kernel functions. In S. A.

REFERENCES

725

Solla, T. K. Leen, and K. R. Müller (Eds.), Ad- Schölkopf, B., A. Smola, and K.-R. Müller (1998).
Nonlinear component analysis as a kernelvances in Neural Information Processing Syseigenvalue problem. Neural Computation 10(5),tems, Volume 12. MIT Press.
1299–1319.
Roweis, S. (1998). EM algorithms for PCA and
SPCA. In M. I. Jordan, M. J. Kearns, and S. A. Schölkopf, B., A. Smola, R. C. Williamson, and P. L.
Solla (Eds.), Advances in Neural Information
Bartlett (2000). New support vector algorithms.
Processing Systems, Volume 10, pp. 626–632.
Neural Computation 12(5), 1207–1245.
MIT Press.
Schölkopf, B. and A. J. Smola (2002). Learning with
Roweis, S. and Z. Ghahramani (1999). A unifying
Kernels. MIT Press.
review of linear Gaussian models. Neural ComSchwarz, G. (1978). Estimating the dimension of aputation 11(2), 305–345.
model. Annals of Statistics 6, 461–464.
Roweis, S. and L. Saul (2000, December). Nonlinear
dimensionality reduction by locally linear em- Schwarz, H. R. (1988). Finite element methods. Academic Press.
bedding. Science 290, 2323–2326.
Rubin, D. B. (1983). Iteratively reweighted least Seeger, M. (2003). Bayesian Gaussian Process Models: PAC-Bayesian Generalization Error Bounds
squares. In Encyclopedia of Statistical Sciences,
andSparse Approximations. Ph. D. thesis, UniVolume 4, pp. 272–275. Wiley.
versity of Edinburg.
Rubin, D. B. and D. T. Thayer (1982). EM algorithms for ML factor analysis. Psychome- Seeger, M., C. K. I. Williams, and N. Lawrence(2003). Fast forward selection to speed up sparse
trika 47(1), 69–76.
Gaussian processes. In C. M. Bishop and B. Frey
Rumelhart, D. E., G. E. Hinton, and R. J. Williams(Eds.), Proceedings Ninth International Work(1986). Learning internal representations by ershop on Artiﬁcial Intelligence and Statistics, Key
ror propagation. In D. E. Rumelhart, J. L. McWest, Florida.
Clelland, and the PDP Research Group (Eds.),
Parallel Distributed Processing: Explorations Shachter, R. D. and M. Peot (1990). Simulation approaches to general probabilistic inference on bein the Microstructure of Cognition, Volume 1:
lief networks. In P. P. Bonissone, M. Henrion,
Foundations, pp. 318–362. MIT Press. Reprinted
L. N. Kanal, and J. F. Lemmer (Eds.), Uncerin Anderson and Rosenfeld (1988).
tainty in Artiﬁcial Intelligence, Volume 5. ElseRumelhart, D. E., J. L. McClelland, and the PDP Revier.
search Group (Eds.) (1986). Parallel Distributed
Processing: Explorations in the Microstruc- Shannon, C. E. (1948). A mathematical theory ofcommunication. The Bell System Technical Jourture of Cognition, Volume 1: Foundations. MIT
nal 27(3), 379–423 and 623–656.
Press.
Sagan, H. (1969). Introduction to the Calculus of Shawe-Taylor, J. and N. Cristianini (2004). Kernel
Methods for Pattern Analysis. Cambridge UniVariations. Dover.
versity Press.
Savage, L. J. (1961). The subjective basis of statistical practice. Technical report, Department of Sietsma, J. and R. J. F. Dow (1991). Creating artiﬁcial neural networks that generalize. Neural NetStatistics, University of Michigan, Ann Arbor.
works 4(1), 67–79.
Schölkopf, B., J. Platt, J. Shawe-Taylor, A. Smola,and R. C. Williamson (2001). Estimating the sup- Simard, P., Y. Le Cun, and J. Denker (1993). Efﬁport of a high-dimensional distribution. Neuralcient pattern recognition using a new transformaComputation 13(7), 1433–1471.
tion distance. In S. J. Hanson, J. D. Cowan, and726

REFERENCES

C. L. Giles (Eds.), Advances in Neural Informa- Tarassenko, L. (1995). Novelty detection for thetion Processing Systems, Volume 5, pp. 50–58.
identiﬁcation of masses in mamograms. In ProMorgan Kaufmann.
ceedings Fourth IEE International Conferenceon Artiﬁcial Neural Networks, Volume 4, pp.
Simard, P., B. Victorri, Y. Le Cun, and J. Denker
442–447. IEE.
(1992). Tangent prop – a formalism for specifying selected invariances in an adaptive network. Tax, D. and R. Duin (1999). Data domain descripIn J. E. Moody, S. J. Hanson, and R. P. Lippmanntion by support vectors. In M. Verleysen (Ed.),(Eds.), Advances in Neural Information ProcessProceedings European Symposium on Artiﬁcial
ing Systems, Volume 4, pp. 895–903. Morgan
Neural Networks, ESANN, pp. 251–256. D. Facto
Kaufmann.
Press.
Simard, P. Y., D. Steinkraus, and J. Platt (2003). Teh, Y. W., M. I. Jordan, M. J. Beal, and D. M. Blei
Best practice for convolutional neural networks(2006). Hierarchical Dirichlet processes. Journal
applied to visual document analysis. In Proof the Americal Statistical Association. to appear.
ceedings International Conference on Document Tenenbaum, J. B., V. de Silva, and J. C. Langford
Analysis and Recognition (ICDAR), pp. 958–
(2000, December). A global framework for non962. IEEE Computer Society.
linear dimensionality reduction. Science 290,
Sirovich, L. (1987). Turbulence and the dynamics
2319–2323.
of coherent structures. Quarterly Applied Math- Tesauro, G. (1994). TD-Gammon, a self-teachingematics 45(3), 561–590.
backgammon program, achieves master-level
Smola, A. J. and P. Bartlett (2001). Sparse greedy
play. Neural Computation 6(2), 215–219.
Gaussian process regression. In T. K. Leen, T. G. Thiesson, B., D. M. Chickering, D. Heckerman, andDietterich, and V. Tresp (Eds.), Advances in NeuC. Meek (2004). ARMA time-series modellingral Information Processing Systems, Volume 13,
with graphical models. In M. Chickering andpp. 619–625. MIT Press.
J. Halpern (Eds.), Proceedings of the Twentieth
Conference on Uncertainty in Artiﬁcial IntelliSpiegelhalter, D. and S. Lauritzen (1990). Sequential
gence, Banff, Canada, pp. 552–560. AUAI Press.
updating of conditional probabilities on directed
graphical structures. Networks 20, 579–605.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal StatisStinchecombe, M. and H. White (1989). Universal
tical Society, B 58, 267–288.
approximation using feed-forward networks with
non-sigmoid hidden layer activation functions. In Tierney, L. (1994). Markov chains for exploring posInternational Joint Conference on Neural Netterior distributions. Annals of Statistics 22(4),works, Volume 1, pp. 613–618. IEEE.
1701–1762.
Stone, J. V. (2004). Independent Component Analy- Tikhonov, A. N. and V. Y. Arsenin (1977). Solutions
sis: A Tutorial Introduction. MIT Press.
of Ill-Posed Problems. V. H. Winston.
Sung, K. K. and T. Poggio (1994). Example-based Tino, P. and I. T. Nabney (2002). Hierarchicallearning for view-based human face detection.
GTM: constructing localized non-linear projecA.I. Memo 1521, MIT.
tion manifolds in a principled way. IEEE TransSutton, R. S. and A. G. Barto (1998). Reinforcement
Learning: An Introduction. MIT Press.

actions on Pattern Analysis and Machine Intelligence 24(5), 639–656.

Svensén, M. and C. M. Bishop (2004). Ro- Tino, P., I. T. Nabney, and Y. Sun (2001). Using directional curvatures to visualize folding
bust Bayesian mixture modelling. Neurocomputpatterns of the GTM projection manifolds. In
ing 64, 235–252.

REFERENCES

727

G. Dorffner, H. Bischof, and K. Hornik (Eds.), Vapnik, V. N. (1982). Estimation of dependences
Artiﬁcial Neural Networks – ICANN 2001, pp.
based on empirical data. Springer.
421–428. Springer.
Vapnik, V. N. (1995). The nature of statistical learning theory. Springer.
Tipping, M. E. (1999). Probabilistic visualisation ofhigh-dimensional binary data. In M. S. Kearns, Vapnik, V. N. (1998). Statistical learning theory. WiS. A. Solla, and D. A. Cohn (Eds.), Advances
ley.
in Neural Information Processing Systems, VolVeropoulos, K., C. Campbell, and N. Cristianiniume 11, pp. 592–598. MIT Press.
(1999). Controlling the sensitivity of support
vector machines. In Proceedings of the InternaTipping, M. E. (2001). Sparse Bayesian learning andtional Joint Conference on Artiﬁcial Intelligence
the relevance vector machine. Journal of Ma(IJCAI99), Workshop ML3, pp. 55–60.
chine Learning Research 1, 211–244.
Tipping, M. E. and C. M. Bishop (1997). Probabilis- Vidakovic, B. (1999). Statistical Modelling by
Wavelets. Wiley.
tic principal component analysis. Technical Report NCRG/97/010, Neural Computing Research Viola, P. and M. Jones (2004). Robust real-time faceGroup, Aston University.
detection. International Journal of Computer Vision 57(2), 137–154.
Tipping, M. E. and C. M. Bishop (1999a). Mixturesof probabilistic principal component analyzers. Viterbi, A. J. (1967). Error bounds for convolutional codes and an asymptotically optimum deNeural Computation 11(2), 443–482.
coding algorithm. IEEE Transactions on InforTipping, M. E. and C. M. Bishop (1999b). Probmation Theory IT-13, 260–267.
abilistic principal component analysis. Journalof the Royal Statistical Society, Series B 21(3), Viterbi, A. J. and J. K. Omura (1979). Principles ofDigital Communication and Coding. McGraw611–622.
Hill.
Tipping, M. E. and A. Faul (2003). Fast marginal
likelihood maximization for sparse Bayesian Wahba, G. (1975). A comparison of GCV and GML
for choosing the smoothing parameter in the genmodels. In C. M. Bishop and B. Frey (Eds.),
eralized spline smoothing problem. Numerical
Proceedings Ninth International Workshop on
Mathematics 24, 383–393.
Artiﬁcial Intelligence and Statistics, Key West,
Wainwright, M. J., T. S. Jaakkola, and A. S. Willsky
Florida.
(2005). A new class of upper bounds on the log
Tong, S. and D. Koller (2000). Restricted Bayes oppartition function. IEEE Transactions on Infortimal classiﬁers. In Proceedings 17th National
mation Theory 51, 2313–2335.
Conference on Artiﬁcial Intelligence, pp. 658–
Walker, A. M. (1969). On the asymptotic behaviour
664. AAAI.
of posterior distributions. Journal of the Royal
Tresp, V. (2001). Scaling kernel-based systems to
Statistical Society, B 31(1), 80–88.
large data sets. Data Mining and Knowledge DisWalker, S. G., P. Damien, P. W. Laud, and A. F. M.
covery 5(3), 197–211.
Smith (1999). Bayesian nonparametric inference
Uhlenbeck, G. E. and L. S. Ornstein (1930). On thefor random distributions and related functions
theory of Brownian motion. Phys. Rev. 36, 823–(with discussion). Journal of the Royal Statisti841.
cal Society, B 61(3), 485–527.
Valiant, L. G. (1984). A theory of the learnable. Watson, G. S. (1964). Smooth regression analysis.
Communications of the Association for ComputSankhyā: The Indian Journal of Statistics. Series
ing Machinery 27, 1134–1142.
A 26, 359–372.

728

REFERENCES

Webb, A. R. (1994). Functional approximation by Williams, O., A. Blake, and R. Cipolla (2005).
feed-forward networks: a least-squares approachSparse Bayesian learning for efﬁcient visualto generalisation. IEEE Transactions on Neuraltracking. IEEE Transactions on Pattern Analysis
Networks 5(3), 363–371.
and Machine Intelligence 27(8), 1292–1304.
Weisstein, E. W. (1999). CRC Concise Encyclopedia Williams, P. M. (1996). Using neural networks tomodel conditional multivariate densities. Neuralof Mathematics. Chapman and Hall, and CRC.
Computation 8(4), 843–854.
Weston, J. and C. Watkins (1999). Multi-class support vector machines. In M. Verlysen (Ed.), Pro- Winn, J. and C. M. Bishop (2005). Variational message passing. Journal of Machine Learning Receedings ESANN’99, Brussels. D-Facto Publicasearch 6, 661–694.
tions.
Whittaker, J. (1990). Graphical Models in Applied Zarchan, P. and H. Musoff (2005). Fundamentals of
Kalman Filtering: A Practical Approach (SecMultivariate Statistics. Wiley.
ond ed.). AIAA.
Widrow, B. and M. E. Hoff (1960). Adaptive
switching circuits. In IRE WESCON Convention
Record, Volume 4, pp. 96–104. Reprinted in Anderson and Rosenfeld (1988).
Widrow, B. and M. A. Lehr (1990). 30 years of adaptive neural networks: perceptron, madeline, andbackpropagation. Proceedings of the IEEE 78(9),1415–1442.
Wiegerinck, W. and T. Heskes (2003). Fractional
belief propagation. In S. Becker, S. Thrun, and
K. Obermayer (Eds.), Advances in Neural Information Processing Systems, Volume 15, pp. 455–
462. MIT Press.
Williams, C. K. I. (1998). Computation with inﬁnite neural networks. Neural Computation 10(5),1203–1216.
Williams, C. K. I. (1999). Prediction with Gaussian
processes: from linear regression to linear prediction and beyond. In M. I. Jordan (Ed.), Learning in Graphical Models, pp. 599–621. MIT
Press.
Williams, C. K. I. and D. Barber (1998). Bayesianclassiﬁcation with Gaussian processes. IEEE
Transactions on Pattern Analysis and MachineIntelligence 20, 1342–1351.
Williams, C. K. I. and M. Seeger (2001). Using the
Nystrom method to speed up kernel machines. In
T. K. Leen, T. G. Dietterich, and V. Tresp (Eds.),
Advances in Neural Information Processing Systems, Volume 13, pp. 682–688. MIT Press.

INDEX

Index
Page numbers in bold indicate the primary source of information for the corresponding topic.
1-of-K coding scheme, 424

backgammon, 3
backpropagation, 241
acceptance criterion, 538, 541, 544
bagging, 656
activation function, 180, 213, 227
basis function, 138, 172, 204, 227
active constraint, 328, 709
batch training, 240
AdaBoost, 657, 658
Baum-Welch algorithm, 618
adaline, 196
Bayes’ theorem, 15
adaptive rejection sampling, 530
Bayes, Thomas, 21
ADF, see assumed density ﬁltering
Bayesian analysis, vii, 9, 21
AIC, see Akaike information criterion
hierarchical, 372
Akaike information criterion, 33, 217
model averaging, 654
α family of divergences, 469
Bayesian information criterion, 33, 216
α recursion, 620
Bayesian model comparison, 161, 473, 483
Bayesian network, 360
ancestral sampling, 365, 525, 613
Bayesian probability, 21annular ﬂow, 679
belief propagation, 403
AR model, see autoregressive model
Bernoulli distribution, 69, 113, 685
arc, 360
mixture model, 444
ARD, see automatic relevance determination
Bernoulli,
Jacob, 69
ARMA, see autoregressive moving average
beta
distribution,
71, 686
assumed density ﬁltering, 510
beta
recursion,621
autoassociative networks, 592
automatic relevance determination, 259, 312, 349, between-class covariance, 189
bias, 27, 149
485, 582
bias
parameter, 138, 181, 227, 346
autoregressive hidden Markov model, 632
bias-variance
trade-off, 147
autoregressive model, 609
BIC,
see
Bayesian
information criterion
autoregressive moving average, 304
binary entropy, 495
back-tracking, 415, 630
binomial distribution, 70, 686

729

730

INDEX

biological sequence, 610
bipartite graph, 401
bits, 49
blind source separation, 591
blocked path, 374, 378, 384
Boltzmann distribution, 387
Boltzmann, Ludwig Eduard, 53
Boolean logic, 21
boosting, 657
bootstrap, 23, 656
bootstrap ﬁlter, 646
box constraints, 333, 342
Box-Muller method, 527
C4.5, 663
calculus of variations, 462
canonical correlation analysis, 565
canonical link function, 212
CART, see classiﬁcation and regression trees
Cauchy distribution, 527, 529, 692
causality, 366
CCA, see canonical correlation analysis
central differences, 246
central limit theorem, 78
chain graph, 393
chaining, 555
Chapman-Kolmogorov equations, 397
child node, 361
Cholesky decomposition, 528
chunking, 335
circular normal, see von Mises distribution
classical probability, 21classiﬁcation, 3classiﬁcation and regression trees, 663
clique, 385
clustering, 3clutter problem, 511
co-parents, 383, 492
code-book vectors, 429
combining models, 45, 653
committee, 655
complete data set, 440
completing the square, 86
computational learning theory, 326, 344
concave function, 56

concentration parameter, 108, 693
condensation algorithm, 646
conditional entropy, 55
conditional expectation, 20
conditional independence, 46, 372, 383
conditional mixture model, see mixture model
conditional probability, 14
conjugate prior, 68, 98, 117, 490
convex duality, 494
convex function, 55, 493
convolutional neural network, 267
correlation matrix, 567
cost function, 41
covariance, 20
between-class, 189
within-class, 189
covariance matrix
diagonal, 84
isotropic, 84
partitioned, 85, 307
positive deﬁnite, 308
Cox’s axioms, 21
credit assignment, 3cross-entropy error function, 206, 209, 235, 631,
666
cross-validation, 32, 161
cumulative distribution function, 18
curse of dimensionality, 33, 36
curve ﬁtting, 4D map, see dependency map
d-separation, 373, 378, 443
DAG, see directed acyclic graph
DAGSVM, 339
data augmentation, 537
data compression, 429
decision boundary, 39, 179
decision region, 39, 179
decision surface, see decision boundary
decision theory, 38
decision tree, 654, 663, 673
decomposition methods, 335
degrees of freedom, 559
degrees-of-freedom parameter, 102, 693
density estimation, 3, 67

INDEXdensity network, 597
dependency map, 392
descendant node, 376
design matrix, 142, 347differential entropy, 53
digamma function, 687
directed acyclic graph, 362
directed cycle, 362
directed factorization, 381
Dirichlet distribution, 76, 687
Dirichlet, Lejeune, 77
discriminant function, 43, 180, 181
discriminative model, 43, 203distortion measure, 424
distributive law of multiplication, 396
DNA, 610
document retrieval, 299
dual representation, 293, 329
dual-energy gamma densitometry, 678
dynamic programming, 411
dynamical system, 548
E step, see expectation step
early stopping, 259
ECM, see expectation conditional maximization
edge, 360
effective number of observations, 72, 101
effective number of parameters, 9, 170, 281
elliptical K-means, 444
EM, see expectation maximization
emission probability, 611
empirical Bayes, see evidence approximationenergy function, 387
entropy, 49
conditional, 55
differential, 53
relative, 55
EP, see expectation propagation-tube, 341-insensitive error function, 340
equality constraint, 709
equivalent kernel, 159, 301
erf function, 211
error backpropagation, see backpropagation
error function, 5, 23

error-correcting output codes, 339
Euler, Leonhard, 465
Euler-Lagrange equations, 705
evidence approximation, 165, 347, 581
evidence function, 161
expectation, 19
expectation conditional maximization, 454
expectation maximization, 113, 423, 440
Gaussian mixture, 435
generalized, 454
sampling methods, 536
expectation propagation, 315, 468, 505
expectation step, 437
explaining away, 378
exploitation, 3
exploration, 3
exponential distribution, 526, 688
exponential family, 68, 113, 202, 490
extensive variables, 490
face detection, 2
face tracking, 355
factor analysis, 583
mixture model, 595
factor graph, 360, 399, 625
factor loading, 584
factorial hidden Markov model, 633
factorized distribution, 464, 476
feature extraction, 2feature map, 268
feature space, 292, 586
Fisher information matrix, 298
Fisher kernel, 298
Fisher’s linear discriminant, 186
ﬂooding schedule, 417
forward kinematics, 272
forward problem, 272
forward propagation, 228, 243
forward-backward algorithm, 618
fractional belief propagation, 517
frequentist probability, 21
fuel system, 376
function interpolation, 299
functional, 462, 703
derivative, 463

731

732

INDEX

gamma densitometry, 678
gamma distribution, 529, 688
gamma function, 71
gating function, 672
Gauss, Carl Friedrich, 79
Gaussian, 24, 78, 688
conditional, 85, 93
marginal, 88, 93
maximum likelihood, 93
mixture, 110, 270, 273, 430
sequential estimation, 94
sufﬁcient statistics, 93
wrapped, 110
Gaussian kernel, 296
Gaussian process, 160, 303
Gaussian random ﬁeld, 305
Gaussian-gamma distribution, 101, 690
Gaussian-Wishart distribution, 102, 475, 478, 690
GEM, see expectation maximization, generalized
generalization, 2
generalized linear model, 180, 213
generalized maximum likelihood, see evidence approximation
generative model, 43, 196, 297, 365, 572, 631
generative topographic mapping, 597
directional curvature, 599
magniﬁcation factor, 599
geodesic distance, 596
Gibbs sampling, 542
blocking, 546
Gibbs, Josiah Willard, 543
Gini index, 666
global minimum, 237
gradient descent, 240
Gram matrix, 293graph-cut algorithm, 390
graphical model, 359
bipartite, 401
directed, 360
factorization, 362, 384
fully connected, 361
inference, 393
tree, 398
treewidth, 417
triangulated, 416

undirected, 360
Green’s function, 299
GTM, see generative topographic mapping
Hamilton, William Rowan, 549
Hamiltonian dynamics, 548
Hamiltonian function, 549
Hammersley-Clifford theorem, 387
handwriting recognition, 1, 610, 614
handwritten digit, 565, 614, 677
head-to-head path, 376
head-to-tail path, 375
Heaviside step function, 206
Hellinger distance, 470
Hessian matrix, 167, 215, 217, 238, 249
diagonal approximation, 250
exact evaluation, 253
fast multiplication, 254
ﬁnite differences, 252
inverse, 252
outer product approximation, 251
heteroscedastic, 273, 311
hidden Markov model, 297, 610
autoregressive, 632
factorial, 633
forward-backward algorithm, 618
input-output, 633
left-to-right, 613
maximum likelihood, 615
scaling factor, 627
sum-product algorithm, 625
switching, 644
variational inference, 625
hidden unit, 227
hidden variable, 84, 364, 430, 559
hierarchical Bayesian model, 372
hierarchical mixture of experts, 673
hinge error function, 337
Hinton diagram, 584
histogram density estimation, 120
HME, see hierarchical mixture of experts
hold-out set, 11
homogeneous ﬂow, 679
homogeneous kernel, 292
homogeneous Markov chain, 540, 608

INDEX
Hooke’s law, 580
hybrid Monte Carlo, 548
hyperparameter, 71, 280, 311, 346, 372, 502
hyperprior, 372
I map, see independence map
i.i.d., see independent identically distributed
ICA, see independent component analysis
ICM, see iterated conditional modes
ID3, 663
identiﬁability, 435
image de-noising, 387
importance sampling, 525, 532
importance weights, 533
improper prior, 118, 259, 472
imputation step, 537
imputation-posterior algorithm, 537
inactive constraint, 328, 709
incomplete data set, 440
independence map, 392
independent component analysis, 591
independent factor analysis, 592
independent identically distributed, 26, 379
independent variables, 17
independent, identically distributed, 605
induced factorization, 485
inequality constraint, 709
inference, 38, 42
information criterion, 33
information geometry, 298
information theory, 48
input-output hidden Markov model, 633
intensive variables, 490
intrinsic dimensionality, 559
invariance, 261
inverse gamma distribution, 101
inverse kinematics, 272
inverse problem, 272
inverse Wishart distribution, 102
IP algorithm, see imputation-posterior algorithm
IRLS, see iterative reweighted least squares
Ising model, 389
isomap, 596
isometric feature map, 596
iterated conditional modes, 389, 415

733

iterative reweighted least squares, 207, 210, 316,
354, 672
Jacobian matrix, 247, 264
Jensen’s inequality, 56
join tree, 416
junction tree algorithm, 392, 416
K nearest neighbours, 125
K-means clustering algorithm, 424, 443
K-medoids algorithm, 428
Kalman ﬁlter, 304, 637
extended, 644
Kalman gain matrix, 639
Kalman smoother, 637
Karhunen-Loève transform, 561
Karush-Kuhn-Tucker conditions, 330, 333, 342,
710
kernel density estimator, 122, 326
kernel function, 123, 292, 294
Fisher, 298
Gaussian, 296
homogeneous, 292
nonvectorial inputs, 297
stationary, 292
kernel PCA, 586
kernel regression, 300, 302
kernel substitution, 292
kernel trick, 292
kinetic energy, 549
KKT, see Karush-Kuhn-Tucker conditions
KL divergence, see Kullback-Leibler divergencekriging, see Gaussian process
Kullback-Leibler divergence, 55, 451, 468, 505
Lagrange multiplier, 707
Lagrange, Joseph-Louis, 329
Lagrangian, 328, 332, 341, 708
laminar ﬂow, 678
Laplace approximation, 213, 217, 278, 315, 354
Laplace, Pierre-Simon, 24
large margin, see marginlasso, 145
latent class analysis, 444
latent trait model, 597
latent variable, 84, 364, 430, 559

734

INDEX

lattice diagram, 414, 611, 621, 629
LDS, see linear dynamical system
leapfrog discretization, 551
learning, 2learning rate parameter, 240
least-mean-squares algorithm, 144
leave-one-out, 33
likelihood function, 22
likelihood weighted sampling, 534
linear discriminant, 181
Fisher, 186
linear dynamical system, 84, 635
inference, 638
linear independence, 696
linear regression, 138
EM, 448
mixture model, 667
variational, 486
linear smoother, 159
linear-Gaussian model, 87, 370
linearly separable, 179
link, 360
link function, 180, 213
Liouville’s Theorem, 550
LLE, see locally linear embedding
LMS algorithm, see least-mean-squares algorithm
local minimum, 237
local receptive ﬁeld, 268
locally linear embedding, 596
location parameter, 118
log odds, 197
logic sampling, 525
logistic regression, 205, 336
Bayesian, 217, 498
mixture model, 670
multiclass, 209
logistic sigmoid, 114, 139, 197, 205, 220, 227, 495
logit function, 197
loopy belief propagation, 417
loss function, 41
loss matrix, 41
lossless data compression, 429
lossy data compression, 429
lower bound, 484
M step, see maximization step

machine learning, vii
macrostate, 51
Mahalanobis distance, 80
manifold, 38, 590, 595, 681
MAP, see maximum posterior
margin, 326, 327, 502
error, 334
soft, 332
marginal likelihood, 162, 165
marginal probability, 14
Markov blanket, 382, 384, 545
Markov boundary, see Markov blanket
Markov chain, 397, 539
ﬁrst order, 607
homogeneous, 540, 608
second order, 608
Markov chain Monte Carlo, 537
Markov model, 607
homogeneous, 612
Markov network, see Markov random ﬁeld
Markov random ﬁeld, 84, 360, 383
max-sum algorithm, 411, 629
maximal clique, 385
maximal spanning tree, 416
maximization step, 437
maximum likelihood, 9, 23, 26, 116
Gaussian mixture, 432
singularities, 480
type 2, see evidence approximation
maximum margin, see margin
maximum posterior, 30, 441
MCMC, see Markov chain Monte Carlo
MDN, see mixture density network
MDS, see multidimensional scaling
mean, 24
mean ﬁeld theory, 465
mean value theorem, 52
measure theory, 19memory-based methods, 292
message passing, 396
pending message, 417
schedule, 417
variational, 491
Metropolis algorithm, 538
Metropolis-Hastings algorithm, 541

INDEX
microstate, 51
minimum risk, 44
Minkowski loss, 48
missing at random, 441, 579
missing data, 579
mixing coefﬁcient, 111
mixture component, 111
mixture density network, 272, 673
mixture distribution, see mixture model
mixture model, 162, 423
conditional, 273, 666
linear regression, 667
logistic regression, 670
symmetries, 483
mixture of experts, 672
mixture of Gaussians, 110, 270, 273, 430
MLP, see multilayer perceptron
MNIST data, 677
model comparison, 6, 32, 161, 473, 483
model evidence, 161
model selection, 162
moment matching, 506, 510
momentum variable, 548
Monte Carlo EM algorithm, 536
Monte Carlo sampling, 24, 523
Moore-Penrose pseudo-inverse, see pseudo-inverse
moralization, 391, 401
MRF, see Markov random ﬁeld
multidimensional scaling, 596
multilayer perceptron, 226, 229
multimodality, 272
multinomial distribution, 76, 114, 690
multiplicity, 51
mutual information, 55, 57
Nadaraya-Watson, see kernel regression
naive Bayes model, 46, 380
nats, 50
natural language modelling, 610
natural parameters, 113
nearest-neighbour methods, 124
neural network, 225
convolutional, 267
regularization, 256
relation to Gaussian process, 319

735

Newton-Raphson, 207, 317
node, 360
noiseless coding theorem, 50
nonidentiﬁability, 585
noninformative prior, 23, 117
nonparametric methods, 68, 120
normal distribution, see Gaussian
normal equations, 142
normal-gamma distribution, 101, 691
normal-Wishart distribution, 102, 475, 478, 691
normalized exponential, see softmax function
novelty detection, 44
ν-SVM, 334
object recognition, 366
observed variable, 364
Occam factor, 217
oil ﬂow data, 34, 560, 568, 678
Old Faithful data, 110, 479, 484, 681
on-line learning, see sequential learning
one-versus-one classiﬁer, 183, 339
one-versus-the-rest classiﬁer, 182, 338
ordered over-relaxation, 545
Ornstein-Uhlenbeck process, 305
orthogonal least squares, 301
outlier, 44, 185, 212
outliers, 103
over-ﬁtting, 6, 147, 434, 464
over-relaxation, 544
PAC learning, see probably approximately correct
PAC-Bayesian framework, 345
parameter shrinkage, 144
parent node, 361
particle ﬁlter, 645
partition function, 386, 554
Parzen estimator, see kernel density estimator
Parzen window, 123
pattern recognition, vii
PCA, see principal component analysis
pending message, 417
perceptron, 192
convergence theorem, 194
hardware, 196
perceptron criterion, 193
perfect map, 392

736

INDEX

periodic variable, 105phase space, 549
photon noise, 680
plate, 363
polynomial curve ﬁtting, 4, 362
polytree, 399
position variable, 548
positive deﬁnite covariance, 81
positive deﬁnite matrix, 701
positive semideﬁnite covariance, 81
positive semideﬁnite matrix, 701
posterior probability, 17posterior step, 537
potential energy, 549
potential function, 386
power EP, 517
power method, 563
precision matrix, 85
precision parameter, 24predictive distribution, 30, 156
preprocessing, 2
principal component analysis, 561, 572, 593
Bayesian, 580
EM algorithm, 577
Gibbs sampling, 583
mixture distribution, 595
physical analogy, 580
principal curve, 595
principal subspace, 561
principal surface, 596
prior, 17
conjugate, 68, 98, 117, 490
consistent, 257
improper, 118, 259, 472
noninformative, 23, 117
probabilistic graphical model, see graphical model
probabilistic PCA, 570
probability, 12
Bayesian, 21
classical, 21
density, 17
frequentist, 21
mass function, 19prior, 45
product rule, 13, 14, 359

sum rule, 13, 14, 359
theory, 12probably approximately correct, 344
probit function, 211, 219
probit regression, 210
product rule of probability, 13, 14, 359
proposal distribution, 528, 532, 538
protected conjugate gradients, 335
protein sequence, 610
pseudo-inverse, 142, 185
pseudo-random numbers, 526
quadratic discriminant, 199
quality parameter, 351
radial basis function, 292, 299
Rauch-Tung-Striebel equations, 637
regression, 3
regression function, 47, 95
regularization, 10
Tikhonov, 267
regularized least squares, 144
reinforcement learning, 3
reject option, 42, 45
rejection sampling, 528
relative entropy, 55
relevance vector, 348
relevance vector machine, 161, 345
responsibility, 112, 432, 477
ridge regression, 10
RMS error, see root-mean-square error
Robbins-Monro algorithm, 95
robot arm, 272
robustness, 103, 185root node, 399
root-mean-square error, 6
Rosenblatt, Frank, 193
rotation invariance, 573, 585
RTS equations, see Rauch-Tung-Striebel equations
running intersection property, 416
RVM, see relevance vector machinesample mean, 27
sample variance, 27
sampling-importance-resampling, 534
scale invariance, 119, 261

INDEX
scale parameter, 119
scaling factor, 627
Schwarz criterion, see Bayesian information criterion
self-organizing map, 598
sequential data, 605
sequential estimation, 94
sequential gradient descent, 144, 240
sequential learning, 73, 143
sequential minimal optimization, 335
serial message passing schedule, 417
Shannon, Claude, 55
shared parameters, 368
shrinkage, 10
Shur complement, 87
sigmoid, see logistic sigmoid
simplex, 76
single-class support vector machine, 339
singular value decomposition, 143
sinusoidal data, 682
SIR, see sampling-importance-resamplingskip-layer connection, 229
slack variable, 331
slice sampling, 546
SMO, see sequential minimal optimizationsmoother matrix, 159
smoothing parameter, 122
soft margin, 332
soft weight sharing, 269
softmax function, 115, 198, 236, 274, 356, 497
SOM, see self-organizing map
sparsity, 145, 347, 349, 582
sparsity parameter, 351
spectrogram, 606
speech recognition, 605, 610
sphereing, 568
spline functions, 139
standard deviation, 24
standardizing, 425, 567
state space model, 609
switching, 644
stationary kernel, 292
statistical bias, see bias
statistical independence, see independent variables

737

statistical learning theory, see computational learning theory, 326, 344
steepest descent, 240
Stirling’s approximation, 51
stochastic, 5stochastic EM, 536
stochastic gradient descent, 144, 240
stochastic process, 305
stratiﬁed ﬂow, 678
Student’s t-distribution, 102, 483, 691
subsampling, 268
sufﬁcient statistics, 69, 75, 116
sum rule of probability, 13, 14, 359
sum-of-squares error, 5, 29, 184, 232, 662
sum-product algorithm, 399, 402
for hidden Markov model, 625
supervised learning, 3
support vector, 330
support vector machine, 225
for regression, 339
multiclass, 338
survival of the ﬁttest, 646
SVD, see singular value decomposition
SVM, see support vector machine
switching hidden Markov model, 644
switching state space model, 644
synthetic data sets, 682
tail-to-tail path, 374
tangent distance, 265
tangent propagation, 262, 263
tapped delay line, 609
target vector, 2
test set, 2, 32
threshold parameter, 181
tied parameters, 368
Tikhonov regularization, 267
time warping, 615
tomography, 679
training, 2
training set, 2transition probability, 540, 610translation invariance, 118, 261
tree-reweighted message passing, 517
treewidth, 417

738

INDEX

trellis diagram, see lattice diagram
triangulated graph, 416
type 2 maximum likelihood, see evidence approximation
undetermined multiplier, see Lagrange multiplierundirected graph, see Markov random ﬁeld
uniform distribution, 692
uniform sampling, 534
uniquenesses, 584
unobserved variable, see latent variable
unsupervised learning, 3utility function, 41
validation set, 11, 32
Vapnik-Chervonenkis dimension, 344
variance, 20, 24, 149variational inference, 315, 462, 635
for Gaussian mixture, 474
for hidden Markov model, 625
local, 493
VC dimension, see Vapnik-Chervonenkis dimension
vector quantization, 429vertex, see nodevisualization, 3
Viterbi algorithm, 415, 629
von Mises distribution, 108, 693
wavelets, 139
weak learner, 657
weight decay, 10, 144, 257
weight parameter, 227
weight sharing, 268
soft, 269
weight vector, 181
weight-space symmetry, 231, 281
weighted least squares, 668
well-determined parameters, 170
whitening, 299, 568
Wishart distribution, 102, 693
within-class covariance, 189
Woodbury identity, 696
wrapped distribution, 110
Yellowstone National Park, 110, 681