Independent
Component
Analysis

Independent
Component
Analysis

Final version of 7 March 2001

Aapo Hyvärinen, Juha Karhunen, and Erkki Oja

A Wiley-Interscience Publication

JOHN WILEY & SONS, INC.
New York / Chichester / Weinheim / Brisbane / Singapore / Toronto

Contents

Preface
1 Introduction
1.1 Linear representation of multivariate data
1.1.1 The general statistical setting
1.1.2 Dimension reduction methods
1.1.3 Independence as a guiding principle
1.2 Blind source separation
1.2.1 Observing mixtures of unknown signals
1.2.2 Source separation based on independence
1.3 Independent component analysis
1.3.1 Definition
1.3.2 Applications
1.3.3 How to find the independent components
1.4 History of ICA

xvii
1
1
1
2
3
3
4
5
6
6
7
7
11

v

vi

CONTENTSPart I MATHEMATICAL PRELIMINARIES
2 Random Vectors and Independence
2.1 Probability distributions and densities
2.1.1 Distribution of a random variable
2.1.2 Distribution of a random vector
2.1.3 Joint and marginal distributions
2.2 Expectations and moments
2.2.1 Definition and general properties
2.2.2 Mean vector and correlation matrix
2.2.3 Covariances and joint moments
2.2.4 Estimation of expectations
2.3 Uncorrelatedness and independence
2.3.1 Uncorrelatedness and whiteness2.3.2 Statistical independence
2.4 Conditional densities and Bayes’ rule
2.5 The multivariate gaussian density
2.5.1 Properties of the gaussian density
2.5.2 Central limit theorem
2.6 Density of a transformation
2.7 Higher-order statistics
2.7.1 Kurtosis and classification of densities
2.7.2 Cumulants, moments, and their properties
2.8 Stochastic processes *2.8.1 Introduction and definition
2.8.2 Stationarity, mean, and autocorrelation
2.8.3 Wide-sense stationary processes
2.8.4 Time averages and ergodicity
2.8.5 Power spectrum
2.8.6 Stochastic signal models
2.9 Concluding remarks and references
Problems

15
15
15
17
18
19
19
20
22
24
24
24
27
28
31
32
34
35
36
37
40
43
43
45
46
48
49
50
51
52

3 Gradients and Optimization Methods
3.1 Vector and matrix gradients
3.1.1 Vector gradient3.1.2 Matrix gradient3.1.3 Examples of gradients

57
57
57
59
59

3.2

3.3

3.4

CONTENTS

vii

3.1.4 Taylor series expansions
Learning rules for unconstrained optimization
3.2.1 Gradient descent3.2.2 Second-order learning
3.2.3 The natural gradient and relative gradient3.2.4 Stochastic gradient descent
3.2.5 Convergence of stochastic on-line algorithms *Learning rules for constrained optimization
3.3.1 The Lagrange method
3.3.2 Projection methods
Concluding remarks and references
Problems

62
63
63
65
67
68
71
73
73
73
75
75

4 Estimation Theory
4.1 Basic concepts
4.2 Properties of estimators
4.3 Method of moments
4.4 Least-squares estimation
4.4.1 Linear least-squares method
4.4.2 Nonlinear and generalized least squares *4.5 Maximum likelihood method
4.6 Bayesian estimation *4.6.1 Minimum mean-square error estimator
4.6.2 Wiener filtering
4.6.3 Maximum a posteriori (MAP) estimator
4.7 Concluding remarks and references
Problems

77
78
80
84
86
86
88
90
94
94
96
97
99
101

5 Information Theory
5.1 Entropy
5.1.1 Definition of entropy
5.1.2 Entropy and coding length
5.1.3 Differential entropy
5.1.4 Entropy of a transformation
5.2 Mutual information
5.2.1 Definition using entropy
5.2.2 Definition using Kullback-Leibler divergence

105
105
105
107
108
109
110
110
110

viii

CONTENTS

5.3

Maximum entropy
5.3.1 Maximum entropy distributions
5.3.2 Maximality property of gaussian distribution
Negentropy
Approximation of entropy by cumulants
5.5.1 Polynomial density expansions
5.5.2 Using expansions for entropy approximation
Approximation of entropy by nonpolynomial functions
5.6.1 Approximating the maximum entropy
5.6.2 Choosing the nonpolynomial functions
5.6.3 Simple special cases
5.6.4 Illustration
Concluding remarks and references
Problems
Appendix proofs

111
111
112
112
113
113
114
115
116
117
118
119
120
121
122

6 Principal Component Analysis and Whitening
6.1 Principal components
6.1.1 PCA by variance maximization
6.1.2 PCA by minimum MSE compression
6.1.3 Choosing the number of principal components
6.1.4 Closed-form computation of PCA
6.2 PCA by on-line learning
6.2.1 The stochastic gradient ascent algorithm
6.2.2 The subspace learning algorithm
6.2.3 The PAST algorithm *6.2.4 PCA and back-propagation learning *6.2.5 Extensions of PCA to nonquadratic criteria *6.3 Factor analysis
6.4 Whitening
6.5 Orthogonalization
6.6 Concluding remarks and references
Problems

125
125
127
128
129
131
132
133
134
135
136
137
138
140
141
143
144

5.4
5.5

5.6

5.7

CONTENTS

ix

Part II BASIC INDEPENDENT COMPONENT ANALYSIS
7 What is Independent Component Analysis?
7.1 Motivation
7.2 Definition of independent component analysis
7.2.1 ICA as estimation of a generative model
7.2.2 Restrictions in ICA
7.2.3 Ambiguities of ICA
7.2.4 Centering the variables
7.3 Illustration of ICA
7.4 ICA is stronger that whitening
7.4.1 Uncorrelatedness and whitening
7.4.2 Whitening is only half ICA
7.5 Why gaussian variables are forbidden
7.6 Concluding remarks and references
Problems

147
147
151
151
152
154
154
155
158
158
160
161
163
164

8 ICA by Maximization of Nongaussianity
8.1 “Nongaussian is independent”
8.2 Measuring nongaussianity by kurtosis
8.2.1 Extrema give independent components
8.2.2 Gradient algorithm using kurtosis
8.2.3 A fast fixed-point algorithm using kurtosis
8.2.4 Examples
8.3 Measuring nongaussianity by negentropy
8.3.1 Critique of kurtosis
8.3.2 Negentropy as nongaussianity measure
8.3.3 Approximating negentropy
8.3.4 Gradient algorithm using negentropy
8.3.5 A fast fixed-point algorithm using negentropy
8.4 Estimating several independent components
8.4.1 Constraint of uncorrelatedness
8.4.2 Deflationary orthogonalization
8.4.3 Symmetric orthogonalization
8.5 ICA and projection pursuit
8.5.1 Searching for interesting directions
8.5.2 Nongaussian is interesting
8.6 Concluding remarks and references

165
166
171
171
175
178
179
182
182
182
183
185
188
192
192
194
194
197
197
197
198

x

CONTENTS

Problems
Appendix proofs

199
201

9 ICA by Maximum Likelihood Estimation
9.1 The likelihood of the ICA model
9.1.1 Deriving the likelihood
9.1.2 Estimation of the densities
9.2 Algorithms for maximum likelihood estimation
9.2.1 Gradient algorithms
9.2.2 A fast fixed-point algorithm
9.3 The infomax principle
9.4 Examples
9.5 Concluding remarks and references
Problems
Appendix proofs

203
203
203
204
207
207
209
211
213
214
218
219

10 ICA by Minimization of Mutual Information
10.1 Defining ICA by mutual information
10.1.1 Information-theoretic concepts
10.1.2 Mutual information as measure of dependence
10.2 Mutual information and nongaussianity
10.3 Mutual information and likelihood
10.4 Algorithms for minimization of mutual information
10.5 Examples
10.6 Concluding remarks and references
Problems

221
221
221
222
223
224
224
225
225
227

11 ICA by Tensorial Methods
11.1 Definition of cumulant tensor
11.2 Tensor eigenvalues give independent components
11.3 Tensor decomposition by a power method
11.4 Joint approximate diagonalization of eigenmatrices
11.5 Weighted correlation matrix approach
11.5.1 The FOBI algorithm
11.5.2 From FOBI to JADE
11.6 Concluding remarks and references
Problems

229
229
230
232
234
235
235
235
236
237

CONTENTS

xi

12 ICA by Nonlinear Decorrelation and Nonlinear PCA
12.1 Nonlinear correlations and independence
12.2 The Hérault-Jutten algorithm
12.3 The Cichocki-Unbehauen algorithm
12.4 The estimating functions approach *12.5 Equivariant adaptive separation via independence
12.6 Nonlinear principal components
12.7 The nonlinear PCA criterion and ICA
12.8 Learning rules for the nonlinear PCA criterion
12.8.1 The nonlinear subspace rule12.8.2 Convergence of the nonlinear subspace rule *12.8.3 Nonlinear recursive least-squares rule12.9 Concluding remarks and references
Problems

239
240
242
243
245
247
249
251
254
254
255
258
261
262

13 Practical Considerations
13.1 Preprocessing by time filtering
13.1.1 Why time filtering is possible
13.1.2 Low-pass filtering
13.1.3 High-pass filtering and innovations
13.1.4 Optimal filtering
13.2 Preprocessing by PCA
13.2.1 Making the mixing matrix square
13.2.2 Reducing noise and preventing overlearning
13.3 How many components should be estimated?
13.4 Choice of algorithm
13.5 Concluding remarks and references
Problems

263
263
264
265
265
266
267
267
268
269
271
272
272

14 Overview and Comparison of Basic ICA Methods
14.1 Objective functions vs. algorithms
14.2 Connections between ICA estimation principles
14.2.1 Similarities between estimation principles
14.2.2 Differences between estimation principles
14.3 Statistically optimal nonlinearities
14.3.1 Comparison of asymptotic variance *14.3.2 Comparison of robustness *14.3.3 Practical choice of nonlinearity

273
273
274
274
275
276
276
277
279

xii

CONTENTS

14.4 Experimental comparison of ICA algorithms
14.4.1 Experimental set-up and algorithms
14.4.2 Results for simulated data
14.4.3 Comparisons with real-world data
14.5 References
14.6 Summary of basic ICA
Appendix Proofs

280
281
282
286
287
287
289

Part III EXTENSIONS AND RELATED METHODS
15 Noisy ICA
15.1 Definition
15.2 Sensor noise vs. source noise
15.3 Few noise sources
15.4 Estimation of the mixing matrix
15.4.1 Bias removal techniques
15.4.2 Higher-order cumulant methods
15.4.3 Maximum likelihood methods
15.5 Estimation of the noise-free independent components
15.5.1 Maximum a posteriori estimation
15.5.2 Special case of shrinkage estimation
15.6 Denoising by sparse code shrinkage
15.7 Concluding remarks

293
293
294
295
295
296
298
299
299
299
300
303
304

16 ICA with Overcomplete Bases
16.1 Estimation of the independent components
16.1.1 Maximum likelihood estimation
16.1.2 The case of supergaussian components
16.2 Estimation of the mixing matrix
16.2.1 Maximizing joint likelihood
16.2.2 Maximizing likelihood approximations
16.2.3 Approximate estimation by quasiorthogonality
16.2.4 Other approaches
16.3 Concluding remarks

305
306
306
307
307
307
308
309
311
313

CONTENTS

17 Nonlinear ICA
17.1 Nonlinear ICA and BSS
17.1.1 The nonlinear ICA and BSS problems
17.1.2 Existence and uniqueness of nonlinear ICA
17.2 Separation of post-nonlinear mixtures
17.3 Nonlinear BSS using self-organizing maps
17.4 A generative topographic mapping approach *17.4.1 Background17.4.2 The modified GTM method
17.4.3 An experiment17.5 An ensemble learning approach to nonlinear BSS
17.5.1 Ensemble learning
17.5.2 Model structure
17.5.3 Computing Kullback-Leibler cost function *17.5.4 Learning procedure *17.5.5 Experimental results
17.6 Other approaches
17.7 Concluding remarks

xiii

315
315
315
317
319
320
322
322
323
326
328
328
329
330
332
333
337
339

18 Methods using Time Structure341
18.1 Separation by autocovariances342
18.1.1 An alternative to nongaussianity342
18.1.2 Using one time lag
343
18.1.3 Extension to several time lags344
18.2 Separation by nonstationarity of variances346
18.2.1 Using local autocorrelations347
18.2.2 Using cross-cumulants
349
18.3 Separation principles unified
351
18.3.1 Comparison of separation principles
351
18.3.2 Kolmogoroff complexity as unifying framework 352
18.4 Concluding remarks
354

xiv

CONTENTS

19 Convolutive Mixtures and Blind Deconvolution
19.1 Blind deconvolution
19.1.1 Problem definition
19.1.2 Bussgang methods
19.1.3 Cumulant-based methods
19.1.4 Blind deconvolution using linear ICA
19.2 Blind separation of convolutive mixtures
19.2.1 The convolutive BSS problem
19.2.2 Reformulation as ordinary ICA
19.2.3 Natural gradient methods
19.2.4 Fourier transform methods
19.2.5 Spatiotemporal decorrelation methods
19.2.6 Other methods for convolutive mixtures
19.3 Concluding remarks
Appendix Discrete-time filters and the z -transform

355
356
356
357
358
360
361
361
363
364
365
367
367
368
369

20 Other Extensions
20.1 Priors on the mixing matrix
20.1.1 Motivation for prior information
20.1.2 Classic priors
20.1.3 Sparse priors
20.1.4 Spatiotemporal ICA
20.2 Relaxing the independence assumption
20.2.1 Multidimensional ICA
20.2.2 Independent subspace analysis
20.2.3 Topographic ICA
20.3 Complex-valued data
20.3.1 Basic concepts of complex random variables
20.3.2 Indeterminacy of the independent components
20.3.3 Choice of the nongaussianity measure
20.3.4 Consistency of estimator
20.3.5 Fixed-point algorithm
20.3.6 Relation to independent subspaces
20.4 Concluding remarks

371
371
371
372
374
377
378
379
380
382
383
383
384
385
386
386
387
387

CONTENTS

xv

Part IV APPLICATIONS OF ICA
21 Feature Extraction by ICA
21.1 Linear representations
21.1.1 Definition
21.1.2 Gabor analysis
21.1.3 Wavelets
21.2 ICA and Sparse Coding
21.3 Estimating ICA bases from images
21.4 Image denoising by sparse code shrinkage
21.4.1 Component statistics
21.4.2 Remarks on windowing
21.4.3 Denoising results
21.5 Independent subspaces and topographic ICA
21.6 Neurophysiological connections
21.7 Concluding remarks

391
392
392
392
394
396
398
398
399
400
401
401
403
405

22 Brain Imaging Applications
22.1 Electro- and magnetoencephalography
22.1.1 Classes of brain imaging techniques
22.1.2 Measuring electric activity in the brain
22.1.3 Validity of the basic ICA model
22.2 Artifact identification from EEG and MEG
22.3 Analysis of evoked magnetic fields
22.4 ICA applied on other measurement techniques
22.5 Concluding remarks

407
407
407
408
409
410
411
413
414

23 Telecommunications
23.1 Multiuser detection and CDMA communications
23.2 CDMA signal model and ICA
23.3 Estimating fading channels
23.3.1 Minimization of complexity
23.3.2 Channel estimation *23.3.3 Comparisons and discussion
23.4 Blind separation of convolved CDMA mixtures *23.4.1 Feedback architecture
23.4.2 Semiblind separation method
23.4.3 Simulations and discussion

417
417
422
424
424
426
428
430
430
431
432

xvi

CONTENTS

23.5 Improving multiuser detection using complex ICA *23.5.1 Data model
23.5.2 ICA based receivers
23.5.3 Simulation results
23.6 Concluding remarks and references

434
435
436
438
439

24 Other Applications
24.1 Financial applications
24.1.1 Finding hidden factors in financial data
24.1.2 Time series prediction by ICA
24.2 Audio separation
24.3 Further applications

441
441
441
443
446
448

References

449

Index

476

Preface

Independent component analysis (ICA) is a statistical and computational techniquefor revealing hidden factors that underlie sets of random variables, measurements, orsignals. ICA defines a generative model for the observed multivariate data, which istypically given as a large database of samples. In the model, the data variables are
assumed to be linear or nonlinear mixtures of some unknown latent variables, andthe mixing system is also unknown. The latent variables are assumed nongaussian
and mutually independent, and they are called the independent components of theobserved data. These independent components, also called sources or factors, can befound by ICA.
ICA can be seen as an extension to principal component analysis and factor
analysis. ICA is a much more powerful technique, however, capable of finding theunderlying factors or sources when these classic methods fail completely.
The data analyzed by ICA could originate from many different kinds of application fields, including digital images and document databases, as well as economic
indicators and psychometric measurements. In many cases, the measurements are
given as a set of parallel signals or time series; the term blind source separation is used
to characterize this problem. Typical examples are mixtures of simultaneous speech
signals that have been picked up by several microphones, brain waves recorded bymultiple sensors, interfering radio signals arriving at a mobile phone, or parallel timeseries obtained from some industrial process.
The technique of ICA is a relatively new invention. It was for the first time introduced in early 1980s in the context of neural network modeling. In mid-1990s,
some highly successful new algorithms were introduced by several research groups,
xvii

xviii

PREFACE

together with impressive demonstrations on problems like the cocktail-party effect,
where the individual speech waveforms are found from their mixtures. ICA became
one of the exciting new topics, both in the field of neural networks, especially unsupervised learning, and more generally in advanced statistics and signal processing.
Reported real-world applications of ICA on biomedical signal processing, audio signal separation, telecommunications, fault diagnosis, feature extraction, financial timeseries analysis, and data mining began to appear.
Many articles on ICA were published during the past 20 years in a large numberof journals and conference proceedings in the fields of signal processing, artificial
neural networks, statistics, information theory, and various application fields. Several
special sessions and workshops on ICA have been arranged recently [70, 348], andsome edited collections of articles [315, 173, 150] as well as some monographs on
ICA, blind source separation, and related subjects [105, 267, 149] have appeared.
However, while highly useful for their intended readership, these existing texts typically concentrate on some selected aspects of the ICA methods only. In the brief
scientific papers and book chapters, mathematical and statistical preliminaries areusually not included, which makes it very hard for a wider audience to gain fullunderstanding of this fairly technical topic.
A comprehensive and detailed text book has been missing, which would coverboth the mathematical background and principles, algorithmic solutions, and practicalapplications of the present state of the art of ICA. The present book is intended to fillthat gap, serving as a fundamental introduction to ICA.
It is expected that the readership will be from a variety of disciplines, suchas statistics, signal processing, neural networks, applied mathematics, neural andcognitive sciences, information theory, artificial intelligence, and engineering. Bothresearchers, students, and practitioners will be able to use the book. We have madeevery effort to make this book self-contained, so that a reader with a basic backgroundin college calculus, matrix algebra, probability theory, and statistics will be able toread it. This book is also suitable for a graduate level university course on ICA,
which is facilitated by the exercise problems and computer assignments given in
many chapters.

Scope and contents of this book
This book provides a comprehensive introduction to ICA as a statistical and computational technique. The emphasis is on the fundamental mathematical principles andbasic algorithms. Much of the material is based on the original research conductedin the authors’ own research group, which is naturally reflected in the weighting of
the different topics. We give a wide coverage especially to those algorithms that arescalable to large problems, that is, work even with a large number of observed variables and data points. These will be increasingly used in the near future when ICA
is extensively applied in practical real-world problems instead of the toy problems
or small pilot studies that have been predominant until recently. Respectively, some-PREFACE

xix

what less emphasis is given to more specialized signal processing methods involving
convolutive mixtures, delays, and other blind source separation techniques than ICA.
As ICA is a fast growing research area, it is impossible to include every reported
development in a textbook. We have tried to cover the central contributions by otherworkers in the field in their appropriate context and present an extensive bibliography
for further reference. We apologize for any omissions of important contributions that
we may have overlooked.
For easier reading, the book is divided into four parts.

 Part I gives the mathematical preliminaries. It introduces the general math-ematical concepts needed in the rest of the book. We start with a crash course
on probability theory in Chapter 2. The reader is assumed to be familiar withmost of the basic material in this chapter, but also some concepts more specific to ICA are introduced, such as higher-order cumulants and multivariate
probability theory. Next, Chapter 3 discusses essential concepts in optimization theory and gradient methods, which are needed when developing ICA
algorithms. Estimation theory is reviewed in Chapter 4. A complementary
theoretical framework for ICA is information theory, covered in Chapter 5.
Part I is concluded by Chapter 6, which discusses methods related to principal
component analysis, factor analysis, and decorrelation.
More confident readers may prefer to skip some or all of the introductory
chapters in Part I and continue directly to the principles of ICA in Part II.

 In Part II, the basic ICA model is covered and solved. This is the linear

instantaneous noise-free mixing model that is classic in ICA, and forms the coreof the ICA theory. The model is introduced and the question of identifiability ofthe mixing matrix is treated in Chapter 7. The following chapters treat different
methods of estimating the model. A central principle is nongaussianity, whose
relation to ICA is first discussed in Chapter 8. Next, the principles of maximum
likelihood (Chapter 9) and minimum mutual information (Chapter 10) arereviewed, and connections between these three fundamental principles areshown. Material that is less suitable for an introductory course is coveredin Chapter 11, which discusses the algebraic approach using higher-order
cumulant tensors, and Chapter 12, which reviews the early work on ICA basedon nonlinear decorrelations, as well as the nonlinear principal component
approach. Practical algorithms for computing the independent components
and the mixing matrix are discussed in connection with each principle. Next,
some practical considerations, mainly related to preprocessing and dimension
reduction of the data are discussed in Chapter 13,including hints to practitioners
on how to really apply ICA to their own problem. An overview and comparisonof the various ICA methods is presented in Chapter 14, which thus summarizes
Part II.

 In Part III, different extensions of the basic ICA model are given. This part is by
its nature more speculative than Part II, since most of the extensions have been
introduced very recently, and many open problems remain. In an introductory

xx

PREFACEcourse on ICA, only selected chapters from this part may be covered. First,
in Chapter 15, we treat the problem of introducing explicit observational noisein the ICA model. Then the situation where there are more independent
components than observed mixtures is treated in Chapter 16. In Chapter 17,
the model is widely generalized to the case where the mixing process can be ofa very general nonlinear form. Chapter 18 discusses methods that estimate alinear mixing model similar to that of ICA, but with quite different assumptions:the components are not nongaussian but have some time dependencies instead.
Chapter 19 discusses the case where the mixing system includes convolutions.
Further extensions, in particular models where the components are no longer
required to be exactly independent, are given in Chapter 20.

 Part IV treats some applications of ICA methods. Feature extraction (Chap-ter 21) is relevant to both image processing and vision research. Brain imaging
applications (Chapter 22) concentrate on measurements of the electrical andmagnetic activity of the human brain. Telecommunications applications aretreated in Chapter 23. Some econometric and audio signal processing applications, together with pointers to miscellaneous other applications, are treated in
Chapter 24.

Throughout the book, we have marked with an asterisk some sections that are
rather involved and can be skipped in an introductory course.
Several of the algorithms presented in this book are available as public domain
software through the World Wide Web, both on our own Web pages and those ofother ICA researchers. Also, databases of real-world data can be found there fortesting the methods. We have made a special Web page for this book, which contains
appropriate pointers. The address iswww.cis.hut.fi/projects/ica/book

The reader is advised to consult this page for further information.
This book was written in cooperation between the three authors. A. Hyv ärinen
was responsible for the chapters 5, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 20, 21, and 22;
J. Karhunen was responsible for the chapters 2, 4, 17, 19, and 23; while E. Oja was
responsible for the chapters 3, 6, and 12. The Chapters 1 and 24 were written jointly
by the authors.

Acknowledgments
We are grateful to the many ICA researchers whose original contributions form thefoundations of ICA and who have made this book possible. In particular, we wish to
express our gratitude to the Series Editor Simon Haykin, whose articles and books on
signal processing and neural networks have been an inspiration to us over the years.

PREFACE

xxi

Some parts of this text are based on close cooperation with other members of ourresearch group at the Helsinki University of Technology. Chapter 21 is largely basedon joint work with Patrik Hoyer, who also made all the experiments in that chapter.
Chapter 22 is based on experiments and material by Ricardo Vig ário. Section 13.2.2
is based on joint work with Jaakko Särelä and Ricardo Vigário. The experiments in
Section 16.2.3 were provided by Razvan Cristescu. Section 20.3 is based on joint
work with Ella Bingham, Section 14.4 on joint work with Xavier Giannakopoulos,
and Section 20.2.3 on joint work with Patrik Hoyer and Mika Inki. Chapter 19 ispartly based on material provided by Kari Torkkola. Much of Chapter 17 is basedon joint work with Harri Valpola and Petteri Pajunen, and Section 24.1 is joint workwith Kimmo Kiviluoto and Simona Malaroiu.
Over various phases of writing this book, several people have kindly agreed toread and comment on parts or all of the text. We are grateful for this to Ella Bingham,
Jean-François Cardoso, Adrian Flanagan, Mark Girolami, Antti Honkela, Jarmo
Hurri, Petteri Pajunen, Tapani Ristaniemi, and Kari Torkkola. Leila Koivisto helpedin technical editing, while Antti Honkela, Mika Ilmoniemi, Merja Oja, and Tapani
Raiko helped with some of the figures.
Our original research work on ICA as well as writing this book has been mainly
conducted at the Neural Networks Research Centre of the Helsinki University of Technology, Finland. The research had been partly financed by the project “BLISS” (European Union) and the project “New Information Processing Principles” (Academyof Finland), which are gratefully acknowledged. Also, A. H. wishes to thank Göte
Nyman and Jukka Häkkinen of the Department of Psychology of the University of
Helsinki who hosted his civilian service there and made part of the writing possible.
AAPO HYVÄRINEN, JUHA KARHUNEN, ERKKI OJA
Espoo, Finland
March 2001

1
Introduction

Independent component analysis (ICA) is a method for finding underlying factors orcomponents from multivariate (multidimensional) statistical data. What distinguishes
ICA from other methods is that it looks for components that are both statisticallyindependent, and nongaussian. Here we briefly introduce the basic concepts, applications, and estimation principles of ICA.

1.1 LINEAR REPRESENTATION OF MULTIVARIATE DATA
1.1.1

The general statistical setting

A long-standing problem in statistics and related areas is how to find a suitable
representation of multivariate data. Representation here means that we somehowtransform the data so that its essential structure is made more visible or accessible.
In neural computation, this fundamental problem belongs to the area of unsupervised learning, since the representation must be learned from the data itself without
any external input from a supervising “teacher”. A good representation is also acentral goal of many techniques in data mining and exploratory data analysis. In
signal processing, the same problem can be found in feature extraction, and also inthe source separation problem that will be considered below.
Let us assume that the data consists of a number of variables that we have observed
together. Let us denote the number of variables by m and the number of observations
by T . We can then denote the data by xi (t) where the indices take the values
i = 1; :::; m and t = 1; :::; T . The dimensions m and T can be very large.
1

2

INTRODUCTION

A very general formulation of the problem can be stated as follows: What could
be a function from an m-dimensional space to an n-dimensional space such that thetransformed variables give information on the data that is otherwise hidden in thelarge data set. That is, the transformed variables should be the underlying factors orcomponents that describe the essential structure of the data. It is hoped that thesecomponents correspond to some physical causes that were involved in the processthat generated the data in the first place.
In most cases, we consider linear functions only, because then the interpretationof the representation is simpler, and so is its computation. Thus, every component,
say yi , is expressed as a linear combination of the observed variables:yi (t) =Xjwij xj (t); for i = 1; :::; n; j = 1; :::; m(1.1)

where the wij are some coefficients that define the representation. The problem
can then be rephrased as the problem of determining the coefficients wij . Using
linear algebra, we can express the linear transformation in Eq. (1.1) as a matrix
multiplication. Collecting the coefficients wij in a matrix , the equation becomes0 x (t) 1
0y (t)1
B x (t) C
BBy (t)CC
B@ ... CA = W BB@ ... CCA
1

1

2

2

yn (t)

xm (t)W(1.2)

A basic statistical approach consists of considering the xi (t) as a set of T realizations of m random variables. Thus each set xi (t); t = 1; :::; T is a sample of
one random variable; let us denote the random variable by xi . In this framework,by the statistical properties of the transformed
we could determine the matrix
components yi . In the following sections, we discuss some statistical properties that
could be used; one of them will lead to independent component analysis.

W

1.1.2

Dimension reduction methodsW

is to limit the number of comOne statistical principle for choosing the matrix
ponents yi to be quite small, maybe only 1 or 2, and to determine
so that theyi contain as much information on the data as possible. This leads to a family oftechniques called principal component analysis or factor analysis.
In a classic paper, Spearman [409] considered data that consisted of school performance rankings given to schoolchildren in different branches of study, complemented
by finding a single
by some laboratory measurements. Spearman then determined
linear combination such that it explained the maximum amount of the variation in
the results. He claimed to find a general factor of intelligence, thus founding factor
analysis, and at the same time starting a long controversy in psychology.

W

W

BLIND SOURCE SEPARATION

3

Fig. 1.1 The density function of the Laplacian distribution, which is a typical supergaussian
distribution. For comparison, the gaussian density is given by a dashed line. The Laplaciandensity has a higher peak at zero, and heavier tails. Both densities are normalized to unit
variance and have zero mean.

1.1.3

Independence as a guiding principle

W

Another principle that has been used for determining
is independence: the components yi should be statistically independent. This means that the value of any oneof the components gives no information on the values of the other components.
In fact, in factor analysis it is often claimed that the factors are independent,
but this is only partly true, because factor analysis assumes that the data has agaussian distribution. If the data is gaussian, it is simple to find components thatare independent, because for gaussian data, uncorrelated components are always
independent.
In reality, however, the data often does not follow a gaussian distribution, and thesituation is not as simple as those methods assume. For example, many real-world
data sets have supergaussian distributions. This means that the random variables
take relatively more often values that are very close to zero or very large. In otherwords, the probability density of the data is peaked at zero and has heavy tails (large
values far from zero), when compared to a gaussian density of the same variance. An
example of such a probability density is shown in Fig. 1.1.
This is the starting point of ICA. We want to find statistically independent components, in the general case where the data is nongaussian.

1.2 BLIND SOURCE SEPARATIONLet us now look at the same problem of finding a good representation, from adifferent viewpoint. This is a problem in signal processing that also shows thehistorical background for ICA.

4

1.2.1

INTRODUCTION

Observing mixtures of unknown signals

Consider a situation where there are a number of signals emitted by some physicalobjects or sources. These physical sources could be, for example, different brain
areas emitting electric signals; people speaking in the same room, thus emitting
speech signals; or mobile phones emitting their radio waves. Assume further that
there are several sensors or receivers. These sensors are in different posit ions, so thateach records a mixture of the original source signals with slightly different weights.
For the sake of simplicity of exposition, let us say there are three underlying
source signals, and also three observed signals. Denote by x1 (t); x2 (t) and x3 (t) the
observed signals, which are the amplitudes of the recorded signals at time point t,
and by s1 (t); s2 (t) and s3 (t) the original signals. The xi (t) are then weighted sumsof the si (t), where the coefficients depend on the distances between the sources andthe sensors:

x1 (t) = a11 s1 (t) + a12 s2 (t) + a13 s3 (t)x2 (t) = a21 s1 (t) + a22 s2 (t) + a23 s3 (t)x3 (t) = a31 s1 (t) + a32 s2 (t) + a33 s3 (t)(1.3)

The aij are constant coefficients that give the mixing weights. They are assumed
unknown, since we cannot know the values of aij without knowing all the propertiesof the physical mixing system, which can be extremely difficult in general. The
source signals si are unknown as well, since the very problem is that we cannot
record them directly.
As an illustration, consider the waveforms in Fig. 1.2. These are three linear
mixtures xi of some original source signals. They look as if they were completely
noise, but actually, there are some quite structured underlying source signals hiddenin these observed signals.
What we would like to do is to find the original signals from the mixtures
x1 (t); x2 (t) and x3 (t). This is the blind source separation (BSS) problem. Blindmeans that we know very little if anything about the original sources.
We can safely assume that the mixing coefficients aij are different enough to makethe matrix that they form invertible. Thus there exists a matrix
with coefficientswij , such that we can separate the si asWW

s1 (t) = w11 x1 (t) + w12 x2 (t) + w13 x3 (t)s2 (t) = w21 x1 (t) + w22 x2 (t) + w23 x3 (t)s3 (t) = w31 x1 (t) + w32 x2 (t) + w33 x3 (t)(1.4)

Such a matrix
could be found as the inverse of the matrix that consists of themixing coefficients aij in Eq. 1.3, if we knew those coefficients aij .
Now we see that in fact this problem is mathematically similar to the one where
we wanted to find a good representation for the random data in xi (t), as in (1.2).
Indeed, we could consider each signal xi (t); t = 1; :::; T as a sample of a randomvariable xi , so that the value of the random variable is given by the amplitudes ofthat signal at the time points recorded.

BLIND SOURCE SEPARATION

5

6

4

2

0

−2

−4

−6

−8

0

50

100

150

200

250

300

350

400

450

500

0

50

100

150

200

250

300

350

400

450

500

0

50

100

150

200

250

300

350

400

450

500

4

2

0

−2

−4

−6

−8
4

2

0

−2

−4

−6

−8

Fig. 1.2 The observed signals that are assumed to be mixtures of some underlying source
signals.

1.2.2

Source separation based on independence

The question now is: How can we estimate the coefficients wij in (1.4)? We wantto obtain a general method that works in many different circumstances, and in factprovides one answer to the very general problem that we started with: finding agood representation of multivariate data. Therefore, we use very general statisticalproperties. All we observe is the signals x1 ; x2 and x3 , and we want to find a matrix
so that the representation is given by the original source signals s1 ; s2 , and s3 .
A surprisingly simple solution to the problem can be found by considering justthe statistical independence of the signals. In fact, if the signals are not gaussian, it
is enough to determine the coefficients wij , so that the signals

W

y1 (t) = w11 x1 (t) + w12 x2 (t) + w13 x3 (t)(1.5)
y2 (t) = w21 x1 (t) + w22 x2 (t) + w23 x3 (t)y3 (t) = w31 x1 (t) + w32 x2 (t) + w33 x3 (t)are statistically independent. If the signals y1 ; y2 , and y3 are independent, then they
are equal to the original signals s1 ; s2 , and s3 . (They could be multiplied by somescalar constants, though, but this has little significance.)
Using just this information on the statistical independence, we can in fact estimate
for the signals in Fig. 1.2. What we obtain are the source
the coefficient matrix
signals in Fig. 1.3. (These signals were estimated by the FastICA algorithm that
we shall meet in several chapters of this book.) We see that from a data set that
seemed to be just noise, we were able to estimate the original source signals, using
an algorithm that used the information on the independence only. These estimatedsignals are indeed equal to those that were used in creating the mixtures in Fig. 1.2

W

6

INTRODUCTION
3

2

1

0

−1

−2

−3

0

50

100

150

200

250

300

350

400

450

500

400

450

500

400

450

500

2

1.5

1

0.5

0

−0.5

−1

−1.5

−2

0

50

100

150

200

250

300

350

4

2

0

−2

−4

−6

−8

0

50

100

150

200

250

300

350

Fig. 1.3 The estimates of the original source signals, estimated using only the observed
mixture signals in Fig. 1.2. The original signals were found very accurately.

(the original signals are not shown, but they really are virtually identical to what thealgorithm found). Thus, in the source separation problem, the original signals werethe “independent components” of the data set.

1.3 INDEPENDENT COMPONENT ANALYSIS
1.3.1

Definition

We have now seen that the problem of blind source separation boils down to finding
a linear representation in which the components are statistically independent. In
practical situations, we cannot in general find a representation where the components
are really independent, but we can at least find components that are as independent
as possible.
This leads us to the following definition of ICA, which will be consideredin more detail in Chapter 7. Given a set of observations of random variables(x1 (t); x2 (t); :::; xn (t)), where t is the time or sample index, assume that they are
generated as a linear mixture of independent components:

0x (t) 1 0s (t) 1
BBx (t) CC BBs (t) CC
B@ .. CA = A B@ .. CA
.
.

A

1

1

2

2

xn (t)sn (t)(1.6)

where is some unknown matrix. Independent component analysis now consists of
estimating both the matrix and the si (t), when we only observe the xi (t). NoteAINDEPENDENT COMPONENT ANALYSIS

7

that we assumed here that the number of independent components si is equal to thenumber of observed variables; this is a simplifying assumption that is not completely
necessary.
Alternatively, we could define ICA as follows: find a linear transformation given by
as in (1.2), so that the random variables yi ; i = 1; :::; n are as independent
a matrix
as possible. This formulation is not really very different from the previous one, sinceafter estimating , its inverse gives .
It can be shown (see Section 7.5) that the problem is well-defined, that is, themodel in (1.6) can be estimated if and only if the components si are nongaussian.
This is a fundamental requirement that also explains the main difference between
ICA and factor analysis, in which the nongaussianity of the data is not taken into
account. In fact, ICA could be considered as nongaussian factor analysis, since infactor analysis, we are also modeling the data as linear mixtures of some underlying
factors.

W

A

1.3.2

W

Applications

Due to its generality the ICA model has applications in many different areas, someof which are treated in Part IV. Some examples are:

 In brain imaging, we often have different sources in the brain emit signals thatare mixed up in the sensors outside of the head, just like in the basic blindsource separation model (Chapter 22).

 In econometrics, we often have parallel time series, and ICA could decompose

them into independent components that would give an insight to the structureof the data set (Section 24.1).

 A somewhat different application is in image feature extraction, where we wantto find features that are as independent as possible (Chapter 21).

1.3.3

How to find the independent components

It may be very surprising that the independent components can be estimated fromlinear mixtures with no more assumptions than their independence. Now we will try
to explain briefly why and how this is possible; of course, this is the main subject of
the book (especially of Part II).
Uncorrelatedness is not enough The first thing to note is that independence
is a much stronger property than uncorrelatedness. Considering the blind source separation problem, we could actually find many different uncorrelated representationsof the signals that would not be independent and would not separate the sources.
Uncorrelatedness in itself is not enough to separate the components. This is also thereason why principal component analysis (PCA) or factor analysis cannot separate
the signals: they give components that are uncorrelated, but little more.

8

INTRODUCTION

Fig. 1.4 A sample of independent components s1 and s2 with uniform distributions.
Horizontal axis: s1 ; vertical axis: s2 .

Fig. 1.5 Uncorrelated mixtures x1 and x2 .
Horizontal axis: x1 ; vertical axis: x2 .

Let us illustrate this with a simple example using two independent components
with uniform distributions, that is, the components can have any values inside a
certain interval with equal probability. Data from two such components are plottedin Fig. 1.4. The data is uniformly distributed inside a square due to the independenceof the components.
Now, Fig. 1.5 shows two uncorrelated mixtures of those independent components.
Although the mixtures are uncorrelated, one sees clearly that the distributions are not
the same. The independent components are still mixed, using an orthogonal mixing
matrix, which corresponds to a rotation of the plane. One can also see that in Fig. 1.5
the components are not independent: if the component on the horizontal axis has avalue that is near the corner of the square that is in the extreme right, this clearly
restricts the possible values that the components on the vertical axis can have.
In fact, by using the well-known decorrelation methods, we can transform any
linear mixture of the independent components into uncorrelated components, in whichcase the mixing is orthogonal (this will be proven in Section 7.4.2). Thus, the trickin ICA is to estimate the orthogonal transformation that is left after decorrelation.
This is something that classic methods cannot estimate because they are based on
essentially the same covariance information as decorrelation.
Figure 1.5 also gives a hint as to why ICA is possible. By locating the edges ofthe square, we could compute the rotation that gives the original components. In thefollowing, we consider a couple more sophisticated methods for estimating ICA.
Nonlinear decorrelation is the basic ICA method One way of stating how
independence is stronger than uncorrelatedness is to say that independence implies
nonlinear uncorrelatedness: If s1 and s2 are independent, then any nonlinear transformations g (s1 ) and h(s2 ) are uncorrelated (in the sense that their covariance isINDEPENDENT COMPONENT ANALYSIS

9zero). In contrast, for two random variables that are merely uncorrelated, such
nonlinear transformations do not have zero covariance in general.
Thus, we could attempt to perform ICA by a stronger form of decorrelation, by
finding a representation where the yi are uncorrelated even after some nonlinear
transformations. This gives a simple principle of estimating the matrix :WW

ICA estimation principle 1: Nonlinear decorrelation. Find the matrix
so thatfor any i = j , the components yi and yj are uncorrelated, and the transformed
components g (yi ) and h(yj ) are uncorrelated, where g and h are some suitablenonlinear functions.

6This is a valid approach to estimating ICA: If the nonlinearities are properly chosen,
the method does find the independent components. In fact, computing nonlinear
correlations between the two mixtures in Fig. 1.5, one would immediately see thatthe mixtures are not independent.
Although this principle is very intuitive, it leaves open an important question:
How should the nonlinearities g and h be chosen? Answers to this question can befound be using principles from estimation theory and information theory. Estimation
theory provides the most classic method of estimating any statistical model: themaximum likelihood method (Chapter 9). Information theory provides exact measuresof independence, such as mutual information (Chapter 10). Using either one of these
theories, we can determine the nonlinear functions g and h in a satisfactory way.
Independent components are the maximally nongaussian components
Another very intuitive and important principle of ICA estimation is maximum nongaussianity (Chapter 8). The idea is that according to the central limit theorem,
sums of nongaussian random variables are closer to gaussian that the original ones.
Therefore, if we take a linear combination y = i bi xi of the observed mixture
variables (which, because of the linear mixing model, is a linear combination of the
independent components as well), this will be maximally nongaussian if it equals
one of the independent components. This is because if it were a real mixture of two
or more components, it would be closer to a gaussian distribution, due to the central
limit theorem.
Thus, the principle can be stated as follows

PP

ICA estimation principle 2: Maximum nongaussianity. Find the local maxima
of nongaussianity of a linear combination y =i bi xi under the constraintthat the variance of y is constant. Each local maximum gives one independent
component.

To measure nongaussianity in practice, we could use, for example, the kurtosis.
Kurtosis is a higher-order cumulant, which are some kind of generalizations ofvariance using higher-order polynomials. Cumulants have interesting algebraic andstatistical properties which is why they have an important part in the theory of ICA.
For example, comparing the nongaussianities of the components given by the axes
in Figs. 1.4 and 1.5, we see that in Fig. 1.5 they are smaller, and thus Fig. 1.5 cannot
give the independent components (see Chapter 8).
An interesting point is that this principle of maximum nongaussianity shows
the very close connection between ICA and an independently developed technique

10

INTRODUCTION

called projection pursuit. In projection pursuit, we are actually looking for maximallynongaussian linear combinations, which are used for visualization and other purposes.
Thus, the independent components can be interpreted as projection pursuit directions.
When ICA is used to extract features, this principle of maximum nongaussianity
also shows an important connection to sparse coding that has been used in neuroscientific theories of feature extraction (Chapter 21). The idea in sparse coding is
to represent data with components so that only a small number of them are “active”
at the same time. It turns out that this is equivalent, in some situations, to finding
components that are maximally nongaussian.
The projection pursuit and sparse coding connections are related to a deep resultthat says that ICA gives a linear representation that is as structured as possible.
This statement can be given a rigorous meaning by information-theoretic concepts(Chapter 10), and shows that the independent components are in many ways easierto process than the original random variables. In particular, independent components
are easier to code (compress) than the original variables.
ICA estimation needs more than covariances There are many other methods for estimating the ICA model as well. Many of them will be treated in thisbook. What they all have in common is that they consider some statistics that are notcontained in the covariance matrix (the matrix that contains the covariances between
all pairs of the xi ).
Using the covariance matrix, we can decorrelate the components in the ordinary
linear sense, but not any stronger. Thus, all the ICA methods use some form of
higher-order statistics, which specifically means information not contained in thecovariance matrix. Earlier, we encountered two kinds of higher-order information:
the nonlinear correlations and kurtosis. Many other kinds can be used as well.
Numerical methods are important In addition to the estimation principle, one
has to find an algorithm for implementing the computations needed. Because theestimation principles use nonquadratic functions, the computations needed usuallycannot be expressed using simple linear algebra, and therefore they can be quite demanding. Numerical algorithms are thus an integral part of ICA estimation methods.
The numerical methods are typically based on optimization of some objectivefunctions. The basic optimization method is the gradient method. Of particularinterest is a fixed-point algorithm called FastICA that has been tailored to exploit theparticular structure of the ICA problem. For example, we could use both of thesemethods to find the maxima of the nongaussianity as measured by the absolute valueof kurtosis.

HISTORY OF ICA

11

1.4 HISTORY OF ICA
The technique of ICA, although not yet the name, was introduced in the early 1980s
by J. Hérault, C. Jutten, and B. Ans [178, 179, 16]. As recently reviewed by Jutten[227], the problem first came up in 1982 in a neurophysiological setting. In a
simplified model of motion coding in muscle contraction, the outputs x1 (t)and x2 (t)were two types of sensory signals measuring muscle contraction, and s1 (t) and s2 (t)were the angular position and velocity of a moving joint. Then it is not unreasonable
to assume that the ICA model holds between these signals. The nervous system
must be somehow able to infer the position and velocity signals s1 (t); s2 (t) from themeasured responses x1 (t); x2 (t). One possibility for this is to learn the inverse modelusing the nonlinear decorrelation principle in a simple neural network. Hérault andJutten proposed a specific feedback circuit to solve the problem. This approach iscovered in Chapter 12.
All through the 1980s, ICA was mostly known among French researchers, withlimited influence internationally. The few ICA presentations in international neuralnetwork conferences in the mid-1980s were largely buried under the deluge of interest in back-propagation, Hopfield networks, and Kohonen’s Self-Organizing Map(SOM), which were actively propagated in those times. Another related field washigher-order spectral analysis, on which the first international workshop was organized in 1989. In this workshop, early papers on ICA by J.-F. Cardoso [60] andP. Comon [88] were given. Cardoso used algebraic methods, especially higher-order
cumulant tensors, which eventually led to the JADE algorithm [72]. The use offourth-order cumulants has been earlier proposed by J.-L. Lacoume [254]. In signal
processing literature, classic early papers by the French group are [228, 93, 408, 89].
A good source with historical accounts and a more complete list of references is[227].
In signal processing, there had been earlier approaches in the related problem ofblind signal deconvolution [114, 398]. In particular, the results used in multichannel
blind deconvolution are very similar to ICA techniques.
The work of the scientists in the 1980’s was extended by, among others, A. Cichocki and R. Unbehauen, who were the first to propose one of the presently mostpopular ICA algorithms [82, 85, 84]. Some other papers on ICA and signal separationfrom early 1990s are [57, 314]. The “nonlinear PCA” approach was introduced by thepresent authors in [332, 232]. However, until the mid-1990s, ICA remained a rather
small and narrow research effort. Several algorithms were proposed that wor ked,
usually in somewhat restricted problems, but it was not until later that the rigorous
connections of these to statistical optimization criteria were exposed.
ICA attained wider attention and growing interest after A.J. Bell and T.J. Sejnowski
published their approach based on the infomax principle [35, 36] in the mid-90’s.
This algorithm was further refined by S.-I. Amari and his co-workers using the natural
gradient [12], and its fundamental connections to maximum likelihood estimation, as
well as to the Cichocki-Unbehauen algorithm, were established. A couple of years
later, the present authors presented the fixed-point or FastICA algorithm, [210, 192,

12

INTRODUCTION

197], which has contributed to the application of ICA to large-scale problems due toits computational efficiency.
Since the mid-1990s, there has been a growing wave of papers, workshops, andspecial sessions devoted to ICA. The first international workshop on ICA was held in
Aussois, France, in January 1999, and the second workshop followed in June 2000
in Helsinki, Finland. Both gathered more than 100 researchers working on ICA andblind signal separation, and contributed to the transformation of ICA to an establishedand mature field of research.

Part I

MATHEMATICALPRELIMINARIES2
Random Vectors andIndependence
In this chapter, we review central concepts of probability theory,statistics, and random
processes. The emphasis is on multivariate statistics and random vectors. Mattersthat will be needed later in this book are discussed in more detail, including, forexample, statistical independence and higher-order statistics. The reader is assumed
to have basic knowledge on single variable probability theory, so that fundamental
definitions such as probability, elementary events, and random variables are familiar.
Readers who already have a good knowledge of multivariate statistics can skip mostof this chapter. For those who need a more extensive review or more information on
advanced matters, many good textbooks ranging from elementary ones to advanced
treatments exist. A widely used textbook covering probability, random variables, andstochastic processes is [353].

2.1 PROBABILITY DISTRIBUTIONS AND DENSITIES
2.1.1

Distribution of a random variable

In this book, we assume that random variables are continuous-valued unless statedotherwise. The cumulative distribution function (cdf) Fx of a random variable x at
point x = x0 is defined as the probability that x  x0 :Fx (x0 ) = P (x  x0 )(2.1)

Allowing x0 to change from ?1 to 1 defines the whole cdf for all values of x.
Clearly, for continuous random variables the cdf is a nonnegative, nondecreasing(often monotonically increasing) continuous function whose values lie in the interval
15

16

RANDOM VECTORS AND INDEPENDENCEσσmFig. 2.1 A gaussian probability density function with mean m and standard deviation  .

0  Fx (x)  1. From the definition, it also follows directly that Fx (?1) = 0, andFx (+1) = 1.

Usually a probability distribution is characterized in terms of its density function
rather than cdf. Formally, the probability density function (pdf) px (x) of a continuous
random variable x is obtained as the derivative of its cumulative distribution function:x (x)px(x0 ) = dFdx

(2.2)

x=x0

In practice, the cdf is computed from the known pdf by using the inverse relationshipFx (x0 ) =

Zx

0

?1

px( )d(2.3)

For simplicity, Fx (x) is often denoted by F (x) and px (x) by p(x), respectively. The
subscript referring to the random variable in question must be used when confusion
is possible.
Example 2.1 The gaussian (or normal) probability distribution is used in numerous
models and applications, for example to describe additive noise. Its density function
is given by)px (x) = p 1 2 exp ? (x 2?m
2
2

2



(2.4)

17

PROBABILITY DISTRIBUTIONS AND DENSITIES

Here the parameter m (mean) determines the peak point of the symmetric density
function, and  (standard deviation), its effective width (flatness or sharpness of thepeak). See Figure 2.1 for an illustration.
Generally, the cdfpof the gaussian density cannot be evaluated in closed form using(2.3). The term 1= 2 2 in front of the density (2.4) is a normalizing factor thatguarantees that the cdf becomes unity when x0 ! 1. However, the values of the
cdf can be computed numerically using, for example, tabulated values of the error
function





Zx
2
1
exp ? 2 d
erf(x) = p
2 0

(2.5)

The error function is closely related to the cdf of a normalized gaussian density, forwhich the mean m = 0 and the variance  2 = 1. See [353] for details.
2.1.2

Distribution of a random vector

Assume now thatx is an n-dimensional random vectorx = (x ; x ; : : : ; xn )T
1

(2.6)

2

where T denotes the transpose. (We take the transpose because all vectors in this bookare column vectors. Note that vectors are denoted by boldface lowercase letters.) The
components x1 ; x2 ; : : : ; xn of the column vector are continuous random variables.
The concept of probability distribution generalizes easily to such a random vector.
In particular, the cumulative distribution function of is defined byxx

Fx (x0 ) = P (x  x0 )

(2.7)

x

where P (:) again denotes the probability of the event in parentheses, and 0 issome constant value of the random vector . The notation  0 means that eachcomponent of the vector is less than or equal to the respective component of thevector 0 . The multivariate cdf in Eq. (2.7) has similar properties to that of a single
random variable. It is a nondecreasing function of each component, with values lying
in the interval 0  Fx ( )  1. When all the components of approach infinity,
Fx ( ) achieves its upper limit 1; when any component xi ! ?1, Fx ( ) = 0.
The multivariate probability density function px ( ) of is defined as the derivativeof the cumulative distribution function Fx ( ) with respect to all components of therandom vector :

x

x

x

x x

x

x

x

x

x

px (x0 ) = @x@ @x@ : : : @x@ Fx (x)
1

Hence

Fx (x0 ) =

x x

Zx

0

?1

px (x)dx =n2

Z x; Z x;
01

02

?1 ?1

:::

Z x ;n
0

?1

x

x=x0

px (x)dxn : : : dx2 dx1

(2.8)

(2.9)

18

RANDOM VECTORS AND INDEPENDENCE

where x0;i is the ith component of the vector

Z 1
+

?1

x . Clearly,0px(x)dx = 1(2.10)

This provides the appropriate normalization condition that a true multivariate probability density px ( ) must satisfy.
In many cases, random variables have nonzero probability density functions only
on certain finite intervals. An illustrative example of such a case is presented below.

x

Example 2.2 Assume that the probability density function of a two-dimensionalrandom vector = (x; y )T iszpz (z) = px;y (x; y) =(37

(2 ? x)(x + y); x 2 [0; 2]; y 2 [0; 1]0;elsewherezz

Let us now compute the cumulative distribution function of . It is obtained byintegrating over both x and y , taking into account the limits of the regions where thedensity is nonzero. When either x  0 or y  0, the density pz ( ) and consequently
also the cdf is zero. In the region where 0 < x  2 and 0 < y  1, the cdf is givenbyFz (z) = Fx;y (x; y) =Z yZ x 3

7 (2 ? )( + )dd
= 73 xy x + y ? 31 x2 ? 14 xy
In the region where 0 < x  2 and y > 1, the upper limit in integrating over y
becomes equal to 1, and the cdf is obtained by inserting y = 1 into the preceding
expression. Similarly, in the region x > 2 and 0 < y  1, the cdf is obtained byinserting x = 2 to the preceding formula. Finally, if both x > 2 and y > 1, thecdf becomes unity, showing that the probability density pz (z) has been normalized



0

0

correctly. Collecting these results yields

8>0;x  0 or y  0>>< xy(x + y ? x ? xy); 0 < x  2; 0 < y  1
Fz (z) = > x(1 + x ? x );0 < x  2; y > 1>y( + y);x > 2; 0 < y  1>:1;
x > 2 and y > 1
3
7
3
7
6
7

2.1.3

2
3

3
4
1
2

1 2
3
1 2
3

1
4

Joint and marginal distributions

The joint distribution of two different random vectors can be handled in a similarmanner. In particular, let be another random vector having in general a dimension
m different from the dimension n of . The vectors and can be concatenated to

y

x

x

y

EXPECTATIONS AND MOMENTS

z

19

x y

a "supervector" T = ( T ; T ), and the preceding formulas used directly. The cdf
that arises is called the joint distribution function of and , and is given byx y
Fx;y (x ; y ) = P (x  x ; y  y )
0

x
y y

y

0

0

(2.11)

0x y
x xHere 0 and 0 are some constant vectors having the same dimensions as and ,respectively, and Eq. (2.11) defines the joint probability of the event  0 and 0.
The joint density function px;y ( ; ) of and is again defined formally by differentiating the joint distribution function Fx;y ( ; ) with respect to all componentsof the random vectors and . Hence, the relationship

xy

x

y

Fx;y (x0 ; y0 ) =

x

y
xy

Zx Zy

0

0

?1 ?1

px;y (; )ddxxy(2.12)

yholds, and the value of this integral equals unity when both 0 ! 1 and 0 ! 1.
The marginal densities px ( ) of and py ( ) of are obtained by integratingover the other random vector in their joint density px;y ( ; ):xxpx(x) =py (y) =

y

Z1

?1

Z1

?1

y

px;y (x;  )d(2.13)

px;y (; y)d(2.14)

Example 2.3 Consider the joint density given in Example 2.2. The marginal densities of the random variables x and y arepx (x) ==py (y) =
=

Z 3
1

( 7
0

3
7

0

elsewhere

2

( 7
2
7

x 2 [0; 2](1 + 23 x ? x2 ) x 2 [0; 2]

Z 3
0(2 ? x)(x + y)dy;(2 ? x)(x + y)dx;

y 2 [0; 1](2 + 3y); y 2 [0; 1]0;elsewhere

2.2 EXPECTATIONS AND MOMENTS
2.2.1

Definition and general properties

In practice, the exact probability density function of a vector or scalar valued randomvariable is usually unknown. However, one can use instead expectations of some20

RANDOM VECTORS AND INDEPENDENCE

functions of that random variable for performing useful analyses and processing. A
great advantage of expectations is that they can be estimated directly from the data,
even though they are formally defined in terms of the density function.
Let ( ) denote any quantity derived from the random vector . The quantity( ) may be either a scalar, vector, or even a matrix. The expectation of ( ) isdenoted by Ef ( )g, and is defined by

gx

gx

x

gx

gx

Ef ( )g =

Z1

?1

gx

g(x)px (x)dx(2.15)

x

Here the integral is computed over all the components of .The integration operation
is applied separately to every component of the vector or element of the matrix,
yielding as a result another vector or matrix of the same size. If ( ) = , we get the
expectation Ef g of ; this is discussed in more detail in the next subsection.
Expectations have some important fundamental properties.

x

gx x

x

x

1. Linearity. Let i , i = 1; : : : ; m be a set of different random vectors, and ai ,i = 1; : : : ; m, some nonrandom scalar coefficients. Then

m
X

Ef

i=1

ai xi g =mX
i=1

ai Efxi g(2.16)

x

2. Linear transformation. Let be an m-dimensional random vector, andsome nonrandom k  m and m  l matrices, respectively. ThenB

Axg = AEfxg;

Ef

3. Transformation invariance. Let
random vector . Then

x

Z1

?1

y

xBg = EfxgB

Ef

A and
(2.17)

y = g(x) be a vector-valued function of theypy (y)dy =Z1?1g(x)px(x)dx(2.18)

gx

Thus Ef g = Ef ( )g, even though the integrations are carried out overdifferent probability density functions.
These properties can be proved using the definition of the expectation operator
and properties of probability density functions. They are important and very helpfulin practice, allowing expressions containing expectations to be simplified without
actually needing to compute any integrals (except for possibly in the last phase).
2.2.2

Mean vector and correlation matrixx

Moments of a random vector are typical expectations used to characterize it. They
are obtained when ( ) consists of products of components of . In particular, the

gx

x

21

EXPECTATIONS AND MOMENTS

first moment of a random vector
as the expectation of :x

x is called the mean vector mx of x. It is defined

mx = Efxg =Each component mxi of the n-vector

mxi = Efxi g =

Z1

?1

Z1
?1

xpx(x)dx(2.19)

mx is given byxi px (x)dx =Z1?1xi pxi (xi )dxi

(2.20)

xwhere pxi (xi ) is the marginal density of the ith component xi of . This is because
integrals over all the other components of reduce to unity due to the definition of
the marginal density.
Another important set of moments consists of correlations between pairs of components of . The correlation rij between the ith and j th component of is givenby the second moment

x

x

x

rij = Efxi xj g =Z1?1xi xj px (x)dx =

Z1Z1

?1 ?1

xi xj pxi;xj (xi ; xj )dxj dxi(2.21)

Note that correlation can be negative or positive.
The n  n correlation matrix

Rx = EfxxT g
x(2.22)of the vector represents in a convenient form all its correlations, rij being theelement in row i and column j of x .
The correlation matrix x has some important properties:

RR

1. It is a symmetric matrix:

Rx = RTx .

2. It is positive semidefinite:aT Rxa  0(2.23)
for all n-vectors a. Usually in practice Rx is positive definite, meaning thatfor any vector a =6 0, (2.23) holds as a strict inequality.
3. All the eigenvalues of Rx are real and nonnegative (positive if Rx is positive
definite). Furthermore, all the eigenvectors of Rx are real, and can always bechosen so that they are mutually orthonormal.

Higher-order moments can be defined analogously, but their discussion is postponed to Section 2.7. Instead, we shall first consider the corresponding central andsecond-order moments for two different random vectors.

22

RANDOM VECTORS AND INDEPENDENCE

2.2.3

Covariances and joint moments

Central moments are defined in a similar fashion to usual moments, but the mean
vectors of the random vectors involved are subtracted prior to computing the expectation. Clearly, central moments are only meaningful above the first order. The
quantity corresponding to the correlation matrix x is called the covariance matrix
x of , and is given by

C

R

x

Cx = Ef(x ? mx)(x ? mx)T g(2.24)

cij = Ef(xi ? mi )(xj ? mj )g(2.25)The elements

Cof the n  n matrix x are called covariances, and they are the central moments
corresponding to the correlations 1 rij defined in Eq. (2.21).
The covariance matrix x satisfies the same properties as the correlation matrix
x. Using the properties of the expectation operator, it is easy to see that

C

R

m

0

Rx = Cx + mxmTx(2.26)

If the mean vector x = , the correlation and covariance matrices become thesame. If necessary, the data can easily be made zero mean by subtracting the(estimated) mean vector from the data vectors as a preprocessing step. This is a usual
practice in independent component analysis, and thus in later chapters, we simply
denote by x the correlation/covariance matrix, often even dropping the subscript
for simplicity.
For a single random variable x, the mean vector reduces to its mean value mx =Efxg, the correlation matrix to the second moment E fx2 g, and the covariance matrix
to the variance of xCx

x2 = Ef(x ? mx)2 g(2.27)The relationship (2.26) then takes the simple form Efx2 g = x2 + m2x .
The expectation operation can be extended for functions ( ; ) of two different
random vectors and in terms of their joint density:xy

g x y)g =Ef ( ;

gxy

Z1Z1

?1 ?1

g(x; y)px;y (x; y)dy dx
x(2.28)y

The integrals are computed over all the components of and .
Of the joint expectations, the most widely used are the cross-correlation matrix

Rxy = EfxyT g
1 In classic statistics, the correlation coefficients(2.29)

ij = (c ccij)1=2 are used, and the matrix consisting ofii jj

them is called the correlation matrix. In this book, the correlation matrix is defined by the formula (2.22),
which is a common practice in signal processing, neural networks, and engineering.

23

EXPECTATIONS AND MOMENTS
5

5

y

4

3

3

2

2

1

1

x

0

0

−1

−1

−2

−2

−3

−3

−4

−4

−5
−5

−4

−3

−2

−1

0

y

4

1

2

3

4

5

Fig. 2.2 An example of negative covariance
between the random variables x and y .

−5
−5

x

−4

−3

−2

−1

0

1

2

3

4

5

Fig. 2.3 An example of zero covariance between the random variables x and y .

and the cross-covariance matrix

Cxy = Ef(x ? mx)(y ? my)T g(2.30)
Note that the dimensions of the vectors x and y can be different. Hence, the crosscorrelation and -covariance matrices are not necessarily square matrices, and they are
not symmetric in general. However, from their definitions it follows easily that

Rxy = RTyx; Cxy = CTyx
(2.31)
If the mean vectors of x and y are zero, the cross-correlation and cross-covariancematrices become the same. The covariance matrix Cx y of the sum of two random
vectors x and y having the same dimension is often needed in practice. It is easy to+

see that

Cx y = Cx + Cxy + Cyx + Cy+

(2.32)

Correlations and covariances measure the dependence between the random variables using their second-order statistics. This is illustrated by the following example.
Example 2.4 Consider the two different joint distributions px;y (x; y ) of the zeromean scalar random variables x and y shown in Figs. 2.2 and 2.3. In Fig. 2.2, x
and y have a clear negative covariance (or correlation). A positive value of x mostly
implies that y is negative, and vice versa. On the other hand, in the case of Fig. 2.3,
it is not possible to infer anything about the value of y by observing x. Hence, theircovariance cxy  0.

24

RANDOM VECTORS AND INDEPENDENCE

2.2.4

Estimation of expectationsxxUsually the probability density of a random vector is not known, but there is often
available a set of K samples 1 ; 2 ; : : : ; K from . Using them, the expectation(2.15) can be estimated by averaging over the sample using the formula [419]

x x

x

K
X
1
g(x )
Efg(x)g j

K j=1

For example, applying (2.33), we get for the mean vector
estimator, the sample meanm^ x = K1

K
X
j =1

m

(2.33)

mx of x its standard

xj(2.34)

where the hat over is a standard notation for an estimator of a quantity.
Similarly, if instead of the joint density px;y ( ; ) of the random vectors and, we know K sample pairs ( 1 ; 1 ); ( 2 ; 2 ); : : : ; ( K ; K ), we can estimate theexpectation (2.28) by

y

x y

xy

x y

g x y)g  K1

Ef ( ;

K
X
j =1

x y

g(xj ; yj )x

(2.35)For example, for the cross-correlation matrix, this yields the estimation formulaR^ xy = K1

K
X
j =1

xj yjT

Similar formulas are readily obtained for the other correlation type matrices
xx, and xy .

CC

(2.36)

Rxx,

2.3 UNCORRELATEDNESS AND INDEPENDENCE
2.3.1

Uncorrelatedness and whitenessTwo random vectors
is a zero matrix:

x and y are uncorrelated if their cross-covariance matrix Cxy
Cxy = Ef(x ? mx)(y ? my)T g = 0(2.37)

This is equivalent to the condition

Rxy = EfxyT g = EfxgEfyT g = mxmTy(2.38)

25

UNCORRELATEDNESS AND INDEPENDENCE

In the special case of two different scalar random variables x and y (for example,
two components of a random vector ), x and y are uncorrelated if their covariance
cxy is zero:zcxy = Ef(x ? mx )(y ? my )g = 0(2.39)

rxy = Efxyg = EfxgEfyg = mx my(2.40)

or equivalently

Again, in the case of zero-mean variables, zero covariance is equivalent to zerocorrelation.
Another important special case concerns the correlations between the components
of a single random vector given by the covariance matrix x defined in (2.24). In
this case a condition equivalent to (2.37) can never be met, because each componentof is perfectly correlated with itself. The best that we can achieve is that different
components of are mutually uncorrelated, leading to the uncorrelatedness condition

x

x

C

x

Cx = Ef(x ? mx)(x ? mx)T g = D(2.41)

D is an n  n diagonal matrix
D = diag(c ; c ; : : : ; cnn) = diag(x1 ; x2 ; : : : ; xn )(2.42)
whose n diagonal elements are the variances xi = Ef(xi ? mxi ) g = cii of thecomponents xi of x.
Here

11

2

22

2

2

2

2

In particular, random vectors having zero mean and unit covariance (and hencecorrelation) matrix, possibly multiplied by a constant variance  2 , are said to bewhite. Thus white random vectors satisfy the conditions

I

mx = 0; Rx = Cx = I(2.43)

where is the n  n identity matrix.
Assume now that an orthogonal transformation defined by an n  n matrix
applied to the random vector . Mathematically, this can be expressed

T isxy = Tx; where TT T = TTT = I(2.44)An orthogonal matrix T defines a rotation (change of coordinate axes) in the ndimensional space, preserving norms and distances. Assuming that x is white, we

get

my = EfTxg = TEfxg = Tmx = 0(2.45)Cy = Ry = EfTx(Tx)T g = TEfxxT gTT= TRx TT = TTT = I(2.46)and26

RANDOM VECTORS AND INDEPENDENCE

yshowing that is white, too. Hence we can conclude that the whiteness property ispreserved under orthogonal transformations. In fact, whitening of the original data
can be made in infinitely many ways. Whitening will be discussed in more detailin Chapter 6, because it is a highly useful and widely used preprocessing step inindependent component analysis.
It is clear that there also exists infinitely many ways to decorrelate the original
data, because whiteness is a special case of the uncorrelatedness property.
Example 2.5 Consider the linear signal modelx = As + nx(2.47)

As

where is an n-dimensional random or data vector, an n  m constant matrix,
an m-dimensional random signal vector, and an n-dimensional random vector thatusually describes additive noise. The correlation matrix of then becomesnx

Rx = EfxxT g = Ef(As + n)(As + n)T g= EfAssT AT g + EfAsnT g + EfnsT AT g + EfnnT g= AEfssT gAT + AEfsnT g + EfnsT gAT + EfnnT g(2.48)= ARs AT + ARsn + RnsAT + Rn
Usually the noise vector n is assumed to have zero mean, and to be uncorrelated withthe signal vector s. Then the cross-correlation matrix between the signal and noise

vectors vanishes:

Rsn = EfsnT g = EfsgEfnT g = 0
Similarly, Rns = 0, and the correlation matrix of x simplifies to
Rx = ARsAT + Rn

(2.49)(2.50)

Another often made assumption is that the noise is white, which means here thatthe components of the noise vector are all uncorrelated and have equal variance2 , so that in (2.50)

n

Rn =  I
2(2.51)

Sometimes, for example in a noisy version of the ICA model (Chapter 15), thecomponents of the signal vector are also mutually uncorrelated, so that the signal
correlation matrix becomes the diagonal matrixsDs = diag(Efs g; Efs g; : : : ; Efsmg)(2.52)
where s ; s ; : : : ; sm are components of the signal vector s. Then (2.50) can be
2
1

1

2
2

2

2

written in the form

Rx = ADsAT +  I =
2

m
X
i=1

aa

Efs2i g i Ti +  2

I(2.53)

27

UNCORRELATEDNESS AND INDEPENDENCEaA

where i is the ith column vector of the matrix .
The noisy linear signal or data model (2.47) is encountered frequently in signal
processing and other areas, and the assumptions made on and vary dependingon the problem at hand. It is straightforward to see that the results derived in thisexample hold for the respective covariance matrices as well.

s

2.3.2

n

Statistical independence

A key concept that constitutes the foundation of independent component analysis isstatistical independence. For simplicity, consider first the case of two different scalarrandom variables x and y . The random variable x is independent of y , if knowing thevalue of y does not give any information on the value of x. For example, x and y canbe outcomes of two events that have nothing to do with each other, or random signals
originating from two quite different physical processes that are in no way related toeach other. Examples of such independent random variables are the value of a dicethrown and of a coin tossed, or speech signal and background noise originating froma ventilation system at a certain time instant.
Mathematically, statistical independence is defined in terms of probability densities. The random variables x and y are said to be independent if and only ifpx;y (x; y) = px (x)py (y)(2.54)
In words, the joint density px;y (x; y ) of x and y must factorize into the productof their marginal densities px (x) and py (y ). Equivalently, independence could be
defined by replacing the probability density functions in the definition (2.54) by therespective cumulative distribution functions, which must also be factorizable.
Independent random variables satisfy the basic property
Efg (x)h(y )g = Efg (x)gEfh(y )g(2.55)

where g (x) and h(y ) are any absolutely integrable functions of x and y , respectively.
This is because
Efg (x)h(y )g =

=

Z1Z1

?1
Z?1
1
?1

g(x)h(y)px;y (x; y)dydx

g(x)px (x)dxZ1?1(2.56)

h(y)py (y)dy = Efg(x)gEfh(y)g

Equation (2.55) reveals that statistical independence is a much stronger property thanuncorrelatedness. Equation (2.40), defining uncorrelatedness, is obtained from theindependence property (2.55) as a special case where both g (x) and h(y ) are linear
functions, and takes into account second-order statistics (correlations or covariances)
only. However, if the random variables have gaussian distributions, independence
and uncorrelatedness become the same thing. This very special property of gaussian
distributions will be discussed in more detail in Section 2.5.
Definition (2.54) of independence generalizes in a natural way for more than
two random variables, and for random vectors. Let ; ; ; : : : ; be random vectors

xyz

28

RANDOM VECTORS AND INDEPENDENCE

which may in general have different dimensions. The independence condition for; ; ; : : : ; is thenxyz

px;y;z;:::(x; y; z; : : : ) = px (x)py (y)pz (z) : : :(2.57)and the basic property (2.55) generalizes tog xg yg z

g x

g y

g z

Ef x ( ) y ( ) z ( ) : : : g = Ef x ( )gEf y ( )gEf z ( )g : : :g x g yg z

(2.58)

xy

where x ( ), y ( ), and z ( ) are arbitrary functions of the random variables , ,and for which the expectations in (2.58) exist.
The general definition (2.57) gives rise to a generalization of the standard notionof statistical independence. The components of the random vector are themselves
scalar random variables, and the same holds for and . Clearly, the components
of can be mutually dependent, while they are independent with respect to thecomponents of the other random vectors and , and (2.57) still holds. A similarargument applies to the random vectors and .

z

x

y

y

z

y
z

x

z

Example 2.6 First consider the random variables x and y discussed in Examples 2.2
and 2.3. The joint density of x and y , reproduced here for convenience,px;y (x; y) =(37

(2 ? x)(x + y); x 2 [0; 2]; y 2 [0; 1]0;elsewhere

is not equal to the product of their marginal densities px (x) and py (y ) computed in
Example 2.3. Hence, Eq. (2.54) is not satisfied, and we conclude that x and y are notindependent. Actually this can be seen directly by observing that the joint density
fx;y (x; y) given above is not factorizable, since it cannot be written as a product of
two functions g (x) and h(y ) depending only on x and y .
Consider then the joint density of a two-dimensional random vector = (x1 ; x2 )T
and a one-dimensional random vector = y given by [419]

(

x

y

px;y (x; y) = (x1 + 3x2 )y; x1 ; x2 2 [0; 1]; y 2 [0; 1]0;elsewherexyUsing the above argument, we see that the random vectors and are statisticallyindependent, but the components x1 and x2 of are not independent. The exact
verification of these results is left as an exercise.

x

2.4 CONDITIONAL DENSITIES AND BAYES’ RULE
Thus far, we have dealt with the usual probability densities, joint densities, andmarginal densities. Still one class of probability density functions consists of conditional densities. They are especially important in estimation theory, which will

CONDITIONAL DENSITIES AND BAYES’ RULE

29

be studied in Chapter 4. Conditional densities arise when answering the following
question: “What is the probability density of a random vector given that anotherrandom vector has the fixed value 0 ?” Here 0 is typically a specific realizationof a measurement vector .
Assuming that the joint density px;y ( ; ) of and and their marginal densities
exist, the conditional probability density of given is defined as

y

y

y

x

y
x

xy
x

y
pxjy (xjy) = pxp;y ((xy;)y)yy(2.59)yx xxThis can be interpreted as follows: assuming that the random vector lies in theregion 0 <  0 + , the probability that lies in the region 0 <  0 +is pxjy ( 0 j 0 ) . Here 0 and 0 are some constant vectors, and both  and 
are small. Similarly,

y y y
x y x

y
x

x

y

x

pyjx(yjx) = pxp;y ((xx;)y)xxy(2.60)yx

In conditional densities, the conditioning quantity, in (2.59) and in (2.60), isthought to be like a nonrandom parameter vector, even though it is actually a random
vector itself.
Example 2.7 Consider the two-dimensional joint density px;y (x; y ) depicted inFig. 2.4. For a given constant value x0 , the conditional distribution(x0 ; y)pyjx (yjx0 ) = px;yp (x )
x0

Hence, it is a one-dimensional distribution obtained by "slicing" the joint distributionp(x; y) parallel to the y-axis at the point x = x0 . Note that the denominator px(x0 ) ismerely a scaling constant that does not affect the shape of the conditional distribution
pyjx(yjx0 ) as a function of y.
Similarly, the conditional distribution pxjy (xjy0 ) can be obtained geometricallyby slicing the joint distribution of Fig. 2.4 parallel to the x-axis at the point y = y0 .
The resulting conditional distributions are shown in Fig. 2.5 for the value x0 = 1:27,
and Fig. 2.6 for y0 = ?0:37.

x

x

y

y

From the definitions of the marginal densities px ( ) of and py ( ) of given in
Eqs. (2.13) and (2.14), we see that the denominators in (2.59) and (2.60) are obtainedby integrating the joint density px;y ( ; ) over the unconditional random vector. This
also shows immediately that the conditional densities are true probability densities
satisfying

xy

Z1

?1

pxjy (jy)d = 1;

x

y

Z1

?1

pyjx(jx)d = 1(2.61)

If the random vectors and are statistically independent, the conditional density

pxjy (xjy) equals to the unconditional density px (x) of x, since x does not depend

30

RANDOM VECTORS AND INDEPENDENCE

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
2
1

2
1

0
0
−1

−1
−2

y

−2

x

Fig. 2.4 A two-dimensional joint density of the random variables x and y .

y

yx

y

in any way on , and similarly pyjx ( j ) = py ( ), and both Eqs. (2.59) and (2.60)
can be written in the form

px;y (x; y) = px (x)py (y)(2.62)

xywhich is exactly the definition of independence of the random vectors and .
In the general case, we get from Eqs. (2.59) and (2.60) two different expressions
for the joint density of and :x ypx;y (x; y) = pyjx(yjx)px (x) = pxjy (xjy)py (y)From this, for example, a solution can be found for the density of:xp (xjy)p (y)pyjx(yjx) = xjy p (x) yx(2.63)y conditioned on(2.64)

where the denominator can be computed by integrating the numerator if necessary:px (x) =Z1?1pxjy (xj )py ( )d

(2.65)

31

THE MULTIVARIATE GAUSSIAN DENSITY
0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0
−2

−1.5

−1

−0.5

0

0.5

1

1.5

0
−2

2

Fig. 2.5 The conditional probability density pyjx(y x = 1:27).

−1.5

−1

−0.5

0

0.5

1

1.5

2

Fig. 2.6 The conditional probability density pxjy (x y = 0:37).

j

j

?

Formula (2.64) (together with (2.65)) is called Bayes’ rule. This rule is important
especially in statistical estimation theory. There typically pxjy ( j ) is the conditional
density of the measurement vector , with denoting the vector of unknown randomparameters. Bayes’ rule (2.64) allows the computation of the posterior density
pyjx( j ) of the parameters , given a specific measurement (observation) vector ,and assuming or knowing the prior distribution py ( ) of the random parameters .
These matters will be discussed in more detail in Chapter 4.
Conditional expectations are defined similarly to the expectations defined earlier,but the pdf appearing in the integral is now the appropriate conditional density. Hence,for example,

x

yx

y

g x y)jyg =

xy

y

y

x
y

Z1

g(; y)pxjy (jy)d(2.66)
This is still a function of the random vector y, which is thought to be nonrandom
while computing the above expectation. The complete expectation with respect toboth x and y can be obtained by taking the expectation of (2.66) with respect to y:
Efg(x; y)g = EfEfg(x; y)jygg(2.67)
Ef ( ;?1

Actually, this is just an alternative two-stage procedure for computing the expectation(2.28), following easily from Bayes’ rule.

2.5 THE MULTIVARIATE GAUSSIAN DENSITY
The multivariate gaussian or normal density has several special properties that make it
unique among probability density functions. Due to its importance, we shall discuss
it more thoroughly in this section.

32

RANDOM VECTORS AND INDEPENDENCE

xConsider an n-dimensional random vector . It is said to be gaussian if theprobability density function of has the form

x





1
exp ? 21 (x ? mx)T C?x 1 (x ? mx)
px (x) = (2)n=2 (det
1
=
2
C)
x

xm

C

(2.68)

Recall that n is the dimension of , x its mean, and x the covariance matrix of. The notation det is used for the determinant of a matrix , in this case x . It
is easy to see that for a single random variable x (n = 1), the density (2.68) reduces
to the one-dimensional gaussian pdf (2.4) discussed briefly in Example 2.1. Notealso that the covariance matrix x is assumed strictly positive definite, which also
implies that its inverse exists.
It can be shown that for the density (2.68)

x

A

A

C

C

x

Ef g =mx ;x ? mx)(x ? mx)T g = Cx

Ef(m

Hence calling x the mean vector andgaussian density is justified.
2.5.1(2.69)

Cx the covariance matrix of the multivariate

Properties of the gaussian density

In the following, we list the most important properties of the multivariate gaussian
density omitting proofs. The proofs can be found in many books; see, for example,[353, 419, 407].
Only first- and second-order statistics are needed Knowledge of the mean
vector x and the covariance matrix x of are sufficient for defining the multivariate gaussian density (2.68) completely. Therefore, all the higher-order moments
must also depend only on x and x . This implies that these moments do not carryany novel information about the gaussian distribution. An important consequence of
this fact and the form of the gaussian pdf is that linear processing methods based on
first- and second-order statistical information are usually optimal for gaussian data.
For example, independent component analysis does not bring out anything new compared with standard principal component analysis (to be discussed later) for gaussiandata. Similarly, linear time-invariant discrete-time filters used in classic statisticalsignal processing are optimal for filtering gaussian data.

m

C

m

x

C

Linear transformations are gaussian If x is a gaussian random vector andy = Ax its linear transformation, then y is also gaussian with mean vector my =Amx and covariance matrix Cy = ACxAT . A special case of this result says thatany linear combination of gaussian random variables is itself gaussian. This result
again has implications in standard independent component analysis: it is impossible
to estimate the ICA model for gaussian data, that is, one cannot blindly separate

33

THE MULTIVARIATE GAUSSIAN DENSITY

gaussian sources from their mixtures without extra knowledge of the sources, as will
be discussed in Chapter 7. 2
Marginal and conditional densities are gaussian Consider now two random
vectors and having dimensions n and m, respectively. Let us collect them in asingle random vector T = ( T ; T ) of dimension n + m. Its mean vector z andcovariance matrix z are

x

y

C

z

x y

m



m 
x ; Cz = Cx Cxy
mz = mCyx Cyy(2.70)CC

Recall that the cross-covariance matrices are transposes of each other: xy = Tyx .
Assume now that has a jointly gaussian distribution. It can be shown that themarginal densities px ( ) and py ( ) of the joint gaussian density pz ( ) are gaussian.
Also the conditional densities pxjy and pyjx are n- and m-dimensional gaussian
densities, respectively. The mean and covariance matrix of the conditional density
pyjx are

z
x

y

z

myjx = my + CyxC?x (x ? mx)
1(2.71)

Cyjx = Cy ? CyxC?x Cxy(2.72)
Similar expressions are obtained for the mean mxjy and covariance matrix Cxjy of1the conditional density pxjy .

Uncorrelatedness and geometrical structure. We mentioned earlier that
uncorrelated gaussian random variables are also independent, a property which is
not shared by other distributions in general. Derivation of this important result is left
to the reader as an exercise. If the covariance matrix x of the multivariate gaussian
density (2.68) is not diagonal, the components of are correlated. Since x is asymmetric and positive definite matrix, it can always be represented in the formCx

Cx = EDET =
E

n
X
i=1

C

i ei eTi

(2.73)e eeHere is an orthogonal matrix (that is, a rotation) having as its columns 1; 2 ; : : : ; n
the n eigenvectors of x , and = diag(1 ; 2 ; : : : ; n ) is the diagonal matrix containing the respective eigenvalues i of x . Now it can readily be verified that
applying the rotation

C

D

C

u = ET (x ? mx)(2.74)

2 It is possible, however, to separate temporally correlated (nonwhite) gaussian sources using their second-order temporal statistics on certain conditions. Such techniques are quite different from standard independent component analysis. They will be discussed in Chapter 18.

34

RANDOM VECTORS AND INDEPENDENCExu

to makes the components of the gaussian distribution of uncorrelated, and hence
also independent.
Moreover, the eigenvalues i and eigenvectors i of the covariance matrix x
reveal the geometrical structure of the multivariate gaussian distribution. The contours of any pdf are defined by curves of constant values of the density, given by theequation px ( ) = constant. For the multivariate gaussian density, this is equivalent
to requiring that the exponent is a constant c:

e

C

x

(x ? mx)T C?x 1 (x ? mx) = c(2.75)

Using (2.73), it is easy to see [419] that the contours of the multivariate gaussian
are hyperellipsoids centered at the mean vector x. The principal axes of thehyperellipsoids are parallel to the eigenvectors i , and the eigenvalues i are therespective variances. See Fig. 2.7 for an illustration.

e

m

e2

x2

e1

1/2

λ2

m

1/2

λ1

x1

Fig. 2.7 Illustration of a multivariate gaussian probability density.

2.5.2

Central limit theorem

Still another argument underlining the significance of the gaussian distribution isprovided by the central limit theorem. Let

xk =

k
X
i=1

zi

(2.76)

be a partial sum of a sequence fzi g of independent and identically distributed randomvariables zi . Since the mean and variance of xk can grow without bound as k ! 1,
consider instead of xk the standardized variables

m xk
yk = xk ?

xk(2.77)

DENSITY OF A TRANSFORMATION

35

where mxk and xk are the mean and variance of xk .
It can be shown that the distribution of yk converges to a gaussian distribution
with zero mean and unit variance when k ! 1. This result is known as the central
limit theorem. Several different forms of the theorem exist, where assumptions on
independence and identical distributions have been weakened. The central limit
theorem is a primary reason that justifies modeling of many random phenomena as
gaussian random variables. For example, additive noise can often be considered toarise as a sum of a large number of small elementary effects, and is therefore naturally
modeled as a gaussian random variable.
The central limit theorem generalizes readily to independent and identically distributed random vectors i having a common mean z and covariance matrix z .
The limiting distribution of the random vectorzmyk = p1

Ck
X(z ? mz)

k i=1 i(2.78)

C

is multivariate gaussian with zero mean and covariance matrix z .
The central limit theorem has important consequences in independent component
analysis and blind source separation. A typical mixture, or component of the data
vector , is of the form

x

xi =

m
X
j =1

aij sj(2.79)

where aij , j = 1; : : : ; m, are constant mixing coefficients and sj , j = 1; : : : ; m,are the m unknown source signals. Even for a fairly small number of sources (say,m = 10) the distribution of the mixture xk is usually close to gaussian. This seems
to hold in practice even though the densities of the different sources are far from eachother and far from gaussianity. Examples of this property can be found in Chapter 8,
as well as in [149].

2.6 DENSITY OF A TRANSFORMATION
Assume now that boththe vector mapping

x and y are n-dimensional random vectors that are related by
y = g(x)(2.80)

x = g? (y)(2.81)

for which the inverse mapping1yexists and is unique. It can be shown that the density py ( ) of
density px ( ) of as follows:xxy is obtained from thepy (y) = j det J g(1g?1 (y))j px(g?1 (y))(2.82)36

RANDOM VECTORS AND INDEPENDENCE

g

Here J is the Jacobian matrix

2 @g x66 @g@x x
J g(x) = 66 @x..
4 .

1( )
1
1( )
2

x
y Ax

@g1 (x)@xn@g2 (x)@x@g2 (1x)
@x2




@g2 (x)
@xn

.

..
.

..



@gn (x) 3@x@gn (1x) 7
@x2 7
7

..
.
@gn (x)@xn

75(2.83)

gx

and gj ( ) is the j th component of the vector function ( ).
In the special case where the transformation (2.80) is linear and nonsingular soand = ?1 , the formula (2.82) simplifies tothat =xx A ypy (y) = j det1 Aj px(A?1 y)(2.84)y

If in (2.84) is multivariate gaussian, then also becomes multivariate gaussian, as
was mentioned in the previous section.
Other kinds of transformations are discussed in textbooks of probability theory[129, 353]. For example, the sum z = x+y , where x and y are statistically independent
random variables, appears often in practice. Because the transformation between the
random variables in this case is not one-to-one,the preceding results cannot be applied
directly. But it can be shown that the pdf of z becomes the convolution integral ofthe densities of x and y [129, 353, 407].
A special case of (2.82) that is important in practice is the so-called probability
integral transformation. If Fx (x) is the cumulative distribution function of a randomvariable x, then the random variable

z = Fx (x)(2.85)

is uniformly distributed on the interval [0; 1]. This result allows generation of randomvariables having a desired distribution from uniformly distributed random numbers.
First, the cdf of the desired density is computed, and then the inverse transformationof (2.85) is determined. Using this, one gets random variables x with the desireddensity, provided that the inverse transformation of (2.85) can be computed.

2.7 HIGHER-ORDER STATISTICS
Up to this point, we have characterized random vectors primarily using their secondorder statistics. Standard methods of statistical signal processing are based on utilization of this statistical information in linear discrete-time systems. Their theory is
well-developed and highly useful in many circumstances. Nevertheless, it is limitedby the assumptions of gaussianity, linearity, stationarity, etc.
From the mid-1980s, interest in higher-order statistical methods began to growin the signal processing community. At the same time, neural networks became
popular with the development of several new, effective learning paradigms. A basic

HIGHER-ORDER STATISTICS

37

idea in neural networks [172, 48] is distributed nonlinear processing of the input
data. A neural network consists of interconnected simple computational units calledneurons. The output of each neuron typically depends nonlinearly on its inputs.
These nonlinearities, for example, the hyperbolic tangent tanh (u), also implicitly
introduce higher-order statistics for processing. This can be seen by expanding thenonlinearities into their Taylor series; for example,

2 u5 ?   tanh(u) = u ? 13 u3 + 15

(2.86)

wx

The scalar quantity u is in many neural networks the inner product u = T of theweight vector of the neuron and its input vector . Inserting this into (2.86) shows
clearly that higher-order statistics of the components of the vector are involved in
the computations.
Independent component analysis and blind source separation require the use ofhigher-order statistics either directly or indirectly via nonlinearities. Therefore, we
discuss in the following basic concepts and results that will be needed later.

w

2.7.1

x

x

Kurtosis and classification of densities

In this subsection, we deal with the simple higher-order statistics of one scalarrandom variable. In spite of their simplicity, these statistics are highly useful in many
situations.
Consider a scalar random variable x with the probability density function px (x).
The j th moment j of x is defined by the expectation

j = Efxj g =Z1?1

 j px( )d;

j = 1; 2; : : :(2.87)and the j th central moment j of x respectively byj = Ef(x ?

j1) g =

Z1?1( ? mx)j px( )d:

j = 1; 2; : : :(2.88)

The central moments are thus computed around the mean mx of x, which equals
its first moment 1 . The second moment 2 = Efx2 g is the average power of x.
The central moments 0 = 1 and 1 = 0 are insignificant, while the second central
moment 2 = x2 is the variance of x.
Before proceeding, we note that there exist distributions for which all the moments are not finite. Another drawback of moments is that knowing them does notnecessarily specify the probability density function uniquely. Fortunately, for mostof the distributions arising commonly all the moments are finite, and their knowledge
is in practice equivalent to the knowledge of their probability density [315].
The third central moment

3 = Ef(x ? mx)3 g(2.89)

38

RANDOM VECTORS AND INDEPENDENCE

is called the skewness. It is a useful measure of the asymmetricity of the pdf. It
is easy to see that the skewness is zero for probability densities that are symmetricaround their mean.
Consider now more specifically fourth-order moments. Higher than fourth ordermoments and statistics are used seldom in practice, so we shall not discuss them.
The fourth moment 4 = Efx4 g is applied in some ICA algorithms because of its
simplicity. Instead of the fourth central moment 4 = Ef(x ? mx )4 g, the fourthorder statistics called the kurtosis is usually employed, because it has some usefulproperties not shared by the fourth central moment. Kurtosis will be derived in thenext subsection in the context of the general theory of cumulants, but it is discussed
here because of its simplicity and importance in independent component analysis andblind source separation.
Kurtosis is defined in the zero-mean case by the equation
kurt(x) = Efx4 g ? 3[Efx2 g]2(2.90)

Alternatively, the normalized kurtosis4~(x) = [EEffxx2 gg]2 ? 3(2.91)

kurt(x) = ~(x) = Efx4 g ? 3(2.92)

can be used. For whitened data Efx2 g = 1, and both the versions of the kurtosis
reduce toThis implies that for white data, the fourth moment Efx4 g can be used instead of the
kurtosis for characterizing the distribution of x. Kurtosis is basically a normalizedversion of the fourth moment.
A useful property of kurtosis is its additivity. If x and y are two statisticallyindependent random variables, then it holds thatkurt(x + y ) = kurt(x) + kurt(y )(2.93)

Note that this additivity property does not hold for the fourth moment, which shows
an important benefit of using cumulants instead of moments. Also, for any scalarparameter ,kurt( x) =4

kurt(x)(2.94)

Hence kurtosis is not linear with respect to its argument.
Another very important feature of kurtosis is that it is the simplest statisticalquantity for indicating the nongaussianity of a random variable. It can be shown that
if x has a gaussian distribution, its kurtosis kurt (x) is zero. This is the sense in which
kurtosis is “normalized” when compared to the fourth moment, which is not zero forgaussian variables.
A distribution having zero kurtosis is called mesokurtic in statistical literature.
Generally, distributions having a negative kurtosis are said to be subgaussian (orHIGHER-ORDER STATISTICS

39

1/2a

−a

a

Fig. 2.8 Example of a zero-mean uniform density.

platykurtic in statistics). If the kurtosis is positive, the respective distribution iscalled supergaussian (or leptokurtic). Subgaussian probability densities tend to be
flatter than the gaussian one, or multimodal. A typical supergaussian probability
density has a sharper peak and longer tails than the gaussian pdf.
Kurtosis is often used as a quantitative measure of the nongaussianity of a randomvariable or signal, but some caution must then be taken. The reason is that the kurtosisof a supergaussian signal can have a large positive value (the maximum is infinity inprinciple), but the negative value of the kurtosis of a subgaussian signal is bounded
below so that the minimum possible value is ?2 (when variance is normalized tounity). Thus comparing the nongaussianity of supergaussian and subgaussian signals
with each other using plain kurtosis is not appropriate. However, kurtosis can beused as a simple measure of nongaussianity if the signals to be compared are of thesame type, either subgaussian or supergaussian.
In computer simulations, an often used subgaussian distribution is the uniform
distribution. Its pdf for a zero-mean random variable x is

px(x) =

(

a;

1
2

0;

x 2 [?a; a]elsewhere(2.95)

where the parameter a determines the width (and height) of the pdf; see Fig. 2.8.
A widely used supergaussian distribution is the Laplacian or doubly exponential
distribution. Its probability density (again assuming zero mean) is

px(x) = 2 exp(?jxj)(2.96)40

RANDOM VECTORS AND INDEPENDENCE

0.4

λ=0.8

0.3

0.2

0.1

λ=0.4
0

Fig. 2.9 Examples of a Laplacian density.

The only parameter  > 0 determines both the variance and the height of the peak of
the Laplacian density. It is easy to see that increasing the parameter  decreases thevariance of the Laplacian distribution and makes its peak value =2 at x = 0 higher;
see Fig. 2.9.
Both the uniform and Laplacian density can be obtained as special cases of thegeneralized gaussian or exponential power family of pdf’s [53, 256]. The general
expression of the densities belonging to this family is (for zero mean)px (x) = C exp ?  Ejfjxjxj g

(2.97)The positive real-valued power  determines the type of distribution, and C is ascaling constant which normalizes the distribution to unit area (see [53]). (Theexpectation in the denominator is a normalizing constant as well.) If the parameter = 2, the usual gaussian density is obtained. The choice  = 1 yields the Laplaciandensity, and  ! 1 the uniform density. The parameter values  < 2 in (2.97)
give rise to supergaussian densities, and  > 2 to subgaussian ones. Impulsive-type
distributions are obtained from (2.97) when 0 <  < 1.
2.7.2

Cumulants, moments, and their properties

Now we proceed to the general definition of cumulants. Assume that x is a realvalued, zero-mean, continuous scalar random variable with probability density function px (x).

HIGHER-ORDER STATISTICS

41

The first characteristic function '(! ) of x is defined as the continuous Fouriertransform of the pdf px (x):p'(!) = Efexp(|!x)g =Z1?1

exp(|!x)px (x)dx(2.98)

where | = ?1 and ! is the transformed variable corresponding to x. Every
probability distribution is uniquely specified by its characteristic function, and viceversa [353]. Expanding the characteristic function '(! ) into its Taylor series yields[353, 149]'(!) =

Z1 X
1 xk (|!)k !
?1 k=0k!

px (x)dx =

1
X

k=0

Efxk g(|!)kk!

(2.99)

Thus the coefficient terms of this expansion are moments Efxk g of x (assuming thatthey exist). For this reason, the characteristic function '(! ) is also called the moment
generating function.
It is often desirable to use the second characteristic function (! ) of x, or cumulant
generating function for reasons to be discussed later in this section. This function is
given by the natural logarithm of the first characteristic function (2.98):

(!) = ln('(!)) = ln(Efexp(|!x)g)(2.100)The cumulants k of x are defined in a similar way to the respective moments as
the coefficients of the Taylor series expansion of the second characteristic function(2.100):(!) =nkX
k (|!)k!

k=0(2.101)

where the k th cumulant is obtained as the derivativek(!)
k = (?|)k d d!
k!=0(2.102)For a zero mean random variable x, the first four cumulants are1 = 0; 2 = Efx2 g; 3 = Efx3 g; and4 = Efx4 g ? 3[Efx2 g]2(2.103)Hence the first three cumulants are equal to the respective moments, and the fourth
cumulant 4 is recognized to be the kurtosis defined earlier in (2.90).
We list below the respective expressions for the cumulants when the mean Efxg
of x is nonzero [319, 386, 149].

1 = Efxg
2 = Efx2 g ? [Efxg]2(2.104)
3 = Efx3 g ? 3Efx2 gEfxg + 2[Efxg]3
4
2 2
3
2
2
4 = Efx g ? 3[Efx g] ? 4Efx gEfxg + 12Efx g[Efxg] ? 6[Efxg]4

42

RANDOM VECTORS AND INDEPENDENCE

These formulas are obtained after tedious manipulations of the second characteristic
function (! ). Expressions for higher-order cumulants become increasingly complex[319, 386] and are omitted because they are applied seldom in practice.
Consider now briefly the multivariate case. Let be a random vector and px ( )its probability density function. The characteristic function of is again the Fouriertransform of the pdf

x'(!) = Efexp(|!x)g =

Z1

?1

x

x

exp(|!x)px (x)dxx(2.105)where ! is now a row vector having the same dimension as , and the integral iscomputed over all components of . The moments and cumulants of are obtainedin a similar manner to the scalar case. Hence, moments of are coefficients of the
Taylor series expansion of the first characteristic function '(! ), and the cumulants
are the coefficients of the expansion of the second characteristic function (! ) =ln('(! )). In the multivariate case, the cumulants are often called cross-cumulants
in analogy to cross-covariances.
It can be shown that the second, third, and fourth order cumulants for a zero mean
random vector are [319, 386, 149]

x

x

x

x

cum(xi ; xj ) =Efxi xj gcum(xi ; xj ; xk ) =Efxi xj xk gcum(xi ; xj ; xk ; xl ) =Efxi xj xk xl g ? Efxi xj gEfxk xl g
? Efxi xk gEfxj xl g ? Efxi xl gEfxj xk g

(2.106)

Hence the second cumulant is equal to the second moment Efxi xj g, which in turn isthe correlation rij or covariance cij between the variables xi and xj . Similarly, thethird cumulant cum(xi ; xj ; xk ) is equal to the third moment Efxi xj xk g. However,the fourth cumulant differs from the fourth moment E fxi xj xk xl g of the randomvariables xi ; xj ; xk ; and xl .
Generally, higher-order moments correspond to correlations used in second-order
statistics, and cumulants are the higher-order counterparts of covariances. Bothmoments and cumulants contain the same statistical information, because cumulants
can be expressed in terms of sums of products of moments. It is usually preferable towork with cumulants because they present in a clearer way the additional information
provided by higher-order statistics. In particular, it can be shown that cumulants havethe following properties not shared by moments [319, 386].

xy

1. Let and be statistically independent random vectors having the same
dimension, then the cumulant of their sum = + is equal to the sum of the
cumulants of and . This property also holds for the sum of more than two
independent random vectors.

x

y

z x y

x

2. If the distribution of the random vector or process is multivariate gaussian,
all its cumulants of order three and higher are identically zero.

STOCHASTIC PROCESSES *43

Thus higher-order cumulants measure the departure of a random vector fro m a gaussian random vector with an identical mean vector and covariance matrix. This
property is highly useful, making it possible to use cumulants for extracting thenongaussian part of a signal. For example, they make it possible to ignore additive
gaussian noise corrupting a nongaussian signal using cumulants.
Moments, cumulants, and characteristic functions have several other properties
which are not discussed here. See, for example, the books [149, 319, 386] for more
information. However, it is worth mentioning that both moments and cumulants
have symmetry properties that can be exploited to reduce the computational load inestimating them [319].
For estimating moments and cumulants, one can apply the procedure introduced in
Section 2.2.4. However, the fourth-order cumulants cannot be estimated directly, but
one must first estimate the necessary moments as is obvious from (2.106). Practicalestimation formulas can be found in [319, 315].
A drawback in utilizing higher-order statistics is that reliable estimation of higherorder moments and cumulants requires much more samples than for second-order
statistics [318]. Another drawback is that higher-order statistics can be very sensitive
to outliers in the data (see Section 8.3.1). For example, a few data samples having thehighest absolute values may largely determine the value of kurtosis. Higher-order
statistics can be taken into account in a more robust way by using the nonlinear
hyperbolic tangent function tanh(u), whose values always lie in the interval (?1; 1),or some other nonlinearity that grows slower than linearly with its argument value.

2.8 STOCHASTIC PROCESSES *2.8.1

Introduction and definition

In this section,3 we briefly discuss stochastic or random processes, defining what
they are, and introducing some basic concepts. This material is not needed in basic
independent component analysis. However, it forms a theoretical basis for blindsource separation methods utilizing time correlations and temporal information in
the data, discussed in Chapters 18 and 19. Stochastic processes are dealt with inmore detail in many books devoted either entirely or partly to the topic; see forexample [141, 157, 353, 419].
In short, stochastic or random processes are random functions of time. Stochastic
processes have two basic characteristics. First, they are functions of time, definedon some observation interval. Second, stochastic processes are random in the sensethat before making an experiment, it is not possible to describe exactly the waveformthat is observed. Due to their nature, stochastic processes are well suited to thecharacterization of many random signals encountered in practical applications, suchas voice, radar, seismic, and medical signals.

3 An asterisk after the section title means that the section is more advanced material that may be skipped.

44

RANDOM VECTORS AND INDEPENDENCE

x 1 (t)First
samplefunctionx 2 (t)Secondsamplefunctionn thsamplefunction

x n(t)
t1

t

Fig. 2.10 Sample functions of a stochastic process.

Figure 2.10 shows an example of a scalar stochastic process represented by theset of sample functions fxj (t)g, j = 1; 2; : : : ; n. Assume that the probability ofoccurrence of the ith sample function xi (t) is Pi , and similarly for the other sample
functions. Suppose then we observe the set of waveforms fxj (t)g, j = 1; 2; : : : ; n,simultaneously at some time instant t = t1 , as shown in Figure 2.10. Clearly, thevalues fxj (t1 )g, j = 1; 2; : : : ; n of the n waveforms at time t1 form a discreterandom variable with n possible values, each having the respective probability ofoccurrence Pj . Consider then another time instant t = t2 . We obtain again a randomvariable fxj (t2 )g, which may have a different distribution than fxj (t1 )g.
Usually the number of possible waveforms arising from an experiment is infinitely
large due to additive noise. At each time instant a continuous random variable havingsome distribution arises instead of the discrete one discussed above. However, thetime instants t1 ; t2 ; : : : ; on which the stochastic process is observed are discretedue to sampling. Usually the observation intervals are equispaced, and the resulting
samples are represented using integer indices xj (1) = xj (t1 ); xj (2) = xj (t2 ); : : : forSTOCHASTIC PROCESSES *45

notational simplicity. As a result, a typical representation for a stochastic process
consists of continuous random variables at discrete (integer) time instants.
2.8.2

Stationarity, mean, and autocorrelation function

Consider a stochastic process fxj (t)g defined at discrete times t1 ; t2 ; : : : ; tk . For
characterizing the process fxj (t)g completely, we should know the joint probability
density of all the random variables fxj (t1 )g, fxj (t2 )g, : : : , fxj (tk )g. The stochastic
process is said to be stationary in the strict sense if its joint density is invariant
under time shifts of origin. That is, the joint pdf of the process depends only on the
differences ti ? tj between the time instants t1 ; t2 ; : : : ; tk but not directly on them.
In practice, the joint probability density is not known, and its estimation fromsamples would be too tedious and require an excessive number of samples even if
they were available. Therefore, stochastic processes are usually characterized in termsof their first two moments, namely the mean and autocorrelation or autocovariancefunctions. They give a coarse but useful description of the distribution. Using
these statistics is sufficient for linear processing (for example filtering) of stochastic
processes, and the number of samples needed for estimating them remains reasonable.
The mean function of the stochastic process fx(t)g is definedmx (t) = Efx(t)g =Z1?1x(t)px(t) (x(t))dx(t)(2.107)

Generally, this is a function of time t. However, when the process fx(t)g is stationary,
the probability density functions of all the random variables corresponding to different
time instants become the same. This common pdf is denoted by px (x). In such acase, the mean function mx (t) reduces to a constant mean mx independent of time.
Similarly, the variance function of the stochastic process fx(t)gx2 (t) = Ef[x(t) ? mx (t)]2 g =Z1?1[x(t) ? mx(t)]2 px(t) (x(t))dx(t)(2.108)

becomes a time-invariant constant x2 for a stationary process.
Other second-order statistics of a random process fx(t)g are defined in a similarmanner. In particular, the autocovariance function of the process fx(t)g is given bycx (t;  ) = cov[x(t); x(t ?  )] = Ef[x(t) ? mx (t)][x(t ?  ) ? mx (t ?  )]g

(2.109)The expectation here is computed over the joint probability density of the randomvariables x(t) and x(t ?  ), where  is the constant time lag between the observation
times t and t ?  . For the zero lag  = 0, the autocovariance reduces to the variancefunction (2.108). For stationary processes, the autocovariance function (2.109) is
independent of the time t, but depends on the lag  : cx (t;  ) = cx ( ).
Analogously, the autocorrelation function of the process fx(t)g is defined byrx (t;  ) = Efx(t)x(t ?  )g(2.110)46

RANDOM VECTORS AND INDEPENDENCE

If fx(t)g is stationary, this again depends on the time lag  only: rx (t;  ) = rx ( ).
Generally, if the mean function mx (t) of the process is zero, the autocovariance andautocorrelation functions become the same. If the lag  = 0, the autocorrelation
function reduces to the mean-square function rx (t; 0) = Efx2 (t)g of the process,which becomes a constant rx (0) for a stationary process fx(t)g.
These concepts can be extended for two different stochastic processes fx(t)g
and fy (t)g in an obvious manner (cf. Section 2.2.3). More specifically, the crosscorrelation function rxy (t;  ) and the cross-covariance function cxy (t;  ) of theprocesses fx(t)g and fy (t)g are, respectively, defined byrxy (t;  ) = Efx(t)y(t ?  )gcxy (t;  ) = Ef[x(t) ? mx (t)][y(t ?  ) ? my (t ?  )]g

(2.111)(2.112)

Several blind source separation methods are based on the use of cross-covariancefunctions (second-order temporal statistics). These methods will be discussed in
Chapter 18.
2.8.3

Wide-sense stationary processes

A very important subclass of stochastic processes consists of wide-sense stationary(WSS) processes, which are required to satisfy the following properties:
1. The mean function mx (t) of the process is a constant mx for all t.

2. The autocorrelation function is independent of a time shift: Efx(t)x(t ?  )g= rx ( ) for all t.
3. The variance, or the mean-square value rx (0) = Efx2 (t)g of the process isfinite.

The importance of wide-sense stationary stochastic processes stems from two facts.
First, they can often adequately describe the physical situation. Many practical
stochastic processes are actually at least mildly nonstationary, meaning that theirstatistical properties vary slowly with time. However, such processes are usuallyon short time intervals roughly WSS. Second, it is relatively easy to develop usefulmathematical algorithms for WSS processes. This in turn follows from limiting theircharacterization by first- and second-order statistics.
Example 2.8 Consider the stochastic processx(t) = a cos(!t) + b sin(!t)(2.113)where a and b are scalar random variables and ! a constant parameter (angularfrequency). The mean of the process x(t) is
mx(t) = Efx(t)g = Efag cos(!t) + Efbg sin(!t)(2.114)

STOCHASTIC PROCESSES *47

and its autocorrelation function can be writtenrx (t;  ) = Efx(t)x(t ?  )g= 12 Efa2 g[cos(!(2t ?  )) + cos(?! )]+ 21 Efb2g[? cos(!(2t ?  )) + cos(?! )]+ Efabg[sin(!(2t ?  )]

(2.115)where we have used well-known trigonometric identities. Clearly, the process x(t)is generally nonstationary, since both its mean and autocorrelation functions dependon the time t.
However, if the random variables a and b are zero mean and uncorrelated withequal variances, so that
Efag = Efbg = Efabg = 0

Efa2 g = Efb2 g

the mean (2.114) of the process becomes zero, and its autocorrelation function (2.115)
simplifies torx ( ) = Efa2 g cos(! )which depends only on the time lag  . Hence, the process is WSS in this special case(assuming that Efa2 g is finite).

Assume now that fx(t)g is a zero-mean WSS process. If necessary, the process
can easily be made zero mean by first subtracting its mean mx . It is sufficient toconsider the autocorrelation function rx ( ) of fx(t)g only, since the autocovariancefunction cx ( ) coincides with it. The autocorrelation function has certain properties
that are worth noting. First, it is an even function of the time lag  :rx (? ) = rx ( )(2.116)Another property is that the autocorrelation function achieves its maximum absolutevalue for zero lag:?rx (0)  rx ( )  rx (0)(2.117)

The autocorrelation function rx ( ) measures the correlation of random variables
x(t) and x(t ?  ) that are  units apart in time, and thus provides a simple measurefor the dependence of these variables which is independent of the time t due to the
WSS property. Roughly speaking, the faster the stochastic process fluctuates withtime around its mean, the more rapidly the values of the autocorrelation functionrx ( ) decrease from their maximum rx (0) as  increases.
Using the integer notation for the samples x(i) of the stochastic process, we canrepresent the last m + 1 samples of the stochastic process at time n using the random
vectorx(n) = [x(n); x(n ? 1); : : : ; x(n ? m)]T

(2.118)

48

RANDOM VECTORS AND INDEPENDENCE

Assuming that the values of the autocorrelation function rx (0); rx (1); : : : ; rx (m) are
known up to a lag of m samples, the (m + 1)  (m + 1) correlation (or covariance)
matrix of the process fx(n)g is defined by

2 r (0)66 rxx(1)Rx = 64 ..
.

rx (1)
rx (0)..
.

rx (2)
rx (1)..
.

   rx (m) 3   rx (m ? 1) 77..

.

rx (m) rx (m ? 1) rx (m ? 2)   ..
.

rx (0)

75(2.119)R

The matrix x satisfies all the properties of correlation matrices listed in Section
2.2.2. Furthermore, it is a Toeplitz matrix. This is generally defined so that on eachsubdiagonal and on the diagonal, all the elements of Toeplitz matrix are the same.
The Toeplitz property is helpful, for example, in solving linear equations, enablinguse of faster algorithms than for more general matrices.
Higher-order statistics of a stationary stochastic process x(n) can be defined in an
analogous manner. In particular, the cumulants of x(n) have the form [315]cumxx (j ) = Efx(i)x(i + j )gcumxxx (j; k ) = Efx(i)x(i + j )x(i + k )g
(2.120)cumxxx (j; k; l) = Efx(i)x(i + j )x(i + k )x(i + l)g? Efx(i)x(j )gEfx(k)x(l)g ? Efx(i)x(k)gEfx(j )x(l)g? Efx(i)x(l)gEfx(j )x(k)g
These definitions correspond to the formulas (2.106) given earlier for a general
random vector . Again, the second and third cumulant are the same as the respective
moments, but the fourth cumulant differs from the fourth moment E fx(i)x(i+j )x(i+k)x(i + l)g. The second cumulant cumxx(j ) is equal to the autocorrelation rx (j )and autocovariance cx (j ).

x

2.8.4

Time averages and ergodicity

In defining the concept of a stochastic process, we noted that at each fixed time
instant t = t0 the possible values x(t0 ) of the process constitute a random variable
having some probability distribution. An important practical problem is that these
distributions (which are different at different times if the process is nonstationary)are not known, at least not exactly. In fact, often all that we have is just one sample ofthe process corresponding to each discrete time index (since time cannot be stopped
to acquire more samples). Such a sample sequence is called a realization of thestochastic process. In handling WSS processes, we need to know in most cases only
the mean and autocorrelation values of the process, but even they are often unknown.
A practical way to circumvent this difficulty is to replace the usual expectations of
the random variables, called ensemble averages, by long-term sample averages or time
averages computed from the available single realization. Assume that this realization
contains K samples x(1); x(2); : : : ; x(K ). Applying the preceding principle, theSTOCHASTIC PROCESSES *49

mean of the process can be estimated using its time averagem^ x (K ) = K1

K
X
k=1

x(k)

(2.121)and the autocorrelation function for the lag value l usingr^x (l; K ) = K 1? l

K
X?l
k=1

x(k + l)x(k)(2.122)

The accuracy of these estimates depends on the number K of samples. Note alsothat the latter estimate is computed over the K ? l possible sample pairs having thelag l that can be found from the sample set. The estimates (2.122) are unbiased, but
if the number of pairs K ? l available for estimation is small, their variance can behigh. Therefore, the scaling factor K ? l of the sum in (2.122) is often replaced by
K in order to reduce the variance of the estimated autocorrelation values r^x (l; K ),even though the estimates then become biased [169]. As K ! 1, both estimates
tend toward the same value.
The stochastic process is called ergodic if the ensemble averages can be equatedto the respective time averages. Roughly speaking, a random process is ergodic withrespect to its mean and autocorrelation function if it is stationary. A more rigorous
treatment of the topic can be found for example in [169, 353, 141].
For mildly nonstationary processes, one can apply the estimation formulas (2.121)
and (2.122) by computing the time averages over a shorter time interval during which
the process can be regarded to be roughly WSS. It is important to keep this inmind. Sometimes formula (2.122) is applied in estimating the autocorrelation values
without taking into account the stationarity of the process. The consequences canbe drastic, for example, rendering eigenvectors of the correlation matrix (2.119)
useless for practical purposes if ergodicity of the process is in reality a grossly invalid
assumption.
2.8.5

Power spectrum

A lot of insight into a WSS stochastic process is often gained by representing it in
the frequency domain. The power spectrum or spectral density of the process x(n)provides such a representation. It is defined as the discrete Fourier transform of the
autocorrelation sequence rx (0); rx (1); : : : :Sx(!) =

p

1
X

k=?1

rx (k) exp(?|k!)(2.123)where | = ?1 is the imaginary unit and ! the angular frequency. The time
domain representation given by the autocorrelation sequence of the process canbe obtained from the power spectrum Sx (! ) by applying the inverse discrete-time

50

RANDOM VECTORS AND INDEPENDENCE

Fourier transformrx (k) = 21

Z?Sx (!) exp(|k!)d!;k = 1; 2; : : :(2.124)

It is easy to see that the power spectrum (2.123) is always real-valued, even, and a
periodic function of the angular frequency ! . Note also that the power spectrum is acontinuous function of ! , while the autocorrelation sequence is discrete. In practice,
the power spectrum must be estimated from a finite number of autocorrelation values.
If the autocorrelation values rx (k ) ! 0 sufficiently quickly as the lag k grows large,
this provides an adequate approximation.
The power spectrum describes the frequency contents of the stochastic process,showing which frequencies are present in the process and how much power they
possess. For a sinusoidal signal, the power spectrum shows a sharp peak at its
oscillating frequency. Various methods for estimating power spectra are discussed
thoroughly in the books [294, 241, 411].
Higher-order spectra can be defined in a similar manner to the power spectrum
as Fourier transforms of higher-order statistics [319, 318]. Contrary to the power
spectra, they retain information about the phase of signals, and have found many
applications in describing nongaussian, nonlinear, and nonminimum-phase signals[318, 319, 315].
2.8.6

Stochastic signal models

A stochastic process whose power spectrum is constant for all frequencies ! is called
white noise. Alternatively, white noise v (n) can be defined as a process for which
any two different samples are uncorrelated:(2
rv (k) = Efv(n)v(n ? k)g = v ; k = 00; k = 1; 2; : : :(2.125)Here v2 is the variance of the white noise. It is easy to see that the power spectrumof the white noise is Sv (! ) = v2 for all ! , and that the formula (2.125) follows
from the inverse transform (2.124). The distribution of the random variable v (n)forming the white noise can be any reasonable one, provided that the samples areuncorrelated at different time indices. Usually this distribution is assumed to begaussian. The reason is that white gaussian noise is maximally random because any
two uncorrelated samples are also independent. Furthermore, such a noise process
cannot be modeled to yield an even simpler random process.
Stochastic processes or time series are frequently modeled in terms of autoregressive (AR) processes. They are defined by the difference equationx(n) = ?

MXi=1

ai x(n ? i) + v(n)(2.126)

CONCLUDING REMARKS AND REFERENCES

51

where v (n) is a white noise process, and a1 ; : : : ; aM are constant coefficients (parameters) of the AR model. The model order M gives the number of previous samples
on which the current value x(n) of the AR process depends. The noise term v (n)introduces randomness into the model; without it the AR model would be completely
deterministic. The coefficients a1 ; : : : ; aM of the AR model can be computed using linear techniques from autocorrelation values estimated from the available data
[419, 241, 169]. Since the AR models describe fairly well many natural stochastic
processes, for example, speech signals, they are used in many applications. In ICA
and BSS, they can be used to model the time correlations in each source processsi (t). This sometimes improves greatly the performance of the algorithms.
Autoregressive processes are a special case of autoregressive moving average(ARMA) processes described by the difference equationx(n) +

MX
i=1

ai x(n ? i) = v(n) +

N
X
i=1

bi v(n ? i)

(2.127)

Clearly, the AR model (2.126) is obtained from the ARMA model (2.127) whenthe moving average (MA) coefficients b1 ; : : : ; bN are all zero. On the other hand,
if the AR coefficients ai are all zero, the ARMA process (2.127) reduces to a MA
process of order N . The ARMA and MA models can also be used to describe
stochastic processes. However, they are applied less frequently, because estimationof their parameters requires nonlinear techniques [241, 419, 411]. See the Appendix
of Chapter 19 for a discussion of the stability of the ARMA model and its utilizationin digital filtering.

2.9 CONCLUDING REMARKS AND REFERENCES
In this chapter, we have covered the necessary background on the theory of random
vectors, independence, higher-order statistics, and stochastic processes. Topics thatare needed in studying independent component analysis and blind source separation
have received more attention. Several books that deal more thoroughly with thetheory of random vectors exist; for example, [293, 308, 353]. Stochastic processes
are discussed in [141, 157, 353], and higher-order statistics in [386].
Many useful, well-established techniques of signal processing, statistics, and otherareas are based on analyzing random vectors and signals by means of their first- andsecond-order statistics. These techniques have the virtue that they are usually fairly
easy to apply. Typically, second-order error criteria (for example, the mean-squareerror) are used in context with them. In many cases, this leads to linear solutions thatare simple to compute using standard numerical techniques. On the other hand, one
can claim that techniques based on second-order statistics are optimal for gaussian
signals only. This is because they neglect the extra information contained in thehigher-order statistics, which is needed in describing nongaussian data. Independent
component analysis uses this higher-order statistical information, and is the reason
for which it is such a powerful tool.

52

RANDOM VECTORS AND INDEPENDENCE

Problems
2.1 Derive a rule for computing the values of the cdf of the single variable gaussian(2.4) from the known tabulated values of the error function (2.5).
2.2 Let x1 ; x2 ; : : : ; xK be independent, identically distributed samples from a
distribution having a cumulative density function Fx (x). Denote by y1 ; x2 ; : : : ; yK
the sample set x1 ; x2 ; : : : ; xK ordered in increasing order.
2.2.1. Show that the cdf and pdf of yK = maxfx1 ; : : : ; xK g are

FyK (yK ) = [Fx (yK )]K
pyK (yK ) = K [Fx (yK )]K ?1 px (yK )2.2.2. Derive the respective expressions for the cdf and pdf of the random variable

y1 = minfx1 ; : : : ; xK g.

2.3 A two-dimensional random vector
functionpx (x) =

(

1
2

0

x = (x ; x )T has the probability density
12(x1 + 3x2 ) x1 ; x2 2 [0; 1]
elsewhere

2.3.1. Show that this probability density is appropriately normalized.
2.3.2. Compute the cdf of the random vector .
2.3.3. Compute the marginal distributions px1 (x1 ) and px2 (x2 ).

x

2.4 Computer the mean, second moment, and variance of a random variable distributed uniformly in the interval [a; b] (b > a).
2.5

Prove that expectations satisfy the linearity property (2.16).

2.6 Consider n scalar random variables xi , i = 1; 2; : : : ; n, having, respectively,
the variances x2i . Show that if the random variables xi are mutually uncorrelated,nthe variance y2 of their sum y = i=1 xi equals the sum of the variances of the xi :

P

y2 =

n
X
i=1

x2i

2.7 Assume that x1 and x2 are zero-mean, correlated random variables. Any
orthogonal transformation of x1 and x2 can be represented in the form

y1 = cos( )x1 + sin( )x2y2 = ? sin( )x1 + cos( )x2
where the parameter defines the rotation angle of coordinate axes. Let Efx21 g = 12 ,Efx22 g = 22 , and Efx1 x2 g = 1 2 . Find the angle for which y1 and y2 become
uncorrelated.

53

PROBLEMS

2.8 Consider the joint probability density of the random vectors
and = y discussed in Example 2.6:yx = (x ; x )T1

2(

px;y (x; y) = (x1 + 3x2 )y x1 ; x2 2 [0; 1]; y 2 [0; 1]0elsewherexy

2.8.1. Compute the marginal distributions px ( ), py ( ), px1 (x1 ), and px2 (x2 ).
2.8.2. Verify that the claims made on the independence of x1 , x2 , and y in
Example 2.6 hold.
2.9
R = ac db

Which conditions should the elements of the matrix

Rsatisfy so that could be a valid autocorrelation matrix of
2.9.1. A two-dimensional random vector?
2.9.2. A stationary scalar-valued stochastic process?
2.10 Show that correlation and covariance matrices satisfy the relationships (2.26)
and (2.32).

Cx

2.11 Work out Example 2.5 for the covariance matrix x of , showing that similar
results are obtained. Are the assumptions required the same?
2.12 Assume that the inverse ?
x 1 of the correlation matrix of the n-dimensional
column random vector exists. Show thatxR

x Rx? xg = n

Ef T

1

2.13 Consider a two-dimensional gaussian random vector= (2; 1)T and covariance matrix

x with mean vector mx

 2 ?1 
Cx =
?1 2

C

2.13.1. Find the eigenvalues and eigenvectors of x .
2.13.2. Draw a contour plot of the gaussian density similar to Figure 2.7.
2.14 Repeat the previous problem for a gaussian random vector
vector x = (?2; 3)T and covariance matrixmCx = ?22 ?52

x that has the mean

2.15 Assume that random variables x and y are linear combinations of two uncorrelated gaussian random variables u and v , defined by

x = 3u ? 4v
y = 2u + v

54

RANDOM VECTORS AND INDEPENDENCE

Assume that the mean values and variances of both u and v equal 1.
2.15.1. Determine the mean values of x and y .
2.15.2. Find the variances of x and y .
2.15.3. Form the joint density function of x and y .
2.15.4. Find the conditional density of y given x.
2.16

Show that the skewness of a random variable having a symmetric pdf is zero.

2.17

Show that the kurtosis of a gaussian random variable is zero.

2.18 Show that random variables having
2.18.1. A uniform distribution in the interval [?a; a]; a > 0, are subgaussian.
2.18.2. A Laplacian distribution are supergaussian.
2.19

The exponential density has the pdf

px(x) =(

exp(? x) x  00
x<0

where is a positive constant.
2.19.1. Compute the first characteristic function of the exponential distribution.
2.19.2. Using the characteristic function, determine the moments of the exponential density.
2.20

A scalar random variable x has a gamma distribution if its pdf is given by

px(x) =( b?

x 1 exp(?cx) x  00
x<0

where b and c are positive numbers and the parameterb= ?(c b)is defined by the gamma function?(b + 1) =Z10

yb exp(?y)dy; b > ?1

The gamma function satisfies the generalized factorial condition ?(b + 1) = b?(b).
For integer values, this becomes ?(n + 1) = n!.
2.20.1. Show that if b = 1, the gamma distribution reduces to the standard
exponential density.
2.20.2. Show that the first characteristic function of a gamma distributed randomvariable isb'(!) = (c ?c |!)b

PROBLEMS

55

2.20.3. Using the previous result, determine the mean, second moment, andvariance of the gamma distribution.
2.21 Let k (x) and k (y ) be the k th-order cumulants of the scalar random variables
x and y, respectively.
2.21.1. Show that if x and y are independent, thenk (x + y) = k (x) + k (y)2.21.2. Show that k ( x) = k k (x), whereis a constant.

2.22 * Show that the power spectrum Sx (! ) is a real-valued, even, and periodic
function of the angular frequency ! .
2.23* Consider the stochastic processy(n) = x(n + k) ? x(n ? k)where k is a constant integer and x(n) is a zero mean, wide-sense stationary stochastic
process. Let the power spectrum of x(n) be Sx (! ) and its autocorrelation sequencerx (0); rx (1); : : : .
2.23.1. Determine the autocorrelation sequence ry (m) of the process y (n).
2.23.2. Show that the power spectrum of y (n) is
Sy (!) = 4Sx (!) sin2 (k!)
2.24 * Consider the autoregressive process (2.126).
2.24.1. Show that the autocorrelation function of the AR process satisfies thedifference equationrx (l) = ?

M
X
i=1

ai rx (l ? i); l > 0

2.24.2. Using this result, show that the AR coefficients ai can be determined fromthe Yule-Walker equations

Rxa = ?rx

Here the autocorrelation matrix
vectorRx defined in (2.119) has the value m = M ? 1, therx = [rx(1); rx(2); : : : ; rx(M )]T

and the coefficient vectora = [a ; a ; : : : ; aM ]T
1

2

2.24.3. Show that the variance of the white noise process v (n) in (2.126) is related
to the autocorrelation values by the formulav2 = rx (0) +

M
X
i=1

ai rx (i)56

RANDOM VECTORS AND INDEPENDENCE

Computer assignments
2.1 Generate samples of a two-dimensional gaussian random vector
mean vector and the covariance matrix

x having zero

 4 ?1 Cx =?1 2

Estimate the covariance matrix and compare it with the theoretical one for the following numbers of samples, plotting the sample vectors in each case.
2.1.1. K = 20.
2.1.2. K = 200.
2.1.3. K = 2000.
2.2 Consider generation of desired Laplacian random variables for simulation purposes.
2.2.1. Using the probability integral transformation, give a formula for generating
samples of a scalar random variable with a desired Laplacian distribution from
uniformly distributed samples.
2.2.2. Extend the preceding procedure so that you get samples of two Laplacian
random variables with a desired mean vector and joint covariance matrix. (Hint: Use
the eigenvector decomposition of the covariance matrix for generating the desired
covariance matrix.)
2.2.3. Use your procedure for generating 200 samples of a two-dimensional
Laplacian random variable with a mean vector x = (2; ?1)T and covariancematrixxCx = ?41 ?21m

Plot the generated samples.
2.3 * Consider the second-order autoregressive model described by the difference
equationx(n) + a1 x(n ? 1) + a2 x(n ? 2) = v(n)Here x(n) is the value of the process at time n, and v (n) is zero mean white gaussian
noise with variance v2 that “drives” the AR process. Generate 200 samples of theprocess using the initial values x(0) = x(?1) = 0 and the following coefficient values.

Plot the resulting AR process in each case.
2.3.1. a1 = ?0:1 and a2 = ?0:8.
2.3.2. a1 = 0:1 and a2 = ?0:8.
2.3.3. a1 = ?0:975 and a2 = 0:95.
2.3.4. a1 = 0:1 and a2 = ?1:0.

3
Gradients and Optimization
Methods
The main task in the independent component analysis (ICA) problem, formulated inthat will give us the independent
Chapter 1, is to estimate a separating matrix
cannot generally be solved in closed form,
components. It also became clear thatthat is, we cannot write it as some function of the sample or training set, whose value
could be directly evaluated. Instead, the solution method is based on cost functions,also called objective functions or contrast functions. Solutions
to ICA are foundat the minima or maxima of these functions. Several possible ICA cost functions will
be given and discussed in detail in Parts II and III of this book. In general, statisticalestimation is largely based on optimization of cost or objective functions, as will beseen in Chapter 4.
Minimization of multivariate functions, possibly under some constraints on thesolutions, is the subject of optimization theory. In this chapter, we discuss sometypical iterative optimization algorithms and their properties. Mostly, the algorithms
are based on the gradients of the cost functions. Therefore, vector and matrix
gradients are reviewed first, followed by the most typical ways to solve unconstrained
and constrained optimization problems with gradient-type learning algorithms.

W

W

W

3.1 VECTOR AND MATRIX GRADIENTS
3.1.1

Vector gradientConsider a scalar valued function g of m variablesg = g(w1 ; :::; wm ) = g(w)57

58

GRADIENTS AND OPTIMIZATION METHODSw

w

where we have used the notation = (w1 ; :::; wm )T . By convention, we define
as a column vector. Assuming the function g is differentiable, its vector gradient withrespect to is the m-dimensional column vector of partial derivatives

w

0 @g 1@g = B @w.. C@w @ . A
1

(3.1)

@g
@wm

The notation @@g
w is just shorthand for the gradient; it should be understood that it
does not imply any kind of division by a vector, which is not a well-defined concept.
Another commonly used notation would be rg or rw g .
In some iteration methods, we have also reason to use second-order gradients. We
define the second-order gradient of a function g with respect to as

w

0 @g
@w
@ g =B
B@ ...
2

2
1

2

@ w2

@g
@wm w1
2

1
CA::: @w@1 wg m
C
2

..
.

@g
@wm2

:::

2

(3.2)

This is an m  m matrix whose elements are second order partial derivatives. It iscalled the Hessian matrix of the function g ( ). It is easy to see that it is always
symmetric.
These concepts generalize to vector-valued functions; this means an n-element
vectorw

0 g (w) 1
g(w) = B@ ... CA1

(3.3)

gn (w)w) are themselves functions of w. The Jacobian matrix of g withwhose elements gi (respect to isw

0 @g :::@ g = B @w..
@w @ .
1
1

@g1
@wm

:::

@gn 1
@w1
..
.

@gn
@wm

CA

(3.4)

w

Thus the ith column of the Jacobian matrix is the gradient vector of gi ( ) withrespect to . The Jacobian matrix is sometimes denoted by J .
For computing the gradients of products and quotients of functions, as well as ofcomposite functions, the same rules apply as for ordinary functions of one variable.

wgVECTOR AND MATRIX GRADIENTS

59

Thus

@f (w)g(w) = @f (w) g(w) + f (w) @g(w)
(3.5)
@w
@w
@w
@f (w)=g(w) = [ @f (w) g(w) ? f (w) @g(w) ]=g2 (w)
(3.6)
@w
@w
@w
@f (g(w)) = f 0 (g(w)) @g(w)
(3.7)
@w
@w
The gradient of the composite function f (g (w)) can be generalized to any numberof nested functions, giving the same chain rule of differentiation that is valid forfunctions of one variable.
3.1.2

Matrix gradient

In many of the algorithms encountered in this book, we have to consider scalar-valued
functions g of the elements of an m  n matrix= (wij ):Wg = g(W) = g(w11 ; :::; wij ; :::; wmn )

(3.8)

W

A typical function of this kind is the determinant of .
Of course, any matrix can be trivially represented as a vector by scanning theelements row by row into a vector and reindexing. Thus, when considering thegradient of g with respect to the matrix elements, it would suffice to use the notionof vector gradient reviewed earlier. However, using the separate concept of matrix
gradient gives some advantages in terms of a simplified notation and sometimes
intuitively appealing results.
In analogy with the vector gradient, the matrix gradient means a matrix of thesame size m  n as matrix , whose ij th element is the partial derivative of g withrespect to wij . Formally we can writeW

0 @g :::@g = B @w..
@W @ .
11

@g
@wm1

:::

@g 1
@w1n
..
.

@g
@wmn

CA

(3.9)

Again, the notation @@g
W is just shorthand for the matrix gradient.
Let us look next at some examples on vector and matrix gradients. The formulas
presented in these examples will be frequently needed later in this book.
3.1.3

Examples of gradients

Example 3.1 Consider the simple linear functional ofg(w) =mX
i=1

w, or inner productai wi = a T w

60

GRADIENTS AND OPTIMIZATION METHODS

where

a = (a :::am)T is a constant vector. The gradient is, according to (3.1),
1

0a 1@g B .. C@w = @ . A
1(3.10)

amawhich is the vector . We can write@ aT w = a@w

Because the gradient is constant (independent ofaT w is zero.

w), the Hessian matrix of g(w) =Example 3.2 Next consider the quadratic formg(w) = wT Aw =
where

m X
m
X
i=1 j =1

wi wj aij

(3.11)A = (aij ) is a square m  m matrix. We have

0 Pm wj a j + Pm wi ai 1
i@g = B j
CA..
. P@w @ P1=1=1m wa +
j =1 j mj
which is equal to the vector Aw + AT w. So,

1@ wT Aw = Aw + AT w
@w

A

Aw

For symmetric , this becomes 2.
The second-order gradient or Hessian becomes@ wT Aw2@ w2

which is equal to the matrix
is equal to 2 .

A(3.12)m wa
i=1 i im

0
=B
@

2a11
..
.

::: a1m + am1

am1 + a1m :::

..
.

2amm

1
CA

(3.13)A + AT . If A is symmetric, then the Hessian of wT Aw

Example 3.3 For the quadratic form (3.11), we might quite as well take the gradient
T Aw= w i wj .
with respect to , assuming now that is a constant vector. Then @ w@aij
Compiling this into matrix form, we notice that the matrix gradient is the m  mT.
matrixAw

ww

Example 3.4 In some ICA models, we must compute the matrix gradient of thedeterminant of a matrix. The determinant is a scalar function of the matrix elements

VECTOR AND MATRIX GRADIENTS

61

consisting of multiplications and summations, and therefore its partial derivatives areis an invertible square
relatively simple to compute. Let us prove the following: Ifm  m matrix whose determinant is denoted det , thenWW

@ det W = (WT )?1 det W:@W(3.14)

This is a good example for showing that a compact formula is obtained using thematrix gradient; ifwere stacked into a long vector, and only the vector gradientwere used, this result could not be expressed so simply.
Instead of starting from scratch, we employ a well-known result from matrix
is obtained as
algebra (see, e.g., [159]), stating that the inverse of a matrix

W

W

W? = det1W adj(W)with adj(W) the so-called adjoint of W. The adjoint is the matrix
10W
adj(W) = @11::: Wn1

W1n ::: Wnn

1
A(3.15)(3.16)

where the scalar numbers Wij are the so-called cofactors. The cofactor Wij isobtained by first taking the (n ? 1)  (n ? 1) submatrix ofthat remains when
the ith row and j th column are removed, then computing the determinant of thissubmatrix, and finally multiplying by (?1)i+j .
The determinant det
can also be expressed in terms of the cofactors:W

W

det W =nX
k=1

wik Wik(3.17)

Row i can be any row, and the result is always the same. In the cofactors Wik , noneof the matrix elements of the ith row appear, so the determinant is a linear functionof these elements. Taking now a partial derivative of (3.17) with respect to one of theelements, say, wij , gives@ det W = W
ij@wij

By definitions (3.9) and (3.16), this implies directly thatW

But adj( )T is equal to (det
(3.14).
This also implies that@ det W = adj(W)T@W

W)(WT )? by (3.15), so we have shown our result
1@ log j det Wj = 1 @ j det Wj = (WT )?1@W
j det Wj @ W(3.18)see (3.15). This is an example of the matrix gradient of a composite function
consisting of the log, absolute value, and det functions. This result will be needed
when the ICA problem is solved by maximum likelihood estimation in Chapter 9.

62

GRADIENTS AND OPTIMIZATION METHODS

3.1.4

Taylor series expansions of multivariate functions

In deriving some of the gradient type learning algorithms, we have to resort to Taylor
series expansions of multivariate functions. In analogy with the well-known Taylor
series expansion of a function g (w) of a scalar variable w,

dg (w0 ? w) + 1=2 d2 g (w0 ? w)2 + :::g(w0 ) = g(w) + dw
dw2

(3.19)w) = g(w ; :::; wm ) of m variables.

we can do a similar expansion for a function g (We have1

2
g(w0 ) = g(w) + ( @@gw )T (w0 ? w) + 1=2(w0 ? w)T @@wg2 (w0 ? w) + :::(3.20)

w
w w

where the derivatives are evaluated at the point . The second term is the innerproduct of the gradient vector with the vector 0 ? , and the third term is a quadratic2
form with the symmetric Hessian matrix @@wg2 . The truncation error depends on the
distance k 0 ? k; the distance has to be small, if g ( 0 ) is approximated using only
the first- and second-order terms.
The same expansion can be made for a scalar function of a matrix variable. The
second order term already becomes complicated because the second order gradient isa four-dimensional tensor. But we can easily extend the first order term in (3.20), theinner product of the gradient with the vector 0 ? , to the matrix case. Rememberthat the vector inner product is defined as

w w

w

w w

( @@gw )T (w0 ? w) =

m @g
X
i=1

( @ w )i (wi0 ? wi )

Pm Pm

0
For the matrix case, this must become the sum i=1 j =1 ( @@gW )ij (wij ? wij ): Thisis the sum of the products of corresponding elements, just like in the vectorial innerproduct. This can be nicely presented in matrix form when we remember that for any
two matrices, say, and ,ABA B) =

trace( T

m
X

m X
m
X

i=1

i=1 j =1

(AT B)ii =(A)ij (B)ij

with obvious notation. So, we have

T 0
g(W0 ) = g(W) + trace[( @@gW ) (W ? W)] + :::for the first two terms in the Taylor series of a function g of a matrix variable.

(3.21)

LEARNING RULES FOR UNCONSTRAINED OPTIMIZATION

63

3.2 LEARNING RULES FOR UNCONSTRAINED OPTIMIZATION
3.2.1

Gradient descent

W

Many of the ICA criteria have the basic form of minimizing a cost function J ( )with respect to a parameter matrix , or possibly with respect to one of its columns. In many cases, there are also constraints that restrict the set of possible solutions.
A typical constraint is to require that the solution vector must have a bounded norm,or the solution matrix has orthonormal columns.
For the unconstrained problem of minimizing a multivariate function, the mostclassic approach is steepest descent or gradient descent. Let us consider in moredetail the case when the solution is a vector ; the matrix case goes through in acompletely analogous fashion.
In gradient descent, we minimize a function J ( ) iteratively by starting fromsome initial point (0), computing the gradient of J ( ) at this point, and thenmoving in the direction of the negative gradient or the steepest descent by a suitabledistance. Once there, we repeat the same procedure at the new point, and so on. For
t = 1; 2; :::;we have the update rule

W

w

w

w

w

w

w(t) = w(t ? 1) ? (t) @ J@ w(w) jw w t?
(3.22)
with the gradient taken at the point w(t ? 1). The parameter (t) gives the length of
=

(

1)

the step in the negative gradient direction. It is often called the step size or learning
rate. Iteration (3.22) is continued until it converges, which in practice happens when
the Euclidean distance between two consequent solutions k (t) ? (t ? 1)k goes
below some small tolerance level.
If there is no reason to emphasize the time or iteration step, a convenient shorthandnotation will be used throughout this book in presenting update rules of the preceding
type. Denote the difference between the new and old value byw

w(t) ? w(t ? 1) = w

w

(3.23)

We can then write the rule (3.22) either as(w)
w = ? @ J@ w

or even shorter as(w)
w / ? @ J@ w

The symbol / is read “is proportional to”; it is then understood that the vector on theleft-hand side,  , has the same direction as the gradient vector on the right-handside, but there is a positive scalar coefficient by which the length can be adjusted. In
the upper version of the update rule, this coefficient is denoted by . In many cases,
this learning rate can and should in fact be time dependent. Yet a third very convenient
way to write such update rules, in conformity with programming languages, isw

w w ? @ J@ w(w)64

GRADIENTS AND OPTIMIZATION METHODS

means substitution, i.e., the value of the right-hand side iswhere the symbol
computed and substituted in .
Geometrically, a gradient descent step as in (3.22) means going downhill. The
graph of J ( ) is the multidimensional equivalent of mountain terrain, and we arealways moving downwards in the steepest direction. This also immediately shows
the disadvantage of steepest descent: unless the function J ( ) is very simple andsmooth, steepest descent will lead to the closest local minimum instead of a global
minimum. As such, the method offers no way to escape from a local minimum.
Nonquadratic cost functions may have many local maxima and minima. Therefore,
good initial values are important in initializing the algorithm.

w

w

w

Local minimum
Gradient vector

Globalminimum

Fig. 3.1 Contour plot of a cost function with a local minimum.

w

As an example, consider the case of Fig. 3.1. A function J ( )is shown there asa contour plot. In the region shown in the figure, there is one local minimum and one
global minimum. From the initial point chosen there, where the gradient vector has
been plotted, it is very likely that the algorithm will converge to the local minimum.
Generally, the speed of convergence can be quite low close to the minimum point,
because the gradient approaches zero there. The speed can be analyzed as follows.
Let us denote by  the local or global minimum point where the algorithm will
eventually converge. From (3.22) we havew

w(t) ? w = w(t ? 1) ? w ? (t) @ J@ w(w) jw w t?
=

(

1)

(3.24)(w)
Let us expand the gradient vector @ J@ w
element by element as a Taylor series aroundthe point, as explained in Section 3.1.4. Using only the zeroth- and first-orderterms, we have for the ith elementw

m @ 2 J (w)@ J (w) j@ J (w) j  + X
jw=w [wj (t ? 1) ? wj ] + :::w=w
w=w(t?1) =
@wi
@wi
@w
w
i
j
j =1

LEARNING RULES FOR UNCONSTRAINED OPTIMIZATION

65

Now, because  is the point of convergence, the partial derivatives of the costfunction must be zero at  . Using this result, and compiling the above expansion
into vector form, yieldsww@ J (w) j@ w w=w(t?1) = H(w )[w(t ? 1) ? w ] + :::where (  )is the Hessian matrix computed at the pointin (3.24) givesHw

w = w. Substituting thisw(t) ? w  [I ? (t)H(w )][w(t ? 1) ? w]This kind of convergence, which is essentially equivalent to multiplying a matrix
many times with itself, is called linear. The speed of convergence depends on thelearning rate and the size of the Hessian matrix. If the cost function J ( ) is very
flat at the minimum, with second partial derivatives also small, then the Hessian issmall and the convergence is slow (for fixed (t)). Usually, we cannot influence theshape of the cost function, and we have to choose (t), given a fixed cost function.
The choice of an appropriate step length or learning rate (t) is thus essential: toosmall a value will lead to slow convergence. The value cannot be too large either: toolarge a value will lead to overshooting and instability, which prevents convergencealtogether. In Fig. 3.1, too large a learning rate will cause the solution point to zigzag
around the local minimum. The problem is that we do not know the Hessian matrix
and therefore determining a good value for the learning rate is difficult.
A simple extension to the basic gradient descent, popular in neural network learning rules like the back-propagation algorithm, is to use a two-step iteration insteadof just one step like in (3.22), leading to the so-called momentum method. Neuralnetwork literature has produced a large number of tricks for boosting steepest descentlearning by adjustable learning rates, clever choice of the initial value, etc. However,in ICA, many of the most popular algorithms are still straightforward gradient descentmethods, in which the gradient of an appropriate contrast function is computed andused as such in the algorithm.

w

3.2.2

Second-order learning

In numerical analysis, a large number of methods that are more efficient than plain
gradient descent have been introduced for minimizing or maximizing a multivariate
scalar function. They could be immediately used for the ICA problem. Their advantage is faster convergence in terms of the number of iterations required, but thedisadvantage quite often is increased computational complexity per iteration. Herewe consider second-order methods, which means that we also use the information
contained in the second-order derivatives of the cost function. Obviously, this information relates to the curvature of the optimization terrain and should help in finding
a better direction for the next step in the iteration than just plain gradient descent.

66

GRADIENTS AND OPTIMIZATION METHODS

A good starting point is the multivariate Taylor series; see Section 3.1.4. Let us
develop the function J ( ) in Taylor series around a point asww
J (w0 ) = J (w) + [ @ J@ w(w) ]T (w0 ? w) + 21 (w0 ? w)T @ @Jw(w) (w0 ? w) + :::(3.25)
2

2

In trying to minimize the function J ( ), we ask what choice of the new point 0
gives us the largest decrease in the value of J ( ). We can write 0 ? =  and2(w) Tminimize the function J ( 0 ) ?J ( ) = [ @ J@ w
]  +1=2 T @ @Jw(2w)  withrespect to  . The gradient of this function with respect to  is (see Example2(w)3.2) equal to @ J@ w
+ @ @Jw(2w)  ; note that the Hessian matrix is symmetric. If the
Hessian is also positive definite, then the function will have a parabolic shape and theminimum is given by the zero of the gradient. Setting the gradient to zero gives

w

w

w

w

w

w w
w
w

w

w

w
w

w

2
(w)
w = ?[ @ @Jw(2w) ]?1 @ J@ w

From this, the following second-order iteration rule emerges:

w0 = w ? [ @ @Jw(w) ]? @ J@ w(w)22

1

(3.26)

where we have to compute the gradient and Hessian on the right-hand side at the
point .
Algorithm (3.26) is called Newton’s method, and it is one of the most efficient
ways for function minimization. It is, in fact, a special case of the well-known
Newton’s method for solving an equation; here it solves the equation that says thatthe gradient is zero.
Newton’s method provides a fast convergence in the vicinity of the minimum,
if the Hessian matrix is positive definite there, but the method may perform poorly
farther away. A complete convergence analysis is given in [284]. It is also shown
there that the convergence of Newton’s method is quadratic; if  is the limit ofconvergence, thenw

w

kw0 ? w k  kw ? w k2where is a constant. This is a very strong mode of convergence. When the error on
the right-hand side is relatively small, its square can be orders of magnitude smaller.
(If the exponent is 3, the convergence is called cubic, which is somewhat better thanquadratic, although the difference is not as large as the difference between linear andquadratic convergence.)
On the other hand, Newton’s method is computationally much more demanding
per one iteration than the steepest descent method. The inverse of the Hessian
has to be computed at each step, which is prohibitively heavy for many practicalcost functions in high dimensions. It may also happen that the Hessian matrix
becomes ill-conditioned or close to a singular matrix at some step of the algorithm,
which induces numerical errors into the iteration. One possible remedy for this is

67

LEARNING RULES FOR UNCONSTRAINED OPTIMIZATIONI

to regularize the algorithm by adding a diagonal matrix  , with  small, to the
Hessian before inversion. This makes it better conditioned. This is the basis of theMarquardt-Levenberg algorithm (see, e.g., [83]).
For error functions that can be expressed as the sum of error squares, one can apply
the so-called Gauss-Newton method instead of Newton’s method. It is intermediate
between the steepest descent and Newton’s method with respect to both the computational load and convergence speed. Also the conjugate gradient method provides asimilar compromise [46, 135, 284, 172, 407].
In ICA, these second-order methods in themselves are not often used, but the
FastICA algorithm uses an approximation of the Newton method that is tailored to
the ICA problem, and provides fast convergence with little computation per iteration.
3.2.3

The natural gradient and relative gradient

The gradient of a function J points in the steepest direction in the Euclidean orthogonal coordinate system. However, the parameter space is not always Euclidean but
has a Riemannian metric structure, as pointed out by Amari [4]. In such a case, thesteepest direction is given by the so-called natural gradient instead. Let us here only
consider the case of nonsingular m  m matrices, that are important in ICA learning
rules. It turns out that their space has a Riemannian structure with a conveniently
computable natural gradient.
and wish to find a direction for a small matrix
Assume that we are at the point
increment  such that the value J ( +  ) is minimized, under the constraintthat the squared norm k k2 is constant. This is a very natural requirement, as
any step in a gradient algorithm for minimization of function g must consist of thedirection of the step and the length. Keeping the length constant, we search for the
optimal direction.
The squared norm is defined as a weighted matrix inner product

W

W
W

W

W

kWk =< W; W >W
2such that<  W ;  W >I =mXi;j =1(wij )2 = trace(WT W)

Amari argues that, due to the Riemannian structure of the matrix space, this innerproduct should have the following invariance:< W; W >W =< WM; WM >WM

M. Putting M = W? , this gives< W; W >W =< WW? ; WW? >I = trace((WT )? WT WW? ):1

for any matrix

1

1

1

1

Keeping this inner product constant, it was shown in [4] that the largest increment
for J ( +  ) is obtained in the direction of the natural gradient

W

W

@J T
@J
@ Wnat = @ W W W

68

GRADIENTS AND OPTIMIZATION METHODS

W

must be multiplied from the right by the matrix
So, the usual gradient at point
T . This results in the following natural gradient descent rule for the costfunction J ( ):WWW

J WT WW / ? @@W(3.27)

This kind of ICA learning rules will be discussed later in this book.
A related result was derived by Cardoso [71] from a slightly different startingpoint. Let us write the Taylor series of J ( +  ):WWJ )T W] + :::J (W + W) = J (W) + trace[( @@W
Let us require that the displacement  W is always proportional to W itself,  W =DW. We haveJ )T DW] + :::J (W + DW) = J (W) + trace[( @@W
J )T ] + :::= J (W) + trace[DW( @@W
because by definition of the trace, trace(M M ) = trace(M M ) for any matrices
M ; M of a suitable size. This becomes@J T T@J T T
trace[D(@ W W ) ] = trace[( @ W W ) D]
J WT , is called the relative gradient by Cardoso.
The multiplier for D, or matrix @@W
It is the usual matrix gradient multiplied by WT .
The largest decrement in the value of J (W + DW) ? J (W) is now obviously
J WT )T D] is minimized, which happens when D
obtained when the term trace[( @@W@J
T
is proportional to ?( @ W W ). Because  W = DW we have a gradient descentlearning rule
J WT )W
W / ?( @@W
1

1

2

2

1

2

which is exactly the same as the natural gradient learning rule (3.27).
3.2.4

Stochastic gradient descent

Up to now, gradient methods were considered from a general point of view, without
assuming any specific form for the cost function J ( ) or J ( ). In this section,
cost functions of a specific kind will be considered.
ICA, like many other statistical and neural network techniques, is a totally datadependent or data-driven technique. If we do not have any observation data, then wecannot solve the problem at all. Therefore the ICA cost functions will depend on theobservation data. Typically, the cost functions have the formW

J (w) = Efg(w; x)gw(3.28)

LEARNING RULES FOR UNCONSTRAINED OPTIMIZATION

x

69

There is the observation vector, that is modeled as a random vector with some
unknown density f ( ). The expectation in (3.28) is with respect to this density. In
practice, there must be a sample (1); (2); :::; of these vectors available. Usually
this is the only data we have.
Especially in constrained optimization problems, the cost functions will be somewhat more complicated, but the core factor has the form of (3.28) and therefore we
consider this simplest case first. The steepest descent learning rule becomes

x

x

x

w(t) = w(t ? 1) ? (t) @@w Efg(w; x(t))gjw w t?
=(

(3.29)

1)w

Again, the gradient vector is computed at the point (t ? 1). In practice, theexpectation is approximated by the sample mean of this function over the sample(1); :::; (T ). This kind of algorithm, where the entire training set is used at every
step of the iteration to form the expectation, is called batch learning.
In principle, it is easy to form the gradient and Hessian of (3.28) because first andsecond derivatives with respect to the elements of vector can be taken inside theexpectation with respect to ; for instance,

x

x

w

x

@ Efg(w; xg = @ Z g(w; )f ()d@w
Z@ w @= [ @ w g(w; )]f ()d(3.30)(3.31)

wx

It suffices that function g ( ; ) is twice differentiable with respect to the elementsof for this operation to be allowed.
However, it may sometimes be tedious to compute the mean values or sample
averages of the appropriate functions at each iteration step. This is especially problematic if the sample (1); (2); :::; is not fixed but new observations keep on comingin the course of the iteration. The statistics of the sample vectors may also be slowly
varying, and the algorithm should be able to track this. In a learning paradigm
called on-line learning, the whole sample is not used in batch at each step of thealgorithm, but only the latest observation vector (t) is used. Effectively, this meansthat the expectation in the learning rule (3.29) is dropped and the on-line learning
rule becomes

w

x

x

x

w(t) = w(t ? 1) ? (t) @@w g(w; x)jw w t? :=((3.32)

1)

This leads to highly fluctuating directions of instantaneous gradients on subsequent
iteration steps, but the average direction in which the algorithm proceeds is stillroughly the direction of the steepest descent of the batch cost function. Generally,stochastic gradient algorithms converge much slower than the respective steepest
descent algorithms. This is compensated by their often very low computational cost.
The computational load at one step of the iteration is considerably reduced; the valueof the function @@w g ( ; ) has to be computed only once, for the vector (t). In

wx

x

70

GRADIENTS AND OPTIMIZATION METHODS

wx

the batch algorithm, when evaluating the function @@w Efg ( ; )g, this value must be
computed T times, once for each sample vector (t), then summed up and divided
by T .
The trade-off is that the on-line algorithm typically needs many more steps forconvergence. If the training sample is fixed, it must be used several times in thealgorithm. Typically the sample vectors (t) are either picked one by one in acyclical order, or by random choice. The random ordering or shuffling is usually abetter choice. By running the algorithm over sufficiently many iterations over the
training set, a reasonable final accuracy can be achieved.
Examples of such stochastic gradient algorithms will be encountered later in thisbook. Several learning algorithms of principal component analysis networks and thewell-known least-mean-squares algorithm, for example, are instantaneous stochastic
gradient algorithms.

x

x

x

x As
x

, with the elements
Example 3.5 We assume that satisfies the ICA model =of the source vector statistically independent and the mixing matrix. The problem
is to solve and , knowing (or in practice, a sample from ). Due to the linearityof this model, it is reasonable to look for the solution as a linear function of . One
way of doing this is to take a scalar variable y = T and try to solve the parameteror weight vector
so that y will become equal to one of the elements of . Oneof the possible criteria for ICA, as we will see in Chapter 8, is to find y = Tsuch that it has maximal fourth order moment: E fy 4 g = Ef( T )4 g = max. The
computational setting is such that we have a sample (1); (2); :::; (T ) of vectors ,and we should solve the vector that maximizes the fourth order moment of T .
Now computing the gradient of the scalar function J ( ) = Ef( T )4 g gives@ J (w) = 4Ef( T )3 g. Thus, a simple batch learning rule for the weight vector@w
maximizing the cost function would be

s

A

s

A

x

s
wx

w

x

w

wx x

w

x
s

wx
w x
x
x
x
wx
w
wx

w(t) = w(t ? 1) + (t)Ef[w(t ? 1)T x(t)] x(t)g3

or, in our shorthand notation introduced earlier,w / Ef(wT x)3 xg(3.33)

Note that the number 4 appearing in the gradient has been absorbed in the learning rate (t), whose magnitude has to be determined anyway. In practice, theexpectation at each step would be computed as the sample average over the sample(1); (2); :::; (T ) as Ef( T )3 g  1=T Tt=1 [ T (t)]3 (t), where is thevalue of the solution vector at that iteration step.
There are some comments relating to this algorithm: first, we have a positive signin front of the gradient term because we actually wish to maximize, not minimize,the cost function. So we are moving in the direction of the gradient, which shows
the direction in which the function grows fastest. Second, this is not a very good
has not
algorithm for solving the maximization problem, because the norm of
been constrained in any way; what would happen is that the norm or magnitude ofwould grow without bounds because it has the effect of increasing the value of

x

x

x

w x x

P wx

x

w

w

w

71

LEARNING RULES FOR UNCONSTRAINED OPTIMIZATION

the cost function. A simple normalization will solve this problem, as we will see inSection 3.3.
3.2.5

Convergence of stochastic on-line algorithms *A valid question is what exactly is the relation of the on-line learning algorithm (3.32)
to the corresponding batch algorithm (3.29): Does the on-line algorithm converge tothe same solution in theory? Mathematically, the two algorithms are quite different.
The batch algorithm is a deterministic iteration because the random vector isaveraged out on the right-hand side. It can thus be analyzed with all the techniques
available for one-step iteration rules, like fixed points and contractive mappings. In
contrast, the on-line algorithm is a stochastic difference equation because the righthand side is a random vector, due to (t). Even the question of the convergence of
the algorithm is not straightforward, because the randomness causes fluctuations thatnever die out unless they are deliberately frozen by letting the learning rate go tozero.
The analysis of stochastic algorithms like (3.32) is the subject of stochastic approximation; see, e.g., [253]. In brief, the analysis is based on the averaged differential
equation that is obtained from (3.32) by taking averages over on the right-handside: the differential equation corresponding to (3.32) is

x

x

x

dw = ? @ Efg(w; x)gdt@w(3.34)

This is in effect the continuous-time counterpart of the batch algorithm (3.29). Notethat the right-hand side is a function of
only; there is no dependence on or t(although, through expectation, it of course depends on the probability density of ).
Such differential equations are called autonomous, and they are the simplest ones toanalyze.
The theory of autonomous differential equations is very well understood. The only
possible points where the solution can converge are the fixed or stationary points,
i.e., roots of the right-hand side, because these are the points where the change inover time becomes zero. It is also well-known how by linearizing the right-handside with respect to a stability analysis of these fixed points can be accomplished.
Especially important are the so-called asymptotically stable fixed points that are local
points of attraction.
Now, if the learning rate (t) is a suitably decreasing sequence, typically satisfying

w

w

x

x

w

1
X
t=1

1
X

wx

t=1

2

(t) = 1(3.35)(t) < 1

(3.36)and the nonlinearity g ( ; ) satisfies some technical assumptions [253], then it canbe shown that the on-line algorithm (3.32) must converge to one of the asymptotically

72

GRADIENTS AND OPTIMIZATION METHODS

stable fixed points of the differential equation (3.34). These are also the convergencepoints of the batch algorithm, so usually the two algorithms indeed result in the same
final solution. In practice, it is often possible to analyze the fixed points even if thefull convergence proof is intractable.
The conditions (3.35) are theoretical and do not necessarily work very well inpractice. Sometimes the learning rate is decreased at first, but then kept at a small
constant value. A good choice satisfying the conditions may be(t) = + t
with an appropriate constant (e.g., = 100). This prevents the learning rate fromdecreasing too quickly in the early phase of the iteration.
In many cases, however, on-line learning is used to provide a fast adaptation to achanging environment. The learning rate is then kept constant. If the input data is
nonstationary, that is, its statistical structure changes as a function of time, this allows
the algorithm to track these changes, and adapt quickly to the changing environment.

Example 3.6 In Chapter 6, on-line PCA is discussed. One of the learning rules is asfollows:

w / xy ? y2 w

wx

(3.37)

xwhere y = T and is a random vector. The question is where this on-line rulemight converge. We can now analyze it by forming the averaged ordinary differential
equation (ODE):

dw = Efx(xT w) ? (wT x)(xT w)wg
dt

= EfxxT gw ? (wT EfxxT gw)w
= Cx w ? (wT Cxw)w

C

xx

x

where x = Ef T g is the covariance matrix of (assumed zero-mean here). Notethat the average is taken over , assuming constant. The fixed points of the ODE
are given by solutions of x ? ( T x ) = 0. As the term T x is a scalar,
we know from elementary matrix algebra that all solutions must be eigenvectorsof the matrix x . The numbers T x are the corresponding eigenvalues. The
principal components of a random vector are defined in terms of the eigenvectors,
as discussed in Chapter 6. With a somewhat deeper analysis, it can be shown [324]that the only asymptotically stable fixed point is the eigenvector corresponding to thelargest eigenvalue, which gives the first principal component.
The example shows how an intractable stochastic on-line rule can be nicely analyzed by the powerful analysis tools existing for ODEs.

Cxw
C w w C ww
w Cw
x

w

w Cw

LEARNING RULES FOR CONSTRAINED OPTIMIZATION

73

3.3 LEARNING RULES FOR CONSTRAINED OPTIMIZATIONw

In many cases we have to minimize or maximize a function J ( ) under some
additional conditions on the solution . Generally, the constrained optimization
problem is formulated asw

min J (w); subject to Hi (w) = 0; i = 1; :::; k(3.38)

where J (w), as before, is the cost function to be minimized and Hi (w) = 0; i =1; :::; k give a set of k constraint equations on w.
3.3.1

The Lagrange method

The most prominent and widely used way to take the constraints into account is themethod of Lagrange multipliers. We form the Lagrangian function

L(w;  ; ::::; k ) = J (w) +
1

k
X
i=1

i Hi (w)

(3.39)

where 1 ; :::; k are called Lagrange multipliers. Their number k is the same as thenumber of separate scalar constraint equations.
The general result is that the minimum point of the Lagrangian (3.39), where
and all the i gives the solution to theits gradient is zero with respect to bothoriginal constrained optimization problem (3.38). The gradient of L( ; 1 ; :::; k )with respect to i is simply the ith constraint function Hi ( ), so putting all these tozero again gives the original constraint equations. The important point is that when
we form the gradient of L( ; 1 ; :::; k ) with respect to and put it to zero,

w

w
w

w

w

k @H (w)@ J (w) + X
i @iw = 0@w
i=1(3.40)

we have changed the minimization problem into a set of equations that is much easier
to solve.
A possible way to solve the two sets of equations, one set given by the constraints,
the other by (3.40), is, e.g., Newton iteration or some other appropriate iteration
method. These methods give learning rules that resemble the ones in the previous
section, but now instead of the gradient of J ( ) only, the gradient of L( ; 1; :::; k )will be used.

w

3.3.2

w

Projection methods

In most of the constrained optimization problems of this book, the constraints areof the equality type and relatively simple: typically, we require that the norm ofis constant, or some quadratic form ofis constant. We can then use anotherconstrained optimization scheme: projections on the constraint set. This means that

w

w

74

GRADIENTS AND OPTIMIZATION METHODS

we solve the minimization problem with an unconstrained learning rule, which might
be a simple steepest descent, Newton’s iteration, or whatever is most suitable, but
after each iteration step, the solution at that time is projected orthogonally onto
the constraint set so that it satisfies the constraints.

w

Example 3.7 We continue Example 3.5 here. Let us consider the following constrained problem: Assuming is a random vector, maximize the fourth-order moment
Ef( T )4 g under the constraint k k2 = 1. In the terminology of (3.38), the costfunction is J ( ) = ?Ef( T )4 g (note that instead of minimization, we maximize,
hence the minus sign), and the only constraint equation is H ( ) = k k2 ? 1 = 0.
Solving this by the Lagrange method, we formulate the Lagrangian as

wx

x
w
wx

w

w

w

L(w; ) = ?Ef(wT x)4 g + (kwk2 ? 1)w

The gradient with respect to  again gives the constraint k k2 ? 1 = 0. The gradientwith respect to gives ?4Ef( T )3 g + (2 ): We might try to solve the rootsof this, e.g., by Newton iteration, or by a simpler iteration of the form

w

wx x

w

w / Ef(wT x)3 xg ? 2 w

with an appropriate learning rate and . When comparing this to the learning rulein Example 3.5, Eq. (3.33), we notice the additive linear term 2 ; by choosing suitably, the growth in the norm of can be controlled.
However, for this simple constraint, a much simpler way is the projection on theconstraint set. Let us consider simple steepest descent as in eq. (3.33). We only
have to normalize after each step, which is equivalent to orthogonal projection of
onto the unit sphere in the m-dimensional space. This unit sphere is the constraint
set. The learning rule becomes

w

w

w

w

w
w

w + Ef(wT x) xg
w=kwk
3(3.41)(3.42)Exactly the same idea as in the preceding example applies to any cost function. This
will be utilized heavily in the ICA learning rules in the following chapters.
Sometimes a computationally easier learning rule can be obtained from an approximation of the normalization. Consider steepest descent with the norm constraint,
and for simplicity let us write the update rule asw
w

w ? g(w)w=kwk

where we have denoted the gradient of the cost function by
write this isw

w ? g(w)kw ? g(w)k(3.43)(3.44)

g(w). Another way toCONCLUDING REMARKS AND REFERENCES

75

Now, assuming the learning rate is small as is usually the case, at least in the later
iteration steps, we can expand this into a Taylor series with respect to and get a
simplified constrained learning rule [323]. Omitting some intermediate steps, thedenominator becomeskw ? g(w)k  1 ? wT g(w)where all terms proportional to2or higher powers are omitted. The final result is

w ? g(w)  w ? g(w) + (g(w)T w)w
kw ? g(w)k
The resulting learning rule has one extra term compared to the unconstrained rule(3.43), and yet the norm of will stay approximately equal to one.

w

3.4 CONCLUDING REMARKS AND REFERENCES
More information on minimization algorithms in general can be found in books dealing with nonlinear optimization, for example, [46, 135, 284], and their applications[172, 407]. The speed of convergence of the algorithms is discussed in [284, 407]. A
good source for matrix gradients in general is [109]. The natural gradient is considered in detail in [118]. The momentum method and other extensions are covered in[172]. Constrained optimization has been extensively discussed in [284]. Projection
on the unit sphere and the short-cut approximation for normalization has been discussed in [323, 324]. A rigorous analysis of the convergence of the stochastic on-line
algorithms is discussed in [253].
Problems
3.1 Show that the Jacobian matrix of the gradient vector @@g
w with respect toequal to the Hessian of g .

w is

W

is defined as the sum of its diagonal
3.2 The trace of an m  m square matrixmelements i=1 wii . Compute its matrix gradient.

P

WT MW) with respect to W, where W is an

3.3

Show that the gradient of trace(

3.4@ log j det
Show that @ W

m  n matrix and M is an m  m matrix, is equal to MW + MT W.
3.5

Wj = (WT )? .

Consider the 2  2 matrix

1

a b

W= c d

3.5.1. Compute the cofactors with respect to the first column, compute the deteras functions of a; b; c; d.
minant, the adjoint matrix, and the inverse of@ log j det j = ( T )?1 .
3.5.2. Verify in this special case that @ W

W

W

W

76

GRADIENTS AND OPTIMIZATION METHODS

w

w x

x

3.6 Consider a cost function J ( ) = G( T ) where we can assume that is aconstant vector. Assume that the scalar function G is twice differentiable.
3.6.1. Compute the gradient and Hessian of J ( ) in the general case and in thecases that G(t) = t4 and G(t) = log cosh(t).
3.6.2. Consider maximizing this function under the constraint that k k = 1.
Formulate the Lagrangian, its gradient (with respect to ), its Hessian, and the
Newton method for maximizing the Lagrangian.

w

w

w

Let p(:) be a differentiable scalar function, x a constant vector, and W =(w1 :::wn ) an m  n matrix with columns wi . Consider the cost function

3.7

J (W) =

xw

n
X
i=1

log p(ui )

W

ux

u

@ J ( ) = ?'( ) T where is the vector with
where ui = T i . Show that @ W
elements ui and '( ) is a certain function, defined element by element. Give theform of this function. (Note: This matrix gradient is used in the maximum likelihood
approach to ICA, discussed in Chapter 9.)
3.8

uConsider a general stochastic on-line ICA learning rule

W / [I ? g(y)yT ]W

y Wx

where =and g is a nonlinear function. Formulate(a) the corresponding batch learning rule,(b) the averaged differential equation.
is such that the elements
Consider a stationary point of (a) and (b). Show that ifof are zero-mean and independent, then is a stationary point.

y

W

W

w

3.9 Assume that we want to maximize a function F ( ) on the unit sphere, i.e.,under the constraint k k = 1. Prove that at the maximum, the gradient of F must
point in the same direction as . In other words, the gradient must be equal tomultiplied by a scalar constant. Use the Lagrangian method.

w

w

w

Computer assignmentsxCreate a sample of two-dimensional gaussian data with zero mean and covariancematrix

3 1
1 2

:

Apply the stochastic on-line learning rule (3.37), choosing a random initial point
and an appropriate learning rate. Try different choices for the learning rate an dsee how it effects the convergence speed. Then, try to solve the same problem
using a batch learning rule by taking the averages on the right-hand side. Compare
the computational efforts of the on-line vs. batch learning rules at one step of theiteration, and the number of steps needed for convergence in both algorithms. (Note:
The algorithm converges to the dominant eigenvector of the covariance matrix, which
can be solved in closed form.)w

4
Estimation Theory

An important issue encountered in various branches of science is how to estimate thequantities of interest from a given finite set of uncertain (noisy) measurements. Thisis studied in estimation theory, which we shall discuss in this chapter.
There exist many estimation techniques developed for various situations; thequantities to be estimated may be nonrandom or have some probability distributions
themselves, and they may be constant or time-varying. Certain estimation methods
are computationally less demanding but they are statistically suboptimal in many
situations, while statistically optimal estimation methods can have a very high computational load, or they cannot be realized in many practical situations. The choiceof a suitable estimation method also depends on the assumed data model, which may
be either linear or nonlinear, dynamic or static, random or deterministic.
In this chapter, we concentrate mainly on linear data models, studying the estimation of their parameters. The two cases of deterministic and random parameters
are covered, but the parameters are always assumed to be time-invariant. The methods that are widely used in context with independent component analysis (ICA) areemphasized in this chapter. More information on estimation theory can be found inbooks devoted entirely or partly to the topic, for example [299, 242, 407, 353, 419].
Prior to applying any estimation method, one must select a suitable model thatwell describes the data, as well as measurements containing relevant information on
the quantities of interest. These important, but problem-specific issues will not be
discussed in this chapter. Of course, ICA is one of the models that can be used. Some
topics related to the selection and preprocessing of measurements are treated later in
Chapter 13.

77

78

ESTIMATION THEORY

4.1 BASIC CONCEPTS
Assume there are T scalar measurements x(1); x(2); : : : ; x(T ) containing information about the m quantities 1 ; 2 ; : : : ; m that we wish to estimate. The quantities iare called parameters hereafter. They can be compactly represented as the parameter
vector = (1 ; 2 ; : : : ; m)T
(4.1)
Hence, the parameter vector  is an m-dimensional column vector having as its

elements the individual parameters. Similarly, the measurements can be represented
as the T -dimensional measurement or data vector1

xT = [x(1); x(2); : : : ; x(T )]T(4.2)
Quite generally, an estimator ^ of the parameter vector  is the mathematicalexpression or function by which the parameters can be estimated from the measurements:^ = h(xT ) = h(x(1); x(2); : : : ; x(T ))(4.3)

For individual parameters, this becomes^i = hi (xT );i = 1; : : : ; m(4.4)

If the parameters i are of a different type, the estimation formula (4.4) can be quite
different for different i. In other words, the components hi of the vector-valued
function can have different functional forms. The numerical value of an estimator^i , obtained by inserting some specific given measurements into formula (4.4), iscalled the estimate of the parameter i .

h

Example 4.1 Two parameters that are often needed are the mean  and variance  2of a random variable x. Given the measurement vector (4.2), they can be estimatedfrom the well-known formulas, which will be derived later in this chapter:^ = T11^2 = T ?
1

T
X

x(j )

(4.5)

[x(j ) ? ^]2

(4.6)

j =1
T
X
j =1

x

T subsequent scalar samples is denoted in this chapter by T for distinguishing it from the ICA mixture vector , whose components consist of different mixtures.
1 The data vector consisting ofx

BASIC CONCEPTS

79

Example 4.2 Another example of an estimation problem is a sinusoidal signal innoise. Assume that the measurements obey the measurement (data) modelx(j ) = A sin(!t(j ) + ) + v(j );j = 1; : : : ; T(4.7)

Here A is the amplitude, ! the angular frequency, and  the phase of the sinusoid,
respectively. The measurements are made at different time instants t(j ), which areoften equispaced. They are corrupted by additive noise v (j ), which is often assumed
to be zero mean white gaussian noise. Depending on the situation, we may wish to
estimate some of the parameters A, ! , and , or all of them. In the latter case, theparameter vector becomes  = (A; !; )T . Clearly, different formulas must be used
for estimating A, ! , and . The amplitude A depends linearly on the measurementsx(j ), while the angular frequency ! and the phase  depend nonlinearly on the x(j ).
Various estimation methods for this problem are discussed, for example, in [242].
Estimation methods can be divided into two broad classes depending on whether
the parameters  are assumed to be deterministic constants, or random. In thelatter case, it is usually assumed that the parameter vector  has an associatedprobability density function (pdf) p ( ). This pdf, called a priori density, is inprinciple assumed to be completely known. In practice, such exact information isseldom available. Rather, the probabilistic formalism allows incorporation of useful
but often somewhat vague prior information on the parameters into the estimation
procedure for improving the accuracy. This is done by assuming a suitable priordistribution reflecting knowledge about the parameters. Estimation methods using
the a priori distribution p ( ) are often called Bayesian ones, because they utilize
the Bayes’ rule discussed in Section 4.6.
Another distinction between estimators can be made depending on whether they
are of batch type or on-line. In batch type estimation (also called off-line estimation),
all the measurements must first be available, and the estimates are then computed
directly from formula (4.3). In on-line estimation methods (also called adaptive orrecursive estimation), the estimates are updated using new incoming samples. Thus
the estimates are computed from the recursive formula^(j + 1) = h1 (^(j )) + h2 (x(j + 1); ^(j ))

^

(4.8)

where (j ) denotes the estimate based on j first measurements x(1); x(2);: : : ; x(j ).
The correction or update term 2 (x(j + 1); (j )) depends only on the new incoming(j + 1)-th sample x(j + 1) and the current estimate (j ). For example, the estimate
^ of the mean in (4.5) can be computed on-line as follows:

h

^

^

^(j ) = j ?j 1 ^(j ? 1) + 1j x(j )

(4.9)

80

ESTIMATION THEORY

4.2 PROPERTIES OF ESTIMATORSNow briefly consider properties that a good estimator should satisfy.
Generally, assessing the quality of an estimate is based on the estimation error,
which is defined by

~ =  ? ^ =  ? h(xT )(4.10)
Ideally, the estimation error ~ should be zero, or at least zero with probability one.
But it is impossible to meet these extremely stringent requirements for a finite data
set. Therefore, one must consider less demanding criteria for the estimation error.

Unbiasedness and consistency The first requirement is that the mean valueof the error E fg should be zero. Taking expectations of the both sides of Eq. (4.10)
leads to the condition~^

Efg = Efg

(4.11)

Estimators that satisfy the requirement (4.11) are called unbiased. The preceding definition is applicable to random parameters. For nonrandom parameters, the respective
definition is^

E f j  g = (4.12)

Generally, conditional probability densities and expectations, conditioned by theparameter vector  , are used throughout in dealing with nonrandom parameters toindicate that the parameters  are assumed to be deterministic constants. In this case,the expectations are computed over the random data only.
If an estimator does not meet the unbiasedness conditions (4.11) or (4.12). it
is said to be biased. In particular, the bias is defined as the mean value of theestimation error:bb = Ef~g, or b = Ef~ j g

(4.13)

If the bias approaches zero as the number of measurements grows infinitely large, theestimator is called asymptotically unbiased.
Another reasonable requirement for a good estimator  is that it should converge
to the true value of the parameter vector , at least in probability,2 when the number ofmeasurements grows infinitely large. Estimators satisfying this asymptotic property
are called consistent. Consistent estimators need not be unbiased; see [407].

^

Example 4.3 Assume that the observations x(1); x(2); : : : ; x(T ) are independent.
The expected value of the sample mean (4.5) is
Ef
^g =

T
1X
1
Efx(j )g = T = 

T j=1

T

2 See for example [299, 407] for various definitions of stochastic convergence.

(4.14)

PROPERTIES OF ESTIMATORS

81

Thus the sample mean is an unbiased estimator of the true mean . It is also consistent,
which can be seen by computing its varianceEf(^ ? )2 g =T
1 X
1 T2 = 2
Ef[x(j ) ? ]2 g =T 2 j=1

T2

T(4.15)The variance approaches zero when the number of samples T ! 1, implying
together with unbiasedness that the sample mean (4.5) converges in probability to thetrue mean .

~

Mean-square error It is useful to introduce a scalar-valued loss function L()for describing the relative importance of specific estimation errors . A popular loss
function is the squared estimation error L() = k  k2 = k  ?  k2 because of its
mathematical tractability. More generally, typical properties required from a valid
loss function are that it is symmetric: L() = L(?); convex or alternatively at least
nondecreasing; and (for convenience) that the loss corresponding to zero error iszero: L( ) = 0. The convexity property guarantees that the loss function decreases
as the estimation error decreases. See [407] for details.
The estimation error  is a random vector depending on the (random) measurement
vector T . Hence, the value of the loss function L() is also a random variable. To
obtain a nonrandom error measure, is is useful to define the performance index or
error criterion E as the expectation of the respective loss function. Hence,

~

~
~

~

^

~

0

~

x

~

E = EfL(~)g or E = EfL(~) j g
(4.16)
where the first definition is used for random parameters  and the second one fordeterministic ones.
A widely used error criterion is the mean-square error (MSE)

EMSE = Efk  ? ^ k g
2(4.17)

If the mean-square error tends asymptotically to zero with increasing number ofmeasurements, the respective estimator is consistent. Another important property ofthe mean-square error criterion is that it can be decomposed as (see (4.13))

EMSE = Efk ~ ? b k g+ k b k
2

~ b

2

(4.18)

The first term Efk  ? k2 g on the right-hand side is clearly the variance of theestimation error . Thus the mean-square error EMSE measures both the varianceand the bias of an estimator  . If the estimator is unbiased, the mean-square error
coincides with the variance of the estimator. Similar definitions hold for deterministic
parameters when the expectations in (4.17) and (4.18) are replaced by conditional
ones.
Figure 4.1 illustrates the bias b and standard deviation  (square root of the variance2 ) for an estimator ^ of a single scalar parameter . In a Bayesian interpretation(see Section 4.6), the bias and variance of the estimator ^ are, respectively, the mean

~

^

82

ESTIMATION THEORYp(^jx)b
E (^)

^

Fig. 4.1 Bias b and standard deviation  of an estimator ^.

and variance of the posterior distribution p^jxT (^ j ) of the estimator ^ given theobserved data T .
Still another useful measure of the quality of an estimator is given by the covariancematrix of the estimation errorxxC~ = Ef~~T g = Ef( ? ^)( ? ^)T g(4.19)It measures the errors of individual parameter estimates, while the mean-square error
is an overall scalar error measure for all the parameter estimates. In fact, the meansquare error (4.17) can be obtained by summing up the diagonal elements of the error
covariance matrix (4.19), or the mean-square errors of individual parameters.
Efficiency An estimator that provides the smallest error covariance matrix among
all unbiased estimators is the best one with respect to this quality criterion. Suchan estimator is called an efficient one, because it optimally uses the information
is said to be smaller thancontained in the measurements. A symmetric matrix
another symmetric matrix , or < , if the matrix ? is positive definite.
A very important theoretical result in estimation theory is that there exists a lower
bound for the error covariance matrix (4.19) of any estimator based on available
measurements. This is provided by the Cramer-Rao lower bound. In the following
theorem, we formulate the Cramer-Rao lower bound for unknown deterministic
parameters.

B A B

A
B A

PROPERTIES OF ESTIMATORS

83^

Theorem 4.1 [407] If  is any unbiased estimator of  based on the measurement
data , then the covariance matrix of error in the estimator is bounded below by the
inverse of the Fisher information matrix J:

x

^

^

Ef( ? )( ? )T j  g 
where

J=E

(
@

J?

1

(4.20)T ) @@  ln p(xT j ) @  ln p(xT j ) j (4.21)

Here it is assumed that the inverse ?1 exists. The term @@ ln p( T j ) is
recognized to be the gradient vector of the natural logarithm of the joint distribution3 p( T j  ) of the measurements T for nonrandom parameters  . The partial
derivatives must exist and be absolutely integrable.
It should be noted that the estimator  must be unbiased, otherwise the preceding
theorem does not hold. The theorem cannot be applied to all distributions (forexample, to the uniform one) because of the requirement of absolute integrability ofthe derivatives. It may also happen that there does not exist any estimator achieving
the lower bound. Anyway, the Cramer-Rao lower bound can be computed for many
problems, providing a useful measure for testing the efficiency of specific estimation
methods designed for those problems. A more thorough discussion of the CramerRao lower bound with proofs and results for various types of parameters can be found,for example, in [299, 242, 407, 419]. An example of computing the Cramer-Rao
lower bound will be given in Section 4.5.

J

x

x

x
^

Robustness In practice, an important characteristic of an estimator is its robustness [163, 188]. Roughly speaking, robustness means insensitivity to grossmeasurement errors, and errors in the specification of parametric models. A ty picalproblem with many estimators is that they may be quite sensitive to outliers, that is,observations that are very far from the main bulk of data. For example, consider theestimation of the mean from 100 measurements. Assume that all the measurements(but one) are distributed between ?1 and 1, while one of the measurements has thevalue 1000. Using the simple estimator of the mean given by the sample averagein (4.5), the estimator gives a value that is not far from the value 10. Thus, thesingle, probably erroneous, measurement of 1000 had a very strong influence on theestimator. The problem here is that the average corresponds to minimization of thesquared distance of measurements from the estimate [163, 188]. The square function
implies that measurements far away dominate.
Robust estimators can be obtained, for example, by considering instead of thesquare error other optimization criteria that grow slower than quadratical ly withthe error. Examples of such criteria are the absolute value criterion and criteriaxx  for notational simplicity. Thisj  of the density function p(
practice is followed in this chapter unless confusion is possible.

3 We have here omitted the subscript

j

)

84

ESTIMATION THEORY

that saturate as the error grows large enough [83, 163, 188]. Optimization criteria
growing faster than quadratically generally have poor robustness, because a few
large individual errors corresponding to the outliers in the data may almost solely
determine the value of the error criterion. In the case of estimating the mean, forexample, one can use the median of measurements instead of the average. This
corresponds to using the absolute value in the optimization function, and gives a very
robust estimator: the single outlier has no influence at all.

4.3 METHOD OF MOMENTS
One of the simplest and oldest estimation methods is the method of moments. It is
intuitively satisfying and often leads to computationally simple estimators, but on theother hand, it has some theoretical weaknesses. We shall briefly discuss the moment
method because of its close relationship to higher-order statistics.
Assume now that there are T statistically independent scalar measurements or data
samples x(1); x(2); : : : ; x(T ) that have a common probability distribution p(x j )
characterized by the parameter vector  = (1 ; 2 ; : : : ; m )T in (4.1). Recall from
Section 2.7 that the j th moment j of x is defined by

j = Efxj j g =Z1

?1

xj p(x j )dx;

j = 1; 2; : : :(4.22)

Here the conditional expectations are used to indicate that the parameters  are(unknown) constants. Clearly, the moments j are functions of the parameters  .
On the other hand, we can estimate the respective moments directly from themeasurements. Let us denote by dj the j th estimated moment, called the j th sample
moment. It is obtained from the formula (see Section 2.2)

dj = T1

TX
i=1[x(i)]j(4.23)The simple basic idea behind the method of moments is to equate the theoretical
moments j with the estimated ones dj :j () = j (1 ; 2 ; : : : ; m ) = dj

(4.24)

j = Ef(x ? 1 )j j g

(4.25)

Usually, m equations for the m first moments j = 1; : : : ; m are sufficient forsolving the m unknown parameters 1 ; 2 ; : : : ; m . If Eqs. (4.24) have an acceptable
solution, the respective estimator is called the moment estimator, and it is denoted in
the following by MM .
Alternatively, one can use the theoretical central moments^

and the respective estimated sample central moments

1
sj = T ?
1

T
X
i=1

[x(i) ? d1 ]j(4.26)METHOD OF MOMENTS

85

to form the m equations

j (1 ; 2 ; : : : ; m ) = sj ;j = 1; 2; : : : ; m(4.27)

for solving the unknown parameters  = (1 ; 2 ; : : : ; m )T .
Example 4.4 Assume now that x(1); x(2); : : : ; x(T ) are independent and identically distributed samples from a random variable x having the pdf

p(x j ) = 1 exp ? (x ? 1 )2

(4.28)

2

where 1 < x < 1 and 2 > 0. We wish to estimate the parameter vector  =(1 ; 2 )T using the method of moments. For doing this, let us first compute thetheoretical moments 1 and 2 :12= Efx j g == Efx j g =
2

Z1x
1

Z 1x

2





(x ? 1 ) dx =  + 
exp
?
1
2


2

(4.29)2 (x ?  ) 

exp ? 2
1 2

1dx = (1 + 2 )2 + 22(4.30)The moment estimators are obtained by equating these expressions with the first two
sample moments d1 and d2 , respectively, which yields

1 +  2 = d 1(1 + 2 )2 + 22 = d2

(4.31)(4.32)

Solving these two equations leads to the moment estimates^1;MM = d1 ? (d2 ? d21 )1=2
^2;MM = (d2 ? d21 )1=2

(4.33)(4.34)The other possible solution ^2;MM = ?(d2 ? d21 )1=2 must be rejected because theparameter 2 must be positive. In fact, it can be observed that ^2;MM equals the
sample estimate of the standard deviation, and ^1;MM can be interpreted as the mean
minus the standard deviation of the distribution, both estimated from the available
samples.
The theoretical justification for the method of moments is that the sample moments

dj are consistent estimators of the respective theoretical moments j [407]. Similarly,
the sample central moments sj are consistent estimators of the true central moments
j . A drawback of the moment method is that it is often inefficient. Therefore, it

is usually not applied provided that other, better estimators can be constructed. In
general, no claims can be made on the unbiasedness and consistency of estimates

86

ESTIMATION THEORY

given by the method of moments. Sometimes the moment method does not even leadto an acceptable estimator.
These negative remarks have implications in independent component analysis. Algebraic, cumulant-based methods proposed for ICA are typically based on estimating
fourth-order moments and cross-moments of the components of the observation (data)
vectors. Hence, one could claim that cumulant-based ICA methods inefficiently utilize, in general, the information contained in the data vectors. On the other hand,these methods have some advantages. They will be discussed in more detail in
Chapter 11, and related methods can be found in Chapter 8 as well.

4.4 LEAST-SQUARES ESTIMATION
4.4.1

Linear least-squares method

The least-squares method can be regarded as a deterministic approach to the estimation problem where no assumptions on the probability distributions, etc., arenecessary. However, statistical arguments can be used to justify the least-squares
method, and they give further insight into its properties. Least-squares estimation is
discussed in numerous books, in a more thorough fashion from estimation point-ofview, for example, in [407, 299].
In the basic linear least-squares method, the T -dimensional data vectors T are
assumed to obey the following model:xxT = H + vT

(4.35)

v

Here  is again the m-dimensional parameter vector, and T is a T -vector whose
components are the unknown measurement errors v (j ); j = 1; : : : ; T . The T  mobservation matrix is assumed to be completely known. Furthermore, the numberof measurements is assumed to be at least as large as the number of unknown
parameters, so that T  m. In addition, the matrix has the maximum rank m.
First, it can be noted that if m = T , we can set T = , and get a unique solution
 = ?1 T . If there were more unknown parameters than measurements ( m > T ),infinitely many solutions would exist for Eqs. (4.35) satisfying the condition =. However, if the measurements are noisy or contain errors, it is generally highly
desirable to have much more measurements than there are parameters to be estimated,in order to obtain more reliable estimates. So, in the following we shall concentrateon the case T > m.
When T > m, equation (4.35) has no solution for which T = . Because themeasurement errors T are unknown, the best that we can then do is to choose an
estimator  that minimizes in some sense the effect of the errors. For mathematicalconvenience, a natural choice is to consider the least-squares criterion

H

H
v 0

H x

v

0

^

v

v

ELS = 21 k vT k = 12 (xT ? H)T (xT ? H)2

0(4.36)LEAST-SQUARES ESTIMATION

87

Note that this differs from the error criteria in Section 4.2 in that no expectation is
involved and the criterion ELS tries to minimize the measurement errors , and not
directly the estimation error  ? .
Minimization of the criterion (4.36) with respect to the unknown parameters leads to so-called normal equations [407, 320, 299]v^

(HT H)^LS = HT xT(4.37)

^LS = (HT H)?1 HT xT = H+ xT(4.38)^

for determining the least-squares estimate LS of . It is often most convenient tosolve  LS from these linear equations. However, because we assumed that the matrix
has full rank, we can explicitly solve the normal equations, gettingH^

where + = ( T )?1 T is the pseudoinverse of (assuming that has maximal
rank m and more rows than columns: T > m) [169, 320, 299].
The least-squares estimator can be analyzed statistically by assuming that themeasurement errors have zero mean: E f T g = . It is easy to see that the leastsquares estimator is unbiased: EfLS j  g = . Furthermore, if the covariancematrix of the measurement errors v = Ef T TT g is known, one can compute thecovariance matrix (4.19) of the estimation error. These simple analyses are left as an
exercise to the reader.

H

HH H

H

v

^
C

H

0

vv

Example 4.5 The least-squares method is commonly applied in various branches ofscience to linear curve fitting. The general setting here is as follows. We try to fit to
the measurements the linear modely(t) =mX
i=1

ai i (t) + v(t)(4.39)

Here i (t), i = 1; 2; : : : ; m, are m basis functions that can be generally nonlinear
functions of the argument t — it suffices that the model (4.39) be linear with respect
to the unknown parameters ai . Assume now that there are available measurementsy(t1 ); y(t2 ); : : : ; y(tT ) at argument values t1 ; t2 ; : : : ; tT , respectively. The linear
model (4.39) can be easily written in the vector form (4.35), where now the parameter
vector is given by = [a1 ; a2 ; : : : ; am ]T(4.40)

xT = [y(t ); y(t ); : : : ; y(tT )]T(4.41)and the data vector by

v

1

2

Similarly, the vector T = [v (t1 ); v (t2 ); : : : ; v (tT )]T contains the error terms v (ti ).
The observation matrix becomes2  (t )  (t )     (t ) 3
66  (t )  (t )    mm(t ) 77
H = 64 ..
75
..
..
..
.
.
.
.
1

1

2

1

1

1

2

2

2

2

1 (tT ) 2 (tT )    m (tT )(4.42)

88

ESTIMATION THEORY

H

Inserting the numerical values into (4.41) and (4.42) one can now determine andT , and then compute the least-squares estimates a^i;LS of the parameters ai of thecurve from the normal equations (4.37) or directly from (4.38).
The basis functions i (t) are often chosen so that they satisfy the orthonormality
conditions

x

T
X
i=1

j (ti )k (ti ) = 1; j = k
0; j 6= k(4.43)

HH I
HH
^ Hx

Now T = , since Eq. (4.43) represents this condition for the elements (j; k ) of
the matrix T . This implies that the normal equations (4.37) reduce to the simple
form LS = T T . Writing out this equation for each component of LS provides
for the least-squares estimate of the parameter ai^a^i;LS =TX
j =1

i (tj )y(tj );i = 1; : : : ; m(4.44)

Note that the linear data model (4.35) employed in the least-squares method resembles closely the noisy linear ICA model =+ to be discussed in Chapter 15.
Clearly, the observation matrix in (4.35) corresponds to the mixing matrix , theparameter vector  to the source vector , and the error vector to the noise vector
in the noisy ICA model. These model structures are thus quite similar, but theassumptions made on the models are clearly different. In the least-squares model theobservation matrix is assumed to be completely known, while in the ICA model
the mixing matrix
is unknown. This lack of knowledge is compensated in ICA
by assuming that the components of the source vector are statistically independent,
while in the least-squares model (4.35) no assumptions are needed on the parameter
vector . Even though the models look the same, the different assumptions lead toquite different methods for estimating the desired quantities.
The basic least-squares method is simple and widely used. Its success in practicedepends largely on how well the physical situation can be described using the linear
model (4.35). If the model (4.35) is accurate for the data and the elements of theare known from the problem setting, good estimation results
observation matrix
can be expected.

H

n

H
A

x As n

s

A

v

s

H

4.4.2

Nonlinear and generalized least-squares estimators *Generalized least-squares The least-squares problem can be generalized byto the criterion (4.36).
adding a symmetric and positive definite weighting matrix
The weighted criterion becomes [407, 299]W

EWLS = (xT ? H)T W(xT ? H)(4.45)

W

It turns out that a natural, optimal choice for the weighting matrix
is the inverse of= ?
the covariance matrix of the measurement errors (noise)v 1 . This is because

W C

LEAST-SQUARES ESTIMATION

89

for this choice the resulting generalized least-squares estimator

^WLS = (HT C?v 1 H)?1 HT Cv?1 xT(4.46)^

also minimizes the mean-square estimation error EMSE = Efk  ?  k2 j g [407,299]. Here it is assumed that the estimator  is linear and unbiased. The estimator(4.46) is often referred to as the best linear unbiased estimator (BLUE) or GaussMarkov estimator.
Note that (4.46) reduces to the standard least-squares solution (4.38) if v =  2 .
This happens, for example, when the measurement errors v (j ) have zero mean andare mutually independent and identically distributed with a common variance  2 . The
choice v =  2 also applies if we have no prior knowledge of the covariance matrix
v of the measurement errors. In these instances, the best linear unbiased estimator(BLUE) minimizing the mean-square error coincides with the standard least-squares
estimator. This connection provides a strong statistical argument supporting theuse of the least-squares method, because the mean-square error criterion directly
measures the estimation error  ?  .

^

C

C

C

I

I

^

Nonlinear least-squares The linear data model (4.35) employed in the linear
least-squares methods is not adequate for describing the dependence between theparameters  and the measurements T in many instances. It is therefore natural toconsider the following more general nonlinear data modelx

xT = f () + vT
f

(4.47)

Here is a vector-valued nonlinear and continuously differentiable function of theparameter vector  . Each component fi ( ) of () is assumed to be a known scalarfunction of the components of .
Similarly to previously, the nonlinear least-squares criterion ENLS is defined as
the squared sum of the measurement (or modeling) errors k T k2 = j [v (j )]2 .
From the model (4.47), we get

f

v

P

ENLS = [xT ? f ()]T [xT ? f ()](4.48)The nonlinear least-squares estimator ^NLS is the value of  that minimizes ENLS .

The nonlinear least-squares problem is thus nothing but a nonlinear optimization
problem where the goal is to find the minimum of the function ENLS . Such problems
cannot usually be solved analytically, but one must resort to iterative numericalmethods for finding the minimum. One can use any suitable nonlinear optimization
method for finding the estimate  NLS . These optimization procedures are discussed
briefly in Chapter 3 and more thoroughly in the books referred to there.
The basic linear least-squares method can be extended in several other directions.
It generalizes easily to the case where the measurements (made, for example, atdifferent time instants) are vector-valued. Furthermore, the parameters can be timevarying, and the least-squares estimator can be computed adaptively (recursively).
See, for example, the books [407, 299] for more information.

^

90

ESTIMATION THEORY

4.5 MAXIMUM LIKELIHOOD METHOD
Maximum likelihood (ML) estimator assumes that the unknown parameters  areconstants or there is no prior information available on them. The ML estimator has
several asymptotic optimality properties that make it a theoretically desirable choiceespecially when the number of samples is large. It has been applied to a wide variety
of problems in many application areas.
The maximum likelihood estimate ML of the parameter vector  is chosen to be
the value ML that maximizes the likelihood function (joint distribution)^^p(xT j ) = p(x(1); x(2); : : : ; x(T ) j )
(4.49)of the measurements x(1); x(2); : : : ; x(T ). The maximum likelihood estimator
corresponds to the value ^ML that makes the obtained measurements most likely.

Because many density functions contain an exponential function, it is often moreconvenient to deal with the log likelihood function ln p( T j  ). Clearly, the maximum likelihood estimator  ML also maximizes the log likelihood. The maximum
likelihood estimator is usually found from the solutions of the likelihood equationx^@ ln p(x j )
=0(4.50)T@
=^MLThe likelihood equation gives the values of  that maximize (or minimize) the

likelihood function. If the likelihood function is complicated, having several local
maxima and minima, one must choose the value  ML that corresponds to the absolute
maximum. Sometimes the maximum likelihood estimate can be found from theendpoints of the interval where the likelihood function is nonzero.
The construction of the likelihood function (4.49) can be very difficult if themeasurements depend on each other. Therefore, it is almost always assumed inapplying the ML method that the observations x(j ) are statistically independent ofeach other. Fortunately, this holds quite often in practice. Assuming independence,
the likelihood function decouples into the product^p(xT j ) =

T
Y
j =1

p(x(j ) j )

(4.51)

where p(x(i) j  ) is the conditional pdf of a single scalar measurement x(j ). Notethat taking the logarithm, the product (4.51) decouples to the sum of logarithms
j ln p(x(j ) j ).
The vector likelihood equation (4.50) consists of m scalar equations

P@ ln p(x j ^ )
= 0;
T ML
@i
=^MLi = 1; : : : ; m(4.52)

for the m parameter estimates ^i;ML , i = 1; : : : ; m. These equations are in general
coupled and nonlinear, so they can be solved only numerically except for simple

MAXIMUM LIKELIHOOD METHOD

91

cases. In several practical applications, the computational load of the maximum
likelihood method can be prohibitive, and one must resort to various approximations
for simplifying the likelihood equations or to some suboptimal estimation methods.
Example 4.6 Assume that we have T independent observations x(1); : : : ; x(T ) ofa scalar random variable x that is gaussian distributed with mean  and variance  2 .
Using (4.51), the likelihood function can be written

3
2
T
X
1
[x(j ) ? ] 5p(xT j ;  ) = (2 )?T= exp 4?
22

222 j=1

2(4.53)The log likelihood function becomesTX
ln p(xT j ; 2 ) = ? T2 ln(22 ) ? 21 2 [x(j ) ? ]2
j =1(4.54)

The first likelihood equation (4.52) isT@ ln p(x j ^ ; ^ 2 ) = 1 X[x(j ) ? ^ML ] = 0
T
ML
ML
2
@
^ML
j =1

(4.55)

Solving this yields for the maximum likelihood estimate of the mean  the samplemean^ML = T1

T
X
j =1

x(j )

(4.56)The second likelihood equation is obtained by differentiating the log likelihood (4.54)
with respect to the variance  2 :T@ ln p(x j ^ ; ^ 2 ) = ? T + 1 X[x(j ) ? ^ML ]2 = 0
T
ML
ML
2
4
@2
2^ML
2^ML
j =1
(4.57)
From this equation, we get for the maximum likelihood estimate of the variance  2
the sample variance
2
^ML
= T1

T
X
j =1

[x(j ) ? ^ML ]2(4.58)

This is a biased estimator of the true variance  2 , while the sample mean ^ML is an2
unbiased estimator of the mean . The bias of the variance estimator ^MLis dueto using the estimated mean ^ML instead of the true one in (4.58). This reduces
the amount of new information that is truly available for estimation by one sample.

92

ESTIMATION THEORY

Hence the unbiased estimator of the variance is given by (4.6). However, the bias ofthe estimator (4.58) is usually small, and it is asymptotically unbiased.
The maximum likelihood estimator is important because it provides estimates thathave certain very desirable theoretical properties. In the following, we list briefly themost important of them. Somewhat heuristic but illustrative proofs can be found in[407]. For more detailed analyses, see, e.g., [477].
1. If there exists an estimator that satisfies the Cramer-Rao lower bound (4.20) as
an equality, it can be determined using the maximum likelihood method.

^

2. The maximum likelihood estimator  ML is consistent.
3. The maximum likelihood estimator is asymptotically efficient. This meansthat it achieves asymptotically the Cramer-Rao lower bound for the estimation
error.
Example 4.7 Let us determine the Cramer-Rao lower bound (4.20) for the mean  ofa single gaussian random variable. From (4.55), the derivative of the log likelihood
function with respect to  isT
@ ln p(x j ; 2 ) = 1 XT@2 j=1 [x(j ) ? ]

(4.59)

Because we are now considering a single parameter  only, the Fisher information
matrix reduces to the scalar quantity

J = E

(
@



2

@ ln p(xT j ;  ) j ; 
2

)

2

9
82
3
>
>
T
=
< 1X
[x(j ) ? ]5 j ;  >= E >4 
;
: j
2

2

2

(4.60)=1

Since the samples x(j ) are assumed to be independent, all the cross covariance terms
vanish, and (4.60) simplifies to

J = 14TX
j =1

Ef[x(j ) ? ]2 j ;  2 g =T2 = T
4 2

(4.61)

Thus the Cramer-Rao lower bound (4.20) for the mean-square error of any unbiased^ of the mean of the gaussian density is
estimator 
Ef( ? ^)2 j g  J ?1 =

2T

(4.62)^ML of  is
In the previous example we found that the maximum likelihood estimator ^ML)2 g of the sample mean
the sample mean (4.56). The mean-square error E f( ? 

93

MAXIMUM LIKELIHOOD METHOD

was shown earlier in Example 4.3 to be  2 =T . Hence the sample mean satisfies the
Cramer-Rao inequality as an equation and is an efficient estimator for independent
gaussian measurements.
The expectation-maximization (EM) algorithm [419, 172, 298, 304] provides a
general iterative approach for computing maximum likelihood estimates. The main
advantage of the EM algorithm is that it often allows treatment of difficult maximum
likelihood problems suffering from multiple parameters and highly nonlinear likelihood functions in terms of simpler maximization problems. However, the applicationof the EM algorithm requires care in general because it can get stuck into a local
maximum or suffer from singularity problems [48]. In context with ICA methods,
the EM algorithm has been used for estimating unknown densities of source signals.
Any probability density function can be approximated using a mixture-of-gaussiansmodel [48]. A popular method for finding parameters of such a model is to use
the EM algorithm. This specific but important application of the EM algorithm is
discussed in detail in [48]. For a more detailed discussion of the EM algorithm, seereferences [419, 172, 298, 304].
The maximum likelihood method has a connection with the least-squares method.
Consider the nonlinear data model (4.47). Assuming that the parameters  are
unknown constants independent of the additive noise (error) T , the (conditional)
distribution p( T j ) of T is the same as the distribution of T at the point T =T ? ( ):

x

x

f

v
v

x

v

pxj (xT j ) = pv (xT ? f () j )

(4.63)v

If we further assume that the noise T is zero-mean and gaussian with the covariancematrix  2 , the preceding distribution becomesIp(xT j ) = exp ? 21 2 [xT ? f ()]T [x ? f ()]

(4.64)

where = (2 )?T=2  ?T is the normalizing term. Clearly, this is maximized whenthe exponent[xT ? f ()]T [xT ? f ()] = k xT ? f () k2(4.65)

is minimized, since is a constant independent of . But the exponent (4.65)
coincides with the nonlinear least-squares criterion (4.48). Hence if in the nonlinear
data model (4.47) the noise T is zero-mean, gaussian with the covariance matrixv = 2 , and independent of the unknown parameters , the maximum likelihood
estimator and the nonlinear least-squares estimator yield the same results.

C

I

v

94

ESTIMATION THEORY

4.6 BAYESIAN ESTIMATION *All the estimation methods discussed thus far in more detail, namely the moment, theleast-squares, and the maximum likelihood methods, assume that the parameters  are
unknown deterministic constants. In Bayesian estimation methods, the parameters
 are assumed to be random themselves. This randomness is modeled using the apriori probability density function p ( ) of the parameters. In Bayesian methods,
it is typically assumed that this a priori density is known. Taken strictly, this is avery demanding assumption. In practice we usually do not have such far-reaching
information on the parameters. However, assuming some useful form for the apriori density p () often allows the incorporation of useful prior information on
the parameters into the estimation process. For example, we may know which is
the most typical value of the parameter i and its typical range of variation. We canthen formulate this prior information for instance by assuming that i is gaussian
distributed with a mean mi and variance i2 . In this case the mean mi and variancei2 contain our prior knowledge about i (together with the gaussianity assumption).
The essence of Bayesian estimation methods is the posterior density pjx ( j T )of the parameters  given the data T . Basically, the posterior density contains all
the relevant information on the parameters . Choosing a specific estimate  forthe parameters  among the range of values of  where the posterior density is highor relatively high is somewhat arbitrary. The two most popular methods for doing
this are based on the mean-square error criterion and choosing the maximum of theposterior density. These are discussed in the following subsections.

x
^

x

4.6.1

Minimum mean-square error estimator for random parameters

In the minimum mean-square error method for random parameters , the optimal
estimator MSE is chosen by minimizing the mean-square error (MSE)^

EMSE = Efk  ? ^ k g
2(4.66)^

with respect to the estimator . The following theorem specifies the optimal estimator.

x

Theorem 4.2 Assume that the parameters  and the observations T have thejoint probability density function p;x ( ; T ). The minimum mean-square estimator
MSE of  is given by the conditional expectationx^^MSE = EfjxT g(4.67)The theorem can be proved by first noting that the mean-square error (4.66) can
be computed in two stages. First the expectation is evaluated with respect to  only,
and after this it is taken with respect to the measurement vector :x
EMSE = Efk  ? ^ k g = Ex fEfk  ? ^ k jxT gg
22(4.68)

95

BAYESIAN ESTIMATION *This expression shows that the minimization can be carried out by minimizing theconditional expectation

^

^T

^T ^

Efk  ?  k2 j T g =   ? 2 Efj T g + EfT j T g

x

x

x

(4.69)^The right-hand side is obtained by evaluating the squared norm and noting that  is afunction of the observations T only, so that it can be treated as a nonrandom vector
when computing the conditional expectation (4.69). The result (4.67) now follows
directly by computing the gradient 2 ? 2Efj T g of (4.69) with respect to  andequating it to zero.
The minimum mean-square estimator MSE is unbiased since

x

^

^

x

^
Ef^MSE g = Ex fEfjxT gg = Efg(4.70)The minimum mean-square estimator (4.67) is theoretically very significant because of its conceptual simplicity and generality. This result holds for all distributions
for which the joint distribution p;x ( ; ) exists, and remains unchanged if a weighting matrix
is added into the criterion (4.66) [407].
However, actual computation of the minimum mean-square estimator is often very
difficult. This is because in practice we only know or assume the prior distributionp () and the conditional distribution of the observations pxj ( j) given the parameters  . In constructing the optimal estimator (4.67), one must first compute theposterior density from Bayes’ formula (see Section 2.4)

x

W

x

p (x j)p ()pjx (jxT ) = xj p T(x ) (4.71)

x T

where the denominator is computed by integrating the numerator:

px(xT ) =Z1

?1pxj (xT j)p ()d(4.72)

The computation of the conditional expectation (4.67) then requires still anotherintegration. These integrals are usually impossible to evaluate at least analytically
except for special cases.
There are, however, two important special cases where the minimum mean-squareestimator MSE for random parameters  can be determined fairly easily. If theestimator  is constrained to be a linear function of the data:  =T , then it can beshown [407] that the optimal linear estimator LMSE minimizing the MSE criterion(4.66) is

^
^

^

^ Lx

(4.73)^LMSE = m + Cx C?x 1 (xT ? mx)
where m and mx are the mean vectors of  and xT , respectively, Cx is thecovariance matrix of xT , and Cx is the cross-covariance matrix of  and xT . The
error covariance matrix corresponding to the optimum linear estimator ^LMSE isEf( ? ^LMSE )( ? ^LMSE )T g = C ? Cx C?
(4.74)
x 1 Cx96

ESTIMATION THEORY

C

where  is the covariance matrix of the parameter vector  . We can conclude that
if the minimum mean-square estimator is constrained to be linear, it suffices to know
the first-order and second-order statistics of the data and the parameters , that is,their means and covariance matrices.
If the joint probability density p;x (; T ) of the parameters  and data T
is gaussian, the results (4.73) and (4.74) obtained by constraining the minimum
mean-square estimator to be linear are quite generally optimal. This is because theconditional density pjx (j T ) is also gaussian with the conditional mean (4.73) andcovariance matrix (4.74); see section 2.5. This again underlines the fact that for
the gaussian distribution, linear processing and knowledge of first and second order
statistics are usually sufficient to obtain optimal results.

x

x

x

x

4.6.2

Wiener filtering

In this subsection, we take a somewhat different signal processing viewpoint to thelinear minimum MSE estimation. Many estimation algorithms have in fact beendeveloped in context with various signal processing problems [299, 171].
Consider the following linear filtering problem. Let be an m-dimensional data
or input vector of the formzz = [z ; z ; : : : ; zm]T2

(4.75)

w = [w ; w ; : : : ; wm ]T(4.76)

1

and
1

2

an m-dimensional weight vector with adjustable weights (elements) wi, i = 1; : : : ; moperating linearly on so that the output of the filter isz

y = wT z(4.77)In Wiener filtering, the goal is to determine the linear filter (4.77) that minimizes
the mean-square error

EMSE = Ef(y ? d) g
2

(4.78)

between the desired response d and the output y of the filter. Inserting (4.77) into
(4.78) and evaluating the expectation yields

EMSE = wT Rz w ? 2wT rzd + Efd g(4.79)
Here Rz = EfzzT g is the data correlation matrix, and rzd = Efzdg is the crosscorrelation vector between the data vector z and the desired response d. Minimizing
the mean-square error (4.79) with respect to the weight vector w provides as the2

optimum solution the Wiener filter [168, 171, 419, 172]w^ MSE = R?z rzd
1(4.80)

BAYESIAN ESTIMATION *97

R

provided that z is nonsingular. This is almost always the case in practice due to thenoise and the statistical nature of the problem. The Wiener filter is usually computedby directly solving the linear normal equations

Rzw^ MSE = rzd(4.81)
In practice, the correlation matrix Rz and the cross-correlation vector rzd areusually unknown. They must then be replaced by their estimates, which can becomputed easily from the available finite data set. In fact the Wiener estimate thenbecomes a standard least-squares estimator (see exercises). In signal processing
applications, the correlation matrix z is often a Toeplitz matrix, since the data
vectors (i) consist of subsequent samples from a single signal or time series (seeSection 2.8). For this special case, various fast algorithms are available for solving
the normal equations efficiently [169, 171, 419].

R

z

4.6.3

Maximum a posteriori (MAP) estimator

Instead of minimizing the mean-square error (4.66) or some other performance index,
we can apply to Bayesian estimation the same principle as in the maximum likelihood
method. This leads to the maximum a posteriori (MAP) estimator MAP , which isdefined as the value of the parameter vector  that maximizes the posterior densitypjx (j T ) of  given the measurements T . The MAP estimator can be interpreted
as the most probable value of the parameter vector  for the available data T . The
principle behind the MAP estimator is intuitively well justified and appealing.
We have earlier noted that the posterior density can be computed from Bayes’
formula (4.71). Note that the denominator in (4.71) is the prior density px ( T ) of thedata T which does not depend on the parameter vector  , and merely normalizes
the posterior density pjx ( j T ). Hence for finding the MAP estimator it suffices tofind the value of  that maximizes the numerator of (4.71), which is the joint density

^

x

x

x

x

x

x

p;x (; xT ) = pxj (xT j)p ()(4.82)^

Quite similarly to the maximum likelihood method, the MAP estimator MAP canusually be found by solving the (logarithmic) likelihood equation. This now has the
form

@
@
@
@  ln p(; xT ) = @  ln p(xT j) + @  ln p() = 0(4.83)

where we have dropped the subscripts of the probability densities for notational
simplicity.
A comparison with the respective likelihood equation (4.50) for the maximum
likelihood method shows that these equations are otherwise the same, but the MAP
likelihood equation (4.83) contains an additional term @ (ln p())=@ , which takes
into account the prior information on the parameters . If the prior density p()is uniform for parameter values  for which p( T j ) is markedly greater than zero,then the MAP and maximum likelihood estimators become the same. In this case,x

98

ESTIMATION THEORY^

they are both obtained by finding the value  that maximizes the conditional densityp( T j). This is the case when there is no prior information about the parameters available. However, when the prior density p( ) is not uniform, the MAP and ML
estimators are usually different.

x

Example 4.8 Assume that we have T independent observations x(1); : : : ; x(T )from a scalar random quantity x that is gaussian distributed with mean x andvariance x2 . This time the mean x is itself a gaussian random variable having mean
zero and the variance 2 . We assume that both the variances x2 and 2 are known
and wish to estimate  using the MAP method.
Using the preceding information, it is straightforward to form the likelihood^MAP and solve it. The solution is (the derivation
equation for the MAP estimator is left as a exercise)T2 X^MAP = 2 +T2 x(j )x j =1(4.84)

The case in which we do not have any prior information on  can be modeled byletting 2 ! 1, reflecting our uncertainty about  [407]. Then clearly

^MAP ! T1

T
X
j =1

x(j )

(4.85)

so that the MAP estimator ^MAP tends to the sample mean. The same limiting value
is obtained if the number of samples T ! 1. This shows that the influence of theprior information, contained in the variance 2 , gradually decreases as the numberof the measurements increases. Hence asymptotically the MAP estimator coincides^ML which we found earlier in (4.56) to bewith the maximum likelihood estimator 
the sample mean (4.85).
Note also that if we are relatively confident about the prior value 0 of the mean, but the samples are very noisy so that x2 >> 2 , the MAP estimator (4.84) forsmall T stays close to the prior value 0 of , and the number T of samples mustgrow large until the MAP estimator approaches its limiting value (4.85). In contrast,
if 2 >> x2 , so that the samples are reliable compared to the prior information on
, the MAP estimator (4.84) rapidly approaches the sample mean (4.85). Thus the
MAP estimator (4.84) weights in a meaningful way the prior information and thesamples according to their relative reliability.
Roughly speaking, the MAP estimator is a compromise between the generalminimum mean-square error estimator (4.67) and the maximum likelihood estimator.
The MAP method has the advantage over the maximum likelihood method that it
takes into account the (possibly available) prior information about the parameters
, but it is computationally somewhat more difficult to determine because a second
term appears in the likelihood equation (4.83). On the other hand, both the ML
and MAP estimators are obtained from likelihood equations, avoiding the generally

CONCLUDING REMARKS AND REFERENCES

99

difficult integrations needed in computing the minimum mean-square estimator. If
the posterior distribution p( j T ) is symmetric around its peak value, the MAP
estimator and MSE estimator coincide.
There is no guarantee that the MAP estimator is unbiased. It is also generally
difficult to compute the covariance matrix of the estimation error for the MAP and
ML estimators. However, the MAP estimator is intuitively sensible, yields in mostcases good results in practice, and it has good asymptotic properties under appropriate
conditions. These desirable characteristics justify its use.

x

4.7 CONCLUDING REMARKS AND REFERENCES
In this chapter, we have dealt with basic concepts in estimation theory and the mostwidely used estimation methods. These include the maximum likelihood method,minimum mean-square error estimator, the maximum a posteriori method, and theleast-squares method for both linear and nonlinear data models. We have also pointed
out their interrelationships, and discussed the method of moments because of its
relationship to higher-order statistics. Somewhat different estimation methods must
be used depending on whether the parameters are considered to be deterministic, inwhich case the maximum likelihood method is the most common choice, or random,in which case Bayesian methods such as maximum a posteriori estimation can be
used.
Rigorous treatment of estimation theory requires a certain mathematical background as well as a good knowledge of probability and statistics, linear algebra, andmatrix differential calculus. The interested reader can find more information on estimation theory in several textbooks, including both mathematically [244, 407, 477]
and signal-processing oriented treatments [242, 299, 393, 419]. There are several
topics worth mentioning that we have not discussed in this introductory chapter.
These include dynamic estimation methods in which the parameters and/or the data
model are time dependent, for example, Kalman filtering [242, 299]. In this chapter,
we have derived several estimators by minimizing error criteria or maxim izing conditional probability distributions. Alternatively, optimal estimators can often be derived
from the orthogonality principle, which states that the estimator and its associated estimation error must be statistically orthogonal, having a zero cross-covariance matrix.

xx

From a theoretical viewpoint, the posterior density pjx ( j T ) contains all theinformation about the random parameters that the measurements T provide. Knowledge of the posterior density allows in principle the use of any suitable optimality
criterion for determining an estimator. Figure 4.2 shows an example of a hypotheticalposterior density p( j ) of a scalar parameter . Because of the asymmetricity ofthis density, different estimators yield different results. The minimum absolute error
estimator ^ABS minimizes the absolute error E fj  ? ^ jg. The choice of a specificestimator is somewhat arbitrary, since the true value of the parameter  is unknown,and can be anything within the range of the posterior density.

x

100

ESTIMATION THEORYp(^jx)^MAP

^ABS ^MMSE

^jx^MAP , minimum
Fig. 4.2 A posterior density p(), and the respective MAP estimate 
MSE estimate ^MSE , and the minimum absolute error estimate ^ABS .

Regrettably, it is generally difficult to determine the posterior distribution in a form
that allows for convenient mathematical analysis [407]. However, various advanced
and approximative techniques have been developed to facilitate Bayesian estimation;see [142]. When the number of measurements increases, the importance of priorinformation gradually decreases, and the maximum likelihood estimator becomes
asymptotically optimal.
Finally, we point out that neural networks provide in many instances a usefulpractical tool for nonlinear estimation, even though they lie outside the range ofclassic estimation theory. For example, the well-known back-propagation algorithm[48, 172, 376] is in fact a stochastic gradient algorithm for minimizing the meansquare error criterion

EMSE = Efk d ? f (; z) k g(4.86)
Here d is the desired response vector and z the (input) data vector. The parameters
 consist of weights that are adjusted so that the mapping error (4.86) is minimized.
The nonlinear function f ( ; z) has enough parameters and a flexible form, so that it2

can actually model with sufficient accuracy any regular nonlinear function. The backpropagation algorithm learns the parameters  that define the estimated input-output
mapping ( ; ). See [48, 172, 376] for details and applications.

f z

PROBLEMS

101

Problems
4.1 Show that:
4.1.1. the maximum likelihood estimator of the variance (4.58) becomes unbiased^ML is replaced in (4.58) by the true one .
if the estimated mean 
4.1.2. if the mean is estimated from the observations, one must use the formula(4.6) for getting an unbiased estimator.
4.2 Assume that ^1 and ^2 are unbiased estimators of the parameter  having
variances var(^1 ) = 12 , var(^2 ) = 22 .
4.2.1. Show that for any scalar 0   1, the estimator ^3 = ^1 + (1 ? )^2
is unbiased.
4.2.2. Determine the mean-square error of ^3 assuming that ^1 and ^2 are statistically independent.
4.2.3. Find the value of that minimizes this mean-square error.
4.3 Let the scalar random variable z be uniformly distributed on the interval [0; ).
There exist T independent samples z (1); : : : ; z (T ) from z . Using them, the estimate^ = max(z (i)) is constructed for the parameter .
4.3.1. Compute the probability density function of ^. (Hint: First construct the
cumulative distribution function.)
4.3.2. Is ^ unbiased or asymptotically unbiased?
4.3.3. What is the mean-square error E f(^ ? )2 j g of the estimate ^?
4.4 Assume that you know T independent observations of a scalar quantity thatis gaussian distributed with unknown mean  and variance  2 . Estimate  and  2using the method of moments.

4.5 Assume that x(1); x(2); : : : ; x(K ) are independent gaussian random variables
having all the mean 0 and variance x2 . Then the sum of their squares

y=

K
X
j =1

[x(j )]2

is 2 -distributed with the mean Kx2 and variance 2Kx4 . Estimate the parameters
K and x2 using the method of moments, assuming that there exist T measurementsy(1); y(2); : : : ; y(T ) on the sum of squares y.
4.6 Derive the normal equations (4.37) for the least-squares criterion (4.36). Justify
why these equations indeed provide the minimum of the criterion.

v
vv

0

4.7 Assume that the measurement errors have zero mean: E f T g = , and thatthe covariance matrix of the measurement errors is v = Ef T TT g. Consider theproperties of the least-squares estimator  LS in (4.38).
4.7.1. Show that the estimator  LS is unbiased.
4.7.2. Compute the error covariance matrix ~ defined in (4.19).
4.7.3. Compute ~ when v =  2 .

^

^

C

C

C

I

C

102

ESTIMATION THEORY

4.8 Consider line fitting using the linear least-squares method. Assume that you
know T measurements x(1); x(2); : : : ; x(T ) on the scalar quantity x made, respectively, at times (or argument values) t(1); t(2); : : : ; t(T ). The task is to fit the line

x = 0 + 1t
to these measurements.
4.8.1. Construct the normal equations for this problem using the standard linear
least-squares method.
4.8.2. Assume that the sampling interval t is constant and has been scaled sothat the measurement times are integers 1; 2; : : : ; T . Solve the normal equations inthis important special case.
4.9 * Consider the equivalence of the generalized least-squares and linear unbiased
minimum mean-square estimators. Show that
4.9.1. The optimal solution minimizing the generalized least-squares criterion(4.45) is^WLS = (HT WH)?1 HT WxT
4.9.2. An unbiased linear mean-square estimator ^MSE = LxT satisfies the con-

LH I

dition= .
4.9.3. The mean-square error can be written in the form

EMSE = Efk  ? ^ k j g = trace(LCv LT )2

4.9.4. Minimization of the preceding criterion EMSE under the constraint

I leads to the BLUE estimator (4.46).

LH =4.10 For a fixed amount of gas, the following connection holds between the pressure
P and the volume V :PV = c;where and c are constants. Assume that we know T pairs of measurements (Pi ; Vi ).
We want to estimate the parameters and c using the linear least-squares method.
Express the situation in the form of a matrix-vector model and explain how theestimates are computed (you need not compute the exact solution).
4.11

Let the probability density function of a scalar-valued random variable z bep(z j ) = 2 ze?z ;z  0;  > 0

Determine the maximum likelihood estimate of the parameter . There are available
T independent measurements z (1); : : : ; z (T ) on z .
4.12 In a signal processing application five sensors placed mutually according toa cross pattern yield, respectively, the measurements x0 , x1 , x2 , x3 , and x4 , that canbe collected to the measurement vector . The measurements are quantized with 7
bits accuracy so that their values are integers in the interval 0; : : : ; 127. The jointx

PROBLEMS

103xdensity p( j ) of the measurements is a multinomial density that depends on theunknown parameter  as follows:p(x j ) = k(x)(1=2)x0 (=4)x1 (1=4 ? =4)x2 (1=4 ? =4)x3 (=4)x4
where the scaling termk(x) = (x0 +xx1!x+!xx2 !+x x!x3 +! x4 )!
0

1

2

3

4

Determine the maximum likelihood estimate of the parameter  in terms of themeasurement vector . (Here, you can here treat the individual measurements in asimilar manner as mutually independent scalar measurements.)x

4.13 Consider the sum z = x1 + x2 + : : : + xK , where the scalar random variables xi
are statistically independent and gaussian, each having the same mean 0 and variancex2 .
4.13.1. Construct the maximum likelihood estimate for the number K of theterms in the sum.
4.13.2. Is this estimate unbiased?
4.14 * Consider direct evaluation of the Wiener filter.
4.14.1. Show that the mean-square filtering error (4.78) can be evaluated to theform (4.79).
4.14.2. What is the minimum mean-square error given by the Wiener estimate?
4.15 The random variables x1 , x2 , and a third, related random variable y are jointly
distributed. Define the random vectorz = [y; x ; x ]T
It is known that z has the mean vector mz and the covariance matrix Cz given by1

2 1=4 3mz = 4 1=2 5 ;
1=2

2

2

3

7 1 1
Cz = 101 4 1 3 ?1 5
1 ?1 3

Find the optimum linear mean-square estimate of y based on x1 and x2 .

z

z

z

4.16 * Assume that you know T data vectors (1); (2); : : : ; (T ) and their corresponding desired responses d(1); d(2); : : : ; d(T ). Standard estimates of the correlation matrix and the cross-correlation vector needed in Wiener filtering are [172]R^ z = 1tTX
i=1z(i)z(i)T ;TX^rzd = 1 z(i)d(i)T
i=1(4.87)

4.16.1. Express the estimates (4.87) in matrix form and show that when they areused in the Wiener filter (4.80) instead of the true values, the filter coincides with aleast-squares solution.

104

ESTIMATION THEORY

4.16.2. What is the discrete data model corresponding to this least-squares estimator?
4.17* The joint density function of the random variables x and y is given by

pxy (x; y) = 8xy; 0  y  x  1;
and pxy (x; y ) = 0 outside the region defined above.
4.17.1. Find and sketch the conditional density pyjx (y j x).
4.17.2. Compute the MAP (maximum a posteriori) estimate of y .
4.17.3. Compute the optimal mean-square error estimate of y .

4.18 * Suppose that a scalar random variable y is of the form y = z + v , where
the pdf of v is pv (t) = t=2 on the interval [0; 2], and the pdf of z is pz (t) = 2t on
the interval [0; 1]. Both the densities are zero elsewhere. There is available a single
measurement value y = 2:5.
4.18.1. Compute the maximum likelihood estimate of y .
4.18.2. Compute the MAP (maximum a posteriori) estimate of y .
4.18.3. Compute the minimum mean-square estimate of y .
4.19 * Consider the MAP estimator (4.84) of the mean .
4.19.1. Derive the estimator.
4.19.2. Express the estimator in recursive form.
Computer assignments
4.1 Choose a suitable set of two-dimensional data. Plenty of real-world data canbe found for example using the links of the WWW page of this book, as well as in[376] and at the following Web sites:
http://ferret.wrc.noaa.gov/
http://www.ics.uci.edu/ mlearn/MLSummary.html
4.1.1. Plot the data (or part of it, if the data set is large).
4.1.2. Based on the plot, choose a suitable function (which is linear with respect
to the parameters), and fit it to your data using the standard least-squares method.
(Alternatively, you can use nonlinear least-squares method if the parameters of thechosen function depend nonlinearly on the data.)
4.1.3. Plot the fitted curve and the fitting error. Assess the quality of your leastsquares model.
4.2 * Use the Bayesian linear minimum mean-square estimator for predicting ascalar measurement from other measurements.
4.2.1. Choose first a suitable data set in which the components of the data vectors
are correlated (see the previous computer assignment for finding data).
4.2.2. Compute the linear minimum mean-square estimator.
4.2.3. Compute the variance of the measurement that you have predicted andcompare it with your minimum mean-square estimation (prediction) error.

5Information Theory

Estimation theory gives one approach to characterizing random variables. This was
based on building parametric models and describing the data by the parameters.
An alternative approach is given by information theory. Here the emphasis is oncoding. We want to code the observations. The observations can then be stored
in the memory of a computer, or transmitted by a communications channel, forexample. Finding a suitable code depends on the statistical properties of the data.
In independent component analysis (ICA), estimation theory and information theory
offer the two principal theoretical approaches.
In this chapter, the basic concepts of information theory are introduced. The latter
half of the chapter deals with a more specialized topic: approximation of entropy.
These concepts are needed in the ICA methods of Part II.

5.1 ENTROPY
5.1.1

Definition of entropy

Entropy is the basic concept of information theory. Entropy H is defined for a
discrete-valued random variable X as

H (X ) = ?

Xi

P (X = ai ) log P (X = ai )(5.1)

where the ai are the possible values of X . Depending on what the base of thelogarithm is, different units of entropy are obtained. Usually, the logarithm withbase 2 is used, in which case the unit is called a bit. In the following the base is
105

106

INFORMATION THEORY

Fig. 5.1 The function f in (5.2), plotted on the interval [0; 1].

not important since it only changes the measurement scale, so it is not explicitly
mentioned.
Let us define the function f as

f (p) = ?p log p;

for 0  p  1(5.2)

This is a nonnegative function that is zero for p = 0 and for p = 1, and positive forvalues in between; it is plotted in Fig. 5.1. Using this function, entropy can be writtenas

H (X ) =Xif (P (X = ai ))(5.3)

Considering the shape of f , we see that the entropy is small if the probabilities
P (X = ai ) are close to 0 or 1, and large if the probabilities are in between.
In fact, the entropy of a random variable can be interpreted as the degree of
information that the observation of the variable gives. The more “random”, i.e.,
unpredictable and unstructured the variable is, the larger its entropy. Assume that theprobabilities are all close to 0, expect for one that is close to 1 (the probabilities mustsum up to one). Then there is little randomness in the variable, since it almost always
takes the same value. This is reflected in its small entropy. On the other hand, if all
the probabilities are equal, then they are relatively far from 0 and 1, and f takes large
values. This means that the entropy is large, which reflects the fact that the variable
is really random: We cannot predict which value it takes.
Example 5.1 Let us consider a random variable X that can have only two values, aand b. Denote by p the probability that it has the value a, then the probability that itis b is equal to 1 ? p. The entropy of this random variable can be computed as

H (X ) = f (p) + f (1 ? p)

(5.4)

ENTROPY

107

Thus, entropy is a simple function of p. (It does not depend on the values a and b.)
Clearly, this function has the same properties as f : it is a nonnegative function thatis zero for p = 0 and for p = 1, and positive for values in between. In fact, it it ismaximized for p = 1=2 (this is left as an exercice). Thus, the entropy is largest whenthe values are both obtained with a probability of 50%. In contrast, if one of thesevalues is obtained almost always (say, with a probability of 99:9%), the entropy of
X is small, since there is little randomness in the variable.
5.1.2

Entropy and coding length

The connection between entropy and randomness can be made more rigorous byconsidering coding length. Assume that we want to find a binary code for a large
number of observations of X , so that the code uses the minimum number of bits
possible. According to the fundamental results of information theory, entropy isvery closely related to the length of the code required. Under some simplifying
assumptions, the length of the shortest code is bounded below by the entropy, andthis bound can be approached arbitrarily close, see, e.g., [97]. So, entropy gives
roughly the average minimum code length of the random variable.
Since this topic is out of the scope of this book, we will just illustrate it with two
examples.
Example 5.2 Consider again the case of a random variable with two possible values,a and b. If the variable almost always takes the same value, its entropy is small. Thisis reflected in the fact that the variable is easy to code. In fact, assume the value ais almost always obtained. Then, one efficient code might be obtained simply bycounting how many a’s are found between two subsequent observations of b, andwriting down these numbers. If we need to code only a few numbers, we are able tocode the data very efficiently.
In the extreme case where the probability of a is 1, there is actually nothing left tocode and the coding length is zero. On the other hand, if both values have the sameprobability, this trick cannot be used to obtain an efficient coding mechanism, andevery value must be coded separately by one bit.
Example 5.3 Consider a random variable X that can have eight different values
with probabilities (1=2; 1=4; 1=8; 1=16; 1=64; 1=64; 1=64; 1=64). The entropy of X
is 2 bits (this computation is left as an exercice to the reader). If we just coded thedata in the ordinary way, we would need 3 bits for every observation. But a more
intelligent way is to code frequent values with short binary strings and infrequentvalues with longer strings. Here, we could use the following strings for the outcomes:
0,10,110,1110,111100,111101,111110,111111. (Note that the strings can be written
one after another with no spaces since they are designed so that one always knows
when the string ends.) With this encoding the average number of bits needed foreach outcome is only 2, which is in fact equal to the entropy. So we have gained a
33% reduction of coding length.

108

INFORMATION THEORY

5.1.3

Differential entropy

The definition of entropy for a discrete-valued random variable can be generalized
for continuous-valued random variables and vectors, in which case it is often called
differential entropy.
The differential entropy H of a random variable x with density px (:) is defined
as:

Z

Z

H (x) = ? px( ) log px( )d = f (px( ))d

(5.5)

Differential entropy can be interpreted as a measure of randomness in the same way
as entropy. If the random variable is concentrated on certain small intervals, itsdifferential entropy is small.
Note that differential entropy can be negative. Ordinary entropy cannot be negative
because the function f in (5.2) is nonnegative in the interval [0; 1], and discrete
probabilities necessarily stay in this interval. But probability densities can be larger
than 1, in which case f takes negative values. So, when we speak of a “small
differential entropy”, it may be negative and have a large absolute value.
It is now easy to see what kind of random variables have small entropies. They
are the ones whose probability densities take large values, since these give strong
negative contributions to the integral in (5.8). This means that certain intervals arequite probable. Thus we again find that entropy is small when the variable is not very
random, that is, it is contained in some limited intervals with high probabilities.
Example 5.4 Consider a random variable x that has a uniform probability distribution in the interval [0; a]. Its density is given by(px ( ) = 1=a; for 0    a0; otherwise(5.6)

The differential entropy can be evaluated as

H (x) = ?

Za1

1

(5.7)
log a d = log a
0 aThus we see that the entropy is large if a is large, and small if a is small. This isnatural because the smaller a is, the less randomness there is in x. In the limit wherea goes to 0, differential entropy goes to ?1, because in the limit, x is no longer
random at all: it is always 0.

The interpretation of entropy as coding length is more or less valid with differential entropy. The situation is more complicated, however, since the coding lengthinterpretation requires that we discretize (quantize) the values of x. In this case, thecoding length depends on the discretization, i.e., on the accuracy with which we want
to represent the random variable. Thus the actual coding length is given by the sumof entropy and a function of the accuracy of representation. We will not go into thedetails here; see [97] for more information.

109

ENTROPY

The definition of differential entropy can be straightforwardly generalized to themultidimensional case. Let be a random vector with density px (:). The differential
entropy is then defined as:

x

Z

Z

H (x) = ? px() log px ()d = f (px())d
5.1.4

(5.8)

Entropy of a transformationx

Consider an invertible transformation of the random vector , say

y = f (x)

(5.9)

y

x
f

In this section, we show the connection between the entropy of and that of .
A short, if somewhat sloppy derivation is as follows. (A more rigorous derivation
is given in the Appendix.) Denote by J ( ) the Jacobian matrix of the function , i.e.,the matrix of the partial derivatives of at point  . The classic relation between thedensity py of and the density px of , as given in Eq. (2.82), can then be formulated
as

y

x

f
f

py () = px (f ?1 ( ))j det J f (f ?1 ())j?1(5.10)

Now, expressing the entropy as an expectation

H (y) = ?E flog py (y)g(5.11)

we getE flog py (y)g = E flog[px(f ?1 (y))j det J f (f ?1 (y))j?1 ]g= E flog[px (x)j det J f (x)j?1 ]g = E flog px (x)g ? E flog j det J f (x)jg (5.12)
Thus we obtain the relation between the entropies as

H (y) = H (x) + E flog j det J f (x)jg(5.13)

fx

In other words, the entropy is increased in the transformation by E flog j det J ( )jg.
An important special case is the linear transformationy = Mx(5.14)

H (y) = H (x) + log j det Mj(5.15)

in which case we obtain

This also shows that differential entropy is not scale-invariant. Consider a randomvariable x. If we multiply it by a scalar constant, , differential entropy changes as

H ( x) = H (x) + log j j(5.16)

Thus, just by changing the scale, we can change the differential entropy. This is why
the scale of x often is fixed before measuring its differential entropy.

110

INFORMATION THEORY

5.2 MUTUAL INFORMATION
5.2.1

Definition using entropy

Mutual information is a measure of the information that members of a set of randomvariables have on the other random variables in the set. Using entropy, we can define
the mutual information I between n (scalar) random variables, xi ; i = 1; :::; n, as
follows

I (x1 ; x2 ; :::; xn ) =

x

n
X
i=1

H (xi ) ? H (x)(5.17)

where is the vector containing all the xi .
Mutual information can be interpreted by using the interpretation of entropy ascode length. The terms H (xi ) give the lengths of codes for the xi when these arecoded separately, and H ( ) gives the code length when is coded as a random
vector, i.e., all the components are coded in the same code. Mutual information thus
shows what code length reduction is obtained by coding the whole vector insteadof the separate components. In general, better codes can be obtained by coding thewhole vector. However, if the xi are independent, they give no information on eachother, and one could just as well code the variables separately without increasingcode length.

x

5.2.2

x

Definition using Kullback-Leibler divergence

Alternatively, mutual information can be interpreted as a distance, using what iscalled the Kullback-Leibler divergence. This is defined between two n-dimensional
probability density functions (pdf’s) p1 and p2 asZ1
(p1 ; p2 ) = p1 () log pp2 (()) d(5.18)The Kullback-Leibler divergence can be considered as a kind of a distance between
the two probability densities, because it is always nonnegative, and zero if and only if
the two distributions are equal. This is a direct consequence of the (strict) convexityof the negative logarithm, and the application of the classic Jensen’s inequality.
Jensen’s inequality (see [97]) says that for any strictly convex function f and any
random variable y , we have

E ff (y)g  f (E fyg)
(5.19)Take f (y ) = ? log(y ), and assume that y = p2 (x)=p1 (x) where x has the distribution
given by p1 . Then we haveZ
2
2
(p1 ; p2 ) = E ff (y)g = E f? log pp1 ((xx)) g = p1 ()f? log pp1 (()) gd
ZZ2 f (E fyg) = ? log p1 ()f pp1 (()) gd = ? log p2 ()d = 0 (5.20)

MAXIMUM ENTROPY

111

Moreover, we have equality in Jensen’s inequality if and only if y is constant. In ourcase, it is constant if and only if the two distributions are equal, so we have proventhe announced property of the Kullback-Leibler divergence.
Kullback-Leibler divergence is not a proper distance measure, though, because it
is not symmetric.
To apply Kullback-Leibler divergence here, let us begin by considering that if
random variables xi were independent, their joint probability density could be factorized according to the definition of independence. Thus one might measure theindependence of the xi as the Kullback-Leibler divergence between the real density
p1 = px() and the factorized density p2 = p1 (1 )p2 (2 ):::pn (n ), where the pi (:)
are the marginal densities of the xi . In fact, simple algebraic manipulations show that
this quantity equals the mutual information that we defined using entropy in (5.17),
which is left as an exercice.
The interpretation as Kullback-Leibler divergence implies the following important
property: Mutual information is always nonnegative, and it is zero if and only if thevariables are independent. This is a direct consequence of the properties of the
Kullback-Leibler divergence.

5.3 MAXIMUM ENTROPY
5.3.1

Maximum entropy distributions

An important class of methods that have application in many domains is given by the
maximum entropy methods. These methods apply the concept of entropy to the taskof regularization.
Assume that the information available on the density px (:) of the scalar randomvariable x is of the formZp( )F i ( )d = ci ; for i = 1; :::; m(5.21)

which means in practice that we have estimated the expectations E fF i (x)g of mdifferent functions F i of x. (Note that i is here an index, not an exponent.)
The question is now: What is the probability density function p0 that satisfies theconstraints in (5.21), and has maximum entropy among such densities? (Earlier, we
defined the entropy of random variable, but the definition can be used with pdf’s aswell.) This question can be motivated by noting that a finite number of observations
cannot tell us exactly what p is like. So we might use some kind of regularization to
obtain the most useful p compatible with these measurements. Entropy can be here
considered as a regularization measure that helps us find the least structured density
compatible with the measurements. In other words, the maximum entropy density
can be interpreted as the density that is compatible with the measurements and makes
the minimum number of assumptions on the data. This is because entropy can beinterpreted as a measure of randomness, and therefore the maximum entropy density

112

INFORMATION THEORY

is the most random of all the pdf’s that satisfy the constraints. For further details on
why entropy can be used as a measure of regularity, see [97, 353].
The basic result of the maximum entropy method (see, e.g. [97, 353]) tells us that
under some regularity conditions, the density p0 ( ) which satisfies the constraints(5.21) and has maximum entropy among all such densities, is of the form

p0 ( ) = A exp(

Xiai F i ( ))(5.22)

Here, A and ai are constants that are determined from the ci , using the constraintsin (5.21) (i.e., by substituting the right-hand side of (5.22) for p in (5.21)), and theconstraint p0 ( )d = 1. This leads in general to a system of n + 1 nonlinear
equations that may be difficult to solve, and in general, numerical methods must be
used.

R

5.3.2

Maximality property of the gaussian distribution

Now, consider the set of random variables that can take all the values on the real line,
and have zero mean and a fixed variance, say 1 (thus, we have two constraints). The
maximum entropy distribution for such variables is the gaussian distribution. This isbecause by (5.22), the distribution has the form

p0 ( ) = A exp(a1  2 + a2  )(5.23)

and all probability densities of this form are gaussian by definition (see Section 2.5).
Thus we have the fundamental result that a gaussian variable has the largest
entropy among all random variables of unit variance. This means that entropy
could be used as a measure of nongaussianity. In fact, this shows that the gaussian
distribution is the “most random” or the least structured of all distributions. Entropy
is small for distributions that are clearly concentrated on certain values, i.e., when thevariable is clearly clustered, or has a pdf that is very “spiky”. This property can begeneralized to arbitrary variances, and what is more important, to multidimensional
spaces: The gaussian distribution has maximum entropy among all distributions witha given covariance matrix.

5.4 NEGENTROPY
The maximality property given in Section 5.3.2 shows that entropy could be used todefine a measure of nongaussianity. A measure that is zero for a gaussian variable andalways nonnegative can be simply obtained from differential entropy, and is callednegentropy. Negentropy J is defined as follows

J (x) = H (xgauss ) ? H (x)(5.24)APPROXIMATION OF ENTROPY BY CUMULANTS

x

where gauss is a gaussian random vector of the same covariance matrix
entropy can be evaluated as

113

 as x. Its

H (xgauss ) = 12 log j det j + n2 [1 + log 2](5.25)

xwhere n is the dimension of .
Due to the previously mentioned maximality property of the gaussian distribution,negentropy is always nonnegative. Moreover, it is zero if and only if has a gaussian
distribution, since the maximum entropy distribution is unique.
Negentropy has the additional interesting property that it is invariant for invertibleT,
we have E f T g =linear transformations. This is because for =and, using preceding results, the negentropy can be computed as

x

y Mx

yy

MM

J (Mx) = 12 log j det(MMT )j + n2 [1 + log 2] ? (H (x) + log jMj)
= 1 log j det j + 2 1 log j det Mj + n [1 + log 2] ? H (x) ? log j det Mj
2

2
2
1
n
= 2 log j det j + 2 [1 + log 2] ? H (x)= H (xgauss ) ? H (x) = J (x)(5.26)

In particular negentropy is scale-invariant, i.e., multiplication of a random variable bya constant does not change its negentropy. This was not true for differential entropy,
as we saw earlier.

5.5 APPROXIMATION OF ENTROPY BY CUMULANTS
In the previous section we saw that negentropy is a principled measure of nongaussianity. The problem in using negentropy is, however, that it is computationally very
difficult. To use differential entropy or negentropy in practice, we could compute theintegral in the definition in (5.8). This is, however, quite difficult since the integral
involves the probability density function. The density could be estimated using basic density estimation methods such as kernel estimators. Such a simple approach
would be very error prone, however, because the estimator would depend on the correct choice of the kernel parameters. Moreover, it would be computationally rather
complicated.
Therefore, differential entropy and negentropy remain mainly theoretical quantities. In practice, some approximations, possibly rather coarse, have to be used. In
this section and the next one we discuss different approximations of negentropy thatwill be used in the ICA methods in Part II of this book.
5.5.1

Polynomial density expansions

The classic method of approximating negentropy is using higher-order cumulants(defined in Section 2.7). These are based on the idea of using an expansion not unlike

114

INFORMATION THEORYa Taylor expansion. This expansion is taken for the pdf of a random variable, say x,
in the vicinity of the gaussian density. (We only consider the case of scalar randomvariables here, because it seems to be sufficient in most applications.) For simplicity,
let us first make x zero-mean and of unit variance. Then, we can make the technicalassumption that the density px ( ) of x is near the standardized gaussian densityp'( ) = exp(? 2 =2)= 2

(5.27)

Two expansions are usually used in this context: the Gram-Charlier expansion
and the Edgeworth expansion. They lead to very similar approximations, so weonly consider the Gram-Charlier expansion here. These expansions use the so-called
Chebyshev-Hermite polynomials, denoted by Hi where the index i is a nonnegative
integer. These polynomials are defined by the derivatives of the standardized gaussian
pdf '( ) by the equation@ i '( ) = (?1)i H ( )'( )(5.28)i@ i
Thus, Hi is a polynomial of order i. These polynomials have the nice property offorming an orthonormal system in the following sense:(

Z(5.29)'( )Hi ( )Hj ( )d = 1; if i = j
0; if i 6= j
The Gram-Charlier expansion of the pdf of x, truncated to include the two firstnonconstant terms, is then given bypx( )  p^x( ) = '( )(1 + 3 (x) H33!( ) + 4 (x) H44!( ) )(5.30)

This expansion is based on the idea that the pdf of x is very close to a gaussianone, which allows a Taylor-like approximation to be made. Thus, the nongaussianpart of the pdf is directly given by the higher-order cumulants, in this case the thirdand fourth-order cumulants. Recall that these are called the skewness and kurtosis,
and are given by 3 (x) = E fx3 g and 4 (x) = E fx4 g ? 3. The expansion has an
infinite number of terms, but only those given above are of interest to us. Note thatthe expansion starts directly from higher-order cumulants, because we standardized
x to have zero mean and unit variance.
5.5.2

Using density expansions for entropy approximation

Now we could plug the density in (5.30) into the definition of entropy, to obtainZ

H (x)  ? p^x ( ) log p^x ( )d

(5.31)

This integral is not very simple to evaluate, though. But again using the idea thatthe pdf is very close to a gaussian one, we see that the cumulants in (5.30) are very

APPROXIMATION OF ENTROPY BY NONPOLYNOMIAL FUNCTIONS

115

small, and thus we can use the simple approximationlog(1 + )   ? 2 =2(5.32)

which givesZ
H (x)  ? '( )(1 + 3 (x) H33!( ) + 4 (x) H44!( ) )[log '( ) +  (x) H3 ( ) +  (x) H4 ( ) ? ( (x) H3 ( ) +  (x) H4 ( ) )2 =2]
3

3!

4

4!

3

3!

4

4!

(5.33)
This expression can be simplified (see exercices). Straightforward algebraic manipulations then giveZ2)2
H (x)  ? '( ) log '( )d ? 23(x3!) ? 24(4!

(5.34)

Thus we finally obtain an approximation of the negentropy of a standardized randomvariable as

1 E fx3 g2 + 1 kurt(x)2
J (x)  12
48

(5.35)

This gives a computationally very simple approximation of the nongaussianity measured by negentropy.

5.6 APPROXIMATION OF ENTROPY BY NONPOLYNOMIALFUNCTIONSIn the previous section, we introduced cumulant-based approximations of(neg)entropy. However, such cumulant-based methods sometimes provide a rather
poor approximation of entropy. There are two main reasons for this. First, finitesample estimators of higher-order cumulants are highly sensitive to outliers: theirvalues may depend on only a few, possibly erroneous, observations with large values.
This means that outliers may completely determine the estimates of cumulants, thus
making them useless. Second, even if the cumulants were estimated perfectly, theymainly measure the tails of the distribution, and are largely unaffected by structurenear the center of the distribution. This is because expectations of polynomials likethe fourth power are much more strongly affected by data far away from zero thanby data close to zero.
In this section, we introduce entropy approximations that are based on an approximative maximum entropy method. The motivation for this approach is that theentropy of a distribution cannot be determined from a given finite number of estimated expectations as in (5.21), even if these were estimated exactly. As explained in116

INFORMATION THEORY

Section 5.3, there exist an infinite number of distributions for which the constraintsin (5.21) are fulfilled, but whose entropies are very different from each other. In
particular, the differential entropy reaches ?1 in the limit where x takes only a finite
number of values.
A simple solution to this is the maximum entropy method. This means that wecompute the maximum entropy that is compatible with our constraints or measurements in (5.21), which is a well-defined problem. This maximum entropy, or further
approximations thereof, can then be used as a meaningful approximation of the entropy of a random variable. This is because in ICA we usually want to minimize
entropy. The maximum entropy method gives an upper bound for entropy, and its
minimization is likely to minimize the true entropy as well.
In this section, we first derive a first-order approximation of the maximum entropy
density for a continuous one-dimensional random variable, given a number of simple
constraints. This results in a density expansion that is somewhat similar to theclassic polynomial density expansions by Gram-Charlier and Edgeworth. Using thisapproximation of density, an approximation of 1-D differential entropy is derived.
The approximation of entropy is both more exact and more robust against outliers
than the approximations based on the polynomial density expansions, without being
computationally more expensive.
5.6.1

Approximating the maximum entropy

Let us thus assume that we have observed (or, in practice, estimated) a number of
expectations of x, of the formZp( )F i ( )d = ci ; for i = 1; :::; m(5.36)

The functions F i are not, in general, polynomials. In fact, if we used simple
polynomials, we would end up with something very similar to what we had in thepreceding section.
Since in general the maximum entropy equations cannot be solved analytically,
we make a simple approximation of the maximum entropy density p0 . This is basedon the assumption that the density p( ) is not very far from the gaussian densityof the same mean and variance; this assumption is similar to the one made using
polynomial density expansions.
As with the polynomial expansions, we can assume that x has zero mean and unit
variance. Therefore we put two additional constraints in (5.36), defined by

F n+1 ( ) = ; cn+1 = 0
F n+2 ( ) =  2 ; cn+2 = 1(5.37)(5.38)

To further simplify the calculations, let us make another, purely technical assumption:
The functions F i ; i = 1; :::; n, form an orthonormal system according to the metricdefined by ' in (5.27), and are orthogonal to all polynomials of second degree. In

APPROXIMATION OF ENTROPY BY NONPOLYNOMIAL FUNCTIONS

other words, for all i; j = 1; :::; nZ('( )F i ( )F j ( )d = 1;0;Zif i = jif i 6= j'( )F i ( ) k d = 0; for k = 0; 1; 2

117

(5.39)(5.40)

Again, these orthogonality constraints are very similar to those of Chebyshev-Hermite
polynomials. For any set of linearly independent functions F i (not containing secondorder polynomials), this assumption can always be made true by ordinary GramSchmidt orthonormalization.
Now, note that the assumption of near-gaussianity implies that all the other ai in(5.22) are very small compared to an+2  ?1=2, since the exponential in (5.22)
is not far from exp(? 2 =2). Thus we can make a first-order approximation of the
exponential function (detailed derivations can be found in the Appendix). This allows
for simple solutions for the constants in (5.22), and we obtain the approximative
maximum entropy density, which we denote by p^( ):p^( ) = '( )(1 +

n
X
i=1

ci F i ( ))(5.41)

where ci = E fF i ( )g.
Now we can derive an approximation of differential entropy using this density
approximation. As with the polynomial density expansions, we can use (5.31) and(5.32). After some algebraic manipulations (see the Appendix), we obtain

J (x)  12

n
X
i=1

E fF i (x)g2(5.42)

Note that even in cases where this approximation is not very accurate, (5.42) canbe used to construct a measure of nongaussianity that is consistent in the sense that
(5.42) obtains its minimum value, 0, when x has a gaussian distribution. This isbecause according to the latter part of (5.39) with k = 0, we have E fF i ( )g = 0.
5.6.2

Choosing the nonpolynomial functions

Now it remains to choose the “measuring” functions F i that define the information
given in (5.36). As noted in Section 5.6.1, one can take practically any set of
linearly independent functions, say Gi ; i = 1; :::; m, and then apply Gram-Schmidt
orthonormalization on the set containing those functions and the monomials  k ; k =0; 1; 2, so as to obtain the set F i that fulfills the orthogonality assumptions in (5.39).
This can be done, in general, by numerical integration. In the practical choice ofthe functions Gi , the following criteria must be emphasized:
1. The practical estimation of E fGi (x)g should not be statistically difficult. In
particular, this estimation should not be too sensitive to outliers.

118

INFORMATION THEORY

2. The maximum entropy method assumes that the function p0 in (5.22) is integrable. Therefore, to ensure that the maximum entropy distribution exists inthe first place, the Gi (x) must not grow faster than quadratically as a functionof jxj, because a function growing faster might lead to the nonintegrability of
p0 .

3. The Gi must capture aspects of the distribution of X that are pertinent in thecomputation of entropy. In particular, if the density p( ) were known, theoptimal function Gopt would clearly be ? log p( ), because ?E flog p(x)g
gives the entropy directly Thus, one might use for Gi the log-densities of someknown important densities.

The first two criteria are met if the Gi (x) are functions that do not grow too fast
(not faster than quadratically) as jxj increases. This excludes, for example, the useof higher-order polynomials, which are used in the Gram-Charlier and Edgeworth
expansions. One might then search, according to criterion 3, for log-densities ofsome well-known distributions that also fulfill the first two conditions. Examples
will be given in the next subsection.
It should be noted, however, that the criteria above only delimit the space offunctions that can be used. Our framework enables the use of very different functions(or just one) as Gi . However, if prior knowledge is available on the distributions
whose entropy is to be estimated, criterion 3 shows how to choose the optimalfunction.
5.6.3

Simple special cases

A simple special case of (5.41) is obtained if one uses two functions G1 and G2 ,which are chosen so that G1 is odd and G2 is even. Such a system of two functions canmeasure the two most important features of nongaussian 1-D distributions. The odd
function measures the asymmetry, and the even function measures the dimension ofbimodality vs. peak at zero, closely related to sub- vs. supergaussianity. Classically,
these features have been measured by skewness and kurtosis, which correspond to
G1 (x) = x3 and G2 (x) = x4 , but we do not use these functions for the reasons
explained in Section 5.6.2. (In fact, with these choices, the approximation in (5.41)
becomes identical to the one obtained from the Gram-Charlier expansion in (5.35).)
In this special case, the approximation in (5.42) simplifies toJ (x)  k1 (E fG1 (x)g)2 + k2 (E fG2 (x)g ? E fG2 ( )g)2
(5.43)
where k1 and k2 are positive constants (see the Appendix). Practical examplesof choices of Gi that are consistent with the requirements in Section 5.6.2 are thefollowing. First, for measuring bimodality/sparsity, one might use, according to therecommendations of Section 5.6.2, the log-density of the Laplacian distribution:

G2a (x) = jxj(5.44)For computational reasons, a smoother version of G2a might also be used. Anotherchoice would be the gaussian function, which can be considered as the log-density

119

APPROXIMATION OF ENTROPY BY NONPOLYNOMIAL FUNCTIONS

of a distribution with infinitely heavy tails (since it stays constant when going to
infinity):G2b (x) = exp(?x2 =2)(5.45)For measuring asymmetry, one might use, on more heuristic grounds, the following
function:G1 (x) = x exp(?x2 =2)(5.46)

that is smooth and robust against outliers.
Using the preceding examples one obtains two practical examples of (5.43):p

Ja (x) = k1 (E fx exp(?x2 =2)g)2 + k2a (E fjxjg ? 2=)2
and(5.47)p

Jb (x) = k1 (E fx exp(?x2 =2)g)2 + k2b (E fexp(?x2 =2)g ? 1=2)2pp(5.48)

with k1 = 36=(8 3 ? 9), k2a = 1=(2 ? 6= ), and k2b = 24=(16 3 ? 27). These
approximations Ja (x) and Jb (x) can be considered more robust and accurate generalizations of the approximation derived using the Gram-Charlier expansion in
Section 5.5.
Even simpler approximations of negentropy can be obtained by using only one
nonquadratic function, which amounts to omitting one of the terms in the preceding
approximations.
5.6.4

Illustration

Here we illustrate the differences in accuracy of the different approximations ofnegentropy. The expectations were here evaluated exactly, ignoring finite-sample
effects. Thus these results do not illustrate the robustness of the maximum entropy
approximation with respect to outliers; this is quite evident anyway.
First, we used a family of gaussian mixture densities, defined byp( ) = '(x) + (1 ? )2'(2(x ? 1))(5.49)

where  is a parameter that takes all the values in the interval 0    1. This
family includes asymmetric densities of both negative and positive kurtosis. The
results are depicted in Fig. 5.2. One can see that both of the approximations Ja and
Jb introduced in Section 5.6.3 were considerably more accurate than the cumulantbased approximation in (5.35).
Second, we considered the exponential power family of density functions:p ( ) = C1 exp(?C2 j j )(5.50)

120

INFORMATION THEORY
0.035

0.03

0.025

0.02

0.015

0.01

0.005

0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Fig. 5.2 Comparison of different approximations of negentropy for the family of mixture
densities in (5.49) parametrized by  ranging from 0 to 1 (horizontal axis). Solid curve:true negentropy. Dotted curve: cumulant-based approximation as in (5.35). Dashed curve:
approximation Ja in (5.47). Dot-dashed curve: approximation Jb in (5.48). The two maximum
entropy approximations were clearly better than the cumulant-based one.

where is a positive constant, and C1 ; C2 are normalization constants that makep a probability density of unit variance. For different values of , the densities inthis family exhibit different shapes. For < 2, one obtains densities of positive= 2, one obtains the gaussian density, and for
kurtosis (supergaussian). For> 2, a density of negative kurtosis. Thus the densities in this family can beused as examples of different symmetric nongaussian densities. In Fig. 5.3, thedifferent negentropy approximations are plotted for this family, using parametervalues 0:5   3. Since the densities used are all symmetric, the first terms in theapproximations were neglected. Again, it is clear that both of the approximations
Ja and Jb introduced in Section 5.6.3 were considerably more accurate than the
cumulant-based approximation in (5.35). Especially in the case of supergaussian
densities, the cumulant-based approximation performed very poorly; this is probably
because it gives too much weight to the tails of the distribution.
5.7 CONCLUDING REMARKS AND REFERENCES
Most of the material in this chapter can be considered classic. The basic definitionsof information theory and the relevant proofs can be found, e.g., in [97, 353]. The
approximations of entropy are rather recent, however. The cumulant-based approximation was proposed in [222], and it is almost identical to those proposed in [12, 89].
The approximations of entropy using nonpolynomial functions were introduced in[196], and they are closely related to the measures of nongaussianity that have been
proposed in the projection pursuit literature, see, e.g., [95].

121PROBLEMS
0.014

0.6

0.012

0.5

0.01

0.4

0.008

0.3

0.006

0.2

0.004

0.1

0.002

0
0.5

1

1.5

2

0
2

2.1

2.2

2.3

2.4

2.5

2.6

2.7

2.8

2.9

3

Fig. 5.3 Comparison of different approximations of negentropy for the family of densities(5.50) parametrized by(horizontal axis). On the left, approximations for densities of
positive kurtosis (0:5
< 2) are depicted, and on the right, approximations for densities of3). Solid curve: true negentropy. Dotted curve: cumulant-based
negative kurtosis (2 <approximation as in (5.35). Dashed curve: approximation Ja in (5.47). Dot-dashed curve:
approximation Jb in (5.48). Clearly, the maximum entropy approximations were much better
than the cumulant-based one, especially in the case of densities of positive kurtosis.

Problems
5.1 Assume that the random variable X can have two values, a and b, as in Example
5.1. Compute the entropy as a function of the probability of obtaining a. Show that
this is maximized when the probability is 1=2.
5.2
5.3

Compute the entropy of X in Example 5.3.

Assume x has a Laplacian distribution of arbitrary variance with pdf

ppx( ) = p1 exp( 2 j j)2

(5.51)

Compute the differential entropy.
5.4

Prove (5.15).

5.5

Prove (5.25).

5.6 Show that the definition of mutual information using Kullback-Leibler divergence is equal to the one given by entropy.
5.7

Compute the three first Chebyshev-Hermite polynomials.

5.8

Prove (5.34). Use the orthogonality in (5.29), and in particular the fact thatH3 and H4 are orthogonal to any second-order polynomial (prove this first!). Fur-thermore, use the fact that any expression involving a third-order monomial of the122

INFORMATION THEORY

higher-order cumulants is infinitely smaller than terms involving only second-ordermonomials (due to the assumption that the pdf is very close to gaussian).
Computer assignments
5.1 Consider random variables with (1) a uniform distribution and (2) a Laplacian
distribution, both with zero mean and unit variance. Compute their differential
entropies with numerical integration. Then, compute the approximations given by
the polynomial and nonpolynomial approximations given in this chapter. Compare
the results.

Appendix proofs
First, we give a detailed proof of (5.13). We have by (5.10)

Z

H (y) = ? py () log py ()dZ=? px (f ? ())j det J f (f ? ())j? log[px (f ? ())j det J f (f ? ())j? ]d
1

1

Z

=

1

1

1

1

? px(f ? ()) log[px (f ? ())]j det J f (f ? ())j? d
1

Z

1

1

1

? px (f ? ()) log[j det J f (f ? ())j? ]j det J f (f ? ())j? d
1

1

1

1

1

(A.1)

Now, let us make the change of integration variable = f ?1 ( )(A.2)

which gives usZ

H (y) = ? px() log[px ()]j det J f ()j?1 j det J f ()jdZ? px() log[j det J f ()j? ]j det J f ()j? j det J f ()jd
1

1

(A.3)

where the Jacobians cancel each other, and we haveZZ

H (y) = ? px () log[px()]d + px () log j det J f ()jd(A.4)

which gives (5.13).
Now follow the proofs connected with the entropy approximations. First, we prove (5.41).
Due to the assumption of near-gaussianity, we can write p0 ( ) as

p0 () = A exp(?2 =2 + an+1  + (an+2 + 1=2)2 +

n
X
i=1

ai Gi ());

(A.5)

123

APPENDIX

where in the exponential, all other terms are very small with respect to the first one. Thus,
using the first-order approximation exp() 1 + , we obtain~ ( )(1 + an+1  + (an+2 + 1=2) 2 +p0 ()  A'

n
X
i=1

ai Gi ());

(A.6)

p

~ = 2A.
where '( ) = (2 )?1=2 exp(  2 =2) is the standardized gaussian density, and A~ and ai become
Due to the orthogonality constraints in (5.39), the equations for solving A
linear and almost diagonal:?

Z

p0 ()d = A~(1 + (an+2 + 1=2)) = 1(A.7)~ n+1 = 0
p0 ()d = Aa

(A.8)

Z

Z
p () d = A~(1 + 3(an + 1=2)) = 1
Z
0

2

+2

~ i = ci ; for i = 1; :::; np0 ()Gi ()d = Aa

(A.9)
(A.10)?

and can be easily solved to yield A~ = 1; an+1 = 0, an+2 = 1=2 and ai = ci ; i = 1; ::; n.
This gives (5.41).
Second, we prove (5.42). Using the Taylor expansion (1+ ) log(1+ ) =  + 2 =2+ o(2 ),one obtainsZ? p^() log p^()dZ=? '()(1 +X

ci Gi ())(log(1 +

Z

=

Z

Z

? '() log '() ? '()? '()[X=X

ci Gi () + 21 (H ( ) ? 0 ? 0 ? 21

X
X

(A.11)

ci Gi ()) + log '())dci Gi () log '()ci Gi ())2 + o((Xc2i + o((XX

ci Gi ())2 )]ci )2 )

(A.12)(A.13)(A.14)(A.15)

due to the orthogonality relationships in (5.39).
Finally, we prove (5.43), (5.47) and (5.48). First, we must orthonormalize the two functions
G1 and G2 according to (5.39). To do this, it is enough to determine constants 1 ; 1 ; 2 ; 2 ; 2
so that the functions F 1 (x) = (G1 (x) + 1 x)=1 and F 2 (x) = (G2 (x) + 2 x2 + 2 )=2
are orthogonal to any second degree polynomials as in (5.39), and have unit norm in the metricdefined by '. In fact, as will be seen below, this modification gives a G1 that is odd and a

124

INFORMATION THEORY

G2 that is even, and therefore the Gi are automatically orthogonal with respect to each other.
Thus, first we solve the following equations:ZZ'()(G1 () + 1 )d = 0(A.16)'()k (G2 () + 2 2 + 2 )d = 0; for k = 0; 2

(A.17)

A straightforward solution gives:

Z

1 =

2

? '()G ()d
1(A.18)ZZ
1= ('()G ()d ? '()G () d)
2
Z
Z
1
2

2 =

2

2

2

(A.19)'()G2 ()2 d ? 3 '()G2 ()d)(

Next note that together with the standardization(A.20)

R '()(G () +  + )d = 0 implies
2

2

2

2

ci = E fF i (x)g = [E fGi (x)g ? E fGi ( )g]=i(A.21)
This implies (5.43), with ki2 = 1=(2i2 ). Thus we only need to determine explicitly the i foreach function. We solve the two equations

Z

Z'()(G1 () + 1 )2 =1 d = 1(A.22)'()(G2 () + 2 2 + 2 )2 =2 d = 1(A.23)

which, after some tedious manipulations, yield:




2
1 =

2
2 =

Z

Z

Z

'()G () d ? ( '()G1 () d)2
1

2

(A.24)Z'()G () d ? ( '()G2 ()d)2

Z

2

2

Z

? 21 ( '()G ()d ? '()G () d) :
2

2

2

2

(A.25)

Evaluating the i for the given functions Gi , one obtains (5.47) and (5.48) by the relation
ki2 = 1=(2i2 ).

6Principal Component
Analysis and Whitening
Principal component analysis (PCA) and the closely related Karhunen-Loève transform, or the Hotelling transform, are classic techniques in statistical data analysis,
feature extraction, and data compression, stemming from the early work of Pearson[364]. Given a set of multivariate measurements, the purpose is to find a smaller set ofvariables with less redundancy, that would give as good a representation as possible.
This goal is related to the goal of independent component analysis (ICA). However,in PCA the redundancy is measured by correlations between data elements, while
in ICA the much richer concept of independence is used, and in ICA the reductionof the number of variables is given less emphasis. Using only the correlations as in
PCA has the advantage that the analysis can be based on second-order statistics only.
In connection with ICA, PCA is a useful preprocessing step.
The basic PCA problem is outlined in this chapter. Both the closed-form solution
and on-line learning algorithms for PCA are reviewed. Next, the related linear
statistical technique of factor analysis is discussed. The chapter is concluded bypresenting how data can be preprocessed by whitening, removing the effect of firstand second-order statistics, which is very helpful as the first step in ICA.

6.1 PRINCIPAL COMPONENTSx

The starting point for PCA is a random vector with n elements. There is available
a sample (1); :::; (T ) from this random vector. No explicit assumptions on theprobability density of the vectors are made in PCA, as long as the first- and secondorder statistics are known or can be estimated from the sample. Also, no generative

x

x

125

126

PRINCIPAL COMPONENT ANALYSIS AND WHITENINGxx

model is assumed for vector . Typically the elements of are measurements like
pixel gray levels or values of a signal at different time instants. It is essential in
PCA that the elements are mutually correlated, and there is thus some redundancy
in , making compression possible. If the elements are independent, nothing can beachieved by PCA.
In the PCA transform, the vector is first centered by subtracting its mean:xxx x ? Efxgxyx

The mean is in practice estimated from the available sample (1); :::; (T ) (seeChapter 4). Let us assume in the following that the centering has been done and thus
Ef g = 0. Next, is linearly transformed to another vector with m elements,
m < n, so that the redundancy induced by the correlations is removed. This isdone by finding a rotated orthogonal coordinate system such that the elements ofin the new coordinates become uncorrelated. At the same time, the variances ofthe projections of on the new coordinate axes are maximized so that the first axis
corresponds to the maximal variance, the second axis corresponds to the maximal
variance in the direction orthogonal to the first axis, and so on.
For instance, if has a gaussian density that is constant over ellipsoidal surfaces
in the n-dimensional space, then the rotated coordinate system coincides with theprincipal axes of the ellipsoid. A two-dimensional example is shown in Fig. 2.7 in
Chapter 2. The principal components are now the projections of the data points on thetwo principal axes, 1 and 2 . In addition to achieving uncorrelated components, thevariances of the components (projections) also will be very different in most applications, with a considerable number of the variances so small that the corresponding
components can be discarded altogether. Those components that are left constitutethe vector .
As an example, take a set of 88 pixel windows from a digital image,an applicationthat is considered in detail in Chapter 21. They are first transformed, e.g., using rowby-row scanning, into vectors whose elements are the gray levels of the 64 pixels
in the window. In real-time digital video transmission, it is essential to reduce thisdata as much as possible without losing too much of the visual quality, because thetotal amount of data is very large. Using PCA, a compressed representation vector
can be obtained from , which can be stored or transmitted. Typically, can have as
few as 10 elements, and a good replica of the original 8  8 image window can still
be reconstructed from it. This kind of compression is possible because neighboring
elements of , which are the gray levels of neighboring pixels in the digital image,are heavily correlated. These correlations are utilized by PCA, allowing almost thesame information to be represented by a much smaller vector . PCA is a linear
technique, so computing from is not heavy, which makes real-time processing
possible.

x

x

x

x

x

e

e

y

x

x

y

x

y

x

y

y

127

PRINCIPAL COMPONENTS

6.1.1

PCA by variance maximization

In mathematical terms, consider a linear combination

y1 =

n
X

k=1

wk1 xk = w1T x

xof the elements x1 ; :::; xn of the vector . The w11 ; :::; wn1 are scalar coefficients or
weights, elements of an n-dimensional vector 1 , and 1T denotes the transpose of1.
The factor y1 is called the first principal component of , if the variance of y1 is
maximally large. Because the variance depends on both the norm and orientation of
the weight vector 1 and grows without limits as the norm grows, we impose theconstraint that the norm of 1 is constant, in practice equal to 1. Thus we look for aweight vector 1 maximizing the PCA criterion

w

w

w

x

w

w
w
J PCA (w ) = Efy g = Ef(wT x) g = wT EfxxT gw = wT Cx w (6.1)
so that kw k = 1(6.2)
There Ef:g is the expectation over the (unknown) density of input vector x, and thenorm of w is the usual Euclidean norm defined as
1

1

2
1

2

1

1

1

1

1

1

1

C

n
X
T
1=2
kw1 k = (w1 w1 ) = [ wk21 ]1=2
k=1

The matrix x in Eq. (6.1) is the n  n covariance matrix offor the zero-mean vector by the correlation matrixxx (see Chapter 4) given

Cx = EfxxT g(6.3)

It is well known from basic linear algebra (see, e.g., [324, 112]) that the solution
to the PCA problem is given in terms of the unit-length eigenvectors 1 ; :::; n of
the matrix x . The ordering of the eigenvectors is such that the corresponding
eigenvalues d1 ; :::; dn satisfy d1  d2  :::  dn . The solution maximizing (6.1) isgiven by

e

C

e

w =e
Thus the first principal component of x is y = eT x.
1

1

1

1

The criterion J1PCA in eq. (6.1) can be generalized to m principal components,
with m any number between 1 and n. Denoting the m-th (1  m  n) principal
T , with m the corresponding unit norm weight vector, thecomponent by ym = mvariance of ym is now maximized under the constraint that ym is uncorrelated withall the previously found principal components:

w x

w

Efym yk g = 0; k < m:Note that the principal components ym have zero means because
Efym g =wmT Efxg = 0

(6.4)

128

PRINCIPAL COMPONENT ANALYSIS AND WHITENING

The condition (6.4) yields:

T )( T
Efym yk g = Ef( mkw x w x)g = wmT Cxwk = 0(6.5)

For the second principal component, we have the condition thatwT Cw = d wT e = 0(6.6)
because we already know that w = e . We are thus looking for maximal varianceEfy g = Ef(wT x) g in the subspace orthogonal to the first eigenvector of Cx . The
solution is given by
w =e
1

2

2
2

1

2

2

1

1

2

1

2

2

Likewise, recursively it follows thatwk = e kThus the k th principal component is yk = eTk x.
Exactly the same result for the wi is obtained if the variances of yi are maximized under the constraint that the principal component vectors are orthonormal, or
wiT wj = ij . This is left as an exercise.
6.1.2

PCA by minimum mean-square error compression

In the preceding subsection, the principal components were defined as weighted sumsof the elements of with maximal variance, under the constraints that the weights
are normalized and the principal components are uncorrelated with each other. It
turns out that this is strongly related to minimum mean-square error compressionof , which is another way to pose the PCA problem. Let us search for a set of morthonormal basis vectors, spanning an m-dimensional subspace, such that the meansquare error between and its projection on the subspace is minimal. Denoting again
the basis vectors by 1 ; :::; m , for which we assume

x

x

x
w

w

wiT wj = ij

x

Pm w x w

the projection of on the subspace spanned by them is i=1 ( iT ) i . The meansquare error (MSE) criterion, to be minimized by the orthonormal basis 1 ; :::; m ,becomes

PCA = Efkx ?
JMSE

w

m
X
T
i=1

(wi x)wi k2g

It is easy to show (see exercises) that due to the orthogonality of the vectors
criterion can be further written as

m
X

w

(6.7)

wi, this

PCA = Efkxk2 g ? Ef (wT x)2 g
JMSE
j

(6.8)

wjT Cxwj(6.9)=

Cx ) ?

trace(

j =1
m
X
j =1

PRINCIPAL COMPONENTS

129

It can be shown (see, e.g., [112]) that the minimum of (6.9) under the orthonormality condition on the i is given by any orthonormal basis of the PCA subspacespanned by the m first eigenvectors 1 ; :::; m . However, the criterion does not specify the basis of this subspace at all. Any orthonormal basis of the subspace will givethe same optimal compression. While this ambiguity can be seen as a disadvantage,
it should be noted that there may be some other criteria by which a certain basis in
the PCA subspace is to be preferred over others. Independent component analysis isa prime example of methods in which PCA is a useful preprocessing step, but oncethe vector has been expressed in terms of the first m eigenvectors, a further rotation
brings out the much more useful independent components.
It can also be shown [112] that the value of the minimum mean-square error of(6.7) is

w

e

e

x

PCA =
JMSE

n
X
i=m+1

di

(6.10)ee

the sum of the eigenvalues corresponding to the discarded eigenvectors m+1 ; :::; n .
If the orthonormality constraint is simply changed to

wjT wk = !k jk(6.11)

where all the numbers !k are positive and different, then the mean-square error
problem will have a unique solution given by scaled eigenvectors [333].
6.1.3

Choosing the number of principal components

From the result that the principal component basis vectors
x , it follows thatC

wi are eigenvectors ei of

e xxT emg = eTmCxem = dm

2
g = Ef Tm
Efym(6.12)The variances of the principal components are thus directly given by the eigenvaluesof x . Note that, because the principal components have zero means, a small eigenvalue (a small variance) dm indicates that the value of the corresponding principal
component ym is mostly close to zero.
An important application of PCA is data compression. The vectors in the original
data set (that have first been centered by subtracting the mean) are approximated by
the truncated PCA expansion

C

x

x^ =

m
X
i=1

yi ei

(6.13)

x x

Then we know from (6.10) that the mean-square error E fk ? ^ k2 g is equal toni=m+1 di . As the eigenvalues are all positive, the error decreases when more andmore terms are included in (6.13), until the error becomes zero when m = n or all
the principal components are included. A very important practical problem is how to

P

130

PRINCIPAL COMPONENT ANALYSIS AND WHITENING

choose m in (6.13); this is a trade-off between error and the amount of data needed
for the expansion. Sometimes a rather small number of principal components are
sufficient.

Fig. 6.1 Leftmost column: some digital images in a 32 32 grid. Second column: meansof the samples. Remaining columns: reconstructions by PCA when 1, 2, 5, 16, 32, and 64
principal components were used in the expansion.

Example 6.1 In digital image processing, the amount of data is typically very large,
and data compression is necessary for storage, transmission, and feature extraction.
PCA is a simple and efficient method. Fig. 6.1 shows 10 handwritten characters thatwere represented as binary 32  32 matrices (left column) [183]. Such images, when
scanned row by row, can be represented as 1024-dimensional vectors. For each of the
10 character classes, about 1700 handwritten samples were collected, and the sample
means and covariance matrices were computed by standard estimation methods. The
covariance matrices were 1024  1024 matrices. For each class, the first 64 principal
component vectors or eigenvectors of the covariance matrix were computed. The
second column in Fig. 6.1 shows the sample means, and the other columns show thereconstructions (6.13) for various values of m. In the reconstructions, the sample
means have been added again to scale the images for visual display. Note how arelatively small percentage of the 1024 principal components produces reasonable
reconstructions.

131

PRINCIPAL COMPONENTS

The condition (6.12) can often be used in advance to determine the number ofprincipal components m, if the eigenvalues are known. The eigenvalue sequenced1 ; d2 ; :::; dn of a covariance matrix for real-world measurement data is usually
sharply decreasing, and it is possible to set a limit below which the eigenvalues,
hence principal components, are insignificantly small. This limit determines how
many principal components are used.
Sometimes the threshold can be determined from some prior information on thevectors . For instance, assume that obeys a signal-noise model

x

x

x=
a

m
X
i=1

a i si + n(6.14)

where m < n. There i are some fixed vectors and the coefficients si are random
numbers that are zero mean and uncorrelated. We can assume that their variances
have been absorbed in vectors i so that they have unit variances. The term iswhite noise, for which Ef T g =  2 . Then the vectors i span a subspace, called
the signal subspace, that has lower dimensionality than the whole space of vectors. The subspace orthogonal to the signal subspace is spanned by pure noise and it iscalled the noise subspace.
It is easy to show (see exercises) that in this case the covariance matrix of has a
special form:

nn

a

I

n

a

x

x

Cx =

m
X
i=1

ai aTi +  I
2(6.15)

Pm a aThe eigenvalues are now the eigenvalues of i=1 i Ti , added by the constant  2 .
m
But the matrix i=1 i Ti has at most m nonzero eigenvalues, and these correspondto eigenvectors that span the signal subspace. When the eigenvalues of x arecomputed, the first m form a decreasing sequence and the rest are small constants,
equal to  2 :

P aa

C

d1 > d2 > ::: > dm > dm+1 = dm+2 = ::: = dn = 2

It is usually possible to detect where the eigenvalues become constants, and puttinga threshold at this index, m, cuts off the eigenvalues and eigenvectors corresponding
to pure noise. Then only the signal part remains.
A more disciplined approach to this problem was given by [453]; see also [231].
They give formulas for two well-known information theoretic modeling criteria,
Akaike’s information criterion (AIC) and the minimum description length criterion(MDL), as functions of the signal subspace dimension m. The criteria depend on thelength T of the sample (1); :::; (T ) and on the eigenvalues d1 ; :::; dn of the matrix
x . Finding the minimum point gives a good value for m.

C

6.1.4

x

x

Closed-form computation of PCAweC

To use the closed-form solution i = i given earlier for the PCA basis vectors, theeigenvectors of the covariance matrix x must be known. In the conventional use of

132

PRINCIPAL COMPONENT ANALYSIS AND WHITENINGx

PCA, there is a sufficiently large sample of vectors available, from which the mean
and the covariance matrix x can be estimated by standard methods (see Chapter 4).
Solving the eigenvector–eigenvalue problem for x gives the estimate for 1 . There
are several efficient numerical methods available for solving the eigenvectors, e.g.,
the QR algorithm with its variants [112, 153, 320].
However, it is not always feasible to solve the eigenvectors by standard numericalmethods. In an on-line data compression application like image or speech coding,the data samples (t) arrive at high speed, and it may not be possible to estimate thecovariance matrix and solve the eigenvector–eigenvalue problem once and for all.
One reason is computational: the eigenvector problem is numerically too demanding
if the dimensionality n is large and the sampling rate is high. Another reason is that thecovariance matrix x may not be stationary, due to fluctuating statistics in the sample
sequence (t), so the estimate would have to be incrementally updated. Therefore,
the PCA solution is often replaced by suboptimal nonadaptive transformations likethe discrete cosine transform [154].

C

C

e

x

x

C

6.2 PCA BY ON-LINE LEARNING
Another alternative is to derive gradient ascent algorithms or other on-line methods
for the preceding maximization problems. The algorithms will then converge to thesolutions of the problems, that is, to the eigenvectors. The advantage of this approachis that such algorithms work on-line, using each input vector (t) once as it becomes
available and making an incremental change to the eigenvector estimates, without
computing the covariance matrix at all. This approach is the basis of the PCA neuralnetwork learning rules.
Neural networks provide a novel way for parallel on-line computation of the PCA
expansion. The PCA network [326] is a layer of parallel linear artificial neurons
shown in Fig. 6.2. The output of the ith unit (i = 1; :::; m) is yi = iT , withdenoting the n-dimensional input vector of the network and i denoting the weightvector of the ith unit. The number of units, m, will determine how many principal
components the network will compute. Sometimes this can be determined in advancefor typical inputs, or m can be equal to n if all principal components are required.
The PCA network learns the principal components by unsupervised learning rules,
by which the weight vectors are gradually updated until they become orthonormal
and tend to the theoretically correct eigenvectors. The network also has the ability to
track slowly varying statistics in the input data, maintaining its optimality when thestatistical properties of the inputs do not stay constant. Due to their parallelism andadaptivity to input data, such learning algorithms and their implementations in neuralnetworks are potentially useful in feature detection and data compression tasks.
In ICA, where decorrelating the mixture variables is a useful preprocessing step,these learning rules can be used in connection to on-line ICA.

x

w

wx

x

PCA BY ON-LINE LEARNING

Input vector

w

133

x

? ? ? ? ? ?
Q

Qc

C##
A ?S
Sc
@
@
Q
AA CC
C
?
A

S
 @
@
S
Q c #C S A C
?S
A @
#c
Q
  @SQAcC @S A C#@
  ?
Q#c@
 SA C@SA CQ
 ?
#
c

C QQ
?
A
S
@

@SAC
#
c

#
?
AC

c
@
SAC
1 
m?@
1S
mQ
.......................

 w 

w

?

?

wT x

wmT ?1x

1

?

wmT x

Fig. 6.2 The basic linear PCA layer

6.2.1

The stochastic gradient ascent algorithmw

In this learning rule, the gradient of y12 is taken with respect to 1 and the normalizing
constraint k 1 k = 1 is taken into account. The learning rule isww (t + 1) = w (t) + (t)[y (t)x(t) ? y (t)w (t)]with y (t) = w (t)T x(t). This is iterated over the training set x(1); x(2); ::::. The
1

1

1

2
1

1

1

1

parameter (t) is the learning rate controlling the speed of convergence.
In this chapter we will use the shorthand notation introduced in Chapter 3 andwrite the learning rule asw1 = (y1 x ? y12 w1 )(6.16)The name stochastic gradient ascent (SGA) is due to the fact that the gradient is notwith respect to the variance Efy12 g but with respect to the instantaneous random value
y12 . In this way, the gradient can be updated every time a new input vector becomes
available, contrary to batch mode learning. Mathematically, this is a stochastic
approximation type of algorithm (for details, see Chapter 3). Convergence requiresthat the learning rate is decreased during learning at a suitable rate. For tracking
nonstationary statistics, the learning rate should remain at a small constant value.
For a derivation of this rule, as well as for the mathematical details of its convergence,see [323, 324, 330]. The algorithm (6.16) is often called Oja’s rule in the literature.
Likewise, taking the gradient of yj2 with respect to the weight vector j and using
the normalization and orthogonality constraints, we end up with the learning rulew

wj = yj [x ? yj wj ? 2Xi<jyi wi ]

(6.17)

134

PRINCIPAL COMPONENT ANALYSIS AND WHITENINGx

On the right-hand side there is a term yj , which is a so-called Hebbian term,
product of the output yj of the j th neuron and the input to it. The other terms
are implicit orthonormality constraints. The case j = 1 gives the one-unit learning
rule (6.16) of the basic PCA neuron. The convergence of the vectors 1 ; :::; m to
the eigenvectors 1 ; :::; m was established in [324, 330]. A modification called thegeneralized Hebbian algorithm (GHA) was later presented by Sanger [391], who also
applied it to image coding, texture segmentation, and the development of receptive
fields.

e

6.2.2

x

w

e

w

The subspace learning algorithm

The following algorithm [324, 458]

wj = yj [x ?

m
X
i=1

yi wi ]

(6.18)

Pm w xis obtained as a constrained gradient ascent maximization of j =1 ( jT )2 , themean of which gives criterion (6.9). The regular structure allows this algorithm to be= ( 1 ::: m )T the m  n matrix
written in a simple matrix form: denoting by
whose rows are the weight vectors j , we have the update rule

W w ww
W = [WxxT ? (WxxT WT )W]:

(6.19)The network implementation of (6.18) is analogous to the SGA algorithm but
still simpler because the normalizing feedback term, depending on the other weightvectors, is the same for all neuron units. The convergence was studied by Williams[458], who showed that the weight vectors 1 ; :::; m will not tend to the eigenvectors 1 ; :::; m but only to some rotated basis in the subspace spanned by them, in
analogy with the minimum mean-square criterion of Section 6.1.2. For this reason,
this learning rule is called the subspace algorithm. A global convergence analysis
was given in [465, 75].
A variant of the subspace algorithm (6.18) is the weighted subspace algorithm

e

w

e

wj = yj [x ? j

w

m
X
i=1

yi wi ]

(6.20)

Algorithm (6.20) is similar to (6.18) except for the scalar parameters 1 ; :::; m , which
are inverses of the parameters !1 ; :::; !m in criterion (6.11). If all of them are chosendifferent and positive, then it was shown by [333] that the vectors 1 ; :::; m willtend to the true PCA eigenvectors 1 ; :::; m multiplied by scalars. The algorithm isappealing because it produces the true eigenvectors but can be computed in a fullyparallel way in a homogeneous network. It can be easily presented in a matrix form,
analogous to (6.19).
Other related on-line algorithms have been introduced in [136, 388, 112, 450].
Some of them, like the APEX algorithm by Diamantaras and Kung [112], are based

e

e

w

w

135

PCA BY ON-LINE LEARNING

on a feedback neural network. Also minor components defined by the eigenvectors
corresponding to the smallest eigenvalues can be computed by similar algorithms[326]. Overviews of these and related neural network realizations of signal processing
algorithms are given by [83, 112].
6.2.3

Recursive least-squares approach: the PAST algorithm *The on-line algorithms reviewed in preceding sections typically suffer from slow
convergence. The learning rate (t) would have to be tuned optimally to speed up
the convergence. One way of doing this is to use the recursive least squares (RLS)
principle.
Recursive least squares methods have a long history in statistics, adaptive signal
processing, and control; see [171, 299]. For example in adaptive signal processing,it is well known that RLS methods converge much faster than the standard stochastic gradient based least-mean-square (LMS) algorithm at the expense of somewhat
greater computational cost [171].
Consider the mean-square error criterion (6.7). The cost function is in practiceestimated from a fixed sample (1); :::; (T ) asx

PCA = 1
J^MSE
T

x

T
X
j =1

[kx(j ) ?

m
X
T
i=1

(wi x(j ))wi k2](6.21)For simplicity of notation, let us write this in matrix form: denoting again( 1 ::: m )T we havew w

PCA = 1
J^MSE

TX

T j=1[kx(j ) ? W Wx(j )k ]T

2

W=
(6.22)

In [466], the following exponentially weighted sum was considered instead:

JMSE (t) =Xt t?jj =1[kx(j ) ? W(t)T W(t)x(j )k2 ](6.23)

The fixed multiplier T1 has now been replaced by an exponential smoother t?j ,where the “forgetting factor” is between 0 and 1. If = 1, all the samples are
given the same weight, and no forgetting of old data takes place. Choosing < 1 is
especially useful in tracking nonstationary changes in the sources. The solution isdenoted by (t) to indicate that it depends on the sample (1); :::; (t) up to timeindex t. The problem is to solve (t) recursively: knowing (t ? 1), we compute(t) from an update rule.
(t). It
Note that the cost function (6.23) is fourth order in the elements ofcan be simplified by approximating the vector(t) (j ) in the sum (6.23) by thevector (j ) =(j ? 1) (j ). These vectors can be easily computed because theestimated weight matrices (j ? 1) for the previous iteration steps j = 1; :::; t are

W

W

y

x
W

W

W

x
W

W x

x

W

136

PRINCIPAL COMPONENT ANALYSIS AND WHITENING

already known at step t. The approximation error is usually rather small after initialconvergence. This approximation yields the modified least-squares-type criterion

0 (t) =JMSE

Xt t?jj =1[kx(j ) ? WT (t)y(j )k]2

(6.24)

The cost function (6.24) is now of the standard form used in recursive least-squares
methods. Any of the available algorithms [299] can be used for solving the weightmatrix (t) iteratively. The algorithm proposed by Yang [466], which he calls the
Projection Approximation Subspace Tracking (PAST) algorithm, is as follows:Wy(t) = W(t ? 1)x(t)h(t) = P(t ? 1)y(t)m(t) = h(t)=( + yT (t)h(t))
P(t) = 1 Tri P(t ? 1) ? m(t)hT (t)e(t) = x(t) ? WT (t ? 1)y(t)W(t) = W(t ? 1) + m(t)eT (t)(6.25)The notation Tri means that only the upper triangular part of the argument iscomputed and its transpose is copied to the lower triangular part, making thus thematrix (t) symmetric. The simplest way to choose the initial values is to set both(0) and (0) to n  n unit matrices.
The PAST algorithm (6.25) can be regarded either as a neural network learningalgorithm or adaptive signal processing algorithm. It does not require any matrix
inversions, because the most complicated operation is division by a scalar. The
computational cost is thus low. The convergence of the algorithm is relatively fast,
as shown in [466].

W

6.2.4

P

P

PCA and back-propagation learning in multilayer perceptrons *Another possibility for PCA computation in neural networks is the multilayer perceptron (MLP) network, which learns using the back-propagation algorithm (see [172])in unsupervised autoassociative mode. The network is depicted in Fig. 6.3.
The input and output layers have n units and the hidden layer has m < n units.
The outputs of the hidden layer are given byh = (W x + b )1

W(6.26)

1b

where 1 is the input-to-hidden-layer weight matrix, 1 is the corresponding bias
vector, and  is the activation function, to be applied elementwise. The output of
the network is an affine linear function of the hidden-layer outputs:

y =W h+b22

y(6.27)

PCA BY ON-LINE LEARNING

n

137

n
m

W ;b
1

W ;b

-

1

2

-

2

h = (W x + b )11xy =W h+b22

Fig. 6.3 The three-layer MLP in autoassociative mode.

with obvious notation.
In autoassociative mode, the same vectors are used both as inputs and as desired
outputs in back-propagation learning. If  is linear, then the hidden layer outputs will
become the principal components of [23]. For the linear network, back-propagation
learning is especially feasible because it can be shown that the "energy" function has
no local minima.
This network with a nonlinear hidden layer was suggested for data compression
by [96], and it was shown to be closely connected to the theoretical PCA by [52].
It is not equivalent to PCA, however, as shown by [220], unless the hidden layer islinear.

x

x

6.2.5

Extensions of PCA to nonquadratic criteria *In the on-line learning rules reviewed earlier, the explicit computation of the eigenvectors of the covariance matrix has been replaced by gradient ascent. This makes it
possible to widely extend the PCA criteria. In fact, any criterion J ( 1 ; :::; n ) suchthat its maximum over the constraint set iT j = ij coincides with the dominanteigenvectors of x , or is a basis of the subspace spanned by them, could now be
used instead. The criterion does not have to be quadratic anymore, like all the criteriapresented in Section 6.1. An advantage might then be faster convergence to the PCA
basis.

C

ww

w

w

138

PRINCIPAL COMPONENT ANALYSIS AND WHITENING

Recently, Miao and Hua [300] proposed such a criterion: again denoting by W =(w1 :::wm )T the matrix whose rows are the weight vectors, this “novel information
criterion” (NIC) is

JNIC (W) = 21 ftrace[log(WCx WT )] ? trace(WWT )g:(6.28)

It was shown in [300] that the matrix gradient is@JNIC (W) = C WT (WC WT )?1 ? WTx
x
@W

(6.29)

C

Setting this to zero gives the eigenvalue–eigenvector equation for x, so the stationary
points of this criterion are given by bases of PCA subspaces.
Other extensions and analysis of both PCA and minor component learning rules
have been given by [340, 480].

6.3 FACTOR ANALYSISThe PCA model was discussed above as a distribution-free method with no underlying
statistical model. However, PCA can also be derived from a generative latent variable
model: assume

x = Ay + ny(6.30)yyICnwhere is gaussian, zero-mean and white, so that Ef T g = , and is zero-meangaussian white noise. It is now easy to formulate the likelihood function, because
the density of , given , is gaussian. Scaled eigenvectors of x are obtained as therows of in the maximum likelihood solution, in the limiting case when the noise
tends to zero.
This approach is one of the methods for the classic statistical technique of factor
analysis (FA). It is called principal factor analysis [166]. Generally, the goal in factor
analysis is different from PCA. Factor analysis was originally developed in social
sciences and psychology. In these disciplines, the researchers want to find relevant
and meaningful factors that explain observed results [166, 243, 454]. The model has
the form of (6.30), with the interpretation that the elements of are the unobservablefactors. The elements aij of matrix are called factor loadings. The elements of
the additive term are called specific factors, instead of noise. Let us make thesimplifying assumption that the data has been normalized to zero mean.
In FA we assume that the elements of (the factors) are uncorrelated and gaussian,and their variances can be absorbed into the unknown matrix so that we can assume

A

x

y

n

y

A

y

yyT g = I:

Ef

n
nn

A

(6.31)The elements of are uncorrelated with each other and also with the factors yi ;denote = Ef T g. It is a diagonal matrix, but the variances of the noise elements

Q

FACTOR ANALYSIS

139

are generally not assumed to be equal or infinitely small, as in the special case ofprincipal FA. We can write the covariance matrix of the observations from (6.30) as

xxT g = Cx = AAT + Q(6.32)
In practice, we have a good estimate of Cx available, given by the sample covariance matrix. The main problem is then to solve the matrix A of factor loadings
and the diagonal noise covariance matrix Q such that they will explain the observed
covariances from (6.32). There is no closed-form analytic solution for A and Q.
Assuming Q is known or can be estimated, we can attempt to solve A from
AAT = Cx ? Q. The number of factors is usually constrained to be much smallerEfthan the number of dimensions in the data, so this equation cannot be exactly solved;
something similar to a least-squares solution should be used instead. Clearly, thisproblem does not have a unique solution: any orthogonal transform or rotation of! , with an orthogonal matrix (for which T = ), will produce exactly
the same left-hand side. We need some extra constraints to make the problem moreunique.
Now, looking for a factor-based interpretation of the observed variables, FA typically tries to solve the matrix in such a way that the variables would have highloadings on a small number of factors, and very low loadings on the remaining factors. The results are then easier to interpret. This principle has been used in suchtechniques as varimax, quartimax, and oblimin rotations. Several classic techniques
for such factor rotations are covered by Harman [166].
There are some important differences between PCA, FA, and ICA. Principal
component analysis is not based on a generative model, although it can be derived
from one. It is a linear transformation that is based either on variance maximization
or minimum mean-square error representation. The PCA model is invertible inthe (theoretical) case of no compression, i.e., when all the principal components are
retained. Once the principal components yi have been found, the original observationsncan be readily expressed as their linear functions as = i=1 yi i , and also theprincipal components are simply obtained as linear functions of the observations:yi = iT .
The FA model is a generative latent variable model; the observations are expressedin terms of the factors, but the values of the factors cannot be directly computed fromthe observations. This is due to the additive term of specific factors or noise whichis considered important in some application fields. Further, the rows of matrix
are generally not (proportional to) eigenvectors of x ; several different estimation
methods exist.
FA, as well as PCA, is a purely second-order statistical method: only covariances
between the observed variables are used in the estimation, which is due to theassumption of gaussianity of the factors. The factors are further assumed to be
uncorrelated, which also implies independence in the case of gaussian data. ICA
is a similar generative latent variable model, but now the factors or independent
components are assumed to be statistically independent and nongaussian — a muchstronger assumption that removes the rotational redundancy of the FA model. In fact,
ICA can be considered as one particular method of determining the factor rotation.

A AT

T

TT

I

A

x P

w

w x

C

A

140

PRINCIPAL COMPONENT ANALYSIS AND WHITENING

The noise term is usually omitted in the ICA model; see Chapter 15 for a detailed
discussion on this point.

6.4 WHITENING
As already discussed in Chapter 1, the ICA problem is greatly simplified if theobserved mixture vectors are first whitened or sphered. A zero-mean random vector= (z1 :::zn )T is said to be white if its elements zi are uncorrelated and have unit
variances:
Efzi zj g = ij

z

zz

I

I

In terms of the covariance matrix, this obviously means that Ef T g = , withthe unit matrix. The best-known example is white noise; then the elements zi would
be the intensities of noise at consequent time points i = 1; 2; ::: and there are no
temporal correlations in the noise process. The term “white” comes from the factthat the power spectrum of white noise is constant over all frequencies, somewhat
like the spectrum of white light contains all colors.
A synonym for white is sphered. If the density of the vector is radially symmetricand suitably scaled, then it is sphered. An example is the multivariate gaussian
density that has zero mean and unit covariance matrix. The opposite does not hold:the density of a sphered vector does not have to be radially symmetric. An example
is a two-dimensional uniform density that has the shape of a rotated square; see
Fig. 7.10. It is easy to see that in this case both the variables z1 andp z2 on thecoordinate axes have unit variance (if the side of the square has length 2 3) and they
are uncorrelated, independently of the rotation angle. Thus vector is sphered, even
if the density is highly nonsymmetric. Note that the densities of the elements zi of asphered random vector need not be the same.
Because whitening is essentially decorrelation followed by scaling, the techniqueof PCA can be used. This implies that whitening can be done with a linear operation.
The problem of whitening is now: Given a random vector with n elements, find alinear transformation into another vector such that

z

z

V

z
z = Vxx

is white (sphered).
The problem has a straightforward solution in terms of the PCA expansion. Let= ( 1 ::: n ) be the matrix whose columns are the unit-norm eigenvectors of
the covariance matrix x = Ef T g. These can be computed from a sampleof the vectors either directly or by one of the on-line PCA learning rules. Let= diag(d1 :::dn ) be the diagonal matrix of the eigenvalues of . Then a linear
whitening transform is given by

E

D

e e

x

C

xx

C

V = D ? = ET
1 2

(6.33)

This matrix always exists when the eigenvalues di are positive; in practice, this is nota restriction. Remember (see Chapter 4) that x is positive semidefinite, in practice
positive definite for almost any natural data, so its eigenvalues will be positive.

CORTHOGONALIZATION

V
EDE

141

of Eq. (6.33) is indeed a whitening transIt is easy to show that the matrix
formation. Recalling that x can be written in terms of its eigenvector and eigenT , with an orthogonal matrix satisfying
value matrices and as x =T =T = , it holds:

E E EE

E

I

C
D C

E

zzT g = VEfxxT gVT = D? = ET EDET ED? = = I
The covariance of z is the unit matrix, hence z is white.
The linear operator V of (6.33) is by no means the only unique whitening matrix.
It is easy to see that any matrix UV, with U an orthogonal matrix, is also a whitening
matrix. This is because for z = UVx it holds:
EfzzT g = UVEfxxT gVT UT = UIUT = I
An important instance is the matrix ED? = ET . This is a whitening matrix
because it is obtained by multiplying V of Eq. (6.33) from the left by the orthogonal?=
matrix E. This matrix is called the inverse square root of Cx , and denoted by Cx ,
Ef

1 2

1 2

1 2

1 2

because it comes from the standard extension of square roots to matrices.
It is also possible to perform whitening by on-line learning rules, similar to the
PCA learning rules reviewed earlier. One such direct rule isV = (I ? VxxT VT )V = (I ? zzT )VIt can be seen that at a stationary point, when the change in the value ofthe average, it holds( ? Ef T g) = 0

z Vx

I

(6.34)

V is zero onzz V

for which a whitened =is a solution. It can be shown (see, e.g., [71]) that thealgorithm will indeed converge to a whitening transformation .

V

6.5 ORTHOGONALIZATION
In some PCA and ICA algorithms, we know that in theory the solution vectors(PCA basis vectors or ICA basis vectors) are orthogonal or orthonormal, but theiterative algorithms do not always automatically produce orthogonality. Then it may
be necessary to orthogonalize the vectors after each iteration step, or at some suitableintervals. In this subsection, we look into some basic orthogonalization methods.
Simply stated, the problem is as follows: given a set of n-dimensional linearly
independent vectors 1 ; :::; m , with m  n, compute another set of m vectors
1 ; :::;
m that are orthogonal or orthonormal (i.e., orthogonal and having unit
Euclidean norm) and that span the same subspace as the original vectors. This meansthat each i is some linear combination of the j .
The classic approach is the Gram-Schmidt orthogonalization (GSO) method [284]:

w

w
w

a

a

a

w = a
j? T
X
wj = aj ? wwTi waj wi
1

1(6.35)

1

i=1

i

i

(6.36)

142

PRINCIPAL COMPONENT ANALYSIS AND WHITENINGw w

As a result, iT j = 0 for i 6= j , as is easy to show by induction. Assume thatthe first j ? 1 basis vectors are already orthogonal; from (6.36) it then follows forj ?1 wT a
any k < j that kT j = kT j ? i=1 wTi wji ( kT i ). In the sum, all the inneri
products kT i are zero except the one where i = k . This term becomes equal to
wkT aj T
T
T
wkT wk ( k k ) = k j , and thus the inner product k j is zero, too.
If in the GSO each j is further divided by its norm, the set will be orthonormal.
The GSO is a sequential orthogonalization procedure. It is the basis of deflation
approaches to PCA and ICA. A problem with sequential orthogonalization is the
cumulation of errors.
In symmetric orthonormalization methods, none of the original vectors i is treateddifferently from the others. If it is sufficient to find any orthonormal basis for thesubspace spanned by the original vectors,without other constraints on the new vectors,
then this problem does not have a unique solution. This can be accomplished for
instance by first forming the matrix= ( 1 ::: m ) whose columns are the vectors
to be orthogonalized, then computing ( T )?1=2 using the eigendecomposition of
the symmetric matrix ( T ), and finally putting

w w

w w
w w w a
w

wa

P

w w

w w

a

A

AA

a a
AA

W = A(AT A)? =

1 2

W

I
A
WU

(6.37)

W W

w

w

T
it holds= , and its columns 1 ; :::; m span
Obviously, for matrix
the same subspace as the columns of matrix . These vectors are thus a suitableorthonormalized basis. This solution to the symmetric orthonormalization problem
with an orthogonal matrix will dois by no means unique; again, any matrix
quite as well.
However, among these solutions, there is one specific orthogonal matrix that isclosest to matrix (in an appropriate matrix norm). Then this matrix is the orthogonal
projection of onto the set of orthogonal matrices [284]. This is somewhat analogous
to the normalization of one vector ; the vector =k k is the projection of onto
the set of unit-norm vectors (the unit sphere). For matrices, it can be shown that thematrix ( T )?1=2 in Eq. (6.37) is in fact the unique orthogonal projection of
onto this set.
This orthogonalization should be preferred in gradient algorithms that minimize
T= . As explained in Chapter 3,
a function J ( ) under the constraint
one iteration step consists of two parts: first, the matrix
is updated by the usual
gradient descent, and second, the updated matrix is projected orthogonally onto theconstraint set. For this second stage, the form given in (6.37) for orthogonalizing the
updated matrix should be used.
There are iterative methods for symmetric orthonormalization that avoid the matrix
eigendecomposition and inversion. An example is the following iterative algorithm[197], starting from a nonorthogonal matrix (0):

A

U

A

a

a a

a

AA A

W

A

W W

I

W

W

W(1) = W(0)=kW(0)k;W(t + 1) = 23 W(t) ? 21 W(t)W(t)T W(t)(6.38)(6.39)

143

CONCLUDING REMARKS AND REFERENCES

W W
I
W W
W W
W W

The iteration is continued until (t)T (t)  . The convergence of this iteration
can be proven as follows [197]: matrices(t)T (t) and (t + 1)T (t +1) = 49 (t)T (t) ? 23 [ (t)T (t)]2 + 41 [ (t)T (t)]3 have clearly the same
eigenvectors, and the relation between the eigenvalues isWWd(t + 1) = 94 d(t) ? 32 d2 (t) + 41 d3 (t)WW(6.40)

This nonlinear scalar iteration will converge on the interval [0; 1] to 1 (see exercises).
Due to the original normalization, all the eigenvalues are on this interval, assumingthat the norm in the normalization is appropriately chosen (it must be a proper normin the space of matrices; most conventional norms, except for the Frobenius norm,
have this property). Because the eigenvalues tend to 1, the matrix itself tends to theunit matrix.

6.6 CONCLUDING REMARKS AND REFERENCES
Good general discussions on PCA are [14, 109, 324, 112]. The variance maximization criterion of PCA covered in Section 6.1.1 is due to Hotelling [185], while in
the original work by Pearson [364], the starting point was minimizing the squared
reconstruction error (Section 6.1.2). These are not the only criteria leading to the
PCA solution; yet another information-theoretic approach is maximization of mutual
information between the inputs and outputs in a linear gaussian channel [112]. An
expansion closely related to PCA is the Karhunen-Loève expansion for continuous
second-order stochastic processes, whose autocovariance function can be expanded
in terms of its eigenvalues and orthonormal eigenfunctions in a convergent series[237, 283].
The on-line algorithms of Section 6.2 are especially suitable for neural network
implementations. In numerical analysis and signal processing, many other adaptivealgorithms of varying complexity have been reported for different computing hardware. A good review is given by Comon and Golub [92]. Experimental results on
PCA algorithms both for finding the eigenvectors of stationary training sets, and for
tracking the slowly changing eigenvectors of nonstationary input data streams, havebeen reported in [324, 391, 350]. An obvious extension of PCA neural networks
would be to use nonlinear units, e.g., perceptrons, instead of the linear units. It turns
out that such “nonlinear PCA” networks will in some cases give the independent
components of the input vectors, instead of just uncorrelated components [232, 233]
(see Chapter 12).
Good general texts on factor analysis are [166, 243, 454]. The principal FA model
has been recently discussed by [421] and [387].

144

PRINCIPAL COMPONENT ANALYSIS AND WHITENING

Problems

T x (m =6.1 Consider the problem of maximizing the variance of ym = wm
1; :::; n) under the constraint that wm must be of unit Euclidean norm and orthogonalw

to all the previously-found principal vectors i ; i < m. Show that the solution is
given by m = m with m the eigenvector of x corresponding to the mth largest
eigenvalue.

w

e

e

C

6.2 Show that the criterion (6.9) is equivalent to the mean-square error (6.7). Show
that at the optimum, if i = i , the value of (6.7) is given by (6.10).

w

e

6.3 Given the data model (6.14), show that the covariance matrix has the form
(6.15).

w x

6.4 The learning rule for a PCA neuron is based on maximization of y = ( T )2
under constraint k k = 1. (We have now omitted the subscript 1 because only one
neuron is involved.)
6.4.1. Show that an unlimited gradient ascent method would compute the newvector fromw

w

w w + (wT x)x

with the learning rate. Show that the norm of the weight vector always grows inthis case.
6.4.2. Thus the norm must be bounded. A possibility is the following update rule:w [w + (wT x)x]=kw + (wT x)xk
Now the norm will stay equal to 1. Derive an approximation to this update rule for asmall value of , by taking a Taylor expansion of the right-hand side with respect toand dropping all higher powers of . Leave only terms linear in . Show that theresult is

w w + [(wT x)x ? (wT x) w]
2

which is the basic PCA learning rule of Eq. (6.16).
6.4.3. Take averages with respect to the random input vector and show that in astationary point of the iteration, where there is no change on the average in the valueof , it holds: x = ( T x ) with x = Ef T g.
6.4.4. Show that the only possible solutions will be the eigenvectors of x .

x

w

6.5

Cw

w C wwCx is 2:5 1:5 

The covariance matrix of vector

Cx = 1:5 2:5Compute a whitening transformation for x.
6.6

xx

C

(6.41)* Based on the first step (6.38) of the orthogonalization algorithm, show that0 < d(1) < 1 where d(1) is any eigenvalue of W(1). Next consider iteration (6.40).
Write d(t + 1) ? 1 in terms of d(t) ? 1. Show that d(t) converges to 1. How fast is
the convergence?

Part II

BASIC INDEPENDENTCOMPONENT ANALYSIS

7
What is Independent
Component Analysis?
In this chapter, the basic concepts of independent component analysis (ICA) aredefined. We start by discussing a couple of practical applications. These serve as
motivation for the mathematical formulation of ICA, which is given in the form of astatistical estimation problem. Then we consider under what conditions this model
can be estimated, and what exactly can be estimated.
After these basic definitions, we go on to discuss the connection between ICA
and well-known methods that are somewhat similar, namely principal component
analysis (PCA), decorrelation, whitening, and sphering. We show that these methods
do something that is weaker than ICA: they estimate essentially one half of the model.
We show that because of this, ICA is not possible for gaussian variables, since little
can be done in addition to decorrelation for gaussian variables. On the positive side,
we show that whitening is a useful thing to do before performing ICA, because it
does solve one-half of the problem and it is very easy to do.
In this chapter we do not yet consider how the ICA model can actually be estimated.
This is the subject of the next chapters, and in fact the rest of Part II.

7.1 MOTIVATION
Imagine that you are in a room where three people are speaking simultaneously. (Thenumber three is completely arbitrary, it could be anything larger than one.) You also
have three microphones, which you hold in different locations. The microphones give
you three recorded time signals, which we could denote by x1 (t); x2 (t) and x3 (t),with x1 ; x2 and x3 the amplitudes, and t the time index. Each of these recorded
147

148

WHAT IS INDEPENDENT COMPONENT ANALYSIS?
0.5

0

−0.5
0

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

1

0

−1
0
1

0

−1
0

Fig. 7.1 The original audio signals.

signals is a weighted sum of the speech signals emitted by the three speakers, which
we denote by s1 (t); s2 (t), and s3 (t). We could express this as a linear equation:

x1 (t) = a11 s1 (t) + a12 s2 (t) + a13 s3 (t)x2 (t) = a21 s1 (t) + a22 s2 (t) + a23 s3 (t)x3 (t) = a31 s1 (t) + a32 s2 (t) + a33 s3 (t)

(7.1)
(7.2)
(7.3)

where the aij with i; j = 1; :::; 3 are some parameters that depend on the distancesof the microphones from the speakers. It would be very useful if you could now
estimate the original speech signals s1 (t); s2 (t), and s3 (t), using only the recorded
signals xi (t). This is called the cocktail-party problem. For the time being, we omit
any time delays or other extra factors from our simplified mixing model. A moredetailed discussion of the cocktail-party problem can be found later in Section 24.2.
As an illustration, consider the waveforms in Fig. 7.1 and Fig. 7.2. The original
speech signals could look something like those in Fig. 7.1, and the mixed signals
could look like those in Fig. 7.2. The problem is to recover the “source” signals in
Fig. 7.1 using only the data in Fig. 7.2.
Actually, if we knew the mixing parameters aij , we could solve the linear equationin (7.1) simply by inverting the linear system. The point is, however, that here weknow neither the aij nor the si (t), so the problem is considerably more difficult.
One approach to solving this problem would be to use some information on
the statistical properties of the signals si (t) to estimate both the aij and the si (t).
Actually, and perhaps surprisingly, it turns out that it is enough to assume that

MOTIVATION

149

1

0

−1
0

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

2

0

−2
0
2
1
0
−1
0

Fig. 7.2 The observed mixtures of the original signals in Fig. 7.1.

10
5
0
−5
0

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

500

1000

1500

2000

2500

3000

5

0

−5
0
5

0

−5
0

Fig. 7.3 The estimates of the original signals, obtained using only the observed signals in
Fig. 7.2. The original signals were very accurately estimated, up to multiplicative signs.

150

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

s1 (t); s2 (t), and s3 (t) are, at each time instant t, statistically independent. This
is not an unrealistic assumption in many cases, and it need not be exactly true inpractice. Independent component analysis can be used to estimate the aij based on
the information of their independence, and this allows us to separate the three original
signals, s1 (t), s2 (t), and s3 (t), from their mixtures, x1 (t), x2 (t), and x2 (t).
Figure 7.3 gives the three signals estimated by the ICA methods discussed in thenext chapters. As can be seen, these are very close to the original source signals(the signs of some of the signals are reversed, but this has no significance.) Thesesignals were estimated using only the mixtures in Fig. 7.2, together with the very
weak assumption of the independence of the source signals.
Independent component analysis was originally developed to deal with problems
that are closely related to the cocktail-party problem. Since the recent increase ofinterest in ICA, it has become clear that this principle has a lot of other interesting
applications as well, several of which are reviewed in Part IV of this book.
Consider, for example, electrical recordings of brain activity as given by anelectroencephalogram (EEG). The EEG data consists of recordings of electricalpotentials in many different locations on the scalp. These potentials are presumably
generated by mixing some underlying components of brain and muscle activity.
This situation is quite similar to the cocktail-party problem: we would like to findthe original components of brain activity, but we can only observe mixtures of thecomponents. ICA can reveal interesting information on brain activity by giving
access to its independent components. Such applications will be treated in detail in
Chapter 22. Furthermore, finding underlying independent causes is a central concern
in the social sciences, for example, econometrics. ICA can be used as an econometric
tool as well; see Section 24.1.
Another, very different application of ICA is feature extraction. A fundamental
problem in signal processing is to find suitable representations for image, audio orother kind of data for tasks like compression and denoising. Data representations
are often based on (discrete) linear transformations. Standard linear transformations
widely used in image processing are, for example, the Fourier, Haar, and cosine
transforms. Each of them has its own favorable properties.
It would be most useful to estimate the linear transformation from the data itself,
in which case the transform could be ideally adapted to the kind of data that isbeing processed. Figure 7.4 shows the basis functions obtained by ICA from patchesof natural images. Each image window in the set of training images would be
a superposition of these windows so that the coefficient in the superposition are
independent, at least approximately. Feature extraction by ICA will be explained in
more detail in Chapter 21.
All of the applications just described can actually be formulated in a unified
mathematical framework, that of ICA. This framework will be defined in the next
section.

DEFINITION OF INDEPENDENT COMPONENT ANALYSIS

151

Fig. 7.4 Basis functions in ICA of natural images. These basis functions can be consideredas the independent features of images. Every image window is a linear sum of these windows.

7.2 DEFINITION OF INDEPENDENT COMPONENT ANALYSIS
7.2.1

ICA as estimation of a generative model

To rigorously define ICA, we can use a statistical “latent variables” model. We
observe n random variables x1 ; :::; xn , which are modeled as linear combinations of
n random variables s1 ; :::; sn :xi = ai1 s1 + ai2 s2 + ::: + ain sn ; for all i = 1; :::; n(7.4)
where the aij ; i; j = 1; :::; n are some real coefficients. By definition, the si are

statistically mutually independent.
This is the basic ICA model. The ICA model is a generative model, which meansthat it describes how the observed data are generated by a process of mixing thecomponents sj . The independent components sj (often abbreviated as ICs) are latentvariables, meaning that they cannot be directly observed. Also the mixing coefficientsaij are assumed to be unknown. All we observe are the random variables xi , and wemust estimate both the mixing coefficients aij and the ICs si using the i . This must
be done under as general assumptions as possible.
Note that we have here dropped the time index t that was used in the previous
section. This is because in this basic ICA model, we assume that each mixture xi aswell as each independent component sj is a random variable, instead of a proper time
signal or time series. The observed values xi (t), e.g., the microphone signals in the

x

152

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

cocktail party problem, are then a sample of this random variable. We also neglect
any time delays that may occur in the mixing, which is why this basic model is often
called the instantaneous mixing model.
ICA is very closely related to the method called blind source separation (BSS) or
blind signal separation. A “source” means here an original signal, i.e., independent
component, like the speaker in the cocktail-party problem. “Blind” means that weknow very little, if anything, of the mixing matrix, and make very weak assumptions
on the source signals. ICA is one method, perhaps the most widely used, forperforming blind source separation.
It is usually more convenient to use vector-matrix notation instead of the sums
as in the previous equation. Let us denote by the random vector whose elements
are the mixtures x1 ; :::; xn , and likewise by the random vector with elements
s1 ; :::; sn . Let us denote by the matrix with elements aij . (Generally, boldlowercase letters indicate vectors and bold uppercase letters denote matrices.) All
vectors are understood as column vectors; thus T , or the transpose of , is a row
vector. Using this vector-matrix notation, the mixing model is written as

x
s

A

x

x

x = As(7.5)
Sometimes we need the columns of matrix A; if we denote them by aj the model
can also be written as

x=

n
X
i=1

ai si

(7.6)

The definition given here is the most basic one, and in Part II of this book,we will essentially concentrate on this basic definition. Some generalizations andmodifications of the definition will be given later (especially in Part III), however.
For example, in many applications, it would be more realistic to assume that there
is some noise in the measurements, which would mean adding a noise term in themodel (see Chapter 15). For simplicity, we omit any noise terms in the basic model,
since the estimation of the noise-free model is difficult enough in itself, and seems tobe sufficient for many applications. Likewise, in many cases the number of ICs andobserved mixtures may not be equal, which is treated in Section 13.2 and Chapter 16,
and the mixing might be nonlinear, which is considered in Chapter 17. Furthermore,
let us note that an alternative definition of ICA that does not use a generative model
will be given in Chapter 10.
7.2.2

Restrictions in ICA

To make sure that the basic ICA model just given can be estimated, we have to makecertain assumptions and restrictions.
1. The independent components are assumed statistically independent.
This is the principle on which ICA rests. Surprisingly, not much more than thisassumption is needed to ascertain that the model can be estimated. This is why ICA
is such a powerful method with applications in many different areas.

DEFINITION OF INDEPENDENT COMPONENT ANALYSIS

153

Basically, random variables y1 ; y2 ; :::; yn are said to be independent if informationon the value of yi does not give any information on the value of yj for i 6= j .
Technically, independence can be defined by the probability densities. Let us denoteby p(y1 ; y2 ; :::; yn ) the joint probability density function (pdf) of the yi , and by pi (yi )the marginal pdf of yi , i.e., the pdf of yi when it is considered alone. Then we saythat the yi are independent if and only if the joint pdf is factorizable in the followingway:p(y1 ; y2 ; :::; yn ) = p1 (y1 )p2 (y2 ):::pn (yn ):(7.7)

For more details, see Section 2.3.
2. The independent components must have nongaussian distributions.
Intuitively, one can say that the gaussian distributions are “too simple”. The higherorder cumulants are zero for gaussian distributions, but such higher-order information
is essential for estimation of the ICA model, as will be seen in Section 7.4.2. Thus,
ICA is essentially impossible if the observed variables have gaussian distributions.
The case of gaussian components is treated in more detail in Section 7.5 below.
Note that in the basic model we do not assume that we know what the nongaussian
distributions of the ICs look like; if they are known, the problem will be considerably
simplified. Also, note that a completely different class of ICA methods, in which theassumption of nongaussianity is replaced by some assumptions on the time structureof the signals, will be considered later in Chapter 18.
3. For simplicity, we assume that the unknown mixing matrix is square.
In other words, the number of independent components is equal to the number ofobserved mixtures. This assumption can sometimes be relaxed, as explained in
Chapters 13 and 16. We make it here because it simplifies the estimation very much.
Then, after estimating the matrix , we can compute its inverse, say , and obtain
the independent components simply byAB

s = Bx(7.8)

It is also assumed here that the mixing matrix is invertible. If this is not the case,there are redundant mixtures that could be omitted, in which case the matrix would
not be square; then we find again the case where the number of mixtures is not equal
to the number of ICs.
Thus, under the preceding three assumptions (or at the minimum, the two firstones), the ICA model is identifiable, meaning that the mixing matrix and the ICs
can be estimated up to some trivial indeterminacies that will be discussed next. We
will not prove the identifiability of the ICA model here, since the proof is quite
complicated; see the end of the chapter for references. On the other hand, in the next
chapter we develop estimation methods, and the developments there give a kind of anonrigorous, constructive proof of the identifiability.

154

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

7.2.3

Ambiguities of ICA

In the ICA model in Eq. (7.5), it is easy to see that the following ambiguities orindeterminacies will necessarily hold:1. We cannot determine the variances (energies) of the independent components.

s

A

The reason is that, both and being unknown, any scalar multiplier in one of thesources si could always be canceled by dividing the corresponding column i ofby the same scalar, say i :

a

x=

X 1
i

( ai )(si i )

A

(7.9)

i

As a consequence, we may quite as well fix the magnitudes of the independent
components. Since they are random variables, the most natural way to do this is to
assume that each has unit variance: E fs2i g = 1. Then the matrix will be adaptedin the ICA solution methods to take into account this restriction. Note that this stillleaves the ambiguity of the sign: we could multiply an independent component by?1 without affecting the model. This ambiguity is, fortunately, insignificant in mostapplications.

A

2. We cannot determine the order of the independent components.

sAbeing unknown, we can freely change the
The reason is that, again both andorder of the terms in the sum in (7.6), and call any of the independent components
the first one. Formally, a permutation matrix and its inverse can be substituted in?1 . The elements of are the original independent
the model to give =?1 is just a new unknown mixing
variables sj , but in another order. The matrix
matrix, to be solved by the ICA algorithms.

P
Ps
AP

x AP Ps

7.2.4

Centering the variables

Without loss of generality, we can assume that both the mixture variables and the
independent components have zero mean. This assumption simplifies the theory andalgorithms quite a lot; it is made in the rest of this book.
If the assumption of zero mean is not true, we can do some preprocessing to makeit hold. This is possible by centering the observable variables, i.e., subtracting theirsample mean. This means that the original mixtures, say 0 are preprocessed byx = x0 ? E fx0g

x(7.10)

before doing ICA. Thus the independent components are made zero mean as well,since

E fsg = A?1 E fxg(7.11)The mixing matrix, on the other hand, remains the same after this preprocessing, so
we can always do this without affecting the estimation of the mixing matrix. After

ILLUSTRATION OF ICA

155

Fig. 7.5 The joint distribution of the independent components s1 and s2 with uniform
distributions. Horizontal axis: s1 , vertical axis: s2 .

estimating the mixing matrix and the independent components for the zero-meandata, the subtracted mean can be simply reconstructed by adding ?1 E f 0 g to thezero-mean independent components.

Ax

7.3 ILLUSTRATION OF ICA
To illustrate the ICA model in statistical terms, consider two independent componentsthat have the following uniform distributions:p(si ) =

(p
1

2

0;

3

p

; if jsi j  3otherwise(7.12)The range of values for this uniform distribution were chosen so as to make themean zero and the variance equal to one, as was agreed in the previous section. The
joint density of s1 and s2 is then uniform on a square. This follows from the basic
definition that the joint density of two independent variables is just the product of
their marginal densities (see Eq. (7.7)): we simply need to compute the product. The
joint density is illustrated in Fig. 7.5 by showing data points randomly drawn fromthis distribution.
Now let us mix these two independent components. Let us take the following
mixing matrix:

A = 105 102
0(7.13)

156

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

Fig. 7.6 The joint distribution of the observed mixtures x1 and x2 . Horizontal axis: x1 ,vertical axis: x2 . (Not in the same scale as Fig. 7.5.)

This gives us two mixed variables, x1 and x2 . It is easily computed that the mixed
data has a uniform distribution on a parallelogram, as shown in Fig. 7.6. Note thatthe random variables x1 and x2 are not independent anymore; an easy way to see thisis to consider whether it is possible to predict the value of one of them, say x2 , fromthe value of the other. Clearly, if x1 attains one of its maximum or minimum values,
then this completely determines the value of x2 . They are therefore not independent.
(For variables s1 and s2 the situation is different: from Fig. 7.5 it can be seen thatknowing the value of s1 does not in any way help in guessing the value of s2 .)The problem of estimating the data model of ICA is now to estimate the mixing
matrix using only information contained in the mixtures x1 and x2 . Actually, from
Fig. 7.6 you can see an intuitive way of estimating : The edges of the parallelogram
are in the directions of the columns of . This means that we could, in principle,
estimate the ICA model by first estimating the joint density of x1 and x2 , and thenlocating the edges. So, the problem seems to have a solution.
On the other hand, consider a mixture of ICs with a different type of distribution,
called supergaussian (see Section 2.7.1). Supergaussian random variables typicallyhave a pdf with a peak a zero. The marginal distribution of such an IC is given in
Fig. 7.7. The joint distribution of the original independent components is given in
Fig. 7.8, and the mixtures are shown in Fig. 7.9. Here, we see some kind of edges,
but in very different places this time.
In practice, however, locating the edges would be a very poor method because it
only works with variables that have very special distributions. For most distributions,such edges cannot be found; we use only for illustration purposes distributions thatvisually show edges. Moreover, methods based on finding edges, or other similar
heuristic methods, tend to be computationally quite complicated, and unreliable.

A

A

A

ILLUSTRATION OF ICA

157

Fig. 7.7 The density of one supergaussian independent component. The gaussian density ifgive by the dashed line for comparison.

Fig. 7.8 The joint distribution of the independent components s1 and s2 with supergaussian
distributions. Horizontal axis: s1 , vertical axis: s2 .

158

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

Fig. 7.9 The joint distribution of the observed mixtures x1 and x2 , obtained from supergaussian independent components. Horizontal axis: x1 , vertical axis: x2 .

What we need is a method that works for any distributions of the independent
components, and works fast and reliably. Such methods are the main subject of thisbook, and will be presented in Chapters 8–12. In the rest of this chapter, however,
we discuss the connection between ICA and whitening.

7.4 ICA IS STRONGER THAT WHITENING
Given some random variables, it is straightforward to linearly transform them into
uncorrelated variables. Therefore, it would be tempting to try to estimate the independent components by such a method, which is typically called whitening or sphering,and often implemented by principal component analysis. In this section, we show that
this is not possible, and discuss the relation between ICA and decorrelation methods.
It will be seen that whitening is, nevertheless, a useful preprocessing technique for
ICA.
7.4.1

Uncorrelatedness and whitening

A weaker form of independence is uncorrelatedness. Here we review briefly therelevant definitions that were already encountered in Chapter 2.
Two random variables y1 and y2 are said to be uncorrelated, if their covariance iszero:cov(y1 ; y2 ) = E fy1 y2 g ? E fy1 gE fy2 g = 0(7.14)ICA IS STRONGER THAT WHITENING

159

In this book, all random variables are assumed to have zero mean, unless otherwise
mentioned. Thus, covariance is equal to correlation corr (y1 ; y2 ) = E fy1 y2 g, anduncorrelatedness is the same thing as zero correlation (see Section 2.2). 1
If random variables are independent, they are uncorrelated. This is because if they1 and y2 are independent, then for any two functions, h1 and h2 , we haveE fh1(y1 )h2 (y2 )g = E fh1(y1 )gE fh2 (y2 )g(7.15)
see Section 2.3. Taking h1 (y1 ) = y1 and h2 (y2 ) = y2 , we see that this implies

uncorrelatedness.
On the other hand, uncorrelatedness does not imply independence. For example,
assume that (y1 ; y2 ) are discrete valued and follow such a distribution that the pair
are with probability 1=4 equal to any of the following values: (0; 1); (0; ?1); (1; 0);and (?1; 0). Then y1 and y2 are uncorrelated, as can be simply calculated. On theother hand,E fy12y22 g = 0 6= 41 = E fy12gE fy22g(7.16)

so the condition in Eq. (7.15) is violated, and the variables cannot be independent.
A slightly stronger property than uncorrelatedness is whiteness. Whiteness of azero-mean random vector, say , means that its components are uncorrelated andtheir variances equal unity. In other words, the covariance matrix (as well as thecorrelation matrix) of equals the identity matrix:yy

E fyyT g = I(7.17)

Consequently, whitening means that we linearly transform the observed data vector
by linearly multiplying it with some matrixxzVz = Vx(7.18)

so that we obtain a new vector that is white. Whitening is sometimes called
sphering.
A whitening transformation is always possible. Some methods were reviewed in
Chapter 6. One popular method for whitening is to use the eigenvalue decomposition(EVD) of the covariance matrix

E

E fxxT g = EDET

(7.19)

xxD

where is the orthogonal matrix of eigenvectors of E f T g and is the diagonal
matrix of its eigenvalues, = diag(d1 ; :::; dn ). Whitening can now be done by thewhitening matrix

D

V = ED? = ET
1 2

1 In statistical literature, correlation is often defined as a normalized version of covariance.

(7.20)

Here, weuse this simpler definition that is more widely spread in signal processing. In any case, the concept of
uncorrelatedness is the same.

160

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

where the matrix ?1=2 is computed by a simple componentwise operation as
1=2?1=2 = diag(d?1=2 ; :::; d?
n ). A whitening matrix computed this way is denoted
1
T
?
1=2
?
1=2
by E fg or. Alternatively, whitening can be performed in connection
with principal component analysis, which gives a related whitening matrix. For
details, see Chapter 6.

D

7.4.2

D

xx

C

Whitening is only half ICA

Now, suppose that the data in the ICA model is whitened, for example, by the matrix
given in (7.20). Whitening transforms the mixing matrix into a new one, ~ . We havefrom (7.5) and (7.18)

A~
z = VAs = As

(7.21)

One could hope that whitening solves the ICA problem, since whiteness or uncorrelatedness is related to independence. This is, however, not so. Uncorrelatedness
is weaker than independence, and is not in itself sufficient for estimation of the ICA
model. To see this, consider an orthogonal transformation of :y = UzU z(7.22)U, we have
E fyyT g = E fUyyT UT g = UIUT = I

Due to the orthogonality ofyz y(7.23)

In other words, is white as well. Thus, we cannot tell if the independent components
are given by or using the whiteness property alone. Since could be any
orthogonal transformation of , whitening gives the ICs only up to an orthogonaltransformation. This is not sufficient in most applications.
On the other hand, whitening is useful as a preprocessing step in ICA. The utility
is orthogonal.
of whitening resides in the fact that the new mixing matrix ~ =This can be seen from

y

z

A VA

E fzzT g = A~ E fssT gA~ T = A~ A~ T = I(7.24)

This means that we can restrict our search for the mixing matrix to the space oforthogonal matrices. Instead of having to estimate the n2 parameters that are theelements of the original matrix , we only need to estimate an orthogonal mixing
matrix ~ . An orthogonal matrix contains n(n ? 1)=2 degrees of freedom. For
example, in two dimensions, an orthogonal transformation is determined by a single
angle parameter. In larger dimensions, an orthogonal matrix contains only about halfof the number of parameters of an arbitrary matrix.
Thus one can say that whitening solves half of the problem of ICA. Because
whitening is a very simple and standard procedure, much simpler than any ICA
algorithms, it is a good idea to reduce the complexity of the problem this way. The
remaining half of the parameters has to be estimated by some other method; several
will be introduced in the next chapters.

AA

WHY GAUSSIAN VARIABLES ARE FORBIDDEN

161

Fig. 7.10 The joint distribution of the whitened mixtures of uniformly distributed independent components.

A graphical illustration of the effect of whitening can be seen in Fig. 7.10, in
which the data in Fig. 7.6 has been whitened. The square defining the distribution is
now clearly a rotated version of the original square in Fig. 7.10. All that is left is theestimation of a single angle that gives the rotation.
In many chapters of this book, we assume that the data has been preprocessed bywhitening, in which case we denote the data by . Even in cases where whitening
is not explicitly required, it is recommended, since it reduces the number of freeparameters and considerably increases the performance of the methods, especiallywith high-dimensional data.

z

7.5 WHY GAUSSIAN VARIABLES ARE FORBIDDENWhitening also helps us understand why gaussian variables are forbidden in ICA.
Assume that the joint distribution of two ICs, s1 and s2 , is gaussian. This means that
their joint pdf is given byp(s1 ; s2 ) = 21 exp(? s1 +2 s2 ) = 21 exp(? ks2k )2

2

2(7.25)(For more information on the gaussian distribution, see Section 2.5.) Now, assume
that the mixing matrix is orthogonal. For example, we could assume that this isso because the data has been whitened. Using the classic formula of transforming
pdf’s in (2.82), and noting that for an orthogonal matrix ?1 = T holds, we get

A

A

A

162

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

Fig. 7.11 The multivariate distribution of two independent gaussian variables.

the joint density of the mixtures x1 and x2 as density is given byp(x1 ; x2 ) = 21 exp(? kA 2xk )j det AT j
T

AA

(7.26)

A xk = kxk and j det Aj = 1; note that

Due to the orthogonality of , we have k T
if is orthogonal, so is T . Thus we have

A

2

2

2

p(x1 ; x2 ) = 21 exp(? kx2k )2(7.27)

and we see that the orthogonal mixing matrix does not change the pdf, since it does
not appear in this pdf at all. The original and mixed distributions are identical.
Therefore, there is no way how we could infer the mixing matrix from the mixtures.
The phenomenon that the orthogonal mixing matrix cannot be estimated for gaussian variables is related to the property that uncorrelated jointly gaussian variables arenecessarily independent (see Section 2.5). Thus, the information on the independenceof the components does not get us any further than whitening.
Graphically, we can see this phenomenon by plotting the distribution of the orthogonal mixtures, which is in fact the same as the distribution of the ICs. This
distribution is illustrated in Fig. 7.11. The figure shows that the density is rotationally symmetric. Therefore, it does not contain any information on the directions of
the columns of the mixing matrix . This is why cannot be estimated.
Thus, in the case of gaussian independent components, we can only estimate the
ICA model up to an orthogonal transformation. In other words, the matrix is not
identifiable for gaussian independent components. With gaussian variables, all we
can do is whiten the data. There is some choice in the whitening procedure, however;
PCA is the classic choice.

A

A

A

CONCLUDING REMARKS AND REFERENCES

163

What happens if we try to estimate the ICA model and some of the components
are gaussian, some nongaussian? In this case, we can estimate all the nongaussiancomponents, but the gaussian components cannot be separated from each other. In
other words, some of the estimated components will be arbitrary linear combinationsof the gaussian components. Actually, this means that in the case of just one gaussiancomponent, we can estimate the model, because the single gaussian component does
not have any other gaussian components that it could be mixed with.

7.6 CONCLUDING REMARKS AND REFERENCESICA is a very general-purpose statistical technique in which observed random data areexpressed as a linear transform of components that are statistically independent fromeach other. In this chapter, we formulated ICA as the estimation of a generative model,
with independent latent variables. Such a decomposition is identifiable, i.e., welldefined, if the independent components are nongaussian (except for perhaps one). To
simplify the estimation problem, we can begin by whitening the data. This estimatespart of the parameters, but leaves an orthogonal transformation unspecified. Using
the higher-order information contained in nongaussian variables, we can estimate
this orthogonal transformation as well.
Practical methods for estimating the ICA model will be treated in the rest of Part II.
A simple approach based on finding the maxima of nongaussianity is presented firstin Chapter 8. Next, the classic maximum likelihood estimation method is applied on
ICA in Chapter 9. An information-theoretic framework that also shows a connection
between the previous two is given by mutual information in Chapter 10. Somefurther methods are considered in Chapters 11 and 12. Practical considerations on
the application of ICA methods, in particular on the preprocessing of the data, aretreated in Chapter 13. The different ICA methods are compared with each other, andthe choice of the “best” method is considered in Chapter 14, which concludes Part II.
The material that we treated in this chapter can be considered classic. The ICA
model was first defined as herein in [228]; somewhat related developments were given
in [24]. The identifiability is treated in [89, 423]. Whitening was proposed in [61] aswell. In addition to this research in signal processing, a parallel neuroscientific lineof research developed ICA independently. This was started by [26, 27, 28], being
more qualitative in nature. The first quantitative results in this area were proposedin [131], and in [335], a model that is essentially equivalent to the noisy versionof the ICA model (see Chapter 15) was proposed. More on the history of ICA
can be found in Chapter 1, as well as in [227]. For recent reviews on ICA, see[10, 65, 201, 267, 269, 149]. A shorter tutorial text is in [212].

164

WHAT IS INDEPENDENT COMPONENT ANALYSIS?

Problemsx

7.1 Show that given a random vector , there is only one symmetric whitening
matrix for , given by (7.20).

x

7.2 Show that two (zero-mean) random variables that have a jointly gaussian distribution are independent if and only if they are uncorrelated. (Hint: The pdf canbe found in (2.68). Uncorrelatedness means that the covariance matrix is diagonal.
Show that this implies that the joint pdf can be factorized.)xs

7.3 If both and could be observed, how would you estimate the ICA model?
(Assume there is some noise in the data as well.)
7.4 Assume that the data
independent components?

x is multiplied by a matrix M. Does this change the

7.5 In our definition, the signs of the independent components are left undetermined. How could you complement the definition so that they are determined aswell?
7.6 Assume that there are more independent components than observed mixtures.
Assume further that we have been able to estimate the mixing matrix. Can we recover
the values of the independent components? What if there are more observed mixtures
than ICs?
Computer assignments
7.1 Generate samples of two independent components that follow a Laplacian
distribution (see Eq. 2.96). Mix them with three different random mixing matrices.
Plot the distributions of the independent components. Can you see the matrix inthe plots? Do the same for ICs that are obtained by taking absolute values of gaussian
random variables.

A

7.2 Generate samples of two independent gaussian random variables. Mix themwith a random mixing matrix. Compute a whitening matrix. Compute the productof the whitening matrix and the mixing matrix. Show that this is almost orthogonal.
Why is it not exactly orthogonal?

8
ICA by Maximization of
Nongaussianity
In this chapter, we introduce a simple and intuitive principle for estimating themodel of independent component analysis (ICA). This is based on maximization of
nongaussianity.
Nongaussianity is actually of paramount importance in ICA estimation. Without
nongaussianity the estimation is not possible at all, as shown in Section 7.5. Therefore, it is not surprising that nongaussianity could be used as a leading principle in
ICA estimation. This is at the same time probably the main reason for the rather lateresurgence of ICA research: In most of classic statistical theory, random variables are
assumed to have gaussian distributions, thus precluding methods related to ICA. (Acompletely different approach may then be possible, though, using the time structureof the signals; see Chapter 18.)
We start by intuitively motivating the maximization of nongaussianity by thecentral limit theorem. As a first practical measure of nongaussianity, we introduce
the fourth-order cumulant, or kurtosis. Using kurtosis, we derive practical algorithms
by gradient and fixed-point methods. Next, to solve some problems associated withkurtosis, we introduce the information-theoretic quantity called negentropy as analternative measure of nongaussianity, and derive the corresponding algorithms forthis measure. Finally, we discuss the connection between these methods and thetechnique called projection pursuit.

165

166

ICA BY MAXIMIZATION OF NONGAUSSIANITY

8.1 “NONGAUSSIAN IS INDEPENDENT”
The central limit theorem is a classic result in probability theory that was presented in
Section 2.5.2. It says that the distribution of a sum of independent random variables
tends toward a gaussian distribution, under certain conditions. Loosely speaking, asum of two independent random variables usually has a distribution that is closer togaussian than any of the two original random variables.
Let us now assume that the data vector is distributed according to the ICA data
model:xx = As(8.1)

i.e., it is a mixture of independent components. For pedagogical purposes, let us
assume in this motivating section that all the independent components have identical distributions. Estimating the independent components can be accomplished byfinding the right linear combinations of the mixture variables, since we can invert the
mixing as

s = A? x
1

(8.2)

bx P
b As
bA

Thus, to estimate one of the independent components, we can consider a linear
combination of the xi . Let us denote this by y = T = i bi xi , where is avector to be determined. Note that we also have y = T . Thus, y is a certain
linear combination of the si , with coefficients given by T . Let us denote thisvector by . Then we haveqy = bT x = q T s =

b

X
i

A

b

qi si

(8.3)

bx

If were one of the rows of the inverse of , this linear combination T would
actually equal one of the independent components. In that case, the corresponding
would be such that just one of its elements is 1 and all the others are zero.
The question is now: How could we use the central limit theorem to determine
so that it would equal one of the rows of the inverse of ? In practice, we cannot
determine such a exactly, because we have no knowledge of matrix , but we canfind an estimator that gives a good approximation.
Let us vary the coefficients in , and see how the distribution of y = T changes.
The fundamental idea here is that since a sum of even two independent randomvariables is more gaussian than the original variables, y = T is usually more
gaussian than any of the si and becomes least gaussian when it in fact equals one of
the si . (Note that this is strictly true only if the si have identical distributions, as we
assumed here.) In this case, obviously only one of the elements qi of is nonzero.
We do not in practice know the values of , but we do not need to, because
T = T by the definition of . We can just let vary and look at the distributionof T .
Therefore, we could take as a vector that maximizes the nongaussianity of
T . Such a vector would necessarily correspond to a = T , which has only

A

b

A
qs

q

qs

qs bx
bx
bx

q
b

q

q

b

q Ab

q
b

“NONGAUSSIAN IS INDEPENDENT”

bx

167

qs
bx

one nonzero component. This means that y = T = T equals one of the
independent components! Maximizing the nongaussianity of T thus gives us oneof the independent components.
In fact, the optimization landscape for nongaussianity in the n-dimensional spaceof vectors has 2n local maxima, two for each independent component, corresponding to si and ?si (recall that the independent components can be estimated only up
to a multiplicative sign).
We can illustrate the principle of maximizing nongaussianity by simple examples.
Let us consider two independent components that have uniform densities. (They also
have zero mean, as do all the random variables in this book.) Their joint distribution
is illustrated in Fig. 8.1, in which a sample of the independent components is plottedon the two-dimensional (2-D) plane. Figure 8.2 also shows a histogram estimate of
the uniform densities. These variables are then linearly mixed, and the mixtures arewhitened as a preprocessing step. Whitening is explained in Section 7.4; let us recall
briefly that it means that is linearly transformed into a random vectorbxz = Vx = VAs
zz

(8.4)

I

whose correlation matrix equals unity: E f T g = . Thus the ICA model still holds,
though with a different mixing matrix. (Even without whitening, the situation would
be similar.) The joint density of the whitened mixtures is given in Fig. 8.3. It is arotation of the original joint density, as explained in Section 7.4.
Now, let us look at the densities of the two linear mixtures z1 and z2 . These are
estimated in Fig. 8.4. One can clearly see that the densities of the mixtures are closer
to a gaussian density than the densities of the independent components shown in
Fig. 8.2. Thus we see that the mixing makes the variables closer to gaussian. Finding
the rotation that rotates the square in Fig. 8.3 back to the original ICs in Fig. 8.1
would give us the two maximally nongaussian linear combinations with uniform
distributions.
A second example with very different densities shows the same result. In Fig. 8.5,
the joint distribution of very supergaussian independent components is shown. The
marginal density of a component is estimated in Fig. 8.6. The density has a large peakat zero, as is typical of supergaussian densities (see Section 2.7.1 or below). Whitened
mixtures of the independent components are shown in Fig. 8.7. The densities of two
linear mixtures are given in Fig. 8.8. They are clearly more gaussian than the originaldensities, as can be seen from the fact that the peak is much lower. Again, we seethat mixing makes the distributions more gaussian.
To recapitulate, we have formulated ICA estimation as the search for directions thatare maximally nongaussian: Each local maximum gives one independent component.
Our approach here is somewhat heuristic, but it will be seen in the next section andChapter 10 that it has a perfectly rigorous justification. From a practical point ofview, we now have to answer the following questions: How can the nongaussianity of
T be measured? And how can we compute the values of that maximize (locally)such a measure of nongaussianity? The rest of this chapter is devoted to answering
these questions.

bx

b

168

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Fig. 8.1 The joint distribution of two independent components with uniform densities.

Fig. 8.2 The estimated density of one uniform independent component, with the gaussian density (dashed curve) given for comparison.

Fig. 8.3 The joint density of two whitened mixtures of independent components with uniform
densities.

“NONGAUSSIAN IS INDEPENDENT”

169

Fig. 8.4 The marginal densities of the whitened mixtures. They are closer to the gaussian
density (given by the dashed curve) than the densities of the independent components.

Fig. 8.5 The joint distribution of the two
independent components with supergaussian
densities.

Fig. 8.6 The estimated density of one supergaussian independent component.

170

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Fig. 8.7 The joint distribution of two whitened mixtures of independent components withsupergaussian densities.

Fig. 8.8 The marginal densities of the whitened mixtures in Fig. 8.7. They are closer to thegaussian density (given by dashed curve) than the densities of the independent components.

MEASURING NONGAUSSIANITY BY KURTOSIS

171

8.2 MEASURING NONGAUSSIANITY BY KURTOSIS
8.2.1

Extrema of kurtosis give independent components

Kurtosis and its properties To use nongaussianity in ICA estimation, we musthave a quantitative measure of nongaussianity of a random variable, say y . In thissection, we show how to use kurtosis, a classic measure of nongaussianity, for ICA
estimation. Kurtosis is the name given to the fourth-order cumulant of a randomvariable; for a general discussion of cumulants; see Section 2.7. Thus we obtain an
estimation method that can be considered a variant of the classic method of moments;
see Section 4.3.
The kurtosis of y , denoted by kurt(y ), is defined bykurt(y ) = E fy 4 g ? 3(E fy 2 g)2(8.5)

Remember that all the random variables here have zero mean; in the general case, thedefinition of kurtosis is slightly more complicated. To simplify things, we can further
assume that y has been normalized so that its variance is equal to one: E fy 2 g = 1.
Then the right-hand side simplifies to E fy 4 g ? 3. This shows that kurtosis is simply
a normalized version of the fourth moment E fy 4 g. For a gaussian y , the fourth
moment equals 3(E fy 2 g)2 . Thus, kurtosis is zero for a gaussian random variable.
For most (but not quite all) nongaussian random variables, kurtosis is nonzero.
Kurtosis can be both positive or negative. Random variables that have a negative kurtosis are called subgaussian, and those with positive kurtosis are calledsupergaussian. In statistical literature, the corresponding expressions platykurtic andleptokurtic are also used. For details, see Section 2.7.1. Supergaussian randomvariables have typically a “spiky” probability density function (pdf) with heavy tails,i.e., the pdf is relatively large at zero and at large values of the variable, while being
small for intermediate values. A typical example is the Laplacian distribution, whose
pdf is given bypp(y) = p1 exp( 2jyj)
2

(8.6)

Here we have normalized the variance to unity; this pdf is illustrated in Fig. 8.9.
Subgaussian random variables, on the other hand, have typically a “flat” pdf, whichis rather constant near zero, and very small for larger values of the variable. A ty picalexample is the uniform distribution, whose density is given byp(y) =

(p
1

2

0;

3

p

; if jyj  3otherwise(8.7)

which is normalized to unit variance as well; it is illustrated in Fig. 8.10.
Typically nongaussianity is measured by the absolute value of kurtosis. The squareof kurtosis can also be used. These measures are zero for a gaussian variable, andgreater than zero for most nongaussian random variables. There are nongaussian
random variables that have zero kurtosis, but they can be considered to be very rare.

172

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Fig. 8.9 The density function of the Laplacian distribution, which is a typical supergaussian
distribution. For comparison, the gaussian density is given by a dashed curve. Both densities
are normalized to unit variance.

Fig. 8.10 The density function of the uniform distribution, which is a typical subgaussian
distribution. For comparison, the gaussian density is given by a dashed line. Both densities
are normalized to unit variance.

MEASURING NONGAUSSIANITY BY KURTOSIS

173

Kurtosis, or rather its absolute value, has been widely used as a measure of
nongaussianity in ICA and related fields. The main reason is its simplicity, bothcomputational and theoretical. Computationally, kurtosis can be estimated simply
by using the fourth moment of the sample data (if the variance is kept constant).
Theoretical analysis is simplified because of the following linearity property: If x1
and x2 are two independent random variables, it holds
kurt(x1 + x2 ) = kurt(x1 ) + kurt(x2 )

(8.8)

and
kurt( x1 ) =4kurt(x1 )(8.9)

where is a constant. These properties can be easily proven using the generaldefinition of cumulants, see Section 2.7.2.
Optimization landscape in ICA To illustrate in a simple example what the
optimization landscape for kurtosis looks like, and how independent components
could be found by kurtosis minimization or maximization, let us look at a 2-D model= . Assume that the independent components s1 ; s2 have kurtosis values
kurt(s1 ); kurt(s2 ), respectively, both different from zero. Recall that they have unit
variances by definition. We look for one of the independent components as y = T .
Let us again consider the transformed vector = T . Then we have y =T = T= T = q1 s1 + q2 s2 . Now, based on the additive property ofkurtosis, we have

x

bx

As

b As

q

qs

bx

Ab

kurt(y ) = kurt(q1 s1 ) + kurt(q2 s2 ) = q14 kurt(s1 ) + q24 kurt(s2 )(8.10)On the other hand, we made the constraint that the variance of y is equal to 1,
based on the same assumption concerning s1 ; s2 . This implies a constraint on :E fy2 g = q12 + q22 = 1. Geometrically, this means that vector is constrained to theunit circle on the 2-D plane.
The optimization problem is now: What are the maxima of the function jkurt(y )j =jq14 kurt(s1 ) + q24 kurt(s2 )j on the unit circle? To begin with, we may assume forsimplicity that the kurtoses are equal to 1. In this case, we are simply considering
the functionq

F (q) = q14 + q24

q(8.11)

Some contours of this function, i.e., curves in which this function is constant, areshown in Fig. 8.11. The unit sphere, i.e., the set where q12 + q22 = 1, is shown as well.
This gives the "optimization landscape" for the problem.
It is not hard to see that the maxima are at those points where exactly one of
the elements of vector is zero and the other nonzero; because of the unit circle
constraint, the nonzero element must be equal to 1 or ?1. But these points are exactly

q

174

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Fig. 8.11 The optimization landscape of kurtosis. The thick curve is the unit sphere, andthe thin curves are the contours where F in (8.11) is constant.

the ones when y equals one of the independent components si , and the problem
has been solved.
If the kurtoses are both equal to ?1, the situation is similar, because taking
the absolute values, we get exactly the same function to maximize. Finally, if
the kurtoses are completely arbitrary, as long as they are nonzero, more involved
algebraic manipulations show that the absolute value of kurtosis is still maximized
when y = T equals one of the independent components. A proof is given in theexercises.
Now we see the utility of preprocessing by whitening. For whitened data , weseek for a linear combination T that maximizes nongaussianity. This simplifies)T and therefore
the situation here, since we have = (

bx

z
wz
q VA w
kqk = (wT VA)(AT VT w) = kwk(8.12)
This means that constraining q to lie on the unit sphere is equivalent to constraining
w to be on the unit sphere. Thus we maximize the absolute value of kurtosis of
wT z under the simpler constraint that kwk = 1. Also, after whitening, the linear
combinations wT z can be interpreted as projections on the line (that is, a 1-D
subspace) spanned by the vector w. Each point on the unit sphere corresponds to one
22

projection.
As an example, let us consider the whitened mixtures of uniformly distributed
such that the linindependent components in Fig. 8.3. We search for a vector
ear combination or projection T has maximum nongaussianity, as illustrated inFig. 8.12. In this two-dimensional case, we can parameterize the points on the unit
sphere by the angle that the corresponding vector makes with the horizontal axis.
Then, we can plot the kurtosis of T as a function of this angle, which is given in

w

wx

w z

w

175

MEASURING NONGAUSSIANITY BY KURTOSIS

Fig. 8.12 We search for projections (which correspond to points on the unit circle) thatmaximize nongaussianity, using whitened mixtures of uniformly distributed independent components. The projections can be parameterized by the angle.

Fig. 8.13. The plot shows kurtosis is always negative, and is minimized at approximately 1 and 2:6 radians. These directions are thus such that the absolute value ofkurtosis is maximized. They can be seen in Fig. 8.12 to correspond to the directions
given by the edges of the square, and thus they do give the independent components.
In the second example, we see the same phenomenon for whitened mixtures ofsupergaussian independent components. Again, we search for a vector such that the
projection in that direction has maximum nongaussianity, as illustrated in Fig. 8.14.
We can plot the kurtosis of T as a function of the angle in which
points, as
given in Fig. 8.15. The plot shows kurtosis is always positive, and is maximized in
the directions of the independent components. These angles are the same as in thepreceding example because we used the same mixing matrix. Again, they correspondto the directions in which the absolute value of kurtosis is maximized.

w
w

w z

8.2.2

Gradient algorithm using kurtosis

In practice, to maximize the absolute value of kurtosis, we would start from some
vector , compute the direction in which the absolute value of the kurtosis of
y = T is growing most strongly, based on the available sample (1); :::; (T )of mixture vector , and then move the vector
in that direction. This idea isimplemented in gradient methods and their extensions.

w
w z

z

w

z

z

176

ICA BY MAXIMIZATION OF NONGAUSSIANITY

−0.5
−0.6

kurtosis

−0.7
−0.8
−0.9
−1
−1.1
−1.2
−1.3
0

0.5

1

1.5

2

2.5

3

3.5

angle of w

Fig. 8.13 The values of kurtosis for projections as a function of the angle as in Fig. 8.12.
Kurtosis is minimized, and its absolute value maximized, in the directions of the independent
components.

Fig. 8.14 Again, we search for projections that maximize nongaussianity, this time withwhitened mixtures of supergaussian independent components. The projections can be parameterized by the angle.

MEASURING NONGAUSSIANITY BY KURTOSIS

177

5.5
5

kurtosis

4.5
4
3.5
3
2.5
2
0

0.5

1

1.5

2

2.5

3

3.5

angle of w

Fig. 8.15 The values of kurtosis for projections in different angles as in Fig. 8.14. Kurtosis,
as well as its absolute value, is maximized in the directions of the independent components.

Using the principles in Chapter 3, the gradient of the absolute value of kurtosis of

wT z can be simply computed as@ jkurt(wT x)j = 4 sign(kurt(wT z))[E fz(wT z) g ? 3wkwk ]
@w
3

wz

2

(8.13)

w

since for whitened data we have E f( T )2 g = k k2 . Since we are optimizing thisfunction on the unit sphere k k2 = 1, the gradient method must be complemented
by projecting
on the unit sphere after every step. This can be done simply bydividing by its norm.
To further simplify the algorithm, note that since the latter term in brackets in(8.13) would simply be changing the norm of in the gradient algorithm, and notits direction, it can be omitted. This is because only the direction of is interesting,and any change in the norm is insignificant because the norm is normalized to unity
anyway.
Thus we obtain the following gradient algorithm:

w

w

w

w

w / sign(kurt(wT z))E fz(wT z)3 g

w w=kwkw(8.14)(8.15)

An on-line (or adaptive) version of this algorithm can be obtained as well. This ispossible by omitting the second expectation operation in the algorithm, yielding:w / sign(kurt(wT z))z(wT z)3w w=kwkz(8.16)(8.17)

Then every observation (t) can be used in the algorithm at once. However, it
must be noted that when computing sign(kurt( T )), the expectation operator in
the definition of kurtosis cannot be omitted. Instead, the kurtosis must be properly

w x

178

ICA BY MAXIMIZATION OF NONGAUSSIANITY

estimated from a time-average; of course, this time-average can be estimated on-line.
Denoting by the estimate of the kurtosis, we could use

 / ((wT z)4 ? 3) ?

(8.18)

This gives the estimate of kurtosis as a kind of a running average.
Actually, in many cases one knows in advance the nature of the distributions of
the independent components, i.e., whether they are subgaussian or supergaussian.
Then one can simply plug the correct sign of kurtosis in the algorithm, and avoid its
estimation.
More general versions of this gradient algorithm are introduced in Section 8.3.4.
In the next subsection we shall introduce an algorithm that maximizes the absolutevalue of kurtosis much more efficiently than the gradient method.
8.2.3

A fast fixed-point algorithm using kurtosis

In the previous subsection, we derived a gradient method for maximizing nongaussianity as measured by the absolute value of kurtosis. The advantage of such gradientmethods, closely connected to learning in neural networks, is that the inputs (t)can be used in the algorithm at once, thus enabling fast adaptation in a nonstationary
environment. A resulting trade-off, however, is that the convergence is slow, anddepends on a good choice of the learning rate sequence. A bad choice of the learning rate can, in practice, destroy convergence. Therefore, some ways to make thelearning radically faster and more reliable may be needed. The fixed-point iteration
algorithms are such an alternative.
To derive a more efficient fixed-point iteration, we note that at a stable point of
the gradient algorithm, the gradient must point in the direction of , that is, thegradient must be equal to multiplied by some scalar constant. Only in such a case,adding the gradient to does not change its direction, and we can have convergence
is not changed(this means that after normalization to unit norm, the value of
except perhaps by changing its sign). This can be proven more rigorously using
the technique of Lagrange multipliers; see Exercice 3.9. Equating the gradient ofkurtosis in (8.13) with , this means that we should have

z

w

w

w

w

w

w / [E fz(wT z) g ? 3kwk w]
3

2(8.19)

This equation immediately suggests a fixed-point algorithm where we first computethe right-hand side, and give this as the new value for :w
w E fz(wT z) g ? 3w
(8.20)
After every fixed-point iteration, w is divided by its norm to remain on the constraint
set. (Thus kwk = 1 always, which is why it can be omitted from (8.19).) The final
vector w gives one of the independent components as the linear combination wT z.
3

In practice, the expectations in (8.20) must be replaced by their estimates.
Note that convergence of this fixed-point iteration means that the old and newvalues of point in the same direction, i.e., their dot-product is (almost) equal tow

MEASURING NONGAUSSIANITY BY KURTOSIS

179

w

w

1. It is not necessary that the vector converges to a single point, since and ?
define the same direction. This is again because the independent components can be
defined only up to a multiplicative sign.
Actually, it turns out that such an algorithm works very well, converging very fastand reliably. This algorithm is called FastICA [210]. The FastICA algorithm hasa couple of properties that make it clearly superior to the gradient-based algorithms
in most cases. First of all, it can be shown (see Appendix), that the convergenceof this algorithm is cubic. This means very fast convergence. Second, contrary to
gradient-based algorithms, there is no learning rate or other adjustable parameters in
the algorithm, which makes it easy to use, and more reliable. Gradient algorithms
seem to be preferable only in cases where fast adaptation in a changing environmentis necessary.
More sophisticated versions of FastICA are introduced in Section 8.3.5.
8.2.4

ExamplesHere we show what happens when we run the FastICA algorithm that maximizes
the absolute value of kurtosis, using the two example data sets used in this chapter.
First we take a mixture of two uniformly distributed independent components. The
mixtures are whitened, as always in this chapter. The goal is now to find a directionin the data that maximizes the absolute value of kurtosis, as illustrated in Fig. 8.12.
as= (1; 0)T .
We initialize, for purposes of the illustration, the vector
Running the FastICA iteration just two times, we obtain convergence. In Fig. 8.16,
the obtained vectors are shown. The dashed line gives the direction of after the
first iteration, and the solid line gives the direction of after the second iteration.
The third iteration did not significantly change the direction of , which means thatthe algorithm converged. (The corresponding vector is not plotted.) The figure showsthat the value of may change drastically during the iteration, because the values
and ? are considered as equivalent. This is because the sign of the vector cannot
be determined in the ICA model.
The kurtoses of the projections T obtained in the iterations are plotted inFig. 8.17, as a function of iteration count. The plot shows that the algorithm steadily
increased the absolute value of the kurtosis of the projection, until it reached convergence at the third iteration.
Similar experiments were performed for the whitened mixtures of two supergaussian independent components, as illustrated in Fig. 8.14. The obtained vectors areshown in Fig. 8.18. Again, convergence was obtained after two iterations. The
kurtoses of the projections T obtained in the iterations are plotted in Fig. 8.19, asa function of iteration count. As in the preceding experiment, the absolute value ofthe kurtosis of the projection steadily increased, until it reached convergence at thethird iteration.
In these examples, we only estimated one independent component. Of course,
one often needs more than one component. Figures 8.12 and 8.14 indicate how thiscan be done: The directions of the independent components are orthogonal in thewhitened space, so the second independent component can be found as the direction

w

w

w

w

w

w

w

w

w

wz

w z

180

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Fig. 8.16 Result of FastICA using kurtosis, for ICs with uniform distributions. Dashed line:
after the first iteration (plotted longer than actual size). Solid line:
after the second
iteration.

w

w

−0.7

−0.8

kurtosis

−0.9

−1

−1.1

−1.2

−1.3
0

0.5

1

1.5

2

2.5

3

iteration countFig. 8.17 The convergence of FastICA using kurtosis, for ICs with uniform distributions.
The value of kurtosis shown as function of iteration count.

MEASURING NONGAUSSIANITY BY KURTOSIS

181

Fig. 8.18 Result of FastICA with kurtosis, this time for supergaussian ICs. Dash-dotted
line: after the first iteration (plotted longer than actual size). Solid line: after the second
iteration.

w

w

5.5

5

kurtosis

4.5

4

3.5

3

2.5
0

0.5

1

1.5

2

2.5

3

iteration countFig. 8.19 The convergence of FastICA using kurtosis, for supergaussian ICs. The value ofkurtosis shown as a function of iteration count.

w

orthogonal to thecorresponding to the estimated independent component. For
more dimensions, we need to rerun the algorithm, always constraining the current
to be orthogonal to the previously estimated vectors . This will be explained in
more detail in Section 8.4.

w

w

182

ICA BY MAXIMIZATION OF NONGAUSSIANITY

8.3 MEASURING NONGAUSSIANITY BY NEGENTROPY
8.3.1

Critique of kurtosis

In the preceding section, we showed how to measure nongaussianity by kurtosis,
thus obtaining a simple ICA estimation method. However, kurtosis also has somedrawbacks in practice, when its value has to be estimated from a measured sample.
The main problem is that kurtosis can be very sensitive to outliers. Assume, forexample, that a sample of 1000 values of a random variable (with zero mean andunit variance, say) contains one value equal to 10. Then the kurtosis equals at least
104=1000 ? 3 = 7, which means that the single value makes kurtosis large. Thus we
see that the value of kurtosis may depend on only a few observations in the tails of
the distribution, which may be erroneous or irrelevant observations. In other words,
kurtosis is not a robust measure of nongaussianity.
Thus, other measures of nongaussianity might be better than kurtosis in some situations. In this section, we shall consider negentropy, which is the second importantmeasure of nongaussianity. Its properties are in many ways opposite to those of
kurtosis: It is robust but computationally complicated. We also introduce computationally simple approximations of negentropy that more or less combine the good
properties of both measures.
8.3.2

Negentropy as nongaussianity measure

Negentropy is based on the information-theoretic quantity of differential entropy,
which we here call simply entropy. Entropy is the basic concept of information
theory; for a more detailed discussion, see Chapter 5. The entropy of a randomvariable is related to the information that the observation of the variable gives. The
more “random”, i.e., unpredictable and unstructured the variable is, the larger its
entropy. The (differential) entropy H of a random vector with density py ( ) isdefined asyZ

H (y) = ? py () log py ()d(8.21)

A fundamental result of information theory is that a gaussian variable has thelargest entropy among all random variables of equal variance (see Section 5.3.2).
This means that entropy could be used as a measure of nongaussianity. In fact, thisshows that the gaussian distribution is the “most random” or the least structured ofall distributions. Entropy is small for distributions that are clearly concentrated on
certain values, i.e., when the variable is clearly clustered, or has a pdf that is very
“spiky”.
To obtain a measure of nongaussianity that is zero for a gaussian variable andalways nonnegative, one often uses a normalized version of differential entropy,
called negentropy. Negentropy J is defined as follows

J (y) = H (ygauss ) ? H (y)(8.22)

MEASURING NONGAUSSIANITY BY NEGENTROPYy183

where gauss is a gaussian random variable of the same correlation (and covariance)
matrix as . Due to the above-mentioned properties, negentropy is always nonnegative, and it is zero if and only if has a gaussian distribution. Negentropy has the
additional interesting property that it is invariant for invertible linear transformations(see Section 5.4).
The advantage of using negentropy, or equivalently, differential entropy, as ameasure of nongaussianity is that it is well justified by statistical theory. In fact,negentropy is in some sense the optimal estimator of nongaussianity, as far as thestatistical performance is concerned, as will be seen in Section 14.3. The problem
in using negentropy is, however, that it is computationally very difficult. Estimating
negentropy using the definition would require an estimate (possibly nonparametric)of the pdf. Therefore, simpler approximations of negentropy are very useful, as will
be discussed next. These will be used to derive an efficient method for ICA.
8.3.3

y

y

Approximating negentropy

In practice, we only need approximation of 1-D (neg)entropies, so we only consider
the scalar case here.
The classic method of approximating negentropy is using higher-order cumulants,
using the polynomial density expansions as explained in Section 5.5. This gives theapproximation

1 E fy3 g2 + 1 kurt(y)2J (y)  12
48

(8.23)The random variable y is assumed to be of zero mean and unit variance. Actually,
this approximation often leads to the use of kurtosis as in the preceding section.
This is because the first term on the right-hand side of (8.23) is zero in the case of
random variables with (approximately) symmetric distributions, which is quite common. In this case, the approximation in (8.23) is equivalent to the square of kurtosis.
Maximization of the square of kurtosis is of course equivalent to maximization of
its absolute value. Thus this approximation leads more or less to the method in
Section 8.2. In particular, this approximation suffers from the nonrobustness encountered with kurtosis. Therefore, we develop here more sophisticated approximationsof negentropy.
One useful approach is to generalize the higher-order cumulant approximation
so that it uses expectations of general nonquadratic functions, or “nonpolynomial
moments”. This was described in Section 5.6. In general we can replace thepolynomial functions y 3 and y 4 by any other functions Gi (where i is an index,
not a power), possibly more than two. The method then gives a simple way of
approximating the negentropy based on the expectations E fGi (y )g: As a simple
special case, we can take any two nonquadratic functions G1 and G2 so that G1 is
odd and G2 is even, and we obtain the following approximation:

J (y)  k1 (E fG1 (y)g)2 + k2 (E fG2 (y)g ? E fG2 ( )g)2

(8.24)

184

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Fig. 8.20 The functions G1 in Eq. (8.26), G2 in Eq. (8.27), given by the solid curve and thedashed curve, respectively. The fourth power, as used in kurtosis, is given for comparison by
the dash-dotted curve.

where k1 and k2 are positive constants, and  is a gaussian variable of zero mean andunit variance (i.e., standardized). The variable y is assumed to have zero mean andunit variance. Note that even in cases where this approximation is not very accurate,(8.24) can be used to construct a measure of nongaussianity that is consistent in thesense that it is always nonnegative, and equal to zero if y has a gaussian distribution.
This is a generalization of the moment-based approximation in (8.23), which is
obtained by taking G1 (y ) = y 3 and G2 (y ) = y 4 .
In the case where we use only one nonquadratic function G, the approximation
becomes

J (y) / [E fG(y)g ? E fG( )g]2
(8.25)
for practically any nonquadratic function G. This is a generalization of the momentbased approximation in (8.23) if y has a symmetric distribution, in which case the firstterm in (8.23) vanishes. Indeed, taking G(y ) = y 4 , one then obtains a kurtosis-based

approximation.
But the point here is that by choosing G wisely, one obtains approximations ofnegentropy that are better than the one given by (8.23). In particular, choosing a
G that does not grow too fast, one obtains more robust estimators. The following
choices of G have proved very useful:

G1 (y) = a1 log cosh a1 y;
1
G2 (y) = ? exp(?y2=2)(8.26)(8.27)

where 1  a1  2 is some suitable constant, often taken equal to one. The functionsin (8.26)–(8.27) are illustrated in Fig. 8.20.
Thus we obtain approximations of negentropy that give a very good compromise
between the properties of the two classic nongaussianity measures given by kurtosis

MEASURING NONGAUSSIANITY BY NEGENTROPY

185

and negentropy. They are conceptually simple, fast to compute, yet have appealing
statistical properties, especially robustness. Therefore, we shall use these objectivefunctions in our ICA methods. Interestingly, kurtosis can be expressed in this same
framework.
8.3.4

Gradient algorithm using negentropy

Gradient algorithm As with kurtosis, we can derive a simple gradient algorithm
for maximizing negentropy. Taking the gradient of the approximation of negentropyin (8.25) with respect to , and taking the normalization E f( T )2 g = k k2 = 1
into account, one obtains the following algorithmww z

w / E fzg(wT z)gw(8.28)(8.29)

w w=kwk
where= E fG(wT z)g ? E fG( )g,  being a standardized gaussian randomvariable. The normalization is necessary to project w on the unit sphere to keep thevariance of wT z constant. The function g is the derivative of the function G usedin the approximation of negentropy. The expectation could be omitted to obtain an
on-line (adaptive) stochastic gradient algorithm.
The constant, , which gives the algorithm a kind of “self-adaptation” quality, canbe easily estimated on-line as follows: / (G(wT z) ? E fG( )g) ?

(8.30)

This constant corresponds to the sign of kurtosis in (8.13).
As for the function g , we can use the derivatives of the functions in (8.26)–(8.27) that give robust approximations of negentropy. Alternatively, we could use
the derivative corresponding to the fourth power as in kurtosis, which leads to themethod that was already described in the previous section. Thus we can choose from:g1 (y) = tanh(a1 y)
g2 (y) = y exp(?y2 =2)
g3 (y) = y3

(8.31)(8.32)(8.33)

where 1  a1  2 is some suitable constant, often taken as a1 = 1. These functions
are illustrated in Fig. 8.21.
The final form of the on-line stochastic gradient algorithm is summarized on
Table 8.1.
This algorithm can be further simplified. First note that the constant does notchange the stationary points of the learning rule. Its sign does affect their stability,
though. Therefore, one can replace the by its sign without essentially affecting the
behavior of the learning rule. This is useful, for example, in cases where we havesome a priori information on the distributions of the independent components. For
example, speech signals are usually highly supergaussian. One might thus roughly

186

ICA BY MAXIMIZATION OF NONGAUSSIANITY

1. Center the data to make its mean zero.

z

2. Whiten the data to give .
3. Choose an initial (e.g., random) vector.

w of unit norm, and an initial value forw / zg(wT z), where g is defined e.g. as in (8.31)-(8.33)
5. Normalize w
w=kwk
6. If the sign of is not known a priori, update  / (G(wT z) ? E fG( )g) ? .

4. Update 

7. If not converged, go back to step 4.
Table 8.1 The on-line stochastic gradient algorithm for finding one maximally nongaussian
direction, i.e., estimating one independent component.

Fig. 8.21 The robust nonlinearities g1 in Eq. (8.31), g2 in Eq. (8.32), given by the solid line
and the dashed line, respectively. The third power in (8.33), as used in kurtosis-based methods,
is given by the dash-dotted line.

MEASURING NONGAUSSIANITY BY NEGENTROPY

187

evaluate E fG(si ) ? G( )g for some supergaussian independent components andthen take this, or its sign, as the value of . For example, if g is the tanh function,then = ?1 works for supergaussian independent components.
Stability analysis * This section contains a theoretical analysis that can beskipped at first reading.
Since the approximation of negentropy in (8.25) may be rather crude, one may
wonder if the estimator obtained from (8.28) really converges to the direction of oneof the independent components, assuming the ICA data model. It can be proventhat this is so, under rather mild conditions. The key to this proof is the following
theorem, proven in the Appendix:
Theorem 8.1 Assume that the input data follows the ICA model with whitened data:=
where is the whitening matrix, and that G is a sufficiently smooth evenfunction. Then the local maxima (resp. minima) of E fG( T )g under the constraint
k k = 1 include those rows of the mixing matrix such that the corresponding
independent components si satisfy

z VAs
w

V

VA

w z

E fsi g(si ) ? g0 (si )g > 0 (resp. < 0)(8.34)
where g (:) is the derivative of G(:), and g 0 (:) is the derivative of g (:).
This theorem shows that practically any nonquadratic function G may be used
to perform ICA. More precisely, any function G divides the space of probability
distributions into two half-spaces, depending on whether the nonpolynomial momentin the theorem is positive or negative. Independent components whose distribution
is in one of the half-spaces can be estimated by maximizing E fG( T )g, and ICs
whose distribution is in the other half-space can be estimated by minimizing the samefunction. The theorem gives the exact boundary between these two half-spaces.
In particular, this theorem implies the following:

wz

Theorem 8.2 Assume that the input data follows the ICA data model in (8.1), andthat G is a sufficiently smooth even function. Then the asymptotically stable points ofalgorithm in (8.28) include the ith row of the inverse of the whitened mixing matrix
such that the corresponding independent component si fulfills

VA

E fsi g(si ) ? g0 (si )g[E fG(si )g ? E fG( )g] > 0(8.35)
where g (:) is the derivative of G(:), and  is a standardized gaussian variable.
Note that if w equals the ith row of (VA)?1 , the linear combination equals the ith
independent component: wT z = si .
This theorem simply says that the question of stability of the gradient learning ruleof the approximation of negentropy boils down to the question: Does the division into
two half-spaces as given by Theorem 8.1 give the same division that is given by thesign of E fG(si ) ? G( )g? This seems to be approximately true for most reasonablechoices of G, and distributions of the si . In particular, if G(y ) = y 4 , we encounter the188

ICA BY MAXIMIZATION OF NONGAUSSIANITY

kurtosis-based criterion, and the condition is fulfilled for any distribution of nonzero
kurtosis.
Theorem 8.1 also shows how to modify the algorithm in (8.28) so that it is(practically) always stable. This is possible by defining the self-adaptation constantas= sign(E fyg(y) ? g0 (y)g)(8.36)The drawback with this definition is that now the algorithm cannot be interpreted as
optimization of an objective function.
8.3.5

A fast fixed-point algorithm using negentropy

As with kurtosis, a much faster method for maximizing negentropy than that given by
the gradient method, can be found using a fixed-point algorithm. The resulting FastICA algorithm [197] finds a direction, i.e., a unit vector , such that the projection
T maximizes nongaussianity. Nongaussianity is here measured by the approximation of negentropy J ( T ) given in (8.25). Recall that the variance of T musthere be constrained to unity; for whitened data, this is equivalent to constraining thenorm of to be unity.
FastICA is based on a fixed-point iteration scheme for finding a maximum of thenongaussianity of T , as measured in (8.25). More rigorously, it can be derived
as an approximative Newton iteration. The FastICA algorithm using negentropy
combines the superior algorithmic properties resulting from the fixed-point iteration
with the preferable statistical properties due to negentropy.

w

w z

wz

w z

w

wz

Derivation of algorithm * In this subsection, we derive the fixed-point algorithm
using negentropy. This can be skipped by the reader not interested in mathematicaldetails.
Looking at the gradient method in (8.28) immediately suggests the following
fixed-point iteration:

w E fzg(wT z)g(8.37)w

which would of course be followed by normalization of . The coefficient can beomitted because it would be eliminated by the normalization anyway.
The iteration in (8.37) does not, however, have the good convergence propertiesof the FastICA using kurtosis, because the nonpolynomial moments do not have thesame nice algebraic properties as real cumulants like kurtosis. Therefore, the iterationin (8.37) has to be modified. This is possible because we can add , multiplied bysome constant , on both sides of (8.37) without modifying the fixed points. In fact,
we havew

w = E fzg(wT z)g,T(1 + )w = E fzg(w z)g + w(8.38)(8.39)

MEASURING NONGAUSSIANITY BY NEGENTROPY

189

w

and because of the subsequent normalization of to unit norm, the latter equation(8.39) gives a fixed-point iteration that has the same fixed points. Thus, by choosing
wisely, it may be possible to obtain an algorithm that converges as fast as the
fixed-point algorithm using kurtosis. In fact, such a can be found, as we show here.
The suitable coefficient , and thus the FastICA algorithm, can be found using
an approximative Newton method. The Newton method is a fast method for solving
equations; see Chapter 3. When it is applied on the gradient, it gives an optimization
method that usually converges in a small number of steps. The problem with the
Newton method, however, is that it usually requires a matrix inversion at every
step. Therefore, the total computational load may not be smaller than with gradientmethods. What is quite surprising is that using the special properties of the ICA
problem, we can find an approximation of the Newton method that does not need amatrix inversion but still converges roughly with the same number of iterations as thereal Newton method (at least in theory). This approximative Newton method gives afixed-point algorithm of the form (8.39).
To derive the approximative Newton method, first note that the maxima of theapproximation of the negentropy of T are typically obtained at certain optima of
E fG( T )g. According to the Lagrange conditions (see Chapter 3), the optima of
E fG( T )g under the constraint E f( T )2 g = k k2 = 1 are obtained at points
where the gradient of the Lagrangian is zero:

w z
w z

w z
w z

w

E fzg(wT z)g + w = 0(8.40)

Now let us try to solve this equation by Newton’s method, which is equivalent tofinding the optima of the Lagrangian by Newton’s method. Denoting the function
on the left-hand side of (8.40) by F , we obtain its gradient (which is the second
derivative of the Lagrangian) as@F = E fzzT g0 (wT z)g + I@w(8.41)

To simplify the inversion of this matrix, we decide to approximate the first termin (8.41). Since the data is sphered, a reasonable approximation seems to be
E f T g0 ( T )g  E f T gE fg0( T )g = E fg0 ( T )g . Thus the gradientbecomes diagonal, and can easily be inverted. Thus we obtain the following approximative Newton iteration:

zz

wz

zz

w z

w z I

w w ? [E fzg(wT z)g + w]=[E fg0(wT z)g + ](8.42)

This algorithm can be further simplified by multiplying both sides of (8.42) by+ E fg0( T )g. This gives, after straightforward algebraic simplification:

w z

w E fzg(wT z) ? E fg0(wT z)gwg
This is the basic fixed-point iteration in FastICA.

(8.43)190

ICA BY MAXIMIZATION OF NONGAUSSIANITY

1. Center the data to make its mean zero.

z

2. Whiten the data to give .

w of unit norm.
E fzg(wT z)g ? E fg0 (wT z)gw, where g is defined, e.g., as in

3. Choose an initial (e.g., random) vectorw

4. Let(8.31)–(8.33).
5. Letw w=kwk.

6. If not converged, go back to step 4.
Table 8.2 The FastICA algorithm for finding one maximally nongaussian direction, i.e.,estimating one independent component. The expectations are estimated in practice as anaverage over the available data sample.

The fixed-point algorithm The preceding derivation gives us the FastICA algorithm that can be described as follows.
First, we choose a nonlinearity g , which is the derivative of the nonquadratic
function G used in (8.25). For example, we can use the derivatives of the functions in(8.26)–(8.27) that give robust approximations of negentropy. Alternatively, we could
use the derivative corresponding to the fourth power as in kurtosis, which leads tothe method that was already described in the previous section. Thus we can choose
from the same functions in (8.31)–(8.33) as with the gradient algorithm, illustrated
in Fig. 8.21.
Then we use the iteration in (8.43), followed by normalization. Thus, the basic
form of the FastICA algorithm is then as described in Table 8.2.
The functions g 0 can be computed as

g10 (y) = a1 (1 ? tanh2 (a1 y))(8.44)
02
2
g2 (y) = (1 ? y ) exp(?y =2)(8.45)
02
g3(y) = 3y(8.46)
Note that since we have constrained E fy 2 g = 1, the derivative in (8.46) is essentiallyreduced to the constant 3.
Note that as above, convergence means that the old and new values of point in
the same direction, i.e., the absolute value of their dot-product is (almost) equal to 1.
It is not necessary that the vector converges to a single point, since and ? define
the same direction.
As already discussed in connection with kurtosis, FastICA has properties thatmake it clearly superior to the gradient-based algorithms when fast adaptation toa changing environment is not needed. Even when using general approximationsof negentropy, convergence is at least quadratic, which means much faster than thelinear convergence obtained by gradient methods. Moreover, there is no learning rate
or other adjustable parameters in the algorithm, which makes it easy to use, and morewww

MEASURING NONGAUSSIANITY BY NEGENTROPY

191

Fig. 8.22 Results with FastICA using negentropy, for ICs with uniform distributions. Dashedline: after the first iteration (plotter longer than actual size). Solid line: after the second
iteration.

w

w

reliable. Using robust approximations of negentropy instead of kurtosis enhances thestatistical properties of the resulting estimator, as discussed in Section 8.3.1.
The algorithm just given estimates only one independent component. To estimate
more independent components, different kinds of decorrelation schemes should be
used; see Section 8.4.
Examples Here we show what happens when we run this version of the FastICAalgorithm that maximizes the negentropy, using the two example data sets usedin this chapter. First we take a mixture of two uniformly distributed independent
components. The mixtures are whitened, as always in this chapter. The goal is now
to find a direction in the data that maximizes the negentropy, as illustrated in 8.12.
For purposes of the illustration,we initialize the vector as = (1; 0)T . Running
the FastICA iteration just two times (using the tanh nonlinearity in 8.31)), we obtain
convergence. In Fig. 8.22, the obtained vectors are shown. The dashed line gives
the direction of after the first iteration, and the solid line gives the direction ofafter the second iteration. The third iteration didn’t make any significant change inthe direction of , which means that the algorithm converged. (The corresponding
vector in not plotted.) The figure shows that the value of may change drasticallyduring the iteration, because the values and ? are considered as equivalent. Thisis because the sign of the vector cannot be determined in the ICA model.
The negentropies of the projections T obtained in the iterations are plotted inFig. 8.23, as a function of iteration count. The plot shows that the algorithm steadily
increased the negentropy of the projection, until it reached convergence at the third
iteration.

w w

w
w

w

w
w
w z

w

w

192

ICA BY MAXIMIZATION OF NONGAUSSIANITY
−4

2.2

x 10

2
1.8

negentropy

1.6
1.4
1.2
1
0.8
0.6
0.4
0.2
0

0.5

1

1.5

2

2.5

3

iteration countFig. 8.23 The convergence of FastICA using negentropy, for ICs with uniform distributions.
The value of negentropy shown as a function of iteration count. (Note that the value ofnegentropy was not properly scaled because a multiplying constant was omitted.)

Similar experiments were performed for the whitened mixtures of two supergaussian independent components, as illustrated in Fig. 8.14. In Fig. 8.24, the obtained
vectors are shown. Convergence was obtained after three iterations. The negentropies of the projections T obtained in the iterations are plotted in Fig. 8.25, asa function of iteration count. As earlier, the negentropy of the projection steadily
increased, until it reached convergence at the third iteration.
In these examples, we only estimated one independent component. In practice,
we have many more dimensions and, therefore, we usually want to estimate morethan one independent component. This can be done using a decorrelation scheme, as
will be discussed next.

w z

8.4 ESTIMATING SEVERAL INDEPENDENT COMPONENTS
8.4.1

Constraint of uncorrelatedness

In this chapter, we have so far estimated only one independent component. This is
why these algorithms are sometimes called “one-unit” algorithms. In principle, we
could find more independent components by running the algorithm many times andusing different initial points. This would not be a reliable method of estimating many
independent components, however.
The key to extending the method of maximum nongaussianity to estimate more
independent component is based on the following property: The vectors i corresponding to different independent components are orthogonal in the whitened
space, as shown in Chapter 7. To recapitulate, the independence of the components requires that they are uncorrelated, and in the whitened space we have
E f( iT )( jT )g = iT j , and therefore uncorrelatedness in equivalent to orthog-w

wz w zw w

193

ESTIMATING SEVERAL INDEPENDENT COMPONENTS

w

Fig. 8.24 Results with FastICA using negentropy, second experiment. Dashed line: after
the first iteration (plotted longer than actual size). Solid line: after the second iteration.

w

−3

6.5

x 10

6

negentropy

5.5
5
4.5
4
3.5
3
2.5
0

0.5

1

1.5

2

2.5

3

iteration countFig. 8.25 The convergence of FastICA using negentropy, for supergaussian ICs. The valueof negentropy shown as function of iteration count. (Again, note that the value of negentropy
was not properly scaled.)

194

ICA BY MAXIMIZATION OF NONGAUSSIANITY

onality. This property is a direct consequence of the fact that after whitening, themixing matrix can be taken to be orthogonal. The i are in fact by definition therows of the inverse of the mixing matrix, and these are equal to the columns of themixing matrix, because by orthogonality ?1 = T .
Thus, to estimate several independent components, we need to run any of the oneunit algorithms several times (possibly using several units) with vectors 1 ; :::; n ,and to prevent different vectors from converging to the same maxima we must
orthogonalize the vectors 1; :::; n after every iteration. We present in the following
different methods for achieving decorrelation.

w

A

A

w

w

8.4.2

w

w

Deflationary orthogonalization

A simple way of orthogonalization is deflationary orthogonalization using the GramSchmidt method. This means that we estimate the independent components one byone. When we have estimated p independent components, or p vectors 1 ; :::; p ,we run any one-unit algorithm for p+1 , and after every iteration step subtractfrom p+1 the projections ( pT+1 j ) j ; j = 1; :::; p of the previously estimatedp vectors, and then renormalize p+1 . More precisely, we alternate the following
steps:

w

w
w w w
w

w

1. Choose m, the number of ICs to estimate. Set p
2. Initialize

wp (e.g. randomly)

3. Do an iteration of a one-unit algorithm on

w

1.

wp .

4. Do the following orthogonalization:

wp w p ?
5. Normalize
6. If

pX
?1
j =1

(wpT wj )wj(8.47)

wp by dividing it by its norm.

wp has not converged, go back to step 3.

7. Set p
step 2.

p + 1. If p is not greater than the desired number of ICs, go back to

In particular, we give the FastICA algorithm with deflationary orthogonalizationin Table 8.3.
8.4.3

Symmetric orthogonalization

In certain applications, it may be desirable to use a symmetric decorrelation, in
which no vectors are “privileged” over others. This means that the vectors i are
not estimated one by one; instead, they are estimated in parallel. One motivation forthis is that the deflationary method has the drawback that estimation errors in the firstw

ESTIMATING SEVERAL INDEPENDENT COMPONENTS

195

vectors are cumulated in the subsequent ones by the orthogonalization. Another one
is that the symmetric orthogonalization methods enable parallel computation of ICs.
Symmetric orthogonalization is done by first doing the iterative step of the oneunit algorithm on every vector i in parallel, and afterwards orthogonalizing all thei by special symmetric methods. In other words:w

w

1. Choose the number of independent components to estimate, say m.
2. Initialize thewi; i = 1; :::; m (e.g., randomly).

wi in parallel.
4. Do a symmetric orthogonalization of the matrix W = (w ; :::; wm )T .
3. Do an iteration of a one-unit algorithm on every

1

5. If not converged, go back to step 3.
In Chapter 6, methods for symmetric orthogonalization were discussed. The
can be accomplished, e.g., by the classic method
symmetric orthogonalization of
involving matrix square roots,W

W (WWT )? = W(8.48)The inverse square root (WWT )? = is obtained from the eigenvalue decompositionof WWT = E diag(d ; :::; dm ) ET as(WWT )? = = E diag(d? = ; :::; d?m = ) ET
(8.49)
1 2

1 2

1

1 2

1

1 2

1 2

A simpler alternative is the following iterative algorithm:

W W=kWk.
2. Let W
W ? WWT W.
3. If WWT is not close enough to identity, go back to step 2.
1. Let3
2

1
2

The norm in step 1 can be almost any ordinary matrix norm, e.g., the 2-norm or thelargest absolute row or column sum (but not the Frobenius norm); see Section 6.5 fordetails.
We give a detailed version of the FastICA algorithm that uses the symmetricorthogonalization in Table 8.4.

196

ICA BY MAXIMIZATION OF NONGAUSSIANITY

1. Center the data to make its mean zero.

z

2. Whiten the data to give .
3. Choose m, the number of ICs to estimate. Set counter p

1.

wp, e.g., randomly.
E fzg(wpT z)g ? E fg0(wpT z)gw, where g is defined, e.g., as in

4. Choose an initial value of unit norm forw

5. Let p(8.31)–(8.33).

6. Do the following orthogonalization:

wp w p ?

pX
?1
j =1

(wpT wj )wj(8.50)

wp wp =kwpk.
8. If wp has not converged, go back to step 5.
9. Set p
p + 1. If p  m, go back to step 4.
7. LetTable 8.3 The FastICA algorithm for estimating several ICs, with deflationary orthogonalization. The expectations are estimated in practice as sample averages.

1. Center the data to make its mean zero.

z

2. Whiten the data to give .
3. Choose m, the number of independent components to estimate.

w

4. Choose initial values for the i ; i = 1; :::; m, each of unit norm. Orthogonalize
as in step 6 below.
the matrix

W

5. For every i = 1; :::; m, let i
E f g( iT )g ? E fg0( iT )g , where g isdefined, e.g., as in (8.31)–(8.33).

w

z w z

w z w

W = (w ; :::; wm)T by
W (WWT )? = W;
(8.51)

6. Do a symmetric orthogonalization of the matrix

1

1 2

or by the iterative algorithm in Sec. 8.4.3.
7. If not converged, go back to step 5.
Table 8.4 The FastICA algorithm for estimating several ICs, with symmetric orthogonalization. The expectations are estimated in practice as sample averages.

ICA AND PROJECTION PURSUIT

197

8.5 ICA AND PROJECTION PURSUIT
It is interesting to note how the approach to ICA described in this Chapter makes
explicit the connection between ICA and another technique: projection pursuit.
8.5.1

Searching for interesting directions

Projection pursuit is a technique developed in statistics for finding “interesting”
projections of multidimensional data. Such projections can then be used for optimal
visualization of the data, and for such purposes as density estimation and regression.
When projection pursuit is used for exploratory data analysis, we usually computea couple of the most interesting 1-D projections. (The definition of interestingness
will be treated in the next section.) Some structure of the data can then be visualizedby showing the distribution of the data in the 1-D subspaces, or on 2-D planes spannedby two of the projection pursuit directions. This method is en extension of the classic
method of using principal component analysis (PCA) for visualization, in which the
distribution of the data is shown on the plane spanned by the two first principal
components.
An example of the problem can be seen in Fig. 8.26. In reality, projection pursuit
is of course used in situations where the number of dimensions is very large, but
for purposes of illustration, we use here a trivial 2-D example. In the figure, theinteresting projection of the data would be on the horizontal axis. This is becausethat projection shows the clustering structure of the data. In contrast, projections
in very different directions (here, projection on the vertical axis) would show only
an ordinary gaussian distribution. It would thus be useful to have a method that
automatically finds the horizontal projection in this example.
8.5.2

Nongaussian is interesting

The basic question in projection pursuit is thus to define what kind of projections areinteresting.
It is usually argued that the gaussian distribution is the least interesting one, andthat the most interesting directions are those that show the least gaussian distribution.
One motivation for this is that distributions that are multimodal, i.e., show someclustering structure, are far from gaussian.
An information-theoretic motivation for nongaussianity is that entropy is maximized by the gaussian distribution, and entropy can be considered as a measure of
the lack of structure (see Chapter 5). This is related to the interpretation of entropy
as code length: a variable that has a clear structure is usually easy to code. Thus,
since the gaussian distribution has the largest entropy, it is the most difficult to code,and therefore it can be considered as the least structured.
The usefulness of using the most nongaussian projections for visualization can beseen in Fig. 8.26. Here the most nongaussian projection is on the horizontal axis; thisis also the projection that most clearly shows the clustered structure of the data. On

198

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Fig. 8.26 An illustration of projection pursuit and the “interesting” directions. The data inthis figure is clearly divided into two clusters. The goal in projection pursuit is to find the
projection (here, on the horizontal axis) that reveals the clustering or other structure of thedata.

the other hand, the projection on the vertical direction, which is also the direction of
the first principal component, fails to show this structure. This also shows that PCA
does not use the clustering structure. In fact, clustering structure is not visible in thecovariance or correlation matrix on which PCA is based.
Thus projection pursuit is usually performed by finding the most nongaussian
projections of the data. This is the same thing that we did in this chapter to estimate the
ICA model. This means that all the nongaussianity measures and the corresponding
ICA algorithms presented in this chapter could also be called projection pursuit“indices” and algorithms.
It should be noted that in the formulation of projection pursuit, no data modelor assumption about independent components is made. If the ICA model holds,
optimizing the ICA nongaussianity measures produce independent components; if
the model does not hold, then what we get are the projection pursuit directions.

8.6 CONCLUDING REMARKS AND REFERENCES
A fundamental approach to ICA is given by the principle of nongaussianity. The
independent components can be found by finding directions in which the data is
maximally nongaussian. Nongaussianity can be measured by entropy-based measures or cumulant-based measures like kurtosis. Estimation of the ICA model canthen be performed by maximizing such nongaussianity measures; this can be doneby gradient methods or by fixed-point algorithms. Several independent components
can be found by finding several directions of maximum nongaussianity under theconstraint of decorrelation.

PROBLEMS

199

This approach is closely connected to projection pursuit, in which the maximallynongaussian directions are considered interesting from the viewpoint of visualizationand exploratory data analysis [222, 137, 138, 95, 160, 151, 316, 414, 139]; a usefulreview is provided in [189]. From a modeling viewpoint, this approach was firstdeveloped in the context of blind deconvolution [114, 399]. Blind deconvolution,
discussed in more detail in Chapter 19, is a linear model not unlike the ICA model,
but the mixing is done by convolution of a one-dimensional signal. In the context
of ICA, the principle of nongaussianity was probably first used in [107], where themaximality property of kurtosis was proven rigorously, and further developed in[197, 210, 211, 291], on which this chapter is based.
Problems
8.1 Prove (8.8) and (8.9)
8.1.1. by algebraically manipulating the formulas
8.1.2. by using the general definition of cumulants (see Section 2.7.2).
8.2

Derive the gradient in (8.13).

8.3

Derive the gradient in (8.28).

8.4 What happens to the ICA algorithms in (8.28) and (8.43) if the nonlinearity gis taken to be linear?
8.5 How does the behavior of the algorithms in (8.28) and (8.43) change if
8.5.1. a linear function is added to g ?
8.5.2. a constant is added to g ?

8.6 Derive a fixed-point algorithm for the third-order cumulant: E fy 3 g. When
could this algorithm be useful? Why is kurtosis preferred in most applications?
8.7 In this exercice, we prove the fundamental maximality property of kurtosis.
More precisely, we prove that the maxima of the function

F (q) = jkurt(qT s)j = jq14 kurt(s1 ) + q24 kurt(s2 )jq(8.52)

qin the constraint set k k2 = 1 are obtained when only one of the components of isnonzero. For simplicity, we consider here the 2-D case first.
8.7.1. Make the change of variables ti = qi2 . What is the geometrical form of theconstraint set of = (t1 ; t2 )? Note that the objective function is now quadratic.
8.7.2. Assume that both of the kurtoses are positive. What is the geometrical shapeof the sets F ( ) = const.? By a geometrical argument, show that the maximum ofF ( ) is obtained when one of the ti is one and the other one is zero. Show how thisproves the maximality property if the kurtoses are both positive.
8.7.3. Assume that both kurtoses are negative. Using exactly the same logic as in
the preceding point, show that the maximality property holds if the kurtoses are both
negative.

t

t

t

200

ICA BY MAXIMIZATION OF NONGAUSSIANITY

8.7.4. Assume that the kurtoses have different signs. What is the geometricalshape of the sets F ( ) = const. now? By geometrical arguments, show the maximality property holds even in this case.
8.7.5. Let us redo the proof algebraically. Express t2 as a function of t1 , andreformulate the problem. Solve it explicitly.

t

8.8 * Now we extend the preceding geometric proof the n dimensions. We willneed some basic concepts of convex analysis [284].
8.8.1. Make the same change of variables. Prove that the constraint set is convex(in fact, it is what is called a simplex).
8.8.2. Assume that the kurtoses of the si are all positive. Show that the objectivefunction is strictly convex.
8.8.3. Show that a strictly convex function defined on a simplex obtains its maxima
in the extremal points.
8.8.4. Show that our objective function is maximized at the points where just oneof the ti is 1 and the others are zero, and that these correspond to the independent
components.
8.8.5. Show the same when all the kurtoses are negative.
8.8.6. In the case of kurtoses of different signs, first show by a simple argumentthat if ti and tj corresponding to components with kurtoses of different signs arenonzero, the objective function can be increased by reducing one of them and increasing the other by the same amount. Conclude that in the maximum, only two of
the ti , corresponding to two different signs of kurtoses, can be nonzero. Show thatthe problem then reduces to what was already shown in the 2-D case.
Computer assignments
8.1 In this computer assignment, the central limit theorem is studied experimentally. Let x(t), t = 1; : : : ; T , be T independent random numbers distributed uniformly on the interval [?1; 1], and

y=

T
X
t=1

x(t)

their sum. Generate 5000 different realizations of the random variable y for thefollowing numbers of terms in the sum: T = 2, T = 4, and T = 12.
8.1.1. Plot the experimental pdf’s of y , and compare it with the gaussian pdf
having the same (zero) mean and variance. (Hint: you can here estimate the pdf
from the generated samples simply by dividing their value range into small bins ofwidth 0.1 or 0.05, count the number of samples falling into each bin, and divide by
the total number of samples. You can compute the difference between the respective
gaussian and the estimated density to get a better idea of the similarity of the two
distributions.)
8.1.2. Plot the kurtoses in each case. Note that you must normalize all thevariables to unit variance. What if you don’t normalize?

APPENDIX

201

8.2 Program the FastICA algorithm in (8.4) in some computer environment.
8.2.1. Take the data x(t) in the preceding assignment as two independent components by splitting the sample in two. Mix them using a random mixing matrix, andestimate the model, using one of the nonlinearity in (8.31).
8.2.2. Reduce the sample size to 100. Estimate the mixing matrix again. Whatdo you see?
8.2.3. Try the different nonlinearities in (8.32)–(8.33). Do you see any difference?
8.2.4. Try the nonlinearity g (u) = u2 . Why does this not work?
Appendix proofs
Proof of Theorem 8.1 Denote by H (w) the function to be minimized/maximized,
E fG(wT z)g. Make the orthogonal change of coordinates q = A2T VT w. Then we can calculate the gradient as @H@ q(q) = E fsg (qT s)g and the Hessian as @ @Hq(2q) = E fssT g 0 (qT s)g.

q e
q e

Without loss of generality, it is enough to analyze the stability of the point = 1 , where= 1 , we get using
1 = (1; 0; 0; 0; :::). Evaluating the gradient and the Hessian at point
the independence of the si ,e

@H (e1 ) = e E fs g(s )g
1
1
1
@q

(A.1)

and

@ 2 H (e1) = diag(E fs2 g0 (s )g; E fg0 (s )g; E fg0 (s )g; :::):
1
1
1
1
@ q2
Making a small perturbation  = (1 ; 2 ; :::), we obtain2
H (e1 + ) = H (e1) + T @H@(qe1) + 12 T @ @Hq(2e1)  + o(kk2 )
X2
1
2 0
2
0
= H (e1 ) + E fs1 g (s1 )g1 + [E fs1 g (s1 )g1 + E fg (s1 )g
i ] + o(kk2 )
2
i>1

p1 ?

Due to the constraint= 1?P

kwk = 1 we get  =
1

(A.2)

(A.3)

p1 ?  ?  ? ::: ? 1. Due to the fact that
2
2

2
3

? =2 + o( ), the term of order  in (A.3) is o(kk ), i.e., of higher order,212

and can be neglected. Using the aforementioned first-order approximation for 1 we obtain22
1 =i>1 i =2 + o(  ), which finally giveskk

H (e1 + ) = H (e1 ) + 12 [E fg0 (s1 ) ? s1 g(s1 )g]
which clearly proves
theorem.

Xi>1

2i + o(kk2 )(A.4)

q = e is an extremum, and of the type implied by the condition of the1

Proof of convergence of FastICA The convergence is proven under the assumptions
that first, the data follows the ICA data model (8.1) and second, that the expectations areevaluated exactly.

202

ICA BY MAXIMIZATION OF NONGAUSSIANITY

Let g be the nonlinearity used in the algorithm. In the case of the kurtosis-based algorithm
in Section 8.2.3, this is the cubic function, so we obtain that algorithm as a special case of thefollowing proof for a general g . We must also make the following technical assumption:

E fsi g(si) ? g0 (si )g 6= 0; for any i(A.5)

which can be considered a generalization of the condition valid when we use kurtosis, thatthe kurtosis of the independent components must be nonzero. If (A.5) is true for a subset ofindependent components, we can estimate just those independent components.
To begin with, make the change of variable = T T , as earlier, and assume thatis in the neighborhood of a solution (say, q1
1 as before). As shown in the proof of
Theorem 8.1, the change in q1 is then of a lower order than the change in the other coordinates,= 1. Then we can expand the terms in (8.43) using a Taylor
due to the constraint
approximation for g and g 0 , first obtaining

q AVw

q



kqk

g(qT s) = g(q1 s1 ) + g0 (q1 s1 )qT?1 s?1 + 21 g00 (q1 s1 )(qT?1 s?1 )2
T
3
4
1 000
+ 6 g (q1 s1 )(q?1 s?1 ) + O (kq?1 k )(A.6)

and theng0 (qT s) =

q

s

1
+2

g0 (q1 s1 ) + g00 (q1 s1 )qT?1 s?1000
g (q1 s1 )(qT?1 s?1 )2 + O(kq?1 k3 )

q

(A.7)

s

q

where ?1 and ?1 are the vectors and without their first components. Denote by + the
new value of (after one iteration). Thus we obtain, using the independence of the si anddoing some tedious but straightforward algebraic manipulations,

q

qi+ =

q1+ = E fs1 g(q1 s1 ) ? g0 (q1 s1 )g + O(kq?1 k2 )(A.8)

E fs3i gE fg00 (s1 )gqi2000
34+ 1 kurt (si )E fg (s1 )gqi + O (kq?1 (k ); for i > 1

(A.9)

1
2

6

We obtain alsoq = q =kq k++(A.10)

This shows clearly that under assumption (A.5), the algorithm converges (locally) to such
T ?1 converges,
a vector that q1 = 1 and qi = 0 for i > 1. This means that = (() ), which implies thatup to the sign, to one of the rows of the inverse of the mixing matrix
T converges to one of the si . Moreover, if E g 00 (s1 ) = 0, i.e., if the si has a symmetric
distribution, as is usually the case, (A.9) shows that the convergence is cubic. In other cases,
the convergence is quadratic. If kurtosis is used, however, we always have E g 00 (s1 ) = 0
and thus cubic convergence. In addition, if G(y ) = y 4 , the local approximations are exact,
and the convergence is global.

q

wz

w



f

g

VA
VA

q

f

g

9
ICA by Maximum
Likelihood Estimation
A very popular approach for estimating the independent component analysis (ICA)
model is maximum likelihood (ML) estimation. Maximum likelihood estimation isa fundamental method of statistical estimation; a short introduction was provided in
Section 4.5. One interpretation of ML estimation is that we take those parametervalues as estimates that give the highest probability for the observations. In thissection, we show how to apply ML estimation to ICA estimation. We also show its
close connection to the neural network principle of maximization of information flow
(infomax).

9.1 THE LIKELIHOOD OF THE ICA MODEL
9.1.1

Deriving the likelihood

It is not difficult to derive the likelihood in the noise-free ICA model. This is basedon using the well-known result on the density of a linear transform, given in (2.82).
According to this result, the density px of the mixture vector

x = As(9.1)

can be formulated aspx(x) = j det Bjps (s) = j det BjYipi (si )

(9.2)
203

204

ICA BY MAXIMUM LIKELIHOOD ESTIMATION= ?1 , and the pi denote the densities of the independent components.
whereThis can be expressed as a function of = ( 1 ; :::; n )T and , giving

B A

B

b

b
Y
px(x) = j det Bj pi (bTi x)

x

(9.3)

i

x

x

x

x
B

Assume that we have T observations of , denoted by (1); (2); :::; (T ). Then thelikelihood can be obtained (see Section 4.5) as the product of this density evaluatedat the T points. This is denoted by L and considered as a function of :L(B) =T YnY
t=1 i=1

pi (bTi x(t))j det Bj(9.4)

Very often it is more practical to use the logarithm of the likelihood, since it isalgebraically simpler. This does not make any difference here since the maximum of
the logarithm is obtained at the same point as the maximum of the likelihood. The
log-likelihood is given bylog L(B) =T X
nX
t=1 i=1

log pi (bTi x(t)) + T log j det Bj(9.5)

The basis of the logarithm makes no difference, though in the following the natural
logarithm is used.
To simplify notation and to make it consistent to what was used in the previous
chapter, we can denote the sum over the sample index t by an expectation operator,and divide the likelihood by T to obtainn
1 log L(B) = E fX
log p (bT x)g + log j det Bj

T

i=1

i i

(9.6)

The expectation here is not the theoretical expectation, but an average computed fromthe observed sample. Of course, in the algorithms the expectations are eventuallyreplaced by sample averages, so the distinction is purely theoretical.
9.1.2

Estimation of the densities

Problem of semiparametric estimation In the preceding, we have expressed
the likelihood as a function of the parameters of the model, which are the elementsof the mixing matrix. For simplicity, we used the elements of the inverse of themixing matrix. This is allowed since the mixing matrix can be directly computedfrom its inverse.
There is another thing to estimate in the ICA model, though. This is the densities of
the independent components. Actually, the likelihood is a function of these densities
as well. This makes the problem much more complicated, because the estimationof densities is, in general, a nonparametric problem. Nonparametric means that it

BTHE LIKELIHOOD OF THE ICA MODEL

205

cannot be reduced to the estimation of a finite parameter set. In fact the number ofparameters to be estimated is infinite, or in practice, very large. Thus the estimationof the ICA model has also a nonparametric part, which is why the estimation issometimes called “semiparametric”.
Nonparametric estimation of densities is known to be a difficult problem. Many
parameters are always more difficult to estimate than just a few; since nonparametricproblems have an infinite number of parameters, they are the most difficult to estimate.
This is why we would like to avoid the nonparametric density estimation in the ICA.
There are two ways to avoid it.
First, in some cases we might know the densities of the independent components
in advance, using some prior knowledge on the data at hand. In this case, we could
simply use these prior densities in the likelihood. Then the likelihood would reallybe a function of only. If reasonably small errors in the specification of these priordensities have little influence on the estimator, this procedure will give reasonable
results. In fact, it will be shown below that this is the case.
A second way to solve the problem of density estimation is to approximate thedensities of the independent components by a family of densities that are specified
by a limited number of parameters. If the number of parameters in the density familyneeds to be very large, we do not gain much from this approach, since the goal wasto reduce the number of parameters to be estimated. However, if it is possible to use
a very simple family of densities to estimate the ICA model for any densities pi , wewill get a simple solution. Fortunately, this turns out to be the case. We can use an
extremely simple parameterization of the pi , consisting of the choice between two
densities, i.e., a single binary parameter.

B

A simple density family It turns out that in maximum likelihood estimation, it isenough to use just two approximations of the density of an independent component.
For each independent component, we just need to determine which one of the two
approximations is better. This shows that, first, we can make small errors when we
fix the densities of the independent components, since it is enough that we use adensity that is in the same half of the space of probability densities. Second, it showsthat we can estimate the independent components using very simple models of theirdensities, in particular, using models consisting of only two densities.
This situation can be compared with the one encountered in Section 8.3.4, where
we saw that any nonlinearity can be seen to divide the space of probability distributionsin half. When the distribution of an independent component is in one of the halves,the nonlinearity can be used in the gradient method to estimate that independent
component. When the distribution is in the other half, the negative of the nonlinearity
must be used in the gradient method. In the ML case, a nonlinearity corresponds toa density approximation.
The validity of these approaches is shown in the following theorem, whose proof
can be found in the appendix. This theorem is basically a corollary of the stability
theorem in Section 8.3.4.

206

ICA BY MAXIMUM LIKELIHOOD ESTIMATION

Theorem 9.1 Denote by p~i the assumed densities of the independent components,and0gi (si ) = @s@ log p~i (si ) = pp~~i ((ssi ))
i

(9.7)

i i

bx

Constrain the estimates of the independent components yi = Ti to be uncorrelated
and to have unit variance. Then the ML estimator is locally consistent, if the assumed
densities p~i fulfill

E fsi gi (si ) ? g0 (si )g > 0(9.8)

for all i.
This theorem shows rigorously that small misspecifications in the densities pi do
not affect the local consistency of the ML estimator, since sufficiently small changes
do not change the sign in (9.8).
Moreover, the theorem shows how to construct families consisting of only two
densities, so that the condition in (9.8) is true for one of these densities. For example,
consider the following log-densities:

log p~+i (s) = 1 ? 2 log cosh(s)log p~?i (s) = 2 ? [s2 =2 ? log cosh(s)](9.9)(9.10)

where 1 ; 2 are positive parameters that are fixed so as to make these two functions
logarithms of probability densities. Actually, these constants can be ignored in thefollowing. The factor 2 in (9.9) is not important, but it is usually used here; also, thefactor 1=2 in (9.10) could be changed.
The motivation for these functions is that p~+
i is a supergaussian density, because
the log cosh function is close to the absolute value that would give the Laplaciandensity. The density given by p~?
i is subgaussian, because it is like a gaussian logdensity, ?s2 =2 plus a constant, that has been somewhat “flattened” by the log coshfunction.
Simple computations show that the value of the nonpolynomial moment in (9.8)
is for p~+i

2E f? tanh(si )si + (1 ? tanh(si )2 )g(9.11)

and for p~?
i it isE ftanh(si )si ? (1 ? tanh(si )2 )g(9.12)
since the derivative of tanh(s) equals 1 ? tanh(s)2 , and E fs2i g = 1 by definition.
We see that the signs of these expressions are always opposite. Thus, for practicallyany distributions of the si , one of these functions fulfills the condition, i.e., has thedesired sign, and estimation is possible. Of course, for some distribution of the si
the nonpolynomial moment in the condition could be zero, which corresponds to theALGORITHMS FOR MAXIMUM LIKELIHOOD ESTIMATION

207

case of zero kurtosis in cumulant-based estimation; such cases can be considered tobe very rare.
Thus we can just compute the nonpolynomial moments for the two prior distributions in (9.9) and (9.10), and choose the one that fulfills the stability condition in (9.8).
This can be done on-line during the maximization of the likelihood. This always
provides a (locally) consistent estimator, and solves the problem of semiparametricestimation.
In fact, the nonpolynomial moment in question measures the shape of the density
function in much the same way as kurtosis. For g (s) = ?s3 , we would actuallyobtain kurtosis. Thus, the choice of nonlinearity could be compared with the choice
whether to minimize or maximize kurtosis, as previously encountered in Section 8.2.
That choice was based on the value of the sign of kurtosis; here we use the sign of a
nonpolynomial moment.
Indeed, the nonpolynomial moment of this chapter is the same as the one encountered in Section 8.3 when using more general measures of nongaussianity. However,it must be noted that the set of nonlinearities that we can use here is more restricted
than those used in Chapter 8. This is because the nonlinearities gi used must correspond to the derivative of the logarithm of a probability density function (pdf). For
example, we cannot use the function g (s) = s3 because the corresponding pdf would
be of the form exp(s4 =4), and this is not integrable, i.e., it is not a pdf at all.
9.2 ALGORITHMS FOR MAXIMUM LIKELIHOOD ESTIMATION
To perform maximum likelihood estimation in practice, we need an algorithm toperform the numerical maximization of likelihood. In this section, we discuss different methods to this end. First, we show how to derive simple gradient algorithms,of which especially the natural gradient algorithm has been widely used. Then weshow how to derive a fixed-point algorithm, a version of FastICA, that maximizes thelikelihood faster and more reliably.
9.2.1

Gradient algorithms

The Bell-Sejnowski algorithm The simplest algorithms for maximizing likelihood are obtained by gradient methods. Using the well-known results in Chapter 3,
one can easily derive the stochastic gradient of the log-likelihood in (9.6) as:

1 @ log L
T ?1
T
T @ B = [B ] + E fg(Bx)x g(9.13)

gy

Here, ( ) = (gi (yi ); :::; gn (yn )) is a component-wise vector function that consistsof the so-called (negative) score functions gi of the distributions of si , defined as0
gi = (log pi )0 = ppi :i(9.14)

208

ICA BY MAXIMUM LIKELIHOOD ESTIMATIONThis immediately gives the following algorithm for ML estimation:

B / [BT ]?1 + E fg(Bx)xT g(9.15)

A stochastic version of this algorithm could be used as well. This means that the
expectation is omitted, and in each step of the algorithm, only one data point is used:B / [BT ]?1 + g(Bx)xT :(9.16)

This algorithm is often called the Bell-Sejnowski algorithm. It was first derived in[36], though from a different approach using the infomax principle that is explainedin Section 9.3 below.
The algorithm in Eq. (9.15) converges very slowly, however, especially due tothe inversion of the matrix that is needed in every step. The convergence can be
improved by whitening the data, and especially by using the natural gradient.

B

The natural gradient algorithm The natural (or relative) gradient method simplifies the maximization of the likelihood considerably, and makes it better conditioned. The principle of the natural gradient is based on the geometrical structure of
the parameter space, and is related to the principle of relative gradient, which uses
the Lie group structure of the ICA problem. See Chapter 3 for more details. In thecase of basic ICA, both of these principles amount to multiplying the right-hand sideof (9.15) by T . Thus we obtainBBB / (I + E fg(y)yT g)B(9.17)

Interestingly, this algorithm can be interpreted as nonlinear decorrelation. This
principle will be treated in more detail in Chapter 12. The idea is that the algorithm converges when E f ( ) T g = , which means that the yi and gj (yj ) are
uncorrelated for i 6= j . This is a nonlinear extension of the ordinary requirementof uncorrelatedness, and, in fact, this algorithm is a special case of the nonlinear
decorrelation algorithms to be introduced in Chapter 12.
In practice, one can use, for example, the two densities described in Section 9.1.2.
For supergaussian independent components, the pdf defined by (9.9) is usually used.
This means that the component-wise nonlinearity g is the tanh function:gyyIg+ (y) = ?2 tanh(y)(9.18)For subgaussian independent components, other functions must be used. For example, one could use the pdf in (9.10), which leads tog? (y) = tanh(y) ? y(9.19)(Another possibility is to use g (y ) = ?y 3 for subgaussian components.) These
nonlinearities are illustrated in Fig. 9.1.
The choice between the two nonlinearities in (9.18) and (9.19) can be made bycomputing the nonpolynomial moment:

E f? tanh(si )si + (1 ? tanh(si )2 )g(9.20)

ALGORITHMS FOR MAXIMUM LIKELIHOOD ESTIMATION

209

Fig. 9.1 The functions g + in Eq. (9.18) and g ? in Eq. (9.19), given by the solid line and thedashed line, respectively.

using some estimates of the independent components. If this nonpolynomial moment
is positive, the nonlinearity in (9.18) should be used, otherwise the nonlinearity in(9.19) should be used. This is because of the condition in Theorem 9.1.
The choice of nonlinearity can be made while running the gradient algorithm,
using the running estimates of the independent components to estimate the nature of
the independent components (that is, the sign of the nonpolynomial moment). Notethat the use of the polynomial moment requires that the estimates of the independent
components are first scaled properly, constraining them to unit variance, as in thetheorem. Such normalizations are often omitted in practice, which may in some cases
lead to situations in which the wrong nonlinearity is chosen.
The resulting algorithm is recapitulated in Table 9.1. In this version, whitening andthe above-mentioned normalization in the estimation of the nonpolynomial moments
are omitted; in practice, these may be very useful.
9.2.2

A fast fixed-point algorithm

Likelihood can be maximized by a fixed-point algorithm as well. The fixed-point
algorithm given by FastICA is a very fast and reliable maximization method that was
introduced in Chapter 8 to maximize the measures of nongaussianity used for ICA
estimation. Actually, the FastICA algorithm can be directly applied to maximizationof the likelihood.
The FastICA algorithm was derived in Chapter 8 for optimization of E fG( T )g
under the constraint of the unit norm of . In fact, maximization of likelihood gives
us an almost identical optimization problem, if we constrain the estimates of the
independent components to be white (see Chapter 7). In particular, this implies thatthe term log j det j is constant, as proven in the Appendix, and thus the likelihood
basically consists of the sum of n terms of the form optimized by FastICA. Thus

w

W

wz

210

ICA BY MAXIMUM LIKELIHOOD ESTIMATION

1. Center the data to make its mean zeroB

2. Choose an initial (e.g., random) separating matrix . Choose initial valuesof i ; i = 1; :::; n, either randomly or using prior information. Choose thelearning rates  and  .
3. Computey = Bx.

4. If the nonlinearities are not fixed a priori:(a) update i = (1 ?  ) i +  E f? tanh(yi )yi + (1 ? tanh(yi )2 )g.
(b) if i > 0, define gi as in (9.18), otherwise define it as in (9.19).
5. Update the separating matrix by

B B + [I + g(y)yT ]B
where(9.21)

g(y) = (g (y ); :::; gn(yn))T .
1

1

6. If not converged, go back to step 3.
Table 9.1 The on-line stochastic natural gradient algorithm for maximum likelihood estimation. Preliminary whitening is not shown here, but in practice it is highly recommended.

we could use directly the same kind of derivation of fixed-point iteration as used in
Chapter 8.
In Eq. (8.42) in Chapter 8 we had the following form of the FastICA algorithm(for whitened data):w w ? [E fzg(wT z)g + w]=[E fg0(wT z)g + ](9.22)
can be computed from (8.40) as = ?E fyi g (yi )g. If we write this inwherematrix form, we obtain:

W W + diag( i )[diag( i ) + E fg(y)yT g]W(9.23)
where i is defined as ?1=(E fg 0(wT z) + i g), and y = Wz. To express this using

nonwhitened data, as we have done in this chapter, it is enough to multiply both sidesof (9.23) from the right by the whitening matrix. This means simply that we replace
by , since we have=
which implies =.
theThus, we obtain the basic iteration of FastICA as:

W B

Wz WVxB WV

B B + diag( i )[diag( i ) + E fg(y)yT g]B(9.24)
where y = Bx, i = ?E fyi g (yi )g, and i = ?1=( i + E fg 0 (yi )g).
After every step, the matrix B must be projected on the set of whitening matrices.
This can be accomplished by the classic method involving matrix square roots,

B (BCBT )? = B
1 2(9.25)

THE INFOMAX PRINCIPLE

C

211

xx

where = E f T g is the correlation matrix of the data (see exercises). The inverse
square root is obtained as in (7.20). For alternative methods, see Section 8.4 andChapter 6, but note that those algorithms require that the data is prewhitened, since
they simply orthogonalize the matrix.
This version of FastICA is recapitulated in Table 9.2. FastICA could be comparedwith the natural gradient method for maximizing likelihood given in (9.17). Then
we see that FastICA can be considered as a computationally optimized version of thegradient algorithm. In FastICA, convergence speed is optimized by the choice of thematrices diag( i ) and diag( i ). These two matrices give an optimal step size to be
used in the algorithm.
Another advantage of FastICA is that it can estimate both sub- and supergaussian
independent components without any additional steps: We can fix the nonlinearity
to be equal to the tanh nonlinearity for all the independent components. The reason
is clear from (9.24): The matrix diag( i ) contains estimates on the nature (sub- or
supergaussian) of the independent components. These estimates are used as in thegradient algorithm in the previous subsection. On the other hand, the matrix diag( i )can be considered as a scaling of the nonlinearities, since we could reformulate[diag( i ) + E f ( ) T g] = diag( i )[ + diag( i?1 )E f ( ) T g]. Thus we cansay that FastICA uses a richer parameterization of the densities than that used in
Section 9.1.2: a parameterized family instead of just two densities.
Note that in FastICA, the outputs yi are decorrelated and normalized to unit
variance after every step. No such operations are needed in the gradient algorithm.
FastICA is not stable if these additional operations are omitted. Thus the optimization
space is slightly reduced.
In the version given here, no preliminary whitening is done. In practice, it is often
highly recommended to do prewhitening, possibly combined with PCA dimension
reduction.

gyy

I

gyy

9.3 THE INFOMAX PRINCIPLE
An estimation principle for ICA that is very closely related to maximum likelihood
is the infomax principle [282, 36]. This is based on maximizing the output entropy,
or information flow, of a neural network with nonlinear outputs. Hence the nameinfomax.
Assume that is the input to the neural network whose outputs are of the formxyi = i (bTi x) + n(9.31)bwhere the i are some nonlinear scalar functions, and the i are the weight vectorsof the neurons. The vector is additive gaussian white noise. One then wants tomaximize the entropy of the outputs:n

H (y) = H (1 (bT1 x); :::; n (bTn x))(9.32)

This can be motivated by considering information flow in a neural network. Efficient
information transmission requires that we maximize the mutual information between

212

ICA BY MAXIMUM LIKELIHOOD ESTIMATION

1. Center the data to make its mean zero. Compute correlation matrix
E f T g.

xx

2. Choose an initial (e.g., random) separating matrix

C=B.

3. Computey = Bxi = ?E fyi g (yi )g; for i = 1; :::; ni = ?1=( i + E fg 0 (yi )g); for i = 1; :::; n(9.26)(9.27)(9.28)

4. Update the separating matrix by

B B + diag( i )[diag( i ) + E fg(y)yT g]B(9.29)

5. Decorrelate and normalize by

B (BCBT )? = B
1 2(9.30)

6. If not converged, go back to step 3.
Table 9.2 The FastICA algorithm for maximum likelihood estimation. This is a version
without whitening; in practice, whitening combined with PCA may often be useful. The
nonlinear function g is typically the tanh function.

EXAMPLES

x

213

y

the inputs and the outputs . This problem is meaningful only if there is some
information loss in the transmission. Therefore, we assume that there is some noisein the network. It can then be shown (see exercices) that in the limit of no noise (i.e.,with infinitely weak noise), maximization of this mutual information is equivalent tomaximization of the output entropy in (9.32). For simplicity, we therefore assume in
the following that the noise is of zero variance.
Using the classic formula of the entropy of a transformation (see Eq. (5.13) wehaveF (x)jg
H (1 (bT1 x); :::; n (bTn x)) = H (x) + E flog j det @@B

Fx

w x

(9.33)

w x

where ( ) = (1 ( 1T ); :::; n ( nT )) denotes the function defined by the neuralnetwork. We can simply calculate the derivative to obtain

F (x)jg = X E flog 0 (bT x)g + log j det Bj
E flog j det @@Bi ii(9.34)

Now we see that the output entropy is of the same form as the expectation of thelikelihood as in Eq. 9.6. The pdf’s of the independent components are here replacedby the functions 0i . Thus, if the nonlinearities i used in the neural network arechosen as the cumulative distribution functions corresponding to the densities pi , i.e.,
0i (:) = pi (:), the output entropy is actually equal to the likelihood. This means that
infomax is equivalent to maximum likelihood estimation.

9.4 EXAMPLES
Here we show the results of applying maximum likelihood estimation to the two
mixtures introduced in Chapter 7. Here, we use whitened data. This is not strictly
necessary, but the algorithms converge much better with whitened data. The algorithms were always initialized so that was the identity matrix.
First, we used the natural gradient ML algorithm in Table 9.1. In the first example, we used the data consisting of two mixtures of two subgaussian (uniformly
distributed) independent components, and took the nonlinearity to be the one in(9.18), corresponding to the density in (9.9). The algorithm did not converge properly, as shown in Fig. 9.2. This is because the nonlinearity was not correctly chosen.
Indeed, computing the nonpolynomial moment (9.20), we saw that it was negative,
which means that the nonlinearity in (9.19) should have been used. Using the correct
nonlinearity, we obtained correct convergence, as in Fig. 9.3. In both cases, several
hundred iterations were performed.
Next we did the corresponding estimation for two mixtures of two supergaussian
independent components. This time, the nonlinearity in (9.18) was the correctone, and gave the estimates in Fig. 9.4. This could be checked by computing the
nonpolynomial moment in (9.20): It was positive. In contrast, using the nonlinearity
in (9.19) gave completely wrong estimates, as seen in Fig. 9.5.

B

214

ICA BY MAXIMUM LIKELIHOOD ESTIMATION

In contrast to the gradient algorithm, FastICA effortlessly finds the independent
components in both cases. In Fig. 9.6, the results are shown for the subgaussian data,and in Fig. 9.7, the results are shown for the supergaussian data. In both cases thealgorithm converged correctly, in a couple of iterations.

9.5 CONCLUDING REMARKS AND REFERENCES
Maximum likelihood estimation, perhaps the most commonly used statistical estimation principle, can be used to estimate the ICA model as well. It is closely related
to the infomax principle used in neural network literature. If the densities of the
independent components are known in advance, a very simple gradient algorithm canbe derived. To speed up convergence, the natural gradient version and especially the
FastICA fixed-point algorithm can be used. If the densities of the independent components are not known, the situation is somewhat more complicated. Fortunately,
however, it is enough to use a very rough density approximation. In the extreme
case, a family that contains just two densities to approximate the densities of the
independent components is enough. The choice of the density can then be basedon the information whether the independent components are sub- or supergaussian.
Such an estimate can be simply added to the gradient methods, and it is automaticallydone in FastICA.
The first approaches to using maximum likelihood estimation for ICA were in[140, 372]; see also [368, 371]. This approach became very popular after the
introduction of the algorithm in (9.16) by Bell and Sejnowski, who derived it using theinfomax principle [36]; see also [34]. The connection between these two approaches
was later proven by [64, 322, 363]. The natural gradient algorithm in (9.17) issometimes called the Bell-Sejnowski algorithm as well. However, the natural gradientextension was actually introduced only in [12, 3]; for the underlying theory, see[4, 11, 118]. This algorithm is actually almost identical to those introduced previously[85, 84] based on nonlinear decorrelation, and quite similar to the one in [255, 71](see Chapter 12). In particular, [71] used the relative gradient approach, which in thiscase is closely related to the natural gradient; see Chapter 14 for more details. Our
two-density family is closely related to those in [148, 270]; for alternative approaches
on modeling the distributions of the ICs, see [121, 125, 133, 464].
The stability criterion in Theorem 9.1 has been presented in different forms by
many authors [9, 71, 67, 69, 211]. The different forms are mainly due to thecomplication of different normalizations, as discussed in [67]. We chose to normalize
the components to unit variance, which gives a simple theorem and is in line withthe approach of the other chapters. Note that in [12], it was proposed that a single
very high-order polynomial nonlinearity could be used as a universal nonlinearity.
Later research has shown that this is not possible, since we need at least two different
nonlinearities, as discussed in this chapter. Moreover, a high-order polynomial leads
to very nonrobust estimators.

CONCLUDING REMARKS AND REFERENCES

215

Fig. 9.2 Problems of convergence with the (natural) gradient method for maximum likelihood estimation. The data was two whitened mixtures of subgaussian independent components.
The nonlinearity was the one in (9.18), which was not correct in this case. The resulting estimates of the columns of the whitened mixing matrix are shown in the figure: they are notaligned with the edges of the square, as they should be.

Fig. 9.3 The same as in Fig. 9.2, but with the correct nonlinearity, given by (9.19). Thistime, the natural gradient algorithm gave the right result. The estimated vectors are alignedwith the edges of the square.

216

ICA BY MAXIMUM LIKELIHOOD ESTIMATION

Fig. 9.4 In this experiment, data was two whitened mixtures of supergaussian independent
components. The nonlinearity was the one in (9.18). The natural gradient algorithm converged
correctly.

Fig. 9.5 Again, problem of convergence with the natural gradient method for maximum
likelihood estimation. The nonlinearity was the one in (9.19), which was not correct in thiscase.

CONCLUDING REMARKS AND REFERENCES

217

Fig. 9.6 FastICA automatically estimates the nature of the independent components, andconverges fast to the maximum likelihood solution. Here, the solution was found in 2 iterations
for subgaussian independent components.

Fig. 9.7 FastICA this time applied on supergaussian mixtures. Again, the solution was
found in 2 iterations.

218

ICA BY MAXIMUM LIKELIHOOD ESTIMATION

Problems
9.1

Derive the likelihood in (9.4).

9.2

Derive (9.11) and (9.12).

9.3

Derive the gradient in (9.13).

9.4 Instead of the function in (9.19), one could use the function g (y ) = ?y 3 . Show
that this corresponds to a subgaussian distribution by computing the kurtosis of the
distribution. Note the normalization constants involved.
9.5 After the preceding problem, one might be tempted to use g (y ) = y 3 for supergaussian variables. Why is this not correct in the maximum likelihood framework?
9.6 Take a linear g (y ) = ?y . What is the interpretation of this in the ML framework? Conclude (once again) that g must be nonlinear.

9.7 Assume that you use the general function family g (y ) = ?2 tanh(a1 y ) instead
the simple ? tanh function in (9.18), where a1  1 is a constant. What is theinterpretation of a1 in the likelihood framework?
9.8 Show that for a gaussian random variable, the nonpolynomial moment in (9.8)
is zero for any g .

9.9 The difference between using ? tanh(y ) or ?2 tanh(y ) in the nonlinearity(9.18) is a matter of normalization. Does it make any difference in the algorithms?
Consider separately the natural gradient algorithm and the FastICA algorithm.
9.10 Show that maximizing the mutual information of inputs and outputs in a
network of the form

yi = i (bTi x) + ni

(9.35)

where ni is gaussian noise, and the output and input spaces have the same dimension,
as in (9.31), is equivalent to maximizing the entropy of the outputs, in the limit of no
zero noise level. (You can compute the joint entropy of inputs and outputs using theentropy transformation formula. Show that it is constant. Then set the noise level toinfinitely small.)
9.11

Show that after (9.25),

y = Bx is white.

Computer assignments
9.1 Take random variables of (1) uniform and (2) Laplacian distribution. Computethe values of the nonpolynomial moment in (9.8), for different nonlinearities g . Are
the moments of different signs for any nonlinearity?
9.2

Reproduce the experiments in Section 9.4.

APPENDIX

219

9.3 The densities of the independent components could be modeled by a density
family given byp( ) = c1 exp(c2 j j )(9.36)

where c1 and c2 are normalization constants to make this a pdf of unit variance. For
different values of , ranging from 0 to infinity, we get distributions of different
properties.
9.3.1. What happens when we have = 2?
9.3.2. Plot the pdf’s, the logarithms of the pdf’s, and the corresponding scorefunctions for the following values of : 0.2,1,2,4,10.
9.3.3. Conclude that for < 2, we have subgaussian densities, and for > 2,
we have supergaussian densities.

Appendix proofs
Here we prove Theorem 9.1. Looking at the expectation of log-likelihood, using assumed
densities p~i :nX

T log L(B) = i=1 E flog p~i (bi x)g + log j det Bj
1

T

(A.1)

f b xgwe see that the first term on the right-hand side is a sum of terms of the form E G( Ti ) , asin the stability theorem in Section 8.3.4. Using that theorem, we see immediately that the first
gives independent components.
term is maximized when =Thus if we prove that the second term remains constant under the conditions of the theorem,
T =the theorem is proven. Now, uncorrelatedness and unit variance of the yi means E
TTE= , which implies

y Bx

fyy g

W fxx gW I
T
T
T
T
det I = 1 = (det WE fxx gW ) = (det W)(det E fxx g)(det W )and this implies that det

W must be constant. Thus the theorem is proven.

(A.2)

10
ICA by Minimization of
Mutual Information
An important approach for independent component analysis (ICA) estimation, inspired by information theory, is minimization of mutual information.
The motivation of this approach is that it may not be very realistic in many cases
to assume that the data follows the ICA model. Therefore, we would like to develop
an approach that does not assume anything about the data. What we want to haveis a general-purpose measure of the dependence of the components of a random
vector. Using such a measure, we could define ICA as a linear decomposition thatminimizes that dependence measure. Such an approach can be developed using
mutual information, which is a well-motivated information-theoretic measure ofstatistical dependence.
One of the main utilities of mutual information is that it serves as a unifying
framework for many estimation principles, in particular maximum likelihood (ML)estimation and maximization of nongaussianity. In particular, this approach gives arigorous justification for the heuristic principle of nongaussianity.

10.1
10.1.1DEFINING ICA BY MUTUAL INFORMATION
Information-theoretic concepts

The information-theoretic concepts needed in this chapter were explained in Chapter 5. Readers not familiar with information theory are advised to read that chapter
before this one.

221

222

ICA BY MINIMIZATION OF MUTUAL INFORMATION

We recall here very briefly the basic definitions of information theory. The
differential entropy H of a random vector with density p( ) is defined as:

Z

y

y

H (y) = ? p(y) log p(y)dy(10.1)

Entropy is closely related to the code length of the random vector. A normalizedversion of entropy is given by negentropy J , which is defined as follows

J (y) = H (ygauss ) ? H (y)y(10.2)

where gauss is a gaussian random vector of the same covariance (or correlation)
matrix as . Negentropy is always nonnegative, and zero only for gaussian random
vectors. Mutual information I between m (scalar) random variables, yi ; i = 1:::m isdefined as followsy

I (y1 ; y2 ; :::; ym ) =10.1.2mX
i=1

H (yi ) ? H (y)(10.3)

Mutual information as measure of dependence

We have seen earlier (Chapter 5) that mutual information is a natural measure of thedependence between random variables. It is always nonnegative, and zero if and only
if the variables are statistically independent. Mutual information takes into accountthe whole dependence structure of the variables, and not just the covariance, likeprincipal component analysis (PCA) and related methods.
Therefore, we can use mutual information as the criterion for finding the ICA
representation. This approach is an alternative to the model estimation approach. We
define the ICA of a random vector as an invertible transformation:xs = BxB(10.4)

where the matrix is determined so that the mutual information of the transformed
components si is minimized. If the data follows the ICA model, this allows estimationof the data model. On the other hand, in this definition, we do not need to assume
that the data follows the model. In any case, minimization of mutual information canbe interpreted as giving the maximally independent components.

MUTUAL INFORMATION AND NONGAUSSIANITY

10.2

223

MUTUAL INFORMATION AND NONGAUSSIANITY

Using the formula for the differential entropy of a transformation as given in (5.13)
of Chapter 5, we obtain a corresponding result for mutual information. We have:for an invertible linear transformation =y Bx

I (y1 ; y2 ; :::; yn ) =Xi

H (yi ) ? H (x) ? log j det Bj(10.5)

Now, let us consider what happens if we constrain the yi to be uncorrelated and ofunit variance. This means E f T g = E f T g T = , which impliesyy
B xx B I
det I = 1 = det(BE fxxT gBT ) = (det B)(det E fxxT g)(det BT )B

(10.6)xxand this implies that det must be constant since det E f T g does not dependon . Moreover, for yi of unit variance, entropy and negentropy differ only by aconstant and the sign, as can be seen in (10.2). Thus we obtain,B

I (y1 ; y2 ; :::; yn ) = const. ?

Xi

J (yi )

(10.7)

B

where the constant term does not depend on . This shows the fundamental relation
between negentropy and mutual information.
We see in (10.7) that finding an invertible linear transformation that minimizes
the mutual information is roughly equivalent to finding directions in which the negentropy is maximized. We have seen previously that negentropy is a measure of
nongaussianity. Thus, (10.7) shows that ICA estimation by minimization of mutual information is equivalent to maximizing the sum of nongaussianities of the estimates of
the independent components, when the estimates are constrained to be uncorrelated.
Thus, we see that the formulation of ICA as minimization of mutual information
gives another rigorous justification of our more heuristically introduced idea of finding
maximally nongaussian directions, as used in Chapter 8.
In practice, however, there are also some important differences between these two
criteria.

B

1. Negentropy, and other measures of nongaussianity, enable the deflationary, i.e.,
one-by-one, estimation of the independent components, since we can look forthe maxima of nongaussianity of a single projection T . This is not possible
with mutual information or most other criteria, like the likelihood.

bx

2. A smaller difference is that in using nongaussianity, we force the estimates of
the independent components to be uncorrelated. This is not necessary whenusing mutual information, because we could use the form in (10.5) directly,
as will be seen in the next section. Thus the optimization space is slightly
reduced.

224

ICA BY MINIMIZATION OF MUTUAL INFORMATION

10.3

MUTUAL INFORMATION AND LIKELIHOOD

Mutual information and likelihood are intimately connected. To see the connection
between likelihood and mutual information, consider the expectation of the loglikelihood in (9.5):n
1 E flog L(B)g = X
E flog p (bT x)g + log j det Bj

T

i=1

i i

(10.8)

bx

If the pi were equal to the actual pdf’s of Ti , the first term would be equal to? i H ( Ti ). Thus the likelihood would be equal, up to an additive constant givenby the total entropy of , to the negative of mutual information as given in Eq. (10.5).
In practice, the connection may be just as strong, or even stronger. This is because
in practice we do not know the distributions of the independent components that areneeded in ML estimation. A reasonable approach would be to estimate the densityof Ti as part of the ML estimation method, and use this as an approximation of thedensity of si . This is what we did in Chapter 9. Then, the pi in this approximationof likelihood are indeed equal to the actual pdf’s Ti . Thus, the equivalency would
really hold.
Conversely, to approximate mutual information, we could take a fixed approximation of the densities yi , and plug this in the definition of entropy. Denote the pdf’s
by Gi (yi ) = log pi (yi ). Then we could approximate (10.5) as

P

bx

x

bx

bx

I (y1 ; y2; :::; yn ) = ?

Xi

E fGi (yi )g ? log j det Bj ? H (x)(10.9)

Now we see that this approximation is equal to the approximation of the likelihood
used in Chapter 9 (except, again, for the global sign and the additive constant given by
H ( )). This also gives an alternative method of approximating mutual informationthat is different from the approximation that uses the negentropy approximations.

x

10.4

ALGORITHMS FOR MINIMIZATION OF MUTUAL INFORMATION

To use mutual information in practice, we need some method of estimating or approximating it from real data. Earlier, we saw two methods for approximating mutual
entropy. The first one was based on the negentropy approximations introduced in
Section 5.6. The second one was based on using more or less fixed approximations
for the densities of the ICs in Chapter 9.
Thus, using mutual information leads essentially to the same algorithms as used for
maximization of nongaussianity in Chapter 8, or for maximum likelihood estimationin Chapter 9. In the case of maximization of nongaussianity, the corresponding
algorithms are those that use symmetric orthogonalization, since we are maximizing
the sum of nongaussianities, so that no order exists between the components. Thus,
we do not present any new algorithms in this chapter; the reader is referred to the two
preceding chapters.

EXAMPLES

225

−4

5

x 10

4.5

mutual information

4
3.5
3
2.5
2
1.5
1
0.5
0
0

0.5

1

1.5

2

2.5

3

iteration countFig. 10.1 The convergence of FastICA for ICs with uniform distributions. The value ofmutual information shown as function of iteration count.

10.5

EXAMPLES

Here we show the results of applying minimization of mutual information to thetwo mixtures introduced in Chapter 7. We use here the whitened mixtures, and the
FastICA algorithm (which is essentially identical whichever approximation of mutual
information is used). For illustration purposes, the algorithm was always initialized
was the identity matrix. The function G was chosen as G1 in (8.26).
so thatFirst, we used the data consisting of two mixtures of two subgaussian (uniformly
distributed) independent components. To demonstrate the convergence of the algorithm, the mutual information of the components at each iteration step is plottedin Fig. 10.1. This was obtained by the negentropy-based approximation. At convergence, after two iterations, mutual information was practically equal to zero.
The corresponding results for two supergaussian independent components are shownin Fig. 10.2. Convergence was obtained after three iterations, after which mutual
information was practically zero.

W

10.6

CONCLUDING REMARKS AND REFERENCES

A rigorous approach to ICA that is different from the maximum likelihood approachis given by minimization of mutual information. Mutual information is a natural
information-theoretic measure of dependence, and therefore it is natural to estimate
the independent components by minimizing the mutual information of their estimates.
Mutual information gives a rigorous justification of the principle of searching for
maximally nongaussian directions, and in the end turns out to be very similar to thelikelihood as well.
Mutual information can be approximated by the same methods that negentropy isapproximated. Alternatively, is can be approximated in the same way as likelihood.

226

ICA BY MINIMIZATION OF MUTUAL INFORMATION
−3

3

x 10

mutual information

2.5

2

1.5

1

0.5

0
0

0.5

1

1.5

2

2.5

3

iteration countFig. 10.2 The convergence of FastICA for ICs with supergaussian distributions. The valueof mutual information shown as function of iteration count.

Therefore, we find here very much the same objective functions and algorithms as
in maximization of nongaussianity and maximum likelihood. The same gradient andfixed-point algorithms can be used to optimize mutual information.
Estimation of ICA by minimization of mutual information was probably first
proposed in [89], who derived an approximation based on cumulants. The idea has,
however, a longer history in the context of neural network research, where it has
been proposed as a sensory coding strategy. It was proposed in [26, 28, 30, 18],that decomposing sensory data into features that are maximally independent is usefulas a preprocessing step. Our approach follows that of [197] for the negentropy
approximations.
A nonparametric algorithm for minimization of mutual information was proposedin [175], and an approach based on order statistics was proposed in [369]. See[322, 468] for a detailed analysis of the connection between mutual information andinfomax or maximum likelihood. A more general framework was proposed in [377].

227

PROBLEMS

Problems
10.1

Derive the formula in (10.5).

10.2

Compute the constant in (10.7).

10.3 If the variances of the yi are not constrained to unity, does this constantchange?
10.4 Compute the mutual information for a gaussian random vector with covariancematrix .

C

Computer assignments
10.1

Create a sample of 2-D gaussian data with the two covariance matrices

3 0
0 2

and

3 1
1 2

(10.10)Estimate numerically the mutual information using the definition. (Divide the data
into bins, i.e., boxes of fixed size, and estimate the density at each bin by computing
the number of data points that belong to that bin and dividing it by the size of the bin.
This elementary density approximation can then be used in the definition.)

11
ICA by Tensorial Methods

One approach for estimation of independent component analysis (ICA) consists ofusing higher-order cumulant tensor. Tensors can be considered as generalizationof matrices, or linear operators. Cumulant tensors are then generalizations of thecovariance matrix. The covariance matrix is the second-order cumulant tensor, andthe fourth order tensor is defined by the fourth-order cumulants cum (xi ; xj ; xk ; xl ).
For an introduction to cumulants, see Section 2.7.
As explained in Chapter 6, we can use the eigenvalue decomposition of thecovariance matrix to whiten the data. This means that we transform the data so that
second-order correlations are zero. As a generalization of this principle, we can use
the fourth-order cumulant tensor to make the fourth-order cumulants zero, or at leastas small as possible. This kind of (approximative) higher-order decorrelation gives
one class of methods for ICA estimation.

11.1

DEFINITION OF CUMULANT TENSOR

We shall here consider only the fourth-order cumulant tensor, which we call for simplicity the cumulant tensor. The cumulant tensor is a four-dimensional array whose
entries are given by the fourth-order cross-cumulants of the data: cum (xi ; xj ; xk ; xl ),where the indices i; j; k; l are from 1 to n. This can be considered as a “fourdimensional matrix”, since it has four different indices instead of the usual two. Fora definition of cross-cumulants, see Eq. (2.106).
In fact, all fourth-order cumulants of linear combinations of xi can be obtained
as linear combinations of the cumulants of xi . This can be seen using the additive
229

230

ICA BY TENSORIAL METHODS

properties of the cumulants as discussed in Section 2.7. The kurtosis of a linear
combination is given by
kurt

X
i

wi xi = cum(Xiwi xi ;

Xj

wj xj ;

=

X
ijkl

X
k

wk xk ;

Xlwl xl )

wi4 wj4 wk4 wl4 cum(xi ; xj ; xk ; xl ) (11.1)

Thus the (fourth-order) cumulants contain all the fourth-order information of the data,
just as the covariance matrix gives all the second-order information on the data. Notethat if the xi are independent, all the cumulants with at least two different indices arezero, and therefore we have the formula that was already widely used in Chapter 8:
kurt i qi si = i qi4 kurt(si ).
The cumulant tensor is a linear operator defined by the fourth-order cumulantscum(xi ; xj ; xk ; xl ). This is analogous to the case of the covariance matrix withelements cov(xi ; xj ), which defines a linear operator just as any matrix defines one.
In the case of the tensor we have a linear transformation in the space of n  n matrices,
instead of the space of n-dimensional vectors. The space of such matrices is a linear
space of dimension n  n, so there is nothing extraordinary in defining the linear
transformation. The i; j th element of the matrix given by the transformation, say
ij , is defined as

P

P

F

Fij (M) =X
kl

mkl cum(xi ; xj ; xk ; xl )where mkl are the elements in the matrix
11.2(11.2)

M that is transformed.

TENSOR EIGENVALUES GIVE INDEPENDENT COMPONENTS

As any symmetric linear operator, the cumulant tensor has an eigenvalue decomposition (EVD). An eigenmatrix of the tensor is, by definition, a matrix
such
that

M

F M

M

F(M) = M(11.3)

i.e., ij ( ) =  ij , where  is a scalar eigenvalue.
The cumulant tensor is a symmetric linear operator, since in the expression
cum(xi ; xj ; xk ; xl ), the order of the variables makes no difference. Therefore, thetensor has an eigenvalue decomposition.
Let us consider the case where the data follows the ICA model, with whiteneddata:

z = VAs = WT s
(11.4)
where we denote the whitened mixing matrix by WT . This is because it is orthogonal,
and thus it is the transpose of the separating matrix W for whitened data.

TENSOR EIGENVALUES GIVE INDEPENDENT COMPONENTS

231

z

The cumulant tensor of has a special structure that can be seen in the eigenvalue
decomposition. In fact, every matrix of the form

M = wmwmT(11.5)w

for m = 1; :::; n is an eigenmatrix. The vector m is here one of the rows of thematrix , and thus one of the columns of the whitened mixing matrix T . To see
this, we calculate by the linearity properties of cumulantsW

Fij (wmwmT ) =WXwmk wml cum(zi ; zj ; zk ; zl )
kl
X
X
X
X
X
= wmk wml cum( wqi sq ; wq0 j sq0 ; wrk sr ; wr0 l sr0 )
q
r
q0
r0
kl
=

X

klqq0 rr0

wmk wml wqi wq0 j wrk wr0 l cum(sq ; sq0 ; sr ; sr0 ) (11.6)

Now, due to the independence of the si , only those cumulants where q = q 0 = r = r0
are nonzero. Thus we have

Fij (wmwmT ) =Xklq

wmk wml wqi wqj wqk wql kurt(sq )(11.7)PW

Due to the orthogonality of the rows of, we have k wmk wqk = mq , andsimilarly for index l. Thus we can take the sum first with respect to k , and then withrespect to l, which gives

Fij (wmwmT ) =Xlq

wml wqi wqj mq wql kurt(sq )=Xqwqi wqj mq mq kurt(sq ) = wmi wmj kurt(sm ) (11.8)

This proves that matrices of the form in (11.5) are eigenmatrices of the tensor. The
corresponding eigenvalues are given by the kurtoses of the independent components.
Moreover, it can be proven that all other eigenvalues of the tensor are zero.
Thus we see that if we knew the eigenmatrices of the cumulant tensor, we could
easily obtain the independent components. If the eigenvalues of the tensor, i.e., thekurtoses of the independent components, are distinct, every eigenmatrix corresponds
T , giving one of the columns of theto a nonzero eigenvalue of the form m mwhitened mixing matrix.
If the eigenvalues are not distinct, the situation is more problematic: The eigenmatrices are no longer uniquely defined, since any linear combinations of the matrices
T corresponding to the same eigenvalue are eigenmatrices of the tensor asm mwell. Thus, every k -fold eigenvalue corresponds to k matrices i ; i = 1; :::; k thatare different linear combinations of the matrices i(j ) iT(j ) corresponding to the k
ICs whose indices are denoted by i(j ). The matrices i can be thus expressed as:

w w

w w

Mi =

k
X
j =1

w w
M

j wi(j ) wiT(j )M(11.9)232

ICA BY TENSORIAL METHODS

Now, vectors that can be used to construct the matrix in this way can be computedby the eigenvalue decomposition of the matrix: The i(j ) are the (dominant) eigenvectors of i .
Thus, after finding the eigenmatrices i of the cumulant tensor, we can decompose them by ordinary EVD, and the eigenvectors give the columns of the mixing
matrix i . Of course, it could turn out that the eigenvalues in this latter EVD are
equal as well, in which case we have to figure out something else. In the algorithms
given below, this problem will be solved in different ways.
This result leaves the problem of how to compute the eigenvalue decompositionof the tensor in practice. This will be treated in the next section.

w

M

M

w

11.3

COMPUTING THE TENSOR DECOMPOSITION BY A POWER
METHODIn principle, using tensorial methods is simple. One could take any method forcomputing the EVD of a symmetric matrix, and apply it on the cumulant tensor.
To do this, we must first consider the tensor as a matrix in the space of n  nmatrices. Let q be an index that goes though all the n  n couples (i; j ). Then wecan consider the elements of an n  n matrix
as a vector. This means that weare simply vectorizing the matrices. Then the tensor can be considered as a q  qsymmetric matrix with elements fqq0 = cum(zi ; zj ; zi0 ; zj 0 ), where the indices(i; j ) corresponds to q, and similarly for (i0 ; j 0 ) and q0 . It is on this matrix that we
could apply ordinary EVD algorithms, for example the well-known QR methods. The
special symmetricity properties of the tensor could be used to reduce the complexity.
Such algorithms are out of the scope of this book; see e.g. [62].
The problem with the algorithm in this category, however, is that the memory
requirements may be prohibitive, because often the coefficients of the fourth-ordertensor must be stored in memory, which requires O(n4 ) units of memory. The
computational load also grows quite fast. Thus these algorithms cannot be used inhigh-dimensional spaces. In addition, equal eigenvalues may give problems.
In the following we discuss a simple modification of the power method, thatcircumvents the computational problems with the tensor EVD. In general, the power
method is a simple way of computing the eigenvector corresponding to the largest
eigenvalue of a matrix. This algorithm consists of multiplying the matrix with therunning estimate of the eigenvector, and taking the product as the new value of thevector. The vector is then normalized to unit length, and the iteration is continued
until convergence. The vector then gives the desired eigenvector.
We can apply the power method quite simply to the case of the cumulant tensor.
Starting from a random matrix , we compute ( ) and take this as the new valueof . Then we normalize
and go back to the iteration step. After convergence,
will be of the form k k i(k) iT(k) . Computing its eigenvectors gives one ormore of the independent components. (In practice, though, the eigenvectors will
not be exactly of this form due to estimation errors.) To find several independent

M

F

M

M

PM

M
w w

FM

TENSOR DECOMPOSITION BY A POWER METHOD

233

components, we could simply project the matrix after every step on the space of
matrices that are orthogonal to the previously found ones.
In fact, in the case of ICA, such an algorithm can be considerably simplified.
Since we know that the matrices i iT are eigenmatrices of the cumulant tensor, we
can apply the power method inside that set of matrices= T only. After every
computation of the product with the tensor, we must then project the obtained matrix
T . A very simple way of doing this is toback to the set of matrices of the form
multiply the new matrix  by the old vector to obtain the new vector  = (which will be normalized as necessary). This can be interpreted as another power
method, this time applied on the eigenmatrix to compute its eigenvectors. Since theTbest way of approximating the matrix  in the space of matrices of the form
is by using the dominant eigenvector, a single step of this ordinary power method
will at least take us closer to the dominant eigenvector, and thus to the optimal vector.
Thus we obtain an iteration of the form

ww

M ww

ww

M

w

M

ww

w wT F(wwT )
or

X X

wi

j

wj

kl

Mw

(11.10)wk wl cum(zi ; zj ; zk ; zl )(11.11)

In fact, this can be manipulated algebraically to give much simpler forms. We haveequivalentlywicum(zi ;Xjwj zj ;PXk

wk zk ;

Xl

wl zl ) = cum(zi ; y; y; y)(11.12)where we denote by y = i wi zi the estimate of an independent component. By
definition of the cumulants, we havecum(zi ; y; y; y ) = E fzi y 3 g ? 3E fzi y gE fy 2 g(11.13)

We can constrain y to have unit variance, as usual. Moreover, we have E fzi y g = wi .
Thus we havew E fzy g ? 3w
3w

(11.14)

where is normalized to unit norm after every iteration. To find several independent components, we can actually just constrain thecorresponding to different
independent components to be orthogonal, as is usual for whitened data.
Somewhat surprisingly, (11.14) is exactly the FastICA algorithm that was derived
as a fixed-point iteration for finding the maxima of the absolute value of kurtosis in
Chapter 8, see (8.20). We see that these two methods lead to the same algorithm.

w

234

ICA BY TENSORIAL METHODS

11.4

JOINT APPROXIMATE DIAGONALIZATION OF EIGENMATRICES

Joint approximate diagonalization of eigenmatrices (JADE) refers to one principle ofsolving the problem of equal eigenvalues of the cumulant tensor. In this algorithm,
the tensor EVD is considered more as a preprocessing step.
Eigenvalue decomposition can be viewed as diagonalization. In our case, the dediagonalizes
velopments in Section 11.2 can be rephrased as follows: The matrix( ) for any . In other words,( ) T is diagonal. This is because thematrix is of a linear combination of terms of the form i iT , assuming that the
ICA model holds.
Thus, we could take a set of different matrices i ; i = 1; :::; k , and try to make( i ) as diagonal as possible. In practice, they cannot be madethe matrices
exactly diagonal because the model does not hold exactly, and there are sampling
errors.
The diagonality of a matrix =( i ) T can be measured, for example,2
as the sum of the squares of off-diagonal elements:
k6=l qkl . Equivalently, sincedoes not change the total sum of squares of a matrix,
an orthogonal matrix
minimization of the sum of squares of off-diagonal elements is equivalent to the
maximization of the sum of squares of diagonal elements. Thus, we could formulate
the following measure:

FM

F

M

W

WF M WM

WF M WW

ww

Q WF M W P

JJADE (W) =Xikdiag(WF(Mi )WT )k2(11.15)where kdiag(:)k2 means the sum of squares of the diagonal. Maximization of JJADE
is then one method of joint approximate diagonalization of the ( i ).
How do we choose the matrices i ? A natural choice is to take the eigenmatricesof the cumulant tensor. Thus we have a set of just n matrices that give all the relevant
information on the cumulants, in the sense that they span the same subspace as the
cumulant tensor. This is the basic principle of the JADE algorithm.
Another benefit associated with this choice of the i is that the joint diagonalization criterion is then a function of the distributions of the =and a clear
link can be made to methods of previous chapters. In fact, after quite complicatedalgebraic manipulations, we can obtain

FM

M

M

JJADE (W) =X

ijkl6=iikly Wzcum(yi ; yj ; yk ; yl )2(11.16)in other words, when we minimize JJADE we also minimize a sum of the squared
cross-cumulants of the yi . Thus, we can interpret the method as minimizing nonlinear
correlations.
JADE suffers from the same problems as all methods using an explicit tensor
EVD. Such algorithms cannot be used in high-dimensional spaces, which pose no
problem for the gradient or fixed-point algorithm of Chapters 8 and 9. In problemsof low dimensionality (small scale), however, JADE offers a competitive alternative.

WEIGHTED CORRELATION MATRIX APPROACH

11.5

235

WEIGHTED CORRELATION MATRIX APPROACH

A method closely related to JADE is given by the eigenvalue decomposition of theweighted correlation matrix. For historical reasons, the basic method is simply called
fourth-order blind identification (FOBI).
11.5.1The FOBI algorithm

Consider the matrix= E fzzT kzk2 g(11.17)

Assuming that the data follows the whitened ICA model, we have

= E fVAssT (VA)T kVAsk2 g = WT E fssT ksk2gW(11.18)VA, and denoted the separating matrix bywhere we have used the orthogonality of= ( )T . Using the independence of the si , we obtain (see exercices)W

VA= WT diag(E fs2i ksk2g)W = WT diag(E fs4i g + n ? 1)W(11.19)Now we see that this is in fact the eigenvalue decomposition of . It consists of theorthogonal separating matrix
and the diagonal matrix whose entries depend on
the fourth-order moments of the si . Thus, if the eigenvalue decomposition is unique,which is the case if the diagonal matrix has distinct elements, we can simply computethe decomposition on , and the separating matrix is obtained immediately.
FOBI is probably the simplest method for performing ICA. FOBI allows the computation of the ICA estimates using standard methods of linear algebra on matricesof reasonable complexity (n  n). In fact, the computation of the eigenvalue decomposition of the matrix is of the same complexity as whitening the data. Thus,
this method is computationally very efficient: It is probably the most efficient ICA
method that exists.
However, FOBI works only under the restriction that the kurtoses of the ICs
are all different. (If only some of the ICs have identical kurtoses, those that havedistinct kurtoses can still be estimated). This restricts the applicability of the method
considerably. In many cases, the ICs have identical distributions, and this method
fails completely.

W

11.5.2

From FOBI to JADE

Now we show how we can generalize FOBI to get rid of its limitations, which actuallyleads us to JADE.
First, note that for whitened data, the definition of the cumulant can be written as

F(M) = E f(zT Mz)zzT g ? 2M ? tr(M)I(11.20)

236

ICA BY TENSORIAL METHODS

which is left as an exercice. Thus, we could alternatively define the weighted
correlation matrix using the tensor as= F(I)(11.21)

F(I) = E fkzk zzT g ? (n + 2)I(11.22)

because we have2

and the identity matrix does not change the EVD in any significant way.
Thus we could take some matrix
and use the matrix ( ) in FOBI insteadof ( ). This matrix would have as its eigenvalues some linear combinations of the
cumulants of the ICs. If we are lucky, these linear combinations could be distinct,
and FOBI works. But the more powerful way to utilize this general definition is totake several matrices ( i ) and jointly (approximately) diagonalize them. But thisis what JADE is doing, for its particular set of matrices! Thus we see how JADE is a
generalization of FOBI.

M

FI

FM

FM

11.6

CONCLUDING REMARKS AND REFERENCES

An approach to ICA estimation that is rather different from those in the previous
chapters is given by tensorial methods. The fourth-order cumulants of mixtures give
all the fourth-order information inherent in the data. They can be used to definea tensor, which is a generalization of the covariance matrix. Then we can apply
eigenvalue decomposition on this matrix. The eigenvectors more or less directly givethe mixing matrix for whitened data. One simple way of computing the eigenvalue
decomposition is to use the power method that turns out to be the same as the FastICAalgorithm with the cubic nonlinearity. Joint approximate diagonalization of eigenmatrices (JADE) is another method in this category that has been successfully used in
low-dimensional problems. In the special case of distinct kurtoses, a computationallyvery simple method (FOBI) can be devised.
The tensor methods were probably the first class of algorithms that performed
ICA successfully. The simple FOBI algorithm was introduced in [61], and the tensor
structure was first treated in [62, 94]. The most popular algorithm in this category
is probably the JADE algorithm as proposed in [72]. The power method given
by FastICA, another popular algorithm, is not usually interpreted from the tensor
viewpoint, as we have seen in preceding chapters. For an alternative form of thepower method, see [262]. A related method was introduced in [306]. An in-depth
overview of the tensorial method is given in [261]; see also [94]. An accessible
and fundamental paper is [68] that also introduces sophisticated modifications of themethods. In [473], a kind of a variant of the cumulant tensor approach was proposedby evaluating the second derivative of the characteristic function at arbitrary points.
The tensor methods, however, have become less popular recently. This is because
methods that use the whole EVD (like JADE) are restricted, for computational reasons, to small dimensions. Moreover, they have statistical properties inferior to thosePROBLEMS

237

methods using nonpolynomial cumulants or likelihood. With low-dimensional data,
however, they can offer an interesting alternative, and the power method that boils
down to FastICA can be used in higher dimensions as well.
Problems

W diagonalizes F(M) as claimed in Section 11.4.

11.1

Prove that

11.2

Prove (11.19)

11.3

Prove (11.20).

Computer assignments
11.1

Compute the eigenvalue decomposition of random fourth-order tensors of size2  2  2  2 and 5  5  5  5. Compare the computing times. What about a tensor
of size 100  100  100  100?

11.2 Generate 2-D data according to the ICA model. First, with ICs of different
distributions, and second, with identical distributions. Whiten the data, and performthe FOBI algorithm in Section 11.5. Compare the two cases.

12
ICA by Nonlinear
Decorrelation and
Nonlinear PCA
This chapter starts by reviewing some of the early research efforts in independent
component analysis (ICA), especially the technique based on nonlinear decorrelation,that was successfully used by Jutten, Hérault, and Ans to solve the first ICA problems.
Today, this work is mainly of historical interest, because there exist several more
efficient algorithms for ICA.
Nonlinear decorrelation can be seen as an extension of second-order methods
such as whitening and principal component analysis (PCA). These methods givecomponents that are uncorrelated linear combinations of input variables, as explainedin Chapter 6. We will show that independent components can in some cases be foundas nonlinearly uncorrelated linear combinations. The nonlinear functions used inthis approach introduce higher order statistics into the solution method, making ICA
possible.
We then show how the work on nonlinear decorrelation eventually lead to the
Cichocki-Unbehauen algorithm, which is essentially the same as the algorithm that
we derived in Chapter 9 using the natural gradient. Next, the criterion of nonlinear
decorrelation is extended and formalized to the theory of estimating functions, andthe closely related EASI algorithm is reviewed.
Another approach to ICA that is related to PCA is the so-called nonlinear PCA.
A nonlinear representation is sought for the input data that minimizes a least meansquare error criterion. For the linear case, it was shown in Chapter 6 that pr incipal
components are obtained. It turns out that in some cases the nonlinear PCA approachgives independent components instead. We review the nonlinear PCA criterion andshow its equivalence to other criteria like maximum likelihood (ML). Then, twotypical learning rules introduced by the authors are reviewed, of which the first one
239

240

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

is a stochastic gradient algorithm and the other one a recursive least mean-squarealgorithm.

12.1

NONLINEAR CORRELATIONS AND INDEPENDENCE

The correlation between two random variables y1 and y2 was discussed in detail in
Chapter 2. Here we consider zero-mean variables only, so correlation and covarianceare equal. Correlation is related to independence in such a way that independent
variables are always uncorrelated. The opposite is not true, however: the variables
can be uncorrelated, yet dependent. An example is a uniform density in a rotatedsquare centered at the origin of the (y1 ; y2 ) space, see e.g. Fig. 8.3. Both y1 andy2 are zero mean and uncorrelated, no matter what the orientation of the square, butthey are independent only if the square is aligned with the coordinate axes. In somecases uncorrelatedness does imply independence, though; the best example is thecase when the density of (y1 ; y2 ) is constrained to be jointly gaussian.
Extending the concept of correlation, we here define the nonlinear correlation of
the random variables y1 and y2 as Eff (y1 )g (y2 )g. Here, f (y1 ) and g (y2 ) are twofunctions, of which at least one is nonlinear. Typical examples might be polynomialsof degree higher than 1, or more complex functions like the hyperbolic tangent. Thismeans that one or both of the random variables are first transformed nonlinearly to
new variables f (y1 ); g (y2 ) and then the usual linear correlation between these newvariables is considered.
The question now is: Assuming that y1 and y2 are nonlinearly decorrelated in thesense
Eff (y1 )g (y2 )g = 0(12.1)

can we say something about their independence? We would hope that by making
this kind of nonlinear correlation zero, independence would be obtained under some
additional conditions to be specified.
There is a general theorem (see, e.g., [129]) stating that y1 and y2 are independent
if and only if
Eff (y1 )g (y2 )g = Eff (y1 )gEfg (y2 )g(12.2)

for all continuous functions f and g that are zero outside a finite interval. Basedon this, it seems very difficult to approach independence rigorously, because thefunctions f and g are almost arbitrary. Some kind of approximations are needed.
This problem was considered by Jutten and Hérault [228]. Let us assume that f (y1)and g (y2 ) are smooth functions that have derivatives of all orders in a neighborhood

NONLINEAR CORRELATIONS AND INDEPENDENCE

241

of the origin. They can be expanded in Taylor series:

f (y1 ) = f (0) + f 0 (0)y1 + 21 f 00 (0)y12 + :::
=

1
X
i=0

fi y1ig(y2 ) = g(0) + g0 (0)y2 + 12 g00 (0)y22 + :::
=

1
X
i=0

gi y2iwhere fi ; gi is shorthand for the coefficients of the ith powers in the series.
The product of the functions is thenf (y1 )g(y2 ) =1X
1X
i=1 j =1

fi gj y1i y2j(12.3)and condition (12.1) is equivalent toEff (y1 )g (y2 )g =1X
1X
i=1 j =1

fi gj Efy1i y2j g = 0(12.4)

Obviously, a sufficient condition for this equation to hold is
Efy1i y2j g = 0(12.5)

for all indices i; j appearing in the series expansion (12.4). There may be othersolutions in which the higher order correlations are not zero, but the coefficients fi ; gj
happen to be just suitable to cancel the terms and make the sum in (12.4) exactly
equal to zero. For nonpolynomial functions that have infinite Taylor expansions, suchspurious solutions can be considered unlikely (we will see later that such spurious
solutions do exist but they can be avoided by the theory of ML estimation).
Again, a sufficient condition for (12.5) to hold is that the variables y1 and y2 are
independent and one of Efy1i g; Efy2j g is zero. Let us require that E fy1i g = 0 for all
powers i appearing in its series expansion. But this is only possible if f (y1 ) is an oddfunction; then the Taylor series contains only odd powers 1; 3; 5; :::, and the powers
i in Eq. (12.5) will also be odd. Otherwise, we have the case that even moments ofy1 like the variance are zero, which is impossible unless y1 is constant.
To conclude, a sufficient (but not necessary) condition for the nonlinear uncorrelatedness (12.1) to hold is that y1 and y2 are independent, and for one of them, say
y1 , the nonlinearity is an odd function such that f (y1 ) has zero mean.
The preceding discussion is informal but should make it credible that nonlinear
correlations are useful as a possible general criterion for independence. Several things
have to be decided in practice: the first one is how to actually choose the functions
f; g. Is there some natural optimality criterion that can tell us that some functions

242

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

x1

y1
-m12
-m21

x2

y2

Fig. 12.1 The basic feedback circuit for the Hérault-Jutten algorithm. The element marked
with + is a summation

are better than some other ones? This will be answered in Sections 12.3 and 12.4.
The second problem is how we could solve Eq. (12.1), or nonlinearly decorrelate twovariables y1 ; y2 . This is the topic of the next section.
12.2

THE HÉRAULT-JUTTEN ALGORITHM

x

As. Let us first look at a 2  2 case, which was
Consider the ICA model =considered by Hérault, Jutten and Ans [178, 179, 226] in connection with the blindseparation of two signals from two linear mixtures. The model is thenx1 = a11 s1 + a12 s2
x2 = a21 s1 + a22 s2
Hérault and Jutten proposed the feedback circuit shown in Fig. 12.1 to solve the problem. The initial outputs are fed back to the system, and the outputs are recomputed
until an equilibrium is reached.
From Fig. 12.1 we have directly

y1 = x1 ? m12 y2
y2 = x2 ? m21 y1(12.6)(12.7)

Before inputting the mixture signals x1 ; x2 to the network, they were normalized tozero mean, which means that the outputs y1 ; y2 also will have zero means. Defining amatrixwith off-diagonal elements m12 ; m21 and diagonal elements equal to zero,these equations can be compactly written asM

y = x ? My
Thus the input-output mapping of the network is

y = (I + M)? x1(12.8)

THE CICHOCKI-UNBEHAUEN ALGORITHM

243

isNote that from the original ICA model we have = ?1 , provided that
invertible. If += , then becomes equal to . However, the problem in blindseparation is that the matrix is unknown.
The solution that Jutten and Hérault introduced was to adapt the two feedback
coefficients m12 ; m21 so that the outputs of the network y1 ; y2 become independent.
Then the matrix has been implicitly inverted and the original sources have beenfound. For independence, they used the criterion of nonlinear correlations. They
proposed the following learning rules:

I M A

A

s
s

y

A x

A

A

m12 = f (y1 )g(y2 )m21 = f (y2 )g(y1 )(12.9)(12.10)with  the learning rate. Both functions f (:); g (:) are odd functions; typically, thefunctionsf (y) = y3 ; g(y) = arctan(y)were used, although the method also seems to work for g (y ) = y or g (y ) = sign(y ).
Now, if the learning converges, then the right-hand sides must be zero on average,
implying
Eff (y1 )g (y2 )g = Eff (y2 )g (y1 )g = 0

Thus independence has hopefully been attained for the outputs y1 ; y2 . A stability
analysis for the Hérault-Jutten algorithm was presented by [408].
In the numerical computation of the matrix according to algorithm (12.9,12.10),
the outputs y1 ; y2 on the right-hand side must also be updated at each step of theiteration. By Eq. (12.8), they too depend on , and solving them requires theinversion of matrix + . As noted by Cichocki and Unbehauen [84], this matrix
inversion may be computationally heavy, especially if this approach is extended tomore than two sources and mixtures. One way to circumvent this problem is to makea rough approximation

M

M

I M

y = (I + M)? x  (I ? M)x1

that seems to work in practice.
Although the Hérault-Jutten algorithm was a very elegant pioneering solution to
the ICA problem, we know now that it has some drawbacks in practice. The algorithm
may work poorly or even fail to separate the sources altogether if the signals are badly
scaled or the mixing matrix is ill-conditioned. The number of sources that the method
can separate is severely limited. Also, although the local stability was shown in [408],
good global convergence behavior is not guaranteed.

12.3

THE CICHOCKI-UNBEHAUEN ALGORITHM

Starting from the Hérault-Jutten algorithm Cichocki, Unbehauen, and coworkers [82,85, 84] derived an extension that has a much enhanced performance and reliability.
Instead of a feedback circuit like the Hérault-Jutten network in Fig. 12.1, Cichocki

244

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

B
B

and Unbehauen proposed a feedforward network with weight matrix , with the
mixture vector for input and with output =. Now the dimensionality of the
problem can be higher than 2. The goal is to adapt the m  m matrix so that theelements of become independent. The learning algorithm for is as follows:

x

y Bx

y

B = [ ? f (y)g(yT )]B

B(12.11)

where  is the learning rate, is a diagonal matrix whose elements determine theamplitude scaling for the elements of (typically,
could be chosen as the unit
matrix ), and f and g are two nonlinear scalar functions; the authors proposed apolynomial and a hyperbolic tangent. The notation f ( ) means a column vector withelements f (y1 ); : : : ; f (yn ).
The argumentation showing that this algorithm will give independent components,
too, is based on nonlinear decorrelations. Consider the stationary solution of thislearning rule defined as the matrix for which E f g = 0, with the expectation
taken over the density of the mixtures . For this matrix, the update is on the average
zero. Because this is a stochastic-approximation-type algorithm (see Chapter 3), suchstationarity is a necessary condition for convergence. Excluding the trivial solution= 0, we must have
? Eff ( )g( T )g = 0

y

I


y

B

x

B



y y

Especially, for the off-diagonal elements, this implies
Eff (yi )g (yj )g = 0(12.12)

which is exactly our definition of nonlinear decorrelation in Eq. (12.1) extended to n
output signals y1 ; :::; yn . The diagonal elements satisfy

ii
showing that the diagonal elements ii of matrix  only control the amplitude scalingof the outputs.
The conclusion is that if the learning rule converges to a nonzero matrix B, thenEff (yi )g (yi )g =the outputs of the network must become nonlinearly decorrelated, and hopefully
independent. The convergence analysis has been performed in [84]; for generalprinciples of analyzing stochastic iteration algorithms like (12.11), see Chapter 3.
The justification for the Cichocki-Unbehauen algorithm (12.11) in the originalarticles was based on nonlinear decorrelations, not on any rigorous cost functions
that would be minimized by the algorithm. However, it is interesting to note that
this algorithm, first appearing in the early 1990’s, is in fact the same as the popularnatural gradient algorithm introduced later by Amari, Cichocki, and Young [12] asan extension to the original Bell-Sejnowski algorithm [36]. All we have to do ischooseas the unit matrix, the function g ( ) as the linear function g ( ) = ,and the function f ( ) as a sigmoidal related to the true density of the sources. The
Amari-Cichocki-Young algorithm and the Bell-Sejnowski algorithm were reviewed
in Chapter 9 and it was shown how the algorithms are derived from the rigorous
maximum likelihood criterion. The maximum likelihood approach also tells us whatkind of nonlinearities should be used, as discussed in Chapter 9.



y

y

y

y

THE ESTIMATING FUNCTIONS APPROACH *

12.4

245

THE ESTIMATING FUNCTIONS APPROACH *Consider the criterion of nonlinear decorrelations being zero, generalized to n randomvariables y1 ; :::; yn , shown in Eq. (12.12). Among the possible roots y1 ; :::; yn ofthese equations are the source signals s1 ; :::; sn . When solving these in an algorithm
like the Hérault-Jutten algorithm or the Cichocki-Unbehauen algorithm, one in factsolves the separating matrix .
This notion was generalized and formalized by Amari and Cardoso [8] to the case, = of estimating functions. Again, consider the basic ICA model =
whereis a true separating matrix (we use this special notation here to avoid any
confusion). An estimation function is a matrix-valued function ( ; ) such thatB

x As s B x
FxBB

F(x; B)g = 0:

Ef(12.13)xThis means that, taking the expectation with respect to the density of , the true
separating matrices are roots of the equation. Once these are solved from Eq. (12.13),
the independent components are directly obtained.

y Bx

Example 12.1 Given a set of nonlinear functions f1 (y1 ); :::; fn (yn ), with =,and defining a vector function ( ) = [f1 (y1 ); :::; fn (yn )]T , a suitable estimating
function for ICA isfy

F(x; B) =  ? f (y)yT =  ? f (Bx)(Bx)T(12.14)
because obviously Eff (y)yT g becomes diagonal when B is a true separating matrix
B and y ; :::; yn are independent and zero mean. Then the off-diagonal elements
become Effi (yi )yj g = Effi (yi )gEfyj g = 0. The diagonal matrix  determines the1scales of the separated sources. Another estimating function is the right-hand side ofthe learning rule (12.11),

F(x; B) = [ ? f (y)g(yT )]BThere is a fundamental difference in the estimating function approach compared tomost of the other approaches to ICA: the usual starting point in ICA is a cost functionthat somehow measures how independent or nongaussian the outputs yi are, and the
independent components are solved by minimizing the cost function. In contrast,
there is no such cost function here. The estimation function need not be the gradientof any other function. In this sense, the theory of estimating functions is very generaland potentially useful for finding ICA algorithms. For a discussion of this approachin connection with neural networks, see [328].
It is not a trivial question how to design in practice an estimation function so that
we can solve the ICA model. Even if we have two estimating functions that bothhave been shaped in such a way that separating matrices are their roots, what is arelevant measure to compare them? Statistical considerations are helpful here. Notethat in practice, the densities of the sources si and the mixtures xj are unknown in246

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

the ICA model. It is impossible in practice to solve Eq. (12.13) as such, because theexpectation cannot be formed. Instead, it has to be estimated using a finite sample of. Denoting this sample by (1); :::; (T ), we use the sample function

x

x

x

F(x; B)g  T1

Ef

T
X
t=1

F(x(t); B)

Its root ^ is then an estimator for the true separating matrix. Obviously (see Chapter4), the root ^ = ^ [ (1); :::; (T )] is a function of the training sample, and it ismeaningful to consider its statistical properties like bias and variance. This gives ameasure of goodness for the comparison of different estimation functions. The best
estimating function is one that gives the smallest error between the true separating
matrix  and the estimate ^ .
A particularly relevant measure is (Fisher) efficiency or asymptotic variance, as
the size T of the sample (1); :::; (T ) grows large (see Chapter 4). The goal is
to design an estimating function that gives the smallest variance, given the set ofobservations (t). Then the optimal amount of information is extracted from thetraining set.
The general result provided by Amari and Cardoso [8] is that estimating functionsof the form (12.14) are optimal in the sense that, given any estimating function ,one can always find a better or at least equally good estimating function (in the senseof efficiency) having the form

B

B

Bx

x

B

B

x

x

x

F

F(x; B) =  ? f (y)yT=  ? f (Bx)(Bx)T(12.15)(12.16)



where is a diagonal matrix. Actually, the diagonal matrix has no effect on theoff-diagonal elements of ( ; ) which are the ones determining the independence
between yi ; yj ; the diagonal elements are simply scaling factors.
The result shows that it is unnecessary to use a nonlinear function g ( ) instead ofas the other one of the two functions in nonlinear decorrelation. Only one nonlinear
function f ( ), combined with , is sufficient. It is interesting that functions of exactly
the type f ( ) T naturally emerge as gradients of cost functions such as likelihood;the question of how to choose the nonlinearity f ( ) is also answered in that case. A
further example is given in the following section.
The preceding analysis is not related in any way to the practical methods for finding
the roots of estimating functions. Due to the nonlinearities, closed-form solutions do
not exist and numerical algorithms have to be used. The simplest iterative stochastic
approximation algorithm for solving the roots of ( ; ) has the form

y

FxB

y

y
yy

y

y

FxB

B = ?F(x; B);(12.17)

with  an appropriate learning rate. In fact, we now discover that the learning rules(12.9), (12.10) and (12.11) are examples of this more general framework.

247

EQUIVARIANT ADAPTIVE SEPARATION VIA INDEPENDENCE

12.5

EQUIVARIANT ADAPTIVE SEPARATION VIA INDEPENDENCE

In most of the proposed approaches to ICA, the learning rules are gradient descentalgorithms of cost (or contrast) functions. Many cases have been covered in previous
chapters. Typically, the cost function has the form J ( ) = EfG( )g, with G
some scalar function, and usually some additional constraints are used. Here again= , and the form of the function G and the probability density of determine
the shape of the contrast function J ( ).
It is easy to show (see the definition of matrix and vector gradients in Chapter 3)
that

B

y Bx

y
x

B

@J (B) = Ef( @G(y) )xT g = Efg(y)xT g(12.18)@B@ywhere g(y) is the gradient of G(y). If B is square and invertible, then x = B?1 y
and we have

@J (B) = Efg(y)yT g(BT )?1(12.19)@B
For appropriate nonlinearities G(y), these gradients are estimating functions inythe sense that the elements of must be statistically independent when the gradientbecomes zero. Note also that in the form Eff ( ) T gg( T )?1 , the first factor( ) T has the shape of an optimal estimating function (except for the diagonal
elements); see eq. (12.15). Now we also know how the nonlinear function ( )can be determined: it is directly the gradient of the function G( ) appearing in theoriginal cost function.
Unfortunately, the matrix inversion ( T )?1 in (12.19) is cumbersome. Matrix
inversion can be avoided by using the so-called natural gradient introduced by Amari[4]. This is covered in Chapter 3. The natural gradient is obtained in this case bymultiplying the usual matrix gradient (12.19) from the right by matrix T , which
gives Ef ( ) T g . The ensuing stochastic gradient algorithm to minimize the costfunction J ( ) is then

gyy

gyy

B

gy

y

B

BB

gyy B
B

B = ?g(y)yT B(12.20)This learning rule again has the form of nonlinear decorrelations. Omitting the
diagonal elements in matrix in ( ) T , the off-diagonal elements have the same
form as in the Cichocki-Unbehauen algorithm (12.11), with the two functions now
given by the linear function and the gradient ( ).
This gradient algorithm can also be derived using the relative gradient introduced
by Cardoso and Hvam Laheld [71]. This approach is also reviewed in Chapter
3. Based on this, the authors developed their equivariant adaptive separation via
independence (EASI) learning algorithm. To proceed from (12.20) to the EASI
learning rule, an extra step must be taken. In EASI, as in many other learning
rules for ICA, a whitening preprocessing is considered for the mixture vectors(see Chapter 6). We first transform linearly to =whose elements zi have

gyy

y

gy

x

z Vx

x

248

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

zz

I

unit variances and zero covariances: E f T g = . As also shown in Chapter 6, anappropriate adaptation rule for whitening isV = (I ? zzT )V(12.21)The ICA model using these whitened vectors instead of the original ones becomesz = VAs, and it is easily seen that the matrix VA is an orthogonal matrix (a rotation).
Thus its inverse which gives the separating matrix is also orthogonal. As in earlier
chapters, let us denote the orthogonal separating matrix by W.
Basically, the learning rule for W would be the same as (12.20). However, as
noted by [71], certain constraints must hold in any updating of W if the orthogonality
is to be preserved at each iteration step. Let us denote the serial update for W using
W + DW, where now D = ?g(y)yT .
the learning rule (12.20), briefly, as W
The orthogonality condition for the updated matrix becomes(W + DW)(W + DW)T = I + D + DT + DDT = I
where WWT = I has been substituted. Assuming D small, the first-order approximation gives the condition that D = ?DT , or D must be skew-symmetric. Applying
this condition to the relative gradient learning rule (12.20) fory Wz

W = ?[g(y)yT ? yg(y)T ]W

W, we have(12.22)where now =. Contrary to the learning rule (12.20), this learning rule also
takes care of the diagonal elements of ( ) T in a natural way, without imposing
any conditions on them.
What is left now is to combine the two learning rules (12.21) and (12.22) into=
just one learning rule for the global system separation matrix. Because =, this global separation matrix is =. Assuming the same learning rates
for the two algorithms, a first order approximation gives

WVx

gyy

B WV

y Wz

B = WV + WV= ?[g(y)yT ? yg(y)T ]WV + [WV ? WzzT WT WV](12.23)= ?[yyT ? I + g(y)yT ? yg(y)T ]BThis is the EASI algorithm. It has the nice feature of combining both whitening
and separation into a single algorithm. A convergence analysis as well as someexperimental results are given in [71]. One can easily see the close connection to thenonlinear decorrelation algorithm introduced earlier.
The concept of equivariance that forms part of the name of the EASI algorithm
is a general concept in statistical estimation; see, e.g., [395]. Equivariance of an
estimator means, roughly, that its performance does not depend on the actual value ofthe parameter. In the context of the basic ICA model, this means that the ICs can be
estimated with the same performance what ever the mixing matrix may be. EASI was
one of the first ICA algorithms which was explicitly shown to be equivariant. In fact,
most estimators of the basic ICA model are equivariant. For a detailed discussion,see [69].

249

NONLINEAR PRINCIPAL COMPONENTS

12.6

NONLINEAR PRINCIPAL COMPONENTS

One of the basic definitions of PCA was optimal least mean-square error compression, as explained in more detail in Chapter 6. Assuming a random m-dimensional
zero-mean vector , we search for a lower dimensional subspace such that theresidual error between and its orthogonal projection on the subspace is minimal,
averaged over the probability density of . Denoting an orthonormal basis of thissubspace by 1 ; :::; n , the projection of on the subspace spanned by the basis is ni=1 ( iT ) i . Now n is the dimension of the subspace. The minimum
mean-square criterion for PCA is

x

P

x

x

w w
w xw

x?

minimize Efk

x

n
X
T
i=1

(wi x)wi k2g(12.24)A solution (although not the unique one) of this optimization problem is given by the
eigenvectors 1 ; :::; n of the data covariance matrix x = Ef T g. Then the linear
factors iT in the sum become the principal components Ti .
For instance, if is two-dimensional with a gaussian density, and we seek for a
one-dimensional subspace (a straight line passing through the center of the density),
then the solution is given by the principal axis of the elliptical density.
We now pose the question how this criterion and its solution are changed if anonlinearity is included in the criterion. Perhaps the simplest nontrivial nonlinear
extension is provided as follows. Assuming g1 (:); ::::; gn (:) a set of scalar functions,as yet unspecified, let us look at a modified criterion to be minimized with respect tothe basis vectors [232]:

wx

e

e
x

C

J (w1 :::wn ) = Efkx ?

n
X
i=1

xx
ex

gi (wiT x)wi k2 g

(12.25)This criterion was first considered by Xu [461] who called it the “least mean-squareerror reconstruction” (LMSER) criterion.
The only change with respect to (12.24) is that instead of the linear factors iT , wenow have nonlinear functions of them in the expansion that gives the approximation
to . In the optimal solution that minimizes the criterion J ( 1 ; :::; n ), suchfactors might be termed nonlinear principal components. Therefore, the technique offinding the basis vectors i is here called “nonlinear principal component analysis”(NLPCA).
It should be emphasized that practically always when a well-defined linear problem
is extended into a nonlinear one, many ambiguities and alternative definitions arise.
This is the case here, too. The term “nonlinear PCA” is by no means unique.
There are several other techniques, like the method of principal curves [167, 264]
or the nonlinear autoassociators [252, 325] that also give “nonlinear PCA”. In thesemethods, the approximating subspace is a curved manifold, while the solution to the
problem posed earlier is still a linear subspace. Only the coefficients corresponding
to the principal components are nonlinear functions of . It should be noted that

x

w

w

x

wx
w

250

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

minimizing the criterion (12.25) does not give a smaller least mean square error thanstandard PCA. Instead, the virtue of this criterion is that it introduces higher-order
statistics in a simple manner via the nonlinearities gi .
Before going into any deeper analysis of (12.25), it may be instructive to see ina simple special case how it differs from linear PCA and how it is in fact related to
ICA.
If the functions gi(y ) were linear, as in the standard PCA technique, and the number
n of terms in the sum were equal to m or the dimension of , then the representation
error always would be zero, as long as the weight vectors are chosen orthonormal.
For nonlinear functions gi (y ), however, this is usually not true. Instead, in somecases, at least, it turns out that the optimal basis vectors i minimizing (12.25) willbe aligned with the independent components of the input vectors.

x

w

x

Example 12.2 Assume that is a two-dimensional random vector that has a uniform
density in a unit square that is not aligned with the coordinate axes x1; x2 , according to
Fig. 12.2. Then it is easily shown that the elements x1 ; x2 are uncorrelated and haveequal variances (equal to 1/3), and the covariance matrix of is therefore equal to
1=3 . Thus, except for the scaling by 1=3, vector is whitened (sphered). However,ofthe elements are not independent. The problem is to find a rotation =such that the elements of the rotated vector are statistically independent. It is
obvious from Fig. 12.2 that the elements of must be aligned with the orientation of
the square, because then and only then the joint density is separable into the productof the two marginal uniform densities.
must
Because of the whitening, we know that the rows of the separating matrix
be orthogonal. This is seen by writing

I

x

x

x

s Wx

s

s

W

ss

Ef T g =WEfxxT gWT = 13 WWT(12.26)w w

Because the elements s1 and s2 are uncorrelated, it must hold that 1T 2 = 0.
The solution minimizing the criterion (12.25), with 1 ;2 orthogonal twodimensional vectors and g1 (:) = g2 (:) = g (:) a suitable nonlinearity, provides now
a rotation into independent components. This can be seen as follows. Assume thatg is a very sharp sigmoid, e.g., g(y) = tanh(10y), which is approximately the signfunction. The term 2i=1 g ( iT ) i in criterion (12.25) becomes

w w

P

w xw
w g(wT x) + w g(wT x)
 w sign(wT x) + w sign(wT x)
Thus according to (12.25), each x should be optimally represented by one of the four
possible points (w ; w ), with the signs depending on the angles between x and
1

1

1

2

1

2

2

1

2

2

the basis vectors. Each choice of the two orthogonal basis vectors divides the squareof Fig. 12.2 into four quadrants, and by criterion (12.25), all the points in a given
quadrant must be represented in the least mean square sense by just one point; e.g.,in the first quadrant where the angles between and the basis vectors are positive,by the point 1 + 2 . From Fig. 12.2, it can be seen that the optimal fit is obtained

w

w

x

THE NONLINEAR PCA CRITERION AND ICA

251

x2
s2
s1
w2

w1
x1

Fig. 12.2

A rotated uniform density

w w

when the basis vectors are aligned with the axes s1 ; s2 , and the point 1 + 2 is thecenter of the smaller square bordered by the positive s1 ; s2 axes.
For further confirmation, it is easy to compute the theoretical value of the costfunction J ( 1 ; 2 ) of Eq. (12.25) when the basis vectors 1 and 2 are arbitrary
orthogonal vectors [327]. Denoting the angle between 1 and the s1 axis in Fig.
12.2 by , we then have the minimal value of J ( 1 ; 2 ) for the rotation  = 0, andthen the lengths of the orthogonal vectors are equal to 0.5. These are the vectors
shown in Fig. 12.2.

w w

w w

w

w

w

x

In the preceding example, it was assumed that the density of is uniform. For
some other densities, the same effect of rotation into independent directions would
not be achieved. Certainly, this would not take place for gaussian densities withequal variances, for which the criterion J ( 1 ; :::; n ) would be independent of
the orientation. Whether the criterion results in independent components, depends
strongly on the nonlinearities gi (y ). A more detailed analysis of the criterion (12.25)
and its relation to ICA is given in the next section.

w

12.7

w

THE NONLINEAR PCA CRITERION AND ICAInterestingly, for prewhitened data, it can be shown [236] that the original nonlinear
PCA criterion of Eq. (12.25) has an exact relationship with other contrast func-

252

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

tions like kurtosis maximization/minimization, maximum likelihood, or the so-called
Bussgang criteria. In the prewhitened case, we use instead of to denote the input
vector. Also, assume that in whitening, the dimension of has been reduced to thatof . We denote this dimension by n. In this case, it has been shown before (seeT= T = .
Chapter 13) that matrix
is n  n and orthogonal: it holds
First, it is convenient to change to matrix formulation. Denoting by=( 1 ::: n )T the matrix that has the basis vectors i as rows, criterion (12.25) becomes

z

s

x

z

W

WW

w w

w

WW I
W

J (w1 ; :::; wn ) = J (W) = Efkz ? WT g(Wz)k2 g:(12.27)
The function g(Wz) is a column vector with elements g1 (w1T z); :::; gn (wnT z). We

can write now [236]kz ? WT g(Wz)k = [z ? WT g(Wz)]T [z ? WT g(Wz)]= [z ? WT g(Wz)]T WT W[z ? WT g(Wz)]= kWz ? WWT g(Wz)k= ky ? g(y)knX[yi ? gi (yi )] ;
=
2

2

2

2

i=1

with

y = Wz. Therefore the criterion J (W) becomes
JNLPCA(W) =

n
X
i=1

Ef[yi ? gi (yi )]2 g(12.28)This formulation of the NLPCA criterion can now be related to several other contrast
functions.
As the first case, choose gi (y ) as the odd quadratic function (the same for all i)gi (y) = y + y;if y  0?y + y; if y < 022Then the criterion (12.28) becomes

Jkurt (W) =

n
X
i=1

Ef(yi ? yi  yi2 )2 g =

n
X
i=1

Efyi4 g(12.29)This statistic was discussed in Chapter 8. Note that because the input data has been
whitened, the variance Efyi2 g = iT Ef g i = iT i = 1, so in the kurtosiskurt(yi ) = Efyi4 g ? 3(Efyi2 g)2 the second term is a constant and can be dropped
in kurtosis maximization/minimization. What remains is criterion (12.29). For thisfunction, minimizing the NLPCA criterion is exactly equivalent to minimizing thesum of the kurtoses of yi .
As a second case, consider the maximum likelihood solution of the ICA model.
The maximum likelihood solution starts from the assumption that the density of ,

w

zz w

w w

s

THE NONLINEAR PCA CRITERION AND ICA

s253

due to independence, is factorizable: p( ) = p1 (s1 )p2 (s2 ):::pn (sn ). Suppose wehave a large sample (1); ::::; (T ) of input vectors available. It was shown in
Chapter 9 that the log-likelihood becomesxx

log L(B) =T X
nX
t=1 i=1xlog pi (bTi x(t)) + T log j det Bj(12.30)= ?1 . In the case of a whitened
where the vectors i are the rows of matrix
sample (1); :::; (T ), the separating matrix will be orthogonal. Let us denote itagain by . We have

z
W

z

b

B A

log L(W) =T X
nX
t=1 i=1

log pi (wiT z(t))(12.31)The second term in (12.30) is zero, because the determinant of the orthogonal matrix
is equal to one.
Because this is a maximization problem, we can multiply the cost function (12.31)
by the constant 1=T . For large T , this function tends toW

JML (W) =n
X
i=1

Eflog pi (yi )g(12.32)w z

with yi = iT .
From this, we can easily derive the connection between the NLPCA criterion(12.28) and the ML criterion (12.32). In minimizing the sum (12.28), an arbitrary additive constant and a positive multiplicative constant can be trivially added.
Therefore, in the equivalence between the two criteria, we can consider the relation(dropping the subscript i from yi for convenience)log pi (y) = ? [y ? gi (y)]2
where

and

(12.33)> 0 are some constants, yieldingpi (y) / exp[? [y ? gi (y)]2 ](12.34)This shows how to choose the function gi (y ) for any given density pi (y ).
As the third case, the form of (12.28) is quite similar to the so-called Bussgang
cost function used in blind equalization (see [170, 171]). We use Lambert’s notation
and approach [256]. He chooses only one nonlinearity g1 (y ) = : : : = gn (y ) = g (y ):0g(y) = ?Efpy(yg)p (y)2(12.35)

The function p(y ) is the density of y and p0 (y )its derivative. Lambert [256] also gives
several algorithms for minimizing this cost function. Note that now in the whitened

254

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

data case the variance of y is equal to one and the function (12.35) is simply the scorefunction0? pp((yy)) = ? dyd log p(y)Due to the equivalence of the maximum likelihood criterion to several other criteria
like infomax or entropic criteria, further equivalences of the NLPCA criterion withthese can be established. More details are given in [236] and Chapter 14.

12.8

LEARNING RULES FOR THE NONLINEAR PCA CRITERION

Once the nonlinearities gi (y ) have been chosen, it remains to actually solve theminimization problem in the nonlinear PCA criterion. Here we present the simplest
learning algorithms for minimizing either the original NLPCA criterion (12.25) or
the prewhitened criterion (12.28). The first algorithm, the nonlinear subspace rule, isof the stochastic gradient descent type; this means that the expectation in the criterion
is dropped and the gradient of the sample function that only depends on the present
sample of the input vector ( or , respectively) is taken. This allows on-line learningin which each input vector is used when it comes available and then discarded;
see Chapter 3 for more details. This algorithm is a nonlinear generalization of thesubspace rule for PCA, covered in Chapter 6. The second algorithm reviewed in thissection, the recursive least-squares learning rule,is likewise a nonlinear generalizationof the PAST algorithm for PCA covered in Chapter 6.

x z

12.8.1

The nonlinear subspace ruleLet us first consider a stochastic gradient algorithm for the original cost function)k2 g.
(12.25), which in matrix form can be written as J ( ) = Efk ? T (This problem was considered by one of the authors [232, 233] as well as by Xu [461].
It was shown that the stochastic gradient algorithm isW

x W g WxW = [F(Wx)WrxT + g(Wx)rT ](12.36)r = x ? WT g(Wx)(12.37)F(Wx) = diag[g0(wT x); :::; g0(wnT x)]:(12.38)

whereis the residual error term and
1

There g 0 (y ) denotes the derivative of the function g (y ). We have made the simplifying
assumption here that all the functions g1 (:); :::; gn (:) are equal; a generalization to thecase of different functions would be straightforward but the notation would become
more cumbersome.
As motivated in more detail in [232], writing the update rule (12.36) for anindividual weight vector i shows that the first term in the brackets on the right-handw

LEARNING RULES FOR THE NONLINEAR PCA CRITERION

255

side of (12.36) affects the update of weight vectors much less than the second term,
if the error is relatively small in norm compared to the input vector . If the firstterm is omitted, then we obtain the following learning rule:

r

with

x

y the vector

W = g(y)[xT ? g(yT )W](12.39)y = Wx(12.40)Comparing this learning rule to the subspace rule for ordinary PCA, eq. (6.19)
in Chapter 6, we see that the algorithms are formally similar and become the sameif g is a linear function. Both rules can be easily implemented in the one-layer PCA
network shown in Fig. 6.2 of Chapter 6; the linear outputs yi = iT must only bechanged to nonlinear versions g (yi ) = g ( iT ). This was the way the nonlinear
PCA learning rule was first introduced in [332] as one of the extensions to numericalon-line PCA computation.
Originally, in [232] the criterion (12.25) and the learning scheme (12.39) weresuggested for signal separation, but the exact relation to ICA was not clear. Even
without prewhitening of the inputs , the method can separate signals to a certain
degree. However, if the inputs are whitened first, the separation performance isgreatly improved. The reason is that for whitened inputs, the criterion (12.25) and theconsequent learning rule are closely connected to well-known ICA objective (cost)functions, as was shown in Section 12.7.

wx

w x

x

12.8.2x

Convergence of the nonlinear subspace rule *Let us consider the convergence and behavior of the learning rule in the case that the
ICA model holds for the data. This is a very specialized section that may be skipped.
The prewhitened form of the learning rule iswithy now the vectorzzz

W = g(y)[zT ? g(yT )W](12.41)y = Wz(12.42)I

and white, thus Ef T g = . We also have to assume that the ICA model holds,
such thati.e., there exists an orthogonal separating matrixss = MzM(12.43)where the elements of are statistically independent. With whitening, the dimensionof has been reduced to that of ; thus bothandare n  n matrices.
To make further analysis easier, we proceed by making a linear transformation to
the learning rule (12.41): we multiply both sides by the orthogonal separating matrix
T , giving

z

M

s

M

W

(WMT ) = g(Wz)[zT MT ? g(zT WT )WMT ](12.44)= g(WMT Mz)[zT MT ? g(zT MT MWT )WMT ]256

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

where we have used the fact thatand using (12.43), we have

MT M = I. Denoting for the moment H = WMTH = g(Hs)[sT ? g(sT HT )H](12.45)This equation has exactly the same form as the original one (12.41). Geometricallythe transformation by the orthogonal matrix T simply means a coordinate change
to a new set of coordinates such that the elements of the input vector expressed inthese coordinates are statistically independent.
The goal in analyzing the learning rule (12.41) is to show that, starting from some
initial value, the matrix
will tend to the separating matrix . For the transformed
T in (12.45), this translates into the requirement thatweight matrix=
should tend to the unit matrix or a permutation matrix. Then =also would
tend to the vector , or a permuted version, with independent components.
However, it turns out that in the learning rule (12.45), the unit matrix or a permutation matrix generally cannot be the asymptotic or steady state solution. Thisis due to the scaling given by the nonlinearity . Instead, we can make the moregeneral requirement thattends to a diagonal matrix or a diagonal matrix times awill become the elements
permutation matrix. In this case the elements of =of the original source vector , in some order, multiplied by some numbers. In viewof the original problem, in which the amplitudes of the signals si remain unknown,
this is actually no restriction, as independence is again attained.
To proceed, the difference equation (12.45) can be further analyzed by writing
down the corresponding averaged differential equation; for a discussion of the technique, see Chapter 3. The limit of convergence of (12.45) is among the asymptoticallystable solutions of the averaged differential equation. In practice, this also requiresthat the learning rate  is decreasing to zero at a suitable rate.
Now taking averages in (12.45) and also using the same symbol= (t) forthe continuous-time counterpart of the transformed weight matrix , we obtain

M

W
H WM
s

M

H

y Hs

H

s

g
y Hs

H H
H
dH=dt = Efg(Hs)sT g ? Efg(Hs)g(sT HT )gH(12.46)The expectations are over the (unknown) density of vector s. We are ready to state
the main result of this section:

Theorem 12.1 In the matrix differential equation (12.46), assume the following:ss

1. The random vector has a symmetric density with Ef g = 0;s

2. The elements of , denoted s1 ; :::; sn , are statistically independent;

3. The function g (:) is odd, i.e., g (y ) = ?g (?y ) for all y , and at least twice
differentiable everywhere;

4. The function g (:) and the density ofhold for all i = 1; :::; n:s are such that the following conditions

Ai = Efs2i g0 ( i si )g ? 2 i Efg( i si )g0 ( i si )si g ? Efg2 ( i si )g < 0(12.47)

LEARNING RULES FOR THE NONLINEAR PCA CRITERION

257

where the i ; i = 1; :::; n are scalars satisfying
Efsi g ( i si )g = i Efg 2 ( i si )g(12.48)

5. Denoting

i2
Bi
Ci
Fi

=
=
=
=

Efs2i gEfg 2 ( i si )g
Efg 0 ( i si )gEfui g ( i si )gboth eigenvalues of the 2  2 matrix  C ? C F ?B
2
i j

i j i
? j Cj Fi

? i Ci F j
j Ci ? j Ci Fj ? Bij(12.49)(12.50)(12.51)(12.52)

2(12.53)have strictly negative real parts for all i; j = 1; :::; n.
Then the matrix

H = diag( ; :::; n)1(12.54)is an asymptotically stable stationary point of (12.46), where i satisfies Eq. (12.48).
The proof, as well as explanations of the rather technical conditions of the theorem,
are given in [327]. The main point is that the algorithm indeed converges to a diagonal
matrix, if the initial value (0)is not too far from it. Transforming back to the original
learning rule (12.41) for , it follows thatconverges to a separating matrix.
Some special cases were given in [327]. For example, if the nonlinearity is chosen
as the simple odd polynomial

HWW

g(y) = yq ; q = 1; 3; 5; 7; :::(12.55)then all the relevant variables in the conditions of the theorem, for any probability
density, will become moments of si . It can be shown (see exercises) that the stability
condition becomes
Efsq+1 g ? q Efs2 gEfsq?1 g > 0(12.56)

Based on this, it can be shown that the linear function g(y ) = y never gives asymptotic
stability, while the cubic function g (y ) = y 3 leads to asymptotic stability provided
that the density of s satisfies
Efs4 g ? 3(Efs2 g)2 > 0(12.57)This expression is exactly the kurtosis or the fourth order cumulant of s [319]. If
and only if the density is positively kurtotic (supergaussian), the stability conditionis satisfied for the cubic polynomial g (y ) = y 3 .

258

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

S1

S2

S3

S4

S5

S6

S7

S8

S9

Fig. 12.3

The original images.

Example 12.3 The learning rule was applied to a signal separation problem in [235].
Consider the 9 digital images shown in Fig. 12.3. They were linearly mixed with arandomly chosen mixing matrix into 9 mixture images, shown in Fig. 12.4.
Whitening, shown in Fig. 12.5, is not able to separate the images. When thelearning rule (12.41) was applied to the mixtures with a tanh nonlinearity and thematrix
was allowed to converge, it was able to separate the images as shown in
Fig. 12.6. In the figure, the images have been scaled to fit the gray levels in use; insome cases, the sign has also been reversed to avoid image negatives.

AW

12.8.3The nonlinear recursive least-squares learning rule

It is also possible to effectively minimize the prewhitened NLPCA criterion (12.27)
using approximative recursive least-squares (RLS) techniques. Generally, RLS algorithms converge clearly faster than their stochastic gradient counterparts, and achievea good final accuracy at the expense of a somewhat higher computational load. TheseLEARNING RULES FOR THE NONLINEAR PCA CRITERION

M1

M2

M3

M4

M5

M6

M7

M8

M9

Fig. 12.4

259

The mixed images.

advantages are the result of the automatic determination of the learning rate parameterfrom the input data, so that it becomes roughly optimal.
The basic symmetric algorithm for the prewhitened problem was derived by one of
the authors [347]. This is a nonlinear modification of the PAST algorithm introduced
by Yang for the standard linear PCA [466, 467]; the PAST algorithm is covered in
Chapter 6. Using index t to denote the iteration step, the algorithm isq(t) = g(W(t ? 1)z(t)) = g(y(t))(12.58)h(t) = P(t ? 1)q(t)(12.59)Tm(t) = h(t)=[ + q (t)h(t)](12.60)1
P(t) = Tri[P(t ? 1) ? m(t)hT (t)](12.61)r(t) = z(t) ? WT (t ? 1)q(t)(12.62)T
W(t) = W(t ? 1) + m(t)r (t):(12.63)The vector variables q(t); h(t); m(t), and r(t) and the matrix variable P(t) areauxiliary variables, internal to the algorithm. As before, z(t) is the whitened input

260

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

W1

W2

−W3

−W4

W5

W6

−W7

−W8

−W9

Fig. 12.5yThe whitened images.

Wgvector, (t) is the output vector, (t) is the weight matrix, and is the nonlinearity
in the NLPCA criterion. The parameter is a kind of “forgetting constant” thatshould be close to unity. The notation Tri means that only the upper triangular partof the matrix is computed and its transpose is copied to the lower triangular part,(0) and (0) can be
making the resulting matrix symmetric. The initial values
chosen as identity matrices.
This algorithm updates the whole weight matrix(t) simultaneously, treating(t) in a symmetric way. Alternatively, it is possible to computeall the rows of
the weight vectors i (t) in a sequential manner using a deflation technique. The
sequential algorithm is presented in [236]. The authors show there experimentallythat the recursive least-squares algorithms perform better and have faster convergencethan stochastic gradient algorithms like the nonlinear subspace learning rule. Yet, therecursive algorithms are adaptive and can be used for tracking if the statistics of thedata or the mixing model are slowly varying. They seem to be robust to initial values
and have relatively low computational load. Also batch versions of the recursivealgorithm are derived in [236].

W

W

W

w

P

CONCLUDING REMARKS AND REFERENCES

Fig. 12.6

12.9−(W+NPCA1)

W+NPCA2−(W+NPCA3)−(W+NPCA4)

W+NPCA5

W+NPCA6

W+NPCA7

W+NPCA8

W+NPCA9

261

The separated images using the nonlinear PCA criterion and learning rule.

CONCLUDING REMARKS AND REFERENCES

The first part of this chapter reviewed some of the early research efforts in ICA,
especially the technique based on nonlinear decorrelations. It was based on thework of Jutten, Hérault, and Ans [178, 179, 16]. A good overview is [227]. The
exact relation between the nonlinear decorrelation criterion and independence wasanalyzed in the series of papers [228, 93, 408]. The Cichocki-Unbehauen algorithm
was introduced in [82, 85, 84]; see also [83]. For estimating functions, the reference is[8]. The EASI algorithm was derived in [71]. The efficiency of estimating functions
can in fact be extended to the notion of superefficiency [7].
Somewhat related methods specialized for discrete-valued ICs were proposed in[286, 379].
The review of nonlinear PCA is based on the authors’ original works [332, 232,
233, 450, 235, 331, 327, 328, 347, 236]. A good review is also [149]. Nonlinear
PCA is a versatile and useful starting point for blind signal processing. It has close
connections to other well-known ICA approaches, as was shown in this chapter; see[236, 329]. It is unique in the sense that it is based on a least mean-square error

262

ICA BY NONLINEAR DECORRELATION AND NONLINEAR PCA

formulation of the ICA problem. Due to this, recursive least mean-square algorithms
can be derived; several versions like the symmetric, sequential, and batch algorithms
are given in [236].
Problems
12.1 In the Hérault-Jutten algorithm (12.9,12.10), let f (y1 ) = y13 and f (y2 ) = y2 .
Write the update equations so that only x1 ; x2 ; m12 , and m21 appear on the right-hand
side.

y Wz

12.2 Consider the cost function (12.29). Assuming =, compute the matrix
gradient of this cost function with respect to . Show that, except for the diagonal
elements, the matrix gradient is an estimating function, i.e., its off-diagonal elements
is a true separating matrix for which= . What are thebecome zero when
diagonal elements?

W

W

Wz s

12.3 Repeat the previous problem for the maximum likelihood cost function(12.32).
12.4 Consider the stationary points of (12.46). Show that the diagonal matrix(12.54) is a stationary point if (12.48) holds.
12.5 * In Theorem 12.1, let the nonlinearity be a simple polynomial: g (y ) = y q
with q an odd positive integer. Assume for simplicity that all the sources si have thesame density, so the subscript i can be dropped in the Theorem.
12.5.1. Solve from Eq. (12.48).
12.5.2. Show that the stability conditions reduce to Eq. (12.56).
12.5.3. Show that the linear function g (y ) = y does not fulfill the stability
condition.
12.6 Consider the nonlinear subspace learning rule for whitened inputs,Eq. (12.41).
Let us combine this rule with the whitening rule (12.21) in the same way as was done toand  = +  .
derive the EASI algorithm (12.23): writing =Like in the EASI derivation, assume thatis approximately orthogonal. Show that
we get the new learning rule

B WV
W

B

B = [g(y)yT ? g(y)g(yT ) + I ? yyT ]:WV W V13
Practical Considerations

In the preceding chapters, we presented several approaches for the estimation of
the independent component analysis (ICA) model. In particular, several algorithms
were proposed for the estimation of the basic version of the model, which has asquare mixing matrix and no noise. Now we are, in principle, ready to apply thosealgorithms on real data sets. Many such applications will be discussed in Part IV.
However, when applying the ICA algorithms to real data, some practical considerations arise and need to be taken into account. In this chapter, we discussdifferent problems that may arise, in particular, overlearning and noise in the data.
We also propose some preprocessing techniques (dimension reduction by principal
component analysis, time filtering) that may be useful and even necessary before theapplication of the ICA algorithms in practice.

13.1

PREPROCESSING BY TIME FILTERINGThe success of ICA for a given data set may depend crucially on performing some
application-dependent preprocessing steps. In the basic methods discussed in theprevious chapters, we always used centering in preprocessing, and often whitening
was done as well. Here we discuss further preprocessing methods that are notnecessary in theory, but are often very useful in practice.

263

264

PRACTICAL CONSIDERATIONS

13.1.1

Why time filtering is possible

In many cases, the observed random variables are, in fact, time signals or time series,
which means that they describe the time course of some phenomenon or system.
Thus the sample index t in xi (t) is a time index. In such a case, it may be very useful
to filter the signals. In other words, this means taking moving averages of the timeseries. Of course, in the ICA model no time structure is assumed, so filtering is notalways possible: If the sample points (t) cannot be ordered in any meaningful way
with respect to t, filtering is not meaningful, either.

x

For time series, any linear filtering of the signals is allowed, since it does notchange the ICA model. In fact, if we filter linearly the observed signals xi (t) toobtain new signals, say xi (t), the ICA model still holds for xi (t), with the same
mixing matrix. This can be seen as follows. Denote by the matrix that contains
the observations (1); :::; (T ) as its columns, and similarly for . Then the ICA
model can be expressed as:

x

X

x

X = AS(13.1)

X

Now, time filtering of corresponds to multiplying
us call it . This gives

MS

X from the right by a matrix, letX = XM = ASM = AS(13.2)

which shows that the ICA model still remains valid. The independent components
are filtered by the same filtering that was applied on the mixtures. They are notmixed with each other in  because the matrix
is by definition a component-wisefiltering matrix.

S

M

Since the mixing matrix remains unchanged, we can use the filtered data in the
ICA estimating method only. After estimating the mixing matrix, we can apply thesame mixing matrix on the original data to obtain the independent components.
The question then arises what kind of filtering could be useful. In the following,
we consider three different kinds of filtering: high-pass and low-pass filtering, as
well as their compromise.

PREPROCESSING BY TIME FILTERING

13.1.2265

Low-pass filtering

Basically, low-pass filtering means that every sample point is replaced by a weighted
average of that point and the points immediately before it.1 This is a form ofin (13.2) would be something likesmoothing the data. Then the matrix

M

1
0
..
.
BB: : : 1 1 1 0 0 0 0 0 : : :CC
BB: : : 0 1 1 1 0 0 0 0 : : :CC
BB: : : 0 0 1 1 1 0 0 0 : : :CC
1
M= 3B
BB: : : 0 0 0 1 1 1 0 0 : : :CCC
BB: : : 0 0 0 0 1 1 1 0 : : :CC
B@: : : 0 0 0 0 0 1 1 1 : : :CA.

(13.3)..

Low-pass filtering is often used because it tends to reduce noise. This is a wellknown property in signal processing that is explained in most basic signal processing
textbooks.
In the basic ICA model, the effect of noise is more or less neglected; see Chapter 15
for a detailed discussion. Thus basic ICA methods work much better with data that
does not have much noise, and reducing noise is thus useful and sometimes evennecessary.
A possible problem with low-pass filtering is that it reduces the information in thedata, since the fast-changing, high-frequency features of the data are lost. It often
happens that this leads to a reduction of independence as well (see next section).
13.1.3High-pass filtering and innovations

High-pass filtering is the opposite of low-pass filtering. The point is to remove slowly
changing trends from the data. Thus a low-pass filtered version is subtracted fromthe signal. A classic way of doing high-pass filtering is differencing, which means
replacing every sample point by the difference between the value at that point andin (13.2) would be
the value at the preceding point. Thus, the matrix

M

0
1
..
BB: : : 1 ?1 0 0. 0 0 0 : : :CC
BB: : : 0 1 ?1 0 0 0 0 : : :CC
BB: : : 0 0 1 ?1 0 0 0 : : :CC
M=B
BB: : : 0 0 0 1 ?1 0 0 : : :CCC
BB: : : 0 0 0 0 1 ?1 0 : : :CC
B@: : : 0 0 0 0 0 1 1 : : :CA.
..

1 To have a causal filter, points after the current point may be left out of the averaging.

(13.4)

266

PRACTICAL CONSIDERATIONSHigh-pass filtering may be useful in ICA because in certain cases it increases
the independence of the components. It often happens in practice that the components have slowly changing trends or fluctuations, in which case they are not very
independent. If these slow fluctuations are removed by high-pass filtering the filtered components are often much more independent. A more principled approach tohigh-pass filtering is to consider it in the light of innovation processes.

sssInnovation processes Given a stochastic process (t), we define its innovation
process ~(t) as the error of the best prediction of (t), given its past. Such a best
prediction is given by the conditional expectation of (t) given its past, because itis the expected value of the conditional distribution of (t) given its past. Thus theinnovation process of ~(t) is defined by

s

s

s

~s(t) = s(t) ? E fs(t)js(t ? 1); s(t ? 2); :::g(13.5)The expression “innovation” describes the fact that ~s(t) contains all the new information about the process that can be obtained at time t by observing s(t).
The concept of innovations can be utilized in the estimation of the ICA model dueto the following property:xss

Theorem 13.1 If (t) and (t) follow the basic ICA model, then the innovation
processes ~ (t) and ~(t) follow the ICA model as well. In particular, the components
s~i (t) are independent from each other.

x

On the other hand, independence of the innovations does not imply the independence of the si (t). Thus, the innovations are more often independent from eachother than the original processes. Moreover, one could argue that the innovations
are usually more nongaussian than the original processes. This is because the si (t)is a kind of moving average of the innovation process, and sums tend to be more
gaussian than the original variable. Together these mean that the innovation process
is more susceptible to be independent and nongaussian, and thus to fulfill the basic
assumptions in ICA.
Innovation processes were discussed in more detail in [194], where it was also
shown that using innovations, it is possible to separate signals (images of faces) thatare otherwise strongly correlated and very difficult to separate.
The connection between innovations and ordinary filtering techniques is that thecomputation of the innovation process is often rather similar to high-pass filtering.
Thus, the arguments in favor of using innovation processes apply at least partly infavor of high-pass filtering.
A possible problem with high-pass filtering, however, is that it may increase noisefor the same reasons that low-pass filtering decreases noise.
13.1.4

Optimal filtering

Both of the preceding types of filtering have their pros and cons. The optimum would
be to find a filter that increases the independence of the components while reducing

PREPROCESSING BY PCA

267

noise. To achieve this, some compromise between high- and low-pass filtering may
be the best solution. This leads to band-pass filtering, in which the highest and thelowest frequencies are filtered out, leaving a suitable frequency band in between.
What this band should be depends on the data and general answers are impossible togive.
In addition to simple low-pass/high-pass filtering, one might also use more sophisticated techniques. For example, one might take the (1-D) wavelet transforms ofthe data [102, 290, 17]. Other time-frequency decompositions could be used as well.

13.2

PREPROCESSING BY PCA

A common preprocessing technique for multidimensional data is to reduce its dimension by principal component analysis (PCA). PCA was explained in more detail in
Chapter 6. Basically, the data is projected linearly onto a subspacex~ = Enx(13.6)

so that the maximum amount of information (in the least-squares sense) is preserved.
Reducing dimension in this way has several benefits which we discuss in the next
subsections.
13.2.1

Making the mixing matrix squareFirst, let us consider the case where the the number of independent components n
is smaller than the number of mixtures, say m. Performing ICA on the mixtures
directly can cause big problems in such a case, since the basic ICA model does nothold anymore. Using PCA we can reduce the dimension of the data to n. After sucha reduction, the number of mixtures and ICs are equal, the mixing matrix is square,
and the basic ICA model holds.
The question is whether PCA is able to find the subspace correctly, so that then ICs can be estimated from the reduced mixtures. This is not true in general, but
in a special case it turns out to be the case. If the data consists of n ICs only, withno noise added, the whole data is contained in an n-dimensional subspace. Using
PCA for dimension reduction clearly finds this n-dimensional subspace, since theeigenvalues corresponding to that subspace, and only those eigenvalues, are nonzero.
Thus reducing dimension with PCA works correctly. In practice, the data is usually
not exactly contained in the subspace, due to noise and other factors, but if the noiselevel is low, PCA still finds approximately the right subspace; see Section 6.1.3. In
the general case, some “weak” ICs may be lost in the dimension reduction process,but PCA may still be a good idea for optimal estimation of the “strong” ICs [313].
Performing first PCA and then ICA has an interesting interpretation in terms offactor analysis. In factor analysis, it is conventional that after finding the factor
subspace, the actual basis vectors for that subspace are determined by some criteria

268

PRACTICAL CONSIDERATIONS

that make the mixing matrix as simple as possible [166]. This is called factor rotation.
Now, ICA can be interpreted as one method for determining this factor rotation, basedon higher-order statistics instead of the structure of the mixing matrix.
13.2.2

Reducing noise and preventing overlearning

A well-known benefit of reducing the dimension of the data is that it reduces noise,as was already discussed in Chapter 6. Often, the dimensions that have been omitted
consist mainly of noise. This is especially true in the case where the number of ICs
is smaller than the number of mixtures.
Another benefit of reducing dimensions is that it prevents overlearning, to which
the rest of this subsection is devoted. Overlearning means that if the number ofparameters in a statistical model is too large when compared to the number ofavailable data points, the estimation of the parameters becomes difficult, maybe
impossible. The estimation of the parameters is then too much determined by theavailable sample points, instead of the actual process that generated the data, which
is what we are really interested in.
Overlearning in ICA [214] typically produces estimates of the ICs that have asingle spike or bump, and are practically zero everywhere else. This is because in thespace of source signals of unit variance, nongaussianity is more or less maximized
by such spike/bump signals. This becomes easily comprehensible if we consider
the extreme case where the sample size T equals the dimension of the data m, andthese are both equal to the number of independent components n. Let us collect
the realizations (t) of as the columns of the matrix , and denote by thecorresponding matrix of the realizations of (t), as in (13.1). Note that now all the(andmatrices in (13.1) are square. This means that by changing the values of
keeping fixed), we can give any values whatsoever to the elements of . This isa case of serious overlearning, not unlike the classic case of regression with equal
numbers of data points and parameters.
that is obtained by ICA
Thus it is clear that in this case, the estimate ofestimation depends little on the observed data. Let us assume that the densities ofthe source signals are known to be supergaussian (i.e., positively kurtotic). Then the
ICA estimation basically consists of finding a separating matrix that maximizes ameasure of the supergaussianities (or sparsities) of the estimates of the source signals.
Intuitively, it is easy to see that sparsity is maximized when the source signals eachhave only one nonzero point. Thus we see that ICA estimation with an insufficient
sample size leads to a form of overlearning that gives artifactual (spurious) source
signals. Such source signals are characterized by large spikes.
An important fact shown experimentally [214] is that a similar phenomenon ismuch more likely to occur if the source signals are not independently and identically
distributed (i.i.d.) in time, but have strong time-dependencies. In such cases the
sample size needed to get rid of overlearning is much larger, and the source signals
are better characterized by bumps, i.e., low-pass filtered versions of spikes. An
intuitive way of explaining this phenomenon is to consider such a signal as being
constant on N=k blocks of k consecutive sample points. This means that the data can

x

x

X

s

X

S

S

B

S
A

HOW MANY COMPONENTS SHOULD BE ESTIMATED?

269

be considered as really having only N=k sample points; each sample point has simply
been repeated k times. Thus, in the case of overlearning, the estimation procedure
gives “spikes” that have a width of k time points, i.e., bumps.
Here we illustrate the phenomenon by separation of artificial source signals.
Three positively kurtotic signals, with 500 sample points each, were used in thesesimulations, and are depicted in Fig. 13.1 a. Five hundred mixtures were produced,
and a very small amount of gaussian noise was added to each mixture separately.
As an example of a successful ICA estimation, Fig. 13.1 b shows the result ofapplying the FastICA and maximum likelihood (ML) gradient ascent algorithms(denoted by “Bell-Sejnowski”) to the mixed signals. In both approaches, the preprocessing (whitening) stage included a dimension reduction of the data into the first
three principal components. It is evident that both algorithms are able to extract all
the initial signals.
In contrast, when the whitening is made with very small dimension reduction (wetook 400 dimensions), we see the emergence of spiky solutions (like Dirac functions),
which is an extreme case of kurtosis maximization (Fig. 13.1 c). The algorithm usedin FastICA was of a deflationary type, from which we plot the first five components
extracted. As for the ML gradient ascent, which was of a symmetric type, we show
five representative solutions to the 400 extracted.
Thus, we see here that without dimension reduction, we are not able to estimate
the source signals.
Fig. 13.1 d presents an intermediate stage of dimension reduction (from the original
500 mixtures we took 50 whitened vectors). We see that the actual source signals are
revealed by both methods, even though each resulting vector is more noisy than theones shown in Fig. 13.1 b.
For the final example, in Fig. 13.1 e, we low-pass filtered the mixed signals, prior to
the independent component analysis, using a 10 delay moving average filter. Taking
the same amount of principal components as in d, we can see that we lose all theoriginal source signals: the decompositions show a bumpy structure corresponding to
the low-pass filtering of the spiky outputs presented in c. Through low-pass filtering,
we have reduced the information contained in the data, and so the estimation isrendered impossible even with this, not very weak, dimension reduction. Thus, we
see that with this low-pass filtered data, a much stronger dimension reduction by
PCA is necessary to prevent overlearning.
In addition to PCA, some kind of prior information on the mixing matrix could be
useful in preventing overlearning. This is considered in detail in Section 20.1.3.

13.3

HOW MANY COMPONENTS SHOULD BE ESTIMATED?

Another problem that often arises in practice is to decide the number of ICs to be
estimated. This problem does not arise if one simply estimates the same numberof components as the dimension of the data. This may not always be a good idea,
however.

270

PRACTICAL CONSIDERATIONS(a)Fast ICA

Bell−Sejnowski

(b)

(c)

(d)

(e)

Fig. 13.1 (From [214]) Illustration of the importance of the degree of dimension reduction
and filtering in artificially generated data, using FastICA and a gradient algorithm for ML
estimation. (a) Original positively kurtotic signals. (b) ICA decomposition in which the
preprocessing includes a dimension reduction to the first 3 principal components. (c) Poor,i.e., too weak dimension reduction. (d) Decomposition using an intermediate dimension
reduction (50 components retained). (e) Same results as in (d) but using low-pass filteredmixtures

CHOICE OF ALGORITHM

271

First, since dimension reduction by PCA is often necessary, one must choose
the number of principal components to be retained. This is a classic problem;
see Chapter 6. It is usually solved by choosing the minimum number of principal
components that explain the data well enough, containing, for example, 90% of thevariance. Often, the dimension is actually chosen by trial and error with no theoreticalguidelines.
Second, for computational reasons we may prefer to estimate only a smaller
number of ICs than the dimension of the data (after PCA preprocessing). This is thecase when the dimension of the data is very large, and we do not want to reduce thedimension by PCA too much, since PCA always contains the risk of not including theICs in the reduced data. Using FastICA and other algorithms that allow estimation ofa smaller number of components, we can thus perform a kind of dimension reduction
by ICA. In fact, this is an idea somewhat similar to projection pursuit. Here, it iseven more difficult to give any guidelines as to how many components should be
estimated. Trial and error may be the only method applicable.
Information-theoretic, Bayesian, and other criteria for determining the number of
ICs are discussed in more detail in [231, 81, 385].

13.4

CHOICE OF ALGORITHM

Now we shall briefly discuss the choice of ICA algorithm from a practical viewpoint.
As will be discussed in detail in Chapter 14, most estimation principles and objectivefunctions for ICA are equivalent, at least in theory. So, the main choice is reduced toa couple of points:

 One choice is between estimating all the independent components in parallel,or just estimating a few of them (possibly one-by-one). This corresponds tochoosing between symmetric and hierarchical decorrelation. In most cases,
symmetric decorrelation is recommended. Deflation is mainly useful in thecase where we want to estimate only a very limited number of ICs, and otherspecial cases. The disadvantage with deflationary orthogonalization is that theestimation errors in the components that are estimated first accumulate andincrease the errors in the later components.

 One must also choose the nonlinearity used in the algorithms. It seems that therobust, nonpolynomial nonlinearities are to be preferred in most applications.
The simplest thing to do is to just use the tanh function as the nonlinearity
g. This is sufficient when using FastICA. (When using gradient algorithms,
especially in the ML framework, a second function needs to be used as well;
see Chapter 9.)

 Finally, there is the choice between on-line and batch algorithms. In mostcases, the whole data set is available before the estimation, which is called
in different contexts batch, block, or off-line estimation. This is the casewhere FastICA can be used, and it is the algorithm that we recommend. On-

272

PRACTICAL CONSIDERATIONS

line or adaptive algorithms are needed in signal-processing applications where
the mixing matrix may change on-line, and fast tracking is needed. In theon-line case, the recommended algorithms are those obtained by stochastic
gradient methods. It should also be noted that in some cases, the FastICAalgorithm may not converge well as Newton-type algorithms sometimes exhibit
oscillatory behavior. This problem can be alleviated by using gradient methods,
or combinations of the two (see [197]).

13.5

CONCLUDING REMARKS AND REFERENCES

In this chapter, we considered some practical problems in ICA. When dealing withtime signals, low-pass filtering of the data is useful to reduce noise. On the other
hand, high-pass filtering, or computing innovation processes is useful to increase theindependence and nongaussianity of the components. One of these, or their combination may be very useful in practice. Another very useful thing to do is to reduce thedimension of the data by PCA. This reduces noise and prevents overlearning. It may
also solve the problems with data that has a smaller number of ICs than mixtures.
Problems
13.1 Take a Fourier transform on every observed signal xi (t). Does the ICA model
still hold, and in what way?
13.2

Prove the theorem on innovations.

Computer assignments
13.1 Take a gaussian white noise sequence. Low-pass filter it by a low-pass filter
with coefficients (...,0,0,1,1,1,1,1,0,0,0,...). What does the signal look like?
13.2
like?

High-pass filter the gaussian white noise sequence. What does the signal look13.3 Generate 100 samples of 100 independent components. Run FastICA on thisdata without any mixing. What do the estimated ICs look like? Is the estimate of themixing matrix close to identity?

14
Overview and Comparisonof Basic ICA Methods
In the preceding chapters, we introduced several different estimation principles andalgorithms for independent component analysis (ICA). In this chapter, we provide
an overview of these methods. First, we show that all these estimation principles
are intimately connected, and the main choices are between cumulant-based vs.
negentropy/likelihood-based estimation methods, and between one-unit vs. multiunit methods. In other words, one must choose the nonlinearity and the decorrelation
method. We discuss the choice of the nonlinearity from the viewpoint of statisticaltheory. In practice, one must also choose the optimization method. We compare thealgorithms experimentally, and show that the main choice here is between on-line(adaptive) gradient algorithms vs. fast batch fixed-point algorithms.
At the end of this chapter, we provide a short summary of the whole of Part II,
that is, of basic ICA estimation.

14.1

OBJECTIVE FUNCTIONS VS. ALGORITHMS

A distinction that has been used throughout this book is between the formulation of
the objective function, and the algorithm used to optimize it. One might express thisin the following “equation”:
ICA method = objective function + optimization algorithm:
In the case of explicitly formulated objective functions, one can use any of theclassic optimization methods, for example, (stochastic) gradient methods and Newton
273

274

OVERVIEW AND COMPARISON OF BASIC ICA METHODS

methods. In some cases, however, the algorithm and the estimation principle may bedifficult to separate.
The properties of the ICA method depend on both of the objective function andthe optimization algorithm. In particular: the statistical properties (e.g., consistency, asymptotic variance, robustness) of
the ICA method depend on the choice of the objective function,

 the algorithmic properties (e.g., convergence speed, memory requirements,
numerical stability) depend on the optimization algorithm.

Ideally, these two classes of properties are independent in the sense that different
optimization methods can be used to optimize a single objective function, and asingle optimization method can be used to optimize different objective functions. In
this section, we shall first treat the choice of the objective function, and then consider
optimization of the objective function.

14.2

CONNECTIONS BETWEEN ICA ESTIMATION PRINCIPLES

Earlier, we introduced several different statistical criteria for estimation of the ICA
model, including mutual information, likelihood, nongaussianity measures, cumulants, and nonlinear principal component analysis (PCA) criteria. Each of thesecriteria gave an objective function whose optimization enables ICA estimation. We
have already seen that some of them are closely connected; the purpose of this section
is to recapitulate these results. In fact, almost all of these estimation principles can beconsidered as different versions of the same general criterion. After this, we discuss
the differences between the principles.
14.2.1

Similarities between estimation principles

Mutual information gives a convenient starting point for showing the similarity between different estimation principles. We have for an invertible linear transformation
= :

y Bx

I (y1 ; y2 ; :::; yn ) =Xi

H (yi ) ? H (x) ? log j det Bj(14.1)

If we constrain the yi to be uncorrelated and of unit variance, the last term on the
anyway (seeright-hand side is constant; the second term does not depend on
Chapter 10). Recall that entropy is maximized by a gaussian distribution, when
variance is kept constant (Section 5.3). Thus we see that minimization of mutual
information means maximizing the sum of the nongaussianities of the estimatedcomponents. If these entropies (or the corresponding negentropies) are approximatedby the approximations used in Chapter 8, we obtain the same algorithms as in thatchapter.

BCONNECTIONS BETWEEN ICA ESTIMATION PRINCIPLES

275

Alternatively, we could approximate mutual information by approximating thedensities of the estimated ICs by some parametric family, and using the obtained
log-density approximations in the definition of entropy. Thus we obtain a methodthat is essentially equivalent to maximum likelihood (ML) estimation.
The connections to other estimation principles can easily be seen using likelihood.
First of all, to see the connection to nonlinear decorrelation, it is enough to compare
the natural gradient methods for ML estimation shown in (9.17) with the nonlinear
decorrelation algorithm (12.11): they are of the same form. Thus, ML estimation
gives a principled method for choosing the nonlinearities in nonlinear decorrelation.
The nonlinearities used are determined as certain functions of the probability density
functions (pdf’s) of the independent components. Mutual information does the samething, of course, due to the equivalency discussed earlier. Likewise, the nonlinear PCA methods were shown to be essentially equivalent to ML estimation (and,therefore, most other methods) in Section 12.7.
The connection of the preceding principles to cumulant-based criteria can be seen
by considering the approximation of negentropy by cumulants as in Eq. (5.35):

1 E fy3 g2 + 1 kurt(y)2J (y)  1248

(14.2)

where the first term could be omitted, leaving just the term containing kurtosis.
Likewise, cumulants could be used to approximate mutual information, since mutual
information is based on entropy. More explicitly, we could consider the following
approximation of mutual information:

I (y)  c1 ? c2Xikurt(yi )2(14.3)

where c1 and c2 are some constants. This shows clearly the connection between
cumulants and minimization of mutual information. Moreover, the tensorial methods
in Chapter 11 were seen to lead to the same fixed-point algorithm as the maximizationof nongaussianity as measured by kurtosis, which shows that they are doing very muchthe same thing as the other kurtosis-based methods.
14.2.2Differences between estimation principles

There are, however, a couple of differences between the estimation principles as well.
1. Some principles (especially maximum nongaussianity) are able to estimate
single independent components, whereas others need to estimate all the components at the same time.
2. Some objective functions use nonpolynomial functions based on the (assumed)
probability density functions of the independent components, whereas others
use polynomial functions related to cumulants. This leads to different nonquadratic functions in the objective functions.
3. In many estimation principles, the estimates of the ICs are constrained to be
uncorrelated. This reduces somewhat the space in which the estimation is

276

OVERVIEW AND COMPARISON OF BASIC ICA METHODS

performed. Considering, for example, mutual information, there is no reason
why mutual information would be exactly minimized by a decomposition thatgives uncorrelated components. Thus, this decorrelation constraint slightly
reduces the theoretical performance of the estimation methods. In practice,
this may be negligible.
4. One important difference in practice is that often in ML estimation,the densitiesof the ICs are fixed in advance, using prior knowledge on the independent
components. This is possible because the pdf’s of the ICs need not be knownwith any great precision: in fact, it is enough to estimate whether they are subor supergaussian. Nevertheless, if the prior information on the nature of the
independent components is not correct, ML estimation will give completely
wrong results, as was shown in Chapter 9. Some care must be taken with ML
estimation, therefore. In contrast, using approximations of negentropy, thisproblem does not usually arise, since the approximations we have used in thisbook do not depend on reasonable approximations of the densities. Therefore,
these approximations are less problematic to use.

14.3

STATISTICALLY OPTIMAL NONLINEARITIESThus, from a statistical viewpoint, the choice of estimation method is more or lessreduced to the choice of the nonquadratic function G that gives information on thehigher-order statistics in the form of the expectation E fG( Ti )g. In the algorithms,
this choice corresponds to the choice of the nonlinearity g that is the derivative of G.
In this section, we analyze the statistical properties of different nonlinearities. Thisis based on the family of approximations of negentropy given in (8.25). This family
includes kurtosis as well. For simplicity, we consider here the estimation of just one
IC, given by maximizing this nongaussianity measure. This is essentially equivalent
to the problem

bx

max

E f(bT x)2 g=1

E fG(bT x)g(14.4)

bx
bb

where the sign of G depends of the estimate on the sub- or supergaussianity of T .
The obtained vector is denoted by . The two fundamental statistical properties ofthat we analyze are asymptotic variance and robustness.

bb

14.3.1Comparison of asymptotic variance *x

In practice, one usually has only a finite sample of T observations of the vector .
Therefore, the expectations in the theoretical definition of the objective function are infact replaced by sample averages. This results in certain errors in the estimator , and
it is desired to make these errors as small as possible. A classic measure of thisperror
T as
is asymptotic (co)variance, which means the limit of the covariance matrix of
T ! 1. This gives an approximation of the mean-square error of , as was already

bb

bb

bb

277

STATISTICALLY OPTIMAL NONLINEARITIES

discussed in Chapter 4. Comparison of, say, the traces of the asymptotic variances of
two estimators enables direct comparison of the accuracy of two estimators. One cansolve analytically for the asymptotic variance of , obtaining the following theorem[193]:

bb

Theorem 14.1 The trace of the asymptotic variance of
estimation of the independent component si , equals

bb as defined above for the

VG = C (A) E f(gE f(ssi )gg(s?)(?E fgs0(isg()sgi))2g) ;
2

2

i

i

i

(14.5)A) is a constant that depends only on A.

where g is the derivative of G, and C (The theorem is proven at the appendix of this chapter.
Thus the comparison of the asymptotic variances of two estimators for two different
nonquadratic functions G boils down to a comparison of the VG . In particular, one
can use variational calculus to find a G that minimizes VG . Thus one obtains thefollowing theorem [193]:
Theorem 14.2 The trace of the asymptotic variance ofthe formbb is minimized when G is of

Gopt (y) = c1 log pi (y) + c2 y2 + c3
(14.6)
where pi is the density function of si , and c1 ; c2 ; c3 are arbitrary constants.
For simplicity, one can choose Gopt (y ) = log pi (y ). Thus, we see that the optimal
nonlinearity is in fact the one used in the definition of negentropy. This shows thatnegentropy is the optimal measure of nongaussianity, at least inside those measuresthat lead to estimators of the form considered here. 1 Also, one sees that the optimalfunction is the same as the one obtained for several units by the maximum likelihood
approach.
14.3.2Comparison of robustness *Another very desirable property of an estimator is robustness against outliers. Thismeans that single, highly erroneous observations do not have much influence on theestimator. In this section, we shall treat the question: How does the robustness ofthe estimator ^ depend on the choice of the function G? The main result is that thefunction G(y ) should not grow fast as a function of jy j if we want robust estimators.
In particular, this means that kurtosis gives nonrobust estimators, which may be very
disadvantagous in some situations.

b

1 One has to take into account, however, that in the definition of negentropy, the nonquadratic function is

not fixed in advance, whereas in our nongaussianity measures, G is fixed. Thus, the statistical propertiesof negentropy can be only approximatively derived from our analysis.

278

OVERVIEW AND COMPARISON OF BASIC ICA METHODSFirst, note that the robustness of ^ depends also on the method of estimation usedin constraining the variance of ^ T to equal unity, or, equivalently, the whitening
method. This is a problem independent of the choice of G. In the following, we
assume that this constraint is implemented in a robust way. In particular, we assume
that the data is sphered (whitened) in a robust manner, in which case the constraint
reduces to k ^ k = 1, where is the value of for whitened data. Several robust
estimators of the variance of ^ T or of the covariance matrix of are presented in
the literature; see reference [163].
The robustness of the estimator ^ can be analyzed using the theory of Mestimators. Without going into technical details, the definition of an M-estimatorcan be formulated as follows: an estimator is called an M-estimator if it is defined as
the solution ^ for  of

b
bx

w

w
wz

b

x

w

E f (z; )g = 0z(14.7)

where is a random vector and is some function defining the estimator. Now, thepoint is that the estimator ^ is an M-estimator. To see this, define  = ( ; ), where is the Lagrangian multiplier associated with the constraint. Using the Lagrange
conditions, the estimator ^ can then be formulated as the solution of Eq. (14.7) whereis defined as follows (for sphered data):w
w

w(z; ) = zg(wT z) + cw 
kwk ? 1(14.8)

2

where c = (Ez fG( ^ T )g ? E fG( )g)?1 is an irrelevant constant.
The analysis of robustness of an M-estimator is based on the concept of an
influence function, IF ( ; ^). Intuitively speaking, the influence function measures
the influence of single observations on the estimator. It would be desirable to havean influence function that is bounded as a function of , as this implies that eventhe influence of a far-away outlier is “bounded”, and cannot change the estimate
too much. This requirement leads to one definition of robustness, which is called
B-robustness. An estimator is called B-robust, if its influence function is bounded
as a function of , i.e., supz kIF ( ; ^)k is finite for every ^. Even if the influencefunction is not bounded, it should grow as slowly as possible when k k grows, toreduce the distorting effect of outliers.
It can be shown that the influence function of an M-estimator equals

w z
z

z

z

z

z

IF (z; ^) = B (z; ^)B(14.9)zwhere is an irrelevant invertible matrix that does not depend on . On the other
hand, using our definition of , and denoting by = T =k k the cosine of theangle between and , one obtains easily

w z z

z w
k (z; (w; ))k = C 1 h (wT z) + C h(wT z) + C
2

1

2

2

2

z

3

(14.10)where C1 ; C2 ; C3 are constants that do not depend on , and h(y ) = yg (y ). Thus we
see that the robustness of ^ essentially depends on the behavior of the function h(u).

w

STATISTICALLY OPTIMAL NONLINEARITIES279

The slower h(u) grows, the more robust the estimator. However, the estimator reallycannot be B-robust, because the in the denominator prevents the influence function
from being bounded for all . In particular, outliers that are almost orthogonal to ^ ,and have large norms, may still have a large influence on the estimator. These results
are stated in the following theorem:

z

w

z

Theorem 14.3 Assume that the data is whitened (sphered) in a robust manner.
Then the influence function of the estimator ^ is never bounded for all . However,if h(y ) = yg (y ) is bounded, the influence function is bounded in sets of the formf j ^ T =k k > g for every  > 0, where g is the derivative of G.

wzz wz zIn particular, if one chooses a function G(y ) that is bounded, h is also bounded,and ^ is quite robust against outliers. If this is not possible, one should at least choose
a function G(y ) that does not grow very fast when jy j grows. If, in contrast, G(y )grows very fast when jy j grows, the estimates depend mostly on a few observations far
from the origin. This leads to highly nonrobust estimators, which can be completely
ruined by just a couple of bad outliers. This is the case, for example, when kurtosis
is used, which is equivalent to using ^ with G(y ) = y 4 .

w

w

14.3.3Practical choice of nonlinearity

It is useful to analyze the implications of the preceding theoretical results by considering the following family of density functions:p (s) = C1 exp(C2 jsj )(14.11)is a positive constant, and C1 ; C2 are normalization constants that ensure

wherethat p is a probability density of unit variance. For different values of alpha, thedensities in this family exhibit different shapes. For 0 < < 2, one obtains a sparse,
supergaussian density (i.e., a density of positive kurtosis). For = 2, one obtains thegaussian distribution, and for > 2, a subgaussian density (i.e., a density of negative
kurtosis). Thus the densities in this family can be used as examples of different
nongaussian densities.
Using Theorem 14.1, one sees that in terms of asymptotic variance, the optimal
nonquadratic function is of the form:

Gopt (y) = jyj(14.12)where the arbitrary constants have been dropped for simplicity. This implies roughlythat for supergaussian (resp. subgaussian) densities, the optimal function is a functionthat grows slower than quadratically (resp. faster than quadratically). Next, recall
from Section 14.3.2 that if G(y ) grows fast with jy j, the estimator becomes highly
nonrobust against outliers. Also taking into account the fact that most ICs encounteredin practice are supergaussian, one reaches the conclusion that as a general-purpose
function, one should choose a function G that resembles rather

Gopt (u) = jyj ; where < 2(14.13)

280

OVERVIEW AND COMPARISON OF BASIC ICA METHODS

The problem with such functions is, however, that they are not differentiable at 0 for 1. This can lead to problems in the numerical optimization. Thus it is better
to use approximating differentiable functions that have the same kind of qualitative
behavior. Considering = 1, in which case one has a Laplacian density, one could
use instead the function G1 (y ) = log cosh a1 y where a1 is a constant. This is very
similar to the so-called Huber function that is widely used in robust statistics as
a robust alternative of the square function. Note that the derivative of G1 is thenthe familiar tanh function (for a1 = 1). We have found 1  a1  2 to provide
a good approximation. Note that there is a trade-off between the precision of theapproximation and the smoothness of the resulting objective function.
< 1, i.e., highly supergaussian ICs, one could approximate
In the case of
the behavior of Gopt for large u using a gaussian function (with a minus sign):
G2 (y) = ? exp(?y2 =2). The derivative of this function is like a sigmoid for smallvalues, but goes to 0 for larger values. Note that this function also fulfills thecondition in Theorem 14.3, thus providing an estimator that is as robust as possible
in this framework.
Thus, we reach the following general conclusions: A good general-purpose function is G(y) = log cosh a y, where 1  a  2
1

1

is a constant.

 When the ICs are highly supergaussian, or when robustness is very important,G(y) = ? exp(?y =2) may be better.
 Using kurtosis is well justified only if the ICs are subgaussian and there are no2

outliers.

In fact, these two nonpolynomial functions are those that we used in the nongaussianity measures in Chapter 8 as well, and illustrated in Fig. 8.20. The functions in
Chapter 9 are also essentially the same, since addition of a linear function does nothave much influence on the estimator. Thus, the analysis of this section justifies theuse of the nonpolynomial functions that we used previously, and shows why caution
should be taken when using kurtosis.
In this section, we have used purely statistical criteria for choosing the function G.
One important criterion for comparing ICA methods that is completely independentof statistical considerations is the computational load. Since most of the objectivefunctions are computationally very similar, the computational load is essentially afunction of the optimization algorithm. The choice of the optimization algorithm
will be considered in the next section.

14.4

EXPERIMENTAL COMPARISON OF ICA ALGORITHMS

The theoretical analysis of the preceding section gives some guidelines as to which
nonlinearity (corresponding to a nonquadratic function G) should be chosen. In
this section, we compare the ICA algorithms experimentally. Thus we are able toEXPERIMENTAL COMPARISON OF ICA ALGORITHMS

281

analyze the computational efficiency of the different algorithms as well. This is doneby experiments, since a satisfactory theoretical analysis of convergence speed does
not seem possible. We saw previously, though, that FastICA has quadratic or cubic
convergence whereas gradient methods have only linear convergence, but this result issomewhat theoretical because it does not say anything about the global convergence.
In the same experiments, we validate experimentally the earlier analysis of statistical
performance in terms of asymptotic variance.
14.4.1

Experimental set-up and algorithms

Experimental setup In the following experimental comparisons, artificial data
generated from known sources was used. This is quite necessary, because only then
are the correct results known and a reliable comparison possible. The experimental
setup was the same for each algorithm in order to make the comparison as fair as
possible. We have also compared various ICA algorithms using real-world data in[147], where experiments with artificial data also are described in somewhat moredetail. At the end of this section, conclusions from experiments with real-world data
are presented.
The algorithms were compared along the two sets of criteria, statistical and computational, as was outlined in Section 14.1. The computational load was measured
as flops (basic floating-point operations, such as additions or divisions) needed forconvergence. The statistical performance, or accuracy, was measured using a performance index, defined as

E1 =

m
m X
m
m X
X
X
( maxjpijjjp j ? 1) + ( maxjpijjpj j ? 1)
k ik
k kj
i=1 j =1

j =1 i=1(14.14)

P BA

where pij is the ij th element of the m  m matrix =. If the ICs have been
separated perfectly, becomes a permutation matrix (where the elements may havedifferent signs, though). A permutation matrix is defined so that on each of its rows
and columns, only one of the elements is equal to unity while all the other elements
are zero. Clearly, the index (14.14) attains its minimum value zero for an ideal
permutation matrix. The larger the value E1 is, the poorer the statistical performanceof a separation algorithm. In certain experiments, another fairly similarly behaving
performance index, E2 , was used. It differs slightly from E1 in that squared values
p2ij are used instead of the absolute ones in (14.14).

P

ICA algorithms used The following algorithms were included in the comparison(their abbreviations are in parentheses): The FastICA fixed-point algorithm. This has three variations: using kurtosis

with deflation (FP) or with symmetric orthogonalization (FPsym), and using
the tanh nonlinearity with symmetric orthogonalization (FPsymth).

 Gradient algorithms for maximum likelihood estimation, using a fixed nonlinearity given by ? tanh. First, we have the ordinary gradient ascent algorithm,

282

OVERVIEW AND COMPARISON OF BASIC ICA METHODS

or the Bell-Sejnowski algorithm (BS). Second, we have the natural gradientalgorithm proposed by Amari, Cichocki and Yang [12], which is abbreviatedas ACY.

 Natural gradient MLE using an adaptive nonlinearity. (Abbreviated as ExtBS,

since this is called the “extended Bell-Sejnowski” algorithm by some authors.)
The nonlinearity was adapted using the sign of kurtosis as in reference [149],
which is essentially equivalent to the density parameterization we used in
Section 9.1.2.

 The EASI algorithm for nonlinear decorrelation, as discussed in Section 12.5.
Again, the nonlinearity used was tanh.

 The recursive least-squares algorithm for a nonlinear PCA criterion (NPCA-

RLS), discussed in Section 12.8.3. In this algorithm, the plain tanh function
could not be used for stability reasons, but a slightly modified nonlinearity waschosen: y ? tanh(y ).

Tensorial algorithms were excluded from this comparison due to the problems ofscalability discussed in Chapter 11. Some tensorial algorithms have been comparedrather thoroughly in [315]. However, the conclusions are of limited value, because
the data used in [315] always consisted of the same three subgaussian ICs.
14.4.2

Results for simulated data

Statistical performance and computational load The basic experimentmeasures the computational load and statistical performance (accuracy) of the tested
algorithms. We performed experiments with 10 independent components that werechosen supergaussian, because for this source type all the algorithms in the comparison worked, including ML estimation with a fixed ? tanh nonlinearity. The
mixing matrix used in our simulations consisted of uniformly distributed random
numbers. For achieving statistical reliability, the experiment was repeated over 100
different realizations of the input data. For each of the 100 realizations, the accuracy
was measured using the error index E1 . The computational load was measured infloating point operations needed for convergence.
Fig. 14.1 shows a schematic diagram of the computational load vs. the statistical
performance. The boxes typically contain 80% of the 100 trials, thus representing
standard outcomes.
As for statistical performance, Fig. 14.1 shows that best results are obtained byusing a tanh nonlinearity (with the right sign). This was to be expected according
to the theoretical analysis of Section 14.3. tanh is a good nonlinearity especiallyfor supergaussian ICs as in this experiment. The kurtosis-based FastICA is clearly
inferior, especially in the deflationary version. Note that the statistical performance
only depends on the nonlinearity, and not on the optimization method, as explainedin Section 14.1. All the algorithms using tanh have pretty much the same statistical
performance. Note also that no outliers were added to the data, so the robustness of
the algorithms is not measured here.

AEXPERIMENTAL COMPARISON OF ICA ALGORITHMS

283

10

9

8

Error index E1

7

FP
6

5

FPsym
EASI

4

NPCA−RLS
3

2
6
10

BS
ExtBS
ACY

FPsymth

7

8

10

10

9

10

Flops

Fig. 14.1 Computational requirements in flops versus the statistical error index E1 .
(Reprinted from [147], reprint permission and copyright by World Scientific, Singapore.)

Looking at the computational load, one sees clearly that FastICA requires thesmallest amount of computation. Of the on-line algorithms, NPCA-RLS converges
fastest, probably due to its roughly optimal determination of learning parameters. For
the other on-line algorithms, the learning parameter was a constant, determined bymaking some preliminary experiments so that a value providing good convergence
was found. These ordinary gradient-type algorithms have a computational load thatis about 20–50 times larger than for FastICA.
To conclude, the best results from a statistical viewpoint are obtained when using
the tanh nonlinearity with any algorithm. (Some algorithms, especially the tensorial
ones, cannot use the tanh nonlinearity, but these were excluded from this comparison
for reasons discussed earlier.) As for the computational load, the experiments showthat the FastICA algorithm is much faster than the gradient algorithms.
Convergence speed of on-line algorithms Next, we studied the convergence
speeds of the on-line algorithms. Fixed-point algorithms do not appear in this comparison, because they are of a different type and a direct comparison is not possible.

284

OVERVIEW AND COMPARISON OF BASIC ICA METHODS

90

ACY
BS
NPCA−RLS
EASI

80

70

Error index E1

60

50

40

30

20

10

0

0

0.5

1

1.5
Flops

2

2.5

3
8

x 10

Fig. 14.2 Convergence speed of on-line ICA algorithms as a function of required floatingpoint operations for 10 supergaussian ICs. (Reprinted from [147], reprint permission andcopyright by World Scientific, Singapore.)

The results (shown in Fig. 14.2) are averages of 10 trials for 10 supergaussian ICs(for which all the algorithms worked without on-line estimation of kurtosis). The
main observation is that the recursive least-squares version of the nonlinear PCA algorithm (NPCA-RLS) is clearly the fastest converging of the on-line algorithms. The
difference between NPCA-RLS and the other algorithms could probably be reduced
by using simulated annealing or other more sophisticated technique for determining
the learning parameters.
For subgaussian ICs, the results were qualitatively similar to those in Fig. 14.2,
except that sometimes the EASI algorithm may converge even faster than NPCA-RLS.
However, sometimes its convergence speed was the poorest among the comparedalgorithms. Generally, a weakness of on-line algorithms using stochastic gradientsis that they are fairly sensitive to the choice of the learning parameters.

EXPERIMENTAL COMPARISON OF ICA ALGORITHMS

285

0.9

FP
FPsymthNPCA−RLS
ACY
EASI
ExtBS

0.8

0.7

2

Error index E1/2

0.6

0.5

0.4

0.3

0.2

0.1

0

0

2

4

6
8
Number of sources

10

12

14

Fig. 14.3 Error as a function of the number of ICs. (Reprinted from [147], reprint permission
and copyright by World Scientific, Singapore.)

Error for increasing number of components We also made a short investigation on how the statistical performances of the algorithms change with increasing
number of components. In Fig. 14.3, the error (square root of the error index E2 ) isplotted as the function of the number of supergaussian ICs. The results are medianvalues over 50 different realizations of the input data. For more than five ICs, thenumber of data samples was increased so that it was proportional to the square of
the number of ICs. The natural gradient ML algorithm (ACY), and its version withadaptive nonlinearity (ExtBS), achieve some of the best accuracies, behaving very
similarly. The basic fixed-point algorithm (FP) using a cubic nonlinearity has thepoorest accuracy, but its error increases only slightly after seven ICs. On the other
hand, the version of the fixed-point algorithm which uses symmetric orthogonalization and tanh nonlinearity (FPsymth) performs as well as the natural gradient ML
algorithm. Again, we see that it is the nonlinearity that is the most important indetermining statistical performance. For an unknown reason, the errors of the EASI
and NPCA-RLS algorithms have a peak around 5–6 ICs. For a larger number ofICs, the accuracy of the NPCA-RLS algorithm is close to the best algorithms, while

286

OVERVIEW AND COMPARISON OF BASIC ICA METHODS

the error of EASI increases linearly with the number of independent components.
However, the error of all the algorithms is tolerable for most practical purposes.
Effect of noise In [147], the effect of additive gaussian noise on the performanceof ICA algorithms has been studied, too. The first conclusion is that the estimation
accuracy degrades fairly smoothly until the noise power increases up to ?20 dB ofthe signal power. If the amount of noise is increased even more, it may happen thatthe studied ICA algorithms are not able to separate all the sources. In practice, noise
smears the separated ICs or sources, making the separation results almost useless if
there is a lot of noise present.
Another observation is that once there is even a little noise present in the data,
the error strongly depends on the condition number of the mixing matrix . The
condition number of a matrix [320, 169] describes how close to singularity it is.

A

14.4.3Comparisons with real-world data

We have compared in [147] the preceding ICA algorithms using three different
real-world data sets. The applications were projection pursuit for well-known crab
and satellite data sets, and finding interesting source signals from the biomedicalmagnetoencephalographic data (see Chapter 22). For the real-world data, the true
independent components are unknown, and the assumptions made in the standard
ICA model may not hold, or hold only approximately. Hence it is only possible tocompare the performances of the ICA algorithms with each other, in the application
at hand.
The following general conclusions can be made from these experiments [147]:
1. ICA is a robust technique. Even though the assumption of statistical independence is not strictly fulfilled, the algorithms converge towards a clear set ofcomponents (MEG data), or a subspace of components whose dimension ismuch smaller than the dimension of the problem (satellite data). This is a goodcharacteristic encouraging the use of ICA as a general data analysis tool.
2. The FastICA algorithm and the natural gradient ML algorithm with adaptive
nonlinearity (ExtBS) yielded usually similar results with real-world data. This
is not surprising, because there exists a close theoretical connection between
these algorithms, as discussed in Chapter 9. Another pair of similarly behaving
algorithms consisted of the EASI algorithm and the nonlinear PCA algorithm
using recursive least-squares (NPCA-RLS).
3. In difficult real-world problems, it is useful to apply several different ICA
algorithms, because they may reveal different ICs from the data. For the MEG
data, none of the compared algorithms was best in separating all types of source
signals.

REFERENCES

14.5

287

REFERENCES

The fundamental connection between cumulants and negentropy and mutual information was introduced in [89]. A similar approximation of likelihood by cumulants
was introduced in [140]. Approximation of negentropy by cumulants was originallyconsidered in [222]. The connection between infomax and likelihood was shownin [363, 64], and the connection between mutual information and likelihood has
been explicitly discussed in [69]. The interpretation of nonlinear PCA criteria as
maximum likelihood estimation was presented in [236]. The connections between
different methods were discussed in such review papers as [201, 65, 269].
The theoretical analysis of the performance of the estimators is taken from [193].
See [69] for more information, especially on the effect of the decorrelation constraint
on the estimator. On robustness and influence functions, see such classic texts as[163, 188].
More details on the experimental comparison can be found in [147].

14.6

SUMMARY OF BASIC ICA

Now we summarize Part II. This part treated the estimation of the basic ICA model,i.e., the simplified model with no noise or time-structure, and a square mixing
matrix. The observed data = (x1 ; :::; xn )T is modeled as a linear transformationof components = (s1 ; :::; sn )T that are statistically independent:sxx = As

(14.15)This is a rather well-understood problem for which several approaches have been
proposed. What distinguished ICA from PCA and classic factor analysis is that thenongaussian structure of the data is taken into account. This higher-order statisticalinformation (i.e., information not contained in the mean and the covariance matrix)
can be utilized, and therefore, the independent components can be actually separated,which is not possible by PCA and classic factor analysis.
Often, the data is preprocessed by whitening (sphering), which exhausts the second
order information that is contained in the covariance matrix, and makes it easier to
use the higher-order information:

VA

z = Vx = (VA)s(14.16)The linear transformationin the model is then reduced to an orthogonal one,so that =i.e., a rotation. Thus, we are searching for an orthogonal matrix
should be good estimates of the independent components.
Several approaches can then be taken to utilize the higher-order information. A
principled, yet intuitive approach is given by finding linear combinations of maximum
nongaussianity, as motivated by the central limit theorem. Sums of nongaussian
random variables tend to be closer to gaussian that the original ones. Therefore if
we take a linear combination y = i wi zi of the observed (whitened) variables,

W

P

y Wz

288

OVERVIEW AND COMPARISON OF BASIC ICA METHODS

this will be maximally nongaussian if it equals one of the independent components.
Nongaussianity can be measured by kurtosis or by (approximations of) negentropy.
This principle shows the very close connection between ICA and projection pursuit,in which the most nongaussian projections are considered as the interesting ones.
Classic estimation theory directly gives another method for ICA estimation: maximum likelihood estimation. An information-theoretic alternative is to minimize themutual information of the components. All these principles are essentially equivalent or at least closely related. The principle of maximum nongaussianity has the
additional advantage of showing how to estimate the independent components oneby-one. This is possible by a deflationary orthogonalization of the estimates of theindividual independent components.
With every estimation method, we are optimizing functions of expectations ofnonquadratic functions, which is necessary to gain access to higher-order information. Nonquadratic functions usually cannot be maximized simply by solving theequations: Sophisticated numerical algorithms are necessary.
The choice of the ICA algorithm is basically a choice between on-line and batchmode algorithms. In the on-line case, the algorithms are obtained by stochastic
gradient methods. If all the independent components are estimated in parallel, themost popular algorithm in this category is natural gradient ascent of likelihood. The
fundamental equation in this method is

W W + [I + g(y)yT ]W(14.17)g

where the component-wise nonlinear function is determined from the log-densitiesof the independent components; see Table 9.1 for details.
In the more usual case, where the computations are made in batch-mode (offline), much more efficient algorithms are available. The FastICA algorithm is a very
efficient batch algorithm that can be derived either from a fixed-point iteration or as
an approximate Newton method. The fundamental iteration in FastICA is, for one
row of :w W

w E fzg(wT z)g ? E fg0(wT z)gw

(14.18)w

where the nonlinearity g can be almost any smooth function, andshould be
normalized to unit norm at every iteration. FastICA can be used to estimate thecomponents either one-by-one by finding maximally nongaussian directions (seeTables 8.3), or in parallel by maximizing nongaussianity or likelihood (see Table 8.4
or Table 9.2).
In practice, before application of these algorithms, suitable preprocessing is often
necessary (Chapter 13). In addition to the compulsory centering and whitening, it isoften advisable to perform principal component analysis to reduce the dimension of
the data, or some time filtering by taking moving averages.

APPENDIX

289

Appendix Proofs
Here we prove Theorem 14.1. Making the change of variable
the optimal solutions ^ becomesqXtstg(^qT st) = Xtq = AT b, the equation defining

stsTt q^(A.1)

where t = 1; :::; T is the sample index, T is the sample size, and  is a Lagrangian multiplier..
Without loss of generality, let us assume that ^ is near the ideal solution = (1; 0; 0; :::). Note
2that due to the constraint E ( T )2 == 1, the variance of the first component of ^ ,denoted by ^1 , is of a smaller order than the variance of the vector of other components, denotedby ^ ?1 . Excluding the first component in (A.1), and making the first-order approximationg(^ T ) = g(s1) + g0(s1 )^ T?1 ?1 , where also ?1 denotes without its first component, one
obtains after some simple manipulationsqq

f b x g kqk

q

q
qs

q s

p1

X

T t

s

s? [g(s ) ? s ] = T1
1

1

1

X
t

q

s

s? [?sT? g0(s ) + sT? ]^q? pT
1

1

1

1

1

(A.2)

where the sample index t has been dropped for simplicity. Making the first-order approximation
 = E s1 g(s1) , one can write (A.2) in the form u = v ^ ?1 T where v converges to theidentity matrix multiplied by E s1 g (s1 )E g0 (s1 ) , and u converges to a variable thathas a normal distribution of zero mean whose covariance matrix equals the identity matrix
2
multiplied by E g 2 (s1 )(E s1 g (s1 ) ) . This implies the theorem, since ^ ?1 = ?1 ^ ,Twhere ?1 is the inverse ofwithout its first row.

f

B

g

f

f

g? f

A

g? f

g

g

q p

q

B b

Part III

EXTENSIONS AND
RELATED METHODS

15
Noisy ICAIn real life, there is always some kind of noise present in the observations. Noisecan correspond to actual physical noise in the measuring devices, or to inaccuraciesof the model used. Therefore, it has been proposed that the independent component
analysis (ICA) model should include a noise term as well. In this section, we consider
different methods for estimating the ICA model when noise is present.
However, estimation of the mixing matrix seems to be quite difficult when noiseis present. It could be argued that in practice, a better approach could often be to
reduce noise in the data before performing ICA. For example, simple filtering oftime-signals is often very useful in this respect, and so is dimension reduction byprincipal component analysis (PCA); see Sections 13.1.2 and 13.2.2.
In noisy ICA, we also encounter a new problem: estimation of the noise-free
realizations of the independent components (ICs). The noisy model is not invertible,
and therefore estimation of the noise-free components requires new methods. This
problem leads to some interesting forms of denoising.

15.1

DEFINITION

Here we extend the basic ICA model to the situation where noise is present. The
noise is assumed to be additive. This is a rather realistic assumption, standard infactor analysis and signal processing, and allows for a simple formulation of the noisy
model. Thus, the noisy ICA model can be expressed as

x = As + n(15.1)293

294

NOISY ICAnwhere = (n1 ; :::; nn ) is the noise vector. Some further assumptions on the noise
are usually made. In particular, it is assumed that
1. The noise is independent from the independent components.
2. The noise is gaussian.

I

The covariance matrix of the noise, say , is often assumed to of the form  2 , but
this may be too restrictive in some cases. In any case, the noise covariance is assumed
to be known. Little work on estimation of an unknown noise covariance has been
conducted; see [310, 215, 19].
The identifiability of the mixing matrix in the noisy ICA model is guaranteed
under the same restrictions that are sufficient in the basic case, 1 basically meaning
independence and nongaussianity. In contrast, the realizations of the independent
components si can no longer be identified, because they cannot be completely separated from noise.

15.2

SENSOR NOISE VS. SOURCE NOISEI

In the typical case where the noise covariance is assumed to be of the form  2 , thenoise in Eq. (15.1) could be considered as “sensor” noise. This is because the noise
variables are separately added on each sensor, i.e., observed variable xi . This is incontrast to “source” noise, in which the noise is added to the independent components(sources). Source noise can be modeled with an equation slightly different from thepreceding, given by

x = A(s + n)

(15.2)

where again the covariance of the noise is diagonal. In fact, we could consider the
noisy independent components, given by s~i = si + ni , and rewrite the model as

x = A~s(15.3)We see that this is just the basic ICA model, with modified independent components.
What is important is that the assumptions of the basic ICA model are still valid: thecomponents of ~ are nongaussian and independent. Thus we can estimate the modelin (15.3) by any method for basic ICA. This gives us a perfectly suitable estimator
for the noisy ICA model. This way we can estimate the mixing matrix and the noisy
independent components. The estimation of the original independent components
from the noisy ones is an additional problem, though; see below.
This idea is, in fact, more general. Assume that the noise covariance has the forms

 = AAT 

2

1 This seems to be admitted by the vast majority of ICA researchers.

proofs of this property, though.

(15.4)We are not aware of any rigorous

FEW NOISE SOURCES

295

Then the noise vector can be transformed into another one ~ = ?1 , which can becalled equivalent source noise. Then the equation (15.1) becomesn A n

x = As + An~ = A(s + n~ )(15.5)~ is  I, and thus the transformed components in
The point is that the covariance of n
s + n~ are independent. Thus, we see again that the mixing matrix A can be estimated2

by basic ICA methods.
To recapitulate: if the noise is added to the independent components and not to theobserved mixtures, or has a particular covariance structure, the mixing matrix can be
estimated by ordinary ICA methods. The denoising of the independent components
is another problem, though; it will be treated in Section 15.5 below.

15.3

FEW NOISE SOURCES

Another special case that reduces to the basic ICA model can be found, when thenumber of noise components and independent components is not very large. In
particular, if their total number is not larger than the number of mixtures, we again
have an ordinary ICA model, in which some of the components are gaussian noise andothers are the real independent components. Such a model could still be estimatedby the basic ICA model, using one-unit algorithms with less units than the dimensionof the data.
In other words, we could define the vector of the independent components as~ = (s1 ; :::; sk ; n1 ; :::; nl )T where the si ; i = 1; :::; k are the “real” independent
components and the ni ; i = 1; :::; l are the noise variables. Assume that the numberof mixtures equals k + l, that is the number of real ICs plus the number of noise
variables. In this case, the ordinary ICA model holds with = ~, whereisa matrix that incorporates the mixing of the real ICs and the covariance structureof the noise, and the number of the independent components in ~ is equal to thenumber of observed mixtures. Therefore, finding the k most nongaussian directions,
we can estimate the real independent components. We cannot estimate the remaining
dummy independent components that are actually noise variables, but we did notwant to estimate them in the first place.
The applicability of this idea is quite limited, though, since in most cases we wantto assume that the noise is added on each mixture, in which case k + l, the numberof real ICs plus the number of noise variables, is necessarily larger than the numberof mixtures, and the basic ICA model does not hold for ~.

s

x As
s

A

s

15.4

ESTIMATION OF THE MIXING MATRIX

Not many methods for noisy ICA estimation exist in the general case. The estimationof the noiseless model seems to be a challenging task in itself, and thus the noise isusually neglected in order to obtain tractable and simple results. Moreover, it may

296

NOISY ICA

be unrealistic in many cases to assume that the data could be divided into signals andnoise in any meaningful way.
Here we treat first the problem of estimating the mixing matrix. Estimation of the
independent components will be treated below.
15.4.1

Bias removal techniques

Perhaps the most promising approach to noisy ICA is given by bias removal techniques. This means that ordinary (noise-free) ICA methods are modified so that thebias due to noise is removed, or at least reduced.
Let us denote the noise-free data in the following by

v = As(15.6)

w v

We can now use the basic idea of finding projections, say T , in which nongaussianity, is locally maximized for whitened data, with constraint k k = 1. As shownin Chapter 8, projections in such directions give consistent estimates of the independent components, if the measure of nongaussianity is well chosen. This approach
could be used for noisy ICA as well, if only we had measures of nongaussianity
which are immune to gaussian noise, or at least, whose values for the original data
can be easily estimated from noisy observations. We have T = T + T ,and thus the point is to measure the nongaussianity of T from the observed T
so that the measure is not affected by the noise T .

w n

w v

w

w x wv w n
wx

Bias removal for kurtosis If the measure of nongaussianity is kurtosis (thefourth-order cumulant), it is almost trivial to construct one-unit methods for noisy
ICA, because kurtosis is immune to gaussian noise. This is because the kurtosis of
T equals the kurtosis of T , as can be easily proven by the basic properties ofkurtosis.
It must be noted, however, that in the preliminary whitening, the effect of noisemust be taken into account; this is quite simple if the noise covariance matrix is
known. Denoting by= E f T g the covariance matrix of the observed noisy
data, the ordinary whitening should be replaced by the operation

w x

w v

C

xx

x~ = (C ? )? = x(15.7)
In other words, the covariance matrix C ?  of the noise-free data should be used inwhitening instead of the covariance matrix C of the noisy data. In the following, we~call this operation “quasiwhitening”. After this operation, the quasiwhitened data x
1 2

follows a noisy ICA model as well:x~ = Bs + n~Bn

(15.8)

where is orthogonal, and ~ is a linear transform of the original noise in (15.1).
Thus, the theorem in Chapter 8 is valid for ~ , and finding local maxima of the absolutevalue of kurtosis is a valid method for estimating the independent components.

x

ESTIMATION OF THE MIXING MATRIX

297

Bias removal for general nongaussianity measures As was argued in
Chapter 8, it is important in many applications to use measures of nongaussianitythat have better statistical properties than kurtosis. We introduced the following
measure:JG (wT v) = [E fG(wT v)g ? E fG( )g]2
(15.9)
where the function G is a sufficiently regular nonquadratic function, and  is astandardized gaussian variable.
Such a measure could be used for noisy data as well, if only we were able toestimate JG ( T ) of the noise-free data from the noisy observations . Denoting
by z a nongaussian random variable, and by n a gaussian noise variable of variance2 , we should be able to express the relation between E fG(z )g and E fG(z + n)gin simple algebraic terms. In general, this relation seems quite complicated, and canbe computed only using numerical integration.
However, it was shown in [199] that for certain choices of G, a similar relation
becomes very simple. The basic idea is to choose G to be the density function ofa zero-mean gaussian random variable, or a related function. These nonpolynomial
moments are called gaussian moments.
Denote by

w v

x

2
'c (x) = 1c '( xc ) = p 1 exp(? 2xc2 )(15.10)2c(k)
the gaussian density function with variance c2 , and by 'c (x) the k th (k > 0)(?k)
derivative of 'c (x). Denote further by 'cintegral function of 'c (x),R x (?k+1) ()d, wherethewekthdefine
(?k)
'(0)
obtained by 'c (x) = 0 'cc (x) = 'c (x). (Thelower integration limit 0 is here quite arbitrary, but has to be fixed.) Then we havethe following theorem [199]:

Theorem 15.1 Let z be any nongaussian random variable, and n an independent
gaussian noise variable of variance  2 . Define the gaussian function ' as in (15.10).
Then for any constant c >  2 , we have

E f'c (z )g = E f'd (z + n)g(15.11)pwith d = c2 ?  2 . Moreover, (15.11) still holds when ' is replaced by '(k) for any
integer index k .
The theorem means that we can estimate the independent components from noisy
observations by maximizing a general contrast function of the form (15.9), where
the direct estimation of the statistics E fG( T )g of the noise-free data is made(k)(k)
possible by using G(u) = 'c (u). We call the statistics of the form E f'c ( T )g
the gaussian moments of the data. Thus, for quasiwhitened data ~ , we maximize thefollowing contrast function:

wv

x

max [E f'(dk(w) ) (wT x~ )g ? E f'(ck) ( )g]2

kwk=1

w v

(15.12)

298

NOISY ICAp

with d( ) = c2 ? T ~ . This gives a consistent (i.e., convergent) method of
estimating the noisy ICA model, as was shown in Chapter 8.
To use these results in practice, we need to choose some values for k . In fact, cdisappears from the final algorithm, so value for this parameter need not be chosen.
Two indices k for the gaussian moments seem to be of particular interest: k = 0 andk = ?2. The first corresponds to the gaussian density function; its use was proposed
in Chapter 8. The case k = ?2 is interesting because the contrast function is thenof the form of a (negative) log-density of a supergaussian variable. In fact, '(?2) (u)can be very accurately approximated by G(u) = 1=2 log cosh u, which was also usedin Chapter 8.

w

w w

FastICA for noisy data Using the unbiased measures of nongaussianity given inthis section, we can derive a variant of the FastICA algorithm [198]. Using kurtosis
or gaussian moments give algorithms of a similar form, just like in the noise-freecase.
The algorithm takes the form [199, 198]:w = E fx~g(wT x~)g ? (I + ~ )wE fg0(wT x~)g
(15.13)where w , the new value of w, is normalized to unit norm after every iteration, and~ is given by(15.14)
~ = E fn~n~ T g = (C ? )? = (C ? )? =
1 2

1 2

The function g is here the derivative of G, and can thus be chosen among the following:

g1 (u) = tanh(u); g2 (u) = u exp(?u2 =2); g3 (u) = u3

(15.15)where g1 is an approximation of '(?1) , which is the gaussian cumulative distributionfunction (these relations hold up to some irrelevant constants). These functions coveressentially the nonlinearities ordinarily used in the FastICA algorithm.
15.4.2Higher-order cumulant methods

A different approach to estimation of the mixing matrix is given by methods using
higher-order cumulants only. Higher-order cumulants are unaffected by gaussian
noise (see Section 2.7), and therefore any such estimation method would be immune
to gaussian noise. Such methods can be found in [63, 263, 471]. The problem is,however, that such methods often use cumulants of order 6. Higher-order cumulants
are sensitive to outliers, and therefore methods using cumulants of orders higher
than 4 are unlikely to be very useful in practice. A nice feature of this approach is,
however, that we do not need to know the noise covariance matrix.
Note that the cumulant-based methods in Part II used both second- and fourthorder cumulants. Second-order cumulants are not immune to gaussian noise, andtherefore the cumulant-based method introduced in the previous chapters would not

ESTIMATION OF THE NOISE-FREE INDEPENDENT COMPONENTS

299

be immune either. Most of the cumulant-based methods could probably be modified
to work in the noisy case, as we did in this chapter for methods maximizing theabsolute value of kurtosis.
15.4.3

Maximum likelihood methods

Another approach for estimation of the mixing matrix with noisy data is given by
maximum likelihood (ML) estimation. First, one could maximize the joint likelihoodof the mixing matrix and the realizations of the independent components, as in[335, 195, 80]. This is given bylog L(A; s(1); :::; s(T )) =

?

T "1
X
t=1 2

kAs(t) ? x(t)k?1 +
2

n
X
i=1

#

fi (si (t)) + C(15.16)

where k k2?1 is defined as T ?1 , the (t) are the realizations of the independent components, and C is an irrelevant constant. The fi are the logarithms of theprobability density functions (pdf’s) of the independent components. Maximizationof this joint likelihood is, however, computationally very expensive.
A more principled method would be to maximize the (marginal) likelihood of
the mixing matrix, and possibly that of the noise covariance, which was done in[310]. This was based on the idea of approximating the densities of the independent
components as gaussian mixture densities; the application of the EM algorithm
then becomes feasible. In [42], the simpler case of discrete-valued independent
components was treated. A problem with the EM algorithm is, however, that thecomputational complexity grows exponentially with the dimension of the data.
A more promising approach might be to use bias removal techniques so as tomodify existing ML algorithms to be consistent with noisy data. Actually, the bias
removal techniques given here can be interpreted as such methods; a related method
was given in [119].
Finally, let us mention a method based on the geometric interpretation of the
maximum likelihood estimator, introduced in [33], and a rather different approachfor narrow-band sources, introduced in [76].

m

15.5

15.5.1m m

s

ESTIMATION OF THE NOISE-FREE INDEPENDENTCOMPONENTS
Maximum a posteriori estimation

In noisy ICA, it is not enough to estimate the mixing matrix. Inverting the mixing
matrix in (15.1), we obtain

Wx = s + Wn

(15.17)

300

NOISY ICA

In other words, we only get noisy estimates of the independent components. Therefore, we would like to obtain estimates of the original independent components s^ithat are somehow optimal, i.e., contain minimum noise.
A simple approach to this problem would be to use the maximum a posteriori(MAP) estimates. See Section 4.6.3 for the definition. Basically, this means that wetake the values that have maximum probability, given the . Equivalently, we takeas s^i those values that maximize the joint likelihood in (15.16), so this could also becalled a maximum likelihood (ML) estimator.
To compute the MAP estimator, let us take the gradient of the log-likelihood(15.16) with respect to the (t); t = 1; :::; T and equate this to 0. Thus we obtain theequationxsA^ T ? A^ ^s(t) ? A^ T ? x(t) + f 0(^s(t)) = 0
1

1(15.18)where the derivative of the log-density, denoted by f 0 , is applied separately on eachcomponent of the vector ^(t).
In fact, this method gives a nonlinear generalization of classic Wiener filtering presented in Section 4.6.2. An alternative approach would be to use the time-structureof the ICs (see Chapter 18) for denoising. This results in a method resembling the
Kalman filter; see [250, 249].

s

15.5.2

Special case of shrinkage estimations

Solving for the ^ is not easy, however. In general, we must use numerical optimization.
A simple special case is obtained if the noise covariance is assumed to be of the same
form as in (15.4) [200, 207]. This corresponds to the case of (equivalent) source
noise. Then (15.18) gives^s = g(A^ ?1 x)(15.19)where the scalar component-wise function g is obtained by inverting the relationg?1(u) = u + 2 f 0 (u)(15.20)Thus, the MAP estimator is obtained by inverting a certain function involving f 0 , orthe score function [395] of the density of s. For nongaussian variables, the scorefunction is nonlinear, and so is g .
In general, the inversion required in (15.20) may be impossible analytically. Here
we show three examples, which will be shown to have great practical value in
Chapter 21, where the inversion can be done easily.
Example 15.1 Assume that s has ap
Laplacianp (or doublepexponential) distribution ofunit variance. Then p(s) = exp(? 2jsj)= 2, f 0 (s) = 2 sign(s), and g takes theformpg(u) = sign(u) max(0; juj ? 22 )(15.21)

301

ESTIMATION OF THE NOISE-FREE INDEPENDENT COMPONENTS(Rigorously speaking, the function in (15.20) is not invertible in this case, but approximating it by a sequence of invertible functions, (15.21) is obtained as the limit.)
The function in (15.21) is a shrinkage function that reduces the absolute value of its
argument by a fixed amount, as depicted in Fig 15.1. Intuitively, the utility of such afunction can be seen as follows. Since the density of a supergaussian random variable(e.g., a Laplacian random variable) has a sharp peak at zero, it can be assumed that
small values of the noisy variable correspond to pure noise, i.e., to s = 0. Thresholding such values to zero should thus reduce noise, and the shrinkage function canindeed be considered a soft thresholding operator.
Example 15.2 More generally, assume that the score function is approximated as alinear combination of the score functions of the gaussian and the Laplacian distributions:f 0 (s) = as + b sign(s)(15.22)

with a; b > 0. This corresponds to assuming the following density model for s:p(s) = C exp(?as2=2 ? bjsj)(15.23)where C is an irrelevant scaling constant. This is depicted in Fig. 15.2. Then we
obtaing(u) = 1 +12 a sign(u) max(0; juj ? b2 )(15.24)This function is a shrinkage with additional scaling, as depicted in Fig 15.1.
Example 15.3 Yet another possibility is to use the following strongly supergaussian
probability density:=2+1)p(s) = 21d (p + 2) [ ( + 1)=2] ( +3)(15.25)[ ( + 1)=2 + js=dj]
with parameters ; d > 0, see Fig. 15.2. When ! 1, the Laplacian density is(

obtained as the limit. The strong sparsity of the densities given by this model can beseen e.g., from the fact that the kurtosis [131, 210] of these densities is always larger
than the kurtosis of the Laplacian density, and reaches infinity for  2. Similarly,p(0) reaches infinity as goes to zero. The resulting shrinkage function given by(15.20) can be obtained after some straightforward algebraic manipulations as:pg(u) = sign(u) max(0; juj ?2 ad + 21 (juj + ad)2 ? 42 ( + 3) )p

(15.26)( + 1)=2, and g(u) is set to zero in case the square root in (15.26)
where a =is imaginary. This is a shrinkage function that has a stronger thresholding flavor, as
depicted in Fig. 15.1.

302

NOISY ICA

Fig. 15.1 Plots of the shrinkage functions. The effect of the functions is to reduce theabsolute value of its argument by a certain amount which depends on the noise level. Smallarguments are set to zero. This reduces gaussian noise for sparse random variables. Solid line:
shrinkage corresponding to Laplacian density as in (15.21). Dashed line: typical shrinkage
function obtained from (15.24). Dash-dotted line: typical shrinkage function obtained from
(15.26). For comparison, the line x = y is given by dotted line. All the densities werenormalized to unit variance, and noise variance was fixed to :3.

1.5

1

0.5

0
−2.5

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

Fig. 15.2 Plots of densities corresponding to models (15.23) and (15.25) of the sparse
components. Solid line: Laplacian density. Dashed line: a typical moderately supergaussian
density given by (15.23). Dash-dotted line: a typical strongly supergaussian density given by(15.25). For comparison, gaussian density is given by dotted line.

303

DENOISING BY SPARSE CODE SHRINKAGE

15.6

DENOISING BY SPARSE CODE SHRINKAGE

Although the basic purpose of noisy ICA estimation is to estimate the ICs, the model
can be used to develop an interesting denoising method as well.
Assume that we observe a noisy version,

x =v+n(15.27)xof the data , which has previously been modeled by ICA

v = As(15.28)
To denoise x, we can compute estimates ^s of the independent components by theabove MAP estimation procedure. Then we can reconstruct the data asv^ = A^s(15.29)The point is that if the mixing matrix is orthogonal and the noise covariance is of
the form  2 , the condition in (15.4) is fulfilled. This condition of the noise is acommon one. Thus we could approximate the mixing matrix by an orthogonal one,for example the one obtained by orthogonalization of the mixing matrix as in (8.48).
This method is called sparse code shrinkage [200, 207], since it means that wetransform the data into a sparse, i.e., supergaussian code, and then apply shrinkage
on that code. To summarize, the method is as follows.

Iv

1. First, using a noise-free training set of , estimate ICA and orthogonalize theT . Estimate amixing matrix. Denote the orthogonal mixing matrix bydensity model pi (si ) for each sparse component, using the models in (15.23)
and (15.25).

Wx

2. Compute for each noisy observation (t) the corresponding noisy sparse com(t). Apply the shrinkage nonlinearity gi (:) as defined inponents (t) =(15.24), or in (15.26), on each component ui (t), for every observation index t.
Denote the obtained components by s^i (t) = gi (ui (t)).

u

Wx

v

3. Invert the transform to obtain estimates of the noise-free data, given by ^ (t) =T ^(t).

Ws

For experiments using sparse code shrinkage on image denoising, see Chapter 21.
In that case, the method is closely related to wavelet shrinkage and “coring” methods[116, 403].

304

NOISY ICA

15.7

CONCLUDING REMARKSIn this chapter, we treated the estimation of the ICA model when additive sensor
noise is present. First of all, it was shown that in some cases, the mixing matrix canbe estimated with basic ICA methods without any further complications. In cases
where this is not possible, we discussed bias removal techniques for estimation of
the mixing matrix, and introduced a bias-free version of the FastICA algorithm.
Next, we considered how to estimate the noise-free independent components, i.e.,
how to denoise the initial estimates of the independent components. In the case of
supergaussian data, it was shown that this led to so-called shrinkage estimation. In
fact, we found an interesting denoising procedure called sparse code shrinkage.
Note that in contrast to Part II where we considered the estimation of the basic
ICA model, the material in this chapter is somewhat speculative in character. The
utility of many of the methods in this chapter has not been demonstrated in practice.
We would like to warn the reader not to use the noisy ICA methods lightheartedly:
It is always advisable to first attempt to denoise the data so that basic ICA methods
can be used, as discussed in Chapter 13.

16
ICA with Overcomplete
Bases
A difficult problem in independent component analysis (ICA) is encountered if thenumber of mixtures xi is smaller than the number of independent components si. Thismeans that the mixing system is not invertible: We cannot obtain the independent
components (ICs) by simply inverting the mixing matrix . Therefore, even if
we knew the mixing matrix exactly, we could not recover the exact values of the
independent components. This is because information is lost in the mixing process.
This situation is often called ICA with overcomplete bases. This is because wehave in the ICA modelA

x = As =

X
i

ai si

(16.1)awhere the number of “basis vectors”, i , is larger than the dimension of the space of: thus this basis is “too large”, or overcomplete. Such a situation sometimes occurs
in feature extraction of images, for example.
As with noisy ICA, we actually have two different problems. First, how to estimate
the mixing matrix, and second, how to estimate the realizations of the independent
components. This is in stark contrast to ordinary ICA, where these two problems are
solved at the same time. This problem is similar to the noisy ICA in another respect
as well: It is much more difficult than the basic ICA problem, and the estimation
methods are less developed.

x

305

306

ICA WITH OVERCOMPLETE BASES

16.1

ESTIMATION OF THE INDEPENDENT COMPONENTS

16.1.1

Maximum likelihood estimation

Many methods for estimating the mixing matrix use as subroutines methods that
estimate the independent components for a known mixing matrix. Therefore, weshall first treat methods for reconstructing the independent components, assumingthat we know the mixing matrix. Let us denote by m the number of mixtures and by
n the number of independent components. Thus, the mixing matrix has size m  nwith n > m, and therefore it is not invertible.
The simplest method of estimating the independent components would be to use
the pseudoinverse of the mixing matrix. This yields^s = AT (AAT )?1 x(16.2)

In some situations, such a simple pseudoinverse gives a satisfactory solution, but inmany cases we need a more sophisticated estimate.
A more sophisticated estimator of can be obtained by maximum likelihood(ML) estimation [337, 275, 195], in a manner similar to the derivation of the ML or
maximum a posteriori (MAP) estimator of the noise-free independent components in
Chapter 15. We can write the posterior probability of as follows:sp(sjx; A) = 1x=AsYispi (si )(16.3)

x As

where 1x=As is an indicator function that is 1 if =and 0 otherwise. The (prior)probability densities of the independent components are given by pi (si ). Thus, we
obtain the maximum likelihood estimator of as^s = arg xmax
=As

s

X
i

log pi (si )(16.4)

Alternatively, we could assume that there is noise present as well. In this case, we
get a likelihood that is formally the same as with ordinary noisy mixtures in (15.16).
The only difference is in the number of components in the formula.
The problem with the maximum likelihood estimator is that it is not easy tocompute. This optimization cannot be expressed as a simple function in analytic
form in any interesting case. It can be obtained in closed form if the si havegaussian distribution: In this case the optimum is given by the pseudoinverse in (16.2).
However, since ICA with gaussian variables is of little interest, the pseudoinverse is
not a very satisfactory solution in many cases.
In general, therefore, the estimator given by (16.4) can only be obtained bynumerical optimization. A gradient ascent method can be easily derived. One
case where the optimization is easier than usual is when the si have a Laplacian
distribution:ppi (si ) = p1 exp( 2jsi j)2(16.5)

307

ESTIMATION OF THE MIXING MATRIX

Ignoring uninteresting constants, we have^s = arg xmax
=As

X
i

jsi j

(16.6)

which can be formulated as a linear program and solved by classic methods for linear
programming [275].
16.1.2The case of supergaussian components

Using a supergaussian distribution, such as the Laplacian distribution, is well justified
in feature extraction, where the components are supergaussian. Using the Laplaciandensity also leads to an interesting phenomenon: The ML estimator gives coefficientss^i of which only m are nonzero. Thus, only the minimum number of the components
are activated. Thus we obtain a sparse decomposition in the sense that the components
are quite often equal to zero.
It may seem at first glance that it is useless to try to estimate the ICs by ML
estimation, because they cannot be estimated exactly in any case. This is not so,
however; due to this phenomenon of sparsity, the ML estimation is very useful. In
the case where the independent components are very supergaussian, most of themare very close to zero because of the large peak of the pdf at zero. (This is related tothe principle of sparse coding that will be treated in more detail in Section 21.2.)
Thus, those components that are not zero may not be very many, and the system
may be invertible for those components. If we first determine which components arelikely to be clearly nonzero, and then invert that part of the linear system, we may
be able to get quite accurate reconstructions of the ICs. This is done implicitly in
the ML estimation method. For example, assume that there are three speech signals
mixed into two mixtures. Since speech signals are practically zero most of the time
(which is reflected in their strong supergaussianity), we could assume that only twoof the signals are nonzero at the same time, and successfully reconstruct those two
signals [272].

16.2
16.2.1ESTIMATION OF THE MIXING MATRIX
Maximizing joint likelihood

To estimate the mixing matrix, one can use maximum likelihood estimation. In
the simplest case of ML estimation, we formulate the joint likelihood of and therealization of the si , and maximize it with respect to all these variables. It is slightly
simpler to use a noisy version of the joint likelihood. This is of the same form as theone in Eq. (15.16):

A

log L(A; s(1); :::; s(T )) = ?

T " 1
X

n
X

#

kAs(t) ? x(t)k + fi (si (t)) + C
2
t=1 2
i=1(16.7)2

308

ICA WITH OVERCOMPLETE BASESs

where  2 is the noise variance, here assumed to be infinitely small, the (t) are therealizations of the independent components, and C is an irrelevant constant. The
functions fi are the log-densities of the independent components.
Maximization of (16.7) with respect toand si could be accomplished by aglobal gradient descent with respect to all the variables [337]. Another approachto maximization of the likelihood is to use an alternating variables technique [195],
in which we first compute the ML estimates of the si (t) for a fixed
and then,using this new , we compute the ML estimates of the si (t), and so on. The ML
estimate of the si (t) for a given is given by the methods of the preceding section,
considering the noise to be infinitely small. The ML estimate of for given si (t)can be computed as

A

A

A

A

X

A

A = ( x(t)x(t)T )?
t

1

X
t

x(t)s(t)T(16.8)

This algorithm needs some extra stabilization, however. For example, normalizing
the estimates of the si to unit norm is necessary. Further stabilization can be obtainedby first whitening the data. Then we have (considering infinitely small noise)

E fxxT g = AAT = I(16.9)A

which means that the rows of form an orthonormal system. This orthonormality
could be enforced after every step of (16.8), for further stabilization.
16.2.2

Maximizing likelihood approximations

Maximization of the joint likelihood is a rather crude method of estimation. Froma Bayesian viewpoint, what we really want to maximize is the marginal posterior
probability of the mixing matrix. (For basic concepts of Bayesian estimation, see
Section 4.6.)
A more sophisticated form of maximum likelihood estimation can be obtainedby using a Laplace approximation of the posterior distribution of . This improves
the stability of the algorithm, and has been successfully used for estimation of
overcomplete bases from image data [274], as well as for separation of audio signals[272]. For details on the Laplace approximation, see [275]. An alternative for the
Laplace approximation is provided by ensemble learning; see Section 17.5.1.
A promising direction of research is given by Monte Carlo methods. These area class of methods often used in Bayesian estimation, and are based on numericalintegration using stochastic algorithms. One method in this class, Gibbs sampling,has been used in [338] for overcomplete basis estimation. Monte Carlo methods
typically give estimators with good statistical properties; the drawback is that they
are computationally very demanding.
Also, one could use an expectation-maximization (EM) algorithm [310, 19]. Using
gaussian mixtures as models for the distributions of the independent components, thealgorithm can be derived in analytical form. The problem is, however, that its
complexity grows exponentially with the dimension of , and thus it can only be used

A

s

309

ESTIMATION OF THE MIXING MATRIX

in small dimensions. Suitable approximations of the algorithm might alleviate thislimitation [19].
A very different approximation of the likelihood method was derived in [195],
in which a form of competitive neural learning was used to estimate overcomplete
bases with supergaussian data. This is a computationally powerful approximationthat seems to work for certain data sets. The idea is that the extreme case ofsparsity or supergaussianity is encountered when at most one of the ICs is nonzero
at any one time. Thus we could simply assume that only one of the components isnonzero for a given data point, for example, the one with the highest value in thepseudoinverse reconstruction. This is not a realistic assumption in itself, but it may
give an interesting approximation of the real situation in some cases.
16.2.3

Approximate estimation using quasiorthogonality

The maximum likelihood methods discussed in the preceding sections give a welljustified approach to ICA estimation with overcomplete bases. The problem withmost of the methods in the preceding section is that they are computationally quite
expensive. A typical application of ICA with overcomplete bases is, however, feature
extraction. In feature extraction, we usually have spaces of very high dimensions.
Therefore, we show here a method [203] that is more heuristically justified, but has
the advantage of being not more expensive computationally than methods for basic
ICA estimation. This method is based on the FastICA algorithm, combined with theconcept of quasiorthogonality.
Sparse approximately uncorrelated decompositions Our heuristic approach is justified by the fact that in feature extraction for many kinds of naturaldata, the ICA model is only a rather coarse approximation. In particular, the numberof potential “independent components” seems to be infinite: The set of such components is closer to a continuous manifold than a discrete set. One evidence forthis is that classic ICA estimation methods give different basis vectors when started
with different initial values, and the number of components thus produced does notseem to be limited. Any classic ICA estimation method gives a rather arbitrary collection of components which are somewhat independent, and have sparse marginal
distributions.
We can also assume, for simplicity, that the data is prewhitened as a preprocessing
step, as in most ICA method in Part II. Then the independent components are simply
given by the dot-products of the whitened data vector with the basis vectors i .
Due to the preceding considerations, we assume in our approach that what is
usually needed, is a collection of basis vectors that has the following two properties:

z

a

az

1. The dot-products Ti of the observed data with the basis vectors have sparse(supergaussian) marginal distributions.

az

2. The Ti should be approximately uncorrelated (“quasiuncorrelated”). Equivalently, the vectors i should be approximately orthogonal (“quasiorthogonal”).

a

310

ICA WITH OVERCOMPLETE BASES

A decomposition with these two properties seems to capture the essential propertiesof the decomposition obtained by estimation of the ICA model. Such decompositions
could be called sparse approximately uncorrelated decompositions.
It is clear that it is possible to find highly overcomplete basis sets that have thefirst property of these two. Classic ICA estimation is usually based on maximizing
the sparseness (or, in general, nongaussianity) of the dot-products, so the existenceof several different classic ICA decompositions for a given image data set shows theexistence of decompositions with the first property.
What is not obvious, however, is that it is possible to find strongly overcomplete
decompositions such that the dot-products are approximately uncorrelated. The main
point here is that this is possible because of the phenomenon of quasiorthogonality.
Quasiorthogonality in high-dimensional spaces Quasiorthogonality [247]
is a somewhat counterintuitive phenomenon encountered in very high-dimensional
spaces. In a certain sense, there is much more room for vectors in high-dimensional
spaces. The point is that in an n-dimensional space, where n is large, it is possible
to have (say) 2n vectors that are practically orthogonal, i.e., their angles are close to
90 degrees. In fact, when n grows, the angles can be made arbitrarily close to 90
degrees. This must be contrasted with small-dimensional spaces: If, for example,n = 2, even the maximally separated 2n = 4 vectors exhibit angles of 45 degrees.
For example, in image decomposition, we are usually dealing with spaces whose
dimensions are of the order of 100. Therefore, we can easily find decompositions of,say, 400 basis vectors, such that the vectors are quite orthogonal, with practically all
the angles between basis vectors staying above 80 degrees.
FastICA with quasiorthogonalization To obtain a quasiuncorrelated sparsedecomposition as defined above, we need two things. First, a method for finding
vectors i that have maximally sparse dot-products, and second, a method of quasiorthogonalization of such vectors. Actually, most classic ICA algorithms can beconsidered as maximizing the nongaussianity of the dot-products with the basis vectors, provided that the data is prewhitened. (This was shown in Chapter 8.) Thus themain problem here is constructing a proper method for quasidecorrelation.
We have developed two methods for quasidecorrelation: one of them is symmetric and the other one is deflationary. This dichotomy is the same as in ordinary
decorrelation methods used in ICA. As above, it is here assumed that the data iswhitened.
A simple way of achieving quasiorthogonalization is to modify the ordinary
deflation scheme based on a Gram-Schmidt-like orthogonalization. This meansthat we estimate the basis vectors one by one. When we have estimated p basis
vectors 1 ; :::; p , we run the one-unit fixed-point algorithm for p+1 , and after
every iteration step subtract from p+1 a certain proportion of the ’projections’T j j ; j = 1; :::; p of the previously estimated p vectors, and then renormalize
p+1

a

a
a aa

a

a

a

ESTIMATION OF THE MIXING MATRIX

ap :
+1

a
a

1. p+1
2. p+1

ap ? Ppj aTp aj aj
ap =kap k
+1
+1

=1

+1

+1

311

(16.10)where is a constant determining the force of quasiorthogonalization. If = 1, we
have ordinary, perfect orthogonalization. We have found in our experiments that anin the range [0:1 ::: 0:3] is sufficient in spaces where the dimension is 64.
In certain applications it may be desirable to use a symmetric version of quasiorthogonalization, in which no vectors are “privileged” over others [210, 197]. Thiscan be accomplished, for example, by the following algorithm:

A

A

AA A

3
? 12 T :1.
2
2. Normalize each column of

A to unit norm

(16.11)

which is closely related to the iterative symmetric orthogonalization method used forbasic ICA in Section 8.4.3. The present algorithm is simply doing one iteration of theiterative algorithm. In some cases, it may be necessary to do two or more iterations,although in the experiments below, just one iteration was sufficient.
Thus, the algorithm that we propose is similar to the FastICA algorithm as described, e.g. in Section 8.3.5 in all other respects than the orthogonalization, whichis replaced by one of the preceding quasiorthogonalization methods.
Experiments with overcomplete image bases We applied our algorithm on
image windows (patches) of 8  8 pixels taken from natural images. Thus, we used
ICA for feature extraction as explained in detail in Chapter 21.
The mean of the image window (DC component) was removed as a preprocessing step, so the dimension of the data was 63. Both deflationary and symmetricquasiorthogonalization were used. The nonlinearity used in the FastICA algorithm
was the hyperbolic tangent. Fig. 16.1 shows an estimated approximately 4 times
overcomplete basis (with 240 components). The sample size was 14000. The results
shown here were obtained using the symmetric approach; the deflationary approachyielded similar results, with the parameter fixed at 0:1.
The results show that the estimated basis vectors are qualitatively quite similar
to those obtained by other, computationally more expensive methods [274]; they
are also similar to those obtained by basic ICA (see Chapter 21). Moreover, bycomputing the dot-products between different basis vectors, we see that the basis is,indeed, quasiorthogonal. This validates our heuristic approach.
16.2.4

Other approaches

We mention here some other algorithms for estimation of overcomplete bases. First,
in [341], independent components with binary values were considered, and a geometrically motivated method was proposed. Second, a tensorial algorithm for theovercomplete estimation problem was proposed in [63]. Related theoretical results
were derived in [58]. Third, a natural gradient approach was developed in [5]. Fur-312

ICA WITH OVERCOMPLETE BASES

Fig. 16.1 The basis vectors of a 4 times overcomplete basis. The dimension of the data is
63 (excluding the DC component, i.e., the local mean) and the number of basis vectors is 240.
The results are shown in the original space, i.e., the inverse of the preprocessing (whitening)
was performed. The symmetric approach was used. The basis vectors are very similar toGabor functions or wavelets, as is typical with image data (see Chapter 21).

CONCLUDING REMARKS

313

ther developments on estimation of overcomplete bases using methods similar to thepreceding quasiorthogonalization algorithm can be found in [208].

16.3

CONCLUDING REMARKSThe ICA problem becomes much more complicated if there are more independent
components than observed mixtures. Basic ICA methods cannot be used as such. In
most practical applications, it may be more useful to use the basic ICA model as an
approximation of the overcomplete basis model, because the estimation of the basicmodel can be performed with reliable and efficient algorithms.
When the basis is overcomplete, the formulation of the likelihood is difficult,since the problem belongs to the class of missing data problems. Methods basedon maximum likelihood estimation are therefore computationally rather inefficient.
To obtain computationally efficient algorithms, strong approximations are necessary.
For example, one can use a modification of the FastICA algorithm that is based on
finding a quasidecorrelating sparse decomposition. This algorithm is computationallyvery efficient, reducing the complexity of overcomplete basis estimation to that ofclassic ICA estimation.

17
Nonlinear ICA

This chapter deals with independent component analysis (ICA) for nonlinear mixing
models. A fundamental difficulty in the nonlinear ICA problem is that it is highly
nonunique without some extra constraints, which are often realized by using a suitable
regularization. We also address the nonlinear blind source separation (BSS) problem.
Contrary to the linear case, we consider it different from the respective nonlinear ICA
problem. After considering these matters, some methods introduced for solving thenonlinear ICA or BSS problems are discussed in more detail. Special emphasis isgiven to a Bayesian approach that applies ensemble learning to a flexible multilayer
perceptron model for finding the sources and nonlinear mixing mapping that havemost probably given rise to the observed mixed data. The efficiency of this method isdemonstrated using both artificial and real-world data. At the end of the chapter, othertechniques proposed for solving the nonlinear ICA and BSS problems are reviewed.

17.1
17.1.1NONLINEAR ICA AND BSS
The nonlinear ICA and BSS problems

In many situations, the basic linear ICA or BSS modelx = As =

n
X
j =1

sj aj

(17.1)xis too simple for describing the observed data adequately. Hence, it is natural toconsider extension of the linear model to nonlinear mixing models. For instantaneous
315

316

NONLINEAR ICA

mixtures, the nonlinear mixing model has the general form

x = f (s)x(17.2)fwhere is the observed m-dimensional data (mixture) vector, is an unknown realvalued m-component mixing function, and is an n-vector whose elements are then unknown independent components.
Assume now for simplicity that the number of independent components n equals
the number of mixtures m. The general nonlinear ICA problem then consists offinding a mapping : Rn ! Rn that gives componentsshy = h(x)(17.3)

that are statistically independent. A fundamental characteristic of the nonlinear
ICA problem is that in the general case, solutions always exist, and they are highly
nonunique. One reason for this is that if x and y are two independent randomvariables, any of their functions f (x) and g (y ) are also independent. An even more
serious problem is that in the nonlinear case, x and y can be mixed and still statisticallyindependent, as will be shown below. This is not unlike in the case of gaussian ICs
in a linear mixing.
In this chapter, we define BSS in a special way to clarify the distinction between
finding independent components, and finding the original sources. Thus, in therespective nonlinear BSS problem, one should find the original source signalsthat have generated the observed data. This is usually a clearly more meaningful
and unique problem than nonlinear ICA defined above, provided that suitable priorinformation is available on the sources and/or the mixing mapping. It is worthemphasizing that if some arbitrary independent components are found for the data
generated by (17.2), they may be quite different from the true source signals. Hencethe situation differs greatly from the basic linear data model (17.1), for which the
ICA or BSS problems have the same solution. Generally, solving the nonlinear BSS
problem is not easy, and requires additional prior information or suitable regularizing
constraints.
An important special case of the general nonlinear mixing model (17.2) consistsof so-called post-nonlinear mixtures. There each mixture has the form

s

1
0n
X
xi = fi @ aij sj A ; i = 1; : : : ; n
j =1(17.4)

Thus the sources sj , j = 1; : : : ; n are first mixed linearly according to the basic
ICA/BSS model (17.1), but after that a nonlinear function fi is applied to them to get
the final observations xi . It can be shown [418] that for the post-nonlinear mixtures,the indeterminacies are usually the same as for the basic linear instantaneous mixing
model (17.1). That is, the sources can be separated or the independent components estimated up to the scaling, permutation, and sign indeterminacies under weak
conditions on the mixing matrix and source distributions. The post-nonlinearity
assumption is useful and reasonable in many signal processing applications, becauseANONLINEAR ICA AND BSS

317

it can be thought of as a model for a nonlinear sensor distortion. In more generalsituations, it is a restrictive and somewhat arbitrary constraint. This model will betreated in more detail below.
Another difficulty in the general nonlinear BSS (or ICA) methods proposed thus
far is that they tend to be computationally rather demanding. Moreover, the computational load usually increases very rapidly with the dimensionality of the problem,
preventing in practice the application of nonlinear BSS methods to high-dimensional
data sets.
The nonlinear BSS and ICA methods presented in the literature could be divided
into two broad classes: generative approaches and signal transformation approaches[438]. In the generative approaches, the goal is to find a specific model that explains
how the observations were generated. In our case, this amounts to estimating boththe source signals and the unknown mixing mapping () that have generated theobserved data through the general mapping (17.2). In the signal transformation
methods, one tries to estimate the sources directly using the inverse transformation(17.3). In these methods, the number of estimated sources is the same as the numberof observed mixtures [438].

x17.1.2sf

Existence and uniqueness of nonlinear ICA

The question of existence and uniqueness of solutions for nonlinear independent
component analysis has been addressed in [213]. The authors show that there always
exists an infinity of solutions if the space of the nonlinear mixing functions is
not limited. They also present a method for constructing parameterized familiesof nonlinear ICA solutions. A unique solution (up to a rotation) can be obtainedin the two-dimensional special case if the mixing mapping is constrained to be a
conformal mapping together with some other assumptions; see [213] for details.
In the following, we present in more detail the constructive method introduced in[213] that always yields at least one solution to the nonlinear ICA problem. This
procedure might be considered as a generalization of the well-known Gram-Schmidt
orthogonalization method. Given m independent variables = (y1 ; : : : ; ym ) and avariable x, a new variable ym+1 = g ( ; x) is constructed so that the set y1 ; : : : ; ym+1
is mutually independent.
The construction is defined recursively as follows. Assume that we have alreadym independent random variables y1 ; : : : ; ym which are jointly uniformly distributed
in [0; 1]m . Here it is not a restriction to assume that the distributions of the yi areuniform, since this follows directly from the recursion, as will be seen below; for asingle variable, uniformity can be attained by the probability integral transformation;see (2.85). Denote by x any random variable, and by a1 ; : : : ; am ; b some nonrandom

f

f

y

y

318

NONLINEAR ICA

scalars. Defineg(a1 ; : : : ; am ; b; py;x) = P (x  bjy1 = a1 ; : : : ; ym = am )R b p (a ; : : : ; a ; )d

= ?1 py;x(a 1; : : : ; a m)
y 1
m

y

(17.5)ywhere py () and py;x () are the marginal probability densities ofand ( ; x),respectively (it is assumed here implicitly that such densities exist), and P (j) denotes
the conditional probability. The py;x in the argument of g is to remind that g depends
on the joint probability distribution of and x. For m = 0, g is simply the cumulative
distribution function of x. Now, g as defined above gives a nonlinear decomposition,
as stated in the following theorem.

yTheorem 17.1 Assume that y1 ; : : : ; ym are independent scalar random variables
that have a joint uniform distribution in the unit cube [0; 1]m . Let x be any scalarrandom variable. Define g as in (17.5), and set

ym+1 = g(y1 ; : : : ; ym ; x; py;x)(17.6)
Then ym+1 is independent from the y1 ; : : : ; ym , and the variables y1 ; : : : ; ym+1 arejointly uniformly distributed in the unit cube [0; 1]m+1 .
The theorem is proved in [213]. The constructive method given above can be used
to decompose n variables x1 ; : : : ; xn into n independent components y1 ; : : : ; yn ,giving a solution for the nonlinear ICA problem.
This construction also clearly shows that the decomposition in independent components is by no means unique. For example, we could first apply a linear trans, and then compute 0
formation on the to obtain another random vector 0 =000= ( ) withbeing defined using the above procedure, where is replaced by
0 . Thus we obtain another decomposition of into independent components. The
resulting decomposition 0 = 0 ( ) is in general different from , and cannot be
reduced to by any simple transformations. A more rigorous justification of thenonuniqueness property has been given in [213].
Lin [278] has recently derived some interesting theoretical results on ICA thatare useful in describing the nonuniqueness of the general nonlinear ICA problem.
Let the matrices s and x denote the Hessians of the logarithmic probability
densities log ps ( ) and log px ( ) of the source vector and mixture (data) vector ,respectively. Then for the basic linear ICA model (17.1) it holds that

x

g x

x

g

y g Lx

y

s

A

x Lx

H

H

x

x

y

x
y

s

Hs = AT HxA

x(17.7)sH

is the mixing matrix. If the components of are truly independent, s
whereshould be a diagonal matrix. Due to the symmetry of the Hessian matrices s andx , Eq. (17.7) imposes n(n ? 1)=2 constraints for the elements of the n  n matrix. Thus a constant mixing matrix can be solved by estimating x at two different
points, and assuming some values for the diagonal elements of s .

H
A

A

H

H
H

SEPARATION OF POST-NONLINEAR MIXTURES

319

If the nonlinear mapping (17.2) is twice differentiable, we can approximate it
is defined by the
locally at any point by the linear mixing model (17.1). There
first order term @ ( )=@ of the Taylor series expansion of ( ) at the desired point.
But now generally changes from point to point, so that the constraint conditions(17.7) still leave n(n ? 1)=2 degrees of freedom for determining the mixing matrix(omitting the diagonal elements). This also shows that the nonlinear ICA problem
is highly nonunique.
Taleb and Jutten have considered separability of nonlinear mixtures in [418, 227].
Their general conclusion is the same as earlier: Separation is impossible without
additional prior knowledge on the model, since the independence assumption alone
is not strong enough in the general nonlinear case.

A

fs s

fs

A

A

17.2

SEPARATION OF POST-NONLINEAR MIXTURES

Before discussing approaches applicable to general nonlinear mixtures, let us briefly
consider blind separation methods proposed for the simpler case of post-nonlinear
mixtures (17.4). Especially Taleb and Jutten have developed BSS methods for thiscase. Their main results have been represented in [418], and a short overview of theirstudies on this problem can be found in [227]. In the following, we present the themain points of their method.
A separation method for the post-nonlinear mixtures (17.4) should generally consist of two subsequent parts or stages:
1. A nonlinear stage, which should cancel the nonlinear distortions fi ; i =1; : : : ; n. This part consists of nonlinear functions gi (i ; u). The parameters
i of each nonlinearity gi are adjusted so that cancellation is achieved (at least
roughly).

v

2. A linear stage that separates the approximately linear mixtures obtained after
the nonlinear stage. This is done as usual by learning a n  n separating matrix
for which the components of the output vector =of the separating
system are statistically independent (or as independent as possible).

B

y Bv

y

Taleb and Jutten [418] use the mutual information I ( ) between the components
y1 ; : : : ; yn of the output vector (see Chapter 10) as the cost function and independence
criterion in both stages. For the linear part, minimization of the mutual information
leads to the familiar Bell-Sejnowski algorithm (see Chapters 10 and 9)@I (y) = ?Ef xT g ? (BT )?1@B
where components i of the vector
the output vector :y(17.8)

are score functions of the components yi of

0

d
p (u)
i (u) = du log pi (u) = pi (u)i(17.9)320

NONLINEAR ICA

Here pi (u) is the probability density function of yi and p0i (u) its derivative. In practice,
the natural gradient algorithm is used instead of the Bell-Sejnowski algorithm (17.8);
see Chapter 9.
For the nonlinear stage, one can derive the gradient learning rule [418]





(

n
@gk (k ; xk ) j@I (y) = ?E @ log j gk0 (k ; xk ) j ? E X
i (yi )bik
@ k
@ k
@ k
i=1

)

Here xk is the k th component of the input vector, bik is the element ik of the matrix, and gk0 is the derivative of the k th nonlinear function gk . The exact computation
algorithm depends naturally on the specific parametric form of the chosen nonlinear
mapping gk (k ; xk ). In [418], a multilayer perceptron network is used for modelingthe functions gk ( k ; xk ), k = 1; : : : ; n.
In linear BSS, it suffices that the score functions (17.9) are of the right type forachieving separation. However, their appropriate estimation is critical for the good
performance of the proposed nonlinear separation method. The score functions (17.9)
must be estimated adaptively from the output vector . Several alternative ways todo this are considered in [418]. An estimation method based on the Gram-Charlier
expansion performs appropriately only for mild post-nonlinear distortions. However,another method, which estimates the score functions directly, also provides very goodresults for hard nonlinearities. Experimental results are presented in [418]. A well
performing batch type method for estimating the score functions has been introduced
in a later paper [417].
Before proceeding, we mention that separation of post-nonlinear mixtures also
has been studied in [271, 267, 469] using mainly extensions of the natural gradientalgorithm.

B

y

17.3

NONLINEAR BSS USING SELF-ORGANIZING MAPS

One of the earliest ideas for achieving nonlinear BSS (or ICA) is to use Kohonen’s
self-organizing map (SOM) to that end. This method was originally introduced by
Pajunen et al. [345]. The SOM [247, 172] is a well-known mapping and visualization
method that in an unsupervised manner learns a nonlinear mapping from the data toa usually two-dimensional grid. The learned mapping from often high-dimensional
data space to the grid is such that it tries to preserve the structure of the data as wellas possible. Another goal in the SOM method is to map the data so that it would be
uniformly distributed on the rectangular (or hexagonal) grid. This can be roughlyachieved with suitable choices [345].
If the joint probability density of two random variables is uniformly distributed
inside a rectangle, then clearly the marginal densities along the sides of the rectangle
are statistically independent. This observation gives the justification for applying
self-organizing map to nonlinear BSS or ICA. The SOM mapping provides theregularization needed in nonlinear BSS, because it tries to preserve the structure321

NONLINEAR BSS USING SELF-ORGANIZING MAPSof the data. This implies that the mapping should be as simple as possible whileachieving the desired goals.
1

10

0.5

5

0

0

−0.5

−5

−1
0

5

10

15

20

25

30

35

40

45

50

−10
0

5

10

15

20

25

30

35

40

45

50

5

10

15

20

25

30

35

40

45

50

Original signals

20

1.5
1

10

0.5

0

0
−0.5

−10
−1
−1.5
0

5

10

15

20

25

30

35

40

45

50

−20
0

Fig. 17.1 Original source signals.

Fig. 17.2 Nonlinear mixtures.

The following experiment [345] illustrates the use of the self-organizing map in
nonlinear blind source separation. There were two subgaussian source signals sishown in Fig. 17.1, consisting of a sinusoid and uniformly distributed white noise.
Each source vector was first mixed linearly using the mixing matrixs 0:7 0:3 x

A = 0:3 0:7(17.10)After this, the data vectors were obtained as post-nonlinear mixtures of the sources
by applying the formula (17.4), where the nonlinearity fi (t) = t3 + t, i = 1; 2. These
mixtures xi are depicted in Fig. 17.2.
15

0
−5

10
−10
−15
−20
0

5

5

10

15

20

25

30

35

40

45

50

0
5

−5

0
−5

−10
−10
−15
0

5

10

15

20

25

30

35

40

45

Fig. 17.3 Signals separated by SOM.

50

−15
−10

−8

−6

−4

−2

0

2

4

6

8

10

Fig. 17.4 Converged SOM map.

The sources separated by the SOM method are shown in Fig. 17.3, and theconverged SOM map is illustrated in Fig. 17.4. The estimates of the source signals
in Fig. 17.3 are obtained by mapping each data vector onto the map of Fig. 17.4,

x

322

NONLINEAR ICA

and reading the coordinates of the mapped data vector. Even though the preceding
experiment was carried out with post-nonlinear mixtures, the use of the SOM method
is not limited to them.
Generally speaking, there are several difficulties in applying self-organizing maps
to nonlinear blind source separation. If the sources are uniformly distributed, thenit can be heuristically justified that the regularization of the nonlinear separating
mapping provided by the SOM approximately separates the sources. But if the true
sources are not uniformly distributed, the separating mapping providing uniform
densities inevitably causes distortions, which are in general the more serious thefarther the true source densities are from the uniform ones. Of course, the SOM
method still provides an approximate solution to the nonlinear ICA problem, but thissolution may have little to do with the true source signals.
Another difficulty in using SOM for nonlinear BSS or ICA is that computational
complexity increases very rapidly with the number of the sources (dimensionality ofthe map), limiting the potential application of this method to small-scale problems.
Furthermore, the mapping provided by the SOM is discrete, where the discretization
is determined by the number of grid points.

17.4

17.4.1

A GENERATIVE TOPOGRAPHIC MAPPING APPROACH TONONLINEAR BSS *BackgroundThe self-organizing map discussed briefly in the previous section is a nonlinear
mapping method that is inspired by neurobiological modeling arguments. Bishop,
Svensen and Williams introduced the generative topographic mapping (GTM) method
as a statistically more principled alternative to SOM. Their method is presented indetail in [49].
In the basic GTM method, mutually similar impulse (delta) functions that are
equispaced on a rectangular grid are used to model the discrete uniform density in thespace of latent variables, or the joint density of the sources in our case. The mappingfrom the sources to the observed data, corresponding in our nonlinear BSS problem
to the nonlinear mixing mapping (17.2), is modeled using a mixture-of-gaussiansmodel. The parameters of the mixture-of-gaussians model, defining the mixing
mapping, are then estimated using a maximum likelihood (ML) method (see Section
4.5) realized by the expectation-maximization (EM) algorithm [48, 172]. After this,the inverse (separating) mapping from the data to the latent variables (sources) canbe determined.
It is well-known that any continuous smooth enough mapping can be approximatedwith arbitrary accuracy using a mixture-of-gaussians model with sufficiently many
gaussian basis functions [172, 48]. Roughly stated, this provides the theoreticalbasis of the GTM method. A fundamental difference of the GTM method comparedwith SOM is that GTM is based on a generative approach that starts by assuming
a model for the latent variables, in our case the sources. On the other hand, SOM

A GENERATIVE TOPOGRAPHIC MAPPING APPROACH *323

tries to separate the sources directly by starting from the data and constructing asuitable separating signal transformation. A key benefit of GTM is its firm theoreticalfoundation which helps to overcome some of the limitations of SOM. This also
provides the basis of generalizing the GTM approach to arbitrary source densities.
Using the basic GTM method instead of SOM for nonlinear blind source separation
does not yet bring out any notable improvement, because the densities of the sources
are still assumed to be uniform. However, it is straightforward to generalize the GTM
method to arbitrary known source densities. The advantage of this approach is that
one can directly regularize the inverse of the mixing mapping by using the known
source densities. This modified GTM method is then used for finding a noncomplex
mixing mapping. This approach is described in the following.
17.4.2The modified GTM method

The modified GTM method introduced in [346] differs from the standard GTM [49]
only in that the required joint density of the latent variables (sources) is defined asa weighted sum of delta functions instead of plain delta functions. The weighting
coefficients are determined by discretizing the known source densities. Only the main
points of the GTM method are presented here, with emphasis on the modifications
made for applying it to nonlinear blind source separation. Readers wishing to gain adeeper understanding of the GTM method should look at the original paper [49].
The GTM method closely resembles SOM in that it uses a discrete grid of points
forming a regular array in the m-dimensional latent space. As in SOM, the dimensionof the latent space is usually m = 2. Vectors lying in the latent space are denoted by(t); in our application they will be source vectors. The GTM method uses a set of L
fixed nonlinear basis functions fj ( )g, j = 1; : : : ; L, which form a nonorthogonalbasis set. These basis functions typically consist of a regular array of sphericalgaussian functions, but the basis functions can at least in principle be of other types.
The mapping from the m-dimensional latent space to the n-dimensional data
space, which is in our case the mixing mapping of Eq. (17.2), is in GTM modeled asa linear combination of basis functions 'j :ss

x = f (s) = M'(s); ' = [' ; ' ; : : : ; 'L]THere M is an n  L matrix of weight parameters.
1

Denote the node locations in the latent space bycorresponding set of reference vectors

2(17.11)i . Eq. (17.11) then defines ami = M'( i)(17.12)in data space. Each of these reference vectors then forms the center of an isotropic
gaussian distribution in data space. Denoting the common variance of these gaussiansby ?1 , we get n=px(x j i) = 22



exp ? 2 k mi ? x k2

(17.13)

324

NONLINEAR ICA

The probability density function for the GTM model is obtained by summing over
all of the gaussian components, yieldingpx (x(t) j M; ) ==KX

P (i)px(x j i)

i=1
K 1 
X
i=1 K

n=

2

2



exp ? 2 k mi ? x k2

(17.14)

Here K is the total number of gaussian components, which is equal to the number ofgrid points in latent space, and the prior probabilities P (i) of the gaussian components
are all equal to 1=K .
GTM tries to represent the distribution of the observed data in the n-dimensional
data space in terms of a smaller m-dimensional nonlinear manifold [49]. The gaussian
distribution in (17.13) represents a noise or error model which is needed because thedata usually does not lie exactly in such a lower dimensional manifold. It is importantto realize that the K gaussian distributions defined in (17.13) have nothing to do withthe basis function 'i , i = 1; : : : ; L. Usually it is advisable that the number L of
the basis functions is clearly smaller than the number K of node locations and theirrespective noise distributions (17.13). In this way, one can avoid overfitting andprevent the mixing mapping (17.11) to become overly complicated.
and the inverse
The unknown parameters in this model are the weight matrix
variance . These parameters are estimated by fitting the model (17.14) to theobserved data vectors (1); (2); : : : ; (T ) using the maximum likelihood method
discussed earlier in Section 4.5. The log likelihood function of the observed data isgiven byxM

x

L(M; ) =

T
X
t=1

x

x

log px(x(t)jM; ) =

Z
T
X
t=1

log px (x(t)js; M; )ps (s)ds(17.15)where ?1 is the variance of given and , and T is the total number of data
vectors (t).
For applying the modified GTM method, the probability density function ps ( )of the source vectors should be known. Assuming that the sources s1 ; s2 ; : : : ; sm
are statistically independent, this joint density can be evaluated as the product of themarginal densities of the individual sources:

x

x

s

M

s

s

ps (s) =mY
i=1

pi (si )(17.16)

Each marginal density is here a discrete density defined at the sampling points
corresponding to the locations of the node vectors.
The latent space in the GTM method usually has a small dimension, typicallym = 2. The method can be applied in principle for m > 2, but its computationalload then increases quite rapidly just like in the SOM method. For this reason, only

325

A GENERATIVE TOPOGRAPHIC MAPPING APPROACH *two sources s1 and s2 are considered in the following. The dimension of the latent
space is chosen to be m = 2, and we use a rectangular K1  K2 grid with equispaced
nodes, so that the total number of nodes is K = K1  K2 . The locations of the nodepoints in the grid are denoted by ij , i = 1; : : : ; K1 ; j = 1; : : : ; K2 . One can thenwrite

ps(s) =

K
K X
X
1

2

i=1 j =1

aij (s ? ij ) =

K
X
q=1

aq (s ? q )(17.17)The coefficients aij = p1 (i)p2 (j ), where p1 (i) and p2 (j ) are the values of the marginaldensities p1 (s1 ) and p2 (s2 ) corresponding to the location of the node point ij . In
(17.17),  () is the Dirac delta function or impulse function which has the special
property that g ( ) ( ? 0 )ds = g ( 0 ) if the integration extends over the point 0 ,otherwise the integral is zero. In the last phase, the node points and their respective
probabilities have been reindexed again using a single index q for easier notation.
This can be done easily by going through all the node points in some prescribedorder, for example rowwise.
Inserting (17.17) into (17.15) yields

R s s sL(M; ) =

s

T
X
t=1

log

s

K
X
q=1

aq px(x(t)j q ; M; )!

(17.18)M

andComputing the gradient of this expression with respect to the weight matrix
setting it to zero, after some manipulations yields the following equation for updating
the weight matrix :M(T Gold )MTnew = T Rold X

X xx(17.19)In this formula, = [ (1); : : : ; (T )]T is the data matrix, and the (q; j )-th element
fqj = 'j ( q ) of the K  L matrix is the value of the j th basis function 'j () at
the q th node point q . Furthermore, is a diagonal matrix with elements


G

Gqq =

T
X
t=1

Rqt (M; )R are
Rqt = PKaq px (x(t)j q ; M; )k ak px (x(t)j k ; M; )(17.20)and the elements of the responsibility matrix(17.21)=1

Then the variance parametercan be updated using the formula

K X
T
1 = 1 X2
Tn q=1 t=1 Rqt k Mnew'( q ) ? x(t) k
newwhere n is the dimension of the data space.

(17.22)

326

NONLINEAR ICA

Fig. 17.5 Source signals.

Fig. 17.6 Separated signals.

In GTM, the EM algorithm is used for maximizing the likelihood. Here the Estep (17.21) consists of computing the responsibilities Rqt , and the M-steps (17.19),(17.22) of updating the parameters and . The preceding derivation is quite similar
to the one as in the original GTM method [49], only the prior density coefficients aij= aq have been added to the model.
After a few iterations, the EM algorithm converges to the parameter values and  that maximize the log likelihood (17.15), at least locally. The optimum
values  and  then specify the estimated probability density (17.14) that GTM
provides for the data vectors . Because the prior density ps ( ) of the sources
is assumed to be known, it is then straightforward to compute the posterior densityp( (t) j (t);  ;  ) of the sources given the observed data using the Bayes’ rule.
As mentioned in Chapter 4, this posterior density contains all the relevant information
about the sources.
However, it is often convenient to choose a specific source estimate (t) corresponding to each data vector (t) for visualizing purposes. An often used estimate
is the mean Ef (t) j (t);  ;  g of the posterior density. It can be computed in
the GTM method from the simple formula [49]

M

M

M

s

x

s

s

x M
s

s

x
x M

^s(t) = Efs(t) j x(t); M ;  g =

K
X
q=1

Rqt q

(17.23)

If the posterior density of the sources is multimodal, the posterior mean (17.23) cangive misleading results. Then it is better to use for example the maximum a posteriori(MAP) estimate, which is simply the source value corresponding to the maximum
responsibility qmax = argmax(Rqt ), q = 1; : : : ; K; for each sample index t.
17.4.3

An experiment

In the following, a simple experiment involving two sources shown in Fig. 17.5
and three noisy nonlinear mixtures is described. The mixed data was generated bytransforming linear mixtures of the original sources using a multilayer perceptron

A GENERATIVE TOPOGRAPHIC MAPPING APPROACH *327

Fig. 17.7 Joint mixture densities with superimposed maps. Top left: Joint density p(x1 ; x2 )of mixtures x1 and x2 . Top right: Joint density p(x1 ; x3 ) of mixtures x1 and x3 . Bottom left:
Joint density p(x2 ; x3 ) of mixtures x2 and x3 . Bottom right: Joint density of the estimatedtwo source signals.

network with a volume conserving architecture (see [104]). Such an architecture waschosen for ensuring that the total mixing mapping is bijective and therefore reversible,and for avoiding highly complex distortions of the source densities. However, thischoice has the advantage that it makes the total mixing mapping more complex thanthe post-nonlinear model (17.4). Finally, gaussian noise was added to the mixtures.
The mixtures were generated using the modelU

x = As + tanh(UAs) + n(17.24)

whereis an upper-diagonal matrix with zero diagonal elements. The nonzero
were drawn from a standard gaussian distribution. The matrix
elements ofensures volume conservation of the nonlinearity applied to.
The modified GTM algorithm presented above was used to learn a separating
mapping. For reducing scaling effects, the mixtures were first whitened. After
whitening the mixtures are uncorrelated and have unit variance. Then the modified
GTM algorithm was run for eight iterations using a 5  5 grid. The number of basis
functions was L = 7.

U

As

U

328

NONLINEAR ICA

The separated sources depicted in Fig. 17.6 can be compared with the original
sources in Fig. 17.5. The waveforms of the original sources are approximately recovered, even though there is some inevitable distortion due to the noise, discretization,and the difficulty of the problem.
The mixtures are not shown here directly because they are in three dimensions.
The two-dimensional marginal densities of the mixtures are shown in Fig. 17.7,
however. They clearly reveal the nonlinearity of the mixing mapping; in particular,
the joint density p(x2 ; x3 ) of the second and third mixtures (components of the data
vector ) is highly nonlinear. Also the joint density of the separated sources is shownin Fig. 17.7 in the subfigure at bottom right. It indicates that a factorizable density
has been approximately obtained. The superimposed maps in the first 3 subfiguresof Figure 17.7 were obtained by mapping a 10  10 grid of source vectors to
the mixture (data) space using the mapping (17.11) learned by the modified GTM
algorithm.

x

s

17.5

AN ENSEMBLE LEARNING APPROACH TO NONLINEAR BSS

In this section, we present a new generative approach for nonlinear blind source
separation or independent component analysis. Here the nonlinear mapping (17.2)
from the unknown sources to the known observations is modeled using the familiar
multilayer perceptron (MLP) network structure [172, 48]. MLP networks have theuniversal approximation property [172] for smooth continuous mappings, and theysuit well for modeling both strongly and mildly nonlinear mappings. However, thelearning procedure is based on unsupervised Bayesian ensemble learning. It is quite
different from standard back-propagation learning which minimizes the mean-square
representation error in MLP networks in a supervised manner [172, 48].

s

17.5.1x

Ensemble learning

A flexible model family, such as MLP networks, provides infinitely many possible
explanations of different complexity for the observed data. Choosing too complex
a model results in overfitting, where the model tries to make up meaningless explanations for the noise in addition to the true sources or independent components.
Choosing too simple a model results in underfitting, leaving hidden some of the true
sources that have generated the data.
An appropriate solution to this problem is that no single model should actually
be chosen. Instead, all the possible explanations should be taken into accountand weighted according to their posterior probabilities. This approach, known as
Bayesian learning [48], optimally solves the trade-off between under- and overfitting.
All the relevant information needed in choosing an appropriate model is containedin the posterior probability density functions (pdf’s) of different model structures.
The posterior pdf’s of too simple models are low, because they leave a lot of the data
unexplained. On the other hand, too complex models occupy little probability mass,AN ENSEMBLE LEARNING APPROACH TO NONLINEAR BSS

329

even though they often show a high but very narrow peak in their posterior pdf’s
corresponding to the overfitted parameters.
In practice, exact computation of the posterior pdf’s of the models is impossible.
Therefore, some suitable approximation method must be used. Ensemble learning[180, 25, 260], also known as variational learning, is a method for parametric approximation of posterior pdf’s where the search takes into account the probability massof the models. Therefore, it does not suffer from overfitting. The basic idea in ensemble learning is to minimize the misfit between the posterior pdf and its parametricapproximation.
Let us denote by X = f (t)jtg the set of available mixture (data) vectors, and by
S = f (t)jtg the respective source vectors. Denote by  all the unknown parametersof the mixture (data) model, which will be described in more detail in the next
subsection. Furthermore, let p(S;  jX ) denote the exact posterior pdf and q (S;  jX )its parametric approximation. The misfit is measured with the Kullback-Leibler (KL)divergence JKL between the densities p and q , which is defined by the cost function

x

s

  Z
jX )
JKL = Eq log pq = q(S; jX ) log pq((S;S; jX ) ddS

(17.25)

The Kullback-Leibler divergence or distance measures the difference in the probability mass between the densities p and q . Its minimum value 0 is achieved when thetwo densities are the same; see Chapter 5.
17.5.2Model structure

As mentioned before, MLP networks are now used for modeling the nonlinear mixing
mapping () in (17.2). In the following, we represent the mixing model [259, 436]
in more detail. Let (t) denote the observed data vector at time t, and (t) the vectorof independent components (source signals) at time t. The matrices and containthe weights of the output and the hidden layers of the network, respectively, andand are the respective bias vectors of the output and hidden layers. The vectorof nonlinear activation functions, applied componentwise, is denoted by (), and(t) is a zero mean gaussian noise vector corrupting the observations. Using thesenotations, the mixing (data) model can be written

f

x

Q

a

s

A

b

g

n

x(t) = Qg(As(t) + a) + b + n(t)(17.26)The hyperbolic tangent g (y ) = tanh(y ) is used as the nonlinear activation function,
which is a typical choice in MLP networks. Other continuous activation functions
could be used, too. The sources are assumed to be independent, and they aremodeled by mixtures of gaussians. The independence assumption is natural, because
the goal of the model is to find the underlying independent components of theobservations. By using mixtures of gaussians, one can model sufficiently well any
nongaussian distribution of the sources [48]. This type of representation has earlier
been successfully applied to the standard linear ICA model in [258].

330

NONLINEAR ICA

A

Q

and , and the
The parameters of the model are (1) the weight matrices
bias vectors and ; (2) the parameters of the distributions of the noise, source
signals and column vectors of the weight matrices; (3) the hyperparameters used for
defining the distributions of the biases and the parameters in the group (2). For a
more detailed description, see [259, 436]. All the parameterized distributions are
assumed to be gaussian, except for the sources, which are modeled as mixtures ofgaussians. This does not limit the generality of the approach severely, but makes
computational implementation simpler and much more efficient. The hierarchicaldescription of the distributions of the parameters of the model used here is a standard
procedure in probabilistic Bayesian modeling. Its strength lies in that knowledge
about equivalent status of different parameters can be easily incorporated. For
example, all the variances of the noise components have a similar status in the model.
This is reflected by the fact that their distributions are assumed to be governed bycommon hyperparameters.

a17.5.3bComputing Kullback-Leibler cost function *In this subsection, the Kullback-Leibler cost function JKL defined earlier in Eq.
(17.25) in considered in more detail. For approximating and then minimizing it, weneed two things: the exact formulation of the posterior density p(S; jX ) and its
parametric approximation q (S; jX ).
According to the Bayes’ rule, the posterior pdf of the unknown variables S and isp(S; jX ) = p(X jS; p)(pX(S) j)p()(17.27)

The probability density p(X jS; ) of the data X given the sources S and theparameters  is obtained from Eq. (17.26). Let us denote the variance of the ith
component of the noise vector (t) by by i2 . The distribution p(xi (t)j (t);  )of the ith component xi (t) of the data vector (t) is thus gaussian with the meanT ( (t) + ) + bi and variance  2 . Here T denotes the ith row vector of theiiiweight matrix , and bi is the ith component of the bias vector . As usually, thenoise components ni (t) are assumed to be independent, and thereforenq g AsxqaQp(X jS; ) =

T Y
n
Y

t=1 i=1p(xi (t)js(t); )sb(17.28)The terms p(S j) and p( ) in (17.27) are also products of simple gaussian distributions, and they are obtained directly from the definition of the model structure[259, 436]. The term p(X ) does not depend on the model parameters and can beneglected.
The approximation q (S;  j X ) must be simple enough for mathematical tractability and computational efficiency. First, we assume that the source signals S are
independent of the other parameters , so that q (S;  j X ) decouples intoq(S;  j X ) = q(S j X )q( j X )(17.29)

331

AN ENSEMBLE LEARNING APPROACH TO NONLINEAR BSS

For the parameters , a gaussian density with a diagonal covariance matrix is used.
This implies that the approximation q ( j X ) is a product of independent gaussian
distributions:q( j X ) =Yjqj (j j X )(17.30)The parameters of each gaussian component density qj (j j X ) are its mean i andvariance ~i .
The source signals si (t), i = 1; : : : ; n are assumed to be mutually statisticallyindependent, and also at different time instants (sample values) t = 1; : : : ; T , so thatq(S j X ) =

T Y
n
Y

t=1 i=1

qti (si (t) j X )(17.31)

Here the component densities qti (si (t) j X ) are modeled as mixtures of gaussian
densities. Inserting Eqs. (17.31) and (17.30) into (17.29) provides the desired approximation q (S;  j X ) of the posterior density.
Both the posterior density p(S;  jX ) and its approximation q (S;  jX ) are productsof simple gaussian or mixture-of-gaussians terms, which simplifies the cost function(17.25) considerably: it splits into expectations of many simple terms. The terms of
the form Eq flog qj (S; j j X )g are negative entropies of gaussians, having the exact
values ?(1+log 2 ~j )=2. The terms of the form ?Eq flog p(xi (t)j (t);  )g are mostdifficult to handle. They are approximated by applying second order Taylor series
expansions of the nonlinear activation functions as explained in [259, 436]. The
remaining terms are expectations of simple gaussian terms, which can be computed
as in [258].
The cost function JKL depends on the posterior means i and variances ~i of theparameters of the network and the source signals. This is because instead of finding
a point estimate, the joint posterior pdf of the sources and parameters is estimatedin ensemble learning. The variances give information about the reliability of theestimates.
Let us denote the two parts of the cost function (17.25) arising from the denominator and numerator of the logarithm by, respectively, Jp = ?Eq flog pg and Jq =Eq flog q g. The variances ~i are obtained by differentiating (17.25) with respect to ~i[259, 436]:s@ JKL = @ Jp + @ Jq = @ Jp ? 1@ ~@ ~ @ ~@ ~ 2~(17.32)Equating this to zero yields a fixed-point iteration for updating the variances:~ = 2 @ J~p
@

?

1

(17.33)The means i can be estimated from the approximate Newton iteration [259, 436]

 2 ?1  ? @@Jp ~  ? @@Jp @ @JKL
2(17.34)

332

NONLINEAR ICA

The formulas (17.33) and (17.34) have a central role in learning.
17.5.4

Learning procedure *Usually MLP networks learn the nonlinear input–output mapping in a supervisedmanner using known input–output training pairs, for which the mean-square mapping
error is minimized using the back-propagation algorithm or some alternative method[172, 48]. In our case, the inputs are the unknown source signals (t), and only
the outputs of the MLP network, namely the observed data vectors (t), are known.
Hence, unsupervised learning must be applied. Detailed account of the learning
method and discussion of potential problems can be found in [259, 436]. In thefollowing, we give an overall description of the learning method.
The practical learning procedure used in all the experiments was the same. First,
linear principal component analysis (PCA) (see Chapter 6) was applied to find sensible
initial values for the posterior means of the sources. Even though PCA is a linear
method, it yields much better initial values than a random choice. The posteriorvariances of the sources are initialized to small values. Good initial values areimportant for the method because it can effectively prune away unused parts of the
MLP network1 .
Initially the weights of the MLP network have random values, and the network
has quite a bad representation for the data. If the sources were adapted from random
values, too, the network would consider many of the sources useless for the representation and prune them away. This would lead to a local minimum from which
the network might not recover. Therefore the sources were fixed at the values given
by linear PCA for the first 50 sweeps through the entire data set. This allows the
MLP network to find a meaningful mapping from the sources to the observations,
thereby justifying using the sources for the representation. For the same reason,
the parameters controlling the distributions of the sources, weights, noise, and thehyperparameters are not adapted during the first 100 sweeps. They are adapted only
after the MLP network has found sensible values for the variables whose distributions
these parameters control.
Furthermore, we first used a simpler nonlinear model where the sources had
standard gaussian distributions instead of mixtures of gaussians. This is called
nonlinear factor analysis model in the following. After this phase, the sources wererotated using the FastICA algorithm. The rotation of the sources was compensated byan inverse rotation of the weight matrix of the hidden layer. The final representationof the data was then found by continuing learning, but using now the mixture-ofgaussians model for the sources. In [259], this representation is called nonlinear
independent factor analysis.

xA

1 Pruning techniques for neural networks are discussed, for example, in [172, 48]

s

333

AN ENSEMBLE LEARNING APPROACH TO NONLINEAR BSS

4

4

4

4

2

2

2

2

0

0

0

0

−2

−2

−2

−2

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

4

4

4

4

2

2

2

2

0

0

0

0

−2

−2

−2

−2

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−2

0

2

4

Fig. 17.8 Original sources are on the x-axis of each scatter plot and the sources estimatedby a linear ICA are on the y -axis. Signal-to-noise ratio is 0.7 dB.

17.5.5Experimental results

In all the simulations, the total number of sweeps was 7500, where one sweep means
going through all the observations (t) once. As explained before, a nonlinear
factor analysis (or nonlinear PCA subspace) representation using plain gaussians asmodel distributions for the sources was estimated first. In the experiments, 2000
first sweeps were used for finding this intermediate representation. After a linear
ICA rotation, the final mixture-of-gaussians representation of the sources was thenestimated during the remaining 5500 sweeps. In the following, experiments withartificially generated nonlinear data are first described, followed by separation results
on real-world process data.

x

Simulated data In the first experiment, there were eight sources, four subgaussianand four supergaussian ones. The data were generated from these sources througha nonlinear mapping, which was obtained by using a randomly initialized MLP
network having 30 hidden neurons and 20 output neurons. Gaussian noise having astandard deviation of 0.1 was added to the data. The nonlinearity used in the hidden
neurons was chosen to be the inverse hyperbolic sine sinh?1 (x). This guarantees
that the nonlinear source separation algorithm using the MLP network with tanhnonlinearities cannot use exactly the same weights.
Several different numbers of hidden neurons were tested in order to optimize
the structure of the MLP network, but the number of sources was assumed to be
known. This assumption is reasonable because it seems to be possible to optimize
the number of sources simply by minimizing the cost function, as experiments withpurely gaussian sources have shown [259, 438]. The MLP network which minimized

334

NONLINEAR ICA

4

4

4

4

2

2

2

2

0

0

0

0

−2

−2

−2

−2

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

4

4

4

4

2

2

2

2

0

0

0

0

−2

−2

−2

−2

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−2

0

2

4

Fig. 17.9 Scatter plots of the sources after 2000 sweeps of nonlinear factor analysis followed
by a rotation with a linear ICA. Signal-to-noise ratio is 13.2 dB.
4

4

4

4

2

2

2

2

0

0

0

0

−2

−2

−2

−2

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

4

4

4

4

2

2

2

2

0

0

0

0

−2

−2

−2

−2

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−4
−4

−2

0

2

4

−2

0

2

4

Fig. 17.10 The final separation results after using a mixture-of-gaussians model for thesources for the last 5500 sweeps. Signal to noise ratio is 17.3 dB. The original sources haveclearly been found.

the Kullback-Leibler cost function turned out to have 50 hidden neurons. The numberof gaussians in the mixtures modeling the distribution of each source was chosen to
be three, and no attempt was made to optimize this number.
The results are depicted in Figs. 17.8, 17.9, and 17.10. Each figure shows eightscatter plots, corresponding to each of the eight original sources. The original sourceAN ENSEMBLE LEARNING APPROACH TO NONLINEAR BSS

335

1
linear FA
nonlinear FA
0.9
0.8

Remaining energy

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0

5

10
15
Number of sources

20

25

Fig. 17.11 The remaining energy of the process data as a function of the number of extracted
components using linear and nonlinear factor analysis

which was used for generating the data appears on the x-axis, and the respective
estimated source in on the y -axis of each plot. Each point corresponds to one sample
point (t). The upper plots of each figure correspond to the supergaussian and thelower plots to the subgaussian sources. The optimal result is a straight line implyingthat the estimated values of the sources coincide with the true values.
Figure 17.8 shows the result given by the linear FastICA algorithm alone. Linear
ICA is able to retrieve the original sources with only 0.7 dB signal-to-noise ratio(SNR). In practice a linear method could not deduce the number of sources, andthe result would be even worse. The poor signal-to-noise ratio shows that the data
really lies in a nonlinear subspace. Figure 17.9 depicts the results after 2000 sweeps
with gaussian sources (nonlinear factor analysis) followed by a rotation with linear
FastICA. Now the SNR is considerably better, 13.2 dB, and the sources have clearly
been retrieved. Figure 17.10 shows the final results after another 5500 sweeps whenthe mixture-of-gaussians model has been used for the sources. The SNR has further
improved to 17.3 dB.

x

Industrial process data Another data set consisted of 30 time series of length2480 measured using different sensors from an industrial pulp process. A humanexpert preprocessed the measured signals by roughly compensating for the time lagsof the process originating from the finite speed of pulp flow through the process.

336

NONLINEAR ICA

Fig. 17.12 The ten estimated sources of the industrial pulp process. Time increases fromleft to right.

Fig. 17.13 The 30 original time series are shown on each plot on top of the reconstruction
made from the sources shown in Fig. 17.12

For studying the intrinsic dimensionality of the data, linear factor analysis wasapplied to the data. The result is shown in Fig. 17.11. The figure also shows theresults with nonlinear factor analysis. It is obvious that the data are quite nonlinear,because nonlinear factor analysis is able to explain the data with 10 components
equally well as linear factor analysis (PCA) with 21 components.
Different numbers of hidden neurons and sources were tested with random initializations using gaussian model for sources (nonlinear factor analysis). It turned outthat the Kullback-Leibler cost function was minimized for an MLP network having

OTHER APPROACHES

337

10 sources (outputs) and 30 hidden neurons. The same network size was chosen for
nonlinear blind source separation based on the mixture-of-gaussians model for thesources. After 2000 sweeps using the nonlinear factor analysis model, the sourceswere rotated with FastICA, and each source was modeled with a mixture of threegaussian distributions. The learning process was then continued using this refinedmodel for 5500 more sweeps. The resulting sources are shown in Fig. 17.12.
Figure 17.13 shows 30 subimages, each corresponding to a specific measurement
made from the process. The original measurement appears as the upper time series
in each subimage, and the lower time series is the reconstruction of this measurement
given by the network. These reconstructions are the posterior means of the outputsof the network when the inputs were the estimated sources shown in Fig. 17.12. The
original measurements show a great variability, but the reconstructions are strikingly
accurate. In some cases it seems that the reconstruction is less noisy than the original
signal. This is not so surprising, because generally an estimate of data reconstructedfrom a smaller number of most significant components is often able to remove somenoise.
The experiments suggest that the estimated source signals can have meaningfulphysical interpretations. The results are encouraging, but further studies are needed
to verify the interpretations of the signals.
The proposed ensemble learning method for nonlinear blind source separation
can be extended in several ways. An obvious extension is inclusion of time delays
into the data model, making use of the temporal information often present in thesources. This would probably help in describing the process data even better. Using
the Bayesian framework, it is also easy to treat missing observations or only partly
known inputs.

17.6

OTHER APPROACHESIn this section, other methods proposed for nonlinear independent component analysis
or blind source separation are briefly reviewed. The interested reader can find more
information on them in the given references.
Already in 1987, Jutten [226] used soft nonlinear mixtures for assessing therobustness and performance of the seminal Hérault-Jutten algorithm introduced forthe linear BSS problem (see Chapter 12). However, Burel [57] was probably the
first to introduce an algorithm specifically for nonlinear ICA. His method is basedon back-propagation type neural learning for parametric nonlinearities, and suffersfrom high computational complexity and problems with local minima. In a series of
papers [104, 357, 355, 105, 358], Deco and Parra with their co-authors developed
methods based on volume conserving symplectic transformations for nonlinear ICA.
The constraint of volume conservation is, however, somewhat arbitrary, and so thesemethods are usually not able to recover the original sources. Also their computationalload tends to be high.
The idea of using the self-organizing map [247] for the general nonlinear ICA
problem, as discussed in Section 17.3, was introduced in [345]. However, this338

NONLINEAR ICA

approach is limited mainly to separation of sources having distributions not too
far from the uniform distribution. The nonlinear mixing process is considered tobe regular: it is required that the inverse of the mixing mapping should be theleast complex mapping which yields independent components [345]. Lin, Grier,and Cowan [279] have independently proposed using SOM for nonlinear ICA in adifferent manner by treating ICA as a computational geometry problem.
The ensemble learning approach to nonlinear ICA, discussed in more detail earlier
in this chapter, is based on using multilayer perceptron networks as a flexible model forthe nonlinear mixing mapping (17.2). Several authors have used autoassociative MLP
networks [172] discussed briefly in Chapter 6 for learning a similar type of mappings.
Both the generative model and its inversion are learned simultaneously, but separately
without utilizing the fact that the models are connected. Autoassociative MLPs haveshown some success in nonlinear data representation [172], but generally they sufferfrom slow learning prone to local minima.
Most works on autoassociative MLPs use point estimates for weights and sources
obtained by minimizing the mean-square representation error for the data. It is then
impossible to reliably choose the structure of the model, and problems with over- or
underfitting can be severe. Hecht-Nielsen [176, 177] proposed so-called replicator
networks for universal optimal nonlinear coding of input data. These networks
are autoassociate MLP networks, where the data vectors are mapped onto a unit
hypercube so that the mapped data is uniformly distributed inside the hypercube.
The coordinates of the mapped data on the axes of the hypercube, called natural
coordinates, then in fact form a nonlinear ICA solution, even though this has not been
noted in the original papers [176, 177].
Hochreiter and Schmidhuber [181] have used in context with MLP networks amethod based on minimum description length, called LOCOCODE. This method
does estimate the distribution of the weights, but has no model for the sources. It
is then impossible to measure the description length of the sources. Anyway, their
method shows interesting connections with ICA; sometimes it provides a nonlinear
ICA solution, sometimes it does not [181]. Another well-known information theoretic
criterion, mutual information, is applied to measuring independence in [2, 469]. In
these papers, methods based on various MLP network structures are also introduced
for nonlinear blind separation. In particular, Yang, Amari, and Cichocki [469] deal
with extensions of the basic natural gradient method (see Chapter 9), for nonlinear
BSS, presenting also an extension based on entropy maximization and experimentswith post-nonlinear mixtures.
Xu has developed the general Bayesian Ying-Yang framework which can beapplied on ICA as well; see e.g. [462, 463].
Other general approaches proposed for solving the nonlinear ICA or BSS problems
include a pattern repulsion based method [295], a state-space modeling approach [86],and an entropy-based method [134]. Various separation methods for the considerably
simpler case of post-nonlinear mixtures (17.4) have been introduced at least in [271,267, 365, 418, 417].
In the ensemble learning method discussed before, the necessary regularization
for nonlinear ICA is achieved by choosing the model and sources that have mostCONCLUDING REMARKS

339

probably generated the observed data. Attias has applied a similar generative modelto the linear ICA problem in [19]. The ensemble learning based method described inthis chapter differs from the method introduced in [19] in that it uses a more general
nonlinear data model, and applies a fully Bayesian treatment to the hyperparametersof the network or graphical model, too. A related extension was proposed in [20].
Connections of the ensemble learning method with other Bayesian approaches are
discussed in more detail in [259, 438].
Nonlinear independent component analysis or blind source separation are generally difficult problems both computationally and conceptually. Therefore, local linear
ICA/BSS methods have received some attention recently as a practical compromise
between linear ICA and completely nonlinear ICA or BSS. These methods are moregeneral than standard linear ICA in that several different linear ICA models are used
to describe the observed data. The local linear ICA models can be either overlapping,as in the mixture-of-ICA methods introduced in [273], or nonoverlapping, as in theclustering-based methods proposed in [234, 349].

17.7

CONCLUDING REMARKSIn this chapter, generalizations of standard linear independent component analysis(ICA) or blind source separation (BSS) problems to nonlinear data models have been
considered. We made a distinction between the ICA and BSS problems, because in the
nonlinear case their relation is more complicated than in the linear case. In particular,
the nonlinear ICA problem is ill-posed without some suitable extra constraints or
regularization, having in general infinitely many qualitatively different solutions.
Solving the nonlinear BSS problem appropriately using only the independence
assumption of the source signals is possible only in simple special cases, for example,
when the mixtures are post-nonlinear. Otherwise, suitable additional information on
the problem is required. This extra information is often provided in the form of
regularizing constraints. Various methods proposed for regularizing the nonlinear
ICA or BSS problems have been briefly reviewed in the previous section. Anotherpossibility is to have more information about the sources or mixtures themselves.
An example of such an approach is the method based on the generative topographic
mapping (GTM), which requires knowledge of the probability distributions of thesources.
A large part of this chapter has been devoted to a recently introduced fully Bayesian
approach based on ensemble learning for solving the nonlinear BSS problem. Thismethod applies the well-known MLP network, which is well-suited to modelingboth mildly and strongly nonlinear mappings. The proposed unsupervised ensemblelearning method tries to find the sources and the mapping that together have mostprobably generated the observed data. This regularization principle has a firm theoretical foundation, and it is intuitively satisfying for the nonlinear source separation
problem. The results are encouraging for both artificial and real-world data. The
ensemble-learning method allows nonlinear source separation for larger scale prob-

340

NONLINEAR ICA

lems than some previously proposed computationally quite demanding methods, andit can be easily extended in various directions.
A lot of work remains to be done in developing suitable methods for the nonlinear
ICA and BSS problems, and understanding better which constraints are most suitablein each situation. A number of different approaches have been proposed, but nocomparisons are yet available for assessing their strengths and weaknesses.

18
Methods using Time
Structure
The model of independent component analysis (ICA) that we have considered so
far consists of mixing independent random variables, usually linearly. In many
applications, however, what is mixed is not random variables but time signals, ortime series. This is in contrast to the basic ICA model in which the samples ofhave no particular order: We could shuffle them in any way we like, and this would
have no effect on the validity of the model, nor on the estimation methods we havediscussed. If the independent components (ICs) are time signals, the situation is quite
different.
In fact, if the ICs are time signals, they may contain much more structure than simple random variables. For example, the autocovariances (covariances over different
time lags) of the ICs are then well-defined statistics. One can then use such additional
statistics to improve the estimation of the model. This additional information canactually make the estimation of the model possible in cases where the basic ICA
methods cannot estimate it, for example, if the ICs are gaussian but correlated overtime.
In this chapter, we consider the estimation of the ICA model when the ICs aretime signals, si (t); t = 1; :::; T , where t is the time index. In the previous chapters,
we denoted by t the sample index, but here t has a more precise meaning, since it
defines an order between the ICs. The model is then expressed byx

Ax(t) = As(t)(18.1)

where is assumed to be square as usual, and the ICs are of course independent. In
contrast, the ICs need not be nongaussian.
In the following, we shall make some assumptions on the time structure of the ICs
that allow for the estimation of the model. These assumptions are alternatives to the
341

342

METHODS USING TIME STRUCTURE

assumption of nongaussianity made in other chapters of this book. First, we shallassume that the ICs have different autocovariances (in particular, they are all different
from zero). Second, we shall consider the case where the variances of the ICs are
nonstationary. Finally, we discuss Kolmogoroff complexity as a general framework
for ICA with time-correlated mixtures.
We do not here consider the case where it is the mixing matrix that changes intime; see [354].

18.1
18.1.1

SEPARATION BY AUTOCOVARIANCES
Autocovariances as an alternative to nongaussianity

The simplest form of time structure is given by (linear) autocovariances. This means
covariances between the values of the signal at different time points: cov(xi (t)xi (t ?
 )) where  is some lag constant,  = 1; 2; 3; :::. If the data has time-dependencies,
the autocovariances are often different from zero.
In addition to the autocovariances of one signal, we also need covariances between
two signals: cov(xi (t)xj (t ?  )) where i 6= j . All these statistics for a given time
lag can be grouped together in the time-lagged covariance matrix

Cx = E fx(t)x(t ?  )T g

(18.2)The theory of time-dependent signals was briefly discussed in Section 2.8.
As we saw in Chapter 7, the problem in ICA is that the simple zero-lagged
covariance (or correlation) matrix does not contain enough parameters to allow theestimation of . This means that simply finding a matrix so that the componentsof the vector

C

A

V

z(t) = Vx(t)(18.3)

are white, is not enough to estimate the independent components. This is because
there is an infinity of different matrices that give decorrelated components. This
is why in basic ICA, we have to use the nongaussian structure of the independent
components, for example, by minimizing the higher-order dependencies as measured
by mutual information.
The key point here is that the information in a time-lagged covariance matrix x
could be used instead of the higher-order information [424, 303]. What we do isso that in addition to making the instantaneous covariances of
to find a matrix(t) = (t) go to zero, the lagged covariances are made zero as well:

V

C

y

Bx

B

E fyi (t)yj (t ?  )g = 0; for all i; j; 
(18.4)The motivation for this is that for the ICs si (t), the lagged covariances are all zero due

to independence. Using these lagged covariances, we get enough extra information
to estimate the model, under certain conditions specified below. No higher-orderinformation is then needed.

343

SEPARATION BY AUTOCOVARIANCES

18.1.2Using one time lag

In the simplest case, we can use just one time lag. Denote by  such a time lag, which
is very often taken equal to 1. A very simple algorithm can now be formulated to finda matrix that cancels both the instantaneous covariances and the ones corresponding
to lag  .
Consider whitened data (see Chapter 6), denoted by . Then we have for theorthogonal separating matrix :zWWz(t) = s(t)Wz(t ?  ) = s(t ?  )(18.5)(18.6)Let us consider a slightly modified version of the lagged covariance matrix as definedin (18.2), given byC z = 21 [Cz + (Cz )T ](18.7)We have by linearity and orthogonality the relationC z = 12 WT [E fs(t)s(t ?  )T g + E fs(t ?  )s(t)T g]W = WT C s W

(18.8)

Due to the independence of the si (t), the time-lagged covariance matrix s =E f (t) (t ?  )g is diagonal; let us denote it by . Clearly, the matrix  s equals
this same matrix. Thus we have

s s

C

D

C

C z = WT DW(18.9)
What this equation shows is that the matrix W is part of the eigenvalue decomposition
 z . The eigenvalue decomposition of this symmetric matrix is simple to compute.
of C

In fact, the reason why we considered this matrix instead of the simple time-lagged
covariance matrix (as in [303]) was precisely that we wanted to have a symmetricmatrix, because then the eigenvalue decomposition is well defined and simple tocompute. (It is actually true that the lagged covariance matrix is symmetric if thedata exactly follows the ICA model, but estimates of such matrices are not symmetric.)
The AMUSE algorithm Thus we have a simple algorithm, called AMUSE [424],
for estimating the separating matrix
for whitened data:W
1. Whiten the (zero-mean) data x to obtain z(t).
 z = [C + CT ], where C =2. Compute the eigenvalue decomposition of C
E fz(t)z(t ?  )g is the time-lagged covariance matrix, for some lag  .
3. The rows of the separating matrix W are given by the eigenvectors.
1
2

An essentially similar algorithm was proposed in [303].

344

METHODS USING TIME STRUCTURE

This algorithm is very simple and fast to compute. The problem is, however, thatit only works when the eigenvectors of the matrix   are uniquely defined. This is
the case if the eigenvalues are all distinct (not equal to each other). If some of theeigenvalues are equal, then the corresponding eigenvectors are not uniquely defined,
and the corresponding ICs cannot be estimated. This restricts the applicability of thismethod considerably. These eigenvalues are given by cov(si (t)si (t ?  )), and thus
the eigenvalues are distinct if and only if the lagged covariances are different for all
the ICs.
As a remedy to this restriction, one can search for a suitable time lag  so thatthe eigenvalues are distinct, but this is not always possible: If the signals si (t) haveidentical power spectra, that is, identical autocovariances, then no value of  makes
estimation possible.

C

18.1.3Extension to several time lags

An extension of the AMUSE method that improves its performance is to consider
several time lags  instead of a single one. Then, it is enough that the covariances forone of these time lags are different. Thus the choice of  is a somewhat less serious
problem.
In principle, using several time lags, we want to simultaneously diagonalize all thecorresponding lagged covariance matrices. It must be noted that the diagonalization
is not possible exactly, since the eigenvectors of the different covariance matrices
are unlikely to be identical, except in the theoretical case where the data is exactly
generated by the ICA model. So here we formulate functions that express the degreeof diagonalization obtained and find its maximum.
One simple way of measuring the diagonality of a matrix
is to use the operator

M) =

off(

X
i6=j

M

m2ij

(18.10)M

which gives the sum of squares of the off-diagonal elements . What we now
want to do is to minimize the sum of the off-diagonal elements of several lagged. As before, we use the symmetric version  y of the lagged
covariances of =covariance matrix. Denote by S the set of the chosen lags  . Then we can write thisas an objective function J ( ):

C

y Wz

w

J (W) =
1X

 2S

WC z WT )

off(

W

(18.11)Minimizing J1 under the constraint thatis orthogonal gives us the estimation
method. This minimization could be performed by (projected) gradient descent.
Another alternative is to adapt the existing methods for eigenvalue decomposition tothis simultaneous approximate diagonalization of several matrices. The algorithm
called SOBI (second-order blind identification) [43] is based on these principles, andso is TDSEP [481].

345

SEPARATION BY AUTOCOVARIANCESW

The criterion J1 can be simplified. For an orthogonal transformation, , the sum
T is constant.1 Thus, the “off” criterionof the squares of the elements ofcould be expressed as the difference of the total sum of squares minus the sum of thesquares on the diagonal. Thus we can formulate

WMW

J (W) = ?
2w

X X T z

 2S i(wi C wi )2(18.12)W

where the iT are the rows of . Thus, minimizing J2 is equivalent to minimizing
J1 .
An alternative method for measuring the diagonality can be obtained using theapproach in [240]. For any positive-definite matrix , we haveXiM
log mii  log j det Mj

and the equality holds only for diagonal
bynality ofM

F (M) =

Xi

(18.13)

M. Thus, we could measure the nondiago-

log mii ? log j det Mj(18.14)C

Again, the total nondiagonality of the  at different time lags can be measured
by the sum of these measures for different time lags. This gives us the following
objective function to minimize:X

J (W) = 213

 2S

F (C y ) = 12

X

 2S

Just as in maximum likelihood (ML) estimation,
ing the logarithm of the determinant. We obtain

J (W) =
3

F (WC z WT )(18.15)W decouples from the term involv-

XX 1

 2S iTC z wi ) ? log j det Wj ? 1 log j det C z j
log(
w
i
2
2

(18.16)W

can be constrained orthogonal, we see
Considering whitened data, in which casethat the term involving the determinant is constant, and we finally have

J (W) =3

XX 1

log(wiT C z wi ) + const.
2
 2S i(18.17)This is in fact rather similar to the function J2 in (18.12). The only difference isthat the function ?u2 has been replaced by 1=2 log(u). What these functions have

W WMM

MM

WMWT WMWT T

1 This is because it equals trace (T ) = trace(
T ).
trace ( T

(

) )

=

WMMT WT

trace (

)

=

346

METHODS USING TIME STRUCTURE

in common is concavity, so one might speculate that many other concave functions
could be used as well.
The gradient of J3 can be evaluated as@ J3 = X Q WC z@ W  2S  (18.18)withQ = diag(WC z WT )?

1(18.19)Thus we obtain the gradient descent algorithm

W /

X

 2S

Q WC z

(18.20)W

Here,should be orthogonalized after every iteration. Moreover, care must
be taken so that in the inverse in (18.19), very small entries do not cause numericalproblems. A very similar gradient descent can be obtained for (18.12), the maindifference being the scalar function in the definition of .
Thus we obtain an algorithm that estimates
based on autocorrelations withseveral time lags. This gives a simpler alternative to methods based on joint approximative diagonalization. Such an extension allows estimation of the model in somecases where the simple method using a single time lag fails. The basic limitation
cannot be avoided, however: if the ICs have identical autocovariances (i.e., identicalpower spectra), they cannot be estimated by the methods using time-lagged covariances only. This is in contrast to ICA using higher-order information, where theindependent components are allowed to have identical distributions.
Further work on using autocovariances for source separation can be found in[11, 6, 106]. In particular, the optimal weighting of different lags has be consideredin [472, 483].

W

18.2

Q

SEPARATION BY NONSTATIONARITY OF VARIANCES

An alternative approach to using the time structure of the signals was introduced in[296], where it was shown that ICA can be performed by using the nonstationarityof the signals. The nonstationarity we are using here is the nonstationarity of thevariances of the ICs. Thus the variances of the ICs are assumed to change smoothly intime. Note that this nonstationarity of the signals is independent from nongaussianityor the linear autocovariances in the sense that none of them implies or presupposes
any of the other assumptions.
To illustrate the variance nonstationarity in its purest form, let us look at the signal
in Fig. 18.1. This signal was created so that it has a gaussian marginal density,and no linear time correlations, i.e., E f (t) (t ?  )T g = 0 for any lag  . Thus,x x

SEPARATION BY NONSTATIONARITY OF VARIANCES

347

4

3

2

1

0

−1

−2

−3

−4

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Fig. 18.1 A signal with nonstationary variance.

ICs of this kind could not be separated by basic ICA methods, or using linear timecorrelations. On the other hand, the nonstationarity of the signal is clearly visible. It
is characterized by bursts of activity.
Below, we review some basic approaches to this problem. Further work can befound in [40, 370, 126, 239, 366].
18.2.1

Using local autocorrelations

Separation of nonstationary signals could be achieved by using a variant of autocorrelations, somewhat similar to the case of Section 18.1. It was shown in [296] that if(t) are uncorrelated at every
we find a matrix so that the components of (t) =time point t, we have estimated the ICs. Note that due to nonstationarity, the covariance of (t) depends on t, and thus if we force the components to be uncorrelated
for every t, we obtain a much stronger condition than simple whitening.
The (local) uncorrelatedness of (t) could be measured using the same measuresof diagonality as used in Section 18.1.3. We use here a measure based on (18.14):

B

y

Bx

y

y

Q(B; t) =Xilog Et fyi (t)2 g ? log j det Et fy(t)y(t)T gj(18.21)The subscript t in the expectation emphasizes that the signal is nonstationary, and the
expectation is the expectation around the time point t. This function is minimized by
the separating matrix .

B

348

METHODS USING TIME STRUCTURE

B = (b ; :::; bn)T we obtain
X
Q(B; t) = log Et f(bTi x(t)) g ? log j det Et fBx(t)x(t)T BT gj
Expressing this as a function of

1

2

=

X
i

i

log Et f(bTi x(t))2 g ? log j det Et fx(t)x(t)T gj ? 2 log j det Bj

x x(18.22)B

Note that the term log j det Et f (t) (t)T gj does not depend on at all. Furthermore,
to take into account all the time points, we sum the values of Q in different time
points, and obtain the objective function

J (B) =
4

Xt

Q(B; t) =Xi;tlog Et f(bTi x(t))2 g ? 2 log j det Bj + const.

(18.23)z

As usual, we can whiten the data to obtain whitened data , and force the separating
matrixto be orthogonal. Then the objective function simplifies toW

J (W) =
4

X

Q(W; t) =t

X

log Et f(wiT z(t))2 g + const.

i;t(18.24)Thus we can compute the gradient of J4 as@ J4 = 2 X diag(E f(wT z(t))2 g?1 )WE fz(t)z(t)T g:
t
t
i
@W
t

(18.25)w z

The question is now: How to estimate the local variances Et f( iT (t))2 g? Wecannot simply use the sample variances, due to nonstationarity, which leads to dependence between these variances and the (t). Instead, we have to use some local
estimates at time point t. A natural thing to do is to assume that the variance changes
slowly. Then we can estimate the local variance by local sample variances. In otherwords:z

E^t f(wiT z(t))2 g =Xh( )(wiT z(t ?  ))2

(18.26)

where h is a moving average operator (low-pass filter), normalized so that the sumof its components is one.
Thus we obtain the following algorithm:

W /

X
t

where after every iteration,diag(E^t f( iT (t))2 g?1 )wz

Wz(t)z(t)T(18.27)

W is symmetrically orthogonalized (see Chapter 6), andE^t is computed as in (18.26). Again, care must be taken that taking the inverseof very small local variances does not cause numerical problems. This is the basic
method for estimating signals with nonstationary variances. It is a simplified formof the algorithm in [296].

SEPARATION BY NONSTATIONARITY OF VARIANCES

349

300

1000

14

12

10

8

6

4

2

0

0

100

200

400

500

600

700

800

900

Fig. 18.2 The energy (i.e., squares) of the initial part of the signal in Fig. 18.1. This isclearly time-correlated.

The algorithm in (18.27) enables one to estimate the ICs using the informationon the nonstationarity of their variances. This principle is different from the ones
considered in preceding chapters and the preceding section. It was implemented byconsidering simultaneously different local autocorrelations. An alternative method
for using nonstationarity will be considered next.
18.2.2Using cross-cumulants

Nonlinear autocorrelations A second method of using nonstationarity is basedon interpreting variance nonstationarity in terms of higher-order cross-cumulants.
Thus we obtain a very simple criterion that expresses nonstationarity of variance.
To see how this works, consider the energy (i.e., squared amplitude) of the signal
in Fig. 18.1. The energies of the initial 1000 time points are shown in Fig. 18.2.
What is clearly visible is that the energies are correlated in time. This is of course a
consequence of the assumption that the variance changes smoothly in time.
Before proceeding, note that the nonstationarity of a signal depends on the timescale and the level of the detail in the model of the signal. If the nonstationarity ofthe variance is incorporated in the model (by hidden Markov models, for example),the signal no longer needs to be considered nonstationary [370]. This is the approachthat we choose in the following. In particular, the energies are not considered
nonstationary, but rather they are considered as stationary signals that are timecorrelated. This is simply a question of changing the viewpoint.
So, we could measure the variance nonstationarity of a signal y (t); t = 1; :::tusing a measure based on the time-correlation of energies: E fy (t)2 y (t ?  )2 g where is some lag constant, often equal to one. For the sake of mathematical simplicity, it
is often useful to use cumulants instead of such basic higher-order correlations. The

350

METHODS USING TIME STRUCTURE

cumulant corresponding to the correlation of energies is given by the fourth-ordercross cumulant
cum(y (t); y (t); y (t ?  ); y (t ?  ))

= E fy(t)2 y(t ?  )2 g ? E fy(t)2 gE fy(t ?  )2 g ? 2(E fy(t)y(t ?  )g)2

(18.28)This could be considered as a normalized version of the cross-correlation of energies.
In our case, where the variances are changing smoothly, this cumulant is positive
because the first term dominates the two normalizing terms.
Note that although cross-cumulants are zero for random variables with jointly
gaussian distributions, they need not be zero for variables with gaussian marginal
distributions. Thus positive cross-cumulants do not imply nongaussian marginal
distributions for the ICs, which shows that the property measured by this crosscumulant is indeed completely different from the property of nongaussianity.
The validity of this criterion can be easily proven. Consider a linear combinationof the observed signals xi (t) that are mixtures of original ICs, as in (18.1). This linear
combination, say T (t), is a linear combination of the ICs T (t) = T (t), say
T (t) =
i qi si (t). Using the basic properties of cumulants, the nonstationarityof such a linear combination can be evaluated as

P bx

qs

bx

b As

b x(t); bT x(t); bT x(t ?  ); bT x(t ?  ))cum( T=Xi

qi4 cum(si (t); si (t); si (t ?  ); si (t ?  )) (18.29)

bx

Now, we can constrain the variance of T to be equal to unity to normalize thescale (cumulants are not scale-invariant). This implies var i qi si = k k2 = 1. Let
us consider what happens if we maximize nonstationarity with respect to . This isequivalent to the optimization problemXP

q4 cum(si (t); si (t); si (t ?  ); si (t ?  ))maxkqk2 =1 i iqb(18.30)This optimization problem is formally identical to the one encountered when kurtosis (or in general, its absolute value) is maximized to find the most nongaussian
directions, as in Chapter 8. It was proven that solutions to this optimization problem
give the ICs. In other words, the maxima of (18.30) are obtained when only oneof the qi is nonzero. This proof applies directly in our case as well, and thus we
see that the maximally nonstationary linear combinations give the ICs.2 Since thecross-cumulants are assumed to be all positive, the problem we have here is in fact
slightly simpler since we can then simply maximize the cross-cumulant of the linear
combinations, and need not consider its absolute value as is done with kurtosis in
Chapter 8.
2 Note that this statement requires that we identify nonstationarity with the energy correlations, which may

or may not be meaningful depending on the context.

351

SEPARATION PRINCIPLES UNIFIED

Thus we see that maximization of the nonstationarity, as measured by the crosscumulant, of a linear combination of the observed mixtures allows for the estimationof one IC. This also gives a one-unit approach to source separation by nonstationarity.
A fixed-point algorithm To maximize the variance nonstationarity as measured
by the cross-cumulant, one can use a fixed-point algorithm derived along the samelines as the FastICA algorithm for maximizing nongaussianity.
To begin with, let us whiten the data to obtain (t). Now, using the principle of
fixed-point iteration as in Chapter 8, let us equate
to the gradient of the crosscumulant of T (t). This gives, after straightforward calculations, the following
update for :

z

wz

w

w
w E fz(t)wT z(t)(wT z(t ?  )) g + E fz(t ?  )wT z(t ?  )(wT z(t)) g? 2w ? 4C z w(wT C z w) (18.31)
22

where we have multiplied the gradient by 1=2 for notational simplicity, and the matrix

C z is equal to [E fz(t)z(t ?  )g + E fz(t)z(t ?  )g], as above. The algorithm
thus consists of iteratively computing the new value of w according to (18.31), andnormalizing w to unit norm after every step.
1
2

The convergence of the algorithm can be proven to be cubic, i.e., very fast. A
detailed proof can be constructed as with kurtosis. Let us only note here that if
the algorithm is expressed with respect to the transformed variable , which can be
simply obtained by computing the gradient of (18.29), we haveqqi

qi3 [4cum(si (t); si (t); si (t ?  ); si (t ?  ))]

(18.32)q

followed by normalization of the norm of . This can be easily seen to lead toconvergence of to a vector where only one of the qi is nonzero. The index i forwhich qi will be nonzero depends on the initial value of .
Thus we have obtained a fast fixed-point algorithm for separating ICs by nonstationarity, using cross-cumulants. This gives an alternative for the algorithm inthe preceding subsection. This algorithm is similar to the FastICA algorithm. The
convergence of the algorithm is cubic, like the convergence of the cumulant-based
FastICA. This was based on the interpretation of a particular cross-cumulant as ameasure of nonstationarity.

q

18.3
18.3.1q

SEPARATION PRINCIPLES UNIFIED
Comparison of separation principles

In this chapter we have discussed the separation of ICs (sources) using their timedependencies. In particular, we showed how to use autocorrelations and variancenonstationarities. These two principles complement the principle of nongaussianitythat was the basis of estimation in the basic ICA model in Part II.

352

METHODS USING TIME STRUCTURE

This raises the question: In what situations should one use each of these methods.
The answer is basically simple: The different criteria make different assumptions on
the data, and the choice of the criterion should be made depending on the data that
we want to analyze.
is aFirst of all, in many cases the data has no time structure at all, i.e.,
random variable, and not a time signal. This means that the order of the samples
is arbitrary and has no significance. In that case, only the basic ICA method basedon nongaussianity can be used. So the alternatives are only meaningful if the data
comes from a source that has a time structure.
If the data does have clear time-dependencies, these usually imply nonzero autocorrelation functions, and the methods based on autocorrelations can be used.
However, such methods only work if the autocorrelations are different for each IC.
In the case where some of the autocorrelations are identical, one could then try to use
the methods based on nonstationarity, since these methods utilize the time structure
but do not require the time structure of each IC to be different. The nonstationarity methods work best, of course, if the time structure does consist of a changing
variance, as with the signal depicted in Fig. 18.1.
On the other hand, the basic ICA methods often work well even when the ICs havetime-dependencies. The basic methods do not use the time structure, but this does
not mean that they would be disturbed by such structure. It must be noted, though,that the basic ICA methods may then be very far from optimal, since they do not use
the whole structure of the data.

x

18.3.2Kolmogoroff complexity as a unifying framework

It is also possible to combine different types of information. For example, methods
that combine nongaussianity with autocorrelations were proposed in [202, 312]. Whatis interesting is that one can formulate a general framework that encompasses all thesedifferent principles. This framework that includes basic ICA and methods using time
structure was proposed by Pajunen [342, 343], based on the information-theoretic
concept of Kolmogoroff complexity.
As has been argued in Chapters 8 and 10, ICA can be seen as a method of findinga transformation into components that are somehow structured. It was argued that
nongaussianity is a measure of structure. Nongaussianity can be measured by the
information-theoretic concept of entropy.
Entropy of a random variable measures the structure of its marginal distribution
only. On the other hand, in this section we have been dealing with time signals thathave different kinds of time structure, like autocorrelations and nonstationarity. How
could one measure such a more general type of structure using information-theoretic
criteria? The answer lies in Kolmogoroff complexity. One can define a very generalform of linear ICA by finding projections of the data that have low complexity in thissense. First, let us look at the definition of Kolmogoroff complexity.
Definition of Kolmogoroff complexity The information-theoretic measuresof structure are based on the interpretation of coding length as structure. Suppose

SEPARATION PRINCIPLES UNIFIED

353

that we want to code a signal s(t); t = 1; :::; T . For simplicity, let us assume thatthe signal is binary, so that every value s(t) is 0 or 1. We can code such a signal
so that every bit in the code gives the value of s(t) for one t; this is the obvious
and trivial code. In general, it is not possible to code this signal with less that T
bits. However, most natural signals have redundancy, i.e., parts of the signal can be
efficiently predicted from other parts. Such a signal can be coded, or compressed,
so that the code length is shorter than the original signal length T . It is well known
that audio or image signals, for example, can be coded so that the code length isdecreased considerably. This is because such natural signals are highly structured.
For example, image signals do not consist of random pixels, but of such higher-orderregularities as edges, contours, and areas of constant color [154].
We could thus measure the amount of structure of the signal s(t) by the amount ofcompression that is possible in coding the signal. For signals of fixed length T , thestructure could be measured by the length of the shortest possible code for the signal.
Note that the signal could be compressed by many different kinds of coding schemes
developed in the coding theory literature, but we are here considering the shortest
possible code, thus maximizing the compression over all possible coding schemes.
For a more rigorous definition of the concept, see [342, 343].
ICA and Kolmogoroff complexity We can now define a generalization of ICA
as follows: Find the transformation of the data where the sum of the coding lengthsof the components is as short as possible. However, an additional operation is needed
here. We also need to consider the code length of the transformation itself [342, 343].
This leads to a framework that is closely related to the minimum description length(MLD) principle [380, 381]. Thus the objective function we are using can be written
as follows:

J (W) = T1

Xi

K (bTi x) ? log j det Wj(18.33)where K (:) denotes the complexity. The latter term is related to the coding length of
the transformation, using the MDL principle.
This objective function can be considered as a generalization of mutual information. If the signals have no time structure, their Kolmogoroff complexities are givenby their entropies, and thus we have in (18.33) the definition of mutual information.
Furthermore, in [344] it was shown how to approximate K (:) by criteria using thetime structure of the signals, which proves that Kolmogoroff complexity gives a generalization of methods using time correlations as well. (More on this connection canbe found in Section 23.3.)
Kolmogoroff complexity is a rather theoretical measure, since its computation
involves finding the best coding scheme for the signal. The number of possible
coding schemes is infinite, so this optimization cannot be performed in practice withmuch accuracy. In some special cases of Kolmogoroff complexity the optimization
can be performed rather accurately however. These include the above-mentioned
cases of mutual information and time-correlations.

354

METHODS USING TIME STRUCTURE

18.4

CONCLUDING REMARKS

In addition to the fundamental assumption of independence, another assumption thatassures that the signals have enough structure is needed for successful separation of
signals. This is because the ICA model cannot be estimated for gaussian randomvariables. In basic ICA, which was treated in Part II, the assumption was the
nongaussianity of the ICs. In this chapter, we used the alternative assumption that
the ICs are time signals that have some time dependencies. Here we have at least
two cases in which separation is possible. First, the case where the signals havedifferent power spectra, i.e., different autocovariance functions. Second, the casewhere they have nonstationary variances. All these assumptions can be consideredin the unifying framework of Kolmogoroff complexity: They can all be derived as
special cases of complexity minimization.

19
Convolutive Mixtures andBlind Deconvolution
This chapter deals with blind deconvolution and blind separation of convolutive
mixtures.
Blind deconvolution is a signal processing problem that is closely related to basic
independent component analysis (ICA) and blind source separation (BSS). In communications and related areas, blind deconvolution is often called blind equalization.
In blind deconvolution, we have only one observed signal (output) and one source
signal (input). The observed signal consists of an unknown source signal mixed withitself at different time delays. The task is to estimate the source signal from theobserved signal only, without knowing the convolving system, the time delays, andmixing coefficients.
Blind separation of convolutive mixtures considers the combined blind deconvolution and instantaneous blind source separation problem. This estimation task appears
under many different names in the literature: ICA with convolutive mixtures, multichannel blind deconvolution or identification, convolutive signal separation, andblind identification of multiple-input-multiple-output (MIMO) systems. In blindseparation of convolutive mixtures, there are several source (input) signals and several observed (output) signals just like in the instantaneous ICA problem. However,the source signals have different time delays in each observed signal due to the finite
propagation speed in the medium. Each observed signal may also contain timedelayed versions of the same source due to multipath propagation caused typicallyby reverberations from some obstacles. Figure 23.3 in Chapter 23 shows an example
of multipath propagation in mobile communications.
In the following, we first consider the simpler blind deconvolution problem, andafter that separation of convolutive mixtures. Many techniques for convolutive mix355

356

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

tures have in fact been developed by extending methods designed originally for either
the blind deconvolution or standard ICA/BSS problems. In the appendix, certain basic
concepts of discrete-time filters needed in this chapter are briefly reviewed.

19.1
19.1.1BLIND DECONVOLUTION
Problem definition

In blind deconvolution [170, 171, 174, 315], it is assumed that the observed discretetime signal x(t) is generated from an unknown source signal s(t) by the convolution
model

x(t) =

1X

k=?1

ak s(t ? k)

(19.1)

Thus, delayed versions of the source signal are mixed together. This situation appears
in many practical applications, for example, in communications and geophysics.
In blind deconvolution, both the source signal s(t) and the convolution coefficientsak are unknown. Observing x(t) only, we want to estimate the source signal s(t). In
other words, we want to find a deconvolution filtery(t) =

1X
k=?1

hk x(t ? k)(19.2)

which provides a good estimate of the source signal s(t) at each time instant. Thisis achieved by choosing the coefficients hk of the deconvolution filter suitably. In
practice the deconvolving finite impulse response (FIR) filter (see the Appendix
for definition) in Eq. (19.2) is assumed to be of sufficient but finite length. Other
structures are possible, but this one is the standard choice.
To estimate the deconvolving filter, certain assumptions on the source signal s(t)must be made. Usually it is assumed that the source signal values s(t) at different
times t are nongaussian, statistically independent and identically distributed (i.i.d.).
The probability distribution of the source signal s(t) may be known or unknown. The
indeterminacies remaining in the blind deconvolution problem are that the estimatedsource signal may have an arbitrary scaling (and sign) and time shift comparedwith the true one. This situation is similar to the permutation and sign indeterminacy
encountered in ICA; the two models are, in fact, intimately related as will be explainedin Section 19.1.4.
Of course, the preceding ideal model usually does not exactly hold in practice.
There is often additive noise present, though we have omitted noise from the model(19.1) for simplicity. The source signal sequence may not satisfy the i.i.d condition,and its distribution is often unknown, or we may only know that the source signal issubgaussian or supergaussian. Hence blind deconvolution often is a difficult signal
processing task that can be solved only approximately, in practice.

BLIND DECONVOLUTION

357

If the linear time-invariant system (19.1) is minimum phase (see the Appendix),then the blind deconvolution problem can be solved in a straightforward way. On theabove assumptions, the deconvolving filter is simply a whitening filter that temporally
whitens the observed signal sequence fx(t)g [171, 174]. However, in many applications, for example, in telecommunications, the system is typically nonminimum
phase [174] and this simple solution cannot be used.
We shall next discuss some popular approaches to blind deconvolution. Blinddeconvolution is frequently needed in communications applications where it is convenient to use complex-valued data. Therefore we present most methods for thisgeneral case. The respective algorithms for real data are obtained as special cases.
Methods for estimating the ICA model with complex-valued data are discussed laterin Section 20.3.
19.1.2

Bussgang methods

Bussgang methods [39, 171, 174, 315] include some of the earliest algorithms [152,392] proposed for blind deconvolution, but they are still widely used. In Bussgang
methods, a noncausal FIR filter structurey(t) =

LX
k=?L

wk (t)x(t ? k)(19.3)of length 2L +1 is used. Here  denotes the complex conjugate. The weights wk (t) of
the FIR filter depend on the time t, and they are adapted using the least-mean-square(LMS) type algorithm [171]wk (t + 1) = wk (t) + x(t ? k)e(t);k = ?L; : : : ; L(19.4)

where the error signal is defined bye(t) = g(y(t)) ? y(t)(19.5)In these equations,  is a positive learning parameter, y (t) is given by (19.3), and g (:)
is a suitably chosen nonlinearity. It is applied separately to the real and imaginary
components of y (t). The algorithm is initialized by setting w0 (0) = 1, wk (0) = 0;
k 6= 0.
Assume that the filter length 2L + 1 is large enough and the learning algorithm
has converged. It can be shown that then the following condition holds for the output
y(t) of the FIR filter (19.3):
Efy (t)y (t ? k )g  Efy (t)g (y (t ? k ))g(19.6)

A stochastic process that satisfies the condition (19.6) is called a Bussgang process.
The nonlinearity g (t) can be chosen in several ways, leading to different Bussgang
type algorithms [39, 171]. The Godard algorithm [152] is the best performing
Bussgang algorithm in the sense that it is robust and has the smallest mean-square

358

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

error after convergence; see [171] for details. The Godard algorithm minimizes thenonconvex cost function

Jp (t) = Ef[jy(t)jp ? p ] g2(19.7)

where p is a positive integer and p is a positive real constant defined by the statisticsof the source signal:
Efjs(t)j2p g
p = Efjs(t)jp g(19.8)

The constant p is chosen in such a way that the gradient of the cost function Jp (t)is zero when perfect deconvolution is attained, that is, when y (t) = s(t). The error
signal (19.5) in the gradient algorithm (19.4) for minimizing the cost function (19.7)
with respect to the weight wk (t) has the forme(t) = y(t)jy(t)jp?2 [ p ? jy(t)jp ](19.9)

In computing e(t), the expectation in (19.7) has been omitted for getting a simpler
stochastic gradient type algorithm. The respective nonlinearity g (y (t)) is given by[171]g(y(t)) = y(t) + y(t)jy(t)jp?2 [ p ? jy(t)jp ](19.10)Among the family of Godard algorithms, the so-called constant modulus algorithm(CMA) is widely used. It is obtained by setting p = 2 in the above formulas. The
cost function (19.7) is then related to the minimization of the kurtosis. The CMA andmore generally Godard algorithms perform appropriately for subgaussian sources
only, but in communications applications the source signals are subgaussian.1. The
CMA algorithm is the most successful blind equalization (deconvolution) algorithm
used in communications due to its low complexity, good performance, and robustness[315].
Properties of the CMA cost function and algorithm have been studied thoroughlyin [224]. The constant modulus property possessed by many types of communications
signals has been exploited also in developing efficient algebraic blind equalizationand source separation algorithms [441]. A good general review of Bussgang typeblind deconvolution methods is [39].
19.1.3Cumulant-based methods

Another popular group of blind deconvolution methods consists of cumulant-based
approaches [315, 170, 174, 171]. They apply explicitly higher-order statistics ofthe observed signal x(t), while in the Bussgang methods higher-order statistics
1 The CMA algorithm can be applied to blind deconvolution of supergaussian sources by using a negative

learning parameter  in (19.4); see [11]

BLIND DECONVOLUTION

359

are involved into the estimation process implicitly via the nonlinear function g ().
Cumulants have been defined and discussed briefly in Chapter 2.
Shalvi and Weinstein [398] have derived necessary and sufficient conditions anda set of cumulant-based criteria for blind deconvolution. In particular, they introduced a stochastic gradient algorithm for maximizing a constrained kurtosis based
criterion. We shall next describe this algorithm briefly, because it is computationallysimple, converges globally, and can be applied equally well to both subgaussian andsupergaussian source signals s(t).
Assume that the source (input) signal s(t) is complex-valued and symmetric,
satisfying the condition Efs(t)2 g = 0. Assume that the length of the causal FIR
deconvolution filter is M . The output z (t) of this filter at discrete time t can then be
expressed compactly as the inner productz (t) = wT (t)y(t)where the M -dimensional filter weight vector
are respectively defined by(19.11)w(t) and output vector y(t) at time ty(t) = [y(t); y(t ? 1); : : : ; y(t ? M + 1)]T
w(t) = [w(t); w(t ? 1); : : : ; w(t ? M + 1)]T

(19.12)(19.13)

Shalvi and Weinstein’s algorithm is then given by [398, 351]

u(t + 1) = u(t) + sign(s )[jz(t)j z(t)]y (t)(19.14)w(t + 1) = u(t + 1)= k u(t + 1) k
Here s is the kurtosis of s(t), k  k is the usual Euclidean norm, and the unnormalized
filter weight vector u(t) is defined quite similarly as w(t) in (19.13).
2

It is important to notice that Shalvi and Weinstein’s algorithm (19.14) requires
whitening of the output signal y (t) for performing appropriately (assuming that s(t)is white, too). For a single complex-valued signal sequence (time series) fy (t)g, thetemporal whiteness condition is
Efy (t)y  (t ? k )g = y2 tk =(y2 ; t = k0; t 6= k(19.15)where the variance of y (t) is often normalized to unity: y2 = 1. Temporal whitening
can be achieved by spectral prewhitening in the Fourier domain, or by using timedomain techniques such as linear prediction [351]. Linear prediction techniques havebeen discussed for example in the books [169, 171, 419].
Shalvi and Weinstein have presented a somewhat more complicated algorithm
for the case Efs(t)2 g 6= 0 in [398]. Furthermore, they showed that there exists a
close relationship between their algorithm and the CMA algorithm discussed in theprevious subsection; see also [351]. Later, they derived fast converging but more
involved super-exponential algorithms for blind deconvolution in [399]. Shalvi and360

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

Weinstein have reviewed their blind deconvolution methods in [170]. Closely related
algorithms were proposed earlier in [114, 457].
It is interesting to note that Shalvi and Weinstein’s algorithm (19.14) can bederived by maximizing the absolute value of the kurtosis of the filtered (deconvolved)
signal z (t) under the constraint that the output signal y (t) is temporally white [398,351]. The temporal whiteness condition leads to the normalization constraint of
the weight vector (t) in (19.14). The corresponding criterion for standard ICA isfamiliar already from Chapter 8, where gradient algorithms similar to (19.14) havebeen discussed. Also Shalvi and Weinstein’s super-exponential algorithm [399] isvery similar to the cumulant-based FastICA as introduced in Section 8.2.3. The
connection between blind deconvolution and ICA is discussed in more detail in thenext subsection.
Instead of cumulants, one can resort to higher-order spectra or polyspectra [319,318]. They are defined as Fourier transforms of the cumulants quite similarly as
the power spectrum is defined as a Fourier transform of the autocorrelation function(see Section 2.8.5). Polyspectra provide a basis for blind deconvolution and moregenerally identification of nonminimum-phase systems, because they preserve phase
information of the observed signal. However, blind deconvolution methods basedon higher-order spectra tend to be computationally more complex than Bussgang
methods, and converge slowly [171]. Therefore, we shall not discuss them here. Theinterested reader can find more information on those methods in [170, 171, 315].

w

19.1.4Blind deconvolution using linear ICA

In defining the blind deconvolution problem, the values of the original signal s(t)were assumed to be independent for different t and nongaussian. Therefore, the blind
deconvolution problem is formally closely related to the standard ICA problem. In
fact, one can define a vector~s(t) = [s(t); s(t ? 1); :::; s(t ? n + 1)]T

(19.16)by collecting n last values of the source signal, and similarly definexsx~(t) = [x(t); x(t ? 1); :::; x(t ? n + 1)]T

(19.17)

Then ~ and ~ are n-dimensional vectors, and the convolution (19.1) can be expressed
for a finite number of values of the summation index k asA

x~ = A~s(19.18)where is a matrix that contains the coefficients ak of the convolution filter as its
rows, at different positions for each row. This is the classic matrix representationof a filter. This representation is not exact near the top and bottom rows, but for a
sufficiently large n, it is good enough in practice.
From (19.18) we see that the blind deconvolution problem is actually (approximately) a special case of ICA. The components of are independent, and the mixing
is linear, so we get the standard linear ICA model.

sBLIND SEPARATION OF CONVOLUTIVE MIXTURES

361

In fact, the one-unit (deflationary) ICA algorithms in Chapter 8 can be directly
used to perform blind deconvolution. As defined above, the inputs (t) should thenconsist of sample sequences x(t); x(t ? 1); :::; x(t ? n + 1) of the signal x(t) to be
deconvolved. Estimating just one “independent component”, we obtain the original
deconvolved signal s(t). If several components are estimated, they correspondto translated versions of the original signal, so it is enough to estimate just one
component.

x

19.2
19.2.1BLIND SEPARATION OF CONVOLUTIVE MIXTURES
The convolutive BSS problem

In several practical applications of ICA, some kind of convolution takes place simultaneously with the linear mixing. For example, in the classic cocktail-party problem,or separation of speech signals recorded by a set of microphones, the speech signals
do not arrive in the microphones at the same time. This is because the sound travels in
the atmosphere with a very limited speed. Moreover, the microphones usually record
echos of the speakers’ voices caused by reverberations from the walls of the roomor other obstacles. These two phenomena can be modeled in terms of convolutive
mixtures. Here we have not considered noise and other complications that often
appear in practice; see Section 24.2 and [429, 430].
Blind source separation of convolutive mixtures is basically a combination ofstandard instantaneous linear blind source separation and blind deconvolution. In the
convolutive mixture model, each element of the mixing matrix in the model (t)=(t) is a filter instead of a scalar. Written out for each mixture, the data model
for convolutive mixtures is given byAAs

xi (t) =n X
X
j =1 kaikj sj (t ? k); for i = 1; :::; nx(19.19)This is a FIR filter model, where each FIR filter (for fixed indices i and j ) is defined by
the coefficients aikj . Usually these coefficients are assumed to be time-independent
constants, and the number of terms over which the convolution index k runs is finite.
Again, we observe only the mixtures xi (t), and both the independent source signalssi (t) and all the coefficients aikj must be estimated.
To invert the convolutive mixtures (19.19), a set of similar FIR filters is typicallyused:yi (t) =n X
X
j =1 kwikj xj (t ? k); for i = 1; :::; n(19.20)The output signals y1 (t); : : : ; yn (t) of the separating system are estimates of thesource signals s1 (t); : : : ; sn (t) at discrete time t. The wikj give the coefficients of
the FIR filters of the separating system. The FIR filters used in separation can be

362

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

either causal or noncausal depending on the method. The number of coefficient ineach separating filter must sometimes be very large (hundreds or even thousands)
for achieving sufficient inversion accuracy. Instead of the feedforward FIR structure,
feedback (IIR type) filters have sometimes been used for separating convolutive
mixtures, an example is presented in Section 23.4. See [430] for a discussion ofmutual advantages and drawbacks of these filter structures in convolutive BSS.
At this point, it is useful to discuss relationships between the convolutive BSS
problem and the standard ICA problem on a general level [430]. Recall first than instandard linear ICA and BSS, the indeterminacies are the scaling and the order of theestimated independent components or sources (and their sign, which can be includedin scaling). With convolutive mixtures the indeterminacies are more severe: the orderof the estimated sources yi (t) is still arbitrary, but scaling is replaced by (arbitrary)filtering. In practice, many of the methods proposed for convolutive mixtures filter
the estimated sources yi (t) so that they are temporally uncorrelated (white). This
follows from the strong independence condition that most of the blind separation
methods introduced for convolutive mixtures try to realize as well as possible. The
temporal whitening effect causes some inevitable distortion if the original source
signals themselves are not temporally white. Sometimes it is possible to get rid of
this by using a feedback filter structure; see [430].
Denote byy(t) = [y (t); y (t); : : : ; yn(t)]T1

2(19.21)the vector of estimated source signals. They are both temporally and spatially whiteif
Efy(t)yH (t ? k )g = tk I =(I; t = k0; t 6= k(19.22)where H denotes complex conjugate transpose (Hermitian operator). The standard
spatial whitening condition Ef (t) H (t)g = is obtained as a special case whent = k. The condition (19.22) is required to hold for all the lag values k for which theseparating filters (19.20) are defined. Douglas and Cichocki have introduced a simple
adaptive algorithm for whitening convolutive mixtures in [120]. Lambert and Nikias
have given an efficient temporal whitening method based on FIR matrix algebra andFourier transforms in [257].
Standard ICA makes use of spatial statistics of the mixtures to learn a spatial blindseparation system. In general, higher-order spatial statistics are needed for achieving
this goal. However, if the source signals are temporally correlated, second-order
spatiotemporal statistics are sufficient for blind separation under some conditions,as shown in [424] and discussed in Chapter 18. In contrast, blind separation of
convolutive mixtures must utilize spatiotemporal statistics of the mixtures to learn aspatiotemporal separation system.
Stationarity of the sources has a decisive role in separating convolutive mixtures,too. If the sources have nonstationary variances, second-order spatiotemporal statistics are enough as briefly discussed in [359, 456].

y y

I

BLIND SEPARATION OF CONVOLUTIVE MIXTURES

363

For convolutive mixtures, stationary sources require higher than second-order
statistics, just as basic ICA, but the following simplification is possible [430]. Spatiotemporal second-order statistics can be used to decorrelate the mixtures. This step
returns the problem to that of conventional ICA, which again requires higher-order
spatial statistics. Examples of such approaches are can be found in [78, 108, 156].
This simplification is not very widely used, however.
Alternatively, one can resort to higher-order spatiotemporal statistics from the
beginning for sources that cannot be assumed nonstationary. This approach has been
adopted in many papers, and it will be discussed briefly later in this chapter.
19.2.2

Reformulation as ordinary ICA

The simplest approach to blind separation of convolutive mixtures is to reformulate
the problem using the standard linear ICA model. This is possible because blind
deconvolution can be formulated as a special case of ICA, as we saw in (19.18).
Define now a vector ~ by concatenating M time-delayed versions of every source
signal:s~s(t) = [s1 (t); s1 (t ? 1); :::; s1(t ? M + 1); s2 (t); s2 (t ? 1); :::; s2 (t ? M + 1);: : : ; sn (t); sn (t ? 1); :::; sn(t ? M + 1)]T (19.23)
and define similarly a vector

x~(t) = [x (t); x (t ? 1); :::; x (t ? M + 1); x (t); x (t ? 1); :::; x (t ? M + 1);: : : ; xn (t); xn (t ? 1); :::; xn (t ? M + 1)]T (19.24)
1

1

1

2

2

2

Using these definitions, the convolutive mixing model (19.19) can be written

x~ = A~ ~s(19.25)where ~ is a matrix containing the coefficients aikj of the FIR filters in a suitableorder. Now one can estimate the convolutive BSS model by applying ordinary ICA
methods to the standard linear ICA model (19.25).
Deflationary estimation is treated in [108, 401, 432]. These methods are based on
finding maxima of the absolute value of kurtosis, thus generalizing the kurtosis-based
methods of Chapter 8. Other examples of approaches in which the convolutive BSS
problem has been solved using conventional ICA can be found in [156, 292].
A problem with the formulation (19.25) is that when the original data vector
is expanded to ~ , its dimension grows very much. The number M of time delays
that needs to be taken into account depends on the application, but it is often tens
or hundreds, and the dimension of model (19.25) grows with the same factor, tonM . This may lead to prohibitively high dimensions. Therefore, depending on theapplication and the dimensions n and M , this reformulation can solve the convolutive
BSS problem satisfactorily, or not.
In blind deconvolution, this is not such a big problem because we have just one
signal to begin with, and we only need to estimate one independent component, which

A

x

x

364

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

is easier than estimating all of them. In convolutive BSS, however, we often needto estimate all the independent components, and their number is nM in the model(19.25). Thus the computations may be very burdensome, and the number of data
points needed to estimate such a large number of parameters can be prohibitive inpractical applications. This is especially true if we want to estimate the separating
system adaptively, trying to track changes in the mixing system. Estimation should
then be fast both in terms of computations and data collection time.
Regrettably, these remarks hold largely for other approaches proposed for blindseparation of convolutive mixtures, too. A fundamental reason of the computationaldifficulties encountered with convolutive mixtures is the fact that the number of theunknown parameters in the model (19.19) is so large. If the filters have length M , it is
M -fold compared with the respective instantaneous ICA model. This basic problem
cannot be avoided in any way.
19.2.3Natural gradient methods

In Chapter 9, the well-known Bell-Sejnowski and natural gradient algorithms were
derived from the maximum likelihood principle. This principle was shown to be quite
closely related to the maximization of the output entropy, which is often called theinformation maximization (infomax) principle; see Chapter 9. These ICA estimation
criteria and algorithms can be extended to convolutive mixtures in a straightforward
way. Early results and derivations of algorithms can be found in [13, 79, 121, 268,
363, 426, 427]. An application to CDMA communication signals will be describedlater in Chapter 23.
Amari, Cichocki, and Douglas presented an elegant and systematic approach forderiving natural gradient type algorithms for blind separation of convolutive mixturesand related tasks. It is based on algebraic equivalences and their nice properties. Theirwork has been summarized in [11], where rather general natural gradient learning
rules have been given for complex-valued data both in the time domain and z transform domain. The derived natural gradient rules can be implemented in eitherbatch, on-line, or block on-line forms [11]. In the batch form, one can use a noncausal
FIR filter structure, while the on-line algorithms require the filters to be causal.
In the following, we represent an efficient natural gradient type algorithm [10, 13]
described also in [430] for blind separation of convolutive mixtures. It can be
implemented on-line using a feedforward (FIR) filter structure in the time domain.
The algorithm is given for complex-valued data.
The separating filters are represented as a sequence of coefficient matrices k (t)at discrete time t and lag (delay) k . The separated output with this notation andcausal FIR filters isWy(t) =xLX
k=0

Wk (t)x(t ? k)(19.26)

Here (t ? k ) is n-dimensional data vector containing the values of the n mixtures(19.19) at the time instant t ? k , and (t) is the output vector whose components areyBLIND SEPARATION OF CONVOLUTIVE MIXTURES

365

yestimates of the source signals si (t); i = 1; : : : ; m. Hence (t) has m components,
with m  n.
This matrix notation allows the derivation of a separation algorithm using thenatural gradient approach. The resulting weight matrix update algorithm, whichtakes into account the causal approximation of a doubly infinite filter by delaying theoutput by L samples, is as follows [13, 430]:

Wk (t) / Wk (t) ? g(y(t ? L))vH (t ? k); k = 0; : : : ; L(19.27)g

Quite similarly as in Chapter 9, each component of the vector applies the nonlinearity gi (:) to the respective component of the argument vector. The optimal nonlinearitygi (:) is the negative score function gi = p0i =pi of the distribution pi of the sourcesi . In (19.27), (t) is reverse-filtered output computed using the L latest samples
backwards from the current sample:v

v(t) =v

L
X
q=0

WLH?q (t)y(t ? q)

(19.28)The vector needs to be stored for the latest L samples to compute the update k (t) of the weight matrix k (t) for all lags l = 0; : : : ; L. The algorithm has
rather modest computational and memory requirements.
Note that if L = 0, the formulas (19.27) and (19.28) reduce to the standard natural
gradient algorithm. In [13], the authors present a speech separation experiment where
about 50 seconds of mixed data were needed to achieve about 10-15 dB enhancementin the quality of separated signals.

W

19.2.4W

Fourier transform methods

Fourier transform techniques are useful in dealing with convolutive mixtures, because
convolutions become products between Fourier transforms in the frequency domain.
It was shown in Chapter 13 that filtering the data is allowed before performing
ICA, since filtering does not change the mixing matrix. Using the same proof, one
can see that applying Fourier transform to the data does not change the mixing matrix
either. Thus we can apply Fourier transform to both sides of Eq. (19.19). Denoting
by Xi (! ), Si (! ), and Aij (! ) the Fourier transforms of xi (t), si (t), and aij (t),respectively, we obtain

Xi (!) =nX
j =1

Aij (!)Si (!); for i = 1; :::; n(19.29)This shows that the convolutive mixture model (19.19) is transformed into a instantaneous linear ICA model in the frequency domain. The price that we have to pay
for this is that the mixing matrix is now a function of the angular frequency ! whilein the standard ICA/BSS problem it is constant.

366

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

To utilize standard ICA in practice in the Fourier domain, one can take short-time
Fourier transforms of the data, instead of the global transform. This means thatthe data is windowed, usually by a smooth windowing function such as a gaussianenvelope, and the Fourier transform is applied separately to each data window. The
dependency of Xi (! ) on ! can be simplified by dividing the values of ! into acertain number of frequency bins (intervals). For every frequency bin, we have then anumber of observations of Xi (! ), and we can estimate the ICA model separately foreach frequency bin. Note that the ICs and the mixing matrix are now complex-valued.
See Section 20.3 on how to estimate the ICA model with complex-valued data.
The problem with this Fourier approach is the indeterminacy of permutation andsign that is ubiquitous in ICA. The permutation and signs of the sources are usuallydifferent in each frequency interval. For reconstructing a source signal si (t) in thetime domain, we need all its frequency components. Hence we a need some method
for choosing which source signals in different frequency intervals belong together.
To this end, various continuity criteria have been introduced by many authors; see[15, 59, 216, 356, 397, 405, 406, 430].
Another major group of Fourier methods developed for convolutive mixtures
avoids the preceding problem by performing the actual separation in the time domain.
Only selected parts of the separation procedure are carried out in the frequency
domain. Separating filters may be easier to learn in the frequency domain because
components are now orthogonal and do not depend on each other like the time domain
coefficients [21, 430]. Examples of methods that apply their separation criterion in
the time domain but do the rest in the frequency domain are reported in [21, 257].
A frequency domain representation of the filters is learned, and they are also appliedin the frequency domain. The final time-domain result is reconstructed using forexample the overlap-save technique of digital signal processing (see [339]). Thus,
the permutation and scaling problem does not exist.
The work by Lambert and Nikias deserves special attention, see the review in[257]. They have introduced methods that utilize the Bussgang family of cost functions and standard adaptive filtering algorithms in blind separation of convolutive
mixtures. FIR matrix algebra introduced in [256] is employed as an efficient tool forsystematic development of methods. Lambert and Nikias [257] have considered three
general classes of Bussgang type cost functions, namely blind least mean-squares(LMS), Infomax, and direct Bussgang costs. Most of these costs can be implemented
in either the time or frequency domain, or in the batch or continuously adaptivemodes. Lambert and Nikias have introduced several efficient and practical algorithms for blind separation of convolutive mixtures having different computational
complexities and convergence speeds. For example, block-oriented frequency domain implementations can be used to perform robust blind separation on convolutive
mixtures which have hundreds or thousands of time delays [257].

367

BLIND SEPARATION OF CONVOLUTIVE MIXTURES

19.2.5

Spatiotemporal decorrelation methods

Consider first the noisy instantaneous linear ICA modelx(t) = As(t) + n(t)(19.30)which has been discussed in more detail in Chapter 15. Making the standard realistic
assumption that the additive noise (t) is independent of the source signals (t), thespatial covariance matrix x (t) of (t) at time t is

C

C

n
C
x
Cx(t) = ACs(t)AT + Cn(t)s(19.31)

where s (t) and n (t) are respectively the covariance matrices of the sources
and the noise at time t. If the sources (t) are nonstationary with respect to theircovariances, then in general s (t) 6= s (t +  ) for  6= 0. This allows to write
multiple conditions for different choices of  to solve for , s (t), and n (t).
Note that the covariances matrices s (t) and n (t) are diagonal. The diagonality ofs (t) follows from the independence of the sources, and n (t) can be taken diagonal
because the components of the noise vector (t) are assumed to be uncorrelated.
We can also look at cross-covariance matrices x (t; t +  ) = Ef (t) (t +  )T g
over time. This approach has been mentioned in the context of convolutive mixturesin [456], and it can be used with instantaneous mixtures as described in Chapter 18.
For convolutive mixtures, we can write in frequency domain for sample averages[359, 356]

s
C

C

C

C

C
n

A C

C

C

C

x x

(19.32)C x(!; t) = A(!)Cs(!; t)AH (!) + Cn(!; t) x is the averaged spatial covariance matrix. If s is nonstationary, one canwhere Cagain write multiple linearly independent equations for different time lags and solve
for unknowns or find LMS estimates of them by diagonalizing a number of matrices
in the frequency domain [123, 359, 356].
If the mixing system is minimum phase, decorrelation alone can provide a uniquesolution, and the nonstationarity of the signals is not needed [55, 280, 402]. Many
methods have been proposed for this case, for example, in [113, 120, 149, 281,
280, 296, 389, 390, 456]. More references are given in [430]. However, suchdecorrelating methods cannot necessarily be applied to practical communications
and audio separation problems, because the mixtures encountered there are often notminimum-phase. For example in the cocktail-party problem the system is minimum
phase if each speaker is closest to his or her “own” microphone, otherwise not [430].
19.2.6

Other methods for convolutive mixtures

Many methods proposed for blind separation of convolutive mixtures are extensionsof earlier methods originally designed for either the standard linear instantaneous BSS(ICA) problem or for the blind deconvolution problem. We have already discussed
some extensions of the natural gradient method in Section 19.2.3 and Bussgang
methods in Section 19.2.4. Bussgang methods have been generalized for convolutive

368

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

mixtures also in [351]. Matsuoka’s method [296] for BSS of nonstationary sources is
modified for convolutive mixtures in [239] using natural gradient learning. Nguyen
Thi and Jutten [420] have generalized the seminal Hérault-Jutten algorithm describedin Chapter 12 to BSS of convolutive mixtures. Their approach has also been studied
in [74, 101]. A state-space approach for blind separation of convolutive mixtures has
been studied in [479].
There exist numerous approaches to convolutive BSS which are based on criteria utilizing directly spatiotemporal higher-order statistics. Methods based on the
maximization of the sum of the squares of the kurtoses to estimate the whole separating system were introduced in [90], and further developed in [307]. Other
methods based on spatiotemporal higher-order statistics have been presented in[1, 124, 145, 155, 218, 217, 400, 416, 422, 434, 433, 470, 471, 474]. More references can be found in [91, 430].

19.3

CONCLUDING REMARKS

Historically, many ideas used in ICA were originally developed in the context ofblind deconvolution, which is an older topic of research than ICA. Later, it was
found that many methods developed for blind deconvolution can be directly applied
for ICA, and vice versa. Blind deconvolution can thus be considered an intellectual
ancestor of ICA. For example, Donoho proposed in [114] that the deconvolution filter(19.2) could be found by finding the filter whose output is maximally nongaussian.
This is the same principle as used for ICA in Chapter 8. Douglas and Haykin have
explored relationships between blind deconvolution and blind source separation in[122]. Elsewhere, it has been pointed out that Bussgang criteria are closely related
to nonlinear PCA criteria [236] and several other ICA methods [11].
In this chapter, we have briefly discussed Bussgang, cumulant, and ICA based
methods for blind deconvolution. Still one prominent class of blind deconvolution
and separation methods for convolutive mixtures consists of subspace approaches[143, 171, 311, 315, 425]. They can be used only if the number of output signals(observed mixtures) strictly exceeds the number of sources. Subspace methods
resort to second-order statistics and fractional sampling, and they are applicable tocyclostationary source signals which are commonplace in communications [91].
General references on blind deconvolution are [170, 171, 174, 315]. Blind deconvolution and separation methods for convolutive mixtures have often been developedin context with blind channel estimation and identification problems in communications. These topics are beyond the scope of our book, but the interested reader canfind useful review chapters on blind methods in communications in [143, 144].
In the second half of this chapter, we have considered separation of convolutive
mixtures. The mixing process then takes place both temporally and spatially, which
complicates the blind separation problem considerably. Numerous methods forhandling this problem have been proposed, but it is somewhat difficult to assess
their usefulness, because comparison studies are still lacking. The large number ofparameters is a problem, making it difficult to apply convolutive BSS methods to large

369

APPENDIX

scale problems. Other practical problems in audio and communications applications
have been discussed in Torkkola’s tutorial review [430]. More information can befound in the given references and recent reviews [257, 425, 429, 430] on convolutive
BSS.
Appendix Discrete-time filters and the z -transformIn this appendix, we briefly discuss certain basic concepts and results of discrete-time signal
processing which are needed in this chapter.
Linear causal discrete-time filters [169, 339] can generally be described by the difference
equationy(n) +

MX
i=1

i y (n ? i) = x(n) +

N
X
i=1

i x(n ? i)

(A.1)

which is mathematically equivalent to the ARMA model (2.127) in Section 2.8.6. In (A.1),
n is discrete time, x(n) is the input signal of the filter, and y(n) its output at time instantn. Causality means that in (A.1) there are no quantities that depend on future time instantsn + j; j > 0, making it possible to compute the filter output y(n) in real time. The constantcoefficients i , i = 1; : : : ; N define the FIR (Finite Impulse Response) part of the filter (A.1),
having the order M . Respectively, the coefficients i , i = 1; : : : ; M define the IIR (Infinite
Impulse Response) part of the filter (A.1) with the order M .
If M = 0, (A.1) defines a pure FIR filter, and if N = 0, a pure IIR filter results. Eitherof these filter structures is typically used in separating convolutive mixtures. The FIR filter
is more popular, because it is always stable, which means that its output y (n) is bounded forbounded input values x(n i) and coefficients i . On the other hand, IIR filter can be unstablebecause of its feedback (recurrent) structure.
The stability and other properties of the discrete-time filter (A.1) can be analyzed conveniently in terms of the z-transform [169, 339]. For a discrete-time real sequence x(k) , thez-transform is defined as the series?

f

X (z) =

1
X

k=?1

x(k)z?kg(A.2)

where z is a complex variable with real and imaginary part. For specifying the z -transform ofa sequence uniquely, one must also know its region of convergence.
The z -transform has several useful properties that follow from its definition. Of particularimportance in dealing with convolutive mixtures is the property that the z -transform of the
convolution sum

y(n) =Xkhk x(n ? k)is the product of the z-transforms of the sequences(A.3)

fhk g and fx(n)g:

Y (z) = H (z)X (z)

(A.4)

370

CONVOLUTIVE MIXTURES AND BLIND DECONVOLUTION

The weights hk in (A.3) are called impulse response values, and the quantity H (z ) =Y (z)=X (z) is called transfer function. The transfer function of the convolution sum (A.3) isthe z -transform of its impulse response sequence.
The Fourier transform of a sequence is obtained from its z -transform as a special case by
constraining the variable z to lie on the unit circle in the complex plane. This can be done bysettingz = exp(|!) = cos(!) + | sin(!)(A.5)
where | is the imaginary unit and ! the angular frequency. The Fourier transform has similar
convolution and other properties as the z -transform [339].
Applying the z -transform to both sides of Eq. (A.1) yields
A(z)Y (z) = B (z)X (z)
(A.6)
where

A(z) = 1 +

M
X
k=1

k z ?k ;Y (z) =

MX
k=0

y(n ? k)z?k(A.7)

A(z) is the z-transform of the coefficients 1; 1 ; : : : ; M where the coefficient 0 = 1
corresponds to y (n), and Y (z ) is the z -transform of the output sequence y (n); : : : ; y (n ? M ).
B (z) and X (z) are defined quite similarly as z-transform of the coefficients 1; 1 ; : : : ; N ,and the respective input signal sequence x(n); : : : ; x(n ? N ).
From (A.6), we get for the transfer function of the linear filter (A.1)

Y (z) = B (z)H (z) = X
(z )
A(z)

(A.8)

Note that for a pure FIR filter, A(z ) = 1, and for pure IIR filter B (z ) = 1. The zeros ofdenominator polynomial A(z ) are called the poles of the transfer function (A.8), and the zerosof numerator B (z ) are called the zeros of (A.8). It can be shown (see for example [339]) thatthe linear causal discrete-time filter (A.1) is stable if all the poles of the transfer function lie
inside the unit circle in the complex plane. This is also the stability condition for a pure IIR
filter.
From (A.8), X (z ) = G(z )Y (z ), where the inverse filter G(z ) has the transfer function
1=H (z ) = A(z )=B (z ). Hence, the inverse filter of a pure FIR filter is a pure IIR filter and viceversa. Clearly, the general stability condition for the inverse filter G(z ) is that the zeros ofB (z) (and hence the zeros of the filter H (z)) in (A.8) are inside the unit circle in the complex
plane. This is also the stability condition for the inverse of a pure FIR filter.
Generally, it is desirable that both the poles and the zeros of the transfer function (A.8)
lie inside the unit circle. Then both the filter and its inverse filter exist and are stable. Suchfilters are called minimum phase filters. The minimum phase property is a necessity in manymethods developed for convolutive mixtures. It should be noted that a filter that has no stablecausal inverse may have a stable noncausal inverse, realized by a nonminimum-phase filter.
These matters are discussed much more thoroughly in many textbooks of digital signal
processing and related areas; see for example [339, 302, 169, 171].

20
Other Extensions

In this chapter, we present some additional extensions of the basic independent
component analysis (ICA) model. First, we discuss the use of prior informationon the mixing matrix, especially on its sparseness. Second, we present models thatsomewhat relax the assumption of the independence of the components. In the model
called independent subspace analysis, the components are divided into subspaces thatare independent, but the components inside the subspaces are not independent. In themodel of topographic ICA, higher-order dependencies are modeled by a topographicorganization. Finally, we show how to adapt some of the basic ICA algorithms to thecase where the data is complex-valued instead of real-valued.

20.1

PRIORS ON THE MIXING MATRIX

20.1.1

Motivation for prior information

No prior knowledge on the mixing matrix is used in the basic ICA model. This has theadvantage of giving the model great generality. In many application areas, however,information on the form of the mixing matrix is available. Using prior information on
the mixing matrix is likely to give better estimates of the matrix for a given numberof data points. This is of great importance in situations where the computationalcosts of ICA estimation are so high that they severely restrict the amount of data thatcan be used, as well as in situations where the amount of data is restricted due to thenature of the application.

371

372

OTHER EXTENSIONSThis situation can be compared to that found in nonlinear regression, where
overlearning or overfitting is a very general phenomenon [48]. The classic wayof avoiding overlearning in regression is to use regularizing priors, which typically
penalize regression functions that have large curvatures, i.e., lots of “wiggles”. This
makes it possible to use regression methods even when the number of parametersin the model is very large compared to the number of observed data points. In theextreme theoretical case, the number of parameters is infinite, but the model can stillbe estimated from finite amounts of data by using prior information. Thus suitablepriors can reduce overlearning that was discussed in Section 13.2.2.
One example of using prior knowledge that predates modern ICA methods is theliterature on beamforming (see the discussion in [72]), where a very specific form of
the mixing matrix is represented by a small number of parameters. Another example
is in the application of ICA to magnetoencephalogaphy (see Chapter 22), where it
has been found that the independent components (ICs) can be modeled by the classic
dipole model, which shows how to constrain the form of the mixing coefficients[246]. The problem with these methods, however, is that they may be applicable to afew data sets only, and lose the generality that is one of the main factors in the current
flood of interest in ICA.
Prior information can be taken into account in ICA estimation by using Bayesianprior distributions for the parameters. This means that the parameters, which in thiscase are the elements of the mixing matrix, are treated as random variables. They
have a certain distribution and are thus more likely to assume certain values thanothers. A short introduction to Bayesian estimation was given in Section 4.6.
In this section, we present a form of prior information on the mixing matrix
that is both general enough to be used in many applications and strong enough toincrease the performance of ICA estimation. To give some background, we firstinvestigate the possibility of using two simple classes of priors for the mixing matrix: Jeffreys’ prior and quadratic priors. We come to the conclusion that these twoclasses are not very useful in ICA. Then we introduce the concept of sparse priors.
These are priors that enforce a sparse structure on the mixing matrix. In other words,
the prior penalizes mixing matrices with a larger number of significantly nonzero
entries. Thus this form of prior is analogous to the widely-used prior knowledge on
the supergaussianity or sparseness of the independent components. In fact, due to thissimilarity, sparse priors are so-called conjugate priors, which implies that estimation
using this kind of priors is particularly easy: Ordinary ICA methods can be simply
adapted to using such priors.

A

20.1.2

Classic priors

In the following, we assume that the estimator B of the inverse of the mixing matrix
A is constrained so that the estimates of the independent components y = Bx are
white, i.e., decorrelated and of unit variance: EfyyT g = I. This restriction greatly
facilitates the analysis. It is basically equivalent to first whitening the data and thenrestricting B to be orthogonal, but here we do not want to restrict the generality of

PRIORS ON THE MIXING MATRIX

these results by whitening. We concentrate here on formulating priors forCompletely analogue results hold for prior on .

A373

B = A? .
1

Jeffreys’ prior The classic prior in Bayesian inference is Jeffreys’ prior. It
is considered a maximally uninformative prior, which already indicates that it isprobably not useful for our purpose.
Indeed, it was shown in [342] that Jeffreys’ prior for the basic ICA model has theform:p(B) / j det B?1 jy(20.1)

Bx

B
W

Now, the constraint of whiteness of the =means thatcan be expressedas=, whereis a constant whitening matrix, andis restricted to
be orthogonal. But we have det = det
det = det , which implies that
Jeffreys’s prior is constant in the space of allowed estimators (i.e., decorrelating ).
Thus we see that Jeffreys’ prior has no effect on the estimator, and therefore cannot
reduce overlearning.

B

WV

V

B

W

V

V

B

Quadratic priors In regression, the use of quadratic regularizing priors is very
common [48]. It would be tempting to try to use the same idea in the context of ICA.
Especially in feature extraction, we could require the columns of , i.e. the features,
to be smooth in the same sense as smoothness is required from regression functions.
In other words, we could consider every column of as a discrete approximation ofa smooth function, and choose a prior that imposes smoothness for the underlying
continuous function. Similar arguments hold for priors defined on the rows of ,i.e., the filters corresponding to the features.
The simplest class of regularizing priors is given by quadratic priors. We willshow here, however, that such quadratic regularizers, at least the simple class that we
define below, do not change the estimator.
Consider priors that are of the form

A

A

B

log p(B) =nXTbi Mbi + const:

i=1(20.2)

where the Ti are the rows of = ?1 , andis a matrix that defines the quadratic
prior. For example, for= we have a “weight decay” prior log p( ) = i k i k2that is often used to penalize large elements in . Alternatively, we could include insome differential operators so that the prior would measure the “smoothnesses”
of the i , in the sense explained above. The prior can be manipulated algebraicallyto yield

b

M

B A
M I

M
B

B

P b

b

n
X
T

n
X

i=1

i=1

bi Mbi =MbibTi ) = tr(MBT B)tr((20.3)

Quadratic priors have little significance in ICA estimation, however. To see this,let us constrain the estimates of the independent components to be white as previously.

374

OTHER EXTENSIONSThis means that we have

E fyyT g = E fBxxT BT g = BCBT = I(20.4)

in the space of allowed estimates, which gives after some algebraic manipulations
T = ?1 . Now we see that

BB C

n
X
T
i=1

bi Mbi = tr(MC? ) = const:
1(20.5)

In other words, the quadratic prior is constant. The same result can be proven for aquadratic prior on . Thus, quadratic priors are of little interest in ICA.

A

20.1.3Sparse priors

Motivation A much more satisfactory class of priors is given by what we callsparse priors. This means that the prior information says that most of the elementsof each row of are zero; thus their distribution is supergaussian or sparse. The
motivation for considering sparse priors is both empirical and algorithmic.
Empirically, it has been observed in feature extraction of images (see Chapter 21)
that the obtained filters tend to be localized in space. This implies that the distributionof the elements bij of the filter i tends to be sparse, i.e., most elements are practicallyzero. A similar phenomenon can be seen in analysis of magnetoencephalography,
where each source signal is usually captured by a limited number of sensors. This isdue to the spatial localization of the sources and the sensors.
The algorithmic appeal of sparsifying priors, on the other hand, is based on thefact that sparse priors can be made to be conjugate priors (see below for definition).
This is a special class of priors, and means that estimation of the model using thisprior requires only very simple modifications in ordinary ICA algorithms.
Another motivation for sparse priors is their neural interpretation. Biologicalneural networks are known to be sparsely connected, i.e., only a small proportionof all possible connections between neurons are actually used. This is exactly whatsparse priors model. This interpretation is especially interesting when ICA is used inmodeling of the visual cortex (Chapter 21).

Bb

Measuring sparsity The sparsity of a random variable, say s, can be measured by
expectations of the form E fG(s)g, where G is a nonquadratic function, for example,
the following

G(s) = ?jsj:(20.6)The use of such measures requires that the variance of s is normalized to a fixedvalue, and its mean is zero. These kinds of measures were widely used in Chapter 8
to probe the higher-order structure of the estimates of the ICs. Basically, this isa robust nonpolynomial moment that typically is a monotonic function of kurtosis.
Maximizing this function is maximizing kurtosis, thus supergaussianity and sparsity.

375

PRIORS ON THE MIXING MATRIXIn feature extraction and probably several other applications as well, the distributions of the elements of of the mixing matrix and its inverse are zero-mean due tosymmetry. Let us assume that the data is whitened as a preprocessing step. Denoteby the whitened data vector whose components are thus uncorrelated and have unit
variance. Constraining the estimates =of the independent components tobe white implies that , the inverse of the whitened mixing matrix, is orthogonal.
This implies that the sum of the squares of the elements j wij is equal to one forcan be then considered a realization of
every i. The elements of each row iT ofa random variable of zero mean and unit variance. This means we could measure theusing a sparsity measure of the form (20.6).
sparsities of the rows ofThus, we can define a sparse prior of the form

x
y Wz

z

W

w

W

P

W

log p(W) =n XnX
i=1 j =1

G(wij ) + const:(20.7)

where G is the logarithm of some supergaussian density function. The function G in
(20.6) is such log-density, corresponding to the Laplacian density, so we see that wehave here a measure of sparsity of the i .
The prior in (20.7) has the nice property of being a conjugate prior. Let us assume
that the independent components are supergaussian, and for simplicity, let us further
assume that they have identical distributions, with log-density G. Now we can takethat same log-density as the log-prior density G in (20.7). Then we can write theprior in the formw

log p(W) =n XnX
i=1 j =1

G(wiT ej ) + const:e(20.8)ewhere we denote by i the canonical basis vectors, i.e., the ith element of i is equal
to one, and all the others are zero. Thus the posterior distribution has the form:

log p(Wjz(1); :::; z (T )) =

n X
T
X

[

i=1 t=1

G(wiT z(t)) +

n
X
j =1

G(wiT ej )] + const:(20.9)

This form shows that the posterior distribution has the same form as the priordistribution (and, in fact, the original likelihood). Priors with this property are called
conjugate priors in Bayesian theory. The usefulness of conjugate priors resides in theproperty that the prior can be considered to correspond to a “virtual” sample. The
posterior distribution in (20.9) has the same form as the likelihood of a sample of size
T + n, which consists of both the observed (t) and the canonical basis vectors i .
In other words, the posterior in (20.9) is the likelihood of the augmented (whitened)data samplezz(t) =e(z(t); if 1  t  T
et?T ; if T < t  T + n(20.10)376

OTHER EXTENSIONS

Thus, using conjugate priors has the additional benefit that we can use exactly thesame algorithm for maximization of the posterior as in ordinary maximum likelihood
estimation of ICA. All we need to do is to add this virtual sample to the data; thevirtual sample is of the same size n as the dimension of the data.
For experiments using sparse priors in image feature extraction, see [209].
Modifying prior strength The conjugate priors given above can be generalized
by considering a family of supergaussian priors given bylog p(W) =n XnX
i=1 j =1

G(wiT ej ) + const:(20.11)

Using this kind of prior means that the virtual sample points are weighted by someparameter . This parameter expresses the degree of belief that we have in the prior.
A large means that the belief in the prior is strong. Also, the parameter could
be different for different i, but this seems less useful here. The posterior distribution
then has the form:

log p(Wjz(1); :::; z(T )) =

T
n X
X

[

i=1 t=1

G(wiT z(t)) +

n
X
j =1

G(wiT ej )] + const:(20.12)The preceding expression can be further simplified in the case where the assumed
density of the independent components is Laplacian, i.e., G(y ) = ?jy j. In this case,the can multiply the j themselves:e

log p(Wjz(1); :::; z(T )) =

T
n X
X
i=1

[?

t=1

jwiT z(t)j ?

n
X
j =1

jwiT ( ej )j] + const:(20.13)

which is simpler than (20.12) from the algorithmic viewpoint: It amounts to theaddition of just n virtual data vectors of the form j to the data. This avoids all
the complications due to the differential weighting of sample points in (20.12), andensures that any conventional ICA algorithm can be used by simply adding the virtual
sample to the data. In fact, the Laplacian prior is most often used in ordinary ICA
algorithms, sometimes in the form of the log cosh function that can be considered asa smoother approximation of the absolute value function.

eWhitening and priors In the preceding derivation, we assumed that the data ispreprocessed by whitening. It should be noted that the effect of the sparse prior isdependent on the whitening matrix. This is because sparseness is imposed on theseparating matrix of the whitened data, and the value of this matrix depends on thewhitening matrix. There is an infinity of whitening matrices, so imposing sparsenesson the whitened separating matrix may have different meanings.
On the other hand, it is not necessary to whiten the data. The preceding framework
can be used for non-white data as well. If the data is not whitened, the meaning of
the sparse prior is somewhat different, though. This is because every row of i is not

b

377

PRIORS ON THE MIXING MATRIX

constrained to have unit norm for general data. Thus our measure of sparsity does
not anymore measure the sparsities of each i . On the other hand, the developmentsof the preceding section show that the sum of squares of the whole matrix ij bij
does stay constant. This means that the sparsity measure is now measuring rather the
global sparsity of , instead of the sparsities of individual rows.
In practice, one usually wants to whiten the data for technical reasons. Then the
problems arises: How to impose the sparseness on the original separating matrix evenwhen the data used in the estimation algorithm needs to be whitened? The preceding
framework can be easily modified so that the sparseness is imposed on the original
separating matrix. Denote by the whitening matrix and by the separating matrix= and = by definition. Now, we canfor original data. Thus, we haveexpress the prior in (20.8) as

b

P

B

V

log p(B) =n XnX
i=1 j =1

WV B

B

z Vx

G(bTi ej ) + const. =n XnX
i=1 j =1

G(wiT (Vej )) + const.

(20.14)zThus, we see that the virtual sample added to (t) now consists of the columns of thewhitening matrix, instead of the identity matrix.
Incidentally, a similar manipulation of (20.8) shows how to put the prior on theoriginal mixing matrix instead of the separating matrix. We always have=( )?1 = T . Thus, we obtain Ti j = Ti T ( ?1 )T j = iT ( ?1 )T j . Thisshows that imposing a sparse prior on is done by using the virtual sample givenby the rows of the inverse of the whitening matrix. (Note that for whitened data,
the mixing matrix is the transpose of the separating matrix, so the fourth logicalpossibility of formulating prior for the whitened mixing matrix is not different fromusing a prior on the whitened separating matrix.)
In practice, the problems implied by whitening can often be solved by using awhitening matrix that is sparse in itself. Then imposing sparseness on the whitened
separating matrix is meaningful. In the context of image feature extraction, a sparse
whitening matrix is obtained by the zero-phase whitening matrix (see [38] for discussion), for example. Then it is natural to impose the sparseness for the whitend
separating matrix, and the complications discussed in this subsection can be ignored.

WW

20.1.4

Spatiotemporal ICA

ae

A

aV V

e

w V

e

VA

When using sparse priors, we typically make rather similar assumptions on both theICs and the mixing matrix. Both are assumed to be generated so that the values
are taken from independent, typically sparse, distributions. At the limit, we mightdevelop a model where the very same assumptions are made on the mixing matrix
and the ICs. Such a model [412] is called spatiotemporal ICA since it does ICA bothin the temporal domain (assuming that the ICs are time signals), and in the spatial
domain, which corresponds to the spatial mixing defined by the mixing matrix.
In spatiotemporal ICA, the distinction between ICs and the mixing matrix iscompletely abolished. To see why this is possible, consider the data as a single= ( (1); :::; (T )), and likewise
matrix of the observed vectors as its columns:

X

x

x

378

OTHER EXTENSIONS

for the ICs. Then the ICA model can be expressed as

X = AS(20.15)

Now, taking a transpose of this equation, we obtain

XT = ST AT(20.16)SA

Now we see that the matrix is like a mixing matrix, with T giving the realizationsof the “independent components”. Thus, by taking the transpose, we flip the roles ofthe mixing matrix and the ICs.
In the basic ICA model, the difference between and is due to the statisticalassumptions made on , which are the independent random variables, and on , which
is a constant matrix of parameters. But with sparse priors, we made assumptions onthat are very similar to those usually made on . So, we can simply consider bothand as being generated by independent random variables, in which case either
one of the mixing equations (with or without transpose) are equally valid. This is thebasic idea in spatiotemporal ICA.
There is another important difference between and , though. The dimensionsofand are typically very different:
is square whereas has many morecolumns than rows. This difference can be abolished by considering that there has
many fewer columns than rows, that is, there is some redundancy in the signal.
The estimation of the spatiotemporal ICA model can be performed in a manner
rather similar to using sparse priors. The basic idea is to form a virtual sample where
the data consists of two parts, the original data and the data obtained by transposing
the data matrix. The dimensions of these data sets must be strongly reduced andmade equal to each other, using PCA-like methods. This is possible because it was
assumed that both and T have the same kind of redundancy: many more rows
than columns. For details, see [412], where the infomax criterion was applied on thisestimation task.

s

s

A
A

S

A

A

20.2

A

s

S

A

A

S

A

S

A

S

RELAXING THE INDEPENDENCE ASSUMPTIONIn the ICA data model, it is assumed that the components si are independent. However, ICA is often applied on data sets, for example, on image data, in which theobtained estimates of the independent components are not very independent, evenapproximately. In fact, it is not possible, in general, to decompose a random vector
linearly into components that are independent. This raises questions on the utility
and interpretation of the components given by ICA. Is it useful to perform ICA on
real data that does not give independent components, and if it is, how should theresults be interpreted?
One approach to this problem is to reinterpret the estimation results. A straightforward reinterpretation was offered in Chapter 10: ICA gives components that are as
independent as possible. Even in cases where this is not enough, we can still justify
the utility by other arguments. This is because ICA simultaneously serves certainx

RELAXING THE INDEPENDENCE ASSUMPTION

379

other useful purposes than dependence reduction. For example, it can be interpreted
as projection pursuit (see Section 8.5) or sparse coding (see Section 21.2). Both ofthese methods are based on the maximal nongaussianity property of the independent
components, and they give important insight into what ICA algorithms are really
doing.
A different approach to the problem of not finding independent components is torelax the very assumption of independence, thus explicitly formulating new data models. In this section, we consider this approach, and present three recently developed
methods in this category. In multidimensional ICA, it is assumed that only certain
sets (subspaces) of the components are mutually independent. A closely related
method is independent subspace analysis, where a particular distribution structure
inside such subspaces is defined. Topographic ICA, on the other hand, attempts
to utilize the dependence of the estimated “independent” components to define a
topographic order.
20.2.1

Multidimensional ICA

In multidimensional independent component analysis [66, 277], a linear generativemodel as in basic ICA is assumed. In contrast to basic ICA, however, the components(responses) si are not assumed to be all mutually independent. Instead, it is assumed
that the si can be divided into couples, triplets or in general k -tuples, such that thesi inside a given k-tuple may be dependent on each other, but dependencies between
different k -tuples are not allowed.
Every k -tuple of si corresponds to k basis vectors i . In general, the dimensionality of each independent subspace need not be equal, but we assume so for simplicity.
The model can be simplified by two additional assumptions. First, even though thecomponents si are not all independent, we can always define them so that they areuncorrelated, and of unit variance. In fact, linear correlations inside a given k -tuple ofdependent components could always be removed by a linear transformation. Second,
we can assume that the data is whitened (sphered), just as in basic ICA.
These two assumptions imply that the i are orthonormal. In particular, theindependent subspaces become orthogonal after whitening. These facts follow directly from the proof in Section 7.4.2, which applies here as well, due to our present
assumptions.
Let us denote by J the number of independent feature subspaces, and by Sj ; j =1; :::; J the set of the indices of the si belonging to the subspace of index j . Assume
that the data consists of T observed data points (t); t = 1; :::; T . Then we can
express the likelihood L of the data, given the model as follows

a

a

x

L(x(t); t = 1; :::; T ; bi; i = 1; :::; n)

=

T
Y

t=1

[j det Bj

J
Y

j =1

pj (bTi x(t); i 2 Sj )]bx(20.17)

where pj (:), which is a function of the k arguments Ti (t); i 2 Sj , gives theprobability density inside the j th k -tuple of si . The term j det j appears here as in

B

380

OTHER EXTENSIONS

any expression of the probability density of a transformation, giving the change involume produced by the linear transformation, as in Chapter 9.
The k -dimensional probability density pj (:) is not specified in advance in thegeneral definition of multidimensional ICA [66]. Thus, the question arises how
to estimate the model of multidimensional ICA. One approach is to estimate thebasic ICA model, and then group the components into k -tuples according to theirdependence structure [66]. This is meaningful only if the independent components
are well defined and can be accurately estimated; in general we would like to utilize
the subspace structure in the estimation process. Another approach is to modelthe distributions inside the subspaces by a suitable model. This is potentially very
difficult, since we then encounter the classic problem of estimating k -dimensional
distributions. One solution for this problem is given by independent subspaces
analysis, to be explained next.
20.2.2

Independent subspace analysis

Independent subspace analysis [204] is a simple model that models some dependencies between the components. It is based on combining multidimensional ICA withthe principle of invariant-feature subspaces.
Invariant-feature subspaces To motivate independent subspace analysis, let us
consider the problem of feature extraction, treated in more detail in Chapter 21. In themost basic case, features are given by linear transformations, or filters. The presenceof a given feature is detected by computing the dot-product of input data with a givenfeature vector. For example, wavelet, Gabor, and Fourier transforms, as well as mostmodels of V1 simple cells, use such linear features (see Chapter 21). The problem
with linear features, however, is that they necessarily lack any invariance with respect
to such transformations as spatial shift or change in (local) Fourier phase [373, 248].
Kohonen [248] developed the principle of invariant-feature subspaces as an abstract approach to representing features with some invariances. The principle ofinvariant-feature subspaces states that one can consider an invariant feature as a linear subspace in a feature space. The value of the invariant, higher-order feature isgiven by (the square of) the norm of the projection of the given data point on thatsubspace, which is typically spanned by lower-order features.
A feature subspace, as any linear subspace, can always be represented by a setof orthogonal basis vectors, say i ; i = 1; :::; k , where k is the dimension of thesubspace. Then the value F ( ) of the feature F with input vector is given byxb

F (x) =

x

k
X
T
i=1

(bi x)2(20.18)x

In fact, this is equivalent to computing the distance between the input vector and ageneral linear combination of the vectors (possibly filters) i of the feature subspace[248].

b

RELAXING THE INDEPENDENCE ASSUMPTION

381

Spherical symmetry Invariant-feature subspaces can be embedded in multidimensional independent component analysis by considering probability distributions
for the k -tuples of si that are spherically symmetric, i.e., depend only on the norm. In
other words, the probability density pj (:) of a k -tuple can be expressed as a functionof the sum of the squares of the si ; i 2 Sj only. For simplicity, we assume further
that the pj (:) are equal for all j , i.e., for all subspaces.
This means that the logarithm of the likelihood L of the data (t); t = 1; :::; T ,can be expressed asxlog L(x(t); t = 1; :::; T ; bi; i = 1; :::; n)

=

P

J
T X
X
t=1 j =1

log p(

X T

i2Sj(bi x(t))2 ) + T log j det Bj(20.19)where p( i2Sj s2i ) = pj (si ; i 2 Sj ) gives the probability density inside the j th
k-tuple of si .
Recall that prewhitening allows us to consider the i to be orthonormal, which
implies that log j det j is zero. This shows that the likelihood in Eq. (20.19) is afunction of the norms of the projections of on the subspaces indexed by j , which
are spanned by the orthonormal basis sets given by i ; i 2 Sj .
In the case of clearly supergaussian components, we can use the following probability distribution:

b

B

x

log p(

Xi2Sj

s2i ) = ? [

b

X

i2Sj

s2i ]1=2 +(20.20)which could be considered a multi-dimensional version of the exponential distribution. The scaling constant and the normalization constant are determined so as
to give a probability density that is compatible with the constraint of unit variance ofthe si , but they are irrelevant in the following. Thus we see that the estimation of
the model consists of finding subspaces such that the norms of the projections of the(whitened) data on those subspaces have maximally sparse distributions.
Independent subspace analysis is a natural generalization of ordinary ICA. In fact,
if the projections on the subspaces are reduced to dot-products, i.e., projections on
one-dimensional (1-D) subspaces, the model reduces to ordinary ICA, provided that,in addition, the independent components are assumed to have symmetric distributions.
It is to be expected that the norms of the projections on the subspaces represent somehigher-order, invariant features. The exact nature of the invariances has not been
specified in the model but will emerge from the input data, using only the priorinformation on their independence.
If the subspaces have supergaussian (sparse) distributions, the dependency impliedby the model is such that components in the same subspace tend to be nonzero at thesame time. In other words, the subspaces are somehow “activated” as a whole, andthen the values of the individual components are generated according to how strongly
the subspaces are activated. This is the particular kind of dependency that is modeledby independent subspaces in most applications, for example, with image data.

382

OTHER EXTENSIONSFor more details on independent subspace analysis, the reader is referred to [204].
Some experiments on image data are reported in Section 21.5 as well.
20.2.3

Topographic ICA

Another way of approaching the problem of nonexistence of independent components
is to try to somehow make the dependency structure of the estimated components
visible. This dependency structure is often very informative and could be utilized infurther processing.
Estimation of the “residual” dependency structure of estimates of independent
components could be based, for example, on computing the cross-cumulants. Typically these would be higher-order cumulants, since second-order cross-cumulants,i.e., covariance, are typically very small, and can in fact be forced to be zero, as we didby orthogonalization after whitening in Part II. However, using such measures raises
the question as to how such numerical estimates of the dependence structure should
be visualized or otherwise utilized. Moreover, there is another serious problem associated with simple estimation of some dependency measures from the estimatesof the independent components. This is due to the fact that often the independent
components do not form a well-defined set. Especially in image decomposition(Chapter 21), the set of potential independent components seems to be larger thanwhat can be estimated at one time, in fact the set might be infinite. A classic ICA
method gives an arbitrarily chosen subset of such independent components. Thus, it isimportant in many applications that the dependency information is utilized during theestimation of the independent components, so that the estimated set of independent
components is one whose residual dependencies can be represented in a meaningful
way. (This is something we already argued in connection with independent subspace
analysis.)
Topographic ICA, introduced in [206], is a modification of the classic ICA modelin which the dependencies of the components are explicitly represented. In particular,
we propose that the residual dependency structure of the independent components,
i.e., dependencies that cannot be canceled by ICA, could be used to define a topographical order between the components. The topographical order is easy torepresent by visualization, and is important in image feature extraction due to its
connections to brain modeling [206].
Our model gives a topographic map where the distance of the components inthe topographic representation is a function of the dependencies of the components.
Components that are near to each other in the topographic representation are strongly
dependent in the sense of higher-order correlations.
To obtain topographic ICA, we generalize the model defined by (20.19) so thatit models a dependence not only inside the k -tuples, but among all neighboring
components. A neighborhood relation defines a topographical order. We define thelikelihood of the model as follows:

log L(B) =

n
T X
n
X
X
t=1 j =1G(i=1h(i; j )(bTi x(t))2 ) + T log j det Bj + const: (20.21)

COMPLEX-VALUED DATA

383

Here, the h(i; j ) is a neighborhood function, which expresses the strength of theconnection between the ith and j th units. It can be defined in the same way as inother topographic maps, like the self-organizing map (SOM) [247]. The function G
is similar to the one in independent subspace analysis. The additive constant depends
only on h(i; j ).
This model thus can be considered a generalization of the model of independent
subspace analysis. In independent subspace analysis, the latent variables si are clearly
divided into k -tuples or subspaces, whereas in topographic ICA, such subspaces arecompletely overlapping: Every neighborhood corresponds to one subspace.
Just as independent subspace analysis, topographic ICA usually models a situation
where nearby components tend to be active (nonzero) at the same time. This seems
to be a common dependency structure for natural sparse data [404]. In fact, thelikelihood given earlier can also be derived as an approximation of the likelihood ofa model where the variance of the ICs is controlled by some higher-order variables,
so that the variances of near-by components are strongly dependent.
For more details on topographic ICA, the reader is referred to [206]. Someexperiments on image data are reported in Chapter 21 as well.

20.3

COMPLEX-VALUED DATA

Sometimes in ICA, the ICs and/or the mixing matrix are complex-valued. For example, in signal processing in some cases frequency (Fourier) domain representations ofsignals have advantages over time-domain representations. Especially in the separation of convolutive mixtures (see Chapter 19) it is quite common to Fourier transformthe signals, which results in complex-valued signals.
In this section we show how the FastICA algorithm can be extended to complexvalued signals. Both the ICs and the observed mixtures assume complex values.
For simplicity, we assume that the number of independent component variables is
the same as the number of observed linear mixtures. The mixing matrix is of fullrank and it may be complex as well, but this need not be the case.
In addition to the assumption of the independence of the components si , anassumption on the dependence of the real and complex parts of a single IC is madehere. We assume that every si is white in the sense that the real and imaginary partsof sj are uncorrelated and their variances are equal; this is quite realistic in practicalproblems.
Related work on complex ICA can be found in [21, 132, 305, 405].

s

x

A

20.3.1

Basic concepts of complex random variables

First, we review some basic concepts of complex random variables; see [419] formore details.
A complex random variable y can be represented as y = u + iv where u and v
are real-valued random variables. The density of y is f (y ) = f (u; v ) 2 R. The

384

OTHER EXTENSIONS

expectation of y is E fy g = E fug + iE fv g. Two complex random variables y1 andy2 are uncorrelated if E fy1 y2 g = E fy1 gE fy2g, where y = u ? iv designates thecomplex conjugate of y . The covariance matrix of a zero-mean complex random
vector y = (y1 ; : : : ; yn ) is2C
E fyyH g = 64 ...

11 C n 3
1

..

..
.

.

Cn1    Cnn

75(20.22)where Cjk = E fyj yk g and yH stands for the Hermitian of y, that is, y transposedand conjugated. The data can be whitened in the usual way.
In our complex ICA model, all ICs si have zero mean and unit variance. Moreover,
we require that they have uncorrelated real and imaginary parts of equal variances.
This can be equivalently expressed as E f H g = and E fssT g = O. In the latter,the expectation of the outer product of a complex random vector without the conjugate
is a null matrix. These assumptions imply that si must be strictly complex; that is,
the imaginary part of si may not in general vanish.
The definition of kurtosis can be easily generalized. For a zero-mean complex
random variable it could be defined, for example, as [305, 319]ssI

kurt(y ) = E fjy j4 g ? E fyy  gE fyy g ? E fyy gE fy y  g ? E fyy  gE fy  y g(20.23)but the definitions vary with respect to the placement of conjugates ( ) — actually,
there are 24 ways to define the kurtosis [319]. We choose the definition in [419],
wherekurt(y ) = E fjy j4 g ? 2(E fjy j2 g)2 ? jE fy 2 gj2 = E fjy j4 g ? 2(20.24)

where the last equality holds if y is white, i.e., the real and imaginary parts of y
are uncorrelated and their variances are equal to 1=2. This definition of kurtosis is
intuitive since it vanishes if y is gaussian.
20.3.2

Indeterminacy of the independent componentss

The independent components in the ICA model are found by searching for a matrix
such that =. However, as in basic ICA, there are some indeterminacies. In
the real case, a scalar factor i can be exchanged between si and a column i of1
without changing the distribution of : i si = ( i i )( ?
i si ). In other words, theorder, the signs and the scaling of the independent components cannot be determined.
Usually one defines the absolute scaling by defining E fs2i g = 1; thus only the signsof the independent components are indetermined.
Similarly in the complex case there is an unknown phase vj for each sj . Let us
write the decomposition

B

s Bx

x a

a

ai si = (vi ai )(vi? sj )1

aA

(20.25)COMPLEX-VALUED DATA

385

where the modulus of vi is equal to one. If si has a spherically symmetric distribution,i.e., the distribution depends on the modulus of si only, the multiplication by a variable
vi does not change the distribution of si. Thus the distribution of remains unchanged
as well. From this indeterminacy it follows that it is impossible to retain the phasesof si , andis a matrix where in each row and each column there is one nonzero
element that is of unit modulus.

xBA

20.3.3

Choice of the nongaussianity measure

Now we generalize the framework in Chapter 8 for complex-valued signals. In
the complex case, the distributions for the complex variables are often sphericallysymmetric, so only the modulus is interesting. Thus we could use a nongaussianity
measure that is based on the modulus only. Based on the measure of nongaussianity
as in (8.25), we use the following:

JG (w) = E fG(jwH zj2 )g(20.26)
where G is a smooth even function, w is an n-dimensional complex vector andE fjwH zj2 g = kwk2 = 1. The data is whitened, as the notation z already indicates.

This can be compared with (20.24), which gives the kurtosis of complex variables:
if we choose G(y ) = y 2 , then JG ( ) = E fj H j4 g. Thus J essentially measures
the kurtosis of H , which is a classic measure in higher-order statistics.
Maximizing JG we estimate one IC. Estimating n independent components ispossible, just as in the real case, by using a sum of n measures of nongaussianity, anda constraint of orthogonality. Thus one obtains the following optimization problem:w

w z

maximize

n
X
j =1

w z

JG (wi ) with respect to wi ; j = 1; : : : ; nw w

under constraint E f kH i g = jk(20.27)

where jk = 1 for j = k and jk = 0 otherwise.
It is highly preferable that the estimator given by the contrast function is robust
against outliers. The more slowly G grows as its argument increases, the more robust
is the estimator. For the choice of G we propose now three different functions, thederivatives g of which are also given:p

G1 (y) = a1 + y; g1 (y) = 2pa1 + y(20.28)1G2 (y) = log(a2 + y); g2 (y) = a 1+ y
G3 (y) = 21 y2 ; g3 (y) = y(20.29)2(20.30)where a1 and a2 are some arbitrary constants (for example, a1  0:1 and a2  0:1
seem to be suitable). Of the preceding functions, G1 and G2 grow more slowly thanG3 and thus they give more robust estimators. G3 is motivated by kurtosis (20.24).

386

OTHER EXTENSIONS

20.3.4

Consistency of estimator

In Chapter 8 it was stated that any nonlinear learning function G divides the space of
probability distributions into two half-spaces. Independent components can be estimated by either maximizing or minimizing a function similar to (20.26), dependingon which half-space their distribution lies in. A theorem for real valued signals waspresented that distinguished between maximization and minimization and gave theexact conditions for convergence. Now we show how this idea can be generalized
to complex-valued random variables. We have the following theorem on the local
consistency of the estimators [47]:
Theorem 20.1 Assume that the input data follows the complex ICA model. The
observed mixtures are prewhitened so that E f H g = . The independent components have zero mean, unit variance, and uncorrelated real and imaginary partsof equal variances. Also, G : R+ [ f0g ! R is a sufficiently smooth even function. Then the local maxima (resp. minima) of E fG(j H j2 )g under the constraint
E fj H j2 g = jj jj2 = 1 include those rows of the inverse of the whitened mixing
matrix VA such that the corresponding independent components sk satisfy

zz

w z

I

w z

w

E fg(jsk j2 ) + jsk j2 g0 (jsk j2 ) ? jsk j2 g(jsk j2 )g < 0 (> 0; resp.)(20.31)

where g () is the derivative of G() and g 0 () is the derivative of g ().

A special case of the theorem is when g (y ) = y , g 0 (y ) = 1. Condition (20.31)
now reads

E fjsk j2 + jsk j2 ? jsk j2 jsk j2 g = ?E fjsk j4 g + 2 < 0 (> 0; resp.):(20.32)Thus the local maxima of E fG(jwH zj2 )g should be found when E fjsk j4 g ? 2 > 0,i.e., the kurtosis (20.24) of sk is positive. This implies that we are actually maximizing
the absolute values of kurtosis, just like in the basic case in Chapter 8.
20.3.5

Fixed-point algorithm

We now give the fixed-point algorithm for complex signals under the complex ICA
model. The algorithm searches for the extrema of E fG(j H j2 )g. The derivation
is presented in [47].
The FastICA algorithm for one unit, using whitened data, is

w z

w E fx(wH x)g(jwH xj )g ? E fg(jwH xj ) + jwH xj g0(jwH xj )gw
w
w jjw(20.33)
jj
2

2

2

2

The one-unit algorithm can be extended to the estimation of the whole ICA
transformation in exactly the same way as in the real case. The orthogonalization
methods in Section 8.4 can be used by simply replacing every transpose operation in
the formulas by the Hermitian operation [47].

CONCLUDING REMARKS

20.3.6387

Relation to independent subspaces

Our approach to complex ICA closely resembles independent subspace methods,
discussed in Section 20.2.2, and multidimensional ICA, discussed in Section 20.2.1.
In our complex ICA, the nongaussianity measure operates on j H j2 which
can be interpreted as the norm of a projection onto a subspace. The subspace
is two-dimensional, corresponding to the real and imaginary parts of a complex
number. In contrast to the subspace method, one of the basis vectors is determinedstraightforward from the other basis vector. In independent subspace analysis, theindependent subspace is determined only up to an orthogonal k  k matrix factor. In
complex ICA however, the indeterminacy is less severe: the sources are determined
up to a complex factor v , jv j = 1.
It can be concluded that complex ICA is a restricted form of independent subspacemethods.

w z

20.4

CONCLUDING REMARKSThe methods presented in the first two sections of the chapter were all related tothe case where we know more about the data than just the blind assumption of
independence. Using sparse priors, we incorporate some extra knowledge on thesparsity of the mixing matrix in the estimation procedure. This was made very easyby the algorithmic trick of conjugate priors.
In the methods of independent subspaces or topographic ICA, on the other hand,
we assume that we cannot really find independent components; instead we canfind groups of independent components, or components whose dependency structure
can be visualized. A special case of the subspace formalism is encountered if the
independent components are complex-valued.
Another class of extensions that we did not treat in this chapter are the so-called
semiblind methods, that is, methods in which much prior information on the mixing
is available. In the extreme case, the mixing could be almost completely known, in
which case the “blind” aspect of the method disappears. Such semiblind methods
are quite application-dependent. Some methods related to telecommunications aretreated in Chapter 23. A closely related theoretical framework is the “principal”
ICA proposed in [285]. See also [415] for a semiblind method in a brain imaging
application.

Part IV

APPLICATIONS OF ICA

21Feature Extraction by ICA

A fundamental approach in signal processing is to design a statistical generativemodel of the observed signals. The components in the generative model then give a
representation of the data. Such a representation can then be used in such tasks ascompression, denoising, and pattern recognition. This approach is also useful from aneuroscientific viewpoint, for modeling the properties of neurons in primary sensory
areas.
In this chapter, we consider a certain class of widely used signals, which wecall natural images. This means images that we encounter in our lives all the time;
images that depict wild-life scenes, human living environments, etc. The working
hypothesis here is that this class is sufficiently homogeneous so that we can build astatistical model using observations of those signals, and then later use this model forprocessing the signals, for example, to compress or denoise them.
Naturally, we shall use independent component analysis (ICA) as the principal
model for natural images. We shall also consider the extensions of ICA introducedin Chapter 20. We will see that ICA does provide a model that is very similar to
the most sophisticated low-level image representations used in image processing andvision research. ICA gives a statistical justification for using those methods that haveoften been more heuristically justified.

391

392

FEATURE EXTRACTION BY ICA

21.1

LINEAR REPRESENTATIONS

21.1.1

Definition

Image representations are often based on discrete linear transformations of the observed data. Consider a black-and-white image whose gray-scale value at the pixel
indexed by x and y is denoted by I (x; y ). Many basic models in image processing
express the image I (x; y ) as a linear superposition of some features or basis functions
ai (x; y):I (x; y) =

n
X
i=1

ai (x; y)si

(21.1)

where the si are stochastic coefficients, different for each image I (x; y ). Alternatively, we can just collect all the pixel values in a single vector = (x1 ; x2 ; :::; xm )T ,in which case we can express the representation asxx = As(21.2)

just like in basic ICA. We assume here that the number of transformed components
equals the number of observed variables, although this need not be the case in general.
This kind of a linear superposition model gives a useful description on a low level
where we can ignore such higher-level nonlinear phenomena as occlusion.
In practice, we may not model a whole image using the model in (21.1). Rather,
we apply it on image patches or windows. Thus we partition the image into patchesof, for example, 8  8 pixels and model the patches with the model in (21.1). Caremust then be taken to avoid border effects.
Standard linear transformations widely used in image processing are, for example,
the Fourier, Haar, Gabor, and cosine transforms. Each of them has its own favorable
properties [154]. Recently, a lot of interest has been aroused by methods that
attempt to combine the good qualities of frequency-based methods (Fourier andcosine transforms) with the basic pixel-by-pixel representation. Here we succinctly
explain some of these methods; for more details see textbooks on the subject, e.g.,[102], or see [290].
21.1.2Gabor analysis

Gabor functions or Gabor filters [103, 128] are functions that are extensively usedin image processing. These functions are localized with respect to three parameters:
spatial location, orientation, and frequency. This is in contrast to Fourier basis
function that are not localized in space, and the basic pixel-by-pixel representationthat is not localized in frequency or orientation.
Let us first consider, for simplicity, one-dimensional (1-D) Gabor functions insteadof the two-dimensional (2-D) functions used for images. The Gabor functions areLINEAR REPRESENTATIONS

393

Fig. 21.1 A pair of 1-D Gabor functions. These functions are localized in space as well as
in frequency. The real part is given by the solid line and the imaginary part by the dashed line.

then of the form

g1d (x) = exp(? 2 (x ? x0 )2 )[cos(2 (x ? x0 ) + ) + i sin(2 (x ? x0 ) + )](21.3)whereis the constant in the gaussian modulation function, which determines thewidth of the function in space.

 x defines the center of the gaussian function, i.e., the location of the function.
 is the frequency of oscillation, i.e., the location of the function in Fourier0

space.

is the phase of the harmonic oscillation.

Actually, one Gabor function as in (21.3) defines two scalar functions: One as its realpart and the other one as its imaginary part. Both of these are equally important, andthe representation as a complex function is done mainly for algebraic convenience.
A typical pair of 1-D Gabor functions is plotted in Fig. 21.1.
Two-dimensional Gabor functions are created by first taking a 1-D Gabor function
along one of the dimensions and multiplying it by a gaussian envelope in the otherdimension:g2d(x; y) = exp(? 2 (y ? y0 )2 )g1d (x)(21.4)

394

FEATURE EXTRACTION BY ICA

Fig. 21.2 A pair of 2-D Gabor functions. These functions are localized in space, frequency,
and orientation. The real part is on the left, and the imaginary part on the right. These functions
have not been rotated.

where the parameter in the gaussian envelope need not be the same in both directions. Second, this function is rotated by an orthogonal transformation of (x; y ) to agiven angle. A typical pair of the real and imaginary parts of a Gabor functions areshown in Fig. 21.2.
Gabor analysis is an example of multi-resolution analysis, which means that theimage is analyzed separately at different resolutions, or frequencies. This is because
Gabor functions can be generated at different sizes by varying the parameter , andat different frequencies by varying .
An open question is what set of values should one choose for the parameters to
obtain a useful representation of the data. Many different solutions exist; see, e.g.,[103, 266]. The wavelet bases, discussed next, give one solution.
21.1.3

Wavelets

Another closely related method of multiresolution analysis is given by wavelets[102, 290]. Wavelet analysis is based on a single prototype function called the motherwavelet (x). The basis functions (in one dimension) are obtained by translations
(x + l) and dilations or rescalings (2?s x) of this basic function. Thus we use thefamily of functions

s;l (x) = 2?s=2 (2?s x ? l)(21.5)The variables s and l are integers that represent scale and dilation, respectively. The
scale parameter, s, indicates the width of the wavelet, while the location index, l,gives the position of the mother wavelet. The fundamental property of wavelets isthus the self-similarity at different scales. Note that  is real-valued.
The mother wavelet is typically localized in space as well as in frequency. Twotypical choices are shown in Fig. 21.3.
A 2-D wavelet transform is obtained in the same way as a 2-D Fourier transform:by first taking the 1-D wavelet transforms of all rows (or all columns), and thenLINEAR REPRESENTATIONS

395

Fig. 21.3 Two typical mother wavelets. On the left, a Daubechies mother wavelet, and on
the right, a Meyer mother wavelet.

Fig. 21.4 Part of a 2-D wavelet basis.

396

FEATURE EXTRACTION BY ICA

the 1-D wavelet transform of the results of this transform. Some 2-D wavelet basis
vectors are shown in Fig. 21.4.
The wavelet representation also has the important property of being localized bothin space and in frequency, just like the Gabor transform. Important differences are
the following: There is no phase parameter, and the wavelets all have the same phase. Thus,

all the basis functions look the same, whereas in Gabor analysis, we have thecouples given by the real and imaginary parts. Thus we have basis vectors of
two different phases, and moreover the phase parameter can be modified. In
Gabor analysis, some functions are similar to bars, and others are similar toedges, whereas in wavelet analysis, the basis functions are usually something
in between.

 The change in size and frequency (parameters

and in Gabor functions) are
not independent. Instead, the change in size implies a strictly corresponding
change in frequency.

 Usually in wavelets, there is no orientation parameter either. The only orientations encountered are horizontal and vertical, which come about when thehorizontal and vertical wavelets have different scales.

 The wavelet transform gives an orthogonal basis of the 1-D space. This is incontrast to Gabor functions, which do not give an orthogonal basis.

One could say that wavelet analysis gives a basis where the size and frequency parameters are given fixed values that have the nice property of giving an orthogonal basis.
On the other hand, the wavelet representation is poorer than the Gabor representationin the sense that the basis functions are not oriented, and all have the same phase.

21.2

ICA AND SPARSE CODING

The transforms just considered are fixed transforms, meaning that the basis vectors
are fixed once and for all, independent of any data. In many cases, however, it would
be interesting to estimate the transform from data. Estimation of the representationin Eq. (21.1) consists of determining the values of si and ai (x; y ) for all i and (x; y ),given a sufficient number of observations of images, or in practice, image patches
I (x; y).
For simplicity, let us restrict ourselves here to the basic case where the ai (x; y )form an invertible linear system, that is, the matrix is square. Then we can invert
the system as

A

si =

X
x;y

wi (x; y)I (x; y)(21.6)ICA AND SPARSE CODING

397

where the wi denote the inverse filters. Note that we have (using the standard ICA
notation)

ai = AAT wi = Cwi(21.7)

which shows a simple relation between the filters wi and the corresponding basis
vectors ai . The basis vectors are obtained by filtering the coefficients in wi by thefiltering matrix given by the autocorrelation matrix. For natural image data, theautocorrelation matrix is typically a symmetric low-pass filtering matrix, so the basis
vectors ai are basically smoothed versions of the filters wi .
The question is then: What principles should be used to estimate a transform fromthe data? Our starting point here is a representation principle called sparse codingthat has recently attracted interest both in signal processing and in theories on thevisual system [29, 336]. In sparse coding, the data vector is represented using a setof basis vectors so that only a small number of basis vectors are activated at the sametime. In a neural network interpretation, each basis vector corresponds to one neuron,and the coefficients si are given by their activations. Thus, only a small number ofneurons is activated for a given image patch.
Equivalently, the principle of sparse coding could be expressed by the property
that a given neuron is activated only rarely. This means that the coefficients si havesparse distributions. The distribution of si is called sparse when si has a probability
density with a peak at zero, and heavy tails, which is the case, for example, with
the Laplacian (or double exponential) density. In general, sparseness can be equatedwith supergaussianity.
In the simplest case, we can assume that the sparse coding is linear, in whichcase sparse coding fits into the framework used in this chapter. One could thenestimate a linear sparse coding transformation of the data by formulating a measureof sparseness of the components, and maximizing the measure in the set of linear
transformations. In fact, since sparsity is closely related to supergaussianity, ordinary
measures of nongaussianity, such as kurtosis and the approximations of negentropy,
could be interpreted as measures of nongaussianity as well. Maximizing sparsity
is thus one method of maximizing nongaussianity, and we saw in Chapter 8 thatmaximizing nongaussianity of the components is one method of estimating the ICs.
Thus, sparse coding can be considered as one method for ICA. At the same time,sparse coding gives a different interpretation of the goal of the transform.
The utility of sparse coding can be seen, for example, in such applications as compression and denoising. In compression, since only a small subset of the components
are nonzero for a given data point, one could code the data point efficiently by coding
only those nonzero components. In denoising, one could use some testing (thresholding) procedures to find out those components that are really active, and set to zero theother components, since their observations are probably almost purely noise. This is
an intuitive interpretation of the denoising method given in Section 15.6.

398

FEATURE EXTRACTION BY ICA

21.3

ESTIMATING ICA BASES FROM IMAGES

Thus, ICA and sparse coding give essentially equivalent methods for estimating
features from natural images, or other kinds of data sets. Here we show the resultsof such an estimation. The set of images that we used consisted of natural scenes
previously used in [191]. An example can be found in Fig. 21.7 in Section 21.4.3,
upper left-hand corner.
First, we must note that ICA applied to image data usually gives one component
representing the local mean image intensity, or the DC component. This component
normally has a distribution that is not sparse; often it is even subgaussian. Thus,
it must be treated separately from the other, supergaussian components, at least if
the sparse coding interpretation is to be used. Therefore, in all experiments we first
subtract the local mean, and then estimate a suitable sparse coding basis for the rest of
the components. Because the data then has lost one linear dimension, the dimensionof the data must be reduced, for example, using principal component analysis (PCA).
Each image was first linearly normalized so that the pixels had zero mean and unit
variance. A set of 10000 image patches (windows) of 16  16 pixels were taken at
random locations from the images. From each patch the local mean was subtracted
as just explained. To remove noise, the dimension of the data was reduced to 160.
The preprocessed dataset was used as the input to the FastICA algorithm, using thetanh nonlinearity.
Figure 21.5 shows the obtained basis vectors. The basis vectors are clearly
localized in space, as well as in frequency and orientation. Thus the features areclosely related to Gabor functions. In fact, one can approximate these basis functions
by Gabor functions, so that for each basis vector one minimizes the squared error
between the basis vector and a Gabor function; see Section 4.4. This gives very goodfits, and shows that Gabor functions are a good approximation. Alternatively, one
could characterize the ICA basis functions by noting that many of them could beinterpreted as edges or bars.
The basis vectors are also related to wavelets in the sense that they represent
more or less the same features in different scales. This means that the frequency andthe size of the envelope (i.e. the area covered by the basis vectors) are dependent.
However, the ICA basis vectors have many more degrees of freedom than wavelets.
In particular, wavelets have only two orientations, whereas ICA vectors have many
more, and wavelets have no phase difference, whereas ICA vectors have very different
phases. Some recent extensions of wavelets, such as curvelets, are much closer to
ICA basis vectors, see [115] for a review.

21.4

IMAGE DENOISING BY SPARSE CODE SHRINKAGE

In Section 15.6 we discussed a denoising method based on the estimation of thenoisy ICA model [200, 207]. Here we show how to apply this method to image
denoising. We used as data the same images as in the preceding section. To reduce
computational load, here we used image windows of 8  8 pixels. As explained inIMAGE DENOISING BY SPARSE CODE SHRINKAGE

399

Fig. 21.5 The ICA basis vectors of natural image patches (windows). The basis vectors
give features that are localized in space, frequency, and orientation, thus resembling Gaborfunctions.

Section 15.6, the basis vectors were further orthogonalized; thus the basis vectors
could be considered as orthogonal sparse coding rather than ICA.
21.4.1

Component statistics

Since sparse code shrinkage is based on the property that individual components inthe transform domain have sparse distributions, we first investigate how well thisrequirement holds. At the same time we can see which of the parameterizations in
Section 15.5.2 can be used to approximate the underlying densities.
Measuring the sparseness of the distributions can be done by almost any nongaussianity measure. We have chosen the most widely used measure, the normalized
kurtosis. Normalized kurtosis is defined as
4(s) = (EEffss2 gg)2 ? 3(21.8)The kurtoses of components in our data set were about 5, on the average. Orthogonalization did not very significantly change the kurtosis. All the components weresupergaussian.
Next, we compared various parametrizations in the task of fitting the observed
densities. We picked one component at random from the orthogonal 8  8 sparse coding transform for natural scenes. First, using a nonparametric histogram technique,400

FEATURE EXTRACTION BY ICA

Fig. 21.6 Analysis of a randomly selected component from the orthogonalized ICA transforms of natural scenes, with window size 8 8. Left: Nonparametrically estimated logdensities (solid curve) vs. the best parametrization (dashed curve). Right: Nonparametricshrinkage nonlinearity (solid curve) vs. that given by our parametrization (dashed curve).
(Reprinted from [207], reprint permission from the IEEE Press c 2001 IEEE.)

we estimated the density of the component, and from this representation derived thelog density and the shrinkage nonlinearity shown in Fig. 21.6. Next, we fitted theparametrized densities discussed in Section 15.5.2 to the observed density. Note thatin each case, the densities were sparser than the Laplacian density, and thus the very
sparse parametrization in (15.25) was used. In can be seen that the density and theshrinkage nonlinearity derived from the density model match quite well those given
by nonparametric estimation.
Thus we see that the components of the sparse coding bases found are highly
supergaussian for natural image data; the sparsity assumption is valid.
21.4.2

Remarks on windowing

The theory of sparse code shrinkage was developed for general random vectors. When
applying this framework to images, certain problems arise. The simplest way to apply
the method to images would be to simply divide the image into windows, and denoise
each such window separately. This approach, however, has a couple of drawbacks:
statistical dependencies across the synthetic edges are ignored, resulting in a blocking
artifact, and the resulting procedure is not translation invariant: The algorithm issensitive to the precise position of the image with respect to the windowing grid.
We have solved this problem by taking a sliding window approach. This means that
we do not divide the image into distinct windows; rather we denoise every possible
N  N window of the image. We then effectively have N 2 different estimated values
for each pixel, and select the final result as the mean of these values. Althoughoriginally chosen on rather heuristic grounds, the sliding window method can bejustified by two interpretations.
The first interpretation is spin-cycling. The basic version of the recently introduced
wavelet shrinkage method is not translation-invariant, because this is not a propertyof the wavelet decomposition in general. Thus, Coifman and Donoho [87] suggested

INDEPENDENT SUBSPACES AND TOPOGRAPHIC ICA

401

performing wavelet shrinkage on all translated wavelet decompositions of the data,
and taking the mean of these results as the final denoised signal, calling the obtained
method spin-cycling. It is easily seen that our sliding window method is then precisely
spin-cycling of the distinct window algorithm.
The second interpretation of sliding windows is due to the method of frames.
Consider the case of decomposing a data vector into a linear combination of a set ofgiven vectors, where the number of given vectors is larger than the dimensionality ofthe data, i.e. given and , where is an m-by-n matrix (m < n), find the vector ,in =: This has an infinite number of solutions. The classic way is to select thesolution with minimum norm, given by ^ = + ; where + is the pseudoinverseof . This solution is often referred to as the method of frames solution [102]. (Moreon this case of “overcomplete bases” can be found in Chapter 16.)
Now consider each basis window in each possible window position of the image
as an overcomplete basis for the whole image. Then, if the transform we use isorthogonal, the sliding window algorithm is equivalent to calculating the method offrames decomposition, shrinking each component, and reconstructing the image.

x As
s
A

21.4.3

x

A

A

s

s A x

A

Denoising results

To begin the actual denoising experiments, a random image from the natural scene
collection was chosen for denoising, and gaussian noise of standard deviation 0:3 wasadded (compared to a standard deviation of 1:0 for the original image). This noisy
version was subsequently denoised using the Wiener filter (Section 4.6.2) to give abaseline comparison. Then, the sparse code shrinkage method was applied using
the estimated orthogonalized ICA transform ( 8  8 windows), with the component
nonlinearities as given by the appropriate estimated parametrization. Figure 21.7
shows a typical result [207]. Visually, it seems that sparse code shrinkage gives thebest noise reduction while retaining the features in the image. The Wiener filter does
not really eliminate the noise. It seems as though our method is performing like afeature detector, in that it retains those features that are clearly visible in the noisy
data but cuts out anything that is probably a result of the noise. Thus, it reduces noise
effectively due to the nonlinear nature of the shrinkage operation.
Thus, we see that sparse code shrinkage is a promising method of image denoising.
The denoising result is qualitatively quite different from those given by traditional
filtering methods, and more along the lines of wavelet shrinkage and coring results[116, 403, 476].

21.5

INDEPENDENT SUBSPACES AND TOPOGRAPHIC ICA

The basic feature extraction schemes use linear features and linear filters as we have so
far used in this chapter. More sophisticated methods can be obtained by introducing
nonlinearities in the system. In the context of ICA, this could mean nonlinearities
that take into account the dependencies of the linear features.

402

FEATURE EXTRACTION BY ICA

Fig. 21.7 Denoising a natural scene (noise level 0.3). Top left: The original image. Topright: Noise added. Bottom left: After wiener filtering. Bottom right: Results after sparsecode shrinkage. (Reprinted from [207], reprint permission from the IEEE Press c 2001 IEEE.)NEUROPHYSIOLOGICAL CONNECTIONS

403

In fact, it is not possible in general to decompose a random vector into independent
components. One can always obtain uncorrelated components, and this is what we
obtain with FastICA. In image feature extraction, however, one can clearly see that
the ICA components are not independent by using any measure of higher-ordercorrelations. Such higher-order correlations were discussed in Section 20.2, in which
extensions of the ICA model were proposed to take into account some of the remaining
dependencies.
Here we apply two of the extensions discussed in Section 20.2, independent
subspace analysis and topographic ICA, to image feature extraction [205, 204, 206].
These give interesting extensions of the linear feature framework. The data andpreprocessing were as in Section 21.3.
Figure 21.8 shows the basis vectors of the 40 feature subspaces, when subspacedimension was chosen to be 4. It can be seen that the basis vectors associated witha single subspace all have approximately the same orientation and frequency. Theirlocations are not identical, but close to each other. The phases differ considerably.
Thus, the norm of the projection onto the subspace is relatively independent of thephase of the input. This is in fact what the principle of invariant-feature subspaces,
one of the inspirations for independent subspace analysis, is all about. Every feature
subspace can thus be considered a generalization of a quadrature-phase filter pair[373], giving a nonlinear feature that is localized in orientation and frequency, butinvariant to phase and somewhat invariant to location shifts. For more details, see[204].
In topographic ICA, the neighborhood function was defined so that every neighborhood consisted of a 3  3 square of 9 units on a 2-D torus lattice [247]. The
obtained basis vectors are shown in Fig. 21.9. The basis vectors are rather similar
to those obtained by ordinary ICA of image data. In addition, they have a clear
topographic organization. In fact, in Section 20.2.3 we discussed the connection
between independent subspace analysis and topographic ICA; this connection can befound in Fig. 21.9. Two neighboring basis vectors in Fig. 21.9 tend to be of the sameorientation and frequency. Their locations are near to each other as well. In contrast,
their phases are very different. This means that a neighborhood of such basis vectors
is similar to an independent subspace. For more details, see [206].

21.6

NEUROPHYSIOLOGICAL CONNECTIONSIn addition to the signal processing applications, we must not forget that sparse coding
was originally developed as a model of the representation of images in the primaryvisual cortex of mammals (V1).
The filters wi (x; y ) can then be identified as the receptive fields of the simple cells
in the cortex, and the si are their activities when presented with a given image patch
I (x; y). It has been shown [336] that when this model is estimated with input dataconsisting of patches of natural scenes, the obtained filters wi (x; y ) have the threeprincipal properties of simple cells in V1: they are localized, oriented, and bandpass.
The obtained filters wi have been compared quantitatively with those measured by

404

FEATURE EXTRACTION BY ICA

Fig. 21.8 Independent subspaces of natural image data. The model gives Gabor-like basis
vectors for image windows. Every group of four basis vectors corresponds to one independentfeature subspace, or complex cell. Basis vectors in a subspace are similar in orientation,
location and frequency. In contrast, their phases are very different. (Adapted from [204].)

CONCLUDING REMARKS

405

Fig. 21.9 Topographic ICA of natural image data. This gives Gabor-like basis vectors as
well. Basis vectors that are similar in orientation, location and/or frequency are close to eachother. The phases of nearby basis vectors are very different, giving each neighborhood a phaseinvariance. (Adapted from [206].)

single-cell recordings of the macaque cortex, and a good match for most of theparameters was found [443, 442].
Independent subspaces, on the other hand, can be considered as a model ofcomplex cells [204], which is the other principal class of cells analyzing visual input
in the cortex. In fact, the invariances that emerge are quite similar to those found incomplex cells. Finally, topographic ICA can be seen as a model of the topographicorganization of the cells in the visual cortex [206].

21.7

CONCLUDING REMARKSICA can be used to extract independent features from different kinds of data. This ispossible by taking patches (windows) from the signals and considering these as themulti-dimensional signals on which ICA is applied. This is closely related to sparse
coding, which means extracting features that have the property that only a small
number of features is simultaneously active. Here we considered feature extraction
from natural images, in which case we obtain a decomposition that is closely related
to the one given by wavelets or Gabor analysis. The features are localized in space aswell as in frequency, and are oriented. Such features can be used in image processingin the same way as wavelet and Gabor methods, and they can also be used as modelsof the receptive fields of simple cells in the visual cortex.

406

FEATURE EXTRACTION BY ICA

In addition to the linear feature extraction by ICA, one can use extensions of
ICA to obtain nonlinear features. Independent subspace analysis gives featureswith invariance with respect to location and phase, and topographic ICA gives a
topographic organization for the features, together with the same invariances. Thesemodels are useful for investigating the higher-order correlations between the basic“independent” components. Higher-order correlations between wavelet or Gaborcoefficients have been investigated in [404, 478] as well. See also [273] for a mixture
model based on ICA, where the different mixtures can be interpreted as contexts.
The ICA framework can also be used for feature extraction from other kinds ofdata, for example, color and stereo images [186, 187], video data [442], audio data[37], and hyperspectral data [360].

22
Brain Imaging Applications

With the advent of new anatomical and functional brain imaging methods, it is now
possible to collect vast amounts of data from the living human brain. It has thus
become very important to extract the essential features from the data to allow aneasier representation or interpretation of their properties. This is a very promisingarea of application for independent component analysis (ICA). Not only is this anarea of rapid growth and great importance; some kinds of brain imaging data also
seem to be quite well described by the ICA model. This is especially the casewith electroencephalograms (EEG) and magnetoencephalograms (MEG), which arerecordings of electric and magnetic fields of signals emerging from neural currents
within the brain. In this chapter, we review some of these brain imaging applications,concentrating on EEG and MEG.

22.1
22.1.1

ELECTRO- AND MAGNETOENCEPHALOGRAPHYClasses of brain imaging techniques

Several anatomical and functional imaging methods have been developed to study theliving human brain noninvasively, that is, without any surgical procedures. One classof methods gives anatomical (structural) images of the brain with a high spatial resolution, and include computerized X-ray tomography (CT) and magnetic resonanceimaging (MRI). Another class of methods gives functional information on whichparts of the brain are activated at a given time. Such brain imaging methods can help
in answering the question: What parts of the brain are needed for a given task?
407

408

BRAIN IMAGING APPLICATIONS

Well-known functional brain mapping methods include positron emission tomography (PET) and functional MRI (fMRI), which are based on probing the changes
in metabolic activity. The time resolution of PET and fMRI is limited, due to theslowness of the metabolic response in the brain, which is in the range of a few
seconds.
Here we concentrate on another type of functional brain imaging methods that arecharacterized by a high time resolution. This is possible by measuring the electricalactivity within the brain. Electrical activity is the fundamental means by which
information is transmitted and processed in the nervous system. These methods are
the only noninvasive ones that provide direct information about the neural dynamics
on a millisecond scale. As a trade-off, the spatial resolution is worse than in fMRI,
being about 5 mm, under favorable conditions. The basic methods in this classare electroencephalography (EEG) and magnetoencephalography (MEG) [317, 165],
which we describe next. Our exposition is based on the one in [447]; see also [162].
22.1.2

Measuring electric activity in the brain

Neurons and potentials The human brain consists of approximately 1010 to1011 neurons [230]. These cells are the basic information-processing units. Signals
between neurons are transmitted by means of action potentials, which are very shortbursts of electrical activity. The action potential is transformed in the receiving
neuron to what is called a postsynaptic potential that is longer in duration, thoughalso weaker. Single action potentials and postsynaptic potentials are very weak andcannot be detected as such by present noninvasive measurement devices.
Fortunately, however, neurons that have relatively strong postsynaptic potentials
at any given time tend to be clustered in the brain. Thus, the total electric current
produced in such a cluster may be large enough to be detected. This can be done bymeasuring the potential distribution on the scalp by placing electrodes on it, whichis the method used in EEG. A more sophisticated method is to measure the magnetic
fields associated with the current, as is done in MEG.
EEG and MEG The total electric current in an activated region is often modeled
as a dipole. It can be assumed that in many situations, the electric activity of the brain
at any given point of time can be modeled by only a very small number of dipoles.
These dipoles produce an electric potential as well as a magnetic field distributionthat can be measured outside the head. The magnetic field is more local, as it does
not suffer from the smearing caused by the different electric conductivities of theseveral layers between the brain and the measuring devices seen in EEG. This is oneof the main advantages of MEG, as it leads to a much higher spatial resolution.
EEG is used extensively for monitoring the electrical activity within the humanbrain, both for research and clinical purposes. It is in fact one of the most widespreadbrain mapping techniques to date. EEG is used both for the measurement of spontaneous activity and for the study of evoked potentials. Evoked potentials are activity
triggered by a particular stimulus that may be, for example, auditory or somatosensory. Typical clinical EEG systems use around 20 electrodes, evenly distributed

ELECTRO- AND MAGNETOENCEPHALOGRAPHY

409

over the head. State-of-the-art EEGs may consist of a couple hundred sensors. The
signal-to noise ratio is typically quite low: the background potential distribution isof the order of 100 microvolts, whereas the evoked potentials may be two orders ofmagnitude weaker.
MEG measurements give basically very similar information to EEG, but with ahigher spatial resolution. MEG is mainly used for basic cognitive brain research.
To measure the weak magnetic fields of the brain, superconducting quantum interference devices (SQUIDs) are needed. The measurements are carried out inside amagnetically shielded room. The superconducting characteristics of the device are
guaranteed through its immersion in liquid helium, at a temperature of ?269C.
The experiments is this chapter were conducted using a Neuromag-122 TM device,
manufactured by Neuromag Ltd., and located at the Low Temperature Laboratory of
the Helsinki University of Technology. The whole-scalp sensor array in this device
is composed of 122 sensors (planar gradiometers), organized in pairs at 61 locations
around the head, measuring simultaneously the tangential derivatives of the magnetic
field component normal to the helmet-shaped bottom of the dewar. The sensors aremainly sensitive to currents that are directly below them, and tangential to the scalp.
22.1.3

Validity of the basic ICA model

The application of ICA to the study of EEG and MEG signals assumes that several
conditions are verified, at least approximately: the existence of statistically independent source signals, their instantaneous linear mixing at the sensors, and thestationarity of the mixing and the independent components (ICs).
The independence criterion considers solely the statistical relations between theamplitude distributions of the signals involved, and not the morphology or physiologyof neural structures. Thus, its validity depends on the experimental situation, andcannot be considered in general.
Because most of the energy in EEG and MEG signals lies below 1 kHz, the socalled quasistatic approximation of Maxwell equations holds, and each time instancecan be considered separately [162]. Therefore, the propagation of the signals is
immediate, there is no need for introducing any time-delays, and the instantaneous
mixing is valid.
The nonstationarity of EEG and MEG signals is well documented [51]. Whenconsidering the underlying source signals as stochastic processes, the requirementof stationarity is in theory necessary to guarantee the existence of a representative
distribution of the ICs. Yet, in the implementation of batch ICA algorithms, the data
are considered as random variables, and their distributions are estimated from thewhole data set. Thus, the nonstationarity of the signals is not really a violation of theassumptions of the model. On the other hand, the stationarity of the mixing matrix A
is crucial. Fortunately, this assumption agrees with widely accepted neuronal source
models [394, 309].

410

BRAIN IMAGING APPLICATIONS

22.2

ARTIFACT IDENTIFICATION FROM EEG AND MEG

As a first application of ICA on EEG and MEG signals, we consider separation of
artifacts. Artifacts mean signals not generated by brain activity, but by some external
disturbances, such as muscle activity. A typical example is ocular artifacts, generated
by eye muscle activity.
A review on artifact identification and removal, with special emphasis on the ocularones, can be found in [56, 445]. The simplest, and most widely used method consistsof discarding the portions of the recordings containing attributes (e.g., amplitude
peak, frequency contents, variance and slope) that are typical of artifacts and exceed
a determined threshold. This may lead to significant loss of data, and to complete
inability of studying interesting brain activity occuring near or during strong eye
activity, such as in visual tracking experiments.
Other methods include the subtraction of a regression portion of one or more
additional inputs (e.g., from electrooculograms, electrocardiograms, or electromyograms) from the measured signals. This technique is more likely to be used in EEG
recordings, but may, in some situations, be applied to MEG. It should be noted that
this technique may lead to the insertion of undesirable new artifacts into the brain
recordings [221]. Further methods include the signal-space projection [190], andsubtracting the contributions of modeled dipoles accounting for the artifact [45]. In
both of these latter methods we need either a good model of the artifactual source
or a considerable amount of data where the amplitude of the artifact is much higher
than that of the EEG or MEG.
ICA gives a method for artifact removal where we do not need an accurate modelof the process that generated the artifacts; this is the blind aspect of the method.
Neither do we need specified observation intervals that contain mainly the artifact,
nor additional inputs; this is the unsupervised aspect of the method. Thus ICA
gives a promising method for artifact identification and removal. It was shown in[445, 446] and [225] that artifacts can indeed be estimated by ICA alone. It turns outthat the artifacts are quite independent from the rest of the signal, and thus even thisrequirement of the model is reasonably well fulfilled.
In the experiments on MEG artifact removal [446], the MEG signals were recorded
in a magnetically shielded room with the 122-channel whole-scalp magnetometer
described above. The test person was asked to blink and make horizontal saccades,in order to produce typical ocular (eye) artifacts. Moreover, to produce myographic(muscle) artifacts, the subject was asked to bite his teeth for as long as 20 seconds.
Yet another artifact was created by placing a digital watch one meter away fromthe helmet into the shielded room. Figure 22.1 presents a subset of 12 artifactcontaminated MEG signals, from a total of 122 used in the experiment. Several
artifact structures are evident from this figure, such as eye and muscle activity.
The results of artifact extraction using ICA are shown in Fig. 22.2. Components
IC1 and IC2 are clearly the activations of two different muscle sets, whereas IC3
and IC5 are, respectively, horizontal eye movements and blinks. Furthermore, otherdisturbances with weaker signal-to-noise ratio, such as the heart beat and a digital
watch, are extracted as well (IC4 and IC8, respectively). IC9 is probably a faulty

411

ANALYSIS OF EVOKED MAGNETIC FIELDS
saccadesblinking

biting

MEG
1
1
2
2
3
3
4
4
5
5
6
6

Fig. 22.1 A subset of 12 spontaneous MEG signals from the frontal, temporal and occipitalareas. The data contains several types of artifacts, including ocular and muscle activity, thecardiac cycle, and environmental magnetic disturbances. (Adapted from [446].)

sensor. ICs 6 and 7 may be breathing artifacts, or alternatively artificial bumps
caused by overlearning (Section 13.2.2). For each component the left, back and rightviews of the field patterns are shown. These field patterns can be computed from thecolumns of the mixing matrix.

22.3

ANALYSIS OF EVOKED MAGNETIC FIELDS

Evoked magnetic fields, i.e., the magnetic fields triggered by an external stimulus, are
one of the fundamental research methods in cognitive brain research. State-of-the-art
approaches for processing magnetic evoked fields are often based on a careful expert
scrutiny of the complete data, which can be either in raw format or averaged over
several responses to repeating stimuli. At each time instance, one or several neuralsources are modeled, often as dipoles, so as to produce as good a fit to the data as
possible [238]. The choice of the time instances where this fitting should be made,as well as the type of source models employed, are therefore crucial. Using ICA, we
can again obtain a blind decomposition without imposing any a priori structure on
the measurements.
The application of ICA in event related studies was first introduced in the blindseparation of auditory evoked potentials in [288]. This method has been further
developed using magnetic auditory and somatosensory evoked fields in [449, 448].
Interestingly, the most significant independent components that were found in thesestudies seem to be of dipolar nature. Using a dipole model to calculate the source

412

BRAIN IMAGING APPLICATIONS

IC1

IC2

IC3

IC4

IC5

IC6

IC7

IC8

IC9
10 s

Fig. 22.2 Artifacts found from MEG data, using the FastICA algorithm. Three views of the
field patterns generated by each independent component are plotted on top of the respective
signal. Full lines correspond to magnetic flux exiting the head, whereas the dashed lines
correspond to the flux inwards. Zoomed portions of some of the signals are shown as well.
(Reprinted from [446], reprint permission and copyright by the MIT Press.)

ICA APPLIED ON OTHER MEASUREMENT TECHNIQUES

413

locations, we have found them to fall on very plausible brain areas. Thus, ICA
validates the conventional dipole modeling assumption in these studies. Futurestudies, though, will probably find cases where the dipole model is too restrictive.
In [448], ICA was shown to be able to differentiate between somatosensory andauditory brain responses in the case of vibrotactile stimulation, which, in addition totactile stimulation, also produced a concomitant sound. Principal component analysis(PCA) has often been used to decompose signals of this kind, but as we have seenin Chapter 7, it cannot really separate independent signals. In fact, computing theprincipal components of these signals, we see that most of the principal components
still represent combined somatosensory and auditory responses [448]. In contrast,computing the ICs, the locations of the equivalent current sources fall on the expected
brain regions for the particular stimulus, showing separation by modality.
Another study was conducted in [449], using only averaged auditory evoked fields.
The stimuli consisted of 200 tone bursts that were presented to the subject’s rightear, using 1 s interstimulus intervals. These bursts had a duration of 100 ms, anda frequency of 1 kHz. Figure 22.3 shows the 122 averages of the auditory evoked
responses over the head. The insert, on the left, shows a sample enlargement of suchaverages, for an easier comparison with the results depicted in the next figures.
Again, we see from Figs 22.4 a and 22.4 b that PCA is unable to resolve thecomplex brain response, whereas the ICA technique produces cleaner and sparser
response components. For each component presented in Fig. 22.4 a and Fig. 22.4 b,
left, top and, right side views of the corresponding field pattern are shown. Note thatthe first principal component exhibits clear dipole-like pattern both over the left andthe right hemispheres, corroborating the idea of an unsuccessful segmentation of theevoked response. Subsequent principal components, however, tend to have less andless structured patterns.
From the field patterns associated with the independent components we see thatthe evoked responses of the left hemisphere are isolated in IC1 and IC4. IC2 hasstronger presence over the right hemisphere, and IC3 fails to show any clear field
pattern structure. Furthermore, we can see that IC1 and IC2 correspond to responses
typically labeled as N1m, with the characteristic latency of around 100 ms after
the onset of the stimulation. The shorter latency of IC1, mainly reflecting activity
contralateral to the stimulated ear, agrees with the known information available forsuch studies.

22.4

ICA APPLIED ON OTHER MEASUREMENT TECHNIQUESIn addition to the EEG/MEG results reported here, ICA has been applied to otherbrain imaging and biomedical signals as well: Functional magnetic resonance images (fMRI). One can use ICA in two different ways, separating either independent spatial activity patterns [297], or
independent temporal activation patterns [50]. A comparison of the two modes

414

BRAIN IMAGING APPLICATIONSMEG sample

Left side

Right side

MEG25

MEG10
(MEG-R)

MEG60(MEG-L)

MEG83-1 0 1 2 3 4 5

Fig. 22.3 Averaged auditory evoked responses to 200 tones, using MEG. Channels MEG10and MEG60 are used in Fig. 22.4 as representatives of one left-hemisphere and one righthemisphere MEG signal. Each tick in the MEG sample corresponds to 100 ms, going from
100 ms before stimulation onset to 500 ms after. (Adapted from [449].)can be found in [367]. A combination of the two modes can be achieved byspatiotemporal ICA, see Section 20.1.4.

 Optical imaging means directly “photographing” the surface of the brain after

making a hole in the skull. Application of ICA can be found in [374, 396]. As
in the case of fMRI signals, this is a case of separating image mixtures as in theexample in Fig. 12.4. Some theory for this particular case is further developedin [164, 301]; also the innovations processes may be useful (see Section 13.1.3
and [194]).

 Outside the area of brain imaging, let us mention applications to the removal of

artifacts from cardiographic (heart) signals [31, 459] and magnetoneurographic
signals [482]. The idea is very similar to the one used in MEG artifact removal.
Further related work is in [32, 460]. Another neuroscientific application is inintracellular calcium spike analysis [375].

22.5

CONCLUDING REMARKSIn this chapter we have shown examples of ICA in the analysis of brain signals.
First, ICA was shown to be suitable for extracting different types of artifacts from
EEG and MEG data, even in situations where these disturbances are smaller than the
background brain activity.

CONCLUDING REMARKS

415

PC1

PC2

PC3

PC4

PC5

-1 0 1 2 3 4 5

a)

IC1

IC2

IC3

IC4

-1 0 1 2 3 4 5b)Fig. 22.4 Principal (a) and independent (b) components found from the auditory evoked fieldstudy. For each component, both the activation signal and three views of the corresponding
field pattern are plotted. (Adapted from [449].)

416

BRAIN IMAGING APPLICATIONS

Second, ICA can be used to decompose evoked fields or potentials. For example,
ICA was able to differentiate between somatosensory and auditory brain responses in
the case of vibrotactile stimulation. Also, it was able to discriminate between the ipsiand contralateral principal responses in the case of auditory evoked potentials. In
addition, the independent components, found with no other modeling assumption thantheir statistical independence, exhibit field patterns that agree with the conventional
current dipole models. The equivalent current dipole sources corresponding to the
independent components fell on the brain regions expected to be activated by theparticular stimulus.
Applications of ICA have been proposed for analysis of other kinds of biomedical
data as well, including fMRI, optical imaging, and ECG.

23
Telecommunications

This chapter deals with applications of independent component analysis (ICA) andblind source separation (BSS) methods to telecommunications. In the following,
we concentrate on code division multiple access (CDMA) techniques, because thisspecific branch of telecommunications provides several possibilities for applying
ICA and BSS in a meaningful way. After an introduction to multiuser detection andCDMA communications, we present mathematically the CDMA signal model andshow that it can be cast in the form of a noisy matrix ICA model. Then we discussin more detail three particular applications of ICA or BSS techniques to CDMA
data. These are a simplified complexity minimization approach for estimating fading
channels, blind separation of convolutive mixtures using an extension of the natural
gradient algorithm, and improvement of the performance of conventional CDMA
receivers using complex-valued ICA. The ultimate goal in these applications is todetect the desired user’s symbols, but for achieving this intermediate quantities suchas fading channel or delays must usually be estimated first. At the end of the chapter,
we give references to other communications applications of ICA and related blindtechniques used in communications.

23.1

MULTIUSER DETECTION AND CDMA COMMUNICATIONS

In wireless communication systems, like mobile phones, an essential issue is divisionof the common transmission medium among several users. This calls for a multiple
access communication scheme. A primary goal in designing multiple access systems
is to enable each user of the system to communicate despite the fact that the other
417

418

TELECOMMUNICATIONS

FDMA

TDMA
time

time

frequency
code

CDMA
time

frequencycode

frequency
code

Fig. 23.1 A schematic diagram of the multiple access schemes FDMA, TDMA, and CDMA
[410, 382].

users occupy the same resources, possibly simultaneously. As the number of users inthe system grows, it becomes necessary to use the common resources as efficiently
as possible. These two requirements have given rise to a number of multiple access
schemes.
Figure 23.1 illustrates the most common multiple access schemes [378, 410, 444].
In frequency division multiple access (FDMA), each user is given a nonoverlapping
frequency slot in which one and only one user is allowed to operate. This prevents
interference of other users. In time division multiple access (TDMA) a similar
idea is realized in the time domain, where each user is given a unique time period(or periods). One user can thus transmit and receive data only during his or her
predetermined time interval while the others are silent at the same time.
In CDMA [287, 378, 410, 444], there is no disjoint division in frequency and time
spaces, but each user occupies the same frequency band simultaneously. The users
are now identified by their codes, which are unique to each user. Roughly speaking,each user applies his unique code to his information signal (data symbols) before
transmitting it through a common medium. In transmission different users’ signals
become mixed, because the same frequencies are used at the same time. Each user’stransmitted signal can be identified from the mixture by applying his unique code at
the receiver.
In its simplest form, the code is a pseudorandom sequence of 1s, also called achip sequence or spreading code. In this case we speak about direct sequence (DS)
modulation [378], and call the multiple access method DS-CDMA. In DS-CDMA,each user’s narrow-band data symbols (information bits) are spread in frequency
before actual transmission via a common medium. The spreading is carried out by
multiplying each user’s data symbols (information bits) by his unique wide-bandchip sequence (spreading code). The chip sequence varies much faster than theMULTIUSER DETECTION AND CDMA COMMUNICATIONS

419+1
binary data
time
-1

+1
spreading code
time
-1

+1
modulated data
time
-1

Tc
T

Fig. 23.2 Construction of a CDMA signal [382]. Top: Binary user’s symbols to be transmitted. Middle: User’s specific spreading code (chip sequence). Bottom: Modulated CDMAsignal, obtained by multiplying user’s symbols by the spreading code.

information bit sequence. In the frequency domain, this leads to spreading of thepower spectrum of the transmitted signal. Such spread spectrum techniques areuseful because they make the transmission more robust against disturbances causedby other signals transmitted simultaneously [444].
Example 23.1 Figure 23.2 shows an example of the formation of a CDMA signal. On
the topmost subfigure, there are 4 user’s symbols (information bits) ?1; +1; ?1; +1
to be transmitted.The middle subfigure shows the chip sequence (spreading code).
It is now ?1; +1; ?1; ?1; +1. Each symbol is multiplied by the chip sequence in asimilar manner. This yields the modulated CDMA signal on the bottom row of Fig.
23.2, which is then transmitted. The bits in the spreading code change in this case 5times faster that the symbols.
Let us denote the mth data symbol (information bit) by bm , and the chip sequenceby s(t). The time period of the chip sequence is T (see Fig. 23.2), so that s(t) 2
f?1; +1g when t 2 [0; T ), and s(t) = 0 when t 2= [0; T ). The length of the chipsequence is C chips, and the time duration of each chip is Tc = T=C . The number ofbits in the observation interval is denoted by N . In Fig. 23.2, the observation interval
contains N = 4 symbols, and the length of the chip sequence is C = 5.

420

TELECOMMUNICATIONSUsing these notations, the CDMA signal r(t) at time t arising in this simple
example can be writtenr(t) =

NX
m=1

bm s(t ? mT )(23.1)In the reception of the DS-CDMA signal, the final objective is to estimate thetransmitted symbols. However, both code timing and channel estimation are often
prerequisite tasks. Detection of the desired user’s symbols is in CDMA systems
more complicated than in the simpler TDMA and FDMA systems used previously inmobile communications. This is because the spreading code sequences of different
users are typically nonorthogonal, and because several users are transmitting theirsymbols at the same time using the same frequency band. However, CDMA systems
offer several advantages over more traditional techniques [444, 382]. Their capacity
is larger, and it degrades gradually with increasing number of simultaneous users
who can be asynchronous [444]. CDMA technology is therefore a strong candidate
for future global wireless communications systems. For example, it has already beenchosen as the transmission technique for the European third generation mobile system
UMTS [334, 182], which will provide useful new services, especially multimedia
and high-bit-rate packet data.
In mobile communications systems, the required signal processing differs in thebase station (uplink) from that in the mobile phone (downlink). In the base station,
all the signals sent by different users must be detected, but there is also much moresignal processing capacity available. The codes of all the users are known but
their time delays are unknown. For delay estimation, one can use for example thesimple matched filter [378, 444], subspace approaches [44, 413], or the optimal but
computationally highly demanding maximum likelihood method [378, 444]. Whenthe delays have been estimated, one can estimate the other parameters such as thefading process and symbols [444].
In downlink (mobile phone) signal processing, each user knows only its own code,
while the codes of the other users are unknown. There is less processing power thanin the base station. Also the mathematical model of the signals differs slightly, sinceusers share the same channel in the downlink communications. Especially the first
two features of downlink processing call for new, efficient and simple solutions.
ICA and BSS techniques provide a promising new approach to the downlink signal
processing using short spreading codes and DS-CDMA systems.
Figure 23.3 shows a typical CDMA transmission situation in an urban environment. Signal 1 arrives directly from the base station to the mobile phone in the car.
It has the smallest time delay and is the strongest signal, because it is not attenuatedby the reflection coefficients of the obstacles in the path. Due to multipath propagation, the user in the car in Fig. 23.3 receives also weaker signals 2 and 3, whichhave longer time delays. The existence of multipath propagation allows the signal
to interfere with itself. This phenomenon is known as intersymbol interference (ISI).
Using spreading codes and suitable processing methods, multipath interference canbe mitigated [444].

421

Magnitude

MULTIUSER DETECTION AND CDMA COMMUNICATIONSTime delay

Fig. 23.3 An example of multipath propagation in urban environment.

There are several other problems that complicate CDMA reception. One of themost serious ones is multiple access interference (MAI), which arises from the factthat the same frequency band is occupied simultaneously. MAI can be alleviated byincreasing the length of the spreading code, but at a fixed chip rate, this decreases
the data rate. In addition, the near–far problem arises when signals from near andfar are received at the same time. If the received powers from different users become
too different, a stronger user will seriously interfere with the weaker ones, even if
there is a small correlation between the users’ spreading codes. In the FDMA and
TDMA systems, the near–far problem does not arise because different users havenonoverlapping frequency or time slots.
The near–far problem in the base station can be mitigated by power control, or bymultiuser detection. Efficient multiuser detection requires knowledge or estimation ofmany system parameters such as propagation delay, carrier frequency, and received
power level. This is usually not possible in the downlink. However, then blind
multiuser detection techniques can be applied, provided that the spreading codes areshort enough [382].
Still other problems appearing in CDMA systems are power control, synchronization, and fading of channels, which is present in all mobile communications systems.
Fading means variation of the signal power in mobile transmission caused for example by buildings and changing terrain. See [378, 444, 382] for more information onthese topics.

422

TELECOMMUNICATIONS

23.2

CDMA SIGNAL MODEL AND ICA

In this section, we represent mathematically the CDMA signal model which is studiedin slightly varying forms in this chapter. This type of models and the formation of
the observed data in them are discussed in detail in [444, 287, 382].
It is straightforward to generalize the simple model (23.1) for K users. The mth
symbol of the k th user is denoted by bkm , and sk () is k :th user’s binary chip sequence(spreading code). For each user k , the spreading code is defined quite similarly as in
Example 23.1. The combined signal of K simultaneous users then becomesr(t) =N X
KX

m=1 k=1

bkm sk (t ? mT ) + n(t)(23.2)

where n(t) denotes additive noise corrupting the observed signal.
The signal model (23.2) is not yet quite realistic, because it does not take into
account the effect of multipath propagation and fading channels. Including thesefactors in (23.2) yields our desired downlink CDMA signal model for the observed
data r(t) at time t:r(t) =N X
KX

m=1 k=1

bkm

L
X
l=1

alm sk (t ? mT ? dl ) + n(t)(23.3)

Here the index m refers to the symbol, k to the user, and l to the path. The termdl denotes the delay of the lth path, which is assumed to be constant during theobservation interval of N symbol bits. Each of the K simultaneous users has L
independent transmission paths. The term alm is the fading factor of the lth pathcorresponding to the mth symbol.
In general, the fading coefficients alm are complex-valued. However, we canapply standard real-valued ICA methods to the data (23.3) by using only the real partof it. This is the case in the first two approaches to be discussed in the next two
sections, while the last method in Section 23.5 directly uses complex data.
The continuous time data (23.3) is first sampled using the chip rate, so that C
equispaced samples per symbol are taken. From subsequent discretized equispaced
data samples r[n], C -length data vectors are then collected:

rm = (r[mC ]; r[mC + 1]; : : : ; r[(m + 1)C ? 1])T

(23.4)

Then the model (23.3) can be written in vector form as [44]

rm =
n

K X
L
X

k=1 l=1[al;m?1 bk;m?1 gkl + al;m bk;m gkl ] + nm

(23.5)

where m denotes the noise vector consisting of subsequent C last samples of noisen(t). The vector kl denotes the “early” part of the code vector, and kl the “late”
part, respectively. These vectors are given bygg

gkl = [ sk [C ? dl + 1]; : : : ; sk [C ]; 0Tdl ]T(23.6)

423

CDMA SIGNAL MODEL AND ICA

gkl = [0Tdl ; sk [1]; : : : ; sk [C ? dl] ]T(23.7)

Here dl is the discretized index representing the time delay, dl 2 f0; : : : ; (C ? 1)=2g,
and Tdl is a row vector having dl zeros as its elements. The early and late parts of thecode vector arise because of the time delay dl , which means that the chip sequence
generally does not coincide with the time interval of a single user’s symbol, but
extends over two subsequent bits bk;m?1 and bk;m . This effect of the time delay canbe easily observed by shifting the spreading code to the right in Fig. 23.2.
The vector model (23.5) can be expressed in compact form as a matrix model.
Define the data matrix0

R = [r ; r ; : : : ; rN ](23.8)
consisting of N subsequent data vectors ri . Then R can be represented as
R = GF + N(23.9)
where the C  2KL matrix G contains all the KL early and late code vectors
G = [ g ; g ; : : : ; gKL; gKL ](23.10)and the 2KL  N matrix F = [f : : : fN ] contains the symbols and fading terms
fm = [a ;m? b ;m? ; a mb m;(23.11)
1

2

11

11

1

1 1

1

1

1

1

: : : ; aL;m?1bK;m?1 ; aLmbKm ]TThe vector fm represents the 2KL symbols and fading terms of all the users andpaths corresponding to the mth pair of early and late code vectors.

From the physical situation, it follows that each path and user are at least approximately independent of each other [382]. Hence every product ai;m?1 bi;m?1 oraim bim of a symbol and the respective fading term can be regarded as an independentsource signal. Because each user’s subsequent transmitted symbols are assumed to
be independent, these products are also independent for a given user i. Denote theindependent sources a1;m?1 b1;m?1 ; : : : ; aLm bKm by yi (m); i = 1; : : : ; 2KL. Hereevery 2L sources correspond to each user, where the coefficient 2 follows from thepresence of the early and late parts.
To see the correspondence of (23.9) to ICA, let us write the noisy linear ICA+ in the matrix form asmodel =x As n

X = AS + N
X(23.12)x

x

SThe data matrix
has as its columns the data vectors (1); (2); : : : ; and andare similarly compiled source and noise matrices whose columns consist of thesource and noise vectors (t) and (t), respectively. Comparing the matrix CDMAsignal model (23.9) with (23.12) shows that it has the same form as the noisy linear
ICA model. Clearly, in the CDMA model (23.9) is the matrix of source signals,
is the observed data matrix, and is the unknown mixing matrix.

N

s

n

G

F

R

424

TELECOMMUNICATIONS

For estimating the desired user’s parameters and symbols, several techniques areavailable [287, 444]. Matched filter (correlator) [378, 444] is the simplest estimator,
but it performs well only if different users’ chip sequences are orthogonal or the users
have equal powers. The matched filter suffers greatly from the near–far problem,
rendering it unsuitable for CDMA reception without a strict power control. The
so-called RAKE detector [378] is a somewhat improved version of the basic matchedfilter which takes advantage of multiple propagation paths. The maximum likelihood(ML) method [378, 444] would be optimal, but it has a very high computational load,
and requires knowledge of all the users’ codes. However, in downlink reception,
only the desired user’s code is known. To remedy this problem while preserving
acceptable performance, subspace approaches have been proposed for example in[44]. But they are sensitive to noise, and fail when the signal subspace dimension
exceeds the processing gain. This easily occurs even with moderate system load
due to the multipath propagation. Some other semiblind methods proposed for theCDMA problem such as the minimum mean-square estimator (MMSE) are discussed
later in this chapter and in [287, 382, 444].
It should be noted that the CDMA estimation problem is not completely blind,because there is some prior information available. In particular, the transmitted
symbols are binary (more generally from a finite alphabet), and the spreading code
(chip sequence) is known. On the other hand, multipath propagation, possibly fading
channels, and time delays make separation of the desired user’s symbols a very
challenging estimation problem which is more complicated than the standard ICA
problem.

23.3
23.3.1ESTIMATING FADING CHANNELS
Minimization of complexity

Pajunen [342] has recently introduced a complexity minimization approach as a true
generalization of standard ICA. In his method, temporal information contained in
the source signals is also taken into account in addition to the spatial independence
utilized by standard ICA. The goal is to optimally exploit all the available informationin blind source separation. In the special case where the sources are temporally white(uncorrelated), complexity minimization reduces to standard ICA [342]. Complexity
minimization has been discussed in more detail in Section 18.3.
Regrettably, the original method for minimizing the Kolmogoroff complexity measure is computationally highly demanding except for small scale problems. But if thesource signals are assumed to be gaussian and nonwhite with significant time correlations, the minimization task becomes much simpler [344]. Complexity minimization
then reduces to principal component analysis of temporal correlation matrices. Thismethod is actually just another example of blind source separation approaches basedon second-order temporal statistics; for example [424, 43], which were discussed
earlier in Chapter 18.

ESTIMATING FADING CHANNELS

425

In the following, we apply this simplified method to the estimation of the fading channel coefficients of the desired user in a CDMA systems. Simulations withdownlink data, propagated through a Rayleigh fading channel, show noticeable performance gains compared with blind minimum mean-square error channel estimation,
which is currently a standard method for solving this problem. The material in thissection is based on the original paper [98].
We thus assume that the fading process is gaussian and complex-valued. Then theamplitude of the fading process is Rayleigh distributed; this case is called Rayleighfading (see [444, 378]). We also assume that a training sequence or a preamble isavailable for the desired user, although this may not always be the case in practice.
Under these conditions, only the desired user’s contribution in the sampled data is
time correlated, which is then utilized. The proposed method has the advantage thatit estimates code timing only implicitly, and hence it does not degrade the accuracyof channel estimation.
A standard method for separating the unknown source signals is based on minimization of the mutual information (see Chapter 10 and [197, 344]) of the separated
signals m = [y1 (m) : : : y2KL (m)]T = :f

J (y) =Xiy

H (yi ) + log j det G j(23.13)where H (yi ) is the entropy of yi (see Chapter 5). But entropy has the interpretationthat it represents the optimum averaged code length of a random variable. Hencemutual information can be expressed by using algorithmic complexity as [344]J (y) =Xi

K (yi ) + log j det G j(23.14)

where K () is the per-symbol Kolmogoroff complexity, given by the number of bits
needed to describe yi . By using prior information about the signals, the coding costs
can be explicitly approximated. For instance, if the signals are gaussian, independence becomes equivalent to uncorrelatedness. Then the Kolmogoroff complexity
can be replaced by the per-symbol differential entropy, which in this case depends on
second-order statistics only.
For Rayleigh type fading transmission channels, the prior information can be formulated by considering that the probability distributions of the mutually independentsource signals yi (m) have zero-mean gaussian distributions. Suppose we want toestimate the channel coefficients of the transmission paths, by sending a given length
constant b1m = 1 symbol sequence to the desired user. We consider the signals
yi (m), i = 1; : : : ; 2L, with i representing the indexes of the 2L sources corresponding to the first user. Then yi (m) will actually represent the channel coefficients of all
the first user’s paths. Since we assume that the channel is Rayleigh fading, then thesesignals are gaussian and time correlated. In this case, blind separation of the sources
can be achieved by using only second-order statistics. In fact, we can express the
Kolmogoroff complexity by coding these signals using principal component analysis[344].

426

TELECOMMUNICATIONS

23.3.2Channel estimation *y

Let i (m) = [yi (m); : : : ; yi (m ? D + 1)] denote the vector consisting of D last
samples of every such source signal yi (m), i = 1; : : : ; 2L. Here D is the number ofdelayed terms, showing what is the range of time correlations taken into account whenestimating the current symbol. The information contained in any of these sources canbe approximated by the code length needed for representing the D principal components, which have variances given by the eigenvalues of the temporal correlation
matrix i = E[ i (m) iT (m)] [344]. Since we assume that the transmission paths aremutually independent, the overall entropy of the source is given by summing up theentropies of the principal components. Using the result that the entropy of a gaussian
random variable is given by the logarithm of the variance, we get for the entropy ofeach source signal

C

y

y

H (yi ) = 21LXklog k2 = 21L log det Ci(23.15)Inserting this into the cost function (23.13) yields

J (y)) =X1

i 2L

log det Ci ? log j det W j(23.16)

where= ?1 is the separating matrix.
can be estimated by using a gradient descent approach
The separating matrix
for minimizing the cost function (23.16), leading to the update rule [344]

W G

W

J (y)
W = ? @ log@ W + W

(23.17)

where  is the learning rate and is the momentum term [172] that can be introduced
to avoid getting trapped into a local minimum corresponding to a secondary path.
Let iT denote the ith row vector of the separating matrix. Since only thecorrelation matrix i of the ith source depends on i , we can express the gradientof the cost function by computing the partial derivatives

w

C

W

w

@ log det Ci@wikw

with respect to the scalar elements of the vector iT = [wi1 ; : : : ; wiC ]. For thesepartial derivatives, one can derive the formula [344]@ log det Ci = 2 trace C?1 E yT @ yi
i @wiki@wik
Since yi (m) =wiT rm, we get@ yi@w = [rk;m ; : : : ; rk;m?L ]
ik

+1



(23.18)(23.19)

ESTIMATING FADING CHANNELS

427

R

defined earlier using
where rk;i is the element (k; j ) of the observation matrix
formulas (23.4) and (23.9).
What is left to do now is to compute the gradient update part due to the mapping
information. It can be written [344]log j det Wj =

C
X
i=1

log k(I ? Pi )wi k(23.20)where i = i ( iT i )?1 iT is a projection matrix onto the subspace spannedby the column vectors of the matrix
i = [ 1 ; : : : ; i?1 ]. Now the cost function
can be separated, and the different independent components can be found one byone, by taking into account the previously estimated components, contained in thesubspace spanned by the columns of the matrix i .
Since our principal interest lies in the transmission path having the largest power,
corresponding usually to the desired user, it is sufficient to estimate the first such
independent component. In this case, the projection matrix 1 becomes a zeromatrix. Then the overall gradient (23.17) for the first row 1T of the separating
matrix can be written

P

W WW

W

W

w

w

W

P
w





@ log J (y) = 1 trace C?1 E yT @ y1
1
1
D
@ w1T
@ w1T



? kw
wT k
T

1

1

(23.21)

It suffices to consider the special case where only the two last samples are taken
into account, so that the the delay D = 2. First, second-order correlations are removed from the data by whitening. This can be done easily in terms of standard
principal component analysis as explained in Chapter 6. After whitening, the subsequent separating matrix will be orthogonal, and thus the second term in Eq. (23.16)
disappears, yielding the cost functionR

J (y) X

log det Ck(23.22)

with the 2  2 autocorrelation matrices given by

Ck =



1

E[yk (m)yk (m ? 1)]

In this case, the separating vectors

E[yk (m)yk (m ? 1)]

1

(23.23)wiT can be found by maximizing sequentiallyE[yi (m)yi (m ? 1)+ yi (m ? 1)yi (m)], which is the first-order correlation coefficientof yi . It follows that the function to be maximized becomes

J (w) = wT E[rm rTm?1 + rm?1 rTm ]w

w(23.24)

So the separating vector 1T corresponding to the most important path is given by
the principal eigenvector of the matrix in Eq. (23.24). We have used a symmetricexpression for the correlation coefficients in order to avoid asymmetry when the428

TELECOMMUNICATIONS
2
1
0
−1
−2

0

100

200

300

400

500

600

700

800

900

1000

0

100

200

300

400

500

600

700

800

900

1000

0

100

200

300

400

500

600

700

800

900

1000

4
2
0
−2
−4
5

0

−5

Fig. 23.4 The original fading process (top), its estimate given by our method (middle), andestimate given by the blind MMSE method (bottom). The signal-to-noise ratio was 10 dB.

observed data set is finite. This usually improves the estimation accuracy. Finally,
we separate the desired channel coefficients by computing the quantities

a11 = w1T ~r(23.25)rr

where ~ denotes whitened data vector . This is done for all the N data vectors
contained in (23.8).
23.3.3Comparisons and discussion

We have compared the method described and derived above to a well-performing
standard method used in multiuser detection, namely the minimum mean-squareerror estimator [452, 287]. In the MMSE method, the desired signal is estimated(up to a scaling) from the formula

aMMSE = g1T Us ?s 1 UTs r

U(23.26)

where s and s are the matrices containing (in the same order) the eigenvalues and
T =N . The vector 1 isthe respective eigenvectors of the data correlation matrix
a column of the matrix defined in (23.10) corresponding to the desired user’s bit
b1;m , that is, either 11 or 11 . The quantity aMMSE is again computed for all the N
data vectors 1 ; : : : ; N . If the pilot signal consists of ones, then aMMSE provides
estimates of the channel coefficients.

r

G
g g
r

RR

g

ESTIMATING FADING CHANNELS

429

0.25

0.2

MMSE
0.15
MSE

blind
estimation
0.1

0.05

0
10

12

14

16

18

20
SNR

22

24

26

28

30

Fig. 23.5 The mean-square errors of the MMSE method and our method as the function of
the signal-to-noise ratio. The number of users was K = 6.

The algorithms were tested in a simulation using length C = 31 quasiorthogonal
gold codes (see [378]). The number of users was K = 6, and the number oftransmission paths was L = 3. The powers of the channel paths were ?5, ?5, and
0 dB respectively for every user, and the signal-to-noise ratio (SNR) varied from 30
dB to 10 dB with respect to the main path. Only the real part of the data was used.
The observation interval was N = 1000 symbols long.
We compared our algorithm with the blind MMSE method, where the pilot signal
corresponding to the first user consisted of ones. The fading coefficients corresponding to the strongest path were estimated using both methods. Fig. 23.4 shows theoriginal fading process and the estimated ones, giving an idea of the achieved accuracy. The figure shows that our method provides somewhat more accurate estimates
than the MMSE method, though the estimated fading process is noisy. Fig. 23.5
presents numerical values of the average mean-square error (MSE) as a functionof SNR. The complexity minimization based method performs clearly better thanthe MMSE method especially at lower signal-to-noise ratios. The convergence ofthe gradient approach took place in this case in 10–15 iterations for the learningparameters  = 1 and = 0:5.
In its current form, the proposed method needs training symbols for providing atemporally correlated structure for the desired user’s signal. If the channel varies
rapidly during the training phase, the method is not able to estimate the channel as
the data modulation is on. This is because the temporal correlatedness of the desired

430

TELECOMMUNICATIONS

signal is lost. A future research topic is to overcome this problem. Finally, we pointout that instead of the simplified complexity minimization approach applied here, one
could have as well tried other blind separation methods based on the time structureof the sources. Such methods are discussed in Chapter 18.

23.4

BLIND SEPARATION OF CONVOLVED CDMA MIXTURES *Now consider estimation of the desired user’s symbol process using a blind source
separation method developed for convolutive mixtures. Such methods have been
discussed earlier in Chapter 19. The model consists of a linear transformation ofboth the independent variables (the transmitted symbols) and their delayed version,
where the delay is one time unit. For separating mixtures of delayed and convolved
independent sources, we use an extension of the natural gradient method based on
the information maximization principle [79, 13, 268, 426]. Experiments show thatthe proposed method has quite competitive detection capabilities compared withconventional symbol estimation methods.
23.4.1

Feedback architecture

The vector model (23.5) can be written

rm =

K "
X

k=1

bk;m?1

L
X
l=1

# X
K "

al gkl +

k=1

bkm

L
X
l=1

#

al gkl + nm

(23.27)This model differs slightly from the fading channel model used in the previous sectionin that the channel is now assumed to stay constant during the block of N symbols.
Hence, the fading coefficients al depend only on the path l but not on the symbol
index m. This type of channel is called block fading. As in the previous section, we
use only the real part of the complex-valued data. This allows application of ICA
and BSS methods developed for real-valued data to CDMA.
The model (23.27) can be further expressed in the matrix-vector form [99]rm = G bm + G bm? + nm
(23.28)where G and G are C  K mixing matrices corresponding to the original and theone time unit delayed symbols. The column vectors of G0 and G1 are given by the
0

0

1

1

1

early and the late parts of the coding vectors multiplied by the fading coefficients,
respectively:

G =
0

G =
1

"X
L
l=1

"X
L
l=1

al g1l ;    ;al g1l ;    ;

L
X
l=1

L
X
l=1

al gKl
al gKl#(23.29)#(23.30)BLIND SEPARATION OF CONVOLVED CDMA MIXTURES *431

b

The symbol vector m contains the binary symbols (information bits) of the K users
at time index m:

bm = [b m; b m; : : : ; bKm]T1b(23.31)

2

The vector m?1 is defined quite similarly.
Eq. (23.28) shows that our CDMA signal model represents a linear mixture ofdelayed and convolved sources in the special case where the maximum time delay is
one unit. Assuming that all the mixing matrices (users’ codes) and symbol sequences
are unknown makes the separation problem blind. One method for solving thisconvolutive BSS problem is to consider a feedback architecture. Assuming that theusers are independent of each other, we can apply to the convolutive BSS problem
the principle of entropy maximization discussed earlier in Section 9.3. The weightsof the network can be optimized using the natural gradient algorithm extended forconvolutive mixtures in [13, 79, 268, 426].
The data vectors m are preprocessed by simultaneously whitening them andreducing their dimension to K (the number of users). Using PCA whitening (Chapter
6), the whitened data matrix becomesr

R~ = ?s 2 UTs R
1(23.32)

We can now write the whitened version of Eq. (23.28)

vm = H bm + H bm?
(23.33)where vm is the whitened input vector, and H and H are whitened square K  K
matrices corresponding to the rectangular matrices G and G . From (23.33), thesymbol vector bm can be expressed in terms of the whitened data vector vm and thepreviously estimated symbol vector bm? as
bm = H? (vm ? H bm? )(23.34)
0

1

1

0

1

0

1

1

0

1

1

1

The arising network architecture is depicted in Fig. 23.6.
23.4.2

Semiblind separation method

Based on this feedback architecture we propose the following algorithm for blindsymbol detection in a CDMA system.

H and H .
2. Compute updates for the matrices H and H from the formulas [79, 268, 426]
H = ?H (I + qm bTm )(23.35)TH = ?(I + H )qm bm?
(23.36)
Here I is a K  K unit matrix, and qm = f (bm ) is a nonlinearly transformed
symbol vector bm . The nonlinear function f is typically a sigmoidal or cubic
nonlinearity, and it is applied componentwise to the elements of bm .

1. Initialize randomly the matrices

0

1

0

0
1

1

0

1

1

432

TELECOMMUNICATIONS

z-1

H1

+

Whitening
rm

-1

H0

vm

bm

Fig. 23.6 A feedback network for a convolutive CDMA signal model

H and H using the rule
Hi + Hi ; i = 1; 2

3. Compute new estimates for the matrices

Hi

where  is a small learning parameter.

01(23.37)b

4. Compute new estimate of the symbol vector m from Eq. (23.34).
5. If the matrices

H and H have not converged, return back to step 2.
01

6. Apply the sign nonlinearity to each component of the final estimate of thesymbol vector m . This quantizes the estimated symbols to the bits +1 or ?1.

b

7. Identify the desired user’s symbol sequence which best fits the training sequence.
If some prior information on the desired user’s transmission delays is available,
it can be utilized in the initialization step 1. The update rules in step 2 have been
adapted for our special case of only one unit delay from the more general convolutive
mixture algorithms described in [268, 426]. Because of the feedback, the choice ofthe learning parameter  in step 3 is essential for the convergence. We have useda constant , but more sophisticated iteration dependent choices would probably
make the convergence faster. In step 5, convergence is verified in terms of the meansquare-error matrix norm. Since the transmission system is binary differential, step
6 provides the most probable value for the estimated symbol. The detector estimates
the symbols of all users up to a permutation. Therefore a pilot training sequence isneeded in step 7 to identify the desired user. Hence the method presented above isan example of a semiblind separation approach.
23.4.3

Simulations and discussion

The method introduced in the previous section has been compared in [99] with twostandard methods used in multiuser detection, namely the matched filter (MF) and theBLIND SEPARATION OF CONVOLVED CDMA MIXTURES *433

minimum mean-square error (MMSE) estimator [378, 444, 382, 287]. The matchedfilter estimator is simply

bMF = sign(g1T r)

r(23.38)rgwhich is again computed for all the data vectors 1 ; : : : ; N . Here 1 is a columnof the matrix 0 defined in (23.29) that corresponds to the desired user’s bit b1;m .
Similarly, the (linear) MMSE symbol estimator is given byG UbMMSE = sign(g1 T Us ?s 1 UTs r)(23.39)The matrices s , s were defined earlier below Eq. (23.26).
It is noteworthy that the formula (23.39) is otherwise the same as (23.26), but
now it provides bit estimates bMMSE instead of fading channel coefficient estimates
aMMSE . The reason for this is that in the previous section the quantities to be
estimated were products of bits and fading channel coefficients, or elements of thevector m defined in (23.11), and the bits were all ones in the pilot training sequence.
On the other hand, in this section the vector 1 also contains the fading coefficientsal (which stay constant during the observation interval) because of the definition of
the matrix 0 in (23.29).
The algorithms were tested using quasiorthogonal gold codes [378] of length
C = 31. The number of users was either K = 4 or K = 8, and the number oftransmission paths was L = 3. The powers of the channel paths were, respectively,?5; ?5 and 0 dB for every user, and the signal-to-noise ratio (SNR) varied from 30
dB to 0 dB with respect to the main path. Only the real part of the data was used.
The observation interval was N = 500. A pilot training sequence of length P = 20
was compared with the preambles of the separated sources for identifying the desired
user. A constant learning parameter  = 0:05 was used. Convergence took about 20
iterations in the above environment.
The experimental results shown in Fig. 23.7 give the bit-error-rates (BERs) forthe compared three methods at different SNRs in the more difficult case of K = 8
users. The results are qualitatively similar for K = 4 users [99], and are therefore notshown here. Figure 23.7 indicates that the proposed semiblind convolutive separation
method yields clearly better detection results than the widely used matched filter orlinear minimum mean-square error estimators. The basic reason for this improved
performance is that standard methods such as the MF and MMSE estimators do not
exploit the independence of the received signals. The MMSE estimator makes thedetected sources uncorrelated [287]. Even this much weaker assumption improves
the performance clearly compared with the simple matched filter. In the studied
scenario the independence of the received signals is a reasonable assumption, and its
power becomes apparent by inspecting the results in Fig. 23.7.
We have in this section considered a batch method based on a feedback architecture for symbol detection, but an adaptive version could have been used instead. The
batch method has the advantage that two observation vectors are used for estimating
the current symbol vector, which improves the estimation. There is also no need forsynchronization as is the case with the MMSE and MF methods. This is because dif-

f

g

G

434

TELECOMMUNICATIONS

0.25

0.2
MF

BER (%)

0.15
MMSE

0.1

conv. ICA
0.05

0

0

5

10

15
SNR (dB)

20

25

30

Fig. 23.7 The bit-error-rate (BER) for the convolutive mixture ICA, minimum mean-squareerror (MMSE), and matched filter (MF) methods. The number of users was K = 8.

ferent users’ path delays are implicitly estimated simultaneously in the basis vectorsof the mixing matrices.

23.5

IMPROVING MULTIUSER DETECTION USING COMPLEX ICA *A general drawback of ICA methods in CDMA applications is that it is difficult or even
impossible to utilize well the available prior information on the problem. However,quite generally in any problem it is highly desirable to apply prior information
whenever available, because it usually improves estimation accuracy and overall
performance if taken into account properly. In this section, a feasible solution tothis issue is presented by using ICA as an additional processing element attached toexisting standard receiver structures.
In this section, two types of receiver structures, RAKE-ICA and MMSE-ICA[382], are studied in a block fading CDMA downlink environment. Numericalresults indicate that the performance of RAKE and subspace MMSE detectors canbe greatly improved in terms of ICA postprocessing. This is mainly due to the facts

435

IMPROVING MULTIUSER DETECTION USING COMPLEX ICA *that ICA efficiently utilizes the independence of the original signals, and that ICA
does not explicitly depend on erroneous timing or channel estimation. On the other
hand, the RAKE and subspace MMSE estimators can apply prior information on theCDMA problem. Since these estimators are complex-valued, an ICA method fitted
to complex data must be used. To this end, complex FastICA algorithm (see Section
20.3 and [47]) is used.
This section is based on the references [382, 383]. The interested reader can findmore information on ICA assisted CDMA reception in them.
23.5.1

Data model

The continuous time signal model is otherwise the same as in (23.3), but the fading
coefficients alm in (23.3) are now replaced by the complex coefficients al . Thus
each path l has its own coefficient al which is assumed to remain constant during
the data block of N symbols bkm , m = 1; 2; : : : ; N . Another difference is that theprocessing window is now two symbols long while its length in the previous sections
is one symbol. Hence the samples are collected into 2C -dimensional vectors

rm = (r[mC ]; r[mC + 1]; : : : ; r[(m + 2)C ? 1])T

(23.40)instead of the C -dimensional data vectors (23.4).
Since sampling is asynchronous with respect to the symbols, the vector sample
m in (23.40) usually contains information on three successive symbols bk;m?1 , bkm ,and bk;m+1 . The two symbols long data window has the advantage that it always
contains one complete symbol. Similarly as in (23.27), the vectors (23.40) can be
expressed in the well-known form

r

rm =

K "
X

L
X

k=1

l=1

bk;m?1

al gkl + bkm

g

L
X
l=1

al gkl + bk;m+1

L
X
l=1

#

al gkl + nm

(23.41)g
g

The “early” and “late” code vectors kl and kl are defined quite similarly as in Eqs.
(23.6) and (23.7). Now they just contain more zeros so that these vectors become 2C dimensional. However, the late code vector kl is now associated with the symbolbk;m+1 , and for the middle symbol bkm the “middle” code vectors

gkl = [0Tdl ; sk [1]; : : : ; sk [C ]; 0TC?dl ]T(23.42)0

are defined, where Tdl is a row vector having dl zeros as its elements.
Again, the data vectors (23.41) can be represented more compactly as

rm = Gbm + nm

(23.43)

which has the form of a noisy linear ICA model. The 2C  3K dimensional code
matrix corresponds to the mixing matrix . It is assumed to have full rank, and

G

A

436

TELECOMMUNICATIONS

contains the code vectors and path strengths:

"X
L

G =

L
X

al g1l ;LX

al g1l ; : : :
l=1
#
L
X
; al gKl ; al gKl ; al gKl
l=1
l=1
l=1
l=1
L
X

al g1l ;l=1
LX(23.44)

The 3K -dimensional symbol vector

bm = [b ;m? ; b m; b ;m ; : : : ; bK;m? ; bKm; bK;m ]T
1

1

1

1

+1

1

+1

(23.45)sGcontains the symbols, and corresponds to the vector of independent (or roughly
independent) sources. Note that both the code matrix and the symbol vector mconsists of subsequent triplets corresponding to the early, middle, and late parts.
23.5.2b

ICA based receivers

In the following, we consider in more detail two ways of initializing the ICA iteration.
For a more thorough discussion, see [382]. The starting point for the receiverdevelopment is to look at the noiseless whitened data1zm = Vbm = ?s 2 UHs Gbm
1



(23.46)

U

where s and s are matrices containing (in the same order) the 3K principal
eigenvalues and -vectors of the data autocorrelation matrix E f m H
m g, respectively.
becomes
It is easy to see that for the data model (23.43), the whitening matrix
H = , because the symbols are uncorrelated: Ef m H g = , andorthonormal:mthe whitened data vectors satisfy the condition = Ef m H
m g.
It suffices to estimate one column of the whitening matrix , say its second
column 2 . This is because we can then estimate the symbols b1m of the desired user
(user k = 1) by applying the vector 2 as follows:

r r

VV

I

I

V
b b I
V

z z

v

v
vH zm = vH Vbm = [0100    0]bm = b m
From the definitions of V and G we see that212

v = s? 2 UHs
1

2

L
X
l=1

a1l g1l(23.47)(23.48)which is exactly the subspace MMSE detector [451] for dispersive channels.
1 Because the data is now complex-valued, the transpose

H , which equals transposition and complex conjugation.

T must be replaced by the Hermitian operator

IMPROVING MULTIUSER DETECTION USING COMPLEX ICA *437

Equation (23.48) can be applied to separating the desired symbol b1m , but it
uses only second-order statistics. In addition, the subspace parameters as well as
the path gains and delays are always subject to estimation errors, degrading theperformance of (23.48) in separation. But we can improve the separation capabilityof the estimator (23.48) by using ICA as a postprocessing tool. This is possible,
since the independence of the original sources is not utilized in deriving (23.48).
Moreover, it is meaningful to apply ICA by using the subspace MMSE estimator(23.48) as the starting point, because it already identifies the desired user. This
identification is not possible by using ICA alone. The proposed DS-CDMA receiverstructure, which we call MMSE-ICA, consists of a subspace MMSE detector refinedby ICA iterations. The complex FastICA algorithm discussed earlier in Section
20.3 and in [47] is a natural choice for the ICA postprocessing method. It can deal
with complex-valued data, and extracts one independent component at a time, which
suffices in this application.
Alternatively, known or estimated symbols can be used for initializing the ICA
iteration. This follows from the uncorrelatedness of the symbols, since then we
have Ef m b1m g = 2 , leading again to the subspace MMSE detector. Because
training symbols are not necessarily implemented in all the DS-CDMA systems, it ispreferable to first use the traditional RAKE receiver [378, 382] or multipath correlator
for detecting symbols. The RAKE estimator is nothing but a simple extension of thematched filter for several paths. Alternatively, one can initially detect the symbols byusing the MMSE method for symbol outputs. The symbol estimates obtained initiallyin terms of the RAKE detector or MMSE symbol estimator are then refined using
the complex FastICA algorithm. These structures are henceforth called as RAKEICA and MMSEbit-ICA detectors, respectively. Global convergence of the complex
FastICA algorithm has been proved in [382]. The sign indeterminacy in the sources
estimated by any ICA method is removed by a comparator element, which chooses
the sign according to the RAKE receiver or subspace MMSE detector, respectively.
The proposed receiver structures are summarizes below. In step 1, an initial
estimate of the k th whitened code vector k is computed using one of the threestandard detection methods mentioned earlier. Steps 2-5 give the procedure for
improving this initial estimate using the complex FastICA algorithm [47].

z

v

v

ICA-based blind interference suppression schemes [382] Let k be theindex of the desired user, and m the whitened data vector corresponding to thesymbol vector m . The constant is 2 for complex-valued symbols, and = 3 forreal-valued symbols (in the latter case, the data itself is complex, but the symbols
are real). An estimate is denoted by the hat ^. Then the iterative algorithms for blindinterference suppression are as follows.

bzw(0) = vk =kvk k, where^ ?s = U^ Hs PLl a^l c^kl .
(a) MMSE-ICA: vk = (b) RAKE-ICA: vk = Efzm^bRAKE
km g.
(c) MMSEbit-ICA: vk = Efzm^bMMSE
km g.

1. Initialize

1 2

^

=1

438

TELECOMMUNICATIONS

Let t = 1.
2. Compute one iteration of the complex FastICA algorithm [47]:w(t) = Efzm(w(t ? 1)H zm) jw(t ? 1)H zmj g ? w(t ? 1)
2

(23.49)w(t) by its norm.
4. If jw(t)H w(t ? 1)j is not close enough to 1, set t = t + 1, and go back to step
3. Divide2.

5. Output the vector
23.5.3w = w(t), where  = sign(Re[w(0)H w(t)]).

Simulation results

The algorithms were tested using simulated DS-CDMA downlink data with a block
fading channel.
In the first experiment gold codes of the length C = 31 were used. The lengthof the block was M = 500 binary phase shift keying (BPSK) symbols. The channel
was fixed during the block period. The number of users was K = 20, and themultiple access interference (MAI) per user was 5 dB. Hence, the total interference
power was 17:8 dB. The number of paths was L = 3. The path gains were gaussian
distributed with a zero mean, and the path delays were chosen randomly from theinterval f0; 1; : : :; (C ? 1)=2g. The delays and the path gains were assumed tobe known. The signal-to-noise ratio (in the chip matched filter output) varied withrespect to the desired user from 5 dB to 35 dB, and 10000 independent trials were
made. A constant = 3 was used in the ICA iteration.
Figure 23.8 shows the achieved bit-error-rates (BERs) for the methods as a functionof the SNR. The performance of RAKE is quite modest due to the near–far situation.
Consequently, RAKE-ICA is able to improve the performance of the RAKE method
only marginally. Subspace MMSE detector suffers from interference floor at higher
SNRs. One reason for this is inaccurate estimation of the signal subspace. Even
though MMSE-ICA uses the same estimate for the signal subspace, it is able toexploit the statistical independence of the source signals, and seems to follow quite
closely the equal length optimal MMSE receiver, denoted as MMSE bound.
Figure 23.9 shows the corresponding block-error-rates (BLER) for these methods.
A block is correctly estimated if all the symbols in the block are estimated correctly.
For speech and data services that do not require real-time processing, raw BLER of
10?1 is sufficient. For real-time data services, raw BLER of the order of 10?2 isrequired. Figure 23.9 shows that RAKE-ICA actually improves the performance of
the RAKE method quite remarkably with respect to BLER, even though the overall
BER has not been improved that much.
More numerical experiments, including simulation results for the case of purely
complex data ( = 2), can be found in [382, 383]. They clearly indicate that theestimates given by the RAKE and subspace MMSE detectors can be greatly improvedby using ICA as a postprocessing tool.

CONCLUDING REMARKS AND REFERENCES

439

0

10

−1

10

−2

10

−3

10

−4

10

−5

10

RAKE
RAKE−ICA
subspace MMSE
MMSE−ICAMMSE bound

−6

10

−7

10

5

10

15

20

25

30

35

Fig. 23.8 Bit-error-rate as a function of SNR ( 5; : : : ; 35 dB). The system includes K = 20
users with the average MAI of 5 dB per interfering user. BPSK data is used.

23.6

CONCLUDING REMARKS AND REFERENCES

In this chapter, we have applied several quite different extensions of basic ICA (or
BSS) techniques to short-code CDMA data. It can be concluded that ICA often
provides significant performance gains in CDMA applications. Basically, this resultsfrom the fact that standard CDMA detection and estimation methods do not exploit
the powerful but realistic independence assumption. At best, they utilize the much
weaker uncorrelatedness condition for the received source signals.
CDMA techniques are currently studied extensively in telecommunications, because they will be used in a form or another in future high performance mobile
communications systems. A specific feature of all telecommunications applicationsof ICA is that they are almost always semiblind problems. The receiver has more orless prior information on the communication system, typically at least the spreadingcode of the desired user is known. This prior information should be combined ina suitable way with blind ICA techniques for achieving optimal results. Anotherimportant design feature is that practical algorithms should not be computationally
too demanding, making it possible to realize them in real time.

440

TELECOMMUNICATIONS

0

10

−1

10

−2

10

RAKE
RAKE−ICA
subspace MMSE
MMSE−ICAMMSE bound

−3

10

−4

10

5

10

15

20

25

30

35

Fig. 23.9 Block-error-rate as a function of SNR ( 5; : : : ; 35 dB). The system includes K = 20
users with the average MAI of 5 dB per interfering user. BPSK data is used.

ICA methods also have been applied to CDMA data, for example, in references[223, 384, 100]. Other applications of blind source separation techniques to various
communications problems can be found for example in [77, 111, 130, 435]. Related
blind identification or blind equalization techniques are discussed in many papers;
see for example [41, 73, 91, 122, 146, 143, 144, 158, 184, 224, 265, 276, 287, 351,
352, 361, 425, 428, 431, 439, 440] and the references therein. Blind identification
techniques used in communications typically exploit second-order temporal statistics(or suitable explicit higher-order statistics) instead of ICA. The interested reader canfind many more references on blind communications techniques in recent conference proceedings and journals dealing with telecommunications and statistical signal
processing.

24
Other Applications

In this chapter, we consider some further applications of independent component
analysis (ICA), including analysis of financial time series and audio signal separation.

24.1
24.1.1FINANCIAL APPLICATIONS
Finding hidden factors in financial data

It is tempting to try ICA on financial data. There are many situations in which parallel
financial time series are available, such as currency exchange rates or daily returnsof stocks, that may have some common underlying factors. ICA might reveal somedriving mechanisms that otherwise remain hidden.
In a study of a stock portfolio [22], it was found that ICA is a complementary tool
to principal component analysis (PCA), allowing the underlying structure of the data
to be more readily observed. If one could find the maximally independent mixturesof the original stocks, i.e., portfolios, this might help in minimizing the risk in theinvestment strategy.
In [245], we applied ICA on a different problem: the cashflow of several stores
belonging to the same retail chain, trying to find the fundamental factors common
to all stores that affect the cashflow. Thus, the effect of the factors specific to any
particular store, i.e., the effect of the managerial actions taken at the individual store
and in its local environment, could be analyzed.
In this case, the mixtures in the ICA model are parallel financial time series xi (t),with i indexing the individual time series, i = 1; : : : ; m and t denoting discrete time.
441

442

OTHER APPLICATIONS

We assume the instantaneous ICA modelxi (t) =Xjaij sj (t)(24.1)

for each time series xi (t). Thus the effect of each time-varying underlying factor orindependent component sj (t) on the measured time series is approximately linear.
The assumption of having some underlying independent components in this specific application may not be unrealistic. For example, factors like seasonal variations
due to holidays and annual variations, and factors having a sudden effect on thepurchasing power of the customers, like price changes of various commodities, canbe expected to have an effect on all the retail stores, and such factors can be assumed
to be roughly independent of each other. Yet, depending on the policy and skills of
the individual manager, e.g., advertising efforts, the effect of the factors on the cash
flow of specific retail outlets are slightly different. By ICA, it is possible to isolate
both the underlying factors and the effect weights, thus also making it possible togroup the stores on the basis of their managerial policies using only the cash flowtime series data.
The data consisted of the weekly cash flow in 40 stores that belong to the sameretail chain, covering a time span of 140 weeks. Some examples of the original dataxi (t) are shown in Fig. 24.1. The weeks of a year are shown on the horizontal axis,
starting from the first week in January. Thus for example the heightened Christmas
sales are visible in each time series before and during week 51 in both of the fullyears shown.
The data were first prewhitened using PCA. The original 40-dimensional signal
vectors were projected to the subspace spanned by four principal components, and thevariances were normalized to 1. Thus the dimension of the signal space was strongly
decreased from 40. A problem in this kind of real world application is that there
is no prior knowledge on the number of independent components. Sometimes the
eigenvalue spectrum of the data covariance matrix can be used, as shown in Chapter
6, but in this case the eigenvalues decreased rather smoothly without indicating any
clear signal subspace dimension. Then the only way is to try different dimensions.
If the independent components that are found using different dimensions for thewhitened data are the same or very similar, we can trust that they are not just artifacts
produced by the compression, but truly indicate some underlying factors in the data.
Using the FastICA algorithm, four independent components (ICs) sj (t); j =1; :::; 4 were estimated. As depicted in Fig. 24.2, the FastICA algorithm has foundseveral clearly different fundamental factors hidden in the original data.
The factors have different interpretations. The topmost factor follows the suddenchanges that are caused by holidays etc.; the most prominent example is Christmas
time. The factor in the bottom row, on the other hand, reflects the slower seasonalvariation, with the effect of the summer holidays clearly visible. The factor in thethird row could represent a still slower variation, something resembling a trend. The
last factor, in the second row, is different from the others; it might be that this factor
follows mostly the relative competitive position of the retail chain with respect to its
competitors, but other interpretations are also possible.

FINANCIAL APPLICATIONS

1

20

40

8

28

48

16

1

20

40

8

28

48

16

1

20

40

8

28

48

16

1

20

40

8

28

48

16

1

20

40

8

28

48

16

443

Fig. 24.1 Five samples of the 40 original cashflow time series (mean removed, normalized
to unit standard deviation). Horizontal axis: time in weeks over 140 weeks. (Adapted from[245].)

If five ICs are estimated instead of four, then three of the found components stayvirtually the same, while the fourth one separates into two new components. Using
the found mixing coefficients aij , it is also possible to analyze the original time series
and cluster them in groups. More details on the experiments and their interpretation
can be found in [245].
24.1.2Time series prediction by ICA

As noted in Chapter 18, the ICA transformation tends to produce component signals,
sj (t), that can be compressed with fewer bits than the original signals, xi (t). They
are thus more structured and regular. This gives motivation to try to predict thesignals xi (t) by first going to the ICA space, doing the prediction there, and thentransforming back to the original time series, as suggested by [362]. The prediction
can be done separately and with a different method for each component, depending
on its time structure. Hence, some interaction from the user may be needed in theoverall prediction procedure. Another possibility would be to formulate the ICA
contrast function in the first place so that it includes the prediction errors — somework along these lines has been reported by [437].
In [289], we suggested the following basic procedure:
1. After subtracting the mean of each time series and prewhitening (after which
each time series has zero mean and unit variance), the independent components

444

OTHER APPLICATIONS

1

20

40

8

28

48

16

1

20

40

8

28

48

16

1

20

40

8

28

48

16

1

20

40

8

28

48

16

Fig. 24.2 Four independent components or fundamental factors found from the cashflow
data. (Adapted from [245].)Asj (t), and the mixing matrix, , are estimated using the FastICA algorithm.
The number of ICs can be variable.
2. For each component sj (t), a suitable nonlinear filtering is applied to reduce theeffects of noise — smoothing for components that contain very low frequencies (trend, slow cyclical variations), and high-pass filtering for components
containing high frequencies and/or sudden shocks. The nonlinear smoothingis done by applying smoothing functions fj on the source signals sj (t),ssj (t) = fj [sj (t + r); : : : ; sj (t); : : : ; sj (t ? k)]:(24.2)

3. Each smoothed independent component is predicted separately, for instanceusing some method of autoregressive (AR) modeling [455]. The prediction isdone for a number of steps into the future. This is done by applying prediction
functions, gj , on the smoothed source signals, ssj (t):spj (t + 1) = gj [ssj (t); ssj (t ? 1); : : : ; ssj (t ? q)](24.3)The next time steps are predicted by gliding the window of length q over themeasured and predicted values of the smoothed signal.
4. The predictions for each independent component are combined by weighing
them with the mixing coefficients, aij , thus obtaining the predictions, xpi (t),for the original time series, xi (t):xp (t + 1) = Asp(t + 1)

and similarly for t + 2; t + 3; : : : .

(24.4)

FINANCIAL APPLICATIONS

445

Fig. 24.3 Prediction of real-world financial data: the upper figure represents the actual
future outcome of one of the original mixtures and the lower one the forecast obtained using
ICA prediction for an interval of 50 values.

To test the method, we applied our algorithm on a set of 10 foreign exchange
rate time series. Again, we suppose that there are some independent factors that
affect the time evolution of such time series. Economic indicators, interest rates, andpsychological factors can be the underlying factors of exchange rates, as they areclosely tied to the evolution of the currencies. Even without prediction, some of theICs may be useful in analyzing the impact of different external phenomena on theforeign exchange rates [22].
The results were promising, as the ICA prediction performed better than direct
prediction. Figure 24.3 shows an example of prediction using our method. The
upper figure represents one of the original time series (mixtures) and the lower one
the forecast obtained using ICA prediction for a future interval of 50 time steps. The
algorithm seemed to predict very well especially the turning points. In Table 24.1
there is a comparison of errors obtained by applying classic AR prediction to theoriginal time series directly, and our method outlined above. The right-most column
shows the magnitude of the errors when no smoothing is applied to the currencies.
While ICA and AR prediction are linear techniques, the smoothing was nonlinear.
Using nonlinear smoothing, optimized for each independent component time series
separately, the prediction of the ICs is more accurately performed and the results also
are different from the direct prediction of the original time series. The noise in thetime series is strongly reduced, allowing a better prediction of the underlying factors.
The model is flexible and allows various smoothing tolerances and different orders
in the classic AR prediction method for each independent component.
In reality, especially in real world time series analysis, the data are distorted bydelays, noise, and nonlinearities. Some of these could be handled by extensions of
the basic ICA algorithms, as reported in Part III of this book.

446

OTHER APPLICATIONS

Table 24.1 The prediction errors (in units of 0.001) obtained with our method and the classic
AR method. Ten currency time series were considered and five independent components wereused. The amount of smoothing in classic AR prediction was varied.

Errors

24.2

Smoothing in
AR prediction

2

0.5

0.1

0.08

0.06

0.05

0

ICA prediction

2.3

2.3

2.3

2.3

2.3

2.3

2.3

AR prediction

9.7

9.1

4.7

3.9

3.4

3.1

4.2

AUDIO SEPARATION

One of the original motivations for ICA research was the cocktail-party problem, as
reviewed in the beginning of Chapter 7. The idea is that there are n sound sources
recorded by a number of microphones, and we want to separate just one of the sources.
In fact, often there is just one interesting signal, for example, a person speaking to
the microphone, and all the other sources can be considered as noise; in this case,
we have a problem of noise canceling. A typical example of a situation where wewant to separate noise (or interference) from a speech signal is a person talking to amobile phone in a noisy car.
If there is just one microphone, one can attempt to cancel the noise by ordinary
noise canceling methods: linear filtering, or perhaps more sophisticated techniques
like wavelet and sparse code shrinkage (Section 15.6). Such noise canceling can berather unsatisfactory, however. It works only if the noise has spectral characteristics
that are clearly different from those of the speech signal. One might wish to remove
the noise more effectively by collecting more data using several microphones. Since
in real-life situations the positions of the microphones with respect to the sources
can be rather arbitrary, the mixing process is not known, and it has to be estimatedblindly. In this case, we find the ICA model, and the problem is one of blind source
separation.
Blind separation of audio signals is, however, much more difficult than one might
expect. This is because the basic ICA model is a very crude approximation of thereal mixing process. In fact, here we encounter almost all the complications that wehave discussed in Part III: The mixing is not instantaneous. Audio signals propagate rather slowly, andthus they arrive in the microphones at different times. Moreover, there are
echos, especially if the recording is made in a room. Thus the problem is more
adequately modeled by a convolutive version of the ICA model (Chapter 19).
The situation is thus much more complicated than with the separation of magnetoencephalographic (MEG) signals, which propagate fast, or with featureAUDIO SEPARATION

447

extraction, where no time delays are possible even in theory. In fact, even thebasic convolutive ICA model may not be enough because the time delays may
be fractional and may not be adequately modeled as integer multiples of thetime interval between two samples.

 Typically, the recordings are made with two microphones only. However, thenumber of source signals is probably much larger than 2 in most cases, since
the noise sources may not form just one well-defined source. Thus we havethe problem of overcomplete bases (Chapter 16).

 The nonstationarity of the mixing is another important problem. The mixing

matrix may change rather quickly, due to changes in the constellation of the
speaker and the microphones. For example, one of these may be moving withrespect to the other, or the speaker may simply turn his head. This implies
that the mixing matrix must be reestimated quickly in a limited time frame,
which also means a limited number of data. Adaptive estimation methods may
alleviate this problem somewhat, but this is still a serious problem due to the
convolutive nature of the mixing. In the convolutive mixing, the number ofparameters can be very large: For example, the convolution may be modeledby filters of the length of 1000 time points, which effectively multiplies thenumber of parameters in the model by 1000. Since the number of data points
should grow with the number of parameters to obtain satisfactory estimates, it
may be next to impossible to estimate the model with the small number of data
points that one has time to collect before the mixing matrix has changed toomuch.

 Noise may be considerable. There may be strong sensor noise, which meansthat we should use the noisy ICA model (Chapter 15). The noise complicatesthe estimation of the ICA model quite considerably,even in the basic case where
noise is assumed gaussian. On the other hand, the effect of overcomplete bases
could be modeled as noise as well. This noise may not be very gaussian,however, making the problem even more difficult.

Due to these complications, it may be that the prior information, independence
and nongaussianity of the source signals, are not enough. To estimate the convolutive
ICA model with a large number of parameters, and a rapidly changing mixing
matrix, may require more information on the signals and the matrix. First, one
may need to combine the assumption of nongaussianity with the different timestructure assumptions in Chapter 18. Speech signals have autocorrelations andnonstationarities, so this information could be used [267, 216]. Second, one may need
to use some information on the mixing. For example, sparse priors (Section 20.1.3)
could be used.
It is also possible that real-life speech separation requires sophisticated modelingof speech signals. Speech signals are highly structured, autocorrelations and nonstationarity being just the very simplest aspects of their time structure. Such approaches
were proposed in [54, 15].

448

OTHER APPLICATIONS

Because of these complications, audio separation is a largely unsolved problem.
For a recent review on the subject, see [429]. One of the main theoretical problems,
estimation of the convolutive ICA model, was described in Chapter 19.

24.3

FURTHER APPLICATIONS

Among further applications, let us mention

 Text document analysis [219, 229, 251]
 Radiocommunications [110, 77] Rotating machine monitoring [475] Seismic monitoring [161] Reflection canceling [127] Nuclear magnetic resonance spectroscopy [321] Selective transmission, which is a dual problem of blind source separation. A

set of independent source signals are adaptively premixed prior to a nondispersive physical mixing process so that each source can be independently
monitored in the far field [117].

Further applications can be found in the proceedings of the ICA’99 and ICA2000
workshops [70, 348].

References

1. K. Abed-Meraim and P. Loubaton. A subspace algorithm for certain blind identification
problems. IEEE Trans. on Information Theory, 43(2):499–511, 1997.
2. L. Almeida. Linear and nonlinear ICA based on mutual information. In Proc. IEEE
2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium(AS-SPCC), pages 117–122, Lake Louise, Canada, October 2000.
3. S.-I. Amari. Neural learning in structured parameter spaces—natural Riemannian gradient. In Advances in Neural Information Processing Systems 9, pages 127–133. MIT
Press, 1997.
4. S.-I. Amari. Natural gradient works efficiently in learning. Neural Computation,
10(2):251–276, 1998.
5. S.-I. Amari. Natural gradient learning for over- and under-complete bases in ICA. NeuralComputation, 11(8):1875–1883, 1999.
6. S.-I. Amari. Estimating functions of independent component analysis for temporally
correlated signals. Neural Computation, 12(9):2083–2107, 2000.
7. S.-I. Amari. Superefficiency in blind source separation. IEEE Trans. on Signal Processing, 47(4):936–944, April 1999.
8. S.-I. Amari and J.-F. Cardoso. Blind source separation—semiparametric statistical approach. IEEE Trans. on Signal Processing, 45(11):2692–2700, 1997.
9. S.-I. Amari, T.-P. Chen, and A. Cichocki. Stability analysis of adaptive blind source
separation. Neural Networks, 10(8):1345–1351, 1997.
10. S.-I. Amari and A. Cichocki. Adaptive blind signal processing—neural network approaches. Proceedings of the IEEE, 86(10):2026–2048, 1998.
449

450

REFERENCES

11. S.-I. Amari, A. Cichocki, and H. H. Yang. Blind signal separation and extraction: Neuraland information-theoretic approaches. In S. Haykin, editor, Unsupervised Adaptive
Filtering, volume 1, pages 63–138. Wiley, 2000.
12. S.-I. Amari, A. Cichocki, and H.H. Yang. A new learning algorithm for blind source
separation. In Advances in Neural Information Processing Systems 8, pages 757–763.
MIT Press, 1996.
13. S.-I. Amari, S. C. Douglas, A. Cichocki, and H. H. Yang. Novel on-line adaptive
learning algorithms for blind deconvolution using the natural gradient approach. In
Proc. IEEE 11th IFAC Symposium on System Identification, SYSID-97, pages 1057–
1062, Kitakyushu, Japan, 1997.
14. T. W. Anderson. An Introduction to Multivariate Statistical Analysis. Wiley, 1958.
15. J. Anemüller and B. Kollmeier. Amplitude modulation decorrelation for convolutive
blind source separation. In Proc. Int. Workshop on Independent Component Analysis
and Blind Signal Separation (ICA2000), pages 215–220, Helsinki, Finland, 2000.
16. B. Ans, J. Hérault, and C. Jutten. Adaptive neural architectures: detection of primitives.
In Proc. of COGNITIVA’85, pages 593–597, Paris, France, 1985.
17. A. Antoniadis and G. Oppenheim, editors. Wavelets in Statistics. Springer, 1995.
18. J.J. Atick. Entropy minimization: A design principle for sensory perception?
Journal of Neural Systems, 3:81–90, 1992. Suppl. 1992.

Int.

19. H. Attias. Independent factor analysis. Neural Computation, 11(4):803–851, 1999.
20. H. Attias. Learning a hierarchical belief network of independent factor analyzers. In
Advances in Neural Information Processing Systems, volume 11, pages 361–367. MIT
Press, 1999.
21. A. D. Back and A. C. Tsoi. Blind deconvolution of signals using a complex recurrent
network. In J. Vlontzos, J. Hwang, and E. Wilson, editors, Neural Networks for Signal
Processing, volume 4, pages 565–574, 1994.
22. A. D. Back and A. S. Weigend. A first application of independent component analysis to
extracting structure from stock returns. Int. J. on Neural Systems, 8(4):473–484, 1997.
23. P. F. Baldi and K. Hornik. Learning in linear neural networks: A survey. IEEE Trans.
on Neural Networks, 6(4):837–858, 1995.
24. Y. Bar-Ness. Bootstrapping adaptive interference cancellers: Some practical limitations.
In The Globecom Conf., pages 1251–1255, Miami, 1982. Paper F3.7.
25. D. Barber and C. Bishop. Ensemble learning in Bayesian neural networks. In M. Jordan,
M. Kearns, and S. Solla, editors, Neural Networks and Machine Learning, pages 215–
237. Springer, 1998.
26. H. B. Barlow. Possible principles underlying the transformations of sensory messages.
In W. A. Rosenblith, editor, Sensory Communication, pages 217–234. MIT Press, 1961.
27. H. B. Barlow. Single units and sensation: A neuron doctrine for perceptual psychology?
Perception, 1:371–394, 1972.
28. H. B. Barlow. Unsupervised learning. Neural Computation, 1:295–311, 1989.
29. H. B. Barlow. What is the computational goal of the neocortex? In C. Koch and J.L.
Davis, editors, Large-Scale Neuronal Theories of the Brain. MIT Press, Cambridge, MA,
1994.

REFERENCES

451

30. H. B. Barlow, T.P. Kaushal, and G.J. Mitchison. Finding minimum entropy codes. NeuralComputation, 1:412–423, 1989.
31. A. Barros, A. Mansour, and N. Ohnishi. Removing artifacts from electrocardiographic
signals using independent component analysis. Neurocomputing, 22:173–186, 1998.
32. A. K. Barros, R. Vigário, V. Jousmäki, and N. Ohnishi. Extraction of event-related
signals from multi-channel bioelectrical measurements. IEEE Trans. on Biomedical
Engineering, 47(5):583–588, 2000.
33. J. Basak and S.-I. Amari. Blind separation of uniformly distributed signals: A generalapproach. IEEE Trans. on Neural Networks, 10(5):1173–1185, 1999.
34. A. J. Bell. Information theory, independent component analysis, and applications. In
S. Haykin, editor, Unsupervised Adaptive Filtering, Vol. I, pages 237–264. Wiley, 2000.
35. A. J. Bell and T. J. Sejnowski. A non-linear information maximization algorithm that
performs blind separation. In Advances in Neural Information Processing Systems 7,
pages 467–474. The MIT Press, Cambridge, MA, 1995.
36. A.J. Bell and T.J. Sejnowski. An information-maximization approach to blind separation
and blind deconvolution. Neural Computation, 7:1129–1159, 1995.
37. A.J. Bell and T.J. Sejnowski. Learning higher-order structure of a natural sound. Network,
7:261–266, 1996.
38. A.J. Bell and T.J. Sejnowski. The ’independent components’ of natural scenes are edge
filters. Vision Research, 37:3327–3338, 1997.
39. S. Bellini. Bussgang techniques for blind deconvolution and equalization. In S. Haykin,editor, Blind Deconvolution, pages 8–59. Prentice Hall, 1994.
40. A. Belouchrani and M. Amin. Blind source separation based on time-frequency signal
representations. IEEE Trans. on Signal Processing, 46(11):2888–2897, 1998.
41. A. Belouchrani and M. Amin. Jammer mitigation in spread spectrum communications
using blind sources separation. Signal Processing, 80(4):723–729, 2000.
42. A. Belouchrani and J.-F. Cardoso. Maximum likelihood source separation by the
expectation-maximization technique: deterministic and stochastic implementation. In
Proc. Int. Symp. on Nonlinear Theory and its Applications (NOLTA’95), pages 49–53,
Las Vegas, Nevada, 1995.
43. A. Belouchrani, K. Abed Meraim, J.-F. Cardoso, and E. Moulines. A blind source
separation technique based on second order statistics. IEEE Trans. on Signal Processing,45(2):434–444, 1997.
44. S. Bensley and B. Aazhang. Subspace-based channel estimation for code divisionmultiple access communication systems. IEEE Trans. on Communications, 44(8):1009–
1020, 1996.
45. P. Berg and M. Scherg. A multiple source approach to the correction of eye artifacts.
Electroencephalography and Clinical Neurophysiology, 90:229–241, 1994.
46. D. Bertsekas. Nonlinear Programming. Athenas Scientific, Belmont, MA, 1995.
47. E. Bingham and A. Hyvärinen. A fast fixed-point algorithm for independent component
analysis of complex-valued signals. Int. J. of Neural Systems, 10(1):1–8, 2000.
48. C. M. Bishop. Neural Networks for Pattern Recognition. Clarendon Press, 1995.

452

REFERENCES

49. C. M. Bishop, M. Svensen, and C. K. I. Williams. GTM: The generative topographic
mapping. Neural Computation, 10:215–234, 1998.
50. B. B. Biswal and J. L. Ulmer. Blind source separation of multiple signal sources of fMRI
data sets using independent component analysis. J. of Computer Assisted Tomography,
23(2):265–271, 1999.
51. S. Blanco, H. Garcia, R. Quian Quiroga, L. Romanelli, and O. A. Rosso. Stationarity of
the EEG series. IEEE Engineering in Medicine and Biology Magazine, pages 395–399,
1995.
52. H. Bourlard and Y. Kamp. Auto-association by multilayer perceptrons and singular value
decomposition. Biological Cybernetics, 59:291–294, 1988.
53. G. Box and G. Tiao. Bayesian Inference in Statistical Analysis. Addison-Wesley, 1973.
54. M.S. Brandstein. On the use of explicit speech modeling in microphone array applications. In Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP’98),
pages 3613–3616, Seattle, Washington, 1998.
55. H. Broman, U. Lindgren, H. Sahlin, and P. Stoica. Source separation: A TITO system
identification approach. Signal Processing, 73:169–183, 1999.
56. C. H. M. Brunia, J. Möcks, and M. Van den Berg-Lennsen. Correcting ocular artifacts—acomparison of several methods. J. of Psychophysiology, 3:1–50, 1989.
57. G. Burel. Blind separation of sources: a nonlinear neural algorithm. Neural Networks,
5(6):937–947, 1992.
58. X.-R. Cao and R.-W. Liu. General approach to blind source separation. IEEE Trans. on
Signal Processing, 44(3):562–571, 1996.
59. V. Capdevielle, C. Serviere, and J.Lacoume. Blind separation of wide-band sources in thefrequency domain. In Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing(ICASSP’95), volume 3, pages 2080–2083, Detroit, Michigan, 1995.
60. J.-F. Cardoso. Blind identification of independent signals. In Proc. Workshop on HigherOrder Specral Analysis, Vail, Colorado, 1989.
61. J.-F. Cardoso. Source separation using higher order moments. In Proc. IEEE Int. Conf.
on Acoustics, Speech and Signal Processing (ICASSP’89), pages 2109–2112, Glasgow,
UK, 1989.
62. J.-F. Cardoso. Eigen-structure of the fourth-order cumulant tensor with application to
the blind source separation problem. In Proc. IEEE Int. Conf. on Acoustics, Speech andSignal Processing (ICASSP’90), pages 2655–2658, Albuquerque, New Mexico, 1990.
63. J.-F. Cardoso. Iterative techniques for blind source separation using only fourth-order
cumulants. In Proc. EUSIPCO, pages 739–742, Brussels, Belgium, 1992.
64. J.-F. Cardoso. Infomax and maximum likelihood for source separation. IEEE Letters on
Signal Processing, 4:112–114, 1997.
65. J.-F. Cardoso. Blind signal separation: statistical principles. Proceedings of the IEEE,
9(10):2009–2025, 1998.
66. J.-F. Cardoso. Multidimensional independent component analysis. In Proc. IEEE Int.
Conf. on Acoustics, Speech and Signal Processing (ICASSP’98), Seattle, WA, 1998.
67. J.-F. Cardoso. On the stability of some source separation algorithms. In Proc. Workshop
on Neural Networks for Signal Processing (NNSP’98), pages 13–22, Cambridge, UK,
1998.

REFERENCES

453

68. J.-F. Cardoso. High-order contrasts for independent component analysis. Neural Computation, 11(1):157–192, 1999.
69. J. F. Cardoso. Entropic contrasts for source separation: Geometry and stability. In
S. Haykin, editor, Unsupervised Adaptive Filtering, volume 1, pages 139–189. Wiley,
2000.
70. J.-F. Cardoso, C. Jutten, and P. Loubaton, editors. Proc. of the 1st Int. Workshop on
Independent Component Analysis and Signal Separation, Aussois, France, January 1115, 1999. 1999.
71. J.-F. Cardoso and B. Hvam Laheld. Equivariant adaptive source separation. IEEE Trans.
on Signal Processing, 44(12):3017–3030, 1996.
72. J.-F. Cardoso and A. Souloumiac. Blind beamforming for non Gaussian signals. IEE
Proceedings-F, 140(6):362–370, 1993.
73. L. Castedo, C. Escudero, and A. Dapena. A blind signal separation method for multiuser
communications. IEEE Trans. on Signal Processing, 45(5):1343–1348, 1997.
74. N. Charkani and Y. Deville. Self-adaptive separation of convolutively mixed signals
with a recursive structure, part i: Stability analysis and optimization of the asymptotic
behaviour. Signal Processing, 73:225–254, 1999.
75. T. Chen, Y. Hua, and W. Yan. Global convergence of Oja’s subspace algorithm forprincipal component extraction. IEEE Trans. on Neural Networks, 9(1):58–67, 1998.
76. P. Chevalier. Optimal separation of independent narrow-band sources: Concept andperformance. Signal Processing, 73:27–47, 1999.
77. P. Chevalier, V. Capdevielle, and P. Comon. Performance of HO blind source separation
methods: Experimental results on ionopheric HF links. In Proc. Int. Workshop on Independent Component Analysis and Signal Separation (ICA’99), pages 443–448, Aussois,
France, 1999.
78. S. Choi and A. Cichocki. Blind signal deconvolution by spatio-temporal decorrelationand demixing. In Proc. IEEE Workshop on Neural Networks for Signal Processing(NNSP·97), pages 426–435, Amelia Island, Florida, September 1997.
79. A. Cichocki, S.-I. Amari, and J. Cao. Blind separation of delayed and convolved sources
with self-adaptive learning rate. In Proc. Int. Symp. on Nonlinear Theory and Applications(NOLTA’96), pages 229–232, Kochi, Japan, 1996.
80. A. Cichocki, S. C. Douglas, and S.-I. Amari. Robust techniques for independent component analysis with noisy data. Neurocomputing, 22:113–129, 1998.
81. A. Cichocki, J. Karhunen, W. Kasprzak, and R. Vigário. Neural networks for blindseparation with unknown number of sources. Neurocomputing, 24:55–93, 1999.
82. A. Cichocki and L. Moszczynski. A new learning algorithm for blind separation ofsources. Electronics Letters, 28(21):1986–1987, 1992.
83. A. Cichocki and R. Unbehauen. Neural Networks for Signal Processing and Optimization. Wiley, 1994.
84. A. Cichocki and R. Unbehauen. Robust neural networks with on-line learning for blindidentification and blind separation of sources. IEEE Trans. on Circuits and Systems,
43(11):894–906, 1996.
85. A. Cichocki, R. Unbehauen, and E. Rummert. Robust learning algorithm for blindseparation of signals. Electronics Letters, 30(17):1386–1387, 1994.

454

REFERENCES

86. A. Cichocki, L. Zhang, S. Choi, and S.-I. Amari. Nonlinear dynamic independent
component analysis using state-space and neural network models. In Proc. Int. Workshop
on Independent Component Analysis and Signal Separation (ICA’99), pages 99–104,
Aussois, France, 1999.
87. R. R. Coifman and D. L. Donoho. Translation-invariant de-noising. Technical report,
Department of Statistics, Stanford University, Stanford, California, 1995.
88. P. Comon. Separation of stochastic processes. In Proc. Workshop on Higher-Order
Specral Analysis, pages 174 – 179, Vail, Colorado, 1989.
89. P. Comon. Independent component analysis—a new concept?
36:287–314, 1994.

Signal Processing,

90. P. Comon. Contrasts for multichannel blind deconvolution. Signal Processing Letters,
3(7):209–211, 1996.
91. P. Comon and P. Chevalier. Blind source separation: Models, concepts, algorithms
and performance. In S. Haykin, editor, Unsupervised Adaptive Filtering, Vol. I, pages
191–235. Wiley, 2000.
92. P. Comon and G. Golub. Tracking a few extreme singular values and vectors in signal
processing. Proceedings of the IEEE, 78:1327–1343, 1990.
93. P. Comon, C. Jutten, and J. Hérault. Blind separation of sources, Part II: Problems
statement. Signal Processsing, 24:11–20, 1991.
94. P. Comon and B. Mourrain. Decomposition of quantics in sums of powers of linear
forms. Signal Processing, 53(2):93–107, 1996.
95. D. Cook, A. Buja, and J. Cabrera. Projection pursuit indexes based on orthonormal
function expansions. J. of Computational and Graphical Statistics, 2(3):225–250, 1993.
96. G.W. Cottrell, P.W. Munro, and D. Zipser. Learning internal representations from grayscale images: An example of extensional programming. In Proc. Ninth Annual Conference of the Cognitive Science Society, pages 462–473, 1987.
97. T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, 1991.
98. R. Cristescu, J. Joutsensalo, J. Karhunen, and E. Oja. A complexity minimization
approach for estimating fading channels in CDMA communications. In Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000), pages
527–532, Helsinki, Finland, June 2000.
99. R. Cristescu, T. Ristaniemi, J. Joutsensalo, and J. Karhunen. Blind separation of convolved mixtures for CDMA systems. In Proc. Tenth European Signal Processing Conference (EUSIPCO2000), pages 619–622, Tampere, Finland, 2000.
100. R. Cristescu, T. Ristaniemi, J. Joutsensalo, and J. Karhunen. Delay estimation in CDMAcommunications using a Fast ICA algorithm. In Proc. Int. Workshop on Independent
Component Analysis and Blind Signal Separation (ICA2000), pages 105–110, Helsinki,
Finland, 2000.
101. S. Cruces and L. Castedo. Stability analysis of adaptive algorithms for blind source
separation of convolutive mixtures. Signal Processing, 78(3):265–276, 1999.
102. I. Daubechies. Ten Lectures on Wavelets. Society for Industrial and Applied Math.,
Philadelphia, 1992.

REFERENCES

455

103. J. G. Daugman. Complete disrete 2-D Gabor transforms by neural networks for image
analysis and compression. IEEE Trans. on Acoustics Speech and Signal Processing,36(7):1169–1179, 1988.
104. G. Deco and W. Brauer. Nonlinear higher-order statistical decorrelation by volumeconserving neural architectures. Neural Networks, 8:525–535, 1995.
105. G. Deco and D. Obradovic. An Information-Theoretic Approach to Neural Computing.
Springer Verlag, 1996.
106. S. Degerine and R. Malki. Second-order blind separation of sources based on canonicalpartial innovations. IEEE Trans. on Signal Processing, 48(3):629–641, 2000.
107. N. Delfosse and P. Loubaton. Adaptive blind separation of independent sources: adeflation approach. Signal Processing, 45:59–83, 1995.
108. N. Delfosse and P. Loubaton. Adaptive blind separation of convolutive mixtures. In
Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP’97), pages
2940–2943, 1996.
109. P. Devijver and J. Kittler. Pattern Recognition: A Statistical Approach. Prentice Hall,1982.
110. Y. Deville and L. Andry. Application of blind source separation techniques to multitag contactless identification systems. In Proc. Int. Symp. on Nonlinear Theory and its
Applications (NOLTA’95), pages 73–78, Las Vegas, Nevada, 1995.
111. Y. Deville, J. Damour, and N. Charkani. Improved multi-tag radio-frequency identification systems based on new source separation neural networks. In Proc. Int. Workshop on
Independent Component Analysis and Blind Source Separation (ICA’99), pages 449–454,
Aussois, France, 1999.
112. K. I. Diamantaras and S. Y. Kung. Principal Component Neural Networks: Theory andApplications. Wiley, 1996.
113. K. I. Diamantaras, A. P. Petropulu, and B. Chen. Blind two-input-two-output FIR
channel identification based on second-order statistics. IEEE Trans. on Signal Processing,48(2):534–542, 2000.
114. D. L. Donoho. On minimum entropy deconvolution. In Applied Time Series Analysis II,
pages 565–608. Academic Press, 1981.
115. D. L. Donoho. Nature vs. math: Interpreting independent component analysis in lightof recent work in harmonic analysis. In Proc. Int. Workshop on Independent Component
Analysis and Blind Signal Separation (ICA2000), pages 459–470, Helsinki, Finland,
2000.
116. D. L. Donoho, I. M. Johnstone, G. Kerkyacharian, and D. Picard. Wavelet shrinkage:asymptopia? Journal of the Royal Statistical Society, Ser. B, 57:301–337, 1995.
117. S. C. Douglas. Equivariant adaptive selective transmission. IEEE Trans. on Signal
Processing, 47(5):1223–1231, May 1999.
118. S. C. Douglas and S.-I. Amari. Natural-gradient adaptation. In S. Haykin, editor,
Unsupervised Adaptive Filtering, volume 1, pages 13–61. Wiley, 2000.
119. S. C. Douglas, A. Cichocki, , and S.-I. Amari. A bias removal technique for blind source
separation with noisy measurements. Electronics Letters, 34:1379–1380, 1998.
120. S. C. Douglas and A. Cichocki. Neural networks for blind decorrelation of signals. IEEE
Trans. on Signal Processing, 45(11):2829–2842, 1997.

456

REFERENCES

121. S. C. Douglas, A. Cichocki, and S.-I. Amari. Multichannel blind separation and deconvolution of sources with arbitrary distributions. In Proc. IEEE Workshop on Neural
Networks for Signal Processing (NNSP’97), pages 436–445, Amelia Island, Florida,
1997.
122. S. C. Douglas and S. Haykin. Relationships between blind deconvolution and blindsource separation. In S. Haykin, editor, Unsupervised Adaptive Filtering, volume 2,
pages 113–145. Wiley, 2000.
123. F. Ehlers and H. Schuster. Blind separation of convolutive mixtures and an application in
automatic speech recognition in a noisy environment. IEEE Trans. on Signal Processing,45(10):2608–2612, 1997.
124. B. Emile and P. Comon. Estimation of time delays between unknown colored signals.
Signal Processing, 69(1):93–100, 1998.
125. R. M. Everson and S. Roberts. Independent component analysis: a flexible nonlinearityand decorrelating manifold approach. Neural Computation, 11(8):1957–1983, 1999.
126. R. M. Everson and S. J. Roberts. Particle filters for non-stationary ICA. In M. Girolami,
editor, Advances in Independent Component Analysis, pages 23–41. Springer-Verlag,
2000.
127. H. Farid and E. H. Adelson. Separating reflections from images by use of independent
component analysis. J. of the Optical Society of America, 16(9):2136–2145, 1999.
128. H. G. Feichtinger and T. Strohmer, editors. Gabor Analysis and Algorithms. Birkhauser,
1997.
129. W. Feller. Probability Theory and Its Applications. Wiley, 3rd edition, 1968.
130. M. Feng and K.-D. Kammayer. Application of source separation algorithms for mobile
communications environment. In Proc. Int. Workshop on Independent Component Analysis and Blind Source Separation (ICA’99), pages 431–436, Aussois, France, January
1999.
131. D.J. Field. What is the goal of sensory coding? Neural Computation, 6:559–601, 1994.
132. S. Fiori. Blind separation of circularly distributed sources by neural extended APEX
algorithm. Neurocomputing, 34:239–252, 2000.
133. S. Fiori. Blind signal processing by the adaptive activation function neurons. Neural
Networks, 13(6):597–611, 2000.
134. J. Fisher and J. Principe. Entropy manipulation of arbitrary nonlinear mappings. In
J. Principe et al., editor, Neural Networks for Signal Processing VII, pages 14–23. IEEE
Press, New York, 1997.
135. R. Fletcher. Practical Methods of Optimization. Wiley, 2nd edition, 1987.
136. P. Földiák. Adaptive network for optimal linear feature extraction. In Proc. Int. Joint
Conf. on Neural Networks, pages 401 – 406, Washington DC, 1989.
137. J. H. Friedman and J. W. Tukey. A projection pursuit algorithm for exploratory data
analysis. IEEE Trans. of Computers, c-23(9):881–890, 1974.
138. J.H. Friedman. Exploratory projection pursuit. J. of the American Statistical Association,
82(397):249–266, 1987.
139. C. Fyfe and R. Baddeley. Non-linear data structure extraction using simple Hebbian
networks. Biological Cybernetics, 72:533–541, 1995.

REFERENCES

457

140. M. Gaeta and J.-L. Lacoume. Source separation without prior knowledge: the maximum
likelihood solution. In Proc. EUSIPCO’90, pages 621–624, 1990.
141. W. Gardner. Introduction to Random Processes with Applications to Signals and Systems.
Macmillan, 1986.
142. A. Gelman, J. Carlin, H. Stern, and D. Rubin. Bayesian Data Analysis. Chapman &Hall/CRC Press, Boca Raton, Florida, 1995.
143. G. Giannakis, Y. Hua, P. Stoica, and L. Tong. Signal Processing Advances in Communications, vol. 1: Trends in Channel Estimation and Equalization. Prentice Hall,
2001.
144. G. Giannakis, Y. Hua, P. Stoica, and L. Tong. Signal Processing Advances in Communications, vol. 2: Trends in Single- and Multi-User Systems. Prentice Hall, 2001.
145. G. Giannakis, Y. Inouye, and J. M. Mendel. Cumulant-based identification of multichannel moving-average processes. IEEE Trans. on Automat. Contr., 34:783–787, July
1989.
146. G. Giannakis and C. Tepedelenlioglu. Basis expansion models and diversity techniques
for blind identification and equalization of time-varying channels. Proceedings of theIEEE, 86(10):1969–1986, 1998.
147. X. Giannakopoulos, J. Karhunen, and E. Oja. Experimental comparison of neuralalgorithms for independent component analysis and blind separation. Int. J. of Neural
Systems, 9(2):651–656, 1999.
148. M. Girolami. An alternative perspective on adaptive independent component analysis
algorithms. Neural Computation, 10(8):2103–2114, 1998.
149. M. Girolami. Self-Organising Neural Networks - Independent Component Analysis andBlind Source Separation. Springer-Verlag, 1999.
150. M. Girolami, editor. Advances in Independent Component Analysis. Springer-Verlag,
2000.
151. M. Girolami and C. Fyfe. An extended exploratory projection pursuit network with linear
and nonlinear anti-hebbian connections applied to the cocktail party problem. Neural
Networks, 10:1607–1618, 1997.
152. D. N. Godard. Self-recovering equalization and carrier tracking in two-dimensional data
communication systems. IEEE Trans. on Communications, 28:1867–1875, 1980.
153. G. Golub and C. van Loan. Matrix Computations. The Johns Hopkins University Press,
3rd edition, 1996.
154. R. Gonzales and P. Wintz. Digital Image Processing. Addison-Wesley, 1987.
155. A. Gorokhov and P. Loubaton. Subspace-based techniques for blind separation of convolutive mixtures with temporally correlated sources. IEEE Trans. Circuits and Systems
I, 44(9):813–820, 1997.
156. A. Gorokhov and P. Loubaton. Blind identification of MIMO-FIR systems: A generalized
linear prediction approach. Signal Processing, 73:105–124, 1999.
157. R. Gray and L. Davisson. Random Processes: A Mathematical Approach for Engineers.
Prentice Hall, 1986.
158. O. Grellier and P. Comon. Blind separation of discrete sources. IEEE Signal Processing
Letters, 5(8):212–214, 1998.

458

REFERENCES

159. S. I. Grossman. Elementary Linear Algebra. Wadsworth, 1984.
160. P. Hall. Polynomial projection pursuit. The Annals of Statistics, 17:589–605, 1989.
161. F.M. Ham and N.A. Faour. Infrasound signal separation using independent component
analysis. In Proc. 21st Seismic Reseach Symposium: Technologies for Monitoring theComprehensive Nuclear-Test-Ban Treaty, Las Vegas, Nevada, 1999.
162. M. Hämäläinen, R. Hari, R. Ilmoniemi, J. Knuutila, and O. V. Lounasmaa.
Magnetoencephalography—theory, instrumentation, and applications to noninvasive
studies of the working human brain. Reviews of Modern Physics, 65(2):413–497, 1993.
163. F.R. Hampel, E.M. Ronchetti, P.J. Rousseuw, and W.A. Stahel. Robust Statistics. Wiley,1986.
164. L. K. Hansen. Blind separation of noisy image mixtures. In M. Girolami, editor, Advancesin Independent Component Analysis, pages 161–181. Springer-Verlag, 2000.
165. R. Hari. Magnetoencephalography as a tool of clinical neurophysiology. In E. Niedermeyer and F. Lopes da Silva, editors, Electroencephalography. Basic principles, clinicalapplications, and related fields, pages 1035–1061. Baltimore: Williams & Wilkins,
1993.
166. H. H. Harman. Modern Factor Analysis. University of Chicago Press, 2nd edition, 1967.
167. T. Hastie and W. Stuetzle. Principal curves. Journal of the American Statistical Association, 84:502–516, 1989.
168. M. Hayes. Statistical Digital Signal Processing and Modeling. Wiley, 1996.
169. S. Haykin. Modern Filters. Macmillan, 1989.
170. S. Haykin, editor. Blind Deconvolution. Prentice Hall, 1994.
171. S. Haykin. Adaptive Filter Theory. Prentice Hall, 3rd edition, 1996.
172. S. Haykin. Neural Networks - A Comprehensive Foundation. Prentice Hall, 2nd edition,
1998.
173. S. Haykin, editor. Unsupervised Adaptive Filtering, Vol. 1: Blind Source Separation.
Wiley, 2000.
174. S. Haykin, editor. Unsupervised Adaptive Filtering, Vol. 2: Blind Deconvolution. Wiley,2000.
175. Z. He, L. Yang, J. Liu, Z. Lu, C. He, and Y. Shi. Blind source separation using clusteringbased multivariate density estimation algorithm. IEEE Trans. on Signal Processing,48(2):575–579, 2000.
176. R. Hecht-Nielsen. Replicator neural networks for universal optimal source coding.
Science, 269:1860–1863, 1995.
177. R. Hecht-Nielsen. Data manifolds, natural coordinates, replicator neural networks, andoptimal source coding. In Proc. Int. Conf. on Neural Information Processing, pages
41–45, Hong Kong, 1996.
178. J. Hérault and B. Ans. Circuits neuronaux à synapses modifiables: décodage de messages
composites par apprentissage non supervisé. C.-R. de l’Académie des Sciences, 299(III13):525–528, 1984.
179. J. Hérault, C. Jutten, and B. Ans. Détection de grandeurs primitives dans un message
composite par une architecture de calcul neuromimétique en apprentissage non supervisé.
In Actes du Xème colloque GRETSI, pages 1017–1022, Nice, France, 1985.

REFERENCES

459

180. G. Hinton and D. van Camp. Keeping neural networks simple by minimizing thedescription length of the weights. In Proc. of the 6th Annual ACM Conf. on Computational
Learning Theory, pages 5–13, Santa Cruz, CA, 1993.
181. S. Hochreiter and J. Schmidhuber. Feature extraction through LOCOCODE. NeuralComputation, 11(3):679–714, 1999.
182. H. Holma and A. Toskala, editors. WCDMA for UMTS. Wiley, 2000.
183. L. Holmström, P. Koistinen, J. Laaksonen, and E. Oja. Comparison of neural andstatistical classifiers - taxonomy and two case studies. IEEE Trans. on Neural Networks,
8:5–17, 1997.
184. M. Honig and V. Poor. Adaptive interference suppression. In Wireless Communications:
Signal Processing Perspectives, pages 64–128. Prentice Hall, 1998.
185. H. Hotelling. Analysis of a complex of statistical variables into principal components.
J. of Educational Psychology, 24:417 – 441, 1933.
186. P. O. Hoyer and A. Hyvärinen. Independent component analysis applied to feature
extraction from colour and stereo images. Network: Computation in Neural Systems,
11(3):191–210, 2000.
187. P. O. Hoyer and A. Hyvärinen. Modelling chromatic and binocular properties of V1
topography. In Proc. Int. Conf. on Neural Information Processing (ICONIP’00), Taejon,
Korea, 2000.
188. P.J. Huber. Robust Statistics. Wiley, 1981.
189. P.J. Huber. Projection pursuit. The Annals of Statistics, 13(2):435–475, 1985.
190. M. Huotilainen, R. J. Ilmoniemi, H. Tiitinen, J. Lavaikainen, K. Alho, M. Kajola, and
R. Näätänen. The projection method in removing eye-blink artefacts from multichannel
MEG measurements. In C. Baumgartner et al., editor, Biomagnetism: Fundamental
Research and Clinical Applications (Proc. Int. Conf. on Biomagnetism), pages 363–367.
Elsevier, 1995.
191. J. Hurri, A. Hyvärinen, and E. Oja. Wavelets and natural image statistics. In Proc.
Scandinavian Conf. on Image Analysis ’97, Lappeenranta, Finland, 1997.
192. A. Hyvärinen. A family of fixed-point algorithms for independent component analysis.
In Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP’97), pages
3917–3920, Munich, Germany, 1997.
193. A. Hyvärinen. One-unit contrast functions for independent component analysis: A
statistical analysis. In Neural Networks for Signal Processing VII (Proc. IEEE Workshop
on Neural Networks for Signal Processing), pages 388–397, Amelia Island, Florida,
1997.
194. A. Hyvärinen. Independent component analysis for time-dependent stochastic processes.
In Proc. Int. Conf. on Artificial Neural Networks (ICANN’98), pages 135–140, Skövde,Sweden, 1998.
195. A. Hyvärinen. Independent component analysis in the presence of gaussian noise by
maximizing joint likelihood. Neurocomputing, 22:49–67, 1998.
196. A. Hyvärinen. New approximations of differential entropy for independent component
analysis and projection pursuit. In Advances in Neural Information Processing Systems,
volume 10, pages 273–279. MIT Press, 1998.

460

REFERENCES

197. A. Hyvärinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE Trans. on Neural Networks, 10(3):626–634, 1999.
198. A. Hyvärinen. Fast independent component analysis with noisy data using gaussian
moments. In Proc. Int. Symp. on Circuits and Systems, pages V57–V61, Orlando,
Florida, 1999.
199. A. Hyvärinen. Gaussian moments for noisy independent component analysis. IEEE
Signal Processing Letters, 6(6):145–147, 1999.
200. A. Hyvärinen. Sparse code shrinkage: Denoising of nongaussian data by maximum
likelihood estimation. Neural Computation, 11(7):1739–1768, 1999.
201. A. Hyvärinen. Survey on independent component analysis. Neural Computing Surveys,
2:94–128, 1999.
202. A. Hyvärinen. Complexity pursuit: Separating interesting components from time-series.
Neural Computation, 13, 2001.
203. A. Hyvärinen, R. Cristescu, and E. Oja. A fast algorithm for estimating overcomplete
ICA bases for image windows. In Proc. Int. Joint Conf. on Neural Networks, pages
894–899, Washington, D.C., 1999.
204. A. Hyvärinen and P. O. Hoyer. Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces. Neural Computation,
12(7):1705–1720, 2000.
205. A. Hyvärinen and P. O. Hoyer. Emergence of topography and complex cell properties
from natural images using extensions of ICA. In Advances in Neural Information
Processing Systems, volume 12, pages 827–833. MIT Press, 2000.
206. A. Hyvärinen, P. O. Hoyer, and M. Inki. Topographic independent component analysis.
Neural Computation, 13, 2001.
207. A. Hyvärinen, P. O. Hoyer, and E. Oja. Image denoising by sparse code shrinkage. In
S. Haykin and B. Kosko, editors, Intelligent Signal Processing. IEEE Press, 2001.
208. A. Hyvärinen and M. Inki. Estimating overcomplete independent component bases fromimage windows. 2001. submitted manuscript.
209. A. Hyvärinen and R. Karthikesh. Sparse priors on the mixing matrix in independent
component analysis. In Proc. Int. Workshop on Independent Component Analysis andBlind Signal Separation (ICA2000), pages 477–452, Helsinki, Finland, 2000.
210. A. Hyvärinen and E. Oja. A fast fixed-point algorithm for independent component
analysis. Neural Computation, 9(7):1483–1492, 1997.
211. A. Hyvärinen and E. Oja. Independent component analysis by general nonlinear Hebbianlike learning rules. Signal Processing, 64(3):301–313, 1998.
212. A. Hyvärinen and E. Oja. Independent component analysis: Algorithms and applications.
Neural Networks, 13(4-5):411–430, 2000.
213. A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence
and uniqueness results. Neural Networks, 12(3):429–439, 1999.
214. A. Hyvärinen, J. Särelä, and R. Vigário. Spikes and bumps: Artefacts generated byindependent component analysis with insufficient sample size. In Proc. Int. Workshop
on Independent Component Analysis and Signal Separation (ICA’99), pages 425–429,
Aussois, France, 1999.

REFERENCES

461

215. S. Ikeda. ICA on noisy data: A factor analysis approach. In M. Girolami, editor,
Advances in Independent Component Analysis, pages 201–215. Springer-Verlag, 2000.
216. S. Ikeda and N. Murata. A method of ICA in time-frequency domain. In Proc. Int.
Workshop on Independent Component Analysis and Signal Separation (ICA’99), pages
365–370, Aussois, France, 1999.
217. Y. Inouye. Criteria for blind deconvolution of multichannel linear time-invariant systems.
IEEE Trans. on Signal Processing, 46(12):3432–3436, 1998.
218. Y. Inouye and K. Hirano. Cumulant-based blind identification of linear multi-inputmulti-output systems driven by colored inputs. IEEE Trans. on Signal Processing,45(6):1543–1552, 1997.
219. C. L. Isbell and P. Viola. Restructuring sparse high-dimensional data for effective retreval.
In Advances in Neural Information Processing Systems, volume 11. MIT Press, 1999.
220. N. Japkowitz, S. J. Hanson, and M. A. Gluck. Nonlinear autoassociation is not equivalent
to PCA. Neural Computation, 12(3):531–545, 2000.
221. B. W. Jervis, M. Coelho, and G. Morgan. Effect on EEG responses of removing ocular
artifacts by proportional EOG subtraction. Medical and Biological Engineering andComputing, 27:484–490, 1989.
222. M.C. Jones and R. Sibson. What is projection pursuit ? J. of the Royal Statistical Society,
Ser. A, 150:1–36, 1987.
223. J. Joutsensalo and T. Ristaniemi. Learning algorithms for blind multiuser detection in
CDMA downlink. In Proc. IEEE 9th International Symposium on Personal, Indoor andMobile Radio Communications (PIMRC ’98), pages 267–270, Boston, Massachusetts,
1998.
224. C. Johnson Jr., P. Schniter, I. Fijalkow, and L. Tong et al. The core of FSE-CMA behavior
theory. In S. Haykin, editor, Unsupervised Adaptive Filtering, volume 2, pages 13–112.
Wiley, New York, 2000.
225. T. P. Jung, C. Humphries, T.-W. Lee, S. Makeig, M. J. McKeown, V. Iragui, and
T. Sejnowski. Extended ICA removes artifacts from electroencephalographic recordings.
In Advances in Neural Information Processing Systems, volume 10. MIT Press, 1998.
226. C. Jutten. Calcul neuromimétique et traitement du signal, analyse en composantes
indépendantes. PhD thesis, INPG, Univ. Grenoble, France, 1987. (in French).
227. C. Jutten. Source separation: from dusk till dawn. In Proc. 2nd Int. Workshop on
Independent Component Analysis and Blind Source Separation (ICA’2000), pages 15–
26, Helsinki, Finland, 2000.
228. C. Jutten and J. Hérault. Blind separation of sources, part I: An adaptive algorithm basedon neuromimetic architecture. Signal Processing, 24:1–10, 1991.
229. A. Kaban and M. Girolami. Clustering of text documents by skewness maximization. In
Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation(ICA2000), pages 435–440, Helsinki, Finland, 2000.
230. E. R. Kandel, J. H. Schwartz, and T. M. Jessel, editors. The Principles of Neural Science.
Prentice Hall, 3rd edition, 1991.
231. J. Karhunen, A. Cichocki, W. Kasprzak, and P Pajunen. On neural blind separation withnoise suppression and redundancy reduction. Int. Journal of Neural Systems, 8(2):219–
237, 1997.

462

REFERENCES

232. J. Karhunen and J. Joutsensalo. Representation and separation of signals using nonlinear
PCA type learning. Neural Networks, 7(1):113–127, 1994.
233. J. Karhunen and J. Joutsensalo. Generalizations of principal component analysis, optimization problems, and neural networks. Neural Networks, 8(4):549–562, 1995.
234. J. Karhunen, S. Malaroiou, and M. Ilmoniemi. Local linear independent component
analysis based on clustering. Int. J. of Neural Systems, 10(6), 2000.
235. J. Karhunen, E. Oja, L. Wang, R. Vigário, and J. Joutsensalo. A class of neural networks
for independent component analysis. IEEE Trans. on Neural Networks, 8(3):486–504,
1997.
236. J. Karhunen, P. Pajunen, and E. Oja. The nonlinear PCA criterion in blind source
separation: Relations with other approaches. Neurocomputing, 22:5–20, 1998.
237. K. Karhunen. Zur Spektraltheorie stochastischer Prozesse. Ann. Acad. Sci. Fennicae,
34, 1946.
238. E. Kaukoranta, M. Hämäläinen, J. Sarvas, and R. Hari. Mixed and sensory nerve stimulations activate different cytoarchitectonic areas in the human primary somatosensory
cortex SI. Experimental Brain Research, 63(1):60–66, 1986.
239. M. Kawamoto, K. Matsuoka, and M. Ohnishi. A method for blind separation forconvolved non-stationary signals. Neurocomputing, 22:157–171, 1998.
240. M. Kawamoto, K. Matsuoka, and M. Oya. Blind separation of sources using temporalcorrelation of the observed signals. IEICE Trans. Fundamentals, E80-A(4):695–704,
1997.
241. S. Kay. Modern Spectral Estimation: Theory and Application. Prentice Hall, 1988.
242. S. Kay. Fundamentals of Statistical Signal Processing - Estimation Theory. Prentice
Hall, 1993.
243. M. Kendall. Multivariate Analysis. Griffin, 1975.
244. M. Kendall and A. Stuart. The Advanced Theory of Statistics, Vols. 1–3. Macmillan,
1976–1979.
245. K. Kiviluoto and E. Oja. Independent component analysis for parallel financial timeseries. In Proc. Int. Conf. on Neural Information Processing (ICONIP’98), volume 2,
pages 895–898, Tokyo, Japan, 1998.
246. H. Knuth. A bayesian approach to source separation. In Proc. Int. Workshop on Independent Component Analysis and Signal Separation (ICA’99), pages 283–288, Aussois,
France, 1999.
247. T. Kohonen. Self-Organizing Maps. Springer, 1995.
248. T. Kohonen. Emergence of invariant-feature detectors in the adaptive-subspace selforganizing map. Biological Cybernetics, 75:281–291, 1996.
249. V. Koivunen, M. Enescu, and E. Oja. Adaptive algorithm for blind separation from noisy
time-varying mixtures. Neural Computation, 13, 2001.
250. V. Koivunen and E. Oja. Predictor-corrector structure for real-time blind separation fromnoisy mixtures. In Proc. Int. Workshop on Independent Component Analysis and Signal
Separation (ICA’99), pages 479–484, Aussois, France, 1999.
251. T. Kolenda, L. K. Hansen, and S. Sigurdsson. Independent components in text. In
M. Girolami, editor, Advances in Independent Component Analysis, pages 235–256.
Springer-Verlag, 2000.

REFERENCES

463

252. M. A. Kramer. Nonlinear principal component analysis using autoassociative neuralnetworks. AIChE Journal, 37(2):233–243, 1991.
253. H.J. Kushner and D.S. Clark. Stochastic approximation methods for constrained andunconstrained systems. Springer-Verlag, 1978.
254. J.-L. Lacoume and P. Ruiz. Sources identification: a solution based on cumulants. In
Proc. IEEE ASSP Workshop, Minneapolis, Minnesota, 1988.
255. B. Laheld and J.-F. Cardoso. Adaptive source separation with uniform performance. In
Proc. EUSIPCO, pages 183–186, Edinburgh, 1994.
256. R. H. Lambert. Multichannel Blind Deconvolution: FIR Matrix Algebra and Separationof Multipath Mixtures. PhD thesis, Univ. of Southern California, 1996.
257. R. H. Lambert and C. L. Nikias. Blind deconvolution of multipath mixtures. In S. Haykin,editor, Unsupervised Adaptive Filtering, Vol. I, pages 377–436. Wiley, 2000.
258. H. Lappalainen. Ensemble learning for independent component analysis. In Proc. Int.
Workshop on Independent Component Analysis and Signal Separation (ICA’99), pages
7–12, Aussois, France, 1999.
259. H. Lappalainen and A. Honkela. Bayesian nonlinear independent component analysis
by multi-layer perceptrons. In M. Girolami, editor, Advances in Independent Component
Analysis, pages 93–121. Springer-Verlag, 2000.
260. H. Lappalainen and J. W. Miskin. Ensemble learning. In M. Girolami, editor, Advancesin Independent Component Analysis, pages 75–92. Springer-Verlag, 2000.
261. L. De Lathauwer. Signal Processing by Multilinear Algebra. PhD thesis, Faculty ofEngineering, K. U. Leuven, Leuven, Belgium, 1997.
262. L. De Lathauwer, P. Comon, B. De Moor, and J. Vandewalle. Higher-order power method,
application in independent component analysis. In Proc. Int. Symp. on Nonlinear Theory
and its Applications (NOLTA’95), pages 10–14, Las Vegas, Nevada, 1995.
263. L. De Lathauwer, B. De Moor, and J. Vandewalle. A technique for higher-order-only blindsource separation. In Proc. Int. Conf. on Neural Information Processing (ICONIP’96),
Hong Kong, 1996.
264. M. LeBlanc and R. Tibshirani. Adaptive principal surfaces. J. of the Amer. Stat. Assoc.,
89(425):53 – 64, 1994.
265. C.-C. Lee and J.-H. Lee. An efficient method for blind digital signal separation of arraydata. Signal Processing, 77:229–234, 1999.
266. T. S. Lee. Image representation using 2D Gabor wavelets. IEEE Trans. on PatternAnalysis and Machine Intelligence, 18(10):959–971, 1996.
267. T.-W. Lee. Independent Component Analysis - Theory and Applications. Kluwer, 1998.
268. T.-W. Lee, A. J. Bell, and R. Lambert. Blind separation of delayed and convolved sources.
In Advances in Neural Information Processing Systems, volume 9, pages 758–764. MIT
Press, 1997.
269. T.-W. Lee, M. Girolami, A.J. Bell, and T.J. Sejnowski. A unifying information-theoretic
framework for independent component analysis. Computers and Mathematics with
Applications, 31(11):1–12, 2000.
270. T.-W. Lee, M. Girolami, and T. J. Sejnowski. Independent component analysis using an
extended infomax algorithm for mixed sub-gaussian and super-gaussian sources. NeuralComputation, 11(2):417–441, 1999.

464

REFERENCES

271. T.-W. Lee, B.U. Koehler, and R. Orglmeister. Blind source separation of nonlinear
mixing models. In Neural Networks for Signal Processing VII, pages 406–415. IEEE
Press, 1997.
272. T.-W. Lee, M.S. Lewicki, M. Girolami, and T.J. Sejnowski. Blind source separation of
more sources than mixtures using overcomplete representations. IEEE Signal Processing
Letters, 4(5), 1999.
273. T.-W. Lee, M.S. Lewicki, and T.J. Sejnowski. ICA mixture models for unsupervisedclassification of non-gaussian sources and automatic context switching in blind signal
separation. IEEE Trans. Pattern Recognition and Machine Intelligence, 22(10):1–12,
2000.
274. M. Lewicki and B. Olshausen. A probabilistic framework for the adaptation and comparison of image codes. J. Opt. Soc. Am. A: Optics, Image Science, and Vision, 16(7):1587–
1601, 1998.
275. M. Lewicki and T.J. Sejnowski. Learning overcomplete representations. Neural Computation, 12:337–365, 2000.
276. T. Li and N. Sidiropoulos. Blind digital signal separation using successive interference
cancellation iterative least squares. IEEE Trans. on Signal Processing, 48(11):3146–
3152, 2000.
277. J. K. Lin. Factorizing multivariate function classes. In Advances in Neural Information
Processing Systems, volume 10, pages 563–569. The MIT Press, 1998.
278. J. K. Lin. Factorizing probability density functions: Generalizing ICA. In Proc. First Int.
Workshop on Independent Component Analysis and Signal Separation (ICA’99), pages
313–318, Aussois, France, 1999.
279. J. K. Lin, D. G. Grier, and J. D. Cowan. Faithful representation of separable distributions.
Neural Computation, 9(6):1305–1320, 1997.
280. U. Lindgren and H. Broman. Source separation using a criterion based on second-order
statistics. IEEE Trans. on Signal Processing, 46(7):1837–1850, 1998.
281. U. Lindgren, T. Wigren, and H. Broman. On local convergence of a class of blindseparation algorithms. IEEE Trans. on Signal Processing, 43:3054–3058, 1995.
282. R. Linsker. Self-organization in a perceptual network. Computer, 21:105–117, 1988.
283. M. Loève. Fonctions aléatoires du second ordre. In P. Lévy, editor, Processus stochastiques et mouvement Brownien, page 299. Gauthier - Villars, Paris, 1948.
284. D. Luenberger. Optimization by Vector Space Methods. Wiley, 1969.
285. J. Luo, B. Hu, X.-T. Ling, and R.-W. Liu. Principal independent component analysis.
IEEE Trans. on Neural Networks, 10(4):912–917, 1999.
286. O. Macchi and E. Moreau. Adaptive unsupervised separation of discrete sources. Signal
Processing, 73:49–66, 1999.
287. U. Madhow. Blind adaptive interference suppression for direct-sequence CDMA. Proceedings of the IEEE, 86(10):2049–2069, 1998.
288. S. Makeig, T.-P. Jung, A. J. Bell, D. Ghahramani, and T. Sejnowski. Blind separationof auditory event-related brain responses into independent components. Proc. National
Academy of Sciences (USA), 94:10979–10984, 1997.

REFERENCES

465

289. S. Malaroiu, K. Kiviluoto, and E. Oja. Time series prediction with independent component analysis. In Proc. Int. Conf. on Advanced Investment Technology, Gold Coast,
Australia, 2000.
290. S. G. Mallat. A theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 11:674–693,
1989.
291. Z. Malouche and O. Macchi. Adaptive unsupervised extraction of one component of alinear mixture with a single neuron. IEEE Trans. on Neural Networks, 9(1):123–138,
1998.
292. A. Mansour, C. Jutten, and P. Loubaton. Adaptive subspace algorithm for blind separationof independent sources in convolutive mixture. IEEE Trans. on Signal Processing,48(2):583–586, 2000.
293. K. Mardia, J. Kent, and J. Bibby. Multivariate Analysis. Academic Press, 1979.
294. L. Marple. Digital Spectral Analysis with Applications. Prentice Hall, 1987.
295. G. Marques and L. Almeida. Separation of nonlinear mixtures using pattern repulsion.
In Proc. Int. Workshop on Independent Component Analysis and Signal Separation(ICA’99), pages 277–282, Aussois, France, 1999.
296. K. Matsuoka, M. Ohya, and M. Kawamoto. A neural net for blind separation of nonstationary signals. Neural Networks, 8(3):411–419, 1995.
297. M. McKeown, S. Makeig, S. Brown, T.-P. Jung, S. Kindermann, A.J. Bell, V. Iragui, and
T. Sejnowski. Blind separation of functional magnetic resonance imaging (fMRI) data.
Human Brain Mapping, 6(5-6):368–372, 1998.
298. G. McLachlan and T. Krishnan. The EM Algorithm and Extensions. Wiley-Interscience,
1997.
299. J. Mendel. Lessons in Estimation Theory for Signal Processing, Communications, andControl. Prentice Hall, 1995.
300. Y. Miao and Y. Hua. Fast subspace tracking and neural network learning by a novel
information criterion. IEEE Trans. on Signal Processing, 46:1967–1979, 1998.
301. J. Miskin and D. J. C. MacKay. Ensemble learning for blind image separation anddeconvolution. In M. Girolami, editor, Advances in Independent Component Analysis,
pages 123–141. Springer-Verlag, 2000.
302. S. Mitra. Digital Signal Processing: A Computer-Based Approach. McGraw-Hill, 1998.
303. L. Molgedey and H. G. Schuster. Separation of a mixture of independent signals using
time delayed correlations. Physical Review Letters, 72:3634–3636, 1994.
304. T. Moon. The expectation-maximization algorithm. IEEE Signal Processing Magazine,
13(6):47–60, 1996.
305. E. Moreau and O. Macchi. Complex self-adaptive algorithms for source separation
based on higher order contrasts. In Proc. EUSIPCO’94, pages 1157–1160, Edinburgh,
Scotland, 1994.
306. E. Moreau and O. Macchi. High order contrasts for self-adaptive source separation. Int.
J. of Adaptive Control and Signal Processing, 10(1):19–46, 1996.
307. E. Moreau and J. C. Pesquet. Generalized contrasts for multichannel blind deconvolutionof linear systems. IEEE Signal Processing Letters, 4:182–183, 1997.

466

REFERENCES

308. D. Morrison. Multivariate Statistical Methods. McGraw-Hill, 1967.
309. J. Mosher, P. Lewis, and R. Leahy. Multidipole modelling and localization from spatiotemporal MEG data. IEEE Trans. Biomedical Engineering, 39:541–557, 1992.
310. E. Moulines, J.-F. Cardoso, and E. Gassiat. Maximum likelihood for blind separationand deconvolution of noisy signals using mixture models. In Proc. IEEE Int. Conf.
on Acoustics, Speech and Signal Processing (ICASSP’97), pages 3617–3620, Munich,Germany, 1997.
311. E. Moulines, J.-F. Cardoso, and S. Mayrargue. Subspace methods for blind identificationof multichannel FIR filters. IEEE Trans. on Signal Processing, 43:516–525, 1995.
312. K.-R. Müller, P. Philips, and A. Ziehe. JADETD : Combining higher-order statistics andtemporal information for blind source separation (with noise). In Proc. Int. Workshop
on Independent Component Analysis and Signal Separation (ICA’99), pages 87–92,
Aussois, France, 1999.
313. J.-P. Nadal, E. Korutcheva, and F. Aires. Blind source processing in the presence of weaksources. Neural Networks, 13(6):589–596, 2000.
314. J.-P. Nadal and N. Parga. Non-linear neurons in the low noise limit: a factorial code
maximizes information transfer. Network, 5:565–581, 1994.
315. A. Nandi, editor. Blind Estimation Using Higher-Order Statistics. Kluwer, 1999.
316. G. Nason. Three-dimensional projection pursuit. Applied Statistics, 44:411–430, 1995.
317. E. Niedermeyer and F. Lopes da Silva, editors. Electroencephalography. Basic Principles, Clinical Applications, and Related fields. Williams & Wilkins, 1993.
318. C. Nikias and J. Mendel. Signal processing with higher-order spectra. IEEE Signal
Processing Magazine, pages 10–37, July 1993.
319. C. Nikias and A. Petropulu. Higher-Order Spectral Analysis - A Nonlinear Signal
Processing Framework. Prentice Hall, 1993.
320. B. Noble and J. Daniel. Applied Linear Algebra. Prentice Hall, 3rd edition, 1988.
321. D. Nuzillard and J.-M. Nuzillard. Application of blind source separation to 1-d and 2-dnuclear magnetic resonance spectroscopy. IEEE Signal Processing Letters, 5(8):209–
211, 1998.
322. D. Obradovic and G. Deco. Information maximization and independent component
analysis: Is there a difference? Neural Computation, 10(8):2085–2101, 1998.
323. E. Oja. A simplified neuron model as a principal component analyzer. J. of Mathematical
Biology, 15:267–273, 1982.
324. E. Oja. Subspace Methods of Pattern Recognition. Research Studies Press, England, andWiley, USA, 1983.
325. E. Oja. Data compression, feature extraction, and autoassociation in feedforward neuralnetworks. In Proc. Int. Conf. on Artificial Neural Networks (ICANN’91), pages 737–745,
Espoo, Finland, 1991.
326. E. Oja. Principal components, minor components, and linear neural networks. Neural
Networks, 5:927–935, 1992.
327. E. Oja. The nonlinear PCA learning rule in independent component analysis. Neurocomputing, 17(1):25–46, 1997.

REFERENCES

467

328. E. Oja. From neural learning to independent components. Neurocomputing, 22:187–199,
1998.
329. E. Oja. Nonlinear PCA criterion and maximum likelihood in independent component
analysis. In Proc. Int. Workshop on Independent Component Analysis and Signal Separation (ICA’99), pages 143–148, Aussois, France, 1999.
330. E. Oja and J. Karhunen. On stochastic approximation of the eigenvectors and eigenvaluesof the expectation of a random matrix. Journal of Math. Analysis and Applications,
106:69–84, 1985.
331. E. Oja, J. Karhunen, and A. Hyvärinen. From neural principal components to neural
independent components. In Proc. Int. Conf. on Artificial Neural Networks (ICANN’97),
Lausanne, Switzerland, 1997.
332. E. Oja, H. Ogawa, and J. Wangviwattana. Learning in nonlinear constrained Hebbian
networks. In Proc. Int. Conf. on Artificial Neural Networks (ICANN’91), pages 385–390,
Espoo, Finland, 1991.
333. E. Oja, H. Ogawa, and J. Wangviwattana. Principal component analysis by homogeneous
neural networks, part I: the weighted subspace criterion. IEICE Trans. on Informationand Systems, E75-D(3):366–375, 1992.
334. T. Ojanperä and R. Prasad. Wideband CDMA for Third Generation Systems. Artech
House, 1998.
335. B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties bylearning a sparse code for natural images. Nature, 381:607–609, 1996.
336. B. A. Olshausen and D. J. Field. Natural image statistics and efficient coding. Network,
7(2):333–340, 1996.
337. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A
strategy employed by V1? Vision Research, 37:3311–3325, 1997.
338. B. A. Olshausen and K. J. Millman. Learning sparse codes with a mixture-of-gaussians
prior. In Advances in Neural Information Processing Systems, volume 12, pages 841–847.
MIT Press, 2000.
339. A. Oppenheim and R. Schafer. Discrete-Time Signal Processing. Prentice Hall, 1989.
340. S. Ouyang, Z. Bao, and G.-S. Liao. Robust recursive least squares learning algorithm forprincipal component analysis. IEEE Trans. on Neural Networks, 11(1):215–221, 2000.
341. P. Pajunen. Blind separation of binary sources with less sensors than sources. In Proc.
Int. Conf. on Neural Networks, Houston, Texas, 1997.
342. P. Pajunen. Blind source separation using algorithmic information theory. Neurocomputing, 22:35–48, 1998.
343. P. Pajunen. Extensions of Linear Independent Component Analysis: Neural andInformation-theoretic Methods. PhD thesis, Helsinki University of Technology, 1998.
344. P. Pajunen. Blind source separation of natural signals based on approximate complexity
minimization. In Proc. Int. Workshop on Independent Component Analysis and Signal
Separation (ICA’99), pages 267–270, Aussois, France, 1999.
345. P. Pajunen, A. Hyvärinen, and J. Karhunen. Nonlinear blind source separation by
self-organizing maps. In Proc. Int. Conf. on Neural Information Processing, pages
1207–1210, Hong Kong, 1996.

468

REFERENCES

346. P. Pajunen and J. Karhunen. A maximum likelihood approach to nonlinear blindsource separation. In Proceedings of the 1997 Int. Conf. on Artificial Neural Networks(ICANN’97), pages 541–546, Lausanne, Switzerland, 1997.
347. P. Pajunen and J. Karhunen. Least-squares methods for blind source separation based on
nonlinear PCA. Int. J. of Neural Systems, 8(5-6):601–612, 1998.
348. P. Pajunen and J. Karhunen, editors. Proc. of the 2nd Int. Workshop on Independent
Component Analysis and Blind Signal Separation, Helsinki, Finland, June 19-22, 2000.
Otamedia, 2000.
349. F. Palmieri and A. Budillon. Multi-class independent component analysis (mucica). In
M. Girolami, editor, Advances in Independent Component Analysis, pages 145–160.
Springer-Verlag, 2000.
350. F. Palmieri and J. Zhu. Self-association and Hebbian learning in linear neural networks.
IEEE Trans. on Neural Networks, 6(5):1165–1184, 1995.
351. C. Papadias. Blind separation of independent sources based on multiuser kurtosis optimization criteria. In S. Haykin, editor, Unsupervised Adaptive Filtering, volume 2, pages
147–179. Wiley, 2000.
352. H. Papadopoulos. Equalization of multiuser channels. In Wireless Communications:
Signal Processing Perspectives, pages 129–178. Prentice Hall, 1998.
353. A. Papoulis. Probability, Random Variables, and Stochastic Processes. McGraw-Hill,
3rd edition, 1991.
354. N. Parga and J.-P. Nadal. Blind source separation with time-dependent mixtures. Signal
Processing, 80(10):2187–2194, 2000.
355. L. Parra. Symplectic nonlinear component analysis. In Advances in Neural Information
Processing Systems, volume 8, pages 437–443. MIT Press, Cambridge, Massachusetts,
1996.
356. L. Parra. Convolutive BBS for acoustic multipath environments. In S. Roberts and
R. Everson, editors, ICA: Principles and Practice. Cambridge University Press, 2000.
in press.
357. L. Parra, G. Deco, and S. Miesbach. Redundancy reduction with information-preserving
nonlinear maps. Network, 6:61–72, 1995.
358. L. Parra, G. Deco, and S. Miesbach. Statistical independence and novelty detection withinformation-preserving nonlinear maps. Neural Computation, 8:260–269, 1996.
359. L. Parra and C. Spence. Convolutive blind source separation based on multiple decorrelation. In Proc. IEEE Workshop on Neural Networks for Signal Processing (NNSP’97),
Cambridge, UK, 1998.
360. L. Parra, C.D. Spence, P. Sajda, A. Ziehe, and K.-R. Müller. Unmixing hyperspectral
data. In Advances in Neural Information Processing Systems 12, pages 942–948. MIT
Press, 2000.
361. A. Paulraj, C. Papadias, V. Reddy, and A.-J. van der Veen. Blind space-time signal
processing. In Wireless Communications: Signal Processing Perspectives, pages 179–
210. Prentice Hall, 1998.
362. K. Pawelzik, K.-R. Müller, and J. Kohlmorgen. Prediction of mixtures. In Proc. Int.
Conf. on Artificial Neural Networks (ICANN’96), pages 127 – 132. Springer, 1996.

REFERENCES

469

363. B. A. Pearlmutter and L. C. Parra. Maximum likelihood blind source separation: A
context-sensitive generalization of ICA. In Advances in Neural Information Processing
Systems, volume 9, pages 613–619, 1997.
364. K. Pearson. On lines and planes of closest fit to systems of points in space. Philosophical
Magazine, 2:559–572, 1901.
365. H. Peng, Z. Chi, and W. Siu. A semi-parametric hybrid neural model for nonlinear blind
signal separation. Int. J. of Neural Systems, 10(2):79–94, 2000.
366. W. D. Penny, R. M. Everson, and S. J. Roberts. Hidden Markov independent component
analysis. In M. Girolami, editor, Advances in Independent Component Analysis, pages
3–22. Springer-Verlag, 2000.
367. K. Petersen, L. Hansen, T. Kolenda, E. Rostrup, and S. Strother. On the independent
component of functional neuroimages. In Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000), pages 251–256, Helsinki, Finland,
2000.
368. D.-T. Pham. Blind separation of instantaneous mixture sources via an independent
component analysis. IEEE Trans. on Signal Processing, 44(11):2768–2779, 1996.
369. D.-T. Pham. Blind separation of instantaneous mixture of sources based on order statistics. IEEE Trans. on Signal Processing, 48(2):363–375, 2000.
370. D.-T. Pham and J.-F. Cardoso. Blind separation of instantaneous mixtures of nonstationary sources. In Proc. Int. Workshop on Independent Component Analysis andBlind Signal Separation (ICA2000), pages 187–193, Helsinki, Finland, 2000.
371. D.-T. Pham and P. Garrat. Blind separation of mixture of independent sources througha quasi-maximum likelihood approach. IEEE Trans. on Signal Processing, 45(7):1712–
1725, 1997.
372. D.-T. Pham, P. Garrat, and C. Jutten. Separation of a mixture of independent sources
through a maximum likelihood approach. In Proc. EUSIPCO, pages 771–774, 1992.
373. D. Pollen and S. Ronner. Visual cortical neurons as localized spatial frequency filters.
IEEE Trans. on Systems, Man, and Cybernetics, 13:907–916, 1983.
374. J. Porrill, J. W. Stone, J. Berwick, J. Mayhew, and P. Coffey. Analysis of optical imaging
data using weak models and ICA. In M. Girolami, editor, Advances in Independent
Component Analysis, pages 217–233. Springer-Verlag, 2000.
375. K. Prank, J. Börger, A. von zur Mühlen, G. Brabant, and C. Schöfl. Independent component analysis of intracellular calcium spike data. In Advances in Neural Information
Processing Systems, volume 11, pages 931–937. MIT Press, 1999.
376. J. Principe, N. Euliano, and C. Lefebvre. Neural and Adaptive Systems - Fundamentals
Through Simulations. Wiley, 2000.
377. J. Principe, D. Xu, and J. W. Fisher III. Information-theoretic learning. In S. Haykin,editor, Unsupervised Adaptive Filtering, Vol. I, pages 265–319. Wiley, 2000.
378. J. Proakis. Digital Communications. McGraw-Hill, 3rd edition, 1995.
379. C. G. Puntonet, A. Prieto, C. Jutten, M. Rodriguez-Alvarez, and J. Ortega. Separationof sources: A geometry-based procedure for reconstruction of n-valued signals. Signal
Processing, 46:267–284, 1995.
380. J. Rissanen. Modeling by shortest data description. Automatica, 14:465–471, 1978.

470

REFERENCES

381. J. Rissanen. A universal prior for integers and estimation by minimum description length.
Annals of Statistics, 11(2):416–431, 1983.
382. T. Ristaniemi. Synchronization and Blind Signal Processing in CDMA Systems. PhD
thesis, University of Jyväskylä, Jyväskylä, Finland, 2000.
383. T. Ristaniemi and J. Joutsensalo. Advanced ICA-based receivers for DS-CDMA systems. In Proc. IEEE Int. Conf. on Personal, Indoor, and Mobile Radio Communications(PIMRC’00), London, UK, 2000.
384. T. Ristaniemi and J. Joutsensalo. On the performance of blind source separation in
CDMA downlink. In Proc. Int. Workshop on Independent Component Analysis andBlind Source Separation (ICA’99), pages 437–441, Aussois, France, January 1999.
385. S. Roberts. Independent component analysis: Source assessment & separation, aBayesian approach. IEE Proceedings - Vision, Image & Signal Processing, 145:149–154,
1998.
386. M. Rosenblatt. Stationary Sequences and Random Fields. Birkhauser, 1985.
387. S. Roweis. EM algorithms for PCA and SPCA. In M. I. Jordan, M. J. Kearns, and S. A.
Solla, editors, Advances in Neural Information Processing Systems, volume 10, pages
626 – 632. MIT Press, 1998.
388. J. Rubner and P. Tavan. A self-organizing network for principal component analysis.
Europhysics Letters, 10(7):693 – 698, 1989.
389. H. Sahlin and H. Broman.
64(1):103–113, 1998.

Separation of real-world signals.

Signal Processing,390. H. Sahlin and H. Broman. MIMO signal separation for FIR channels: A criterion andperformance analysis. IEEE Trans. on Signal Processing, 48(3):642–649, 2000.
391. T.D. Sanger. Optimal unsupervised learning in a single-layered linear feedforward
network. Neural Networks, 2:459–473, 1989.
392. Y. Sato. A method for self-recovering equalization for multilevel amplitude-modulation
system. IEEE Trans. on Communications, 23:679–682, 1975.
393. L. Scharf. Statistical Signal Processing: Detection, Estimation, and Time Series Analysis.
Addison-Wesley, 1991.
394. M. Scherg and D. von Cramon. Two bilateral sources of the late AEP as identified by aspatio-temporal dipole model. Electroencephalography and Clinical Neurophysiology,
62:32 – 44, 1985.
395. M. Schervish. Theory of Statistics. Springer, 1995.
396. I. Schiessl, M. Stetter, J.W.W. Mayhew, N. McLoughlin, J.S.Lund, and K. Obermayer.
Blind signal separation from optical imaging recordings with extended spatial decorrelation. IEEE Trans. on Biomedical Engineering, 47(5):573–577, 2000.
397. C. Serviere and V. Capdevielle. Blind adaptive separation of wide-band sources. In
Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP’96), Atlanta,
Georgia, 1996.
398. O. Shalvi and E. Weinstein. New criteria for blind deconvolution of nonminimum phasesystems (channels). IEEE Trans. on Information Theory, 36(2):312–321, 1990.
399. O. Shalvi and E. Weinstein. Super-exponential methods for blind deconvolution. IEEE
Trans. on Information Theory, 39(2):504:519, 1993.

REFERENCES

471

400. S. Shamsunder and G. B. Giannakis. Multichannel blind signal separation and reconstruction. IEEE Trans. on Speech and Aurdio Processing, 5(6):515–528, 1997.
401. C. Simon, P. Loubaton, C. Vignat, C. Jutten, and G. d’Urso. Separation of a class of
convolutive mixtures: A contrast function approach. In Proc. Int. Conf. on Acoustics,
Speech, and Signal Processing (ICASSP’99), Phoenix, AZ, 1999.
402. C. Simon, C. Vignat, P. Loubaton, C. Jutten, and G. d’Urso. On the convolutive mixture
source separation by the decorrelation approach. In Proc. Int. Conf. on Acoustics, Speech,
and Signal Processing (ICASSP’98), pages 2109–2112, Seattle, WA, 1998.
403. E. P. Simoncelli and E. H. Adelson. Noise removal via bayesian wavelet coring. In Proc.
Third IEEE Int. Conf. on Image Processing, pages 379–382, Lausanne, Switzerland,
1996.
404. E. P. Simoncelli and O. Schwartz. Modeling surround suppression in V1 neurons with astatistically-derived normalization model. In Advances in Neural Information Processing
Systems 11, pages 153–159. MIT Press, 1999.
405. P. Smaragdis. Blind separation of convolved mixtures in the frequency domain. Neurocomputing, 22:21–34, 1998.
406. V. Soon, L. Tong, Y. Huang, and R. Liu. A wideband blind identification approach tospeech acquisition using a microphone array. In Proc. Int. Conf. ASSP-92, volume 1,
pages 293–296, San Francisco, California, March 23–26 1992.
407. H. Sorenson. Parameter Estimation - Principles and Problems. Marcel Dekker, 1980.
408. E. Sorouchyari. Blind separation of sources, Part III: Stability analysis. Signal Processing, 24:21–29, 1991.
409. C. Spearman. General intelligence, objectively determined and measured. American J.
of Psychology, 15:201–293, 1904.
410. R. Steele. Mobile Radio Communications. Pentech Press, London, 1992.
411. P. Stoica and R. Moses. Introduction to Spectral Analysis. Prentice Hall, 1997.
412. J. V. Stone, J. Porrill, C. Buchel, and K. Friston. Spatial, temporal, and spatiotemporal
independent component analysis of fMRI data. In R.G. Aykroyd K.V. Mardia and
I.L. Dryden, editors, Proceedings of the 18th Leeds Statistical Research Workshop on
Spatial-Temporal Modelling and its Applications, pages 23–28. Leeds University Press,
1999.
413. E. Ström, S. Parkvall, S. Miller, and B. Ottersten. Propagation delay estimation in
asynchronous direct-sequence code division multiple access systems. IEEE Trans. Communications, 44:84–93, January 1996.
414. J. Sun. Some practical aspects of exploratory projection pursuit. SIAM J. of Sci. Comput.,
14:68–80, 1993.
415. K. Suzuki, T. Kiryu, and T. Nakada. An efficient method for independent componentcross correlation-sequential epoch analysis of functional magnetic resonance imaging.
In Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation(ICA2000), pages 309–315, Espoo, Finland, 2000.
416. A. Swami, G. B. Giannakis, and S. Shamsunder. Multichannel ARMA processes. IEEE
Trans. on Signal Processing, 42:898–914, 1994.

472

REFERENCES

417. A. Taleb and C. Jutten. Batch algorithm for source separation in post-nonlinear mixtures.
In Proc. First Int. Workshop on Independent Component Analysis and Signal Separation(ICA’99), pages 155–160, Aussois, France, 1999.
418. A. Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Trans. on
Signal Processing, 47(10):2807–2820, 1999.
419. C. Therrien. Discrete Random Signals and Statistical Signal Processing. Prentice Hall,
1992.
420. H.-L. Nguyen Thi and C. Jutten. Blind source separation for convolutive mixtures. Signal
Processing, 45:209–229, 1995.
421. M. E. Tipping and C. M. Bishop. Mixtures of probabilistic principal component analyzers.
Neural Computation, 11:443–482, 1999.
422. L. Tong, Y. Inouye, and R. Liu. A finite-step global convergence algorithm for theparameter estimation of multichannel MA processes. IEEE Trans. on Signal Processing,
40:2547–2558, 1992.
423. L. Tong, Y. Inouye, and R. Liu. Waveform preserving blind estimation of multiple
independent sources. IEEE Trans. on Signal Processing, 41:2461–2470, 1993.
424. L. Tong, R.-W. Liu, V.C. Soon, and Y.-F. Huang. Indeterminacy and identifiability ofblind identification. IEEE Trans. on Circuits and Systems, 38:499–509, 1991.
425. L. Tong and S. Perreau. Multichannel blind identification: From subspace to maximum
likelihood methods. Proceedings of the IEEE, 86(10):1951–1968, 1998.
426. K. Torkkola. Blind separation of convolved sources based on information maximization.
In Proc. IEEE Workshop on Neural Networks and Signal Processing (NNSP’96), pages
423–432, Kyoto, Japan, 1996.
427. K. Torkkola. Blind separation of delayed sources based on information maximization. In
Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP’96), pages
3509–3512, Atlanta, Georgia, 1996.
428. K. Torkkola. Blind separation of radio signals in fading channels. In Advances in Neural
Information Processing Systems, volume 10, pages 756–762. MIT Press, 1998.
429. K. Torkkola. Blind separation for audio signals – are we there yet? In Proc. Int. Workshop
on Independent Component Analysis and Signal Separation (ICA’99), pages 239–244,
Aussois, France, 1999.
430. K. Torkkola. Blind separation of delayed and convolved sources. In S. Haykin, editor,
Unsupervised Adaptive Filtering, Vol. I, pages 321–375. Wiley, 2000.
431. M. Torlak, L. Hansen, and G. Xu. A geometric approach to blind source separation for
digital wireless applications. Signal Processing, 73:153–167, 1999.
432. J. K. Tugnait. Identification and deconvolution of multichannel nongaussian processes
using higher-order statistics and inverse filter criteria. IEEE Trans. on Signal Processing,
45:658–672, 1997.
433. J. K. Tugnait. Adaptive blind separation of convolutive mixtures of independent linear
signals. Signal Processing, 73:139–152, 1999.
434. J. K. Tugnait. On blind separation of convolutive mixtures of independent linear signals
in unknown additive noise. IEEE Trans. on Signal Processing, 46(11):3117–3123,
November 1998.

REFERENCES

473

435. M. Valkama, M. Renfors, and V. Koivunen. BSS based I/Q imbalance compensationin communication receivers in the presence of symbol timing errors. In Proc. Int.
Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000),
pages 393–398, Espoo, Finland, 2000.
436. H. Valpola. Nonlinear independent component analysis using ensemble learning: Theory.
In Proc. Int. Workshop on Independent Component Analysis and Blind Signal Separation(ICA2000), pages 251–256, Helsinki, Finland, 2000.
437. H. Valpola. Unsupervised learning of nonlinear dynamic state-space models. Technical Report A59, Lab of Computer and Information Science, Helsinki University ofTechnology, Finland, 2000.
438. H. Valpola, X. Giannakopoulos, A. Honkela, and J. Karhunen. Nonlinear independent
component analysis using ensemble learning: Experiments and discussion. In Proc. Int.
Workshop on Independent Component Analysis and Blind Signal Separation (ICA2000),
pages 351–356, Helsinki, Finland, 2000.
439. A.-J. van der Veen. Algebraic methods for deterministic blind beamforming. Proceedingsof the IEEE, 86(10):1987–2008, 1998.
440. A.-J. van der Veen. Blind separation of BPSK sources with residual carriers. Signal
Processing, 73:67–79, 1999.
441. A.-J. van der Veen. Algebraic constant modulus algorithms. In G. Giannakis, Y. Hua,P. Stoica, and L. Tong, editors, Signal Processing Advances in Wireless and Mobile
Communications, Vol. 2: Trends in Single-User and Multi-User Systems, pages 89–130.
Prentice Hall, 2001.
442. J. H. van Hateren and D. L. Ruderman. Independent component analysis of natural image
sequences yields spatiotemporal filters similar to simple cells in primary visual cortex.
Proc. Royal Society, Ser. B, 265:2315–2320, 1998.
443. J. H. van Hateren and A. van der Schaaf. Independent component filters of natural
images compared with simple cells in primary visual cortex. Proc. Royal Society, Ser. B,
265:359–366, 1998.
444. S. Verdu. Multiuser Detection. Cambridge University Press, 1998.
445. R. Vigário. Extraction of ocular artifacts from EEG using independent component
analysis. Electroenceph. Clin. Neurophysiol., 103(3):395–404, 1997.
446. R. Vigário, V. Jousmäki, M. Hämäläinen, R. Hari, and E. Oja. Independent component analysis for identification of artifacts in magnetoencephalographic recordings. In
Advances in Neural Information Processing Systems, volume 10, pages 229–235. MIT
Press, 1998.
447. R. Vigário, J. Särelä, V. Jousmäki, M. Hämäläinen, and E. Oja. Independent component approach to the analysis of EEG and MEG recordings. IEEE Trans. Biomedical
Engineering, 47(5):589–593, 2000.
448. R. Vigário, J. Särelä, V. Jousmäki, and E. Oja. Independent component analysis indecomposition of auditory and somatosensory evoked fields. In Proc. Int. Workshop
on Independent Component Analysis and Signal Separation (ICA’99), pages 167–172,
Aussois, France, 1999.
449. R. Vigário, J. Särelä, and E. Oja. Independent component analysis in wave decompositionof auditory evoked fields. In Proc. Int. Conf. on Artificial Neural Networks (ICANN’98),
pages 287–292, Skövde, Sweden, 1998.

474

REFERENCES

450. L.-Y. Wang and J. Karhunen. A unified neural bigradient algorithm for robust PCA andMCA. Int. J. of Neural Systems, 7(1):53–67, 1996.
451. X. Wang and H. Poor. Blind equalization and multiuser detection in dispersive CDMAchannels. IEEE Trans. on Communications, 46(1):91–103, 1998.
452. X. Wang and H. Poor. Blind multiuser detection: A subspace approach. IEEE Trans. on
Information Theory, 44(2):667–690, 1998.
453. M. Wax and T. Kailath. Detection of signals by information-theoretic criteria. IEEE
Trans. on Acoustics, Speech and Signal Processing, 33:387–392, 1985.
454. A. Webb. Statistical Pattern Recognition. Arnold, 1999.
455. A. S. Weigend and N.A. Gershenfeld. Time series prediction. In Proc. of NATO Advanced
Research Workshop on Comparative Time Series Analysis, Santa Fe, New Mexico, 1992.
456. E. Weinstein, M. Feder, and A. V. Oppenheim. Multi-channel signal separation bydecorrelation. IEEE Trans. on Signal Processing, 1:405–413, 1993.
457. R. A. Wiggins. Minimum entropy deconvolution. Geoexploration, 16:12–35, 1978.
458. R. Williams. Feature discovery through error-correcting learning. Technical report,
University of California at San Diego, Institute of Cognitive Science, 1985.
459. J. O. Wisbeck, A. K. Barros, and R. G. Ojeda. Application of ICA in the separation ofbreathing artifacts in ECG signals. In Proc. Int. Conf. on Neural Information Processing(ICONIP’98), pages 211–214, Kitakyushu, Japan, 1998.
460. G. Wubbeler, A. Ziehe, B. Mackert, K. Müller, L. Trahms, and G. Curio. Independent
component analysis of noninvasively recorded cortical magnetic dc-field in humans.
IEEE Trans. Biomedical Engineering, 47(5):594–599, 2000.
461. L. Xu. Least mean square error reconstruction principle for self-organizing neural nets.
Neural Networks, 6:627–648, 1993.
462. L. Xu. Bayesian Kullback Ying-Yang dependence reduction theory. Neurocomputing,
22:81–111, 1998.
463. L. Xu. Temporal BYY learning for state space approach, hidden markov model, andblind source separation. IEEE Trans. on Signal Processing, 48(7):2132–2144, 2000.
464. L. Xu, C. Cheung, and S.-I. Amari. Learned parameter mixture based ICA algorithm.
Neurocomputing, 22:69–80, 1998.
465. W.-Y. Yan, U. Helmke, and J. B. Moore. Global analysis of Oja’s flow for neuralnetworks. IEEE Trans. on Neural Networks, 5(5):674 – 683, 1994.
466. B. Yang. Projection approximation subspace tracking. IEEE Trans. on Signal Processing,43(1):95–107, 1995.
467. B. Yang. Asymptotic convergence analysis of the projection approximation subspacetracking algorithm. Signal Processing, 50:123–136, 1996.
468. H. H. Yang and S.-I. Amari. Adaptive on-line learning algorithms for blind separation:
Maximum entropy and minimum mutual information. Neural Computation, 9(7):1457–
1482, 1997.
469. H. H. Yang, S.-I. Amari, and A. Cichocki. Information-theoretic approach to blindseparation of sources in non-linear mixture. Signal Processing, 64(3):291–300, 1998.
470. D. Yellin and E. Weinstein. Criteria for multichannel signal separation. IEEE Trans. on
Signal Processing, 42:2158–2167, 1994.

REFERENCES

475

471. D. Yellin and E. Weinstein. Multichannel signal separation: Methods and analysis. IEEE
Trans. on Signal Processing, 44:106–118, 1996.
472. A. Yeredor. Blind separation of gaussian sources via second-order statistics with asymptotically optimal weighting. IEEE Signal Processing Letters, 7(7):197–200, 2000.
473. A. Yeredor. Blind source separation via the second characteristic function. Signal
Processing, 80:897–902, 2000.
474. K. Yeung and S. Yau. A cumulant-based super-exponential algorithm for blind deconvolution of multi-input multi-output systems. Signal Processing, 67(2):141–162, 1998.
475. A. Ypma and P. Pajunen. Rotating machine vibration analysis with second-order independent component analysis. In Proc. Int. Workshop on Independent Component
Analysis and Signal Separation (ICA’99), pages 37–42, Aussois, France, 1999.
476. T. Yu, A. Stoschek, and D. Donoho. Translation- and direction- invariant denoising of
2-D and 3-D images: Experience and algorithms. In Proceedings of the SPIE, Wavelet
Applications in Signal and Image Processing IV, pages 608–619, 1996.
477. S. Zacks. Parametric Statistical Inference. Pergamon, 1981.
478. C. Zetzsche and G. Krieger. Nonlinear neurons and high-order statistics: New approaches
to human vision and electronic image processing. In B. Rogowitz and T.V. Pappas,
editors, Human Vision and Electronic Imaging IV (Proc. SPIE vol. 3644), pages 2–33.
SPIE, 1999.
479. L. Zhang and A. Cichocki. Blind separation of filtered sources using state-space approach.
In Advances in Neural Information Processing Systems, volume 11, pages 648–654. MIT
Press, 1999.
480. Q. Zhang and Y.-W. Leung. A class of learning algorithms for principal component
analysis and minor component analysis. IEEE Trans. on Neural Networks, 11(1):200–
204, 2000.
481. A. Ziehe and K.-R. Müller. TDSEP—an efficient algorithm for blind separation using
time structure. In Proc. Int. Conf. on Artificial Neural Networks (ICANN’98), pages
675–680, Skövde, Sweden, 1998.
482. A. Ziehe, K.-R. Müller, G. Nolte, B.-M. Mackert, and G. Curio. Artifact reduction
in magnetoneurography based on time-delayed second-order correlations. IEEE Trans.
Biomedical Engineering, 47(1):75–87, 2000.
483. A. Ziehe, G. Nolte, G. Curio, and K.-R. Müller. OFI: Optimal filtering algorithms forsource separation. In Proc. Int. Workshop on Independent Component Analysis and Blind
Signal Separation (ICA2000), pages 127–132, Helsinki, Finland, 2000.

Index

Akaike’s information criterion, 131
Algorithm
AMUSE, 343
Bell-Sejnowski, 207
Cichocki-Unbehauen, 244
EASI, 247
eigenvalue decompositionof cumulant tensor, 230of weighted correlation, 235
fixed-point (FastICA)for complex-valued data, 386
for maximum likelihood estimation, 209
for tensor decomposition, 232
using kurtosis, 178
using negentropy, 188
FOBI, 235
gradient
for maximum likelihood estimation, 207
using kurtosis, 175
using negentropy, 185
Herault-Jutten, 242
JADE, 234
natural gradientfor maximum likelihood estimation, 208, 430
nonlinear RLS, 259
SOBI, 344
TDSEP, 344
Algorithms
experimental comparison, 280
choice of, 271

476

connections between, 274
effect of noise, 286
performance index, 281
vs. objective functions, 273
AMUSE, 343
APEX, 135
Applications
audio separation, 446
brain imaging, 407
brain modeling, 403
communications, 358, 417
econometric, 441
financial, 441
image denoising, 398
image feature extraction, 311, 391
industrial process monitoring, 335
miscellaneous, 448
vision research, 403
visualization, 197
ARMA process, 51
Artifacts
in EEG and MEG, 410
Asymptotic variance, 276
Autoassociative learning, 136, 249
Autocorrelations, 45, 47
as an alternative to nongaussianity, 342
ICA estimation using, 342
in telecommunications, 424
Autoregressive (AR) process, 50, 445
Back-propagation learning, 136

INDEX

Basis vectors
and factor rotation, 268
Gabor, 394
ICA, 398
in overcomplete ICA, 305
of independent subspace, 380
of PCA subspace, 128
relation to filters in ICA, 396
wavelet, 396
Batch learning, 69
Bayes’ rule, 31
Bias, 80
Blind deconvolution, 355–356
multichannel, 355, 361
Bussgang methods, 357
CMA algorithm, 358
cumulant-based methods, 358
Godard algorithm, 357
Shalvi-Weinstein algorithm, 359
using linear ICA, 360
Blind equalization, see blind deconvolution
Blind source separation, 147
Brain imaging, 407
Bussgang criterion, 253
CDMA (Code Division Multiple Access), 417
CDMA signal model, 422
Centering, 154
Central limit theorem, 34, 166
Central moment, 37, 84
Characteristic function, 41
Chip sequence, 418
Cichocki-Unbehauen algorithm, 244
Cocktail-party problem, 147, 361, 446
Code lengthand entropy, 107
and Kolmogoroff complexity, 352
and mutual information, 110
Complex-valued data, 383
Complexity minimization, 353, 424
Compression
by PCA, 126
Conjugate gradients, 67
Consistency, 80
of ICA methods, 187, 205
Convergenceof on-line algorithms, 71
speed, 65
Convolution, 369
Convolutive mixtures, 355, 361
application in CDMA, 430
Bussgang type methods, 367
Fourier transform methods, 365
natural gradient methods, 364
using autocovariances, 367
using higher-order statistics, 367
using spatiotemporal decorrelation, 367

477

Correlation matrix, 21–22, 26, 48
Correlation, 21
and independence, 240
nonlinear, 240
Covariance matrix, 22of estimation error, 82, 95
Covariance, 22
Cramer-Rao lower bound, 82, 92
Cross-correlation function, 46
Cross-correlation matrix, 22
Cross-covariance function, 46
Cross-covariance matrix, 23
Cross-cumulants, 42
Cumulant generating function, 41
Cumulant tensor, 229
Cumulants, 41–42
Cumulative distribution function, 15, 17, 27, 36
joint, 19Curve fitting, 87
Cyclostationarity, 368
Decorrelation, 132, 140
nonlinear, 239–240, 244
Denoising of images, 398
Density, see probability density
Density expansions
Edgeworth, 113
Gram-Charlier, 113
polynomial, 113
Discrete-valued components, 261, 299, 311
Distribution, see probability density
EASI algorithm, 247
Edgeworth expansion, 113
EEG, 407
Electrocardiography, 413
Electroencephalography, 407
EM algorithm, 93
Ensemble learning, 328
Entropy, 222
approximation, 113, 115
by cumulants, 113
by nonpolynomial functions, 115
definition, 105differential, 108
maximality of gaussian distribution, 112
maximum, 111of transformation, 109
Equivariance, 248
Ergodicity, 49
Error criterion, 81
Estimate, 78
Estimating function, 245
Estimation, 77
adaptive, 79
asymptotically unbiased, 80
batch, 79
Bayesian, 79, 94

478

INDEX

consistent, 80
efficient, 82
error, 80
linear minimum MSE error, 95
maximum a posteriori (MAP), 97
maximum likelihood, 90
minimum mean-square error, 94, 428, 433
moment, 84
of expectation, 24
off-line, 79
on-line, 79
recursive, 79
robust, 83
unbiased, 80
Estimator, see estimation (for general entry);algorithm (for ICA entry)
Evoked fields, 411
Expectation, 19
conditional, 31
properties, 20
Expectation-maximization (EM) algorithm, 322
Factor analysis, 138
and ICA, 139, 268
nonlinear independent, 332
nonlinear, 332
principal, 138
Factor rotation, 139–140, 268
FastICAfor complex-valued data, 437
for maximum likelihood estimation, 209
for tensor decomposition, 232
using kurtosis, 178
using negentropy, 188
Feature extraction
by ICA, 150, 398
by independent subspace analysis, 401
by topographic ICA, 401
using overcomplete bases, 311
Feedback architecture, 431
Filteringhigh-pass, 265
linear, 96
low-pass, 265
optimal, 266
taking innovation processes, 266
Wiener, 96
Financial time series, 441
FIR filter, 369
Fisher information matrix, 83
Fixed-point algorithm, see FastICA
FMRI, 407, 413
FOBI, 235
Fourier transform, 370
Fourth-order blind identification, 235
Gabor analysis, 392
and ICA, 398

Gauss-Newton method, 67
Gaussian density, 16
forbidden in ICA, 161
multivariate, 31
properties, 32
Generalized Hebbian algorithm (GHA), 134
Generative topographic mapping (GTM), 322
Gradient descentdeterministic, 63
stochastic, 68
Gradient, 57
natural, 67, 208, 244, 247
of function, 57
relative, 67, 247
Gram-Charlier expansion, 113
Gram-Schmidt orthogonalization, 141
Herault-Jutten algorithm, 242
Hessian matrix, 58
Higher-order statistics, 36
ICA
ambiguities in, 154complex-valued case, 384
and factor rotation, 140, 268
and feature extraction, 398
definition, 151
identifiability, 152, 154complex-valued case, 384
multidimensional, 379
noisy, 293
overview of estimation principles, 287
restrictions in, 152
spatiotemporal, 377
topographic, 382
applications on images, 401
with complex-valued data, 383, 435
with convolutive mixtures, 355, 361, 430with overcomplete bases, 305–306
with subspaces, 380
IIR filter, 369
Independence, 27, 30, 33
Independent component analysis, see ICA
Independent subspace analysis, 380
and complex-valued data, 387
applications on images, 401
Infomax, 211, 430
Innovation process, 266
Intersymbol interference (ISI), 420
Jacobian matrix, 58
JADE, 234
Jeffreys’ prior, 373
Joint approximate diagonalization, 234
Karhunen-Loève transform, 143
Kolmogoroff complexity, 351, 424–425
Kullback-Leibler divergence, 110
Kurtosis, 38
as nongaussianity measure, 171

INDEX

nonrobustness, 182, 184
relation with nonlinear PCA, 252
Lagrange method, 73
Laplacian density, 39, 171
Learning
algorithms, 63batch, 69
on-line, 69
rate, 63
Least mean-square error, 249
Least-squares method, 86
generalized, 88
linear, 86
nonlinear, 89, 93
normal equations, 87
Likelihood, 90
and mutual information, 224
and nonlinear PCA, 253
and posterior density, 97
of ICA model, 203
See also maximum likelihood
Loss function, 81
Magnetic resonance imaging, 407, 413
Magnetoencephalography, 407
Magnetoneurography, 413
MAP, see maximum a posteriori
Marquardt-Levenberg algorithm, 67
Matched filter, 424, 432
Matrixdeterminant, 61
gradient of function, 59
Jacobian, 36
trace, 62
Maximization of function, 57
Maximum a posteriori, 97, 299, 303, 306, 326
Maximum entropy, 111
Maximum likelihood, 90, 203, 322
consistency of, 205
in CDMA, 424
See also likelihood
Mean function, 45
Mean vector, 21
Mean-square error, 81, 94
minimization for PCA, 128
MEG, 407
Minimization of function, 57
Minimum description length, 131
Minimum-phase filter, 370
Minor components, 135
Mixture of gaussians, 322, 329
ML, see maximum likelihood
MMSE estimator, 424
MMSE-ICA detector, 434, 437–438
Model orderchoosing, 131, 271
Modified GTM method, 323

Moment generating function, 41
Moment method, 84
Moments, 20, 37, 41–42
central, 22
nonpolynomial, 207
Momentum term, 426
Moving average (MA) process, 51
Multilayer perceptron, 136, 328
Multipath propagation, 420
Multiple access communications, 417
Multiple access interference (MAI), 421
Multiuser detection, 421
Mutual information, 221–222, 319
and Kullback-Leibler divergence, 110
and likelihood, 224
and nongaussianity, 223
approximation of, 223–224
definition, 110
minimization of, 221
Near–far problem, 421, 424
Negentropy, 222
approximation, 113, 115, 183
by cumulants, 113
by nonpolynomial functions, 115
as measure of nongaussianity, 182
as nongaussianity measure, 182
definition, 112
optimality, 277
Neural networks, 36
Neurons, 408
Newton’s method, 66
Noise, 446
as independent components, 295
in the ICA model, 293
reduction by low-pass filtering, 265
reduction by nonlinear filtering, 300
reduction by PCA, 268
reduction by shrinkage, 300
application on images, 398
sensor vs. source, 294
Noisy ICA
application
image processing, 398
telecommunications, 423
estimation of ICs, 299
by MAP, 299
by maximum likelihood, 299
by shrinkage, 300
estimation of mixing matrix, 295
bias removal techniques, 296
by cumulant methods, 298
by FastICA, 298
by maximum likelihood, 299
Nongaussianity, 165
and projection pursuit, 197
is interesting, 197

479

480

INDEX

measured by kurtosis, 171, 182
measured by negentropy, 182
optimal measure is negentropy, 277
Nonlinear BSS, 315
definition, 316
Nonlinear ICA, 315
definition, 316
existence and uniqueness, 317
post-nonlinear mixtures, 319
using ensemble learning, 328
using modified GTM method, 323
using self-organizing map (SOM), 320
Nonlinear mixing model, 315
Nonlinearity in algorithm
choice of, 276, 280
Nonstationarity
and tracking, 72, 133, 135, 178
definition, 46
measuring by autocorrelations, 347
measuring by cross-cumulants, 349
separation by, 346
Oja’s rule, 133
On-line learning, 69
Optical imaging, 413
Optimization methods, 57
constrained, 73
unconstrained, 63
Order statistics, 226
Orthogonalization, 141
Gram-Schmidt, 141
symmetric, 142
Overcomplete bases
and image feature extraction, 311
estimation of ICs, 306
by maximum likelihood, 306
estimation of mixing matrix, 307
by FastICA, 309
by maximum likelihood, 307
Overlearning, 268
and PCA, 269
and priors on mixing, 371
Parameter vector, 78
PAST, 136
Performance index, 81
PET, 407
Positive semidefinite, 21
Post-nonlinear mixtures, 316
Posterior, 94
Power method
higher-order, 232
Power spectrum, 49
Prediction of time series, 443
Preprocessing, 263
by PCA, 267
centering, 154
filtering, 264

whitening, 158
Principal component analysis, 125, 332
and complexity, 425
and ICA, 139, 249, 251
and whitening, 140
by on-line learning, 132closed-form computation, 132
nonlinear, 249
number of components, 129
with nonquadratic criteria, 137
Principal curves, 249
Prior, 94
conjugate, 375
for mixing matrix, 371
Jeffreys’, 373
quadratic, 373
sparse, 374
for mixing matrix, 375
Probability density, 16
a posteriori, 94a priori, 94
conditional, 28
double exponential, 39, 171
gaussian, 16, 42
generalized gaussian, 40
joint, 19, 22, 27, 30, 45
Laplacian, 39, 171
marginal, 19, 27, 29, 33
multivariate, 17of a transformation, 35
posterior, 31, 328
prior, 31
uniform, 36, 39, 171
Projection matrix, 427
Projection method, 73
Projection pursuit, 197, 286
Pseudoinverse, 87
Quasiorthogonality, 310
in FastICA, 310
RAKE detector, 424, 434, 437–438
RAKE-ICA detector, 434, 438
Random variable, 15
Random vector, 17
Recursive least-squares
for nonlinear PCA, 259
for PCA, 135
Robustness, 83, 182, 277
Sample mean, 24
Sample moment, 84
Self-organizing map (SOM), 320
Semiblind methods, 387, 424, 432
Semiparametric, 204
Skewness, 38
Smoothing, 445
SOBI, 344
Sparse code shrinkage, 303, 398

INDEX

Sparse coding, 396
Sparsity
measurement of, 374
Spatiotemporal ICA, 377
Spatiotemporal statistics, 362
Sphered random vector, 140
Spreading code, 418
Stability, see consistency
Stationarity
wide-sense, 46
strict sense, 45
Stochastic approximation, 71
Stochastic gradient ascent (SGA), 133
Stochastic processes, 43
Subgaussian, 38
Subspace MMSE detector, 434, 436, 438
Subspacelearning algorithm for PCA, 134
noise, 131
nonlinear learning rule, 254
signal, 131
Subspaces
independent, 380
invariant-feature, 380
Superefficiency, 261
Supergaussian, 39
Taylor series, 62
TDSEP, 344

Tensor methods for ICA, 229
Time averages, 48
Time structure, 43
ICA estimation using, 341
Toeplitz matrix, 48
Tracking in a nonstationary environment, 72
Transfer function, 370
Unbiasedness, 80
Uncorrelatedness, 24, 27, 33
constraint of, 192
Uniform density, 36, 39
rotated, 250
Variance, 22
maximization, 127
Vectorgradient of function, 57
valued function, 58
Visual cortex, 403
Wavelets, 394
and ICA, 398
as preprocessing, 267
White noise, 50
Whiteness, 25Whitening, 140
as preprocessing in ICA, 158
by PCA expansion, 140
Wiener filtering, 96
nonlinear, 300
Z-transform, 369

481