Tutorial n

Chemometrics and Intelligent

Laboratory

Systems, 2 (1987)

37-52

Elsevier Science Publishers B.V., Amsterdam - Printed in The Netherlands

Principal Component Analysis
SVANTE WOLD *Research Group for Chemometrics, Institute of Chemistry, Umei University, S 901 87 Urned (Sweden)
KIM ESBENSEN and PAUL GELADINorwegian Computing Center, P.B. 335 Blindern, N 0314 Oslo 3 (Norway) and Research Group for
Chemometrics, Institute of Chemistry, Umed University, S 901 87 Umeci (Sweden)

CONTENTS...............
Intr~uction:
history of principal component analysis ......
Problem definition for multivariate data ................
...............
A chemical example ..............................
...............
Geometric interpretation of principal component analysis ...
...............
...............
Majesticaldefi~tion of principal component analysis ....
...............
Statistics; how to use the residuals ....................
...............
Plots .........................................
...............
Applications of principal component analysis ............
...............
8.1 Overview (plots) of any data table .................
...............
8.2 Dimensionality reduction .......................
8.3 Similarity models .............................
...............
Data pre-treatment..............................
...............
10 Rank, or dim~sion~ty,of a principal components model. ..........................
11 Extensions; two-block regression and many-way tables .............................
12 Summary .............................................................
References..
.............................................................
Appendix ................................................................

1 INTRODUCTION: HISTORY OF PRINCIPAL COMPONENT ANALYSIS
Principal component analysis (PCA) in many
ways forms the basis for multiv~ate data analysis. PCA provides an approximation of a datatable, a data matrix, X, in terms of the product of
two small matrices T and P’. These matrices, T
and P’, capture the essential data patterns of X.
Plotting the columns of T gives a picture of thedominant “object patterns” of X and, analo0169-7439/87/$03.50

0 1987 Elsevier Science Publishers B.V.

.......
.......
.......
.......
.......
.......
.......
.......
. ..*...
.......
...,...
*....s.

37
38
40
41
41
43
44
46
46
46
47
47
48
49
50
50
51

gously, plotting the rows of P’ shows the complementary “ variable patterns”.
Consider, as an example, a data matrix containing absorbances at K = 100 frequencies measured
on N = 10 mixtures of two chemical constituent.
This matrix is well approximated by a (10 X 2)
matrix T times a (2 x 100) matrix P’, where T
describes the concentrations of the constituentsand P describes their spectra.
PCA was first formulated in statistics by Pearson [l], who formulated the analysis as finding37

nChemometrics

and Intelligent Laboratory

Systems“lines and planes of closest fit to systems ofpoints in space”. This geometricinterpretation
will be further discussed in Section 4. PCA wasbriefly mentioned by Fisher and MacKenzie [2] asmore suitable than analysis of variance for themodelling of response data. Fisher and MacKenzie also outlined
the NIPALS
algorithm,
later
rediscoveredby Wold [3]. Hotelling[4] further
developed PCA to its present stage. In the 1930s
the developmentof factor analysis(FA) wasstarted by Thurstone and other psychologists. This
needs mentioning
here because FA is closely related to PCA and often the two methods are
confused and the two names are incorrectly
usedinterchangeably.
Since then, the utility of PCA has been rediscovered in many diverse scientific fields, resulting
in, amongst other things, an abundanceof redundant terminology.
PCA now goes under many
names. Apart from those already mentioned,
singular value decomposition(SVD) is used innumerical
analysis [5,6] and Karhunen-LoCve
expansion [7,8] in electrical engineering.
Eigenvector
analysis and characteristic
vector analysis are often
used in the physical sciences. In image analysis,
the term Hotellingtransformation
is often used
for a principal component
projection.
Correspondence analysis is a special double-scaled
variant of
PCA that is much favoured in French-speaking
countries and Canada and in some scientific fields.
Many good statisticaltextbooksthat include
this subject have been published, e.g., by Gnanadesikan[9], Mardia
et al. [lo], Johnson
andWichern [ll] and Joliffe [12]. The latter is devoted
solely to PCA and is strongly recommendedforreading.
In chemistry,PCA
wasintroducedbyMalinowski
around 1960 under the name principal factor analysis, and after 1970 a large number of chemical applications
have been published(see Malinowski
and Howery [13]), and Kowalski
et al. [14]).
In geology, PCA has lived a more secluded life,
partly overshadowed
by its twin brother factor
analysis (FA), which has seen ups and downs in
the past 15-20 years. The one eminent textbook inthis field of geological factor analysis is that by
Joreskog, Klovan and Reyment [15]. Davis [16],38

who has set the standards
for statistics and data
analysis in geology for more than a decade, also
included a lucid introduction
to PCA.

2 PROBLEM

DEFINITION

FOR MULTIVARIATEDATAThe starting point in all multivariate
data analysis is a data matrix (a data table) denoted by X.
The N rows in the table are termed “objects”.
These often correspondto chemical or geologicalsamples. The K columns are termed “variables”
and comprise the measurements
made on the objects. Fig. 1 gives an overview of the different
goals one can have for analysing
a data matrix.
These are defined by the problem at hand and not
all of them have to be consideredat the sametime.
Fig. 2 gives a graphical overview of the matrices
and vectors used in PCA. Many of the goals of
PCA are concerned with finding relationships
between objects. One may be interested,
for example, in finding classes of similar objects. The classmembershipmay be known in advance, but it may
also be found by exploration of the available data.
Associatedwith this is the detection of outliers,
since outliers do not belong to known classes.

A matrixof data,measuredfor

N

SIMPLIFICATIONDATA REDUCTION
MODELINGOUTLIER DETECTIONVARIABLE
SELECTIOn
CLASSIFICATION
PREDICTIONUNMIXING
Fig. 1. Principal component
many goals.

analysis on a data matrix can have

Tutorial

k

K

1

t,

U

5

X

i

Fig. 3. The data matrix X can be regarded as a combination ofan underlying structure (PC model) M and noise E. The
underlying structure can be known in advance or one may haveto estimate it from X.

N 0

E

P’,
p;

E
[

X=

lii+TP’+E

Fig. 2. A data matrix X with its first two principal components.
Index i is used for objects (rows) and index k for variables(columns). There are N objects and K variables. The matrix E
contains the residuals, the part of the data not “explained” by
the PC model.

Anothergoal could be data reduction.
This isuseful when large amountsof data may be approximatedby a moderately complex model structure.
In general, almost any data matrix can be simplified by PCA. A large table of numbers is one of
the more difficult things for the human mind tocomprehend.
PCA can be used together with awell selected set of objects and variables to build amodel of how a physical or chemical system behaves, and this model can be used for prediction
when new data are measured for the same system.
PCA has also been used for unmixing
constantsum mixtures. This branch is usually called curve
resolution [17,18].
PCA estimates the correlation
structure of thevariables. The importanceof a variable in a PC
model is indicatedby the size of its residual
variance. This is often used for variable selection.
Fig. 3 gives a graphical explanationof PCA asa tool for separating
an underlying
systematic

data structure from noise. Fig. 4a and b indicate
the projection properties of PCA. With adequate
interpretation,such projections
reveal the dominating characteristics
of a given multivariate
data
set.
Fig. 5 contains a small (3 x 4) numerical illustration that will be used as an example.
(0)

El
X

lb)

clli

P’

Fig. 4. (a) Projecting the matrix X into a vector I is the same asassigning a scalar to every object (row). The projection ischosen such that the values in r have desirable properties andthat the noise contributes as little as possible. (b) projecting thematrix X into a vector p’ is the same as assigning a scalar to
every variable (column). The projection is chosen such that thevalues in p’ have desirable properties and that the noise
contributed as little as possible.

39

Chemometrics and Intelligent Laboratory Systems

n

0
4

3

4

3

5

5

6

4

0

w
0

scalingweights

scaledtest objects

Xt -x

Xt

, scaled

Fig. 5. A training data matrix used as an illustration. Two
extra objects are included as a test set. The actions of meancentring and variance-scaling are illustrated.

3 A CHEMICAL

Fig. 6. Plot of the first two PC score vectors (t, and 1s) of theSwede data of Cole and Phelps [19]. The data were logarithmed
and centred but not scaled before the analysis. Objects l-7
and 15 are fresh samples, whereas 8-14 and 16 are stored
samples.

data classes is even better. Further details of theanalysis of this particular
data set will be given
below.

M

EXAMPLE0
Cole and Phelps [19] presented data relating toa classical multivariate
discrimination
problem.
On 16 samples of fresh and stored swedes (vegetables), they measured
8 chromatographic
peaks.
Two data classes are present: fresh swedes andswedes that have been stored for some months.
The food administration
problem is clear: can we
distinguishbetween these two categories on thebasis of the chromatographic
data alone?
Fig. 6 shows the PC score plot (explained later)
for all 16 samples. Two strong features stand out:
sample 7 is an outlier and the two classes, freshand stored, are indeed separated from each other.
Fig. 7 shows the corresponding
plot calculated on
the reduced data set where object 7 has been
deleted. Here the separation
between these two
40

A

Fig. 7. PC plot of the same data as in Fig. 6 but calculated
after object 7 was deleted.

Tutorial

4 GEOMETRIC INTERPRETATIONCOMPONENT ANALYSIS

OF PRINCIPAL

n

31:

1

A data matrix X with N objects and K variables can be represented as an ensemble of N
points in a K-dimensional space. This space may
be termed M-space for measurement space or
multivariate space or K-space to indicate its dimensionality. An M-space is difficult to visualize
when K > 3. However, mathematically, such aspace is similar to a space with only two or three
dimensions. Geometrical concepts such as points,
lines, planes, distances and angles all have thesame properties in M-space as in 3-space. As ademonstration, consider the following BASIC program, which calculates the distance between thetwo points Z and J in 3-space:
100 KDIM = 3
110 DIST = 0
120 FOR L = 1 TO KDIM
130 DIST = DIST + (X(1, L) - X(J, L)) * *2

1

Fig. 8. A data matrix X is represented as a swarm with N
Points in a K-dimensional space. This figure shows a 3-spacewith a straight line fitted to the points: a one-componentPCmodel. The PC score of an object ( ri) is its orthogonal projection on the PC line. The direction coefficientsthe loading vector pk.

of the line from

140 NEXT L
150 DIST = SQR(DIST)
How can be change this program to calculate thedistance between two points in a space with, say, 7
or 156 dimensions? Simply change statement 100
to KDIM = 7 or KDIM = 156.
A straight line with direction coefficients P(K)passing through a point with coordinates C(E()
has the same equation in any linear space. All
points (I) on the line have coordinates X( I, K)
obeying the relationship (again in BASIC notation)X(1, K) = C(K) + T(1) *P(K)Hence, one can use 2-spaces and 3-spaces as illustrations for what happens in any space we discuss

henceforth.

Fig. 8 shows a 3-space with a point
swarm approximatedby a one-componentPCmodel: a straight line. A two-component PC model
is a plane - defined by two orthogonal lines and an A-components PC model is an A-dimensional hyperplane. From Fig. 8 it may be realizedthat the fitting of a principal component line to anumber of data points is a least squares process.
Lines, planes and hyperplanes can be seen as

spaces with one, two and more dimensions. Hence,
we can see PCA also as the projection of a point
swarm in M-space down on a lower-dimensional
subspace with A dimensions.
Another way to think about PCA is to regard
the subspaces as a windows into M-space. The
data are projected on to the window, which gives a
picture of their configuration in M-space.

5 MATHEMATICAL DEFINITION OF PRINCIPAL COMPONENT ANALYSISThe projection of X down on an A-dimensional
subspace by means of the projection matrix P’
gives the object coordinates in this plane, T. The
columns in T, f,, are called score vectors and therows in P’, pi, are called loading vectors. The
latter comprise the direction coefficients of the PC(hyper) plane. The vectors r, and p. are orthogonal, i.e., p,!pj = 0 and t,!tj = 0, for i f j.
The deviations between projections and theoriginal coordinates are termed the residuals. These41

W Chemometrics

and Intelligent Laboratory

Systems

are collected in the matrix E. PCA in matrix
is the least squares model:form

X=lx+TP’+E
Here the mean vector X is explicitly included in
the model formulation,
but this is not mandatory.
The data may be projected on a hyperplanepassing through the origin. Fig. 9 gives a graphicalrepresentationof this formula.
The sizes of the vectors t, and p, in a PC
dimension
are undefinedwith respect to a multiplicative constant, c, as tp = (tc)( p/c). Hence it isnecessaryto anchor the solution in some way.
This is usually done by normalizing
the vectors p,
to length 1.0. In addition, it is useful to constrain
its largest element to be positive. In this way, theambiguity for c = - 1 is removed.
An anchoring often used in FA is to have thelength of p, be the square root of the corresponding eigenvalue I,. This makes the elements in p,correspond directly to correlation
coefficients andthe score vectors t, be standardized
to length 1.0.
It is instructive
to make a comparison
with thesingular value decomposition(SVD) formulation:
X=l.f+UDV’+E
In this instance, V’ is identical with P’. U contains the same column vectors as does T, but
normalized
to length one. D is a diagonal matrix
containing
the lengths of the column vectors of T.
These diagonal elements of D are the square rootsof the eigenvalues of X’X.
In the statistical literature, PCA has two slightly
different meanings. Traditionally,
PCA has beenFig. 9. A data matrix X can be decomposed as a sum of
matrices Mi and a residual E. The M, can be seen as consisting of outer products of a score vector t, and a loading vector
,
PI.
42

viewed as an expansion of X in as many components as min(N, K). This corresponds
to expressing X in new orthogonalvariables,i.e., atransformation
to a new coordinate
system. The
one which is discussed here refers to PCA as theapproximationof the matrix X by a model with arelatively small number of columns in T and P.
The possibility of deciding on a specific cut-off ofthe number of components
gives a flexible tool for
problem-dependent
data analysis: several contributions to this issue make ample use of these PCA
facilities.
A basic assumption
in the use of PCA is thatthe score and loading vectors corresponding
to thelargest eigenvalues contain the most useful information relating to the specific problem, and thatthe remaining ones mainly comprise noise. Therefore, these vectors are usually written in order ofdescending eigenvalues.
Often the obtained PC model is rotated by therotation matrix R to make the scores and loadingeasier to interpret. This is possible because of theequivalence
TP’ = m-‘P’= SQ’

The FA literature contains numerous
discussions
about various rotation schemes, which we refrainfrom adding to here because the choice of rotationis very problem specific and often problematic.
Once the PC model has been developed for a“training
matrix”, new objects or variables may
be fitted to the model giving scores, t, for the new
objects, or loadings,
p, for the new variables,
respectively. In addition, the variance of the residuals, e, is obtained for each fitted item, providing
a measure of similarity between the item and the“trainingdata”. If this residual variance is larger
than that found in the training stage, it can beconcludedthat the new object (or variable) does
not belong to the training population.
Hypothesis
tests can be applied to this situation. The residuals
may alternatively
be interpreted
as residual distances with respect to a pertinent PC model.
The formulae for a new object x are as follows:
multiply by the loadings from the training stage ofobtain the estimated scores t:
t=xP

Tutorial

6 STATISTICS;

Factorq54.9 %’

45.1 %

2Analysis

Eigenvalues

ExplaIned

SS

Explained

Variance

Fig. 10. Results for the PC model built from the training set in
Fig. 5.

x is projected into the A-dimensional
was developed in the training stage.
Calculated the residuals vector e:

space thate-x-rP’ore=x(I-PP’)
Here I is the identity
matrix of size K. This
calculationof the new scores t, or loadings p, isequivalent
to linear regression because of the orthogonalityof the vectors.
Figs. 10 and 11 show the results of a PCA ofthe (3 X 4) matrix in Fig. 5.

Traimng

Test

Fig. 11. Scores obtained for training and test set in Fig. 5.

n

HOW TO USE THE RESIDUALSThis section describes
two different,
but related, statistics of a data matrix and its residuals,
the usual variance statistics and influence statistics(leverage).
The amount of explained variance can be expressed in different
ways. In FA the data arealways scaled to unit variance and the loadings are
traditionallycalculatedby diagonalizationof thecorrelation
matrix X’X. To obtain a measure corresponding
to the factor analyticaleigenvalue, in
PCA one may calculate
the fraction of the explained
sum of squares (SS) multipliedby thenumber of variables, K. Thus, in Fig. 13 the firstPC explains
83.1% of the SS. Hence the first
eigenvalue is 0.831 X 4 = 3.324.
For a centred
matrix (column
means subtracted), the variance is the sum of squares (SS)divided by the numberof degrees of freedom.
Sums of squares can be calculated for the matrix
X and the residuals E. The number of degrees of
freedom depends on the number of PC dimensions calculated. It is (N - A - l)( K - A) for the
Ath dimension
when the data have been centred(columnaverages
subtracted),otherwise(N A)( K - A).It is also practical to list the residual
variance and modelling
power for each variable(see Fig. 10).
The modelingpower is defined as explained
standard
deviation
per variable (1 - s,Js~~)_ A
variable is completely relevant when its modelingpower is 1. Variables with a low modeling power,
below ca. (A/K), are of little relevance.
Thisfollows from the eigenvalue larger than one rule ofsignificance(see below).
The total sum of squares and the sums ofsquares over rows and columns can all be used to
calculate variance statistics. These can be shownas histograms and allow one to follow the evolution of the PCA model as more dimensionsarecalculated. Fig. 12 gives an idea of how this would
look in the case of variable statistics.
The topic of influential
data has been introduced recently, mainly in multiple regression analysis [20-231. A measure of influence that can bevisualized
geometricallyis leverage.
The term
leverage is based on the Archimedianidea that43

nChemometrics

and Intelligent Laboratory

UNEXPLAINEDVARIANCE

Systems

PC 0

PC I

I

PC2

illustration of the effect of a high leverage point
on a principal component. It can be seen that highleverage is not necessarily bad. A high leverageobservation falling near a PC axis only reinforces
the PC model. A high leverage observation lying
far away from a PC line causes a rotation of thePC.
Leverage is calculated as follows [24,25]:
Ha = T(T’T) -IT’
The diagonal element tr,, of He is the leverage forthe i th object. The hi values are between 0 and 1.
For the variables, leverage is calculated as

IPC3

H, = PP’VARIABLE NURSER--)

Fig. 12. Statistics for the variables. The data are shown ashistogram bars representing the variance per variable formean-centred and variance-scaled data. With 0 PCs, all variables have the same variance. For PC models with increasing
dimensionality,
the variance of the variables is used up. The
hatched bar shows a variable that contributes little to the PC
model. A similar reasoning can be. used for object variances.

anything can be lifted out of balance if the lifter
has a long enough lever.
The least squares method used for fitting principal components to object points in M-spacemakes leverage useful for PCA. Fig. 13 shows an

Fig. 13. Leverage. A point of high leverage (indicated by asquare) can rotate the principal component axis over an anglerb.

44

The diagonal elements h,, of JZ, are the leverages
for the variables.
The interpretation of leverage is closely related
to the concepts of outliers and of construction of aset of samples, i.e., the experimental (or sampling)
design.

7 PLOTS

Perhaps the most common use of PCA is in theconversion of a data matrix to a few informative
plots. By plotting the columns t, in the scorematrix T against each other, one obtains a pictureof the objects and their configuration in M-space.
The first few component plots, the i,-t, or t,-t,,etc., display the most dominant patterns in X. As
was commented upon above, this tacitly assumes
that the directions of maximum variance represent
the directions of m~mum info~ation, This need
not apply to all types of data sets, but it is a wellsubstantiated empirical finding.
Fig. 14 shows the loading plot corresponding to
fig. 6 for the swedes example. In this plot one candirectly identify which variables cause No. 7 to be
an outlier and which variables are responsible forthe separation of the two classes, fresh and stored.
The directions in Fig. 6 correspond directly to thedirections in Fig. 14. The horizontal direction
separates No. 7 from the others in Fig. 6. Hence,variables far from zero in the horizontal directionin Fig. 14 (Nos. 5 and 6) are those responsible forthis. Analogously, the vertical direction in Fig. 6

Tutorial

Loadingplot(pi-p21

of

log

W

SWEDES

2

Da

D5
1
N
0

0

D2

-1

Fig. 14. Plot of the first two loading

Mergedscore

vectors

and(p,

and p2) corresponding

loadingplots

ofto Fig. 6.

log

SWEOES

1.7
1.6
i.6
i.4
i.3
i.2

t

i., -

N
P
pf

I

-

.9

-

.6
.7
.6

-

.5

-

.4
.3
.2

-

.1
0

_

.a
014

Q&g
Qi3
05

oi2

-.i

05

-.2
-.I

-

-.4

-

-.I
-.6

-

-.7

-

-.6

-

.5

011

.2
02

.6

t1 pi
Fig. 15. Figs. 6 and 14 superimposed(originin the same place). The loadings(triangles)

are scaled up by a factor

of 3.

45

n

Chemometrics

and Intelligent

Laboratory

Systems

separates the two classes. Hence, variables vertically far from zero in Fig. 7 (all except No. 2, butmainly 1, 3, 7 and 8) contribute to this separation.
One may also superimpose these two plots (Figs.
6 and 14) to obtain Fig. 15, which simultaneously
displays both the objects and the variables. This
type of plot is common in correspondence
analysis, but can also be accomplishedby ordinary
PCA. It is largely a matter of choice whether one
wishes to inspect these two complementaryplots
in one or two separate figures. For larger data sets
it is probably best to keep the two plots separate
to improve clarity.
8 APPLICATIONSYSISOF PRINCIPALCOMPONENTANAL-

PCA can be applied to any data matrix (properly transformed
and scaled, see Section 9). This isalso recommended
as an initial step of any multivariate analysis to obtain a first look at the structure of the data, to help identify outliers, delineateclasses, etc. However, when the objective is classification (pattern recognitionlevels 1 or 2 [26]) or
relating
one set of variables
to another(e.g.,
calibration),
there are extensions of PCA that are
more efficient for these problems.
Here a rather partisan
view of PCA is presented. It reflects the experience that well considered projections
encompassa surprisingly
large
range of typical scientific problem formulations[27,28].

component
mixtures.
In two-dimensional
highperformanceliquid chromatographic
analysis, forinstance, this is a way to find the smallest numberof species in a sample. Reviews of the numerous
applicationsof these types can be found in
Kowalski et al. [14] and Malinowski
and Howery1131.
The first few score vectors, t,, may be seen as
latent variables that express most of the information in the data. This has recently been used tofind“principal
properties”
of aminoacids,
solvents and catalysts, which then later find use in
the quantitativedescriptionof the selectionofthese entities [30,31]. See Fig. 16 for an example.
PCA and FA are used in many other types ofapplications.
As PC models can be also calculated
for matrices with incomplete
data, PCA may be
used to predict the values for the “holes” in a data
matrix. This, however, is more efficiently
donewith partial least quares analysis, mentionedinSection 11.

7
t

16
*

8.1 Overview (plots) of any data table
The score plot of the first two or three score
outliers and othervectors, 1,) shows groupings,
strong patterns in the data. This can be seen in thescore plot of the swedes data. Many other applications can be found in this issue.
8.2 Dimensionality

reduction

As pointed out by Frank and Kowalski [29],two main groups of FA and PCA applications
canbe seen in analytical chemistry, namely the extraction of the underlying
factors - the latent variables - and the resolutionof spectra of multi46

Fig. 16. Plot of the first and second PC score vectors of a tablewith 20 properties for the 20 common “natural” amino acids.
The grouping indicates a relationship between the physicalchemical properties of the amino acids and which nucleotide isused in the second codon in the amino acid synthesis (adenosine, uracil and cytosine). The amino acids coded for by
guanine (G) do not seem to participate in this relationship [46].
These PC score vectors numerically describe the structuralchange within families of peptides and thus constitute a basis
for structure-activity
relationships for peptides 1301.

Tutorial n

8.3 Similarity

modelsWold [32] showed that a PC model has thesame approximation
property for a data table ofsimilar objects as does a polynomial
for bivariate
data in a limited interval;
the PC model can beseen as a Taylor expansionof a data table. The
closer the similarity between the objects, the fewer
terms are needed in the expansion
to achieve acertain approximation
goodness. The prerequisites
for this interpretationof PCA are a few assumptions about differentiability
and continuityof thedata generating process, which have been found to
hold in many instances.
This explains, at least
partly, the practical utility of PCA.
Hence, a pattern recognition
method may bebased on separate PC models, one for each classof objects. New objects are classified according totheir fit or lack of fit to the class models, which
gives a probabilisticclassification.
This is the basis
for the SIMCA method (Soft Independent
Modeling of Class Analogies)[33]. If one wishes todistinguish
optimally
between the two classes ofswedes, a PC model can be fitted to each of theclasses. The results show that indeed there is similarity within the classes (two- or three-component
models adequately describe the data) and that theclasses are well separated(fresh samples fit the“stored class” badly and vice versa); see Wold etal. [34].

9 DATA PRE-TREATMENT

The principal
components
model parameters
depend on the data matrix, its transformation
andits scaling. Hence, these must be explicitly defined.
First, the data matrix: In statistics it is usuallycustomary to put all available data into the matrix
and analyse
the lot. This conforms
with theconvention
that all data reflect legitimate
phenomena that have been sampled. When one has
full control over the sampling, or over the experimental design, this approach may be correct.
Practical experience from many sciences often
outlines a less ideal reality, however. One is frequently faced with outlying data in real data sets.
One also often knows certain pertinent
externalfacts (such that cannot be coded into the data
matrix itself) relating to the problem formulation.
In chemistry, for example, one often has one set ofobjects (e.g., analytical samples) about which certain essential properties are known. In the swedes
example, for instance, there were 16 samples thatwere known or assumed
to be either fresh or
stored. The analysis confirmed this, but the PCA
recognized that one sample did not comply. This
type of external information
can be used to compute a polished PC model. On may subsequently
investigate new objects and project them on to thesame scores plot without letting the new samples
influence the PC model as such.
Hence, it is practical to distinguishbetween a
training set that is used for calculatingproblemdependent PC models and a test set of objects thatare later subjected to the same projection as thatdeveloped in the training phase.
From the least squares formulationof the PC
models above it is seen that the scores, t, can beviewed as linear combinationsof the data with thecoefficientsp’. Conversely,
the loadings, p’, canalso to be understoodas linear combinationsofthe data with the coefficientst. This duality has
resulted in the coining of the term bilinear modeling (BLM) [35]. From this follows that if one
wishes to have precise r-values, i.e., precise information about the objects, one should have many
variables per object and vice versa. The rule formultiple regression that the number of variables
must be much smaller that the number of objects
does not apply to PCA.
As PCA is a least squares method,
outlier
severely influence the model. Hence it is essential
to find and correct or eliminate outliers before thefinal PC model is developed. This is easily doneby means of a few initial PC plots; outliers thathave a critical influence on the model will reveal
themselves clearly. Irrelevant variables can also be
identified
by an initial PC analysis (after taking
care of outliers). Variables with little explained
variance may be removed without changing
thePC model. However, usually it does not matter ifthey are left in. The reductionof the data set
should be made only if there is a cost (computational or experimental)
connected
with keeping
the variables in the model.
41

nChemometrics

and Intelligent

Laboratory

Systems

Second, the data matrix can be subjectedtotransformations
that make the data more symmetrically distributed,
such as a logarithmictransformation, which was used in the swedes example
above. Taking the logarithm of positively skewed
data makes the tail of the data distribution
shrink,often making the data more centrally distributed.
This is commonly done with chromatographic
data
and trace element concentration
data. Autocorrelation or Fourier transforms
are recommended
ifmass spectrometricdata are used for classification
purposes [36]. There exist a great many potential
transformations
that may be useful in specialized
contexts. Generally, one should exercise some discipline in applying univariate transformationswithdifferent parameters
for different variables, lest a“shear” be introducedin the transformed
covariante (correlation)
matrix relative to that pertaining to the original data.
Centring
the data by subtracting
the column
averages corresponds
to moving the coordinate
system to the centre of the data.
Third, the scaling of the data matrix must be
specified.
Geometrically,
this corresponds
tochanging the length of the coordinate
axes. The
scaling is essential because PCA is a least squares
method, which makes variables with large variance
have large loadings.
To avoid this bias, it iscustomary to standardize
the data matrix so thateach column has a variance1.0. This variancescaling makes all coordinate
axes have the samelength, giving each variable the same influence on
the PC model. This is reasonable in the first stagesof a multivariate
data analysis. This scaling makes
the PC loadings be eigenvectors of the correlation
matrix.
Variance scaling can be recommended
in mostinstances, but some care must be taken if variables
that are almost constant have been included in thedata set. The scaling will then scale up thesevariables
substantially.
If the variationof thesevariables is merely noise, this noise will be scaled
up to be more influential
in the analysis. Rule: If
the standard deviation of a variable over the data
set is smaller than about four times its error ofmeasurement,
leave that variable unscaled.
Theother variables may still be variance scaled if sodesired.
When different types of variables are present,48

say 6 infrared absorbancesand 55 gas chromatographic peak sizes, a blockwise scaling may be
performed
so that the total variance is the samefor each type of variables with autoscaling
withineach block. This is accomplishedby dividing eachvariable by is standard deviation times the squareroot of the number of variables of that type in theblock.
When both objects and variables are centred
and normalized
to unit variance, the PCA of thisderived matrix is called correspondence
analysis.
In correspondence
analysis the “sizes” of the objects are removed from the model. This may bedesirable in some applications,e.g., contingency
tables and the like, but certainly not in general.
This double-scaling/
centring
in effect brings
about a complete equivalence
between variables
and objects, which can confuse the interpretationof the results of the data analysis. The loading andscore plots can now be superimposed.
However,with proper scaling this can also be done withordinary PCA, as shown in Fig. 15.
The treatmentof missing data forms a special
topic here. There may often be “holes” in the data
matrix. The treatment of these can proceed mainly
in two ways. One is to “guess” a value fromknowledge of the populationof objects or of theproperties of the measuring instrument.
A simpleguess that does not affect the PCA result too
much is to replace the missing value with theaverage for the rest of the column. Secondly, thePC algorithm may be able to cope with missing
values (see Appendix). This allows the calculationof a model without having to fill in the missing
value.
No matter how missing values are dealt with,an evaluation of the residuals can give extra hints.
Objects (or variables)with missing values thatshow up as outliers are to be treated with suspicion. As a rule of thumb, one can state that eachobject and variable should have more than fivedefined values per PC dimension.

10 RANK,ORCOMPONENTSDIMENSIONALITY,
MODEL

OF

A PRINCIPAL

When PCA is used as an exploratory
took, thefirst two or three components
are always ex-Tutorial

tracted. These are used for studying the data
structure in terms of plots. In many instances thisserves well to clean the data of typing errors,
sampling errors, etc. This process sometimes must
be carried out iteratively in several rounds inorder to pick out successively less outlying data.
When the purpose is to have a model of X,
however, the correct number of components, A, isessential. Several criteria may be used to determine A.
It should be pointed out here that rank anddimensionality are not used in the strict mathematical sense. The ideal case is to have a number
A of eigenvalues different from zero [A smaller or
equal to min( K, N)] and all the others being zero.
Measurement noise, sampling noise or other irregularities cause almost all data matrices to be offull rank [A = min( K, N)]. This is the most difficult aspect of using PC models on “noisy” data
sets.
Often, as many components are extracted as
are needed to make the variance of the residual of
the same size as the error of measurement of thedata in X. However, this is based on the assumption that all systematic chemical and physical variations in X can be explained by a PC model, anassumption that is often dubious. Variables can bevery precise and still contain very little chemical
information. Therefore, some statistical criterion
is needed to estimate A.
A criterion that is popular, especially in FA, is
to use factors with eigenvalues larger than one.
This corresponds to using PCs explaining at leastone K th of the total sum of squares, where K is
the number of variables. This, in turn, ensures that
the PCs used in the model have contributions
from at least two variables. Joreskog et al. [15]discussed aspects in more detail.
Malinowski and Howery [13] and others haveproposed several criteria based on the rate ofdecrease of the remaining residual sum of squares,
but these criteria are not well understood theoretically and should be used with caution [10,15].
Criteria based on bootstrapping and crossvalidation (CV) have been developed for statisticalmodel testing 1371.Bootstrapping uses the residuals to simulate a large number of data sets similar
to the original and thereafter to study the distribu-ntion of the model parameters over these data.
With CV the idea is to keep parts of the dataout of the model development, then predict the
kept out data by the model, and finally compare
the predicted values with the actual values. The
squared differences between predicted and observed values are summed to form the prediction
sum of squares (PRESS).
This procedure is repeated several times, keeping out different parts of the data until each data
element has been kept out once and only once andthus PRESS has one contribution from each data
element. PRESS then is a measure of the predictive power of the tested model.
In PCA, CV is made for consecutive model
dimensions starting with A = 0. For each additional dimension, CV gives a PRESS, which iscompared with the error one would obtain by justguessing the values of the data elements, namely
the residual sum of squares (RSS) of the previous
dimension. When PRESS is not significantly
smaller than RSS, the tested dimension is considered insi~ficant
and the model building is
stopped.
CV for PCA was first developed by Wold [38],using the NIPALS algorithm (see Appendix) andlater by Eastment and Krzanowski [39], using
SVD for the computations. Experience shows that
this method works well and that with the proper
algorit~s
it is not too computation~ly demanding. CV is slightly conservative, i.e., leads to too
few components rather than too many. In practical work, this is an advantage in that the data are
not ove~nte~reted and false leads are not created.

11 EXTENSIONS;
TWO-BLOCK
MANY-WAY TABLESREGRESSIONAND

PCA can be extended to data matrices divided
into two or more blocks of variables and is thencalled partial least squares (PLS) analysis (partial
least squares projection to latent structures). The
two-block PLS regression is related to multiple
linear regression, but PLS applies also to the casewith several Y-variables. See, e.g., Geladi andKowalski [40], for a tutorial on PLS. Wold et al.
[41] detailed the theoretical background for PLS
49

n

Chcmometrics and Intelligent Laboratory Systems

regression. Several contributions to the present
issue illustrate the diverse application potential for
PLS modeling and prediction.
Multivariate image analysis can use PCA beneficially, as explained by Esbensen et al. [42].
Multivariate image analysis is important in thechemical laboratory [43] and Geographical Information Systems (GIS).
PCA and PLS have been extended to three-way
and higher order matrices by Lohmiiller and Wold
1441, and others. See the expository article by
Wold et al. [45] for the theoretical backgroundand application possibilities for these new bilinear
concepts.

12 SUMMARY

Principal component analysis of a data matrix
extracts the dominant patterns in the matrix interms of a complementary set of score and loadingplots. It is the r~ponsibility of the data analyst to
formulate the scientific issue at hand in terms of
PC projections, PLS regressions, etc. Ask yourself,
or the investigator, why the data matrix was collected, and for what purpose the experiments andmeasurements were made. Specify before the analysis what kinds of patterns you would expect andwhat you would find exciting.
The results of the analysis depend on the scaling of the matrix, which therefore must be specified. Variance scaling, where each variable is scaled
to unit variance, can be recommended for generaluse, provided that almost constant variables areleft unscaled. Combining different types of variables warrants blockscaling.
In the initial analysis, look for outliers andstrong groupings in the plots, indicating that thedata matrix perhaps should be “polished” orwhether disjoint modeling is the proper course.
For plotting purposes, two or three principal
components are usually sufficient, but for modeling purposes the number of significant components should be properly determined, e.g. bycross-validation.
Use the resulting principal components to guide
your continued investigation or chemical experimentation, not as an end in itself.
50

APPENDIX

Denoting the centred and scaled data matrix by

U, the loading vectors p, are eigenvectors to thecovariance matrix U'U and the score vectors t,
are eigenvectors to the association matrix UU’.
Therefore, principal components have earlier been
computed by diagonalizing U’U or UU’. However, singular value decomposition [5] is more
efficient when all PCs are desired, while the
NIPALS method ]3] is faster if just the first few
PCs are to be computed. The NIPALS algorithm
is so simple that it can be formulated in a few
lines of programming and also gives an interesting
interpretation of vector-matrix multiplication as
the partial least squares estimation of a slope. The
NIPALS algorithm also has the advantage ofworking for matrices with moderate amounts ofrandomly distributed missing observations.
The algorithm is as follows. First, scale the data
matrix X and subtract the column averages if
desired. Then, for each dimension, u:(i) From a start for the score vector t, e.g., thecolumn in X with the largest variance.
(ii) Calculate a loading vector as p’ = t’X/t’t.
The elements in p can be interpreted as the slopesin the linear regressions (without intercept) of t on
the corresponding column in X.
(iii) Normalize p to length one by multiplying
by c = l/e(or anchor it otherwise).
(iv) Calculate a new score vector 2 = Xp/p’p.
The ith element in t can be interpreted as theslope in the linear regression of p’ on the ith row
in X.
(v) Check the convergence, for instance using
the sum of squared differences between all elements in two consecutive score vectors. If convergence, continue with step vi, otherwise return tostep ii. If convergence has not been reached in,say, 25 iterations, break anyway. The data arethen almost (h~er)spheri~,with no strongly
preferred direction of maximum variance.
(vi) Form the residual E = X - tp’. Use E as X
in the next dimension.
Inserting the expression for t in step iv into
step ii gives p = X’Xp * c/t’t (c is the normalization constant in step iii). Hence p is an eigenvector to X’X with the eigenvalue t’t/c and we seeTutorialthat the NIPALS algorithm is a variant of thepower method used for matrix diagonalization(see, e.g., Golub and VanLoan [5]). As indicatedpreviously, the eigenvalue is the amount of variance explained by the corresponding component
multiplied by the number of variables, K.
REFERENCES
1 K. Pearson, On lines and planes of closest fit to systems ofpoints in space, Philosophical Magazine, (6) 2 (1901)
559-572.

2 R. Fisher and W. MacKenzie, Studies in crop variation. II.
The manurial response of different potato varieties, Journalof Agricultural Science, 13 (1923) 311-320
3 H. Wold, Nonlinear estimation by iterative least squares
procedures, in F. David (Editor), Research Papers in Statistics, Wiley, New York, 1966, pp. 411-444.
4 H. Hotelling, Analysis of a compfex of statistical variables
into principal #m~nents,
Journal of E~catjonaf
Psychology, 24 (1933) 417-441 and 498520.
G. Gdub and C. VanLoan, Matrix Computations, The
Johns Hopkins University Press, Oxford, 1983.
J. Mandel, Use of the singular value decomposition inregression analysis, American Statistician, 36 (1982) 15-24.
K. Karhunen, Uber lineare Methoden in der Wahrscheinlichkeitsrechnung, Annales Acudemiae Scientiarum Fennicae, Series A, 137 (1947).
8 M. Lobe, Fonctions aleatoires de seconde ordre, in P. Levy
(Editor), Processes Stochastiques et Mouvement Brownien,
Hermann, Paris, 1948.
9 R. Gnanadesikan, Methods for Statistical Data Analysis ofMultivariate Observations, Wiley, New York, 1977.
10 K. Mardia, J. Kent and J. Bibby, ~ultiva~ute AnuZysis,
Academic Press, London, 1979.
11 R. Johnson and D. Wichem, Applied Multivariate Statistical Analysis, Prentice-Hall, Englewood Cliffs, NJ, 1982.
12 J. Johffe, Principal Component Analysis, Springer, Berlin,
1986.
13 F. Malinowski and D. Howery, Factor Analysis in Chemistry, Wiley, New York, 1980.
14 L.S. Ramos, K.R. Beebe, W.P. Carey, E. Sanchez, B.C.
Erickson, B.E. Wilson, L.E. Wangen and B.R. Kowalski,
Chemometrics, Analytical Chemistry, 58 (1986) 294R-315R.
15 K. Jareskog, J. Klovan and R. Reyment, Geological Factor
Analysis, Elsevier, Amsterdam, 1976.
16 J. Davis, Statistics and Data Analysis in Geology, Wiley,
New York, 1973 and 1986.
17 W. Lawton and E. Sylvestre, Self modeling curve resolution, Technometrics, 13 (1971) 617-633.
18 W. Full, R. Ehrlich and J. Klovan, Extended Q
model - Objective definition of external end members in
the analysis of mixtures, Mathe~t~caI Geology, 13 (1981)
331-334.

19 R. Cole and K. Phelps, Use of canonical variate analysis in
the differentiation of Swede cultivars by gas-liquid chro-nmatography of volatile hydrolysis products, Journal of the
Science of Food and Agriculture,

30 (1979) 669-676.

20 R. Hocking, Developments in linear regression methodology: 1959-1982, Technometrics, 25 (1983) 219-230.
21 R. Beckman and R. Cook, Outlier...~, Technometrics, 25(1983) 119-149.

22 D. Belsley, E. Kuh and R. Welsch, Zdenti~ing Zn~uentiu~
Data and Sources of Collinearity, Wiley, new York, 1980.
23 D. Cook and S. Weisber, Residuals and Znfluence in Regression, Chapman and Hall, New York, 1982.
24 P. Velleman and R. Welsch, Efficient computing a regression diagnostics, American Statistician, 35 (1981) 234-242.
25 H. Martens, Mu~tivuriate Calibration, Thesis, University ofTrondheim, 1985.
26 C. Albano, W. Dunn, U. Edlund, E. Johansson, B. Horden,
M. SjBstrSm and S. Weld, Four levels of pattern recognition, Analytica Chimica Acta, 103 (1978) 429-443.
27 S. Weld, C. Albano, W. Dunn, U. Edlund, K. Esbensen, P.
Geladi, S. Hellberg, E. Johansson, W. Lindberg and M.
Sjiistrom, Multivariate data analysis in chemistry, in B.R.
Kowalski (Editor), Chemometrics: Mathematics and Statistics in Chemistry, Reidel, Dordrecht, 1984, pp. 17-95.
28 S. Weld, C. Albano, W.J. Dunn III, K. Esbensen, P.
Geladi, S. Hellberg, E. Johansson, W. Lindberg, M.
Sj&triim, B. Skagerberg, C. Wikstrijm and J. Ghman, Multivatiate data anatysis: Converting chemical data tables toplots, in I. Ugi and J. Brandt (Editors), Proceeding of the
7th International Conference on Computers in Chemical Research and Education (ZCCCREJ, held in Garmisch-Partenkirchen, BDR, June 10-14, 1985, in press.
29 I. Frank and B.R. Kowalski, Chemometrics, Analytical
Chemistry, 54 (1982) 232R-243R.

30 S. Hellberg, M. SjijstrCm and S. Wold, The prediction ofbradykinin potentiating potency of pentapeptides. An example of a peptide quantitative structure-activity relationship, Acta Chemica Scandinavica, Series B, 40 (1986)
135-140.

31 R. Carbon, T. Lundstedt and C. Albano, Screening ofsuitable solvents in organic synthesis. Strategies for solvent
selection, Acta Chemica Scandinaviea, Series B, 39 (1985)
79-84.

32 S. Weld. A theoretical foundation of extrathe~~yn~crelationships (linear free energy relationships), Chemica
Scripta,

5 (1974) 97-106.

33 S. Wold, Pattern recognition by means of disjoint principal
components models, Pattern Recognition, 8 (1976) 127-139.
34 S. Wold, C. Albano, W. Dunn, K. Esbensen, S. Hellberg, E.
Johansson and M. Sjostrom, Pattern recognition; finding
and using regularities in multivariate data, in H. Martensand H. Russwurm (Editors), Food Research and Data Analysis, Applied Science Publishers, London, 1983, pp,147-188.
35 J. Kruskal, Bilinar methods, in W. Kruskal and J. Tanur(Editors), Znte~atjonuf Encyclope~a of Statistics, Vol. I,
The Free Press, New York, 1978.
36 S. Wdd and 0. Christie, Extraction of mass spectral information by a combination of autocorrelation and principal
51

nChemometrics and Intelligent Laboratory Systems

components models. Analyricu Chimicu Acta, 165 (1984)
51-59.
37 P. Diaconis and B. Efron, Computer-intensive methods instatistics, Scienfific American, May (1983) 96-108.
38 S. Wold, Cross validatory estimation of the number ofcomponents in factor and principal components models,Technometrics, 20 (1978) 397-406.
39 H. Eastment and W. Krzanowski, Crossvalidatory choice ofthe number of components from a principal component
analysis, Technometrics, 24 (1982) 73-77.
40 P. Geladi and B.R. Kowalski, Partial least squares regression (PLS): a tutorial, Anut’yticu Chimicu Actu, 185 (1986)
l-17.

41 S. Wold, A. Ruhe, H. Weld and W.J. Dunn III, The
collinearity problem in linear regression. The partial leastsquares approach to generalized inverses, SIAM Journal of
Scientific and Stutisticul

Computations,

5 (1984) 735-743.

42 K. Esbensen, P. Geladi and S. Weld, Bilinear analysis ofmultivariate images (BAMID), in N. Raun (Editor), Pro-

52

Symposium i Anvendt Stutistik, Danmarks EDB-center for Forskning og Uddannelse, Copenhagen, 1986, pp. 279-297.
43 P. Geladi, K. Esbensen and S. Weld, Image analysis, chemical information in images and chemometrics, Anulyticu
Chimica Actu, (1987) in press.
44 J. Lohmoller and H. Wold, Three-mode path models withlatent variables and partial least squares (PLS) parameter
estimation, Puper presented at the European Meeting of thepsychometrics
Society, Groningen, Holland
1980, Forschungsbericht 80.03 Fachbereich Pedagogik, Hochschule
der Bundeswehr, Munich.
45 S. Weld, P. Geladi, K. Esbensen and J. &man, Multi-way
principal components and PLS-analysis, Journal of Chemometrics, 1 (1987) 41-56.
46 M. Sjiistrbm and S. Weld, A multivariate study of therelationshipbetweenthe genetic
code and thephysical-chemical properties of amino acids, Journal of
ceedings of Nordisk

Molecular Evolution,

22 (1985) 272-277.