Probabilistic Latent Semantic Indexing
Thomas Hofmann

International Computer Science Institute, Berkeley, CA &EECS Department, CS Division, UC Berkeley

hofmann@cs.berkeley.edu

Abstract

return the relevant data in some amenable representation,e.g., in form of a ranked list of relevant documents. Many
retrieval methods are based on simple word matching strategies to determine the rank of relevance of a document withrespect to a query. Yet, it is well known that literal term
matching has severe drawbacks, mainly due to the ambivalence of words and their unavoidable lack of precision as wellas due to personal style and individual di erences in word
usage.
Latent Semantic Analysis (LSA) [1] is an approach toautomatic indexing and information retrieval that attempts
to overcome these problems by mapping documents as wellas terms to a representation in the so{called latent semantic space. LSA usually takes the (high dimensional) vector space representation of documents based on term frequencies [14] as a starting point and applies a dimension
reducing linear projection. The speci c form of this mapping is determined by a given document collection and isbased on a Singular Value Decomposition (SVD) of the corresponding term/document matrix. The general claim isthat similarities between documents or between documents
and queries can be more reliably estimated in the reduced
latent space representation than in the original representation. The rationale is that documents which share frequently
co-occurring terms will have a similar representation in thelatent space, even if they have no terms in common. LSA
thus performs some sort of noise reduction and has the potential bene t to detect synonyms as well as words that referto the same topic. In many applications this has proven to
result in more robust word processing.
Although LSA has been applied with remarkable successin di erent domains including automatic indexing (Latent
Semantic Indexing, LSI) [1, 3], it has a number of de cits,
mainly due to its unsatisfactory statistical foundation. The
primary goal of this paper is to present a novel approach
to LSA and factor analysis { called Probabilistic Latent Semantic Analysis (PLSA) { that has a solid statistical foundation, since it is based on the likelihood principle and denes a proper generative model of the data. This implies inparticular that standard techniques from statistics can beapplied for questions like model tting, model combination,and complexity control. In addition, the factor representation obtained by PLSA allows to deal with polysemouswords and to explicitly distinguish between di erent meanings and di erent types of word usage.

Probabilistic Latent Semantic Indexing is a novel approach
to automated document indexing which is based on a statistical latent class model for factor analysis of count data.
Fitted from a training corpus of text documents by a generalization of the Expectation Maximization algorithm, theutilized model is able to deal with domain{speci c synonymy
as well as with polysemous words. In contrast to standard
Latent Semantic Indexing (LSI) by Singular Value Decomposition, the probabilistic variant has a solid statistical foundation and de nes a proper generative data model. Retrieval
experiments on a number of test collections indicate substantial performance gains over direct term matching methods as well as over LSI. In particular, the combination ofmodels with di erent dimensionalities has proven to be advantageous.

1 Introduction
With the advent of digital databases and communication
networks, huge repositories of textual data have become
available to a large public. Today, it is one of the greatchallenges in the information sciences to develop intelligent interfaces for human{machine interaction which support computer users in their quest for relevant information. Although the use of elaborate ergonomic elements likecomputer graphics and visualization has proven to be extremely fruitful to facilitate and enhance information access,progress on the more fundamental question of machine intelligence is ultimately necessary to ensure substantial progresson this issue. In order for computers to interact more naturally with humans, one has to deal with the potential ambivalence, impreciseness, or even vagueness of user requests,
and has to recognize the di erence between what a usermight say or do and what she or he actually meant or intended.
One typical scenario of human{machine interaction in information retrieval is by natural language queries: the user
formulates a request, e.g., by providing a number of keywords or some free-form text, and expects the system to

ACM SIGIR Forum

211

Vol. 51 No. 2, July 2017

2 The Aspect Model

3 Model Fitting with Tempered EM

The core of PLSA is a statistical model which has been called
aspect model [7, 15]. The latter is a latent variable model for
general co-occurrence data which associates an unobserved
class variable z 2 Z = fz1 ; : : : ; zK g with each observation,i.e., with each occurrence of a word w 2 W = fw1 ; : : : ; wM g
in a document d 2 D = fd1 ; : : : ; dN g. In terms of a generativemodel it can be de ned in the following way: select a document d with probability P (d), pick a latent class z with probability P (z jd), generate a word w with probability P (wjz ).
As a result one obtains an observed pair (d; w), while thelatent class variable z is discarded.
Translating this process into a joint probability model
results in the expression

The standard procedure for maximum likelihood estimationin latent variable models is the Expectation Maximization(EM) algorithm [2]. EM alternates two steps: (i) an expectation (E) step where posterior probabilities are computedfor the latent variables z , based on the current estimates ofthe parameters, (ii) an maximization (M) step, where parameters are updated for given posterior probabilities computed in the previous E{step.
For the aspect model in the symmetric parameterization
Bayes' rule yields the E{stepP (z )P (djz )P (wjz)
P (z jd; w) =;(5)z P (z )P (djz )P (wjz )XP (d; w)

= P (d)P (wjd) ; where
P (wjd) =P (wjz )P (z jd) :z2Z

PX X n(d; w) log P (d; w) ;d2D w2W

z2Z

d

0

d;w0

0

w

0

d0 ;w

R

0

d;w

d;w

Alternating (5) with (6){(8) de nes a convergent procedurethat approaches a local maximum of the log{likelihood in(3).
So far we have focused on maximum likelihood estimation or, equivalently, word perplexity reduction. One has,
however, to distinguish between the predictive performanceof the model on training data and the expected performanceon unseen test data. In particular, it is to naive to assume
that a model will generalize well on new data just basedon the fact that it might achieve low perplexity on training
data. To derive conditions under which generalization on
unseen data can be guaranteed is actually the fundamental
problem of statistical learning theory. Here, we propose ageneralization of maximum likelihood for mixture models {called tempered EM (TEM) { which is based on entropic
regularization and is closely related to a method known as
deterministic annealing [13].
Since a principled derivation of TEM is beyond the scopeof this paper (the interested reader is referred to [12, 7]), we
will present the necessary modi cation of standard EM in an
ad hoc manner. Essentially, one introduces a control parameter (inverse computational temperature) and modi es theE-step in (5) according to(3)

P (z jd; w)=

P PP(z()z [)P[(Pd(jzd)jzP)(Pw(jzw)]jz )] :
z0

0

0

0

(9)

Notice that = 1 results in the standard E{step, while for< 1 the likelihood part in Bayes' formula is discounted(additively on the log{scale).
It can be shown, that TEM minimizes an objective function known as the free energy [11] and hence de nes a convergent algorithm. While temperature{based generalizations of
EM and related algorithms for optimization are often used

(4)This is just a re-parameterized version of the generativemodel described by (1), (2).

ACM SIGIR Forum0

P n(d; w)P (zjd;w);(6)
P (wjz ) = Pn(d; w )P (z jd; w )P n(d; w)P (zjd;w)
P (djz ) = P;(7)n(d ; w)P (z jd ; w)
1 X n(d; w)P (z jd;w); R  X n(d; w) : (8)P (z ) =(1)(2)

where n(d; w) denotes the term frequency, i.e., the number oftimes w occurred in d. It is worth noticing that an equivalent
symmetric version of the model can be obtained by invertingthe conditional probability P (z jd) with the help of Bayes'
rule, which results in

X P (z)P (wjz)P (djz) :P (d; w) =0

which is the probability that a word w in a particular document or context d is explained by the factor corresponding
to z . By standard calculations one arrives at the followingM{step re-estimation equations

Essentially, to derive (2) one has to sum over the possible
choices of z which could have generated the observation.
The aspect model is a statistical mixture model [9] which
is based on two independence assumptions: First, observation pairs (d; w) are assumed to be generated independently;
this essentially corresponds to the `bag{of{words' approach.
Secondly, the conditional independence assumption is madethat conditioned on the latent class z , words w are generated independently of the speci c document identity d.
Given that the number of states is smaller than the numberof documents (K  N ), z acts as a bottleneck variable inpredicting w conditioned on d.
Notice that in contrast to document clustering models
document{speci c word distributions P (wjd) are obtainedby a convex combination of the aspects or factors P (wjz ).
Documents are not assigned to clusters, they are characterized by a speci c mixture of factors with weights P (z jd).
These mixing weights o er more modeling power and areconceptually very di erent from posterior probabilities inclustering models and (unsupervised) naive Bayes models(cf. [7]).
Following the likelihood principle, one determines P (d),P (z jd), and P (wjz ) by maximization of the log{likelihood
function
L=

0

0

212

Vol. 51 No. 2, July 2017

as a homotopy or continuation method to avoid unfavorable
local extrema, the main advantage of TEM in our context
is to avoid over tting. Somewhat contrary to the spirit ofannealing as a continuation method we propose to utilize(9) to temper EM by \heating". In order to determine theoptimal value of we propose to make use of some held{outportion of the data. This idea can be implemented by thefollowing scheme:
i. Set1 and perform EM until the performance onheld{out data deteriorates (early stopping).
ii. Decrease , e.g., by setting
 with some rate
parameter  < 1.
iii. As long as the performance on held-out data improves
continue TEM iterations at this value of .
iv. Stop on , i.e., stop when decreasing does not yieldfurther improvements, otherwise goto step (ii).
v. Perform some nal iterations using both, training andheld-out data.
In our experiments, the typical number of iterations TEM
performed starting from randomized initial conditions was
40 , 60, where each iteration requires one pass through thedata, i.e., of the order of R  K arithmetical operations.

embeddingsimplexspannedsub-simplex

KL divergenceprojection

P(w|z 2)
P(w|z 3)Figure 1: Sketch of the probability sub-simplex spanned by
the aspect model.
probability simplex, this can also be thought of in terms of
dimensionality reduction and the sub-simplex can be identi ed with a probabilistic latent semantic space.

4.3 Mixture Decomposition vs. Singular Value Decomposition

To stress this point and to clarify the relation to LSA, let
us rewrite the aspect model as parameterized by (4) in ma^ = (P (d jz )) ,trix notation. Hence de ne matrices by U
V^ = (P (w jz )) , and ^ = diag(P (z )) . The joint probability model P can then be written as a matrix product
P = U^ ^ V^ . By comparing this decomposition with the
SVD decomposition in LSA, one can point out the following
re-interpretation of concepts of linear algebra:
i. The weighted sum over outer products between rows ofU^ and V^ re ects conditional independence in PLSA.
ii. The left/right eigenvectors in SVD are seen to correspond to the factors P (wjz ) and the component distributions P (djz ) of the aspect model.
iii. The mixing proportions P (z ) in PLSA substitute thesingular values of the SVD in LSA.
Despite this similarity, there is also a fundamental difference between PLSA and LSA, which is the objective function utilized to determine the optimal decomposition/approximation. In LSA, this is the L2 {norm or Frobenius norm, which corresponds to an implicit additive Gaussian noise assumption on counts. In contrast, PLSA relieson the likelihood function of multinomial sampling and aims
at an explicit maximization of the predictive power of themodel. On the modeling side this o ers important advantages, for example, the mixture approximation P of the cooccurrence table is a well-de ned probability distribution
and factors have a clear probabilistic meaning in terms of
mixture component distributions.
ij

As mentioned in the introduction, the key idea of LSA[1] is to map documents (and by symmetry terms) to avector space of reduced dimensionality, the latent semantic space. This mapping is computed by decomposing theterm/document matrix N with SVD, N = UV , where
U and V are orthogonal matrices U U = V V = I andthe diagonal matrix  contains the singular values of N.
The LSA approximation of N is computed by thresholding
all but the largest K singular values in  to zero (= ~ ),which is rank K optimal in the sense of the L2 -matrix norm
as is well-known from linear algebra, i.e., one obtains the~ = UV~  UV = N. Note that theapproximation N~ to
L2 {norm approximation does not prohibit entries of N
be negative.
tt

4.2 Geometry of the Aspect Model

Now consider the class-conditional multinomial distributions P (jz ) over the vocabulary in the aspect model which
can be represented as points on the M , 1 dimensional simplex of all possible multinomials. Via its convex hull, this setof K points de nes a K , 1 dimensional sub-simplex. The
modeling assumption expressed by (2) is that all conditional
distributions P (jd) are approximated by a multinomial representable as a convex combination of the class-conditionals
P (jz ). In this geometrical view, the mixing weights P (z jd)
correspond exactly to the coordinates of a document in that
sub-simplex. A simple sketch of the geometry is shown in
Figure 1. This demonstrates that despite of the discreteness of the latent variables introduced in the aspect model,
a continuous latent space is obtained within the space of all
multinomial distributions. Since the dimensionality of the
sub-simplex is K , 1 as opposed to M , 1 for the complete

ACM SIGIR Forum

k

j;k

k

k

i;k

k

t

t

t

P(w|z 1)0

4 Probabilistic Latent Semantic Analysis
4.1 Latent Semantic Analysist+ P(w|d)4.4 Kullback{Leibler Projection vs. Orthogonal
Projection

Returning to the geometrical view of the aspect model assketched in Figure 1, it is interesting to reveal the projection principle which is implicitly used in the aspect model.

213

Vol. 51 No. 2, July 2017

\plane"\space shuttle"

plane
airportcrash
ight
safetyaircraftair
passengerboard
airlinespace
shuttlemissionastronauts
launchstationcrew
nasa
satellite
earth\family"homefamily
like
love
kids
motherlife
happy
friends
cnn\Hollywood"
lm
movie
music
new
best
hollywood
love
actor
entertainment
star

Table 1: Four factors from a 128 factor decomposition of
the TDT-1 corpus. Factor are represented by their 10 mostprobable words, i.e., the words are ordered according to
P (w jz ).
\Bosnia"\Iraq"\Rwanda"\Kobe"

uniraq
bosnianiraqi
serbs
sanctionsbosniakuwaitserb
unsarajevo
councilnato
gulf
peacekeepers saddam
nations
baghdadpeace
husseinrefugeesaid
rwandareliefpeople
camps
zaire
camp
food
rwandanbuilding
citypeoplerescuebuildings
workerskobe
victimsarea
earthquake

L=

X

"

()

n d

d2D

X n(d; w)

w 2W#( ) log P (wjd) + log P (d) : (10)

1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

1

1

1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

1

1

1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

1

1

1

1

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

aid
food
medicalpeople
un
war

aid
food
medicalpeople
un
war

aid
food
medicalpeople
un
war

aid
food
medicalpeopleunwar

iments with the TDT-1 collection, which contains 15,862
documents of broadcast news stories [8].1 Stop words havebeen eliminated by a standard stop word list, no stemmingor further preprocessing has been performed. Table 1 showsa reduced representation of 4 factors from a 128 factor solution.
The rst two factors have been selected as the ones
with the highest probablity to generate the word \ ight",
the last two factors have the highest probability to generate the word \love". It is interesting to see that the rst
two factors indeed capture two di erent types of usage forthe term \ ight": ights with planes and ights with space
ships/shuttles. Similarly the last two factors capture two
distinguishable contexts in which the word \love" occurs inn d

The rst term in brackets corresponds to the negative
Kullback{Leibler (KL) divergence (or cross{entropy) between the empirical distribution of words in a document^ (wjd)  n(d; w)=n(d) and the model distribution P (wjd).
P
For xed factors P (wjz ) maximizing the log{likelihood
w.r.t the mixing proportions P (z jd) thus amounts to projecting P^ (wjd) on the subspace spanned by the factors basedon the KL{divergence. This is very di erent from any typeof squared deviation which would result in an orthogonal
projection (cf. [10] for more details on the geometry of statistical models).

4.5 Factor Representation: An Example

1 Since the TDT-1 collection contains documents on topics andevents most readers will be familiar with, this collection has been
preferred over the test collections utilized in Section 6.

In order to visualize the factor solution found by PLSA we
present an elucidating example. We have performed exper-

ACM SIGIR Forum

1

Figure 2: Folding in a query conisting of the terms \aid",\food", \medical", \people", \UN", and \war": evolutionof posterior probabilities and the mixing proportions P (z jq)(rightmost column in each bar plot) for the four factors depicted in Table 2 after 1 ( rst row), 2 (second row), 3 (thirdrow), and 20 (fourth row) iterations.

Table 2: Four additional factors from the 128 factor decomposition of the TDT-1 corpus (cf. Table 1).
Rewriting the log{likelihood in (3) one arrives at1214Vol. 51 No. 2, July 2017

MED
CRAN
CACM
CISI
precision improvement precision improvement precision improvement precision improvement
cos+tfLSI
PLSI-U
PLSI-Q
PLSI-U
PLSI-Q
cos+t df
LSI
PLSI-U
PLSI-Q
PLSI-U
PLSI-Q

44.3
51.7
63.1
63.9

+16.7
+42.4
+44.2

29.9
28.7
32.8
35.1
33.3



-4.0
+9.7
+17.4
+11.4

17.9
16.0
19.2
22.9
19.5



-11.6
+7.2
+27.9
+8.9

12.7
12.7
14.0
18.8
14.7

0:0
+10.2
+48.0
+15.7

67.5

+52.4
+49.7

37.5

+25.4

26.8

+49.7

20.1

+58.3

49.0
64.6
69.5
63.2

+31.8
+41.8
+29.0

35.2
38.7
38.9
38.6

+9.9
+10.5
+9.7

21.9
23.8
25.3
26.6
27.6

+8.7
+15.5
+21.5
+26.0

20.2
21.9
23.3
23.1

+8.4
+15.3
+14.4

66.3

72.1
66.3

+47.1
+35.3

40.4
40.1

+14.8
+13.9

28.3

+29.2

24.6
24.4

+21.8
+20.8

Table 3: Average precision results and relative improvement w.r.t. the baseline method (cos+tf and cos+t df, respectively)
for the 4 standard test collections. Compared are LSI, PLSI, and the two PLSI variants (PLSI-U, PLSI-Q) as well as results
obtained by combining PLSI models (PLSI-U and PLSI-Q , respectively). An asterix for LSI indicates that no performancegain could be achieved over the baseline, the result at 256 dimensions with a 1 : 2 combination with the baseline score isreported in this case.
drastically increases the probability that this particular occurrence of \UN" is related to the events in Rwanda. The
same mechanism is able to detect \true" polysems [6].

the TDT-1 collection: real love in the context of family life
as opposed to staged love in the sense of \Hollywood".

4.6 Folding-In Queries

5 Probabilistic Latent Semantic Indexing
5.1 Vector-Space Models and LSI

Folding-in refers to the problem of computing a representation for a document or query that was not contained in
the original training collection. In the LSA approach, this issimply done by a linear mapping that e ectively represents
a document or query by the center of its constituent terms(with an appropriate term weighting) [1]. In PLSA, mixing proportions can be computed by EM iteration, where
the factors are xed such that only the mixing proportions
P (zjq ) are adapted in each M{step.
Table 2 shows some more factors for the TDT-1 collection which clearly re ect the vocabulary dealing with certain
events: the war in Bosnia and Iraq, the crisis in Rwanda,
and the earthquake in Kobe. Based on this four factors,
we have computed a representation for a test query consisting of the terms \aid", \food", \medical", \people", \UN",and \war". Figure 2 visualizes the evolution of the posterior
probabilities and the mixing proportions in the course of the
EM procedure. The query has been designed such that only
the \Rwanda" factor is matching all query terms (e.g., theUN was not involved in the Kobe earthquake, there was no
medical aid provided for the Iraq during the Gulf war, etc.).
As can be seen this factor has indeed the highest weight after the rst iteration, but notice that the other factors stillaccount for more than half of the probability. However thischanges after some EM iterations, since the aspect model
introduces feedback between the terms. For example, although a term like \UN" would by itself be best explainedby the \Bosnia" factor, the context of the other query terms

ACM SIGIR ForumOne of the most popular families of information retrieval
techniques is based on the Vector{Space Model (VSM) for
documents [14]. A VSM variant is characterized by threeingredients: (i) a transformation function (also called local
term weight), (ii) a term weighting scheme (also called global
term weight), and (iii) a similarity measure. In our experiments we have utilized (i) a representation based on the (untransformed) term frequencies (tf) n(d; w) which has beencombined with (ii) the popular inverse document frequency(idf) term weights, and the (iii) standard cosine matching
function. The same representation applies to queries q suchthat the matching function for the baseline methods can be
written as
w n^ (d; w)^n(q; w)s(d; q ) =
;
(11)
2
2
w n^ (d; w)
w n^ (q; w)

pP

P

pP

where n^ (d; w) = idf(w)  n(d; w) are the weighted word frequencies.
In latent semantic indexing, the original vector space representation of documents is replaced by a representation in
the low{dimensional latent space and the similarity is computed based on that representation. Queries or documents
which were not part of the original collection can be folded
in by a simple matrix multiplication (cf. [1] for details). In

215

Vol. 51 No. 2, July 2017

100

80

70

70

MED, tf

60

60

CRAN, tf

CACM, tf

precision [%]

60

50

40

50

60

40
40

30
30

40

30

20

20

20
20
cos−tfLSI
PLSI−U*cos−tfLSI
PLSI−Q*10

050
recall [%]

100

100

0
0

50
recall [%]

100

80

50
recall [%]100

CACM, tfidf

50

100

40

50

60

50
recall [%]CISI, tfidf

50

80
60

0

60

60

CRAN, tfidf

cos−tfLSI
PLSI−Q*

0
0

70

70

MED, tfidf

10

cos−tfLSI
PLSI−Q*

10

00

precision [%]CISI, tf

50

80

40
40

30
30

40

30

20

20

20
20
cos−tfidf
LSI
PLSI−U*cos−tfidf
LSI
PLSI−U*

10

0

0
0

50
recall [%]

100

10

cos−tfidf
LSI
PLSI−Q*

10
0

0

50
recall [%]100

cos−tfidf
LSI
PLSI−U*0050
recall [%]

100

0

50
recall [%]100

Figure 3: Precision{recall curves for the 4 test collections with idf term weighting (lower row) and without (upper row).
Depicted are curves for direct term matching, LSI, and the best performing PLSI variant.
version of the empirical distribution P^ (wjd) = n(d; w)=n(d).
We propose to utilize P (wjd) (thought of as a document vector) in order to compute a matching score between a document an a query. Notice that P (wjd) is a representation in
the original (word) space obtained by back{projection fromthe probabilistic latent space. The vector P (jd) can (optionally) be weighted with the inverse document frequencies and compared with the (weighted) query by the cosine.2 We have considered two ways of combining PLSAU with the standard VSM: (i) by linearly combining thecosine similarities as discussed above for LSI, and (ii) byadditively combining the multinomials like in interpolation
methods for language modeling, i.e., by using the representation P~ (wjd) = P^ (wjd) + (1 , )P (wjd). Both methods
have empirically shown almost identical performance and wewill only report results of variant (i), because this scheme
has also been used in the case of LSI.

our experiments, we have actually considered linear combinations of the original similarity score (11) (weight ) andthe one derived from the latent space representation (weight
1 , ), as suggested in [3] (cf. [16] for a more detailed empirical investigation of linear combination schemes for information retrieval systems).

5.2 Variants of Probabilistic Latent Semantic Indexing

Two di erent schemes to exploit PLSA for indexing havebeen investigated: (i) as a context{dependent unigram
model to smoothen the empirical word distributions in documents (PLSI-U), (ii) as a latent space model which provides
a low{dimensional document/query representation (PLSIQ):

PLSI-U For each document d in the collection, PLSA provides a multinomial distribution P (wjd) over the vocabulary

2 Folding{in queries, though possible, has empirically shown no advantages in the PLSI-U scheme.

as given by (2). This distribution will in general be a smooth

ACM SIGIR Forum

216

Vol. 51 No. 2, July 2017

1000
950
900
850
800
Perplexity
750
700
0.7

0.75

0.8

0.85

0.9

0.95

1

0.9

0.95

1

5
10%

30%

0

50%
90%
−5
0.7

0.75

0.8

70%
0.85

Figure 4: Model performance K = 256 on the Cran eld collection in terms of perplexity (upper plot) and precision (lower plot,
absolute gain vs. baseline for di erent recall levels) at di erent values of . The model has been annealed (0:7 ! ! 1:0)
and trained up to convergence; no early stopping was performed.

6 Experimental Results

PLSI-Q In this scheme we use the low{dimensional representation P (z jd) and P (z jq) to evaluate similarities. Therefore, queries have to be folded in, which is done by xing the
P (wjz ) parameters and calculating weights P (z jq) by TEM.

The performance of PLSI has been systematically comparedwith the standard term matching method based on the raw
term frequencies (tf) and their combination with the inverse
document frequencies (t df), as well as with LSI. We have
utilized the following four medium{sized standard document
collection: (i) MED (1033 document abstracts from the National Library of Medicine), (ii) CRAN (1400 document abstracts on aeronautics from the Cran eld Institute of Technology), (iii) CACM (3204 abstracts from the CACM journal), and (iv) CISI (1460 abstracts in library science fromthe Institute for Scienti c Information). The condensed results in terms of average precision recall (at the 9 recall
levels 10% , 90%) are summarized in Table 3. A selectionof average precision recall curves can be found in Figure 3.
Here are some details of the experimental setup: PLSA
models at K = 32; 48; 64; 80; 128 have been trained by
TEM for each data set with 10% held{out data. For PLSIU/PLSI-Q we report the best result obtained by any of thesemodels, for LSI we report the best result obtained for the
optimal dimension (exploring 32{512 dimensions at a step
size of 8). The combination weight  with the cosine baseline
score has been coarsely optimized by hand, MED, CRAN:
 = 1=2, CACM, CISI:  = 2=3; in general slightly smaller
weights have been utilized for the combined models.
The experiments consistently validate the advantages of
PLSI over LSI. Substantial performance gains have beenachieved for all 4 data sets and both term weighting schemes.
In particular, PLSI-Q/PLSI-Q work particularly well on
the raw term frequencies, where LSI on the other hand
may even fail completely (in accordance with the results

How to optimally take into account (global) term weights
in PLSI-Q is an only partially resolved problem. We haveused the ad hoc approach to reweight the di erent modelcomponents by the quantities w P (wjz )  idf(w), but thismay not make optimal use of the term weight priors.
One advantage of using statistical models vs. SVD techniques is that it allows us to systematically combine di erent models. While this should optimally be done according to a Bayesian model combination scheme, we have utilized a much simpler approach in our experiments which has
nevertheless shown excellent performance and robustness.
In the PLSI-U we have combined the probability estimates
P (wjd) for models with di erent number of components K
additively with uniform weights. In the PLSI-Q scheme,
we have simply combined the cosine scores of all models
with a uniform weight. The resulting methods are referred
to as PLSI-U and PLSI-Q , respectively. Empirically wehave found the performance to be very robust w.r.t. di erent (non-uniform) weights and also w.r.t. the {weight usedin combination with the original cosine score. This is dueto the noise reducing bene ts of (model) averaging. Noticethat LSA representations for di erent K form a nested sequence, which is not true for the statistical models which
are expected to capture a larger variety of reasonable decompositions.

P

ACM SIGIR Forum

217

Vol. 51 No. 2, July 2017

Indexing by latent semantic analysis. Journal of theAmerican Society for Information Science (1990).

reported in [1]). We explain this by the fact that large frequencies dominate the squared error deviation used in SVD
and a dampening (e.g., by idf weighting) is necessary to geta reasonable decomposition of the term/document matrix.
Since PLSI-Q can not take much advantage from the term
weighting scheme, PLSI-U/PLSI-U performs slightly better in this case. We suspect that even better results could
be achieved by an improved integration of term weights in
PLSI-Q. The bene ts of model combination are also very
substantial. In all cases the (uniformly) combined model
performed better than the best single model. As a sighte ect, model averaging also deliberates from selecting the\optimal" model dimensionality.
In terms of computational complexity, despite of the iterative nature of EM, the computing time for TEM model
tting at K = 128 was roughly comparable to SVD in astandard implementation. For larger data sets one may also
consider speeding up TEM by on-line learning [11]. Noticethat the PLSI-Q scheme has the advantage that documents
can be represented in a low{dimensional vector space (asin LSI), while PLSI-U requires the calculation of the high{dimensional multinomials P (wjd) which o ers advantages interms of the space requirements for the indexing informationthat has to be stored.
Finally, we have also performed an experiment to stressthe importance of tempered EM over standard EM{basedmodel tting. Figure 4 plots the performance of a 128 factor model trained on CRAN in terms of perplexity and interms of precision as a function of . It can be seen that
it is crucial to control the generalization performance of themodel, since the precision is inversely correlated with theperplexity. In particular, notice that the model obtained by
maximum likelihood estimation (at = 1) actually deteriorates the retrieval performance.

[2] Dempster, A., Laird, N., and Rubin, D. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Statist. Soc. B 39 (1977), 1{38.
[3] Dumais, S. T. Latent semantic indexing (lsi): Trec-3
report. In Proceedings of the Text REtrieval Conference (TREC-3) (1995), D. Harman, Ed., pp. 219{30.
[4] Gildea, D., and Hofmann, T. Topic-based language models using em. In Proceedings of the 6th Eu-

ropean Conference on Speech Communication andTechnology(EUROSPEECH) (1999).

[5] Hofmann, T. Latent class models for collaborative
ltering. In Proceedings of the 16th International

Joint Conference on Arti cial Intelligence (IJCAI)

Acknowledgment(1999).
[6] Hofmann, T. Probabilistic latent semantic analysis.
In Proceedings of the 15th Conference on Uncertainty in AI (1999).
[7] Hofmann, T., Puzicha, J., and Jordan, M. I.
Unsupervised learning from dyadic data. In Advancesin Neural Information Processing Systems (1999),
vol. 11.
[8] Linguistic Data Consortium. TDT pilot study
corpus. Catalog no. LDC98T25, 1998.
[9] McLachlan, G., and Basford, K. E. Mixture
Models. Marcel Dekker, INC, New York Basel, 1988.
[10] Murray, M. K., and Rice, J. W. Di erential geometry and statistics. No. 48 in Monographs
on statistics and applied probability. Chapman & Hal,
1993.
[11] Neal, R., and Hinton, G. A view of the EM algorithm that justi es incremental and other variants.
In Learning in Graphical Models, M. Jordan, Ed.
Kluwer Academic Publishers, 1998, pp. 355{368.
[12] Pereira, F., Tishby, N., and Lee, L. Distributional clustering of english words. In Proceedings of
the ACL (1993), pp. 183{190.
[13] Rose, K., Gurewitz, E., and Fox, G. A deterministic annealing approach to clustering. Pattern
Recognition Letters 11, 11 (1990), 589{594.
[14] Salton, G., and McGill, M. J. Introduction toModern Information Retrieval. McGraw{Hill, 1983.
[15] Saul, L., and Pereira, F. Aggregate and mixed{
order Markov models for statistical language processing. In Proceedings of the 2nd International Con-

References[16] Vogt, C. C., and Cottrell, G. W. Predicting
the performance of linearly combined IR systems. In

7 Conclusion and Outlook
We have presented a novel method for automated indexing
based on a statistical latent class model. This approach has
important theoretical advantages over standard LSI, since it
is based on the likelihood principle, de nes a generative data
model, and directly minimizes word perplexity. It can also
take advantage of statistical standard methods for model
tting, over tting control, and model combination. The
empirical evaluation has clearly con rmed the bene ts ofProbabilistic Latent Semantic Indexing which achieves signi cant gains in precision over both, standard term matching and LSI. Further investigation is needed to take full advantage of the prior information provided by term weighting
schemes. Recent work has also shown that the bene ts of
PLSA extend beyond document indexing and that a similar
approach can be utilized, e.g., for language modeling [4] andcollaborative ltering [5].

ference on Empirical Methods in Natural Language
Processing (1997).

This work has been supported by a DAAD postdoctoral fellowship.

Proceedings of the 21st ACM-SIGIR International
Conference on Research and Development in Information Retrieval, Melbourne, Australia (1998).

[1] Deerwester, S., Dumais, S. T., Furnas,
G. W., Landauer, T. K., and Harshman, R.

ACM SIGIR Forum218

Vol. 51 No. 2, July 2017