Factorized Topic Models
Authors: Cheng Zhang, Carl Henrik Ek, Hedvig Kjellstrom
ICLR 2013

Abstract: In this paper we present a new type of latent topic model, which exploits supervision to produce a factorized representation of the observed data. The structured parameterization separately encodes variance that is shared between classes from variance that is private to each class by the introduction of a new prior. The approach allows for a more efficient inference and provides an intuitive interpretation of the data in terms of an informative signal together with structured noise. The factorized representation is shown to enhance inference performance for both image and text classification.

arXiv:1301.3461v7  [cs.LG]  23 Apr 2013
Factorized Topic Models
Cheng Zhang
CV AP/CAS, KTH
Stockholm, Sweden
chengz@kth.se
Carl Henrik Ek
CV AP/CAS, KTH
Stockholm, Sweden
chek@kth.se
Andreas Damianou
University of Shefﬁeld
Shefﬁeld, UK
andreas.damianou@sheffield.ac.uk
Hedvig Kjellstr¨om
CV AP/CAS, KTH
Stockholm, Sweden
hedvig@kth.se
Abstract
In this paper we present a modification to a latent topic mode l, which makes the
model exploit supervision to produce a factorized represen tation of the observed
data. The structured parameterization separately encodes variance that is shared
between classes from variance that is private to each class by the introduction of a
new prior over the topic space. The approach allows for a moreefficient inference
and provides an intuitive interpretation of the data in terms of an informative signal
together with structured noise. The factorized representation is shown to enhance
inference performance for image, text, and video classification.
1 Introduction
Representing data in terms of latent variables is an important tool in many applications. A generative
latent variable model provides a parameterization that enc odes the variations in the observed data,
relating them to an underlying representation, e.g., a set o f classes, using some kind of mapping. It
is important to note that any modeling task is inherently ill -conditioned as there exists an infinite
number of combinations of mappings and parameterizations t hat could have generated the data.
To that end, we choose different models, based on different a ssumptions and preferences that will
induce different representations, motivated by how well th ey fit the data and for what purpose we
wish to use the representation.
Inference in generative models meets difﬁculties if the var iations in the observed data are not rep-
resentative of the variations in the underlying state to be inferred. As an example, consider a visual
animal classiﬁer, trained with, e.g., SIFT [20] features ex tracted from training images of horses,
cows and cats with a variation of fur texture. The task is now to classify an image of a spotted horse.
Based on the features, which will mostly pick up the fur textu re, the classiﬁer will be unsure of the
class, since there are spotted horses, cows and cats in the tr aining data. The core of the problem
is that fur texture is a weak cue to animal class given this dat a: Horses, cows and cats can all be
red, spotted, brown, black and grey. Shape is on the other han d a strong cue to distinguish between
these classes. However, the visual features will mostly cap ture texture information – the shape in-
formation (signal) is “hidden” among the signiﬁcantly rich er texture information (structured noise)
making up the dominant part of the variation in the data.
In this paper we address this issue by explicitly factorizin g the data into a structured noise part,
whose variations are shared between all classes, and a signal part, whose variations are characteristic
of a certain class. For our purposes, it is very useful to thin k about data as composed of topics.
Probabilistic topic models [23, 11, 4, 2] model a data exampl e as a collection of words (in the case
of images, visual words), each sampled from a latent distribution oftopics. The topics can be thought
of as different aspects of the data – a topic model trained with the data in our animal example above
might model one topic for shape and another for fur texture, a nd a certain data instance is modeled
as a combination of a certain shape and a certain texture.
Our approach is to encourage the topics to assume either a very high correlatio n or a very low
correlation with class. The class can then be inferred using only the class-speciﬁc topics, while the
This research has been supported by the EU through TOMSY , IST-FP7- Collaborative Project-270436, and
the Swedish Research Council (VR).
1
shared topics are used toexplain away the aspects of the data that are not interesting to this particular
inference problem. We present a variant of a Latent Dirichlet Allocation (LDA) [2] model which is
able to model the signal and structured noise separately fro m the data. This new model is trained
using a factorizing prior, which partitions the topic spaceinto a private signal part and a shared noise
part. The model is described in Section 3.
Experiments in Section 4 show that the proposed model outper forms both the standard LDA and
a supervised variant, SLDA [3], on classification of images , text, and video. Furthermore, the
explicit noise model increases the sparsity of the topic rep resentation. This is encouraging for two
reasons: firstly, it indicates that the factorized LDA model is a better model of class compared to the
unrestricted LDA; enabling better performance on any infer ence or data synthesis task. Secondly,
it enables a more economical data representation in terms of storage and computation; crucial for
applications with very large data sets. The factorization method can be applied to other topic models
as well, and the sparse factorized topic representation is b eneficial not only for classification, as
shown here, but also for synthesis [5], ambiguity modeling [7], and domain transfer [21].
2 Related Work
In this section we will create a context for the model that we a re about to propose by relating it
to factorized latent variable models in general and topic mo dels in specific. Providing a complete
review of either is beyond the scope of this paper, why here wewill focus on only the most relevant
subset of work needed to motivate the model.
The motivation for learning a latent variable model is to exploit the structure of the new representa-
tion to perform tasks such as synthesis or prediction of novel data, or to ease an association task such
as classiﬁcation. For continuous observations, several cl assic algorithms such as Principal Compo-
nent Analysis (PCA) and Canonical Correlation Analysis (CCA) can be interpreted as latent variable
models [1, 16, 17, 27]. Another modeling scenario is when observations are provided in the form of
collections of discrete entities. An example is text data where a document consists of a collection of
words. One approach to encode such data is using a latent representation that groups words in terms
of topics. Several approaches for automatically learning topics from data have been suggested in the
literature. A first proposal of a generative topic model was Probabilistic Latent Semantic Indexing
(pLSI) [11]. The model represents each document as a mixture of topics. The next important devel-
opment in terms of a Bayesian version of pLSI by adding a prior to the mixture weights. This was
done by the adaptation of a Dirichlet layer and referred to as Latent Dirichlet Allocation (LDA) [4].
Central to the work presented in this paper is a specific late nt structure simultaneously proposed
by several authors [7, 13, 14, 18]. Given multiple observati on modalities of a single underlying
state, the purpose of these models is to learn a representati on that separately encodes the modality-
independent variance from the modality-dependent. The lat ent representation is factorized such
that the modality-independent and modality-dependent are encoded in separate subspaces [6]. This
factorization has an intuitive interpretation in that the private space encodes variations that exists in
only one modality and does therefore encode variations repr esenting the ambiguities between the
modalities [7].
In this paper we will exploit a similar type of factorization within a topic model, but instead of ex-
ploiting correlations between observation modalities, we employ a single observation modality and
a class label associated with each observation. In speciﬁc, our approach will encourage a factoriza-
tion relating to class, such that the topics will be split int o those encoding within-class variations
from those that encode between-class variations. Such a fac torization becomes interesting for in-
ferring the class label from unseen data; the class-shared t opics can be considered as representing
“structured noise” while only the private class topics contain the relevant for class inference.
However, it is not easy to directly transfer the above factor ization, formulated between modalities
and described for continuous data, to topic models, which ar e inherently discrete. Results have
been presented [12, 25, 28] for the case of two conditionally independent observation modalities,
addressing the image and text cross-modal multimedia retri eval problem with topic representation.
In [12] a model that can be seen as a Markov random field of LDA t opic models is presented.
The topic distribution of each topic model affect the underlying topic spaces of other topic models,
connected to that model through the Markov random field. Fur ther, in [25] CCA is applied to the
topic space of the text data, which in turn has been learned fr om LDA and the image feature space.
2
α θ
c
z w
βπ
C N
M
K
(a) LDA with class label [8]
α θ
c
z w
βπθ
N
M
KM
(b) Our factorized LDA
Figure 1: Graphic representation of LDA structures. The notation in (b) is adopted from Jia et al. [12].
LDA and CCA are used as two separate steps. Differently, [28] instead use a Hierarchical Dirichlet
Process (HDP) based method which has a complexity selectionproperty. It takes the topics that only
describe variance in only one modality as the private space,which explains away the information that
cannot be matched between different modalities. This is an e xtension of [22] to multi-modalities,
hence it can not be generalized to other topic models, such as LDA or pLSI and it can not be used
to model the private and shared information with only one modality.
Differently from [12, 28, 25], which need to model the shared topics and private topics in the joint
topic space across different observation modalities, our factorization takes place over one modality
across different classes, where the structured noise is mod eled in the class-shared topics and the
signal is modeled in class-private topics. Furthermore, an d importantly, our approach is ﬂexible
and can be easily transferred to any type of topic model. Our c hoice of LDA stems from the fact
that it has previously been successfully applied for a large range of data and has desirable sparsity
properties that makes for an efficient model.
Topic models, and the LDA model in specific, are motivated by the benefit of representations that
are sparse in terms of the distribution of topics for each doc ument. In addition to this, the model
we are about to present aims to encourage a specific structur e of the topic themselves. This notion
is not new and have been proposed by several other authors. In [9] the topics are represented as
combinations of a small number of latent components as such leading to a more compact model. In
[29] the each topic is constrained by the words in the vocabul ary. However, none of these models
aim to learn a topic structure that is related to class.
3 Model
As described in the introduction, we add factorization to a m odel that describes variations of data
in terms of a set of latent topics. We seek a structured representation that encodes topics containing
within-class, or class private, variations separately fro m those containing variations that are shared
between the classes. We apply our factorization framework t o an adaptation of LDA, which in-
corporates additional class information to recover such a f actorized latent space. In this section,
the traditional LDA model [4] is first revisited, followed b y the description of our factorized topic
model.
3.1 LDA Revisited
Formally a document w consist of a collection of words w = [w1, . . . , w N ] from a vocabulary
indexed by {1, . . . , V }. Within a topic model each document of N words is described as a mixture
of K topics such that each word is associated with a specific topi c: z = [z1, . . . , z N ], where zn ∈
{1, . . . , K }. The mixture is defined as
p(w|z, β ) =
N∏
n=1
K∑
k=1
p(wn|zn, β k) (1)
where β k is the distribution over the vocabulary for topic k. The novelty, and the reason for the
success, of the LDA model is how the topicsz and the topic vocabularyβ are constructed within the
framework. The underpinning intuition is that the topics sh ould present a compact representation
with K ≪ N, and that the structure of the topics should be sparse such to achieve a robust and
interpretable model. Assuming the topics z to be governed by a multinomial distribution, z ∼
Multi(z|θ), sparsity can be achieved by choosing the parameters θ as governed by a Dirichlet
distribution, θ ∼ Dir(θ|α ). By the same motivation a Dirichlet prior is placed over the t opic-
vocabulary distributionβ ∼ Dir(β |π). As the Dirichlet is conjugate to the multinomial distribution,
3
the marginal likelihood can be reached analytically by comb ining the likelihood with the prior and
performing the integration,
p(w|α, π ) =
∫
p(θ|α )
( N∏
n=1
[∑
zn
p(zn|θ)p(wn|zn, β zn )
]
p(β |π)
)
dθ (2)
from which the parameters of the model can be learned.
One way of incorporating class information within the LDA framework was suggested in [8] where
the use of a class dependent topic distribution was proposed . This was implemented by using the
class variable c as a “switch”; p(θ|α , c ) = ∏ C
j=1 Dir(θ|α j )δcj where δij is the Kronecker delta
function. Using this model the class can be inferred for a new document w∗ through a maximum
likelihood procedure ˆc = arg maxc p(w∗ |α , π, c ) [8].
In this paper we take inspiration from the work presented in [8]. However, we choose to incorporate
the class information in a slightly different manner. In spe cific, we use a factorizing prior over the
topic distribution, which firstly encourages sparsity, and secondly introduces a preference for a class
conditioned structure, such that separate topics encode wi thin-class variations and between-class
variations in the data. Thus, the model we will propose have a stronger class dependency compared
to [8]. We will now proceed to describe and motivate the relevance of this class dependency.
3.2 Factorized Topic Model
As motivated in Section 1, our idea is to separate the topic sp ace into two parts, where the class-
private part explains the class-dependent information (signal) and the shared part explains the class-
independent information (structured noise). To achieve this we introduce an additional prior p(θ) to
the model presented in [8]. This will encourage a factorized structure such that the K topics can be
“softly” split into Kp class-private topics andKs shared topics where Kp +Ks = K. The advantage
of such a structured topic space is that it will be more compact than a regular model; all aspects of the
data that correlate with class will be pushed into the class- private part of the topic space. Since the
other, class-shared, part of the topic space will then only contain noise, the class of a new document
w∗ will in effect be inferred using only the class-private part . Further, in our model, we will use
the same sparsity prior α over the topics for all classes. This removes the additional ﬂexibility of
allowing a different topic sparsity for each class – which can be relevant in certain special cases —
but the gain is a more robust model with fewer free parameters, requiring less training data.
In the following, let θclass be the topic distributions of all classes, obtained by marginalizing θ over
class. Its rows are defined as θclass
c ∝ ∑ M
m=1 θmδcmc, where cm indicates the class label of the
mth document and δij the Kronecker delta function. Examples of θclass distributions can be seen in
Figures 3, 4, 5, and 6.
Intuitively, the private topics would concentrate to a cert ain class in θclass, while the shared top-
ics would be more spread among all classes (more uniformly di stributed over a column in θclass).
Information entropy, widely used in different fields [19, 2 4], provides a good measurement of this
property. In this case, we employ an entropy-like measure H(k) over class for each topic k:
H(k) = 1
log(1/C )
C∑
c=1
θclass
c,k
∑ C
ξ=1 θclass
ξ,k
log(
θclass
c,k
∑ C
ξ=1 θclass
ξ,k
) (3)
where θclass
c,k is the element in row c and column k of θclass. H(k) ∈ [0, 1], 0 if all the probability in
the topic k is concentrated to one class, 1 if all classes are equally probable to contain the topic k.
To split the topics into a private and shared part, we wish the prior p(θ|κ) to encourage topics k to
either have a low H(k) (be very class-specific) or high H(k) (be very class-unspecific). Hence, we
introduce a function as:
A(k) =H(k)2 − H(k) + 1. (4)
The prior is defined as:
p(θ) ∝
K∏
k=1
A(k) . (5)
This prior thus treats each column of θclass independently. With the additional prior (Figure 1(b)),
the generative model becomes:
4
p(w|α, π, c ) =
∫
p(θ|α, c )p(θ)
( N∏
n=1
[∑
zn
p(zn|θ)p(wn|zn, β )
]
p(β |π)
)
dθ . (6)
Learning. We use Gibbs sampling for learning the parameters of the mode l, more speciﬁcally,
collapsed Gibbs sampling [10] in the same manner as [12]. The factorizing prior presents itself in
the learning as an additional factor in the objective functi on over z, compared to the original LDA
model. It should be noted that the factorizing prior in Equat ion 5 is independent of the type of
learning procedure – the model in Equation 6 can also be trained using, e.g., a variational method.
When training the model, the topics are initialized randoml y, which means that they all have a H
close to 1. During Gibbs sampling, it would be very unlikely t o find a topic with low H, given
the bimodality of A in Equation (4). To address this problem, we introduce an “au to-annealing”
procedure, where A is replaced with a dynamic cooling function starting off by e ncouraging low
H only, and gradually encouraging high H more and more, as the average H decreases (i.e., when
some topics have found a class-specific state). Hence, A is changed to a dynamic function
A(k) =H(k)2 − 2 ˆHH (k) + 1 (7)
where the average H, ˆH = ∑ K
k=1 H(k)/K , is used as an annealing parameter in the function. As
with other annealing procedures, the “auto-annealing” pro cedure means that the factorizing prior
p(θ) changes in each step of the iterative learning procedure. In a normal annealing procedure, this
change would be actuated by changing the annealing paramete r. Here, ˆH can be thought of as an
autonomous annealing parameter since it converges automat ically to a value reﬂecting the fraction
of the class-dependent versus class-independent variation in the data. For example, the text data set
(Figure 4) has a lower ˆH than the natural scene dataset (Figure 5).
Segmenting the topic space. When the model have been trained we can evaluate the structure of
the learned topic space by computing H(k) for each topic k. We consider topics with low H as
class-dependent while topics with high H are considered as independent. As such the topic space
can be “softly” segmented and interpreted in a class conditioned manner. As an example, the words
building up the shared topics can be considered as stop words. In text processing, there is usually
some standard stop words list, which can be used to pre-process the text. However, these stop words
are predefined, for example, “the”, “at” etc. However, they sometimes also provide class-relevant
information, for example, some topics are more location dependent or have more nouns. On the other
hand, there are words, like “learning”, “performance” etc, which do not carry much information in,
say, a machine learning conference corpus. In our model, we automatically learn the real stop words
for the given domain. Furthermore, while it is easy to predef ine the stop words in text data, this
problem becomes much more challenging in computer vision applications. The “stop-visual-words”
are ill-defined and much less intuitive to find, why an algor ithm which automatically learns them,
such as the one we propose, is very beneficial. We would like t o emphasize again that there is still
only one topic space; no hard splitting or removal of topics i s done, neither for learning, nor for
inference.
4 Experiments
The proposed model is evaluated on four different classiﬁcation tasks, and compared to two baselines
consisting of a regular LDA model with class label [8], and a model with stronger class-supervision
in the topic learning, SLDA [3].
4.1 Object Classification
We first demonstrate how the factorization works using a toy dataset. The dataset, shown in Fig-
ure 2, is constructed to have a very high degree of structured noise. There are four object classes:
bulb, car, duck, and mug. All 8 instances of a certain class ha ve the same shape and image loca-
tion. However, there is a very high intra-class variability in foreground and background texture.
Furthermore, all four classes contain the exact same foregr ound-background texture combinations.
Thus, the texture (which will dominate the variation among f eatures from any visual extractor) can
be regarded as structured noise, while the true signal relates to shape. The properties of this dataset
can also be found to some extent in natural images: most reali stic image and object classes display
large intra-class appearance variation, and different classes share appearance aspects. Furthermore,
5
Figure 2: All the instances in the toy object dataset.
(a) θclass with regular LDA
 (b) θclass with SLDA
 (c) θclass with factorized LDA
Figure 3: Toy object dataset. (a) Regular LDA topic distribution marginalized over class θclass, topics sorted
in ascending order of class-specificity. (b) SLDA topic distribution marginalized over classθclass, topics sorted
in ascending order of class-specificity. (c) Factorized LD A topic distribution marginalized over class θclass,
topics sorted in ascending order of class-specificity, red line indicating partition between θp and θs.
the backgrounds in natural scenes are often complex and vary ing, introducing even more variation
among training data for a class.
SIFT features on two different scales are densely extracted from all images, and a 64-word vocabu-
lary is learned in which all SIFT features are represented. Thus, each image is represented by a bag
of visual words in this vocabulary.
The experiment is performed in a hold-one-out manner, whereeach image in turn is classified using
a model trained on the other 31 images. In the following, we will by “regular LDA” mean the regular
LDA with upstream supervision presented in [8], but trained using Gibbs sampling in the same way
as our model, with the same value ofα for all documents. With ”SLDA”, we mean the more strongly
supervised LDA variant with downstream supervision presented in [3], implemented by Blei et al.
Our proposed factorized LDA, as well as regular LDA and SLDA, are trained with 15 topics, α =
0. 1 and π = 0. 2. The classification performance for each class is found by a veraging over the
performances for the 8 images of that class.
It should be noted that the test image always will have a textu re that is different from the training
images of that class. However, the same texture can be found i n other classes. A classifier that
tries to explain all variation in the data in terms of class va riation will therefore have difficulties
in modeling this data set; a regular LDA or SLDA model trained with this data will be forced to
represent texture as well as shape in the same topics, since t he Dirichlet prior will promote topic
sparsity. Thus, very few topics will purely represent one class, as shown in Figures 3(a) and 3(b).
However, our model, which explicitly factorizes the topics into those private to a certain class and
those shared between all classes, will allow the relevant shape variation to be represented separately
from the texture variation, which will just confuse the classification in this case. Figure 3(c) shows
the factorized topic distribution; it is clear that the topics in θp are private to a certain class, while the
noise topics in θs are shared equally over all classes; all the structured nois e has thus been pushed
into θs. Thus, even though the full topic space is used for classific ation, it is effectively only based
on θp, while the shared topics θs (right of the red line in Figure 3(c)) are effectively disregarded in
the classification since they appear with equal probability in all classes.
As expected, the explicit noise model greatly improves classification on this dataset: the factorized
LDA reaches 81.25%, while a regular LDA reaches a classifica tion rate of 34.38%, only slightly
6
!"#
"$%&'
'!$(
)$!*(
*(+'$',+
-ï./
,0*1
+$!&'
2 3 4 5 6 7 8 9 : 2; 22 23 24 25 26 27 28 29 2: 3;
(a) θclass with regular LDA
 (b) θclass with SLDA
!"#
"$%&'
'!$(
)$!*(
*(+'$',+
-ï./
,0*1
+$!&'
2 3 4 5 6 7 8 9 : 2; 22 23 24 25 26 27 28 29 2: 3;
(c) θclass with factorized LDA
Figure 4: Reuters 21578 R8 dataset. (a) Regular LDA topic distribution marginalized over classθclass, topics
sorted in ascending order of class-specificity. (b) SLDA topic distribution marginalized over classθclass, topics
sorted in ascending order of class-specificity. (c) Factor ized LDA topic distribution marginalized over class
θclass, topics sorted in ascending order of class-specificity, red line indicating partition between θp and θs.
(a) θclass with regular LDA
 (b) θclass with SLDA
 (c) θclass with factorized LDA
Figure 5: Natural scene dataset. (a) Regular LDA topic distribution m arginalized over class θclass, topics
sorted in ascending order of class-specificity. (b) SLDA topic distribution marginalized over classθclass, topics
sorted in ascending order of class-specificity. (c) Factor ized LDA topic distribution marginalized over class
θclass, topics sorted in ascending order of class-specificity, red line indicating partition between θp and θs.
above chance, and SLDA who is forced by the stronger supervision to represent all variation (where
texture is dominating) in terms of class achieves a result of 0% since the texture of the test image is
not present in the training data of the same class.
4.2 Text Classification
We now evaluate the proposed model in a realistic text classi ﬁcation scenario. We use the stan-
dard R8 training and testing set from the Reuters 21578 datas et [26], which contains 5485 training
documents and 2189 testing documents. The all-terms versio n of the data is used since we want to
illustrate how our model deals with noise.
The regular LDA, SLDA and factorized LDA models are trained w ith 20 topics, and parameter
settings α = 0. 5 and π = 0. 1.The topic distributions are shown in Figures 4(a), 4(b), an d 4(c).
The factorized class-private topic distribution θp (left of the red line in Figure 4(c)) is noticeably
cleaner than the regular distribution θclass (Figure 4(a)). In the factorized LDA, only θp contributes
to the classification, while the shared topics θs (right of the red line in Figure 4(c)) are effectively
disregarded since they appear with equal probability in all classes. The topics of the SLDA model
are sparser (Figure 4(b)), but all topics are forced to be class-specific by the stronger supervision.
There is a significant classification improvement using the factorized topic space, from 74.63% with
regular LDA and 63.75% with SLDA to 83.91% with factorized LDA.
4.3 Scene Classification
We also evaluate the proposed model on a challenging natural scene dataset used in [8]. There are
four classes: forest, mountain, open country and coast, with 100 training images and 50 test images
per class. From each image, SIFT features on two different scales are densely extracted, and labeled
according to a 192-word vocabulary learned from the features, as in [8].
The regular LDA, SLDA, and factorized LDA models are trained with 20 topics, and parameter
settings α = 0. 5 and π = 0. 1.Figures 5(a), 5(b), and 5(c) show the respective topic dist ributions;
notably, the class-specific topic space θp effectively used for classiﬁcation in our factorized LDA
only contains 8 topics, while 12 topics ( θs) are devoted to modeling structured noise. Thus, the
factorized representation is notably sparser than a regula r LDA representation, which gives the op-
portunity to save both storage space and computation time during classiﬁcation – an important factor
to take into account for large datasets.
In addition to rendering a notably sparser data representat ion, the factorized LDA reaches a
marginally higher performance rate than with a regular LDA a nd SLDA: 84.50% for our model
compared to 80.50% for the regular LDA and 84.00% for SLDA. Al l performances are slightly
better than the original implementation of the regular LDA [8], which reaches 76.0%.
7
(a) θclass with regular LDA
 (b) θclass with SLDA
 (c) θclass with factorized LDA
Figure 6: Action dataset. (a) Regular LDA topic distribution margina lized over class θclass, topics sorted in
ascending order of class-specificity. (b) SLDA topic distr ibution marginalized over class θclass, topics sorted
in ascending order of class-specificity. (c) Factorized LD A topic distribution marginalized over class θclass,
topics sorted in ascending order of class-specificity, red line indicating partition between θp and θs.
4.4 Action Classification
We proceed to evaluate the methods on a dataset with more vari ation independent of class. The
dataset consists of three actions from the KTH Action dataset [15]: boxing, handclapping and hand-
waving. There are 100 short video sequences of each action, w hich show 25 different people per-
forming the action, recorded in four shooting conditions (zooming and panning of camera, different
background ). The shooting condition has large inﬂuence on t he motion in the video, as each zoom
or panning motion adds global motion to the video and backgro unds contribute to the motion fea-
tures as well. However, the variation in shooting condition is not at all correlated with action class
in the dataset. Just as in the toy experiment above (but now in a more realistic setting), a large pro-
portion of the data variation is thus independent of the acti on class. Due to the low signal-to-noise
ratio, a topic model without factorization will have difﬁculties capturing the aspects of data relevant
for discriminating activity class.
The experiment was performed by separating out from the training data all the 25 images of an action
filmed with a certain shooting condition. The topic models w ere then trained with all other data,
and evaluated with the 25 removed images. Hence, the certain combination of action and camera
condition in the test data was not present in the training dat a. This was done for all actions in turn,
and the result was averaged over actions.
STIP features [15] were extracted from all sequences and clustered into a vocabulary of 128 spatio-
temporal words. This representation was used to train the re gular LDA, SLDA and factorized LDA
models with 10 topics, α = 0. 1 and π = 0. 1.
Figure 6 shows the topic distributions corresponding to the se three models. We can see that Fac-
torized LDA is able to model the class-dependent informatio n (left of the red line) and the class-
independent information ( right of the red line), which makes it be able to archive better performance
in noisy data. For the regular LDA, although the topics are no t shared, however, it models all the
information and assigned that to different classes with newtopics which made the topics themselves
became noisy. So does SLDA which models the ”noise” as the useful topics.
Factorized LDA gives an accuracy of 65.22%, which is far better than both regular LDA, 38%, and
SLDA, 51.33%. This confirms that the findings of the toy expe riment above applies to realistic
settings as well. Confusion matrices are shown in Figures 6(a), 6(b), and 6(c) respectively.
5 Conclusions
We present a factorized latent topic model, which explicitl y represents aspects of the data which
are not correlated with model state. Specifically, we train an LDA class model with an additional
factorizing prior, which encourages topics to either be ver y class-specific or evenly shared among
classes. The topic space θ is thus partitioned into one part θp whose topics are private to certain
classes, and another part θs with topics shared between classes. Only θp contributes effectively to
classification.
Experiments show the factorized LDA model to give consisten tly better classiﬁcation performance
and sparser topic representations than both a regular LDA mo del [8] and SLDA [3]. Sparse repre-
sentations are advantageous for large datasets since they s ave storage space and computation time
during classiﬁcation.
Future work includes investigating the effect of this facto rization prior on other topic models, such
as HDP, and to integrate the prior into models with multiple data views, such as in [12, 28, 30].
8
References
[1] F. R. Bach and M. I. Jordan. A probabilistic interpretati on of canonical correlation analysis. Technical
report, 2005.
[2] D. M. Blei. Probabilistic topic models. Communications of the ACM, 55(4):77–84, 2012.
[3] D. M. Blei and J. D. McAuliffe. Supervised topic models, arxiv:1003.0783, 2010.
[4] D. M. Blei, A. Y . Ng, and M. I. Jordan. Latent Dirichlet all ocation. Journal of Machine Learning
Research, 3:993–1022, 2003.
[5] A. Damianou, C. H. Ek, M. Titsias, and N. D. Lawrence. Mani fold Relevance Determination. In ICML,
2012.
[6] C. H. Ek. Shared Gaussian Process Latent Variable Models, PhD Thesis, 2009.
[7] C. H. Ek, J. Rihan, P. H. S. Torr, G. Rogez, and N. D. Lawrenc e. Ambiguity modeling in latent spaces.
Machine Learning for Multimodal Interaction, 2008.
[8] L. Fei-Fei and P. Perona. A Bayesian hierarchical model f or learning natural scene categories. In CVPR,
2005.
[9] M. R. Gormley, M. Dredze, B. Van Durme, and J. Eisner. Shar ed components topic models. In NAACL,
2012.
[10] G. Heinrich. Parameter estimation for text analysis. T echnical report, 2005.
[11] T. Hofmann. Probabilistic latent semantic analysis. I n UAI, 1999.
[12] Y . Jia, M. Salzmann, and T. Darrell. Learning cross-mod ality similarity for multinomial data. In ICCV,
2011.
[13] A. Klami and S. Kaski. Generative models that discover d ependencies between data sets. In IEEE
Workshop on Machine Learning for Signal Processing, 2006.
[14] A. Klami and S. Kaski. Probabilistic approach to detect ing dependencies between data sets. Neurocom-
puting, 72(1-3):39–46, 2008.
[15] I. Laptev. On space-time interest points. IJCV, 64(2/3):107–123, 2005.
[16] N.D. Lawrence. Probabilistic non-linear principal co mponent analysis with Gaussian process latent vari-
able models. Journal of Machine Learning Research, 6:1783–1816, 2005.
[17] G. Leen and C. Fyfe. A Gaussian process latent variable m odel formulation of canonical correlation
analysis. In European Symposium on Artiﬁcial Neural Networks, 2006.
[18] G. Leen and C. Fyfe. Learning shared and separate featur es from two related data sets using GPLVM’s.
In Learning from Multiple Sources Workshop, Neural Information Processing, 2008.
[19] B. Leibe, A. Leonardis, and B. Schiele. Robust object de tection with interleaved categorization and
segmentation. IJCV, 77(1):259–289, 2008.
[20] D. G. Lowe. Distinctive image features from scale-inva riant keypoints. IJCV, 60(2):91–110, 2004.
[21] R. Navaratnam, A. W. Fitzgibbon, and R. Cipolla. The joi nt manifold model for semi-supervised multi-
valued regression. In ICCV, 2007.
[22] J. Paisley, C. Wang, and D. Blei. The discrete inﬁnite lo gistic normal distribution. arXiv preprint
arXiv:1103.4789, 2011.
[23] C. H. Papadimitriou, P. Raghavan, and H. Tamaki. Latent semantic indexing: a probabilistic analysis.
Journal of Computer and System Sciences, (61):217–235, 2000.
[24] J. P.W. Pluim, J.B. A. Maintz, and M. A. Viergever. Inter polation artefacts in mutual information-based
image registration. CVIU, 77(2):1077–3142, 2000.
[25] N. Rasiwasia, J. Pereira, E. Coviello, G. Doyle, G. Lanc kriet, R. Levy, and N. Vasconcelos. A new
approach to cross-modal multimedia retrieval. In MMM, 2010.
[26] Reuters-21578. http://csmining.org/index.php/r52 -and-r8-of-reuters-21578.html.
[27] M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal
Statistical Society: Series B (Statistical Methodology), 61(3):611–622, 1999.
[28] S. Virtanen, Y . Jia, A. Klami, and T. Darrell. Factorized multi-modal topic model. In UAI, 2012.
[29] C. Wang and D. M. Blei. Decoupling sparsity and smoothness in the discrete hierarchical dirichlet process.
In NIPS, 2009.
[30] C. Wang, D. M. Blei, and L. Fei-Fei. Simultaneous image c lassiﬁcation and annotation. In CVPR, 2009.
9


Peer review:
[Review]:
We would like to thank the reviewers for their insightful comments about the paper. We will first provide general comments in response to issues raised by more than one reviewer, and then discuss each of the reviews in more detail.

From reading the reviews, we realize that the main contribution of the paper seems to have been obscured in the presentation - for example, due to a formulation in the beginning of the abstract (which now is changed). We do not propose a new topic model, but rather introduce a method for latent factorization in topic models. The method that we propose is general and can be adopted to many different topic models. 

Several tasks benefit from a factorized topic space; classification - the one we use to exemplify with in the paper - is just one. Factorized models produce interpretable latent spaces, which has been exploited in continous models for synthesis, as in [A], or for ambiguity modelling or domain transfer, as in [5] (Ek et al.), [B]. We believe the benefits of this transfer to topic models as well.

It would be very interesting to evaluate the benefit of a factorized topic space for a much larger range of topic models than what we do in this paper - this is beyond the scope of this paper but will definitely be pursued in a future journal version. 

In a revised version of the paper, which is now uploaded to ArXiv, we have however added results from the SLDA model of Blei and McAuliffe, as a second baseline in the experiments, as suggested by reviewers c82a and fda8. The factorized LDA consistently performs better than both the regular LDA and SLDA.

To stress the focus on factorization rather than a specific classification application, we have furthermore  added an experiment with video classification. Other changes, as described below, are also included in this new paper version.

New references (included in the new version):
[A]	A. C. Damianou, C. H. Ek, M. Titsias, and N. D. Lawrence, “Manifold Relevance Determination,” International Conference on Machine Learning, 2012.
[B]	R. Navaratnam, A. W. Fitzgibbon, and R. Cipolla, “The joint manifold model for semi-supervised multi-valued regression,” IEEE International Conference on Computer Vision, 2007.


Reviewer c82a:

We agree with reviewer c82a that we used the word entropy in a rather sloppy manner. We have strived to make the distinction clear in the revised version.

In Figure 1(b), theta in the main plate is connected with another theta outside, since we use all the topics in theta to compute the entropy-like information measure for each topic theta_m. In this, we adopt a graphical notation similar to [9] (Jia et al.).  This is explained more thoroughly in the revised version of the paper.

Moreover, p(theta | kappa) is proportional to F(k) in Equation (8). In the revised version of the paper, we explicitly state the form of the proposed prior.

Finally, as reviewer c82a clearly states, topic models do not produce state-of-the-art results for scene classification (however, they do produce state-of-the-art results in other domains, such as text). The motivation for using the current classification tasks is that we find that they provide a nice intuition into why one would want a factorized representation, which is able to model separately the 'important information' (class-dependent) and the 'unimportant information/noise' (class-independent). 


Reviewer 232f:

As reviewer 232f correctly states, the class-dependent and the class-independent topics jointly encode the variations in the data. The argument is not, as reviewer 232f suggests, to throw the class-independent topics away - they are important for explaining parts of the data variation. This is not suggested anywhere in the paper.  There are many motivations for learning a factorization. In the example application used in the paper, classification, the class dependent topics are the important ones. However, in a transfer learning scenario, the class-independent information is highly relevant. The manner in which factorization is used is highly application and domain specific; in this paper we exemplify one use for classification.

As reviewer 232f points out, using a feature that has been created for discriminative methods in a generative framework might not be particularly sensible. Our motivation for still taking this approach is to make a fair comparison to other topic models, for example, [6] (Fei-Fei and Perona).

We have replaced the term 'view' with 'modality' in the revised version of the paper, and also clarified the relation of our factorization method to the multi-modality methods cited in Section 2. In the literature on factorized latent variable models the word 'view' is predominantly used, but we think that 'modality' is clearer here.


Reviewer fda8:

As reviewer fda8 points out, we could achieve the same effect by using a beta distribution instead of A in Equation (7). However, it would still require a entropy-like measurement to steer the beta distribution so as to achieve the desired factorization. 

As described above, we have added results using SLDA, and show that the factorized LDA consistently performs better than both regular LDA and SLDA. However, we did not have time to implement other variants suggested by reviewer fda8 - this is definitely something which is interesting to do for a journal version.