Discovering Topics With Neural Topic Models Built From PLSA Loss
Authors: sileye ba
Venue:  (2020)
Keywords: neural network
topic model
neural topic model
bag-of-words
PLSA

Abstract: In this paper we present a model for unsupervised topic discovery in texts corpora. The proposed model uses documents, words, and topics lookup table embedding as neural network model parameters to build probabilities of words given topics, and probabilities of topics given documents. These probabilities are used to recover by marginalization probabilities of words given documents. For very large corpora where the number of documents can be in the order of billions, using a neural auto-encoder based document embedding is more scalable then using a lookup table embedding as classically done. We thus extended the lookup based document embedding model to continuous auto-encoder based model. Our models are trained using probabilistic latent semantic analysis (PLSA) assumptions. We evaluated our models on six datasets with a rich variety of contents. Conducted experiments demonstrate that the proposed neural topic models are very effective in capturing relevant topics. Furthermore, considering perplexity metric, conducted evaluation benchmarks show that our topic models outperform latent Dirichlet allocation (LDA) model which is classically used to address topic discovery tasks.

Under review as a conference paper at ICLR 2020
DISCOVERING TOPICS WITH NEURAL TOPIC MODELS BUILT FROM PLSA Loss
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we present a model for unsupervised topic discovery in texts corpora. The proposed model uses documents, words, and topics lookup table embedding as neural network model parameters to build probabilities of words given topics, and probabilities of topics given documents. These prohabilities are used to re- cover by marginalization probabilities of words given documents. For very large corpora where the number of documents can be in the order of billions, using neural auto-encoder based document embedding is more scalable then using lookup table embedding as classically done. We thus extended the lookup based document embedding model to continuous auto-encoder based model. Our mod- els are trained using probabilistic latent semantic analysis (PLSA) assumptions. We evaluated our models six datasets with rich variety of contents. Con- ducted experiments demonstrate that the proposed neural topic models are very effective in capturing relevant topics. Furthermore, considering perplexity metric. conducted evaluation benchmarks show that our topic models outperform latent Dirichlet allocation (LDA) model which is classically used address topic dis- covery tasks.
INTRODUCTION
Nowadays, with the digital era, electronic text corpora are ubiquitous. These corpora can be com- pany emails, news groups articles, online journal articles, Wikipedia articles, video metadata (titles, descriptions, tags). These corpora can be very large, thus requiring automatic analysis methods that are investigated by the researchers working on text content analysis (Collobert al. 201 Cambria & White, 2014). Investigated methods are about named entity recognition, text classification, etc
(Nadeau & Sekine 2007 S. 2002).
An important problem in text analysis is about structuring texts corpora around topics (Daud et al., 2010 Liu Zhang 2012). Developed tools would allow to summarize very large amount of text documents into limited, human understandable, number of topics. In computer science many definitions of the concept of topic can encountered. Two definitions are very popular. The first one defines topic an entity of knowledge graph such as Freebase Wikidata (Bollacker al.) 2008 Vrandežie & Krötzsch, 2014). The second one defines a topic as probability distribution over words of a given vocabulary (Hofmann) |000] Blei et al., 2003). When topics are represented as knowledge graph entities, documents can be associated to identified concepts with very precise meaning. The main drawback that knowledge graphs in general composed of very large number of entities. For example, in 2019, Wikidata counts about 40 million entities. Automatically identifying these entities requires building extreme classifiers trained with expensive labelled data (Puurula et al.) 2014, Liu et al. 2017). When topics are defined as probability distribution over words of vocabulary, they can be identified using unsupervised methods that automatically extract them from text corpora. A precursor of such methods is the latent semantic analysis (LSA) model which is based on the word-document co-occurrence counts matrix factorization (Dumais) 1990).
Since then, LSA has been extended to various probabilistic based models (Hofmann) 2001 Blei
et al. 2003), and more recently to neural network based models (Salakhutdinov & Hinton 2009
Larochelle & Lauly 2012 Wan al. 2011 Yao et al.] 2017 Dieng et al. 2017))
In this paper, we propose a novel neural network based model to automatically, in an unsupervised fashion, discover topics in text corpus. The first variation of the model is based neural

---

Under review as a conference paper at ICLR 2020
networks that uses as input or parameters documents, words, and topics discrete lookup table em- bedding to represent probabilities of words given documents, probabilities of words given topics, and probabilities of topics given documents. However because in a given corpus, the number of documents can be very large, discrete lookup table embedding explicitly associating to each docu- ment given embedded vector be unpractical. For example, for the case of online stores such as Amazon, or video platforms such as Dailymotion or Youtube, the number of documents are in the order of billions. To overcome this limitation, we propose model that generates continuous document embedding using a neural auto-encoder (Kingma & Welling 2013). Our neural topic models are trained using cross entropy loss exploiting probabilistic latent semantic analysis (PLSA) assumptions stating that given topics, words and documents can be considered independent.
The proposed models are evaluated on six datasets: KOS, NIPS, NYtimes, TewetyNewsGrroup.
Wikipedia English 2012, and Dailymotion English. The four first datasets are classically used to benchmark topic models based on bag-of-word representation (Dua & Graff 2017). Wikipedia, and Dailymotion are large scale datasets counting about one million documents. These latter datasets are used to qualitatively assess how our models behave on large scale datasets. Conducted experiments demonstrate that the proposed models effective in discovering latent topics. Furthermore, eval- uation results show that our models achieve lower perplexity than latent Dirichlet allocation (LDA) trained on the same datasets.
The remainder of this paper is organized as follows. Section 2 discusses related work. Section briefly presents principles of topics generation with PLSA. S the first version of the model we propose which is based on discrete topics, documents, and words embedding. Section 5 gives details about the second version of the model which is based on embedding documents using a continuous neural auto-encoder. Section 6 provides details about the experiments conducted to assess the effectiveness of the proposed models. Finally Section derives conclusions and gives future research directions.
2 RELATED WORK
Unsupervised text analysis with methods related latent semantic analysis (LSA) has long re- search history. Latent semantic analysis takes high dimensional text vector representation and apply linear dimensionality reduction methods such as singular value decomposition (SVD) to the word-document counts matrix (Dumais 1990). The main drawback of LSA is related to it's lack of statistical foundations limiting the model interpretability.
Probabilistic latent semantic analysis (PLSA) was proposed by Hofmann 2001 to ground LSA on solid statistical foundations. PLSA is based on a well defined generative model for text generation based on the bag-of-words assumption. PLSA can be interpreted as probabilistic matrix factori- sation the word-document counts matrix. Because PLSA model defines a probabilistic mixture model, it's parameters can be estimated using the classical Expectation Dnprectatioo-Maxmizzztiii (EM) algo- rithm (Moon) 1996). PLSA has been exploited in many applications related to text modelling by Hofmann (001), to collaborative filtering by PPopescul al. (2001). , to web links analysis by Cohn & Hofmann (2001), and to visual scene classification by Quelhas et al.) (2007).
The main drawback of PLSA is that it is generative model of the training data. It does not apply to unseen data. To extend PLSA to unseen data, Blei et al. (2003) proposed the latent Dirichlet Alloca- tion (LDA) which models documents via hidden Dirichlet random variables specifying probabilities on a lower dimensional hidden spaces. The distribution over words of an unseen document is a continuous mixture over document space and a discrete mixture over all possible topics. Modeling with LDA has been thoroughly investigated resulting in dynamic topic models to account for topics temporal dynamics by Blei Lafferty| (2006); Wang et al. (2008); [Shalit et af. y Varadarajan et al. (2013); Farrahi & Gatica-Perez] (2014), hierarchical topic models to account topic hierarchical (Frrrah
structures by Blei et al. (2004). and multi-lingua, topic model to account for multi-lingual corpora by Boyd-Grabber & Blei 22099); Vulic al. (2015), and supervised topic model to account for cor- pora composed by categorized documents (Blei & McAuliffe 2008). Beside text modelling, LDA has been applied to discover people's 'ss socio- -eoorrphgc routines from mobiles phones data by |aa- rahi & Gatica-Perez (2010) 2011 2014), mining recurrent activities from long term videos logs by
Varadarajan et al. (213).
2

---

Under review as a conference paper at ICLR 2020
Learning topic models based LSA, PLSA or LDA requires considering jointly all words, doe- uments, and topics. This is a strong limitation when the vocabulary and the number of documents very large. For example, for PLSA or LDA, learning the model requires maintaining large ma- trix containing the probability of topics given words and documents (Hofmann) [2001] Blei et al.)
2003). To overcome this limitation Hoffman et al. (20T0) proposed online training of LDA models using stochastic variational inference.
Recently, with the rise of deep learning with neural networks that are trained using stochastic gra- dient descent on sample batches, novel topic models based on neural networks have been proposed. Salakhutdinov & Hinton (2009) proposed a two layer restricted Boltzmann machine (RBM) called the replicated-soofmaax to extract low level latent topics from a large collection of unstructured doc- uments. The model is trained using the contrastive divergence formalism proposed by Carreira- Perpiñán & Hinton (2005). Benchmarking the model performance against LDA showed improve- ment in term on unseen documents' perplexity and accuracy on retrieval tasks. [Larochelle & Lauly 2012) proposed neural auto-regressive topic model inspired from the replicated softmax model but replacing the RBM model with neural auto-regressive distribution estimator (NADE) which is generative model over vectors of binary observations (Larochelle & Murray 2011). An advan- tage of the NADE over the RBM is that during training, unlike for the RBM, computing the data negative log- nogllikelihoo''s gradient with respect to the model parameters does not requires Monte Carlo approximation. (Srivastava et al. (2013) generalized the replicated softmax model proposed by Salakhutdinov & Hinton (2009) deep RBM which has more representation power.
Cao et al. 2015) proposed neural topic model (NTM), and it's supervised extension (sNTM) where words and documents embedding are combined. This work goes beyond the bag-of-words represen- lation by embedding word n-grams with word2vec embedding as proposed by Mikolov et al. (2013).
Moody (2166) proposed the Ida2vec, model combining Dirichlet topic model Blei et al. (2003)
and word embedding as Mikolov et al. (2013). The goal of Ida2vec is to embed both words and documents the same space in order to learn both representations simultaneously.
Other interesting works combine probabilistic topic models such as LDA, and neural network mod- elling (Wan et al.] [011] Yao et al., 2017 Dieng et al. (017). Wan et al. (201T) proposed hybrid model combining neural network and latent topic model. The neural network provides lower dimensional embedding of the input data while the topic model extracts further structure from the neural network output features. The proposed model was validated computer vision tasks. Yao et al. (2017) proposed to integrate knowledge graph embedding into probabilistic topic modelling by using observation for the probabilistic topic model document-level word counts and knowledge graph entities embedded into vector forms. Dieng et al. (2077) integrated to a recurrent neural net- work based language model global word semantic information extracted using probabilistic topic model.
3 TOPIC MODELLING WITH PROBABILISTIC LATENT SEMANTIC ANALYSIS
Probabilistic latent semantic analysis (PLSA) proposed by Hofmann (200T) is based on the bag-of- words representation defined in the following.
3.1 BAG OF WORDS REPRESENTATION
The grounding assumption of the bag-of-word representation is, that for text content representation, only word occurrences matter. Word orders can be ignored without harm to understanding.
Let us assume available a corpus of documents 'D = {doc1, doc2, .... doc;. doc,}. Ev- ery document is represented as the occurrence count of words of given vocabulary W (word,, wordg, word,,... wordx). Let us denote by c(word,, doc,) the occurrence count of the n tht vocabulary word into the v'th document. The normalized bag-of-words representation of the i'th document is given by the empirical word occurrences probabilities:
frs e(word,, doc;)
1,..,,N. N.
(1)
Em=1 c(wordm. doc;)"
With the Bug-of-wordd assumption, fni. is an empirical approximation of the probability that word, appears in document doc; denoted p( wowdd, /doc.).
3

---

Under review as a conference paper at ICLR 2020
p(ssordi/doc)
y
plaard, (sbns)
Jai
y
pisonddy does)
(a) Document embedding with lookup table.
(b) Embedding with an auto-encoder.
2(topi (dfoc)
pitops (doo.)
Z
x(topaldoe))
y
p(topw)docs)
6p1
#(topu
p(topx (dacs)
(c) Probabilitics topics given document.
(d) Probabilitics of words given topic.
Figure 1: Discrete and continuous neural topic model. Neural network biases omitted for figure clarity.
3.2 PROBABILISTIC LATENT SEMANTIC ANALYSIS
Prohabilistic latent semantic analysis (PISAA) modelling is based the assumption that there is set unobserved topics T = (top,- top2.-.- topx } that explains occurrences words documents. Given topics, words and documents can be assumed independent. Thus, under the PLSA assumption the probability of the occurrence a word word,, in a document doc, can be decomposed as:
p(word, doc,)
pl
doc.,)
(2)
Hofmann 2001) used the expectation maximization algorithm (Moon 1996) to estimate probabili- ties of words given topics p( (word,, topp.), ). and probabilities of topics given documents p(top,, |do;;).
It is important note that PLSA, well as LDA, are trained the counts matrix (c(wordm: doc;)) and not the normalized counts matrix (fri). The normalized counts matrix are used by the models we propose the following sections.
DISCRETE NEURAL TOPIC MODEL
The discrete neural topic model propose is based on a neural network representation of probabili- ties involved in representing the occurrences of words in documents; p( word,,/do;)) p( (word,,, ttop.). ) and p(top,, /oc;). These probabilities are parametrized by the documents, the words, and the topics discrete lookup table embeddings.
Lets us denote by x; (za)) D-dimensional embedded vector representing the i'th document doc;, Similarly, we define yn (yd)) and Z (=ax) D-dimensional embedded vectors respectively representing word word,, and topic top,- Using these discrete lookup embeddings as parameters, the probability of words given documents can be written as:
exp(yI.x, + b,,)
pl ((word,/doc)
(3)
Σ D==1 exp(yIxx; + 6m)
Similarly, probabilities of words given topic are defined as:
pl p(word,.loop.) = exp(yIIze + b,))
(4)
m== exp(y + bm)
x; is column vector in RD
4

---

Under review as a conference paper at ICLR 2020
and the probability of topic given a document is defined
exp(z + bk)
p(top (doc.)
(5)
2i=1 exp(z/ x4 + bi)
In Equations 3 and in following equations, although different, all neural networks biases are denoted by We used this convention to avoid burdening the reader with too many notations.
Figure Ta Figure lb and c give schematic representation of the neural network representation of probabilities of words given documents, probabilities of words given topics, and topics given doc- uments. It is worth noticing that, because of the form of the probability of words given documents (see Equation
( which based on sealar product between word and document vectors, the higher probability of word occurring in document, the closer it's vector yn to the document's vector x; will be. Similar analysis can be derived about the proximity of word and topic vectors, and topic and document vectors.
The probabilities of words given topics (4), and topics given documents ( can be combined accord- ing the PLSA assumptions (Equation 2 recover probabilities of the words given documents
exp(y bn.)
exp(z X; bb)
p(word,]doc;
2= explymnzs bm)
.
exp (2) x; b1)
To train the model, we optimize, using stochastic gradient descent, according to a cross entropy loss, embedding and biases parameters that probabilities of words given documents in Equations and 6 match the empirical bag-of- of-wrddssrr frequencies defined Equation
5 CONTINUOUS NEURAL TOPIC MODEL
The discrete neural topic model described in Section 4 has two main drawbacks. First, it only models training data, and can not be applied to unseen data, Second, requires building an explicit vector representation X; for every document i = 1,2, .... J. In practise the number of documents can be very large. possibly in the order of billions. solution to these issues to use continuous embeddings to represent document instead of discrete lookup table embeddings (Vincent et al. 2010).
Continuous document embeddings are built using neural auto-encoder representation that maps input documents doc; represented by their word empirical frequencies fns onto themselves through a D-dimensional bottleneck layer x, (za)) which is then taken as the document embedding.
This is done
fni + bai
exp Ed= Ddn de + ban)
fns
(8)
N Em=n exp(El= Dam + bam)
where a is the rectified linear (ReLU) unit activation function. Variables y (yan), and ý (jan)
neural network parameters, and is taken to be word embeddings.
Figure [b] gives a schematic visualization the continuous document embedding model. Because of it's continuous embeddings, this model can encode an unlimited number of documents as far as embedding dimension D is large enough.
Similarly then for the discrete topic model, documents, words, and topics vector representation x, and Yn. and topics vectors 2ž are combined to compute probabilities words given topics using Equation 4 probabilities of topics given documents using Equation 5 and probabilities of words given documents using Equation
To train the continuous neural topic model optimized, using stochastic gradient descent, with respect to cross entropy loss, parameters x,, yn and Z₁ such that the models in Equations and match the empirical bag-of-words frequencies in Equation
5

---

Under review as a conference paper at ICLR 2020
Table 1: Evaluation corpora statistics: is the corpus size, and N the vocabulary size.
Datasets
NIPS
KOS
NYTimes
20NewsGroup
Wikipedia
Dailymotion 1500000
I
1500
3430
300000
18282
900000
N
12419
6906
102660
24164
50000
50000
It has to be noticed, apart from biases, our models parameters are constituted by the embedding parameters. This allows build model with limited of parameters, exploiting parameters sharing as regularization procedure. For the auto-encoder model in Equations 7 we chose different encoding (y) and decoding (ý) parameters avoid constraiingg the model. However, if further reduction of the model number of parameters is targeted, these two variables can be considered as the transposed of one another.
6 EXPERIMENTS
6.1 EVALUATION PROTOCOL
We evaluated our models on six datasets, Four of them are classical dataset used to evaluate bag-of- words models: NIPS full papers, KOS blog entries, NYTimes news articles, and the Twenty News Group dataset. The three first datasets can be obtained from the UCI machine learning repository created by Dua Graff (2017). The Twenty News Group dataset part of datasets available with the well kown Python Scikit-Learn package.
The two other datasets are the Wikipedia English 2012, and the Dailymotion English used assess qualitatively how our models perform datasets with very large number of documents. Apart from the Dailymotion dataset, all other ones are publicly available, and can be used for model benchmarking. Table [ gives the corpora's statistics. These corpora are very diverse in term of corpus sizes, vocabulary sizes, and document contents.
We evaluate the discrete neural topic model (D-NTM) presented in Section 4 and it's continuous extension (C-NTM) presented Section 5 These models are compared to the latent Dirichlet allocation (LDA) model, developed by Blei al. (2003), taken baseline. We considered this
baseline as it outperforms the PLSA models. We used the LDA implementation available in the Python Scikit-Learn package based on Hoffman al. (2010) implementation.
To assess the performances of the models, as proposed by Hofmann (2001): Blei et al. (2033 we
the perplexity measure defined
pp exp El logr(word, , ... word,,,.. word doc;)
Σ
Ed=1
N
where word word., wordy; is the sequence of possibly duplicated words composing the i'th document doc,, and
K
p(word: word,,, wordy,, doc;) = II Σ
ltop (doc:
(10)
The perplexity represents the exponential of data negative log-likelihood of estimated models. Thus, the smaller the better. This measure is classically used to assess language models, and topic models performances.
Our models comprise two hyper-parameters: the embedding dimension D, and the number of topics K. As we are optimizing our models using stochastic gradient descent, their training involves three parameters: a learning rate set to X 0.01, number of descent steps set to 100, and a batch size that was set to 64. Our models were implemented in the Tensorflow framework. Neural network parameters were initialized with Xavier initializers, and model optimization is performed with Adam Optimizers
Our model implementation and benchmarking scripts will he made available upon conference reviews.

---

Under review as a conference paper at ICLR 2020
Table Topic discovery benchmarking results. K the number of topics. For D-NTM and C-NTM results are presented for embedding dimensions of D = 100, 200, 300.
D-NTM
C-NTM
Dataset
K
LDA
100
200
300
100
200
300
50
1785.2
1786.1
1811.5
1957.3
1922.9
1893.4
2232.9
100
1601.7
1546.2
1514.5
1708.8
1606.4
1583.8
2197.6
KOS
200
1421.5
1283.7
1256.1
1511.5
1356.2
1292.7
2123.0
300
1322.2
1159.8
I112.5
1434.7
1246.0
1160.2
2126.6
50
3849.8
3902.3
3896.8
4328.1
4357.3
4428.8
4331.6
100
3713.9
3623.4
3634.7
4048.7
3964.5
3926.
4317.5
NIPS
200
3537.9
3357.1
3317.8
3747.8
3588.7
3574.5
4316.1
300
3458.2
3247.4
3086.9
3631.2
3520.0
3321.2
4302.3
50
4635.1
4611.1
4692.0
4646.0
4609.9
4494.9
4793.0
100
3989.3
3857.9
3849.1
4067.3
3815.5
3721.2
4511.7
20NewsGroup
200
3544.9
3273.0
3193.3
3577.5
3214.8
3023.1
4367.2
300
3258.7
2964.3
2813.4
3304.2
2858.2
2739.9
4342.5
6.2 RESULTS
We investigate neural topic models training performances for varying embedding dimension and number of topics K. We tested number of topics of K 50, 100, 200, 300, and embedding dimen- sions of D 100, 200,000000 300.
Table 2 gives training perplexity for the models the KOS, the NIPS, and the Twenty News Group datasets. These results show that the training perplexity decreases with the number topics until a number where it stagnates. Also, the training perplexity is higher when the embedding dimension is about a 100, while for 200 and 300, exhibit close perplexity values. This trend with perplexity de- creasing with increasing embedding dimension and number of topics is expected larger dimension implies higher neural network learning capacity.
Table gives the comparison of training perplexity between D-NTM, C-NTM, and LDA. These results show that training perplexity is much lower for neural network based topic model than for LDA. They also show that, in general, D-NTM is more efficient achieving low perplexity than C-NTM. For an embedding dimension D = 300, for the KOS and the NIPS dataset, the D-NTM model achieves better performances, while for the Twenty News Group, the C-NTM achieves better performances.
Figure 2 gives samples topics discovered using the continuous neural topic model (C-NTM) over large scale dataset: NYtimes, Wikipedia 2012, and Dailymotion. We only considered this model as it scales better than the discrete neural topic model (D-NTM) to large scale datasets. Discovered topics are displayed in form of word clouds where the size of each word is proportional to the probability the words occurs in the considered topic pl (oord, (top;). This figure shows that the about energy plants, medecine,
model find relevant topics. For NYtimes, the discovered topics and court law. For Wikipedia displayed topics are about books and novels, universities and schools, and new species. For Dailymotion, discovered topics about movies, videos productions, and Super Bowl. These qualitative results show that found topics are consistent and centered around concepts human being can identify and expect. These examples are just few sample topics, other non displayed topics are about news, sport, music, religion, science, economy, etc.
7 CONCLUSIONS
In this paper we presented novel neural topic model. The proposed model has two variations. The first variation based on discrete documents, words, and topics discrete lookup table embeddings. The second variation exploits continuous neural auto-encoder embedding to allow scaling to very large corpora. Proposed models are evaluated on six datasets. Conducted evaluation demonstrate that proposed models outperform LDA with respect to perplexity metric.

---

Under review as a conference paper at ICLR 2020
enee OF
Oy
Fisksyrtton Lrea tment
producer
RCO
BLNN COr UedER
ires
electricity
AFF
CORS
med
found Study
care e
SOOTCTION
a
FUP
Lawyer
gnt
hospital
Hear
per cent.
official
case
planiet
der
health th pein
TUU
W
power
compantes
wi
cance
Tedge
indus try
officill
ppttte enttenrrree
legal
hearing wedeereederaaaeee aaaiii
onnental
eoggnie conds fion
(a) NYtimes sample topics
editor
EROAY
ter
LLEB
award
S.o.iiie y un Velr-errsss eitystty..
ma
Leepaann t
cenus
(imall Eat DOOD
nove
art WOIS K
PUbLICE CUCBLLCREAO
Joor Lo
TH!
Teacline
rectraleyy
notiona
Campus
-faculltt Ins cuttinn
educc
Shhoo
FRO OHUP Joan member
profossor SEDE
Ca
coll rege-yyr year
tion
program
eseerrhhe
state
LO
cource
turic
connin geat
S
animal
omerica
fish
seee
ound
mBM
sma
Le
ofteR
DO
known
stay
st student
inst
institur
tute
science
human
tone
LikE
breedt
SO
Bublished
THE
degree
bird
writing Col
Tema
e
(b) Wikipedia sample topics
upscon
rector wrriistdiirect
films
group
OF dee
ehis ine 1
supe
BRDLY
Econe
role
simmons
ho
patr
neit offficn
ccm!
Torady
dream
mov
1
e
Timi
producer
author boom
tom
Follout
aud
audio
matt
marve wo
warss
S
aa
disney
MUSHE unA ver sa bill york br ian laet
sengs
home
is
pub
land dene GULLERF
sherreeordsa
film
peneng aand
hugh
starrirg
ccatt
mcrrdibbes
Clip
composer
Stevent rus test
Lils
se
(c) Dailymotion sample topics
Figure Samples topics discovered using C-NTM for large scale datasets.
The proposed model can be extend in many directions. The continuous document embedding model is based on a simple singl--hidden-layer auto-encoder. The use of more sophisticated models such as variational auto-encoders such as proposed by Kingma Welling (013)) could be investigated. In the direction of using more sophisticated neural networks, proposed probabilities of words given topics and topics given documents models could be represented with deeper neural networks which are known to have high representation power. This could lead to decisive improvements, specially for large scale corpora.
Another possible direction would be to integrate proposed neural topic models into models combin- ing probabilistic topic models and neural network model such as done by [Deng et al. (2017) who combines LDA to capture global words semantic to recurrent neural network language models. This would allow design a model into a single neural network framework. The model would be fully trainable with stochastic gradient descent on sample batches.
REFERENCES
D. M. Blei and J. D. Lafferty. Dynamic topic models. International Conference on Machine Learn- ing (ICML), pp. 113120, 2006.
D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research (JMLR), pp. 993-1022, 2003.
D.M. Blei and D. McAuliffe. Supervised topic models. Advances in Neural Information Process- ing Systems (NIPS), 2008
8

---

Under review as a conference paper at ICLR 2020
D.M. Blei, T. L. Griffiths, M.I. Jordan, and J. Tenenbaum. Hierarchical topic models and the nested chinese restaurant process. Advances in Neural Information Processing Systems (NIPS), 2004.
K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. Freebase: a collaborativeyy created graph database for structuring human knowledge. ACM SIGMOD international conference on Management of data, 2008
Boyd-Grabbed and D. M. Blei. Multilingual topic models for unaligned text. Conference on Uncertainty in Artificial Intelligence (UAI), 2009.
E. Cambria and B. White. Jumping nlp curves: review natural language processing research. IEEE Computational Intelligence Magazine, 2014.
Z. Cao, S Li, Y. Liu, W. Li, and H. Ji. A novel neural topic model and its supervised extension. AAAI Conference Artificial Intelligence, pp. 2210-2216, 2015.
M. Carreira-Perpiiie and G. E. Hinton. On contrastive divergence learning. International Confer- ence on Artificial Intelligence and Statisties(AASTA75)) 2005.
D. Cohn and T. Hofmann. The missing link-a probabilistic model of document content and hypertext connectivity. Advances in Neural Information Processing Systems, 13, 2001.
R. Collobert, Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12, 2011.
A. Daud, Li, Zhou, and Muhammad. Knowledge discovery through directed probabilistic topic models: a survey. Frontiers of computer science in China, 2010.
A.B. Dieng, C. Wang. J. gao, and J. Paisley. Topicrnn: A reccurent neural network with lon range semantic dependency. International Conference on Learning Representation, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. ittp:ttp://rccciiee. ics. wci.ccc.. adci.edu/ml
S. Dumais. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391-44) --09, 1990.
K. Farrahi and D. Geree.-Perez. Probabilistic mining of socio-geographic routines from mobile phone data. IEEE Journal of Selected Topics in Signal Processing, pp. 746-755, 2010.
K. Farrahi and D. Gatica-Perez. Discovering routines from large-scale human locations using prob- abilistic topic models. ACM Transactions on Intelligent Systems and Technology (TIST), 2011.
K. Farrahi and D. Geteca-Perez. A probabilistic approach to mining mobile phone data sequences. Personal and ubiquitous computing, pp. 223-238, 2014.
M. D. Hoffman, D. M. Blei, and F. Bach. Online learning for latent dirichlet allocation. NIPS Proceedings of the 23rd International Conference on Neural Information Processing Systems- Volume 1, 2010.
Thomas Hofmann. Unsupervised learning by probabilistic latent semantic analysis. Machine learn- ing, 42(1-2):177--16, 2001.
D. Kingma and Ma. Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
H. Larochelle and Lauly. Neural autoregressive topic model. Advances in Neural Information Processing Systems, pp. 1607-1614, 2012.
H. Larochelle and I. Murray. Neural autoregressive distribution estimator. International Conference on Artificial Intelligence and Statisties(AISTATS)) pp. 29-37, 2011.
B. Liu and I.. Zhang, A survey of opinion mining and sentiment analysis. Mining text data, 2012.

---

Under review as a conference paper at ICLR 2020
J. Liu, W.-C. Chang, Y. Wu, and Y. Yang. Deep learning for extreme multi-label text classification. International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017.
T. Mikolov, Sutskever, K. Chen, G. Corrado, and Dean. Distributed representations of words and phrases. Advances in Neural Information Processing Systems, pp. 3111-3119, 2013.
C.E. Moody. Mixing dirichlet topic models and word embeddings to make Ida2vec. arXiv: 02019 (Ls. 2016.
K. Moon. The expectation-maximizziii maximization algorithm. IEEE Signal processing magazine, 1996.
D. Nadeau and S. Sekine. A survey of named entity recognition and classification. Lingvisticae Investigationes, 2007.
A. Popescul, D.M. Penncock, and Lawrence. The missing link-a probabilistic model of document content and hypertext connectivity. Conference on Uncertainty in Artificial Intelligence (UAI), pp. 437-444, 2001.
A. Puurula, J. Read, and A. Bifet. Kaggle LSHTC4 winning solution. CoRR, abs/1405.0546 2014.
Quelhas, J.-M. Odobez, D. Gatica-Perez, and T. Tuytelaars. A thousand words scene. Trans- actions Pattern Analysis and Machine Intelligence (PAMI), 9:1575-1589, 2007.
Fabrizio S. Machine learning in automated text categorization. ACM computing surveys, 2002.
R.R. Salakhutdinov and E. Hinton. Replicated softmax: an undirected topic model. Advances in Neural Information Processing Systems, pp. 1607-1614, 2009.
U. Shalit, D. Weinshall, and G. Chechik. Modeling musical influence with topic models. Journal of Machine Learning Research (JMLR), 2013.
N. Srivastava, R. Salakhutdinov, and G. Hinton. Modelling documents with deep boltzmann ma- chine. Conference on Uncertainty Artificial Intelligence (UAI), pp. 616-624, 2013.
Varadarajan, Emonet, and J.-M. Odobez. sequential topic model for mining recurrent activi- ties from long term video logs. International Journal of Computer Vision, pp. 100-126, 2013.
P. Vincent, H. Larochelle, I. Lajoie, and Y. Bengio P.E. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 2010.
D. Vrandečić and M. Krötzsch. Wikidata: a free collaborative knowledge base. Communications of the ACM, 2014.
I. Vulic, W. De Smet, J. Tang, and M.-F. Moens. Probabilistic topic models in multilingual set- tings: an overview of its methodology and application. Elsevier Information Processing and Management, 2015.
L. Wan, L. zhu, and R. Fergus. A hybrid neural network topic model. International Conference Artificial Intelligence and Atatistiss(ASSTA7S)) pp. 1287-1294, 2011.
C. Wang, D. M. Blei, and J. D. Lafferty. Continuous time dynamic topic models. International Conference on Machine Learning (ICML), 2008.
L. Yao, Y. Zhang, B. Wei, Jin, Zhang, Y. Zhang, and Q. Chen. Incorporating knowledge graph embedding into topic modelling. AAAI Conference on Artificial Intelligence, pp. 3119-3126, 2017.
10


Reviews:
[{"review_id": "rJlIw4RqKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First, some minor issues.  I didn't understand equation (3).  It seems to be a variant of equation (4), and seems to be in disagreement with equation (6).  Might be better if the equation was just dropped.  For equation (9), you should have brackets \"()\" around the argument to the exp.\n\nSecond, in terms of comparisons, the paper lacks adequate related work.  Some non-parametric but non-neural\nmodels not implemented in GPUs substantially beat LDA, and will run on all the big data sets you list, though perhaps\nnot quickly!   There has also been a number of neural and hybrid topic models developed.  \nDocNADE and LLA (Zaheer etal), for instance, work very well in PPL.  Then there are many new deep topic models.  Some use the amortised inference that you adopt in section 5.    Some incorporate word embeddings or document metadata to\nfurther improve performance metrics.  Note some of the earlier ICLR/NeurIPS papers with deep models didn't\ndo extensive comparative empirical testing, so may not work well against DocNADE or more recent algorithms.\n\nIn terms of related work, topic models is a bit of a mine-field because there is a huge amount of work in\na huge number of venues, and few authors do a good job of covering related work.  What you have listed are mainly the\nolder works.  Recent work also includes Poisson Matrix Factorisation and its variants, as well as hierarchical\nvariants of LDA, much better than the 2004 paper you mention.\n\nTo do the coherence comparisons, easiest way is to use the Palmetto software.\nYou can also evaluate models by using them as features in a classification task.\n\nIt was interesting that you only did one layer for your networks, i.e.,  equations (4)-(6).  Why was this?\nI would have liked to have seen the impact of more layers. However, your model is remarkably simple \nso if it works well, that is good.\n\nAnyway, the experimental evaluation shows good results on all three datasets for your models, but its hard to be sure\nsince you only have one comparison, an old LDA, and nothing recent.  So promising work, but\nrelated work and experimental work need to be improved.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rJg_OocPcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am unimpressed with the quality of writing and presentation, to begin with. There are numerous grammatical errors and typos that make the paper a very difficult read. The presentation also follows an inequitable pattern where the backgrounds and related works are overemphasized and the actual contribution of the paper seems very limited. In its current form, this paper is not ready for publication in ICLR.\n\nThe idea of representing a document as an average of the embeddings of the words is a rather crude idea. Paragraph2vec and many of its derivatives have shown significant improvements with document modelling. The perplexity improvements are nice to have, but I would have liked to see the embeddings being applied to some supervised problems to assess their utilities. \n\nThere are quite a few computationally expensive normalization terms. I am curious to understand how these summations do not slow the training process down without further approximations. The authors may present some computational complexity measures to convince readers about the practical applications of the proposed models.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1lgvje0Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural topic model that aim to discover topics by minimizing a version of the PLSA loss. According to PLSA, a document is presented as a mixture of topics, while a topic is a probability distribution over words, with documents and words assumed independent given topics. Thanks to this assumption, each of these probability distributions (word|topic, topic|document, and word|document) can essentially be expressed as a matrix multiplication of the other two, and EM is usually adopted for the optimization. This paper proposes to embed these relationships in a neural network and then optimize the model using SGD.\n\nI believe the paper should be rejected because: 1) most aspects of this paper are a little dated 2) novelty is little 3) experimental section is very limited and unconvincing.\n\nTo elaborate on the experimental section:\n- Only LDA has been presented as baseline. There's plenty of neural topic models to compare against (you mentioned some in your related work section) but no comparison with any of those is presented. If the concern is their training time on large datasets, they should be at least presented as comparison for the smaller datasets. For the large datasets there's other approaches that would scale and should be presented as baselines: 1) train on a sample of the dataset 2) co-occurrence based topic methods on sliding windows of text are extremely fast (eg see \"A Biterm Topic Model\", \"A Practical Algorithm for Topic Modeling with Provable Guarantees\", and \"A Reduction for Efficient LDA Topic Reconstruction\" which could fit your scenario with large datasets where topics most likely have small overlap with each other and are almost separable by anchor words.)\n- Even regarding just LDA: what hyper-parameters \\alpha and \\beta did you set for LDA? Tuning \\beta to a small value might have an impact for large datasets.\n- Metrics: only perplexity is presented and metrics but it's well known that perplexity on its own is quite limited and often is not correlated to human judgment. Consider adding topic coherence measures as well.\n- The section on continuous document embeddings is confusing and the explanation should be improved and the formalism tightened.\n\n\nOther (did not impact the score):\n- Biases: you're adding biases to your probability estimation equations. This is not in line  with the PLSA assumption. What happens if no biases are used?\n\nThe paper has several typos and grammatical errors, e.g.:\n- page 2, L#1: networks -> network\n- page 4, sec 3.2: set unobserved -> set of unobserved\n- page 5, sec 5: pratise -> practice\n- several places: it's -> its\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a neural topic model with the goal of improving topic discovery with a PLSA loss. Reviewers point out major limitations including the following:

1) Empirical comparison is done only with LDA when there are many newer models that perform much better.
2) Related work section is incomplete, especially for the newer models.
3) Writing is unclear in many parts of the paper.

For these reasons, I recommend that the authors make major improvements to the paper before resubmitting to another venue.
Decision: Reject