Discovering Topics With Neural Topic Models Built From PLSA Loss

1 INTRODUCTION . Nowadays , with the digital era , electronic text corpora are ubiquitous . These corpora can be company emails , news groups articles , online journal articles , Wikipedia articles , video metadata ( titles , descriptions , tags ) . These corpora can be very large , thus requiring automatic analysis methods that are investigated by the researchers working on text content analysis ( Collobert et al. , 2011 ; Cambria & White , 2014 ) . Investigated methods are about named entity recognition , text classification , etc ( Nadeau & Sekine , 2007 ; S. , 2002 ) . An important problem in text analysis is about structuring texts corpora around topics ( Daud et al. , 2010 ; Liu & Zhang , 2012 ) . Developed tools would allow to summarize very large amount of text documents into a limited , human understandable , number of topics . In computer science many definitions of the concept of topic can be encountered . Two definitions are very popular . The first one defines a topic as an entity of a knowledge graph such as Freebase or Wikidata ( Bollacker et al. , 2008 ; Vrandečić & Krötzsch , 2014 ) . The second one defines a topic as probability distribution over words of a given vocabulary ( Hofmann , 2001 ; Blei et al. , 2003 ) . When topics are represented as knowledge graph entities , documents can be associated to identified concepts with very precise meaning . The main drawback is that knowledge graphs are in general composed of a very large number of entities . For example , in 2019 , Wikidata counts about 40 million entities . Automatically identifying these entities requires building extreme classifiers trained with expensive labelled data ( Puurula et al. , 2014 ; Liu et al. , 2017 ) . When topics are defined as probability distribution over words of vocabulary , they can be identified using unsupervised methods that automatically extract them from text corpora . A precursor of such methods is the latent semantic analysis ( LSA ) model which is based on the word-document co-occurrence counts matrix factorization ( Dumais , 1990 ) . Since then , LSA has been extended to various probabilistic based models ( Hofmann , 2001 ; Blei et al. , 2003 ) , and more recently to neural network based models ( Salakhutdinov & Hinton , 2009 ; Larochelle & Lauly , 2012 ; Wan et al. , 2011 ; Yao et al. , 2017 ; Dieng et al. , 2017 ) . In this paper , we propose a novel neural network based model to automatically , in an unsupervised fashion , discover topics in a text corpus . The first variation of the model is based on a neural networks that uses as input or parameters documents , words , and topics discrete lookup table embedding to represent probabilities of words given documents , probabilities of words given topics , and probabilities of topics given documents . However because in a given corpus , the number of documents can be very large , discrete lookup table embedding explicitly associating to each document a given embedded vector can be unpractical . For example , for the case of online stores such as Amazon , or video platforms such as Dailymotion or Youtube , the number of documents are in the order of billions . To overcome this limitation , we propose a model that generates continuous document embedding using a neural auto-encoder ( Kingma & Welling , 2013 ) . Our neural topic models are trained using cross entropy loss exploiting probabilistic latent semantic analysis ( PLSA ) assumptions stating that given topics , words and documents can be considered independent . The proposed models are evaluated on six datasets : KOS , NIPS , NYtimes , TwentyNewsGroup , Wikipedia English 2012 , and Dailymotion English . The four first datasets are classically used to benchmark topic models based on bag-of-word representation ( Dua & Graff , 2017 ) . Wikipedia , and Dailymotion are large scale datasets counting about one million documents . These latter datasets are used to qualitatively assess how our models behave on large scale datasets . Conducted experiments demonstrate that the proposed models are effective in discovering latent topics . Furthermore , evaluation results show that our models achieve lower perplexity than latent Dirichlet allocation ( LDA ) trained on the same datasets . The remainder of this paper is organized as follows . Section 2 discusses related work . Section 3 briefly presents principles of topics generation with PLSA . Section 4 presents the first version of the model we propose which is based on discrete topics , documents , and words embedding . Section 5 gives details about the second version of the model which is based on embedding documents using a continuous neural auto-encoder . Section 6 provides details about the experiments conducted to assess the effectiveness of the proposed models . Finally Section 7 derives conclusions and gives future research directions . 2 RELATED WORK . Unsupervised text analysis with methods related to latent semantic analysis ( LSA ) has a long research history . Latent semantic analysis takes a high dimensional text vector representation and apply linear dimensionality reduction methods such as singular value decomposition ( SVD ) to the word-document counts matrix ( Dumais , 1990 ) . The main drawback of LSA is related to it ’ s lack of statistical foundations limiting the model interpretability . Probabilistic latent semantic analysis ( PLSA ) was proposed by Hofmann ( 2001 ) to ground LSA on solid statistical foundations . PLSA is based on a well defined generative model for text generation based on the bag-of-words assumption . PLSA can be interpreted as a probabilistic matrix factorisation of the word-document counts matrix . Because PLSA model defines a probabilistic mixture model , it ’ s parameters can be estimated using the classical Expectation-Maximization ( EM ) algorithm ( Moon , 1996 ) . PLSA has been exploited in many applications related to text modelling by Hofmann ( 2001 ) , to collaborative filtering by Popescul et al . ( 2001 ) , to web links analysis by Cohn & Hofmann ( 2001 ) , and to visual scene classification by Quelhas et al . ( 2007 ) . The main drawback of PLSA is that it is a generative model of the training data . It does not apply to unseen data . To extend PLSA to unseen data , Blei et al . ( 2003 ) proposed the latent Dirichlet Allocation ( LDA ) which models documents via hidden Dirichlet random variables specifying probabilities on a lower dimensional hidden spaces . The distribution over words of an unseen document is a continuous mixture over document space and a discrete mixture over all possible topics . Modeling with LDA has been thoroughly investigated resulting in dynamic topic models to account for topics temporal dynamics by Blei & Lafferty ( 2006 ) ; Wang et al . ( 2008 ) ; Shalit et al . ( 2013 ) ; Varadarajan et al . ( 2013 ) ; Farrahi & Gatica-Perez ( 2014 ) , hierarchical topic models to account topic hierarchical structures by Blei et al . ( 2004 ) , and multi-lingual topic model to account for multi-lingual corpora by Boyd-Grabber & Blei ( 2009 ) ; Vulic et al . ( 2015 ) , and supervised topic model to account for corpora composed by categorized documents ( Blei & McAuliffe , 2008 ) . Beside text modelling , LDA has been applied to discover people ’ s socio-geographic routines from mobiles phones data by Farrahi & Gatica-Perez ( 2010 ; 2011 ; 2014 ) , mining recurrent activities from long term videos logs by Varadarajan et al . ( 2013 ) . Learning a topic models based on LSA , PLSA or LDA requires considering jointly all words , documents , and topics . This is a strong limitation when the vocabulary and the number of documents are very large . For example , for PLSA or LDA , learning the model requires maintaining a large matrix containing the probability of a topics given words and documents ( Hofmann , 2001 ; Blei et al. , 2003 ) . To overcome this limitation Hoffman et al . ( 2010 ) proposed online training of LDA models using stochastic variational inference . Recently , with the rise of deep learning with neural networks that are trained using stochastic gradient descent on sample batches , novel topic models based on neural networks have been proposed . Salakhutdinov & Hinton ( 2009 ) proposed a two layer restricted Boltzmann machine ( RBM ) called the replicated-softmax to extract low level latent topics from a large collection of unstructured documents . The model is trained using the contrastive divergence formalism proposed by CarreiraPerpiñán & Hinton ( 2005 ) . Benchmarking the model performance against LDA showed improvement in term on unseen documents ’ perplexity and accuracy on retrieval tasks . Larochelle & Lauly ( 2012 ) proposed a neural auto-regressive topic model inspired from the replicated softmax model but replacing the RBM model with neural auto-regressive distribution estimator ( NADE ) which is a generative model over vectors of binary observations ( Larochelle & Murray , 2011 ) . An advantage of the NADE over the RBM is that during training , unlike for the RBM , computing the data negative log-likelihood ’ s gradient with respect to the model parameters does not requires Monte Carlo approximation . Srivastava et al . ( 2013 ) generalized the replicated softmax model proposed by Salakhutdinov & Hinton ( 2009 ) to deep RBM which has more representation power . Cao et al . ( 2015 ) proposed neural topic model ( NTM ) , and it ’ s supervised extension ( sNTM ) where words and documents embedding are combined . This work goes beyond the bag-of-words representation by embedding word n-grams with word2vec embedding as proposed by Mikolov et al . ( 2013 ) . Moody ( 2016 ) proposed the lda2vec , a model combining Dirichlet topic model as Blei et al . ( 2003 ) ) and word embedding as Mikolov et al . ( 2013 ) . The goal of lda2vec is to embed both words and documents in the same space in order to learn both representations simultaneously . Other interesting works combine probabilistic topic models such as LDA , and neural network modelling ( Wan et al. , 2011 ; Yao et al. , 2017 ; Dieng et al. , 2017 ) . Wan et al . ( 2011 ) proposed a hybrid model combining a neural network and a latent topic model . The neural network provides a lower dimensional embedding of the input data , while the topic model extracts further structure from the neural network output features . The proposed model was validated on computer vision tasks . Yao et al . ( 2017 ) proposed to integrate knowledge graph embedding into probabilistic topic modelling by using as observation for the probabilistic topic model document-level word counts and knowledge graph entities embedded into vector forms . Dieng et al . ( 2017 ) integrated to a recurrent neural network based language model global word semantic information extracted using a probabilistic topic model . 3 TOPIC MODELLING WITH PROBABILISTIC LATENT SEMANTIC ANALYSIS . Probabilistic latent semantic analysis ( PLSA ) proposed by Hofmann ( 2001 ) is based on the bag-ofwords representation defined in the following . 3.1 BAG OF WORDS REPRESENTATION . The grounding assumption of the bag-of-word representation is , that for text content representation , only word occurrences matter . Word orders can be ignored without harm to understanding . Let us assume available a corpus of documents D = { doc1 , doc2 , ... , doci , ... , docI } . Every document is represented as the occurrence count of words of a given vocabulary W = { word1 , word2 , ... , wordn , ... , wordN } . Let us denote by c ( wordn , doci ) the occurrence count of the n ’ th vocabulary word into the i ’ th document . The normalized bag-of-words representation of the i ’ th document is given by the empirical word occurrences probabilities : fni = c ( wordn , doci ) ∑N m=1 c ( wordm , doci ) , n = 1 , ... , N. ( 1 ) With the bag-of-words assumption , fni , is an empirical approximation of the probability that wordn appears in document doci denoted p ( wordn|doci ) .

Summary: This paper proposes a neural topic model that aim to discover topics by minimizing a version of the PLSA loss. According to PLSA, a document is presented as a mixture of topics, while a topic is a probability distribution over words, with documents and words assumed independent given topics. Thanks to this assumption, each of these probability distributions (word|topic, topic|document, and word|document) can essentially be expressed as a matrix multiplication of the other two, and EM is usually adopted for the optimization. This paper proposes to embed these relationships in a neural network and then optimize the model using SGD.