Discovering Topics With Neural Topic Models Built From PLSA Loss

1 INTRODUCTION . Nowadays , with the digital era , electronic text corpora are ubiquitous . These corpora can be company emails , news groups articles , online journal articles , Wikipedia articles , video metadata ( titles , descriptions , tags ) . These corpora can be very large , thus requiring automatic analysis methods that are investigated by the researchers working on text content analysis ( Collobert et al. , 2011 ; Cambria & White , 2014 ) . Investigated methods are about named entity recognition , text classification , etc ( Nadeau & Sekine , 2007 ; S. , 2002 ) . An important problem in text analysis is about structuring texts corpora around topics ( Daud et al. , 2010 ; Liu & Zhang , 2012 ) . Developed tools would allow to summarize very large amount of text documents into a limited , human understandable , number of topics . In computer science many definitions of the concept of topic can be encountered . Two definitions are very popular . The first one defines a topic as an entity of a knowledge graph such as Freebase or Wikidata ( Bollacker et al. , 2008 ; Vrandečić & Krötzsch , 2014 ) . The second one defines a topic as probability distribution over words of a given vocabulary ( Hofmann , 2001 ; Blei et al. , 2003 ) . When topics are represented as knowledge graph entities , documents can be associated to identified concepts with very precise meaning . The main drawback is that knowledge graphs are in general composed of a very large number of entities . For example , in 2019 , Wikidata counts about 40 million entities . Automatically identifying these entities requires building extreme classifiers trained with expensive labelled data ( Puurula et al. , 2014 ; Liu et al. , 2017 ) . When topics are defined as probability distribution over words of vocabulary , they can be identified using unsupervised methods that automatically extract them from text corpora . A precursor of such methods is the latent semantic analysis ( LSA ) model which is based on the word-document co-occurrence counts matrix factorization ( Dumais , 1990 ) . Since then , LSA has been extended to various probabilistic based models ( Hofmann , 2001 ; Blei et al. , 2003 ) , and more recently to neural network based models ( Salakhutdinov & Hinton , 2009 ; Larochelle & Lauly , 2012 ; Wan et al. , 2011 ; Yao et al. , 2017 ; Dieng et al. , 2017 ) . In this paper , we propose a novel neural network based model to automatically , in an unsupervised fashion , discover topics in a text corpus . The first variation of the model is based on a neural networks that uses as input or parameters documents , words , and topics discrete lookup table embedding to represent probabilities of words given documents , probabilities of words given topics , and probabilities of topics given documents . However because in a given corpus , the number of documents can be very large , discrete lookup table embedding explicitly associating to each document a given embedded vector can be unpractical . For example , for the case of online stores such as Amazon , or video platforms such as Dailymotion or Youtube , the number of documents are in the order of billions . To overcome this limitation , we propose a model that generates continuous document embedding using a neural auto-encoder ( Kingma & Welling , 2013 ) . Our neural topic models are trained using cross entropy loss exploiting probabilistic latent semantic analysis ( PLSA ) assumptions stating that given topics , words and documents can be considered independent . The proposed models are evaluated on six datasets : KOS , NIPS , NYtimes , TwentyNewsGroup , Wikipedia English 2012 , and Dailymotion English . The four first datasets are classically used to benchmark topic models based on bag-of-word representation ( Dua & Graff , 2017 ) . Wikipedia , and Dailymotion are large scale datasets counting about one million documents . These latter datasets are used to qualitatively assess how our models behave on large scale datasets . Conducted experiments demonstrate that the proposed models are effective in discovering latent topics . Furthermore , evaluation results show that our models achieve lower perplexity than latent Dirichlet allocation ( LDA ) trained on the same datasets . The remainder of this paper is organized as follows . Section 2 discusses related work . Section 3 briefly presents principles of topics generation with PLSA . Section 4 presents the first version of the model we propose which is based on discrete topics , documents , and words embedding . Section 5 gives details about the second version of the model which is based on embedding documents using a continuous neural auto-encoder . Section 6 provides details about the experiments conducted to assess the effectiveness of the proposed models . Finally Section 7 derives conclusions and gives future research directions . 2 RELATED WORK . Unsupervised text analysis with methods related to latent semantic analysis ( LSA ) has a long research history . Latent semantic analysis takes a high dimensional text vector representation and apply linear dimensionality reduction methods such as singular value decomposition ( SVD ) to the word-document counts matrix ( Dumais , 1990 ) . The main drawback of LSA is related to it ’ s lack of statistical foundations limiting the model interpretability . Probabilistic latent semantic analysis ( PLSA ) was proposed by Hofmann ( 2001 ) to ground LSA on solid statistical foundations . PLSA is based on a well defined generative model for text generation based on the bag-of-words assumption . PLSA can be interpreted as a probabilistic matrix factorisation of the word-document counts matrix . Because PLSA model defines a probabilistic mixture model , it ’ s parameters can be estimated using the classical Expectation-Maximization ( EM ) algorithm ( Moon , 1996 ) . PLSA has been exploited in many applications related to text modelling by Hofmann ( 2001 ) , to collaborative filtering by Popescul et al . ( 2001 ) , to web links analysis by Cohn & Hofmann ( 2001 ) , and to visual scene classification by Quelhas et al . ( 2007 ) . The main drawback of PLSA is that it is a generative model of the training data . It does not apply to unseen data . To extend PLSA to unseen data , Blei et al . ( 2003 ) proposed the latent Dirichlet Allocation ( LDA ) which models documents via hidden Dirichlet random variables specifying probabilities on a lower dimensional hidden spaces . The distribution over words of an unseen document is a continuous mixture over document space and a discrete mixture over all possible topics . Modeling with LDA has been thoroughly investigated resulting in dynamic topic models to account for topics temporal dynamics by Blei & Lafferty ( 2006 ) ; Wang et al . ( 2008 ) ; Shalit et al . ( 2013 ) ; Varadarajan et al . ( 2013 ) ; Farrahi & Gatica-Perez ( 2014 ) , hierarchical topic models to account topic hierarchical structures by Blei et al . ( 2004 ) , and multi-lingual topic model to account for multi-lingual corpora by Boyd-Grabber & Blei ( 2009 ) ; Vulic et al . ( 2015 ) , and supervised topic model to account for corpora composed by categorized documents ( Blei & McAuliffe , 2008 ) . Beside text modelling , LDA has been applied to discover people ’ s socio-geographic routines from mobiles phones data by Farrahi & Gatica-Perez ( 2010 ; 2011 ; 2014 ) , mining recurrent activities from long term videos logs by Varadarajan et al . ( 2013 ) . Learning a topic models based on LSA , PLSA or LDA requires considering jointly all words , documents , and topics . This is a strong limitation when the vocabulary and the number of documents are very large . For example , for PLSA or LDA , learning the model requires maintaining a large matrix containing the probability of a topics given words and documents ( Hofmann , 2001 ; Blei et al. , 2003 ) . To overcome this limitation Hoffman et al . ( 2010 ) proposed online training of LDA models using stochastic variational inference . Recently , with the rise of deep learning with neural networks that are trained using stochastic gradient descent on sample batches , novel topic models based on neural networks have been proposed . Salakhutdinov & Hinton ( 2009 ) proposed a two layer restricted Boltzmann machine ( RBM ) called the replicated-softmax to extract low level latent topics from a large collection of unstructured documents . The model is trained using the contrastive divergence formalism proposed by CarreiraPerpiñán & Hinton ( 2005 ) . Benchmarking the model performance against LDA showed improvement in term on unseen documents ’ perplexity and accuracy on retrieval tasks . Larochelle & Lauly ( 2012 ) proposed a neural auto-regressive topic model inspired from the replicated softmax model but replacing the RBM model with neural auto-regressive distribution estimator ( NADE ) which is a generative model over vectors of binary observations ( Larochelle & Murray , 2011 ) . An advantage of the NADE over the RBM is that during training , unlike for the RBM , computing the data negative log-likelihood ’ s gradient with respect to the model parameters does not requires Monte Carlo approximation . Srivastava et al . ( 2013 ) generalized the replicated softmax model proposed by Salakhutdinov & Hinton ( 2009 ) to deep RBM which has more representation power . Cao et al . ( 2015 ) proposed neural topic model ( NTM ) , and it ’ s supervised extension ( sNTM ) where words and documents embedding are combined . This work goes beyond the bag-of-words representation by embedding word n-grams with word2vec embedding as proposed by Mikolov et al . ( 2013 ) . Moody ( 2016 ) proposed the lda2vec , a model combining Dirichlet topic model as Blei et al . ( 2003 ) ) and word embedding as Mikolov et al . ( 2013 ) . The goal of lda2vec is to embed both words and documents in the same space in order to learn both representations simultaneously . Other interesting works combine probabilistic topic models such as LDA , and neural network modelling ( Wan et al. , 2011 ; Yao et al. , 2017 ; Dieng et al. , 2017 ) . Wan et al . ( 2011 ) proposed a hybrid model combining a neural network and a latent topic model . The neural network provides a lower dimensional embedding of the input data , while the topic model extracts further structure from the neural network output features . The proposed model was validated on computer vision tasks . Yao et al . ( 2017 ) proposed to integrate knowledge graph embedding into probabilistic topic modelling by using as observation for the probabilistic topic model document-level word counts and knowledge graph entities embedded into vector forms . Dieng et al . ( 2017 ) integrated to a recurrent neural network based language model global word semantic information extracted using a probabilistic topic model . 3 TOPIC MODELLING WITH PROBABILISTIC LATENT SEMANTIC ANALYSIS . Probabilistic latent semantic analysis ( PLSA ) proposed by Hofmann ( 2001 ) is based on the bag-ofwords representation defined in the following . 3.1 BAG OF WORDS REPRESENTATION . The grounding assumption of the bag-of-word representation is , that for text content representation , only word occurrences matter . Word orders can be ignored without harm to understanding . Let us assume available a corpus of documents D = { doc1 , doc2 , ... , doci , ... , docI } . Every document is represented as the occurrence count of words of a given vocabulary W = { word1 , word2 , ... , wordn , ... , wordN } . Let us denote by c ( wordn , doci ) the occurrence count of the n ’ th vocabulary word into the i ’ th document . The normalized bag-of-words representation of the i ’ th document is given by the empirical word occurrences probabilities : fni = c ( wordn , doci ) ∑N m=1 c ( wordm , doci ) , n = 1 , ... , N. ( 1 ) With the bag-of-words assumption , fni , is an empirical approximation of the probability that wordn appears in document doci denoted p ( wordn|doci ) .

Summary: I am unimpressed with the quality of writing and presentation, to begin with. There are numerous grammatical errors and typos that make the paper a very difficult read. The presentation also follows an inequitable pattern where the backgrounds and related works are overemphasized and the actual contribution of the paper seems very limited. In its current form, this paper is not ready for publication in ICLR.