Contrastive Learning for Neural Topic Model

1 Introduction . Topic models have been successfully applied in Natural Language Processing with various applications such as information extraction , text clustering , summarization , and sentiment analysis [ 1–6 ] . The most popular conventional topic model , Latent Dirichlet Allocation [ 7 ] , learns document-topic and topic-word distribution via Gibbs sampling and mean field approximation . To apply deep neural network for topic model , Miao et al . [ 8 ] proposed to use neural variational inference as the training method while Srivastava and Sutton [ 9 ] employed the logistic normal prior distribution . However , recent studies [ 10 , 11 ] showed that both Gaussian and logistic normal prior fail to capture multimodality aspects and semantic patterns of a document , which are crucial to maintain the quality of a topic model . To cope with this issue , Adversarial Topic Model ( ATM ) [ 10–13 ] was proposed with adversarial mechanisms using a combination of generator and discriminator . By seeking the equilibrium between the generator and discriminator , the generator is capable of learning meaningful semantic patterns of the document . Nonetheless , this framework has two main limitations . First , ATM relies on the key ingredient : leveraging the discrimination of the real distribution from the fake ( negative ) distribution to guide the training . Since the sampling of the fake distribution is not conditioned on the real distribution , it barely generates positive samples which largely preserves the semantic content of the real sample . This limits the behavior concerning the mutual information in the positive sample and the real one , which has been demonstrated as key driver to learn useful representations ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in unsupervised learning [ 14–18 ] . Second , ATM takes random samples from a prior distribution to feed to the generator . Previous work [ 19 ] has shown that incorporating additional variables , such as metadata or the sentiment , to estimate the topic distribution aids the learning of coherent topics . Relying on a pre-defined prior distribution , ATM hinders the integration of those variables . To address the above drawbacks , in this paper we propose a novel method to model the relations among samples without relying on the generative-discriminative architecture . In particular , we formulate the objective as an optimization problem that aims to move the representation of the input ( or prototype ) closer to the one that shares the semantic content , i.e. , positive sample . We also take into account the relation of the prototype and the negative sample by forming an auxiliary constraint to enforce the model to push the representation of the negative farther apart from the prototype . Our mathematical framework ends with a contrastive objective , which will be jointly optimized with the evidence lower bound of neural topic model . Nonetheless , another challenge arises : how to effectively generate positive and negative samples under neural topic model setting ? Recent efforts have addressed positive sampling strategies and methods to generate hard negative samples for images [ 20–23 ] . However , relevant research to adapt the techniques to neural topic model setting has been neglected in the literature . In this work , we introduce a novel sampling method that mimics the way human being seizes the similarity of a pair of documents , which is based on the following hypothesis : Hypothesis 1 . The common theme of the prototype and the positive sample can be realized due to their relative frequency of salient words . We use the example in Fig . 1 to explain the idea of our method . Humans are able to tell the similarity of the input with positive sample due to the reason that the frequency of salient words such as “ league ” and “ teams '' is proportional to their counterpart in the positive sample . On the other hand , the separation between the input and the negative sample can be induced since those words in the input do not occur in negative sample , though they both contain words “ billions '' and “ dollars '' , which are not salient in the context of the input . Based on this intuition , our method generates the positive and negative samples for topic model by maintaining the weights of salient entries and altering those of unimportant ones in the prototype to construct the positive samples while performing the opposite procedure for the negative ones . Inherently , since our method is not depended on a fixed prior distribution to draw our samples , we are not restrained in incorporating external variables to provide additional knowledge for better learning topics . In a nutshell , the contributions of our paper are as follows : • We target the problem of capturing meaningful representations through modeling the relations among samples from a new mathematical perspective and propose a novel contrastive objective which is jointly optimized with evidence lower bound of neural topic model . We find that capturing the mutual information between the prototype and its positive samples provides a strong foundation for constructing coherent topics , while differentiating the prototype from the negative samples plays a less important role . • We propose a novel sampling strategy that is motivated by human behavior when comparing different documents . By relying on the reconstructed output , we adapt the sampling to the learning process of the model , and produce the most informative samples compared with other sampling strategies . • We conduct extensive experiments in three common topic modeling datasets and demonstrate the effectiveness of our approach by outperforming other state-of-the-art approaches in terms of topic coherence , on both global and topic-by-topic basis . 2 Related Work . Neural Topic Model ( NTM ) has been studied to encode a large set of documents using latent vectors . Inspired by Variational Autoencoder , NTM inherit most techniques from VAE-specific early works , such as reparameterization trick [ 24 ] and neural variational inference [ 25 ] . Subsequent works attempting to apply for topic model [ 9 , 26 , 8 ] focus on studying various prior distributions , e.g . Gaussian or logistic normal . Recently , researches directly target to improve topic coherence through formulating it as an optimizing objective [ 27 ] , incorporating contextual language knowledge [ 28 ] , or passing external information , e.g . sentiment , group of documents , as input [ 19 ] . Generating topics that are human-interpretable has become the goal of a wide variety of latest efforts . Adversarial Topic Model [ 4 ] is a topic modeling approach that models the topics with GAN-based architecture . The key components in that architecture consist of a generator projecting randomly sampled document-topic distribution to gain the most realistic document-word distribution as possible and a discriminator trying to distinguish between the generated and the true sample [ 10 , 11 ] . To better learn informative representations of a document , Hu et al . [ 12 ] proposed adding two cycle-consistent constraints to encourage the coordination between the encoder and generator . Contrastive Framework and Sampling Techniques There are various efforts studying contrastive method to learn meaningful representations . For visual information , contrastive framework is applied for tasks such as image classification [ 29 , 30 ] , object detection [ 31–33 ] , image segmentaion [ 34–36 ] , etc . Other applications different from image include adversarial training [ 37–39 ] , graph [ 40–43 ] , and sequence modeling [ 44–46 ] . Specific positive sampling strategies have been proposed to improve the performance of contrastive learning , e.g . applying view-based transformations that preserve semantic content in the image [ 22 , 17 , 18 ] . On the other hand , there is a recent surge of interest in studying negative sampling methods . Chuang et al . [ 20 ] propose a debiasing method which is to correct the fact in false negative samples . For object detection , Jin et al . [ 47 ] employ temporal structure of video to generate negative examples . Although widely studied , little effort has been made to adapt contrastive techniques to neural topic model . In this paper , we re-formulate our goal of learning document representations in neural topic model as a contrastive objective . The form of our objective is mostly related to Robinson et al . [ 21 ] . However , there are two key differences : ( 1 ) As they use the weighting factor associated with the impact of negative sample as a tool to search for the distribution of hard negative samples , we consider it as an adaptive parameter to control the impact of the positive and negative sample on the learning . ( 2 ) We regard the effect of positive sample as the main driver to achieve meaningful representations , while they exploit the impact of negative ones . Our approach is more applicable to topic modeling , as proven in the investigation into human behavior of distinguishing among documents . 3 Methodology . 3.1 Notations and Problem Setting . In this paper , we focus on improving the performance of neural topic model ( NTM ) , measured via topic coherence . NTM inherits the architecture of Variational Autoencoder , where the latent vector is taken as topic distribution . Suppose the vocabulary has V unique words , each document is represented as a word count vector x ∈ RV and a latent distribution over T topics : z ∈ RT . NTM assumes that z is generated from a prior distribution p ( z ) and x is generated from the conditional distribution over the topic pφ ( x|z ) by a decoder φ . The aim of model is to infer the document-topic distribution given the word count . In other words , it must estimate the posterior distribution p ( z|x ) , which is approximated by the variational distribution qθ ( z|x ) modelled by an encoder θ. NTM is trained by minimizing the following objective LVAE ( x ) = −Eqθ ( z|x ) [ log pφ ( x|z ) ] +KL [ qθ ( z|x ) ||p ( z ) ] ( 1 ) Algorithm 1 Approximate β Input : Dataset D = { xi } Ni=1 , model parameter θ , model f , total training steps T 1 : Randomly pick a batch of L samples from the training set 2 : for each sample xl in the chosen batch do 3 : Draw the negative sample x−l and a positive sample x + l 4 : Obtain the latent distribution associated with the drawn samples : z−l = f ( x − l ) , f ( x + l ) = z + l 5 : Obtain the candidate β value with γl = ( z · z+ ) / ( z · z− ) . 6 : end for 7 : Initialize β as the mean of the candidate list β0 = 1L · ∑L l=1 γl 8 : for t = 1 to T do 9 : Train the model with βt = 12 − 1 T ∣∣T 2 − t ∣∣+ β0 10 : end for 3.2 Contrastive objective derivation . Let X = { x } denote the set of document bag-of-words . Each vector x is associated with a negative sample x− and a positive sample x+ . We assume a discrete set of latent classes C , so that ( x , x+ ) have the same latent class while ( x , x− ) does not . In this work , we choose to use the semantic dot product to measure the similarity between prototype x and the drawn samples . Our goal is to learn a mapping function fθ : RV → RT of the encoder θ which transforms x to the latent distribution z ( x− and x+ are transformed to z− and z+ , respectively ) . A reasonable mapping function must fulfill two qualities : ( 1 ) x and x+ are mapped onto nearby positions ; ( 2 ) x and x− are projected distantly . Regarding goal ( 1 ) as the main objective and goal ( 2 ) as the constraint enforcing the model to learn the relations among dissimilar samples , we specify the constrained optimization problem , in which denotes the strength of the constraint max θ Ex∼X ( z · z+ ) subject to Ex∼X ( z · z− ) < ( 2 ) Rewriting Eq . 2 as a Lagragian under KKT conditions [ 48 , 49 ] , we attain : F ( θ , x , x+ , x− ) = Ex∼X ( z · z+ ) − α · [ Ex∼X ( z · z− ) − ] ( 3 ) where the positive KKT multiplier α is the regularisation coefficient that controls the effect of the negative sample on training . Eq . 3 can be derived to arrive at the weighted-contrastive loss . F ( θ , x , x+ , x− ) ≥ Lcont ( θ , x , x+ , x− ) = Ex∼X [ log exp ( z · z+ ) exp ( z · z+ ) + β · exp ( z · z− ) ] ( 4 ) where α = exp ( β ) . The full proof of ( 4 ) can be found in the Appendix . Previous works [ 39 , 35 , 40 , 29 , 20 , 50 ] consider the positive and negative sample equally likely as setting β = 1 . In this paper , we leverage different values of β to guide the model concentration on the sample which is distinct from the input . In consequence , a reasonable value of β will provide a clear separation among topics in the dataset . We demonstrate our procedure to estimate β in the following section .

Summary: In this paper, the authors propose a novel contrastive objective for the neural topic model, which models the relations among samples without relying on the generative-discriminative architecture. In this work, they reveal that capturing the mutual information between the prototype and its positive samples provides a strong foundation for constructing coherent topics; while differentiating the prototype from the negative samples plays a less important role. Motivated by human behavior when comparing different documents, they also develop a novel sampling strategy to generate positive and negative samples under neural topic model setting, producing informative samples compared with other sampling strategies. Their experimental results show that their proposed method outperforms other state-of-the-art neural topic models in terms of topic coherence.