One Objective for All Models --- Self-supervised Learning for Topic Models

1 INTRODUCTION . Recently researchers have successfully trained large-scale models like BERT ( Devlin et al. , 2018 ) and GPT ( Radford et al. , 2018 ) , which offers extremely powerful representations for many NLP tasks ( see e.g . Liu et al . ( 2021 ) ; Jaiswal et al . ( 2021 ) and references therein ) . To train these models , often one starts with sentences in a large text corpus , mark random words as “ unknown ” and ask the neural network to try to predict the unknown words . This approach is known as self-supervised learning . Despite many attempts , there is still no complete understanding of why self-supervised learning would lead to useful representations . A major bottleneck is that a “ useful ” representation often has no precise mathematical definition in the practical settings . One approach to get a precise definition for “ useful ” representation is to apply self-supervised learning on synthetic data generated by probabilistic models . For example , many topic models ( such as Latent Dirichlet Allocation ( LDA ) ( Blei et al. , 2003 ) , Correlated Topic Model ( CTM ) ( Blei & Lafferty , 2007 ) , Pachinko Allocation Model ( PAM ) ( Li & McCallum , 2006 ) ) treat documents as a bag-of-words and use a two-step procedure of generating a document : first each document is viewed as a probability distribution of topics ( often called the topic proportions w ) , and each topic is viewed as a probability distribution over words . If we use a matrixA to represent a topic-word matrix , where Aij represents the probability of using word i in topic j , then the words in this document will have probabilities according to Aw . Since the topic-word matrix A is shared across the entire corpus , and the topic distribution vector is specific to documents , it makes sense to definew as the representation for the document . In this setting , the questions we try to answer are : Can self-supervised learning learn a representation that contains information about w ? Why is self-supervised learning better than the traditional approach of learning the probabilistic models and applying inference ? In this paper , we show that if the standard reconstruction-based objective ( see Equation ( 1 ) , similar to the objective used in Pathak et al . ( 2016 ) ; Devlin et al . ( 2018 ) ) can be minimized , then the network will necessarily learn information about posterior of w ( see Theorem 3 ) . We also highlight one particular benefit of self-supervised learning – as long as the objective can be minimized , selfsupervised learning does not depend on what topic model was used to generate the data , while traditional inference methods rely heavily on having the correct model . Self-supervised learning is much more robust to model mis-specification . Of course , the theory still needs to assume that either self-supervised learning objective can be successfully minimized ( or at least approximately minimized ) . We demonstrate that this is indeed the case in practice in Section 5 . In particular , we consider four different topic models ( pure , LDA , CTM and PAM ) and show that using the same reconstruction-based objective , it can extract information about the topic proportion w. Our experiments show that the performance of the self-supervised model is competitive against posterior inference using the correct model , while outperforming posterior inference using mis-specified model . However , our experiments also show that it can be beneficial to adapt the architecture of the network to the complexity of the topic model ( i.e. , larger/more complex networks for CTM/PAM compared to pure and LDA models ) . Therefore if we take optimization into consideration , self-supervised learning is still more robust but not completely oblivious to the exact topic model . Finally in Section 6 , we also demonstrate that minimizing this objective can lead to meaningful representations for real datasets ( which are not generated by topic models ) . 1.1 RELATED WORKS . Self-Supervised Learning Self-supervised learning recently has been shown to be able to learn useful representation , which is later used for downstream tasks . See for example Bachman et al . ( 2019 ) ; Caron et al . ( 2020 ) ; Chen et al . ( 2020a ; b ; c ) ; Grill et al . ( 2020 ) ; Chen & He ( 2021 ) ; Tian et al . ( 2020a ) ; He et al . ( 2020 ) and references therein . In particular , Devlin et al . ( 2018 ) proposed BERT , which shows that self-supervised learning has the ability to train large-scale language models and could provide powerful representations for downstream natural language processing tasks . Theoretical Understanding of Self-Supervised Learning Given the recent success of selfsupervise learning , many works have been tried to provide theoretical understanding on contrastive learning ( Arora et al. , 2019 ; Wang & Isola , 2020 ; Tosh et al. , 2020 ; Tian et al. , 2020b ; HaoChen et al. , 2021 ; Wen & Li , 2021 ; Zimmermann et al. , 2021 ) and reconstruction-based learning ( Lee et al. , 2020 ; Saunshi et al. , 2020 ; Teng & Huang , 2021 ) . Also , several papers considered the problem from a multi-view perspective ( Tsai et al. , 2020 ; Tosh et al. , 2021 ) , which covers both contrastive and reconstruction-based learning . Moreover , Wei et al . ( 2020 ) and Tian et al . ( 2021 ) studied the theoretical properties of self-training and the contrastive learning without the negative pairs respectively . Saunshi et al . ( 2020 ) investigated the benefits of pre-trained language models for downstream tasks . They showed that if one can predict the next word accurately , then the learned representation can be used to solve downstream linear task . Our setting is different from theirs as we focus on the probabilistic models . Most relevant to our paper , Tosh et al . ( 2020 ) considered the contrastive learning in the topic models setting . Our theoretical results extend their theory to reconstructionbased objective ( while also removing some assumptions ) , and our empirical results show that the reconstruction-based objective can be effectively minimized . Theoretical Analysis of Topic Models Topic models are popular probabilistic models that has proved successful in many applications while still have clear structure to allow theoretical studies . Many works have proposed provable algorithms to learn topic models , such as method of moment based approaches ( Anandkumar et al. , 2012 ; 2013 ; 2014 ; 2015 ) and anchor word based approaches ( Papadimitriou et al. , 2000 ; Arora et al. , 2012 ; 2016a ; Gillis & Vavasis , 2013 ; Bittorf et al. , 2012 ) . Much less is known about provable inference for topic models . Sontag & Roy ( 2011 ) showed that MAP estimation can be NP-hard even for LDA model . Arora et al . ( 2016b ) considered approximate inference algorithms . 1.2 OUTLINE . We first introduce the basic concepts of topic models and our objectives in Section 2 . Then in Section 3 we prove guarantees for the reconstruction-based objective . Section 4 connects the contrastive objective to reconstruction-based objective which allows us to prove a stronger guarantee for the former . We then demonstrate the ability of self-supervised learning to adapt to different models in synthetic experiments in Section 5 . Finally , we also evaluate the reconstruction-based objective on real-data to show that it extracts high-quality representations . 2 PRELIMINARIES . In this section we first introduce some general notations . Then we briefly describe the topic models we consider in Section 2.1 . Finally we define the self-supervised learning objectives in Section 2.2 and give our main results . Notation We use [ n ] to denote set { 1 , 2 , . . . , n } . For vector x ∈ Rd , denote ‖x‖ as its ` 2 norm and ‖x‖1 as its ` 1 norm . For matrix A ∈ Rm×n , we use Ai ∈ Rm to denote its i-th column . When matrix A has full column rank , denote its left pseudo-inverse as A† = ( A > A ) −1A > . For matrix or general tensor T ∈ Rd1×···×dl , we use vector vec ( T ) ∈ Rd1···dl to represent its vectorization . Let ∆K to denote K − 1 dimensional probability simplex . For two probability vectors p , q , define their total variation ( TV ) distance as TV ( p , q ) = ‖p− q‖1 /2 . 2.1 TOPIC MODELS . As we mentioned earlier , the generative process of many topic models can be viewed as a two-step procedure . We consider a general scenario where V is a finite set of vocabulary with size V and K is a set of K topics , where each topic is a distribution over V . We denote the topic-word matrix as A with Aij = P ( word i | topic j ) , so that each column in A represents the word distribution of a topic . Each document corresponds to a convex combination of different topics with proportions w. For each word in the document , first sample a topic z according to the topic proportions w , and then sample the word from the corresponding topic ( equivalently , one can also sample a word from distribution Aw ) . Different topic models differ in how they generate w , which we formulate in the following definition : Definition 1 ( General Topic Model ) . A general topic model specifies a distribution ∆ ( K ) for each number of topics K. Given a topic matrix A and ∆ ( K ) , to generate a document , one first sample w ∼ ∆ ( K ) and then sample each word in the document from the distribution Aw . Here ∆ ( K ) is a prior distribution of w which is crucial when we are trying to infer the true topic proportions w given the words in a document . Of course , different topic models may also specify different priors for the topic matrix A . However , given a large number of documents generated from the topic model , in many settings one can hope to learn the topic matrix A and the prior distribution ∆ ( K ) ( see e.g. , Arora et al . ( 2012 ; 2016a ) ) , therefore we consider the following inference problem : Definition 2 ( Topic Inference ) . Given a topic matrix A , prior distribution ∆ ( K ) for topic proportions w , and a document x , the topic inference problem tries to compute the posterior distribution w|x , A given the prior w ∼ ∆ ( K ) . Note that our general topic model can capture many standard topic models , including pure topic model , LDA , CTM and PAM . Semi-supervised Learning Setup As our goal is to understand the representation learning aspect of self-supervised learning , we consider a semi-supervised setting where each document also has a label y which is determined as a function of w. We are given large number of unlabeled documents and a small number of labeled documents , the goal is to apply self-supervised learning on the unlabeled documents to get a good representation , and then use this representation to predict the label y . Of course , if one knows the actual parameters of the model , the best way predictor for y would be to first estimate the posterior distribution w and then apply the function that maps w to label y . We will show that for functions that are approximable by low degree polynomials self-supervised learning can always provide a good representation .

Summary: This paper considers the problem of performing posterior inference for probabilistic topic models. A "general" topic model consists of two key parts:  - $A$ : the topic-word probability matrix - $\Delta$ : the prior distribution over doc-topic probability vector w  Given these parts and a new document x, the goal is to infer the posterior over w: $p(w | x)$.   It is well-known that this posterior can be estimated in several ways using standard approximate Bayesian inference methods: MCMC, variational, etc. The detailed steps of these methods are usually customized to the choice of prior.  This paper suggests a new possibility: that a learned transformation -- denoted f(x) -- trained using a so-called *reconstruction* loss (see Eq 1), can be then translated into the desired posterior via a linear function. Surprisingly, the paper suggests that learning f(x) does not depend on the prior \Delta, so the big idea is that this representation is robust to misspecification of the prior.  Concretely, the contributions of the paper seem to be:  1) Theorem 3, which claims that there exist linear weights \theta that, when applied to the vector produced by the ideal learned transformation f(x) that minimizes the reconstruction objective (Eq 1), can exactly equal the expected value of any polynomial summary function of the doc-topic probability vector w under the posterior.  2) Theorem 4, which claims that even if f is not an exact minimizer, if it is within an additive tolerance \epsilon of the ideal loss, then there exist again weights \theta such that the squared error between the true posterior summary and the linear function has bounded error for all documents.  3) Theorem 5, which looks at the contrastive objective in Eq 2, and shows that there exists a linear function that can recover the expected polynomical summaries of random variable w under the posterior.  Experiments in Sec. 5 look at toy data, with results in Fig 2 and Table 1. Essentially, these results show that the proposed SSL method produces lower error posterior estimates (in terms of  total-variation distance from ground truth) than using Bayesian inference with the wrong prior.   Experiments in Sec. 6 look at real data (the AG news dataset), comparing the representations learned by SSL at a downstream classification task to baselines of bag-of-words, word2vec, and the contrastive method of Tosh et al. Fig. 3 suggests that the proposed method offers a few percentage points absolute gain in accuracy over word2vec, and is similar to the previous NCE method (Tosh et al) in accuracy.