One Objective for All Models --- Self-supervised Learning for Topic Models

1 INTRODUCTION . Recently researchers have successfully trained large-scale models like BERT ( Devlin et al. , 2018 ) and GPT ( Radford et al. , 2018 ) , which offers extremely powerful representations for many NLP tasks ( see e.g . Liu et al . ( 2021 ) ; Jaiswal et al . ( 2021 ) and references therein ) . To train these models , often one starts with sentences in a large text corpus , mark random words as “ unknown ” and ask the neural network to try to predict the unknown words . This approach is known as self-supervised learning . Despite many attempts , there is still no complete understanding of why self-supervised learning would lead to useful representations . A major bottleneck is that a “ useful ” representation often has no precise mathematical definition in the practical settings . One approach to get a precise definition for “ useful ” representation is to apply self-supervised learning on synthetic data generated by probabilistic models . For example , many topic models ( such as Latent Dirichlet Allocation ( LDA ) ( Blei et al. , 2003 ) , Correlated Topic Model ( CTM ) ( Blei & Lafferty , 2007 ) , Pachinko Allocation Model ( PAM ) ( Li & McCallum , 2006 ) ) treat documents as a bag-of-words and use a two-step procedure of generating a document : first each document is viewed as a probability distribution of topics ( often called the topic proportions w ) , and each topic is viewed as a probability distribution over words . If we use a matrixA to represent a topic-word matrix , where Aij represents the probability of using word i in topic j , then the words in this document will have probabilities according to Aw . Since the topic-word matrix A is shared across the entire corpus , and the topic distribution vector is specific to documents , it makes sense to definew as the representation for the document . In this setting , the questions we try to answer are : Can self-supervised learning learn a representation that contains information about w ? Why is self-supervised learning better than the traditional approach of learning the probabilistic models and applying inference ? In this paper , we show that if the standard reconstruction-based objective ( see Equation ( 1 ) , similar to the objective used in Pathak et al . ( 2016 ) ; Devlin et al . ( 2018 ) ) can be minimized , then the network will necessarily learn information about posterior of w ( see Theorem 3 ) . We also highlight one particular benefit of self-supervised learning – as long as the objective can be minimized , selfsupervised learning does not depend on what topic model was used to generate the data , while traditional inference methods rely heavily on having the correct model . Self-supervised learning is much more robust to model mis-specification . Of course , the theory still needs to assume that either self-supervised learning objective can be successfully minimized ( or at least approximately minimized ) . We demonstrate that this is indeed the case in practice in Section 5 . In particular , we consider four different topic models ( pure , LDA , CTM and PAM ) and show that using the same reconstruction-based objective , it can extract information about the topic proportion w. Our experiments show that the performance of the self-supervised model is competitive against posterior inference using the correct model , while outperforming posterior inference using mis-specified model . However , our experiments also show that it can be beneficial to adapt the architecture of the network to the complexity of the topic model ( i.e. , larger/more complex networks for CTM/PAM compared to pure and LDA models ) . Therefore if we take optimization into consideration , self-supervised learning is still more robust but not completely oblivious to the exact topic model . Finally in Section 6 , we also demonstrate that minimizing this objective can lead to meaningful representations for real datasets ( which are not generated by topic models ) . 1.1 RELATED WORKS . Self-Supervised Learning Self-supervised learning recently has been shown to be able to learn useful representation , which is later used for downstream tasks . See for example Bachman et al . ( 2019 ) ; Caron et al . ( 2020 ) ; Chen et al . ( 2020a ; b ; c ) ; Grill et al . ( 2020 ) ; Chen & He ( 2021 ) ; Tian et al . ( 2020a ) ; He et al . ( 2020 ) and references therein . In particular , Devlin et al . ( 2018 ) proposed BERT , which shows that self-supervised learning has the ability to train large-scale language models and could provide powerful representations for downstream natural language processing tasks . Theoretical Understanding of Self-Supervised Learning Given the recent success of selfsupervise learning , many works have been tried to provide theoretical understanding on contrastive learning ( Arora et al. , 2019 ; Wang & Isola , 2020 ; Tosh et al. , 2020 ; Tian et al. , 2020b ; HaoChen et al. , 2021 ; Wen & Li , 2021 ; Zimmermann et al. , 2021 ) and reconstruction-based learning ( Lee et al. , 2020 ; Saunshi et al. , 2020 ; Teng & Huang , 2021 ) . Also , several papers considered the problem from a multi-view perspective ( Tsai et al. , 2020 ; Tosh et al. , 2021 ) , which covers both contrastive and reconstruction-based learning . Moreover , Wei et al . ( 2020 ) and Tian et al . ( 2021 ) studied the theoretical properties of self-training and the contrastive learning without the negative pairs respectively . Saunshi et al . ( 2020 ) investigated the benefits of pre-trained language models for downstream tasks . They showed that if one can predict the next word accurately , then the learned representation can be used to solve downstream linear task . Our setting is different from theirs as we focus on the probabilistic models . Most relevant to our paper , Tosh et al . ( 2020 ) considered the contrastive learning in the topic models setting . Our theoretical results extend their theory to reconstructionbased objective ( while also removing some assumptions ) , and our empirical results show that the reconstruction-based objective can be effectively minimized . Theoretical Analysis of Topic Models Topic models are popular probabilistic models that has proved successful in many applications while still have clear structure to allow theoretical studies . Many works have proposed provable algorithms to learn topic models , such as method of moment based approaches ( Anandkumar et al. , 2012 ; 2013 ; 2014 ; 2015 ) and anchor word based approaches ( Papadimitriou et al. , 2000 ; Arora et al. , 2012 ; 2016a ; Gillis & Vavasis , 2013 ; Bittorf et al. , 2012 ) . Much less is known about provable inference for topic models . Sontag & Roy ( 2011 ) showed that MAP estimation can be NP-hard even for LDA model . Arora et al . ( 2016b ) considered approximate inference algorithms . 1.2 OUTLINE . We first introduce the basic concepts of topic models and our objectives in Section 2 . Then in Section 3 we prove guarantees for the reconstruction-based objective . Section 4 connects the contrastive objective to reconstruction-based objective which allows us to prove a stronger guarantee for the former . We then demonstrate the ability of self-supervised learning to adapt to different models in synthetic experiments in Section 5 . Finally , we also evaluate the reconstruction-based objective on real-data to show that it extracts high-quality representations . 2 PRELIMINARIES . In this section we first introduce some general notations . Then we briefly describe the topic models we consider in Section 2.1 . Finally we define the self-supervised learning objectives in Section 2.2 and give our main results . Notation We use [ n ] to denote set { 1 , 2 , . . . , n } . For vector x ∈ Rd , denote ‖x‖ as its ` 2 norm and ‖x‖1 as its ` 1 norm . For matrix A ∈ Rm×n , we use Ai ∈ Rm to denote its i-th column . When matrix A has full column rank , denote its left pseudo-inverse as A† = ( A > A ) −1A > . For matrix or general tensor T ∈ Rd1×···×dl , we use vector vec ( T ) ∈ Rd1···dl to represent its vectorization . Let ∆K to denote K − 1 dimensional probability simplex . For two probability vectors p , q , define their total variation ( TV ) distance as TV ( p , q ) = ‖p− q‖1 /2 . 2.1 TOPIC MODELS . As we mentioned earlier , the generative process of many topic models can be viewed as a two-step procedure . We consider a general scenario where V is a finite set of vocabulary with size V and K is a set of K topics , where each topic is a distribution over V . We denote the topic-word matrix as A with Aij = P ( word i | topic j ) , so that each column in A represents the word distribution of a topic . Each document corresponds to a convex combination of different topics with proportions w. For each word in the document , first sample a topic z according to the topic proportions w , and then sample the word from the corresponding topic ( equivalently , one can also sample a word from distribution Aw ) . Different topic models differ in how they generate w , which we formulate in the following definition : Definition 1 ( General Topic Model ) . A general topic model specifies a distribution ∆ ( K ) for each number of topics K. Given a topic matrix A and ∆ ( K ) , to generate a document , one first sample w ∼ ∆ ( K ) and then sample each word in the document from the distribution Aw . Here ∆ ( K ) is a prior distribution of w which is crucial when we are trying to infer the true topic proportions w given the words in a document . Of course , different topic models may also specify different priors for the topic matrix A . However , given a large number of documents generated from the topic model , in many settings one can hope to learn the topic matrix A and the prior distribution ∆ ( K ) ( see e.g. , Arora et al . ( 2012 ; 2016a ) ) , therefore we consider the following inference problem : Definition 2 ( Topic Inference ) . Given a topic matrix A , prior distribution ∆ ( K ) for topic proportions w , and a document x , the topic inference problem tries to compute the posterior distribution w|x , A given the prior w ∼ ∆ ( K ) . Note that our general topic model can capture many standard topic models , including pure topic model , LDA , CTM and PAM . Semi-supervised Learning Setup As our goal is to understand the representation learning aspect of self-supervised learning , we consider a semi-supervised setting where each document also has a label y which is determined as a function of w. We are given large number of unlabeled documents and a small number of labeled documents , the goal is to apply self-supervised learning on the unlabeled documents to get a good representation , and then use this representation to predict the label y . Of course , if one knows the actual parameters of the model , the best way predictor for y would be to first estimate the posterior distribution w and then apply the function that maps w to label y . We will show that for functions that are approximable by low degree polynomials self-supervised learning can always provide a good representation .

Summary: This paper introduces a new method for performing inference on topic models, based on self supervised learning. Their approach is generalizable to multiple types of topic models. Specifically, the authors show that the expectation of some polynomial function of the topic posterior will be a linear function of the output of function that optimizes the self-supervised learning objective. The authors explore 2 types of self-supervised learning objectives, one based on reconstruction and one based on contrastive learning that was explored in prior work. The authors provide proofs for their main result as well a result robust to approximate minimization of the self-supervised objective. In their experiments the authors show that their approach outperforms inference with a misspecified model. The authors also run semi-supervised learning experiments on their approach.