One Objective for All Models --- Self-supervised Learning for Topic Models

1 INTRODUCTION . Recently researchers have successfully trained large-scale models like BERT ( Devlin et al. , 2018 ) and GPT ( Radford et al. , 2018 ) , which offers extremely powerful representations for many NLP tasks ( see e.g . Liu et al . ( 2021 ) ; Jaiswal et al . ( 2021 ) and references therein ) . To train these models , often one starts with sentences in a large text corpus , mark random words as “ unknown ” and ask the neural network to try to predict the unknown words . This approach is known as self-supervised learning . Despite many attempts , there is still no complete understanding of why self-supervised learning would lead to useful representations . A major bottleneck is that a “ useful ” representation often has no precise mathematical definition in the practical settings . One approach to get a precise definition for “ useful ” representation is to apply self-supervised learning on synthetic data generated by probabilistic models . For example , many topic models ( such as Latent Dirichlet Allocation ( LDA ) ( Blei et al. , 2003 ) , Correlated Topic Model ( CTM ) ( Blei & Lafferty , 2007 ) , Pachinko Allocation Model ( PAM ) ( Li & McCallum , 2006 ) ) treat documents as a bag-of-words and use a two-step procedure of generating a document : first each document is viewed as a probability distribution of topics ( often called the topic proportions w ) , and each topic is viewed as a probability distribution over words . If we use a matrixA to represent a topic-word matrix , where Aij represents the probability of using word i in topic j , then the words in this document will have probabilities according to Aw . Since the topic-word matrix A is shared across the entire corpus , and the topic distribution vector is specific to documents , it makes sense to definew as the representation for the document . In this setting , the questions we try to answer are : Can self-supervised learning learn a representation that contains information about w ? Why is self-supervised learning better than the traditional approach of learning the probabilistic models and applying inference ? In this paper , we show that if the standard reconstruction-based objective ( see Equation ( 1 ) , similar to the objective used in Pathak et al . ( 2016 ) ; Devlin et al . ( 2018 ) ) can be minimized , then the network will necessarily learn information about posterior of w ( see Theorem 3 ) . We also highlight one particular benefit of self-supervised learning – as long as the objective can be minimized , selfsupervised learning does not depend on what topic model was used to generate the data , while traditional inference methods rely heavily on having the correct model . Self-supervised learning is much more robust to model mis-specification . Of course , the theory still needs to assume that either self-supervised learning objective can be successfully minimized ( or at least approximately minimized ) . We demonstrate that this is indeed the case in practice in Section 5 . In particular , we consider four different topic models ( pure , LDA , CTM and PAM ) and show that using the same reconstruction-based objective , it can extract information about the topic proportion w. Our experiments show that the performance of the self-supervised model is competitive against posterior inference using the correct model , while outperforming posterior inference using mis-specified model . However , our experiments also show that it can be beneficial to adapt the architecture of the network to the complexity of the topic model ( i.e. , larger/more complex networks for CTM/PAM compared to pure and LDA models ) . Therefore if we take optimization into consideration , self-supervised learning is still more robust but not completely oblivious to the exact topic model . Finally in Section 6 , we also demonstrate that minimizing this objective can lead to meaningful representations for real datasets ( which are not generated by topic models ) . 1.1 RELATED WORKS . Self-Supervised Learning Self-supervised learning recently has been shown to be able to learn useful representation , which is later used for downstream tasks . See for example Bachman et al . ( 2019 ) ; Caron et al . ( 2020 ) ; Chen et al . ( 2020a ; b ; c ) ; Grill et al . ( 2020 ) ; Chen & He ( 2021 ) ; Tian et al . ( 2020a ) ; He et al . ( 2020 ) and references therein . In particular , Devlin et al . ( 2018 ) proposed BERT , which shows that self-supervised learning has the ability to train large-scale language models and could provide powerful representations for downstream natural language processing tasks . Theoretical Understanding of Self-Supervised Learning Given the recent success of selfsupervise learning , many works have been tried to provide theoretical understanding on contrastive learning ( Arora et al. , 2019 ; Wang & Isola , 2020 ; Tosh et al. , 2020 ; Tian et al. , 2020b ; HaoChen et al. , 2021 ; Wen & Li , 2021 ; Zimmermann et al. , 2021 ) and reconstruction-based learning ( Lee et al. , 2020 ; Saunshi et al. , 2020 ; Teng & Huang , 2021 ) . Also , several papers considered the problem from a multi-view perspective ( Tsai et al. , 2020 ; Tosh et al. , 2021 ) , which covers both contrastive and reconstruction-based learning . Moreover , Wei et al . ( 2020 ) and Tian et al . ( 2021 ) studied the theoretical properties of self-training and the contrastive learning without the negative pairs respectively . Saunshi et al . ( 2020 ) investigated the benefits of pre-trained language models for downstream tasks . They showed that if one can predict the next word accurately , then the learned representation can be used to solve downstream linear task . Our setting is different from theirs as we focus on the probabilistic models . Most relevant to our paper , Tosh et al . ( 2020 ) considered the contrastive learning in the topic models setting . Our theoretical results extend their theory to reconstructionbased objective ( while also removing some assumptions ) , and our empirical results show that the reconstruction-based objective can be effectively minimized . Theoretical Analysis of Topic Models Topic models are popular probabilistic models that has proved successful in many applications while still have clear structure to allow theoretical studies . Many works have proposed provable algorithms to learn topic models , such as method of moment based approaches ( Anandkumar et al. , 2012 ; 2013 ; 2014 ; 2015 ) and anchor word based approaches ( Papadimitriou et al. , 2000 ; Arora et al. , 2012 ; 2016a ; Gillis & Vavasis , 2013 ; Bittorf et al. , 2012 ) . Much less is known about provable inference for topic models . Sontag & Roy ( 2011 ) showed that MAP estimation can be NP-hard even for LDA model . Arora et al . ( 2016b ) considered approximate inference algorithms . 1.2 OUTLINE . We first introduce the basic concepts of topic models and our objectives in Section 2 . Then in Section 3 we prove guarantees for the reconstruction-based objective . Section 4 connects the contrastive objective to reconstruction-based objective which allows us to prove a stronger guarantee for the former . We then demonstrate the ability of self-supervised learning to adapt to different models in synthetic experiments in Section 5 . Finally , we also evaluate the reconstruction-based objective on real-data to show that it extracts high-quality representations . 2 PRELIMINARIES . In this section we first introduce some general notations . Then we briefly describe the topic models we consider in Section 2.1 . Finally we define the self-supervised learning objectives in Section 2.2 and give our main results . Notation We use [ n ] to denote set { 1 , 2 , . . . , n } . For vector x ∈ Rd , denote ‖x‖ as its ` 2 norm and ‖x‖1 as its ` 1 norm . For matrix A ∈ Rm×n , we use Ai ∈ Rm to denote its i-th column . When matrix A has full column rank , denote its left pseudo-inverse as A† = ( A > A ) −1A > . For matrix or general tensor T ∈ Rd1×···×dl , we use vector vec ( T ) ∈ Rd1···dl to represent its vectorization . Let ∆K to denote K − 1 dimensional probability simplex . For two probability vectors p , q , define their total variation ( TV ) distance as TV ( p , q ) = ‖p− q‖1 /2 . 2.1 TOPIC MODELS . As we mentioned earlier , the generative process of many topic models can be viewed as a two-step procedure . We consider a general scenario where V is a finite set of vocabulary with size V and K is a set of K topics , where each topic is a distribution over V . We denote the topic-word matrix as A with Aij = P ( word i | topic j ) , so that each column in A represents the word distribution of a topic . Each document corresponds to a convex combination of different topics with proportions w. For each word in the document , first sample a topic z according to the topic proportions w , and then sample the word from the corresponding topic ( equivalently , one can also sample a word from distribution Aw ) . Different topic models differ in how they generate w , which we formulate in the following definition : Definition 1 ( General Topic Model ) . A general topic model specifies a distribution ∆ ( K ) for each number of topics K. Given a topic matrix A and ∆ ( K ) , to generate a document , one first sample w ∼ ∆ ( K ) and then sample each word in the document from the distribution Aw . Here ∆ ( K ) is a prior distribution of w which is crucial when we are trying to infer the true topic proportions w given the words in a document . Of course , different topic models may also specify different priors for the topic matrix A . However , given a large number of documents generated from the topic model , in many settings one can hope to learn the topic matrix A and the prior distribution ∆ ( K ) ( see e.g. , Arora et al . ( 2012 ; 2016a ) ) , therefore we consider the following inference problem : Definition 2 ( Topic Inference ) . Given a topic matrix A , prior distribution ∆ ( K ) for topic proportions w , and a document x , the topic inference problem tries to compute the posterior distribution w|x , A given the prior w ∼ ∆ ( K ) . Note that our general topic model can capture many standard topic models , including pure topic model , LDA , CTM and PAM . Semi-supervised Learning Setup As our goal is to understand the representation learning aspect of self-supervised learning , we consider a semi-supervised setting where each document also has a label y which is determined as a function of w. We are given large number of unlabeled documents and a small number of labeled documents , the goal is to apply self-supervised learning on the unlabeled documents to get a good representation , and then use this representation to predict the label y . Of course , if one knows the actual parameters of the model , the best way predictor for y would be to first estimate the posterior distribution w and then apply the function that maps w to label y . We will show that for functions that are approximable by low degree polynomials self-supervised learning can always provide a good representation .

Summary: The main strength of this paper lies in the thorough theoretical analysis and corresponding proofs. The authors not only prove that a new reconstruction-based objective can also extract posterior topic information for a general topic model, but also strengthen the guarantee for contrastive objective in Tosh et al. (2020) by removing some of their assumptions and the necessity of landmark documents.