One Objective for All Models --- Self-supervised Learning for Topic Models
Authors: Zeping Luo
Cindy Weng
Shiyou Wu
Mo Zhou
Rong Ge
Venue: ICLR 2022 Submitted (2022)
Keywords: self-supervised learning
topic models

Abstract: Self-supervised learning has significantly improved the performance of many NLP tasks. In this paper, we highlight a key advantage of self-supervised learning - when applied to data generated by topic models, self-supervised learning can be oblivious to the specific model, and hence is less susceptible to model mis-specification. In particular, we prove that commonly used self-supervised objectives based on reconstruction or contrastive samples can both recover useful posterior information for general topic models. Empirically, we show that the same objectives can perform competitively against posterior inference using the correct model, while outperforming posterior inference using mis-specified model.

Under review as a conference paper at ICLR 2022
ONE OBJECTIVE FOR ALL MODELS SELF-SUPERVISER LEARNING FOR TOPIC MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Self-supervised leamning has significantly improved the performance of many NLP tasks. In this paper, we highlight key advantage of self-supervised learning when applied to data generated by topic models, self-supervised learning can be oblivious to the specific model, and hence is less susceptible to model mis- specification. In particular, we prove that commonly used self-supervised objec- tives based on reconstruction or contrastive samples can both recover useful pos- terior information for general topic models. Empirically, we show that the same objectives can perform competitively against posterior inference using the correct model, while outperforming posterior inference using mis-specified model.
INTRODUCTION
Recently researchers have successfully trained large-scale models like BERT (Devlin et al., 2018) and GPT (Radford et al. 2018), which offers extremely powerful representations for many NLP tasks (see e.g. Liu et al. 2021); Jaiswal al. (202T) and references therein). To train these models, often one starts with sentences in a large text corpus, mark random words as "unknown" and ask the neural network to try predict the unknown words. This approach is known as self-supervised learning. Despite many attempts, there is still no complete understanding of why self-supervised learning would lead useful representations. A major bottleneck is that a "useful" representation often has no precise mathematical definition in the practical settings.
One approach to get a precise definition for "useful" representation is to apply sulf-supe learn- ing on synthetic data generated by probabilistic models. For example, many topic models (such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003), Correlated Topic Model (CTM) (Blei & Lafferty| (2007). Pachinko Allocation Model (PAM) (Li & McCallum, (0066) treat documents as first each document is viewed as probability distribution of topics (often called the topic proportions w), and each topic is viewed as probability distribution over words. If we use a matrix A to represent a topic-word matrix, where Ay represents the probability of using word i in topie j. then the words in this document will have probabilities according to Aw. Since the topic-word matrix A is shared across the entire corpus, and the topic distribution vector specific to documents, it makes sense define the representation for the document. In this setting, the questions we try to answer are: Can self-supervised learning learn a representation that contains information about w? Why relf-supervised learning better than the traditional approach of learning the probabilistic models and applying inference?
bag-of-words and use a two-step procedure of generating a document:
In this paper, we show that if the standard reconstruction- objective (see Equation @.. similar to the objective used in Pathak et al. (2016); Devlin et al. (2018)) can be minimized, then the network will necessarily learn information about posterior of w (see Theorem 3} We also highlight one particular benefit of supervspee learning as long as the objective can be minimized, self- supervised learning does not depend on what topic model was used to generate the data, while traditional inference methods rely heavily on having the correct model. Self-supervised learning is much more robust to model mis- sissspeciiiin.ion.
Of course, the theory still needs assume that either self-supervised learning objective can be suc- cessfully minimized (or at least approximately minimized). We demonstrate that this is indeed the case in practice in Section In particular, consider four different topic models (pure, LDA, CTM and PAM) and show that using the same reconstruction based objective, it can extract information about the topic proportion Our experiments show that the performance of the self-supervised

---

Under review as a conference paper at ICLR 2022
model is competitive against posterior inference using the correct model, while outperforming pos- terior inference using mis-specified model. However, our experiments also show that it can be bene- ficial to adapt the architecture of the network to the complexity of the topic model (i.c., larget/more complex networks for CTM/PAM compared to pure and LDA models). Therefore we take opti- mization into consideration, self-supervised learning is still more robust but not completely oblivious to the exact topic model. Finally in Section 6 we also demonstrate that minimizing this objective can lead to meaningful representations for real datasets (which are not generated by topic models).
1.1 RELATED WORKS
Self-Supervieed Learning Self-supervised learning recently has been shown to be able to learn useful representation, which is later used for downstream tasks. See for example Bachman et al. (2019); : Caron et al.] (2020); [Chnn et al.] (0020|/bch Grill et al. (2020); Chen & He (2021): Tian al. (2020a); He al. (2020) and references therein. In particular, Devlin et al. (20T8) proposed BERT, which shows that self-supervised learning has the ability to train large-scale language models and could provide powerful representations for downstream natural language processing tasks.
Theoretical Understanding of Self-Supervised Learning Given the recent success of self-
supervise learning, many works have been tried to provide theoretical understanding on contrastive learning (Arora et al. 2019, Wang & Isola 2020) Tosh et al. 2020; Tian et al. 2020b HaoChen
2021 Wen & Li] 2021 Zimmermann et al. (2021) and reconstruction- based learning (Lee
et al. 2020 Saunshi et al. 2020; Teng & Huang (2021). Also, several papers considered the prob-
lem from mueti-view perspective (Tsai al. [2020] Tosh et (021). which covers both contrastive
Tosh
and reconstruction-bast bbased learning. Moreover, Wei et al. (2020) and Tian et al.] (2021) studied the
theoretical properties of self-training and the contrastive learning without the negative pairs respec- tively. (Saunshi et al. (2020) investigated the benefits of pre-trained language models for downstream tasks. They showed that one can predict the next word accurately, then the learned representation can be used to solve downstream linear task. Our setting is different from theirs we focus on the probabilistic models. Most relevant to our paper, Tosh et al. (2020) considered the contrastive learning in the topic models setting. Our theoretical results extend their theory reconstruction- based objective (while also removing some assumptions), and our empirical results show that the reconstruction-basee based objective can be effectively minimized.
Theoretical Analysis of Topic Models Topic models are popular probabilistic models that has proved successful in many applications while still have clear structure to allow theoretical studies. Many works have proposed provable algorithms to learn topic models, such as method of moment based approaches (Anandkumar et al. [2012] 2013 2014 2015) and anchor word based approaches (Papadimitriou et al., 2000; Arora et al. 2012 201 6a) Gillis & Vavasis 2013 Bittorf et al.] (012)) Much less is known about provable inference for topic models. [Sontag & Roy (201) 1) showed that MAP estimation can be NP-hard even for LDA model. Arora et al. (2016b) considered approximate inference algorithms.
1.2 OUTLINE
Then in Section 3 we prove guarantees for the reconstruction-basee objective. Section connects the con- trastive objective to reconstruction-bue based objective which allows us to prove a stronger guarantee for the former. We then demonstrate the ability of self-supervised learning to adapt to different models in synthetic experiments in Section 5] Finally, we also evaluate the reconstruction-baat objective on real-data to show that it extracts high-quality representations.
We first introduce the basic concepts of topic models and our objectives in Section 2
2 PRELIMINARIES
In this section we first introduce some general notations. Then briefly describe the topic models we consider in Section 2.1 Finally we define the self-supervised learning objectives in Section 2.2 and give our main results.
Notation We use [n] to denote set (1,2,..., n). For vector x E Rd, denote ||>:|| as its &g norm and ||>||| as its & norm. For matrix A IRmxn, we use A, R to denote its i-th column. When
2

---

Under review as a conference paper at ICLR 2022
matrix A has full column rank, denote its left pseudo-inverse A (AT AAA For matrix or general tensor T E R4 xxxx,, we use vector vee(T) E Rdhnd to represent its vectorization. Let A to denote K - 1 dimensional probability simplex. For two probability vectors p, define their total variation (TV) distance as TV(p.)) q) = -1pp- /2.
2.1 TOPIC MODELS
As we mentioned earlier, the generative process of many topic models can be viewed as two-step procedure. We consider a general scenario where V is a finite set of vocabulary with size V and K. is set of K topics, where each topic is distribution over V. We denote the topic- word matrix as with Ag = (wword i topic j), so that each column in represents the word distribution of topic. Each document corresponds to convex combination of different topics with proportions w. For each word in the document, first sample topic & according to the topic proportions w. and then sample the word from the corresponding topic (equivalently, one also sample word from distribution Aw). Different topic models differ in how they generate w, which we formulate in the following definition:
Definition (General Topic Model). general topic model specifies a distribution ((K) for each number of topics K. Given topic matrix A and (KK), generate document. one first sample W ~ (K) and then sample each word in the document from the distribution Aw.
Here (KK) is a prior distribution of w which is crucial when we are trying to infer the true topic proportions w given the words document. Of course, different topic models may also specify different priors for the topic matrix A. However, given a large number of documents generated from the topic model, in many settings one can hope to learn the topic matrix and the prior distribution A(K) (see e.g.. Arora et al. (2012 2016 therefore we consider the following inference problem:
Definition 2 (Topic Inference). Given a topic matrix A, prior distribution ((K) for topic propor- tions w, and document x, the topic inference problem tries to compute the posterior distribution w xr A given the prior ((K).
Note that our general topic model can capture many standard topic models, including pure topic model, LDA, CTM and PAM.
Semi-supervised Learning Setup As our goal is to understand the representation learning aspect of self-supervised learning, we consider a sume-supeeeieed setting where each document also has a label which is determined as function of w. We are given large number of unlabeled documents and small number of labeled documents, the goal is to apply self-supervised learning on the unla- beled documents to get good representation, and then use this representation to predict the label y. Of course, if one knows the actual parameters of the model, the best way predictor for y would be to first estimate the posterior distribution and then apply the function that maps w to label y.- We will show that for functions that are approximable by low degree polynomials self supe-vupee learning can always provide a good representation.
2.2 SELF-SUPERVISED LEARNING
In general, self-supervised learning tries to turn unlabeled data into a supervised learning problem by hiding some known information. There are many ways to achieve this goal (see e.g.. Liu et al. (2ass)pa et al. (202T)) In this paper, we focus on two different approaches for self-supervised learning: reconstruction-basse objective and contrastive objective. We first formally define the ob- jective and then discuss our corresponding result.
Objective One common approach of self-supervised learning is to first mask part of the data and then try to find a function f to reconstruct the missing part given the unmasked part of input. This is commonly used for language modeling, In the context of topic modelling, since each word is i.i.d. sampled, given the document Tansup that generated from word distribution Aw, pick t random words from the documents and mark them unknown, then ask the learner to predict these t words given the remaining words in the documents. Specifically, let y be the i words that select and let be the document with these / words removed, we aim to
Reconstruction-Bant
3

---

Under review as a conference paper at ICLR 2022
select predictor f that minimizes the following reconstruction objective:
min (rscons)))) & E..yl((f(2)) y)l.
f
where ((0.y) :) -M- log Uk is the cross entropy loss. Here we slightly abuse the notation to use y as one-hot vector for [V]l classes and also outputs probability distribution on V Depending on the context, we will use to denote either the actual next i words or its corresponding one-hot label. Now are ready to present our result for the reconstruction-basse objective:
Theorem 1. (Informal) Consider the general topic model setting as Definition [ suppose function
f minimizes the reconstruction- basedd objective (1). Then, any polynomial P(w) of the posterior w|r, A with degree at most t can be represented by linear function of fx)
The theorem shows that if we want to get basic information about posterior distribution (such as mean, variance), then it suffices to predict small constant number of words (1 for mean and 2 for variance). See Theorem 3 for the formal statement.
Contrastive Objective Another common approach in self-superviser learning is the contrastive learning. contrastive learning. the training data is usually a pair of data (ær, x²) with a label E {0, 1), where label I means (Ir, is positive sample (r and x' are similar) and label 0 means (I, x') is a negative sample (I and x' are dissimilar). The task is to find function f such that it can distinguish the positive sample and negative sample, i.e., f(z, æ') y. Formally, we want to select a predictor f such that it minimizes the following contrastive objective:
min Leontrast Eg,e' y (2(f( v)].
(2)
In the context of topic models, following previous work Tosh et al. 2020). we generate the data (1r, x', y) as follows. We first generate a document x from the word distribution Aw, then (i) with half probability
generate t words from the same distribution Aw to form the document x' and set y 1; (ii) with half probability we generate words from a different word distribution Aw' with w/ (KC) (so that # w') and set 0. For loss function l, consider the square loss E(g. y) = (0 y)² However, our results hold for other loss function such as logistic loss.
We now give informal result contrastive objective. See Theorem 5 for the formal statement. Note that this theorem generalizes Theorem in Tosh et al. (2020).
Theorem 2. (Informal) Consider the general topic model setting as Definition 7 suppose function I minimizes the contrastive objective (2). Then we can use I and enough documents to construct a representation g(z) such that any polynomial P(w) of the posterior w jr A with degree at most can be represented by a linear function of g(æ).
3 GUARANTEES FOR THE RECONSTRUCTION-BALN
OBJECTIVE
In this section, we consider the reconstruction based objective [ and provide theoretical guarantees for its performance. We first show that if such objective with unknown words can be minimized, then any polynomial of topic posterior w pr, A with degree at most t can be represented by linear function of the learned representation.
Theorem (Main Result). Consider the general topic model setting as Definition 7 suppose topic matrix A satisfies rank( A) K and function minimizes the reconstruction- objective ( Then, any polynomial P(w) of the posterior w|r, A with degree at most t is linear in f(x),, that is there exists 0 E V such that for all documents
Eng [n = 0T fxx)
To understand this theorem, we can first think about warm-up example where = (see Sec- tion i appendix). Intuitively, in this case the best way to predict the missing word is to estimate the topic proportion w, and then predict the word using Aw where A is the topic matrix. Therefore, if the output of self-supervised learning fxx (which is V Vddimensional vector indexed by words)
minimizes the loss, then f(x) must have the form fx) = AE (w jz). When is full rank multiply- ing by the pseudo-inverse of A recovers the expectation of the posterior. The proof for the general case of predicting t-words requires more careful characterization of the optimal prediction and its relationship to E,, (w: (r], which we defer to Section A.2
4

---

Under review as a conference paper at ICLR 2022
focus the case when func- tion is exactly the minimizer of reconstruction-basee objective (T) i.e., Tecooms (f) = Tfcconst A miny [f)) However, in practice one cannot hope to find such a function exactly. the fol- lowing, we provide a robust version of Theorem 3 such that it allows solution instead of the exact optimal solution.
Robustness for approximate minimizer In Theorem
find approximate
To present result, need first introduce the following notion of condition number, which was used in many previous works, such as collaborative filtering system (Kleinberg & Sandler (2088) and topic models (Arora et al. 2016b). Intuitively, x( ()) measures how large vector would change after multiplying with B the { norm sense.
Definition 3 (&1 Condition Number). For matrix B E Rmxx, define its l condition number k(B) as (BB & mingeRmazz0 |x|l] / ||=||] max [ where B; is the i-th column of B.
Let W post E XX.... be the topic posterior tensor for the t unknown words y = (1/1.... Yt given document That is, for each entry [W post
[P(4 is the topic of word yı for i E.,[w,, ... Ws, [x]] and W post E. [w Therefore, any polynomial P(w) of degree at most t can be represented as E.. (P(w))a] = 3 vec post) for some B, where vec(W post) R& the
[t][²)
vectorization of W post-
We now are ready to present the robust version of Theorem 3 shows that we only find function whose loss is at most larger than the optimal loss, then a linear transformation of our learned representation can still give good approximation of the target polynomial within a Oe)) error.
Theorem 4 (Robust Version). Consider the general topic model setting as Definition ! suppose topic matrix A satisfies rank( A) K and function f satisfies [reconal))) S T.reconst + 6 for some é <> O. Then, any polynomial P(w) of the posterior w/x., with degree at most is approximately
linear in (x)), that is there exists 0 E VRVV such that
Ex
(Ew [(w))x] 0T ((=)))) $21381 (AT)),
where E. [[w))() = Vec(Wpost)
Note that the dependency on ||||| (A²) expected, since this is the norm ||@| that we would have if e = 0, i... the 6 we would have in Theorem 3 Thus. this quantity should be understood as the complexity of the target function for the downstream task. Empirically we show that x(At) small in Section C.6 The proof is deferred to Section A.3
4 GUARANTEES FOR THE CONTRASTIVE OBJECTIVE
In this section, we consider the contrastive objective (2) for self-supervised learning and provide similar provable guarantees on its performance as the reconstruction based objective.
We use the same approach as Tosh et al. 2020 to construct representation g(æ) based on (ar,.). Given a set of landmark documents {4 with length [,| as references, the representation defined as
g (4)))) = (( g(x,2')- = f(x,2 2')
(3)
1- f(xxx'))
The following theorem gives the theoretical guarantee for this representation. Similar to Theo- rem for reconstruction-base objective, shows that any polynomial P(w) of degree at most t representation by linear function of the learned representation. Our proof here relies the observation that having g(x, {4 (m 1) for all short landmark documents of length gives similar information as f for the reconstruction objective. This observation allows remove the anchor words assumption needed in (Tosh et al. 2020).
'To simply the notations, WLOG assume topics set K. [K] that can use topics 2; [K] as indices.
5

---

Under review as a conference paper at ICLR 2022
Theorem 5. Consider the general topic model setting as Definition
1 suppose topic matrix A sat-
isfies rank( = K and function f minimizes the contrastive objective 2 If we randomly sampled
K* different landmark documents {4 f== and construct g(z, {4 1
(3) then any poly-
nomial P( ()) of the posterior w/a, A with degree at most t. is linear in glo, (74 10))) that is there
exists a T such that for all documents
E. [=(((w))=)=01 00 g(2, (412))
In fact, one can use the same set of documents for both representation and downstream task so that we do not need additional landmark documents. We call this case as self-reference. The following corollary shows that solving the downstream task is equivalent to solve a kernel regression (in some sense the landmark documents are just random features for this kernel (Rahimi et al. (007)). See more discussions in Section B3
Corollary 6. Denote {1 9=) as the downstream dataset, where Tt E.. (ww)) is the target for document T4. If we set landmarks {1:)t= to be {ai]e then solve the downstream task equivalent to kernel regression, that is
m
arg mii
Us
0
g(z, (4))))) arg min 1 GØ||2
where ūj (.1,..., Dm)) and G E T mmxm is kernel matrix such that Gij (x11,
5 SYNTHETIC EXPERIMENTS
In this section, optimize the reconstruction-basee objective ( on data generated by several topic models to show that the objective can adapt to different priors and recover topic posterior accurately.
5.1 TOPIC MODELS
We consider four types of topic models in our experiments. Our first topic model is the pure- topic model, where each document's topic comes from discrete uniform distribution over the K- dimensional linear basis, namely {E1, e2, .... e..,.k Our second topic model is the Latent Dirichlet Allocation (LDA) model, where A(K) is symmetric Dirichlet distribution Dir(1/K).
We are also interested in topic models that involve more subtle topic correlations and similarities between different topic's 's word distributions. To this end, we consider the Correlated Topic Model (CTM) (Blei & Lafferty| (2007) and the Pachinko Allocation Model (PAM) (Li & McCallum 2006). Our goal is to construct settings where the correlation between topics provide useful information for inference. To achieve this goal, in CTM, we construct groups of 4 topics. Within the group, topic pairs (0,2) and (1,3) are highly correlated (as specified by the prior), while topics 0 and I share many words (see Figure for an illustration of the setting). In this case, if we observe a document with words that could either belong to topic 0 or 1, but this same document also has words that are associated with topic 2, we can infer that the first set of words are likely from topic 0, not topic 1. We construct such correlations in CTM model by setting the diagonal entries of its Gaussian covariance matrix to and the covariance between correlated pairs of topics to 0.99 times the diagonal entries, with the remaining entries set to zero. We construct similar examples for Pachinko Allocation Model (PAM), see Section C in appendix for details.
5.2 SIMULATION SETUP
Document generation Our documents are synthetically generated through the following steps: we first construct V K topic-word matrix A. Then, for each document, we determine the document length from Poisson distribution Pois(A), draw topic distribution from A(K), and draw words i.i.d. from the word distribution given by Aw. In our simulation, we set K = 20, V = 5000, À {30, 60). Often A will be drawn from dirichlet distribution Dir(a/K), and we take a = 1, 3, 5, 7, 9 to vary the difficulty level of the inference problem, where a larger a introduces more similarity between topics.

---

Under review as a conference paper at ICLR 2022
CIN leg- Procerion
30
01
12
02
Figure I: One example of a group of 4 topics in the Correlated Topic Model. Left: Weight of cach topic in document's topic proportion. In this example topics 1 and have large proportions they are correlated. Right: Pairwise topic similarity.
Majer Topicts) Recovery
Distance
Document Type
Decument Type
Pure
LDA
CTM
PAM
Pure
LDA
CTM
PAM
0.0148
0:0757
0,355
00089
10000
0,9050
0.9755
0.5022
0.0020
0.0033
0.0037
0,0000
0,0406
0.02255
+00330
0,0308
0.0076
0,0899
0,0053
10.0018
0,799
0.0003
00712
3
LOMDO
0,0000
0.3750
0.0455
100311
0.9900
100334
0,8950
03050]
0.1091
0,970
0.787
5
LOXDD
0,3000
038975
0.8675
0.0030
0.0062
0,0057
0,0043
0,0000
+ 0,0416
1 0.0296
4 0.0407
0,0399
0.1233
0.1071
0.0960
LORDD
0.9150
0.8675
0,8675
0.0022
0.0071
0,0068
0.0057
0.0000
0.0387
0,033
0.0407
0,0517
0.0045
0.1358
0.0089
0.1101
0.0064
0.971
00,0033
9
0.9588
0.0097
0.9100
0,0397
0.0330
0.550
0.332
0.0461
Table 1: Left: TV distance between recovered topic posterior and true topic posterior for different topic models. Right: Major topic(s) recovery rate for different topic models. The 95% confidence interval is reported in both tables.
Neural network models We transform bag-of-word document representation into full document by repeating each word by its frequency and concatenating them in random order; afterwards, the full document will be our model's input. We find that fully-connected neural network with residual connections performs the best on recovering topic posterior distribution for pure-topic and LDA documents, and attention-based architecture (Vaswani et al. (017) performs the best for recovering topic posterior distribution for CTM and PAM documents. More specifically. the attention- model contains 8 transformer blocks where each transformer block consists of a 768-dimensional attention layer and a feed-forward layer, with residual connection applied around every block. The model's second to last layer averages over the outputs of the last transformer block and the final layer projects it to a V V-dimensional word distribution.
Training setup During training. we resample 60,000 new documents after every 2 epochs, and the total amount of training data varies from 720K documents to 6M documents. The loss function is the reconstruction- based objective ( with To reduce the variance during the training, in each training document, the last 6 words are chosen as the prediction target and hidden from the model. The trained model is evaluated on test documents generated from the same topic prior the training documents. We use 5,000 test documents for the pure-topic prior and 200 test documents for the remaining priors. For each test document, we use Markov Chain Monte Carlo No U-Tum Sampler (Salvatier et al.) (016)) assuming the document's correct topic prior to approximate the ground truth expected topic posterior distribution. Then, we measure topic posterior recovery loss as the Total Variation (TV) distance between the recovered topic posterior and the ground truth topic posterior.
1, i.e., we want the model to predict one missing word.
5.3 RESULTS
Topic posterior recovery loss As illustrated in Table after 200 training epochs, our model can accurately recover the topic posterior distribution. Meanwhile, it can be observed that for larger values of the Dirichlet hyperparameter a, the recovery loss gets higher. This is expected because higher a leads to more similar topics, which makes the learning problem more difficult. This effect also captured by Definition and we show the computed condition numbers in Section C.6
7

---

Under review as a conference paper at ICLR 2022
TV Distance
Document
Type (o
Major Topic(s) Recovery
Document Type (a
LDA
CTM
Method
LDA
CTM
6.0406
0.1182
0.1218
0,9150
0,8500
0,800
LDA
LDA
0.0387
+00360
0.2083
0.2060
0.3154
0,5000
0.9125
0,3000
= 0.0038
0,0064
0.0082
CTM
0,0416
0.0275
0,0370
0.3782
0.3459
0.3939
0,8750
0,7250
0.9050
= 0,0038
= 0,00128
0.0099
PAM
0.0458
0.0397
0.0320
SSL (ours)
= 0.0020
0.0148
= 0.0033
0,0757
0,0550
0.0033
0,0489
0.0025
(ours)
0,9055
0.0406
0.9175
0.0275
0,9025
0.0330
Table 2: Left: TV distance between recovered topic posterior and true topic posterior of our self- supervised learning approach versus posterior inference via Markov Chain Monte Carlo assuming specific prior for a = 1. Right: Major topie recovery rate of our approach versus posterior inference via Markov Chain Monte Carlo assuming specific prior for 1. In both the Left and Right table, the 95% confidence interval is reported.
Major topic recovery We examine the extent recovered topic posterior captures the major topics in a given document. For pure-topic and LDA documents, we measure major topic recovery probability of correctly estimating the topic with largest proportion. Considering that topies in CTM and PAM documents are correlated in pairs, we measure the major topic recovery rate for CTM and PAM the top-2 topic overlap rate on their test documents. Table shows that our algorithm is successful throughout all settings we' 'e tried.
Robustness of self-supervised learning We compare the self-supervised approach to traditional topic inference. For each value, we take 200 test documents from each category of documents, and we run posterior inference assuming specific topic model and calculate the TV distance be- tween this posterior and the ground truth posterior. We exclude assuming pure-topic prior from comparison because gives invalid results for documents with mixed topics. For a = 1, as shown in Table Left, the topic posterior recovered from approach is closer to the ground truth topic posterior than that recovered from a mis-specified topic prior.
We also test the major topic recovery rate for model against topic inference using both correct and incorrect model (see Table 2 Right). The major topic recovery rate of our approach is similar to that of posterior inference assuming the correct prior. For pure topic model, all four methods get 100% major topics recovery rate on pure-topic documents: for LDA, again all four methods per- form similarly well since the instance more complicated CTM and PAM models. For these models, the self-supervised learning approach performs similarly to posterior inference using the correct model, and both of them are significantly better than posterior inference using mis-specified models. The results for a = 3, 5, 7, 9 are pre- sented in Section C The comparison further reveals the robustness of self-superivised learning.
difficult. However observe significant difference for
6 EXPERIMENTS ON REAL DATA
Even though theory suggests that self supervisdd learning can learn useful representations for general topic model, is still unclear whether this simple approach learns any reasonable represen- tation in practice the documents not really generated by any topic model. In this section, show that the simple approach at least generates a better representation than several baselines.
6.1 EXPERIMENT SETUP
The dataset used is the AG news dataset (Zhang al. 2015), in which each document has label of one out four categories: world, sports, business, and sci/tech. Thus, the task falls into pure topic model. We generally follow experiment setup done by Tosh et al. (2020) (see more details Section D.I). but the representation we used is generated by our reconstruction. based objective.
Our training involves two major phases: unsupervised phase and supervised phase. In the unsu- pervised phase, we trained our self-supervised model on most of our data that is later used for generating document representations; and then in the supervised phase, trained linear classifier
8

---

Under review as a conference paper at ICLR 2022
Auprasentziin C Corpaaaton
Figure Representation Comparison RBL, NCE with baselines of BOW and Word2vec.
using multi-class Logistics Regression, implemented by Scikit-leam(Pedreoosa et al. (2011)), with 3-fold Cross Validation for parameter tuning (l2 regularization term and solver).
We chose residual blocks as the basic building block for neural network architecture, with vary- ing width and depth. The network used for Figure 2 has 3 residual blocks and a width of 4096 neu- rons per layer. We explore other depths/weight combinations in Section D.3 in Appendix. We used AMSGrad optimizer (Reddi et al. 2019) with weight decay of 0.01. We resample our reconstruction- based document-label pair every 2 epochs. We trained the model using reconstruction-batte objec- tive I with = 1. Using the same variance-reduction technique in synthetic experiments, in actual training. we sampled words our labels and the loss an average of the cross entropy loss of each word evaluated separately.
To extract our representation, we use an identity matrix to replace the last layer. That is, we take a softmax function on top of the second-to-last layer of the neural network. This is different from our theory but we found that effectively reduces the high output dimension (equal to vocabulary size, around 16,700 in our experiments) and improves performance. We included more details in Section D.2 in Appendix.
6.2 EXPERIMENT RESULT
The performance of our reconsttructinn-bssee learning representation (RBL) is shown in Figure 2 where the test accuracy on representation is plotted against number of training samples used to train the classifier in supervised learning phase. Our representation RBL performs better than both BOW and Word2vec baselines. Notably, Word2vec representation is inferior only by small margin, and works well in general even when limited training samples are provided. On the contrary, although BOW representation has decent test accuracy when training samples are abundant, it performs significantly poorly on smaller training set.
We also compared our result to previous noise contrastive estimation (NCE) representation bench- mark achieved by Tosh et al. (2020) using contrastive objective where their reported best accuracy was around 87.5% when full 4000 training samples are used, slightly higher than our RBL corre- our attempt to reproduce their result, parameter tuning yields the best accuracy of 86% when full training samples were used. We plot our own reproduced results on in Figure 2 since parameter tuning did not achieve a close benchmark to their result.
sponding test accuracy of 87.1%.
CONCLUSION
"All models are wrong but some are useful." If one self-supervised objective can capture all models, then it would be able to extract useful information. In this paper, we studied the self-supervised learning in the topic models setup and showed that it can provide useful information about the topic posterior matter what topic model is used. Our results generalized previous work (Tosh et al.) 2020) to both contrastive learning and reconstruction- depend on weaker assumptions. We also empirically showed that the reconstruction-balt performs better than the posterior inference under specified models, and can provide useful representation for the topic inference problem. An immediate open problem is what other models (especially those that captures ordering of words) can be captured by the much more general self- supervised learning approaches used in practice.
learning and our techniques allow us to learning

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
For our theoretical results, we explain the detailed setup in Section 2 give the formal statement of the results in Section 3 and Section and present the complete proofs Section A and Section [B] For our experiments, we report our results and general setup in Section and Section c and explain detailed setup in Section and Section D We also include our codes that generate the results supplement material.
REFERENCES
Anima Anandkumar, Dean P Foster, Daniel Hsu, Sham M Kakade, and Yi-Kai Liu. A spectral algorithm for latent dirichlet allocation. Algorithmica, 12(1):193-214, 2015.
Animashree Anandkumar, Daniel Hsu, and Sham M Kakade. A method of moments for mixture models and hidden markov models. In Conference on Learning Theory, pp. 33-1. JMLR Work- shop and Conference Proceedings, 2012.
Animashree Anandkumar, Rong Ge, Daniel Hsu, and Sham Kakade. A tensor spectral approach to learning mixed membership community models. Conference on Learning Theory, pp. 867-881.
PMLR, 2013.
Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of machine learning research, 15: 2773-2832, 2014.
Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topie models beyond svd. In 2012 IEEE 53rd annual symposium on foundations of computer science, pp. 1-10. IEEE, 2012.
Sanjeev Arora, Rong Ge, Ravi Kannan, and Ankur Moitra. Computing nonnegative matrix factorization-provbbbll SIAM Journal on Computing, 45(4):1582-1611, 2016a.
Sanjeev Arora, Rong Ge, Frederic Koehler, Tengyu Ma, and Ankur Moitra. Provable algorithms for inference in topic models. In International Conference on Machine Learning, pp. 2859-2867.
PMLR, 2016b.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 2019.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint
2019.
Victor Bittorf, Benjamin Recht, Christopher Re, and Joel A Tropp. Factoring nonnegative matrices with linear programs. arXiv preprint arXiv: 1270, 2012.
David M Blei and John D Lafferty. correlated topic model of science. The annals of applied statistics, 1(1):17-35, 2007.
David M Blei, Andrew Ng, and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022, 2003.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint (xXiv:2006.09822 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1697- PMLR, 2020a.
Ting Chen, Simon Korblithh Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised leamerrs arXiv preprint arXiv:2006. 10029, 2020b.
10

---

Under review as a conference paper at ICLR 2022
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 20t.::2003004297, 2020c.
Jacob Devlin, Ming-Weit Wei Chang. Kenton Lee, and Kristina Toutanova. Bert:
Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv: /81o:/48000 2018.
Nicolas Gillis and Stephen A Vavasis. Fast and robust recursive algorithmsfor separable nonnegative matrix factorization. IEEE transactions on pattern analysis and machine intelligence, 36(4):698- 714, 2013.
Ban-BBstien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your latent: A new approach to self-supervised learning. arXiv preprini arXiv:2006.07733 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self supervieee deep learning with spectral contrastive loss. arXiv preprint arXiv:2106.04155 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self sule-sue learning. Technologies, 9(1):2, 2021.
Jon Kleinberg and Mark Sandler. Using mixture models for collaborative filtering. Journal of Computer and System Sciences, 74(1):49-69, 2008.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008.00 0/064, 2020.
Wei Li and Andrew McCallum. Pachinko allocation: Dag-structured mixture models topic corre- lations. In Proceedings of the 23rd international conference on Machine learning, pp. 577-584, 2006.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang. Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEFE Transactions on Knowledge and Data En- gineering, 2021.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-31111 2013.
Christos H Papadimitriou. Prabhakar Raghavan, Hisao Tamaki, and Santosh Vempala. Latent mantic indexing: A probabilistic analysis. Journal of Computer and System Sciences, 61))):
217-235, 2000.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825- 2830, 2011.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NIPS, volume 3, pp. 5. Citeseer, 2007.

---

Under review as a conference paper at ICLR 2022
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.090 092377, 2019.
Radim Rehurek and Petr Sojka. Gensim-python framework for vector space modelling, NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2), 2011.
John Salvatier, Thomas V Wiecki, and Christopher Fonnesbeck. Probabilistic programming in python using pymc3. PeerJ Computer Science, 2:e55, 2016.
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648, 2020.
David Sontag and Dan Roy. Complexity of inference in latent dirichlet allocation. Advances in neural information processing systems, 24: 1400--00116, 2011.
Jiaye Teng and Weiran Huang. Can pretext-based based self self-super learning be boosted by down- stream data? theoretical analysis. arXiv preprint (rxiv:2103.03560 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020a.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578, 2020b.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. arXiv preprint arXiv:2102.0666 000100 2021.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. arXiv preprint arX1v:2003302234, 2020.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206. PMLR, 2021.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from multi-view perspective. arXiv preprint arXiv:2006.0555 055006 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In International Conference Machine Learning, pp- 9929-9939. PMLR, 2020.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self- self-training with deep networks on unlabeled data. arXiv preprint (Xiv:2000003662) 2020.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervieed contrastive learning. arXiv preprint arXh::2105.15134 2021.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28:649-657, 2015.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. arXiv preprint arXiv:2 088.0880,, 2021.
12

---

Under review as a conference paper at ICLR 2022
A OMITTED PROOFS IN SECTION 3
In this section, we give the omitted proof's in Section We first give warmuu example to illustrate our proof idea in Section A.I Then we give the proot of our main result (Theorem) in Section A.2 and the proof of robust version (Theorem in Section A.3
A.1 WARM- UP EXAMPLE: RECONSTRUCT THE UNKNOWN WORD
In this warm-up up example, we consider the simple setting where we try to predict the only one unknown word the document using reconstruction-base objective ( The following result is the special case of Theorem 3 with i - 1, which shows that the learned representation is able to give any linear function of the topic posterior.
Theorem A.1. Consider the general topic model setting as Definition 7 suppose topic matrix A satisfies rank K and function J minimizes the reconstruction--uusee objective ① Then, any linear function P(w) of w is linear in f(x). that is for any P(w) = B w, there exists a R such that for all documents
Ew B w|/] 00 f(x).
Following the proof idea described in Section 3 we first give characterization of the optimal function f. The following lemma shows that f(x) must be the word posterior vector given the document The proof of this lemma is simply based on the property of the cross-entropy loss function and we defer it to Section A.I.I Recall that our vocabulary set is V = {VI,..., By and Tursup (z, y) the given document, where is unmasked part and y word marked unknown.
Lemma A.2. If f minimizes the reconstruction-bassee objective
then we have for all document [
f(x) = (P = 211=). P(y = uv [r)=))
Based on the above lemma, we are able to prove Theorem A. It is easy to see that the word posterior distribution is AT [w]
have (xx A w lr). Since the columns of A are linearly independent, we know Ew [[xx] = A f(x). Thus, for any linear function 8 w. there exists / (At) 8 such that E. [3 w [r] O f(x). The formal proof of the general case given in Section A.2
A.1.1 PROOF OF LEMMA A.2
Instead of focusing the i = 1 case, we directly give the corresponding lemma of Lemma A.2 for general Recall that Tannuup = (r, y) is the given document, ær the unmasked part, y = (V1, Mt) the unknown words that want predict and V {ui..... vv is the set of vocabulary. shows that the optimal ff is the words posterior distribution. Note that the words posterior distribution is linear in the topic posterior of t-th moment, which is useful for the later analysis.
Lemma A.3 (Lemma A.2 General Case). If minimizes the reconstruction-basse objective then we have for all document x
f() (p(y- (01, 81.... m))) (F2, 01,-..,
(1)))), 01,...,;1))))) P, (vv, .v.... uy (x))
Proof. Denote the word posterior distribution as p" (x). We will show f(x) = p" (x). By the law of total expectation, we have
Liccnss
Ex, E...]((((()))))) E. By v][=]
We know the probability of y given x p² (x)). Since is cross-entropy loss, have
Lrrecnnt 2 Ey -
Σ
Ip" (x))a logle" (x))as
kE(Ve)
where the equality is obtained when f(x) = px (z). Thus, when minimizes the reconstruction- based objective (T). have f(x) p" (x))
13

---

Under review as a conference paper at ICLR 2022
A.2 PROOF OF THEOREM 3
In this section, give the proof of Theorem 3 Theorem A.1 is covered by Theorem as the special case of / = 1. The proof follows the same idea as for Theorem A.1 We first show that [() the 1 case where we can recover topic posterior with A f(x), we can also recover topic posterior of t-th moment with f(r) with some matrix A. The result follows by the observation that for polynomial P(w) with degree most (P()(()) is a linear function of the topic posterior of t-th moment.
t-words posterior (Lemma A.3). Then just
Theorem (Main Result). Consider the general lopic model setting as Definition 7 suppose topic matrix A satisfies rank(A) = K and function J minimizes the reconstruction-busee objective ( Then, any polynomial P(w) of the posterior w|x, A with degree at most is linear in f(x), that there exists VRV such that for all documents x
P( = -0
Proof. Since f is the minimizer of reconstruction-basee objective given an input document x,
① by Lemma A.3
we know
f(x) = (P(y (v1, v1 01)) (V2, v1, - (1))),,)))),,,,, P(y = (uv: uv. ................................ uv [=))
We will show that f(x) is linear in the topic posterior. In the following, we focus on [f Bee which is word posterior probability of the unknown words being V1:-... Vt
((...>>......>>>>>>>>>>>
V given document x. Recall A K-1 be the K- dimensional probability simplex. By the law of total probability, we have
[f(z)] Wa......
P(11 U2. .2..... Vel22)
I WWEAK-
P(v1. V2. Ve, w wl
P(y V/...... 3e w x P((()))
WWAAK-
Σ
Ply,, U2. ..... St. 21, 22 2.. (( (w
JWEAK-1
[|
Σ
2/l] Mel 21 Re, w W)P((waa (w))).
WEAK-1 K-1
Note that 2; is the topic of word ys Since we consider the general topic model as Definition we know
P( .e.... Vel 11; #2, Ze, w) (V(11, U2. ........, V/ [1, 22 =))
P(y (2))
II A Aw.s
i=1
where the topic matrix. Hence,
Σ
É II
A /
P( #2, ze (e/wo)P((w())ddd
WEAK
21, E(K) i=1
t
Σ
II
A
II
Wz. (w)) (r)
WEAN
N-
1=1
f
Σ
II
A
Exe
W
T
$1. ( E(K [[K]i=]
14

---

Under review as a conference paper at ICLR 2022
Recall that the topie posterior tensor is W post Eu(u=*[2] IR KX.... where each entry [W WWpostls......e = (25 is the topic word ys for E [t|z) = Ew [22,... Ws [z]. Therefore,
(f(1))e1
E
II
Ag. a. [W woostler.e.....
[[K] 1=1
f(x) (A 2 A @ & A vee posst)
times
where the Kronecker product, A - AQ.............................. A xx' and ve E KK the vectorization of Wosst. Note that AAt At @ At, so we have vec(W post))) = At f(x).
Since P(w) is polynomial of degree at most t. we know there exists 3 such that E. P(w)|=] 3 vec(W Thus, let 0 (At)* B. we have [w[ x 3 vec(W) 0 f(x).
A.3 PROOF OF THBOREM 4
Similar to Lemma A.2 we can show that the representation ((x) is close to / words posterior. Then, the key lemma to obanin Theorem is the following result. suggests that if function ff is c-close to the optimal function f (word posterior distribution) under cross-entropy loss, then can still recover the topic posterior with proper B up to O error. For example, we will choose B A when t 1. See the proof of Theorem 4 for the choice of B in the general case.
Lemma A.4. For cross-entropy loss & and any two probability vectors P,p", if ((p.p") s p 1220 E(. p²) + for any matrix B we have Bpp Bp S x(
Proof. Recall that is cross cross-entropy loss, we know ((p.p")
P.E. ((p.p') DRL(P p²) D
min
By Pinsker's Inequality, we have that
1p-pp"l p'll S /2DKz(p' 11 S V2c.
Then, for any matrix B we have
- BB- S x(B)llp- p"111 S x(B)
where we use the definition of L condition number (Definition 3).
We now are ready to give the proof of Theorem which is based on Lemma A.4 and the proof of Theorem
Theorem 4 (Robust Version). Consider the general topic model setting as Desinition 7 suppose topic matrix A satisfies rank(A) (A) = K and function f satisfies Drecoms(f) I Dicomss + for some e > Then, any polynomial P(w) of the posterior w|x, A with degree most is approximately
linear in fx), that is there exists a E VR. such that
Ex [P())x] 0T f(x))
(())
where Eng [P()))]]] = TT vee (W post)..
Proof. Recall from the proof of Theorem that [P(w)(=] = 3T vec(Wpoet) = 3T (t)
where A = At x X At and k(At) = xet (AA). Same as the proof of Theorem 3 let
(AT)T B. We have
E
(P())\ AT f(=))
=F, (BT AT
<Bn
[|3e|a At (z) (z)-
< 11 E. [ (z) -
15

---

Under review as a conference paper at ICLR 2022
We going use Lemma A.4 to bound (x) ft))l (())) By Lemma A.3 we know (*)
is the word posterior, so f Bylr [x])] Since é is cross-entrope loss, we know
(+)()) [f(x)) x (x)
E U)) e(f"))))) E
Σ yk log
kEIVY
which implies f(f(z), min, f (x)) S c. Therefore, by Lemma know
A.4 with = AA, we
< <<(At) - n'(At)Vee.
Given the desired bound, we have
Eg ([u](((u) 0 [())) <211812 (At)e.
16

---

Under review as a conference paper at ICLR 2022
B OMITTED PROOFS IN SECTION 4
In this section, give the omitted proofs in Section 4 tion and proof of Lemma B.1 in Section B.2
We give the proof of Theorem 5 in Sec-
B. PROOF OF THEOREM 5
Before presenting the proof of Theorem 5 first give characterization of the representation g(x,2" z²), which would be useful the later analysis. Note that this the same as the one shown in Tosh al. (220)) We provide its proof for completeness in Section B.2
Lemma B.1. f minimizes the contrastive objective
2 then have
f(x,x")
Ply 1|,, lx)
g(x,x') 2 1 f(0,x')
Pl 0/x,x")
Now we are ready to proof Theorem
5
Theorem 5. Consider the general topic model setting as Definition isfies rank( K and function f minimizes the contrastive objective 2 If we randomly sampled
/ suppose topic matrix A sat-
m Kt different landmark documents (1,121 and construct g(z, (4 1==1 as (3), then any poly-
nomial P(w) of the posterior w [x, A with degree at most t is linear g(z, (4 m (=))), that is there exists a E I such that for all documents
En (P(W)(] = 9Tr (4
Proof. Since f is the minimizer of contrastive objective 2 by Lemma B.1 we know
Ply 1/)r,
g(r, x²)
Pyy = 0 0) x')
Similar to the proof of Theorem 3 fixed. By Bayes' rule, have
we will show that g(3, x²) is linear in the topic posterior when x
P(x, x'1y 1)) 11//(//11/P({,,,,1 xx,,
P(z, x'ly = 1)
g(x, =
P(x, x' O)T = 0)/P(r,x')) P(x, = 0)
where we use Ply = 0) Ply 1/2.
Denote the words in x' as .i...., 2/t and their corresponding topics as ,1,...,... Then, by our way of generating x, x', we know
P(x, x' y = 0) = P(x))(('),
P 1) / 2f,,.,,,,,,,,,,,, 1)P(w, xly 1)dw
/
Σ P( 21 w P) zelw)P(w, r)dw,
which implies
g(x,x')= P(x')
Σ F ".....,!/.,,.......... 2///))
[[[]]
Note that
"(14,..,,1/la ..... 2)
I P())2() II
Amt
==1
17

---

Under review as a conference paper at ICLR 2022
where A is the topic matrix. Hence,
g(x,z')= F(x')
Σ
II
Aut.2.
P(Z z (w) P(w (wol))d
[|]]] i=1
Σ
II
II
w. IP ((w2x)dw
P(a')
21 [(KK i=1
Σ
II
Ag4 Bu
II
x
P(x')
21 21 [[E] iml
Recall that the topic posterior tensor is W post [W [post]:....... En W2,, ... W2, [z]. Therefore,
[w(w8(/2)
R xx..XXX where each entry
g(x,x0) P(x')
Σ 21. E(K]
II A (W woostleneea....
A|x'] vec vec(W (eo(W) poet).
P(x')
where A= A A E is the row of A that correspond to x',, , 2/1
and vec(Wpoat) 12* the vectorization of W Wpost-
Recall we have m randomly sampled different documents {4)=-1 Denote D E mm as a diagonal matrix such that D, P(4). and Å R=*k" x submatrix of A such that each row of Á is Al4.). Since are randomly sampled, so Di. > O. Thus, we know
g(,, (4) = D- Avec(Wppot))
which implies vec(W = At Dg(x, {4 (i=)) Note that we need to show At is well-defined. Since A has full column rank, we know A also has full column rank. Thus, the submatrix A also has full column rank since
K This implies At is well-defined.
Since P(w) is polynomial of degree at most f, we know there exists B such that E. [[))] B Tee vec(W p. Thus, let 0 = D(At)T B, we have
By (P()) = BT vee([ppost post) = g(r, (4)
B.2 PROOF OF LEMMA B.
Lemma B.1. f minimizes the contrastive objective
2 then we have
f(x,x')
Ply 1|,, x²) Ply 20 x²)
g(x, x') 1- f(x,
Proof. Since f is the minimizer of contrastive objective and
Teconrast (f) = EEz,'' yy [(S x') - y) = E [Eyyee ((((xx²)) y)
is easy to see that f is the Bayes optimal predictor IP(y la,, x'). Therefore, we know
g( = Ply = 1/2,
P 00
18

---

Under review as a conference paper at ICLR 2022
B.3 DISCUSSIONS ON THE SELF-REFERENCING
As discussed in Corollary we show that one can use the same set of documents for both rep- resentation and downstream task. Recall that in Theorem 5 we need large number of landmark documents to generate the representation and then use for the downstream task (e.g., fit a poly- nomial of topic posterior). However, one may not have enough additional documents be used as landmarks. Therefore, the benefit of the self-refcrencing is that we do not need additional landmark documents generate the representation.
The key observation in Corollary i that the learned representation g(z, z') can be viewed kernel. Suppose the documents for the downstream task are #1..... Zm- Construct kernel matrix G' E Rmxm such that Cij = g (2, xz). Then, it is easy to see
0 g( {-y)ー=) A GG,,
where G is the i-th column of G. Thus, 0 can be found by the following kernel regression:
min
| - G01
0
where ÿ 90m....,,m))). T and = E.. (P())
19

---

Under review as a conference paper at ICLR 2022
C SYNTHETIC EXPERIMENT ADDITIONAL DETAILS
In this section, we include more details about our synthetic experiments Section In Section C.1 describe the document generation for PAM model. Additional results topic posterior recovery loss and major topic recovery are included in Section CC2] and Section C.3 Section C.44 and Section C.5 report the results about different training epochs and hyperparameters tuning. & condition number is reported in Section C.6 and we give some visualizations for topic correlations between models in Section C.7
C.I PACHINKO ALLOCATION MODEL TOPIC PRIOR GENERATION
The topic proportion of the Pachinko Allocation Model (PAM) described in Section 5 is generated from the following process: for each document, we first sample a "super-topic" proportion from symmetric Dirichlet distribution Dir(1/K, and super-topie-to-topi proportion from symmetric Dirichlet distribution (30( Then, we sample each word by first sample a super-topic according to the super-topic proportion and then sample the actual topic from the syper-topiccto-tope proportion. In our experiment, we set K2 = 10. Figure C.T offers a few examples of PAM topic proportion generated from the PAM model.
Topic Proportions
0.8
0.8
0.8
0.8
06
06
06
0.
02
02
02
02
0.
0.0
0.0
0.0
10
20
10
Figure C.1: PAM topic proportion
C.2 MAJOR TOPIC RECOVERY ADDITIONAL RESULTS
In Section 5.3 we measure the major topics recovery of CTM and PAM documents as the top-2 topic overlap
Since topics are correlated in pairs in both of these topic models, are also interested in the top-4 and top-6 topics overlap rate. As shown in Figure C.2 for both CTM and PAM documents, the average top-4 overlap rate is above 66% and the average top-6 overlap rate is above 62% on all values we test on. These results show that our model can accurately capture more than just the major pair of correlated topics, but also some correlated pairs of less dominant topics.
Top Topic Recovery
Top Topic Recovery
RAM
1.0
1.0
0.
06.
0.6
02
02-
Agga
Apha
Figure C.2: Recovery rate for top-2, top-4, and top-6 topics, measured on CTM (left) and PAM (right) test documents.
C.3 SELF-SUPERVISED LEARNING VERSUS ASSUMING SPECIFIC TOPIC PRIOR
We present in Section 5.3 a comparison between the performance of our model and Markov Chain Monte Carlo assuming specific topic model for on the task of TV distance and major topic
20

---

Under review as a conference paper at ICLR 2022
recovery. In this subsection, supplement the experiment results for larger values of along with a = In particular, we consider cul-rvueee learning with million training documents and with 120 thousand training documents after 200 training epochs, and expect that the former can allow our model to approximate the optimal predictor f described in Theorem 3 and the latter may offer some insights on approach's performance in dataset with limited size. Additionally, when training with 120 thousand documents, we randomly shuffle each document and reassign the target words in every epoch, to make sure our model learns enough information from the limited amount of training data. We report our results in the form of 95% confidence interval.
practical settings, such
real world
Document Type (a = I)
Method
Pure
LDA
CTM
PAM 0.1218 0.0096 0.3154 + 0.0082
LDA
0.0406 0.0016
0.1182 = 0.0099
CTM
0.2083 0.0038
0.2060 0.0064
PAM
0.3782 0.0038
0.3459
0.0128
0.3939
0.0096
SSL-6M (ours)
0.0148 0.0020
0.0757 + 0.0033
0.0550 + 0.0037
0.0489 + 0.0025
SSLK (ours)
0.0311 0.0025
0.0878 + 0.0041
0.0678 + 0.0038
0.0600 + 0.0028
Document Type (a = 3)
Method
Pure
LDA
CTM
PAM
LDA
0.0561 = 0.0030
0.1942 + 0.0156
0.1813 + 0.0146
CTM
0.2347 0.0048
0.2299 0.0082
0.3362
0.0086
PAM
0.4031 0.0046
0.3665
0.0117
0.4510 = 0.0100
SSL-6M (ours)
0.0308 = 0.0070
0.0899 = 0.0053
0.0799 = 0.0048
0.0712 = 0.0038
SSL-110K (ours)
0.0546 0.0097
0.1292 0.0067
0.1062 0.0058
0.0988
0.0048
Document Type (a 5)
Method
Pure
LDA
CTM
PAM
LDA
0.0731 0.0040
0.2123 0.0159
0.1995
0.0144
CTM
0.2655
0.0059
0.2413
0.0090
0.3428
0.0090
PAM
0.4337 + 0.0055
0.3608 + 0.0089
0.4794 + 0.0097
SSL-6M (ours)
0.0501 0.0030
0.1041
0.0062
0.0970
0.0057
0.0787
0.0043
SSL-120K (ours)
0.0832 0.0174
0.1629 0.0082
0.1245 0.0071
0.1077 0.0054
Document Type (a 7)
Method
Pure
LDA
CTM
PAM
LDA
0.0883 0.0050
0.2438 0.0169
0.2138 0.0158
CTM
0.2840 + 0.0060
0.2556 + 0.0089
0.3530 + 0.0099
PAM
0.4561 0.0052
0.3707
0.0089
0.4979
0.0101
SSL-6M (ours)
0.0391 0.0022
0.1233
0.0071
0.1071
0.0068
0.0960 0.0057
SSL-120K (ours)
+ 0.0212
0.1851
0.0079
0.1340 0.0078
0.1358 # 0.0070
Document Type (a = 9)
Method
Pure
LDA
CTM
PAM
LDA
0.1051 0.0065
0.2559 0.0162
0.2281 0.0152 0.3556 + 0.0084
CTM
0.3129 + 0.0079
0.2580 0.0092
PAM
0.4805 0.0071
0.3776
0.0093
0.5156 0.0088
6SL-6M (ours)
0.0517 = 0.0045
0.1358
0.0089
0.1101 # 0.0064
0.0971 # 0.0053
SSL-120K (ours)
0.1121 + 0.0202
0.2221 + 0.0123
O. 0.1449 + 0.0077
0. 0.1460 + 0.0070
Table C.I: TV between our recovered topic posterior and the true topic posterior of our self- supervised learning approach versus topic inference via Markov Chain Monte Carlo assuming specific prior for a = 1,3,5, 7,7, 7,9. The 95% confidence interval reported.
Table shows that our approach with 6 million training documents consistently outperforms mis- specified topic priors by yielding much lower TV distance from the true topic posterior. Note that the entries with correct topic model are omitted, because in our experiment the Markov Chain Monte Carlo recovered topic posterior assuming the correct topic model is exactly what as
21

---

Under review as a conference paper at ICLR 2022
ground truth topic posterior. Meanwhile, approach with 120 thousand training documents outperforms misspecified prior in almost every scenario, except when compared to LDA prior on pure topic documents. This is is likely because the LDA topic prior is relatively concentrated one specific topic and thus similar to the pure topie prior.
We also present comparison between self-supervised learning approach and posterior inference assuming specific topic prior on the task of recovering major topics in the topic proportion for = 1, 3, 5, 7, as supplement to Section [.3 As shown in Table C.2 in almost every scenario we test on, our model can perform competitively against the MCMC-inferred posterior assuming the correct topic prior and outperform misspecified topic prior.
Document Type (a CTM
Method
LDA 0.9150 + 0.0387
PAM
LDA
0.8850 + 0.0344
0.8500 + 0.0360
CTM
0,9000 = 0.0416
0.9175 = 0.0275
0,8300 = 0.0370
PAM
0,8750 # 0.0458
0.7250 # 0.0397
0.9050 # 0.0320
SSL-6M (ours)
0.9050 + 0.0406
0.9175 = 0.0275
0.9025 = 0.0330
SSL-120K (ours)
0.9150 = 0.0387
0.9100 = 0.0292
0.9200 = 0.0289
Document Type (a 3)
Method
LDA
CTM
PAM
LDA
0.8900 = 0.0434
0.8200 = 0.0354
0.8200 + 0.0404
CTM
0,8800 # 0.0450
0.8900 + 0.0326
0.7650 + 0.0404
PAM
0.8550 + 0.0488
0.6375 + 0.0392
0.8950 + 0.0357
SSL-6M (ours)
0,8750 = 0.0458
0.8900 = 0.0311
0.8950 + 0.0344
SSL-120K (ours)
0,8700 = 0,0466
0.8900 + 0.0312
0,8950 + 0.0357
Document Type (a = 5)
Method
LDA
CTM
PAM
LDA
0.9050 = 0.0406
0.8500 = 0.0339
0.7750 # 0.0390
CTM
0.9050 + 0.0406
0.8975 + 0.0304
0.6775 + 0.0421
PAM
0.8850 + 0.0442
0.6150 = 0.0371
0.8825 # 0.0379
SSL-6M (ours)
0,9000 + 0.0416
0.8975 + 0.0296
0,8675 + 0.0407
SSL-120K (ours)
0.8800 + 0.0450
0.8900 + 0.0303
0.8750 + 0.0390
Document Type (a 7)
Method
LDA 0.8800 + 0.0450
CTM
PAM
LDA
0.7750 + 0.0397
0.7475 + 0.0433
CTM
0.8650 + 0.0474
0.8825 + 0.0353
0.6250 + 0.0390
PAM
0,8450 # 0.0502
0.5725 # 0.0356
0,8700 # 0.0411
SSL-6M (ours)
0.9150 + 0.0387
0.8675 + 0.0383
0.8675 + 0.0407
SSL-120K (ours)
0.9350 = 0.0342
0.8775 # 0.0370
0.8725 # 0.0398
Document Type (a CTM
Method
LDA 0.8850 + 0.0442
PAM
LDA
0.7750 = 0.0384
0.7350 = 0.0432
CTM
0,8950 # 0.0425
0,8800 # 0.0348
0.5925 = 0.0392
PAM
0.8650 # 0.0474
0.5675 # 0.0358
0.8600 + 0.0428
SSL-6M (ours)
0.9100 + 0.0397
0.8850 + 0.0330
0.8325 + 0.0461
SSL-120K (ours)
0.8600 + 0.0481
0.8800 = 0.0341
0.8425 + 0.0430
Table C.2: Major topic recovery rate of our approach versus posterior inference via Markov Chain
Monte Carlo assuming specific prior for a = 1,3, 5, 7, 9. We report the 95% confidence interval.
To further investigate the performance of self-supervised learning, we compare our approach with Variational Inference assuming specific topic prior the task of recovering major topics in test document's topic proportion. Table C.3 shows that, similar to our findings from Table C222 our model can perform competitively against the Variational Inference posterior assuming the correct
22

---

Under review as a conference paper at ICLR 2022
topic prior, particularly for relative small values, and outperform misspecified topie prior in almost every test scenario.
Document Type (a =
Method
LDA
CTM
PAM
LDA
0.8950 + 0.0424
0.8325 = 0.0382
0.8625 = 0.0339
CTM
0,9050 = 0.0410
0.9125 = 0.0283
0.8150 = 0.0354
PAM
0.8600 + 0.0481
0.7125 + 0.0382
0,9050 + 0.0325
SSL-6M (ours)
0.9050 = 0.0406
0.9175 = 0.0275
0.9025 = 0.0330
SSL-120K (ours)
0.9150 = 0.0387
0.9100 = 0.0292
0.9200 = 0.0289
Document Type (a = 3)
Method
LDA
CTM
PAM
LDA
0.8750 = 0.0452
0.7325 = 0.0396
0.7875 # 0.0382
CTM
0,8800 # 0.0450
0,8800 # 0.0325
0.6975 # 0.0410
PAM
0.8550 + 0.0488
0.6075 = 0.0354
0.8825 + 0.0367
SSL-6M (ours)
0.8750 = 0.0458
0.8900 = 0.0311
0,8950 # 0.0344
SSL.- (ours)
0.8700 = 0.0466
0.8900 = 0.0312
0.8950 + 0.0357
Document Type (a = 5)
Method
LDA 0,8800 # 0.0452
CTM
PAM
LDA
0,7575 # 0.0354
0.7375 + 0.0410
CTM
0.8750 + 0.0452
0.8975 + 0.0283
0.5925 + 0.0396
PAM
0.8500 = 0.0495
0.5675 = 0.0297
0.8875 # 0.0368
SSL-6M (ours)
0.9000 + 0.0416
0.8975 + 0.0296
0,8675 + 0,0407
SSL-1205 (ours)
0.8800 + 0.0450
0.8900 = 0.0303
0.8750 + 0.0390
Document Type (a = 7)
Method
LDA
CTM
PAM
LDA
0.9250 + 0.0368
0.7175 + 0.0396
0.6875 + 0.0410
CTM
0.9300 # 0.0354
0.8700 # 0.0382
0.5700 # 0.0354
PAM
0.9050 # 0.0410
0.3400 = 0.0325
0.8750 # 0.0396
SSL-6M (ours)
0.9150 + 0.0387
0.8675 = 0.0383
0.8675 & 0.0407
SSL-120K (ours)
0.9350 = 0.0342
0.8775 # 0.0370
0.8725 # 0.0398
Document Type (a 9)
Method
LDA
CTM
PAM
LDA
0.9050 = 0.0410
0.7250 = 0.0396
0.6975 + 0.0410
CTM
0.9200 = 0.0382
0.8725 # 0.0339
0.5425 # 0.0368
PAM
0.8750 + 0.0452
0.5300 + 0.0325
0,8425 + 0.0452
SSL-6M (ours)
0.9100 = 0.0397
0.8850 # 0.0330
0.8325 = 0.0461
SSL-120K (ours)
0.8600 # 0.0481
0.8800 # 0.0341
0.8425 + 0.0430
Table C.3: Major topic recovery rate of our approach versus posterior inference via Variational Inference assuming a specific prior for a = 1,3, 7, We report the 95% confidence interval.
C.4 THE EFFECT OF TRAINING EPOCHS
We measure how the number of training epochs may influence topic posterior recovery loss by varying the number of training epochs. The results we present in Section 5 are based models trained for 200 epochs for all types documents. Here, we present the topic recovery loss, measured the Total Variation distance between recovered topic posterior and the true topic posterior, for epochs=25, 50, 100, 200 using the same model architecture and same sampling scheme as in Section 5
Table C.4 shows that topic posterior recovery loss steadily decreases as the number of epochs gets larger. Interestingly, even with just training 50 epochs, our model recover the topic posterior
23

---

Under review as a conference paper at ICLR 2022
within a Total Variation distance of less than 0.3 off the true topic posterior for documents generated from all four topic models.
Dirichlet hyperparameter a
Document type
Number of epochs
3
5
7
Pure-topic
25
0.0505
0.1025
0.1756
0.6222
0.8285
50
0,0310
0.0744
0.0991
0.1522
0.2873
100
0.0160
0.0363
0,0511
0.0911
0.1257
200
0.0148
0.0308
0.0501
0.0391
0.0517
LDA
25
0.1236
0.1583
0.1812
0.2022
0.2218
50
0.0967
0.1428
0.1387
0.1953
0.2117
100
0,0805
0.1018
0.1101
0.1361
0.1538
200
0.0709
0.0951
0.1045
0.1222
0.1377
CTM
25
0.1089
0.1599
0.1563
0.1747
0.1971
50
0,0900
0.1132
0.1425
0.1455
0.1501
100
0.0623
0.0948
0.1102
0.1187
0.1243
200
0.0550
0.0799
0.0970
0.1071
0.1101
PAM
25
0.1072
0.1342
0.1480
0.1688
0.1754
50
0,0820
0.1036
0.1246
0.1185
0.1395
100
0.0543
0.0755
0.0859
0.1087
0.1095
200
0.0436
0.0659
0.0787
0.0953
0.0971
Table C.4: Topic posterior recovery loss of our self-supervised learning approach, measured in Total Variation distance, for all four types of documents and a 1,3, 5, 7.9. The number of training epochs ranges from 25 200, and sample 60K new training documents in every epochs, which corresponds to 720K to 6M training documents.
C.5 HYPERPARAMETER TUNING EXPERIMENTS
As described in Section the two types of neural network architecture we use are fully-connected neural networks and attention-based neural networks (see Figure In this section, we present some of our TV results of different combinations of hyperparameters for both types of models. Note that our attention-brut based neural network slightly differs from typical transformers (Vaswani et al., 2017) in that use batch normalization instead of layer normalization in every residual connection. We only use one attention head in each multi-head attention layer. During training, we AMSGrad optimizer and initial learning rate of 0.0001 0.0002. We reduce the learning rate by 50% whenever validation loss does not decrease in 10 epochs.
The main hyperparameters in fully-connected neural networks include the hidden dimension, the number of layers, and whether we apply residual connections. Table|C.5 shows the TV distance results when we vary these hyperparameters for the LDA scenario when connected neural networks perform poorly in CTM and PAM case. For instance, when = 1 and hidden dimension 4096, we find that hyperparameters that work well in the LDA case longer yield satisfactory results (Table C.6).
1. However, fully-
TV
Hidden dimension
layers (residual)
1024
2048
4096
0.1509
0.1112
0.95
0.7902
0.7893
0.7901
(residual)
0.0871
0.0867
0.0862
(residual) 0.0822
0.0835
0.0709
Table C.5: Fully-connected neural network's 's performance in the LDA = 1 scenario.
For attention-based neural networks applied to CTM and PAM scenarios, mainly consider the number layers and the hidden dimension per attention layer. Table C.7 presents the TV distance between recovered topic posterior and true topic posterior in the CTM setting, with varying number of layers and attention dimension for & 1,3, 5 and using a more coarse estimate to the true topic posterior than what
our final true CTM topic posterior. We find that the model performs
24

---

Under review as a conference paper at ICLR 2022
Output word distribution
Addd Norm
Forward
Multiheed Attertionn
Add Norm
Food Forward
Add Narm
Mult-head
Embedding
Input document
Figure C.3: Our attention based neural network architecture
TV
Layer Type
layers
regular
residual
0.1824
0.1665
0.1724
0.1656
0.1700
0.1556
Table C.6: Fully-connected neural network's performance the CTM a scenario with 4096 hidden dimensions.
the best when attention dimension 768 or 1024, and when the attention dimension increases to 2048 the model gives higher TV (see Table C.8
TV
layers
Attention dimension
4
768
0.0902
0.0844
0.0814
1024
0.0851
0.0890
0.0836
3
768
0.1471
0.1429
0. 1398
1024
0.1440
0.1384
0.1383
5
768
0.1794
0.1767
0.1708
1024
0.1878
0.1787
0.1782
Table C.7: Attention preetion-bbsed neural network's performance on CTM documents for a = -,,,5 5.
TV
Attention dimension
a
768
1024
2048
3
0.1471
0.1440
0.1670
5
0.1794
0.7878
0.1922
Table C.8: 4-layer attention-base neural network's performance on CTM documents when attention layer's dimension varies, for = 3, 5.
25

---

Under review as a conference paper at ICLR 2022
C.6 TOPIC POSTERIOR RECOVERY LOSS AND CONDITION NUMBER
We have proved in Theorem that the upper bound of topic posterior recovery loss depends on the e Condition Number x( =(A)). In this section, we show that x(At) is small for every topic matrix A in our experiments (Figure C.4,. Note that we use the same topic matrix for pure-topic model and the LDA model. For CTM and PAM, we use the same topic matrix up to reordering of the topics that
corresponds with pairwise topic correlations. Therefore, pure-topic model and LDA model share the same x(A b(AT), and CTM and PAM share the same k: (At).
30-
Agre
Apho
Figure C.4: & Condition Number for topic matrix A
C.7 VISUALIZING TOPIC CORRELATIONS BETWEEN MODELS
For documents generated by the PAM model, we plotted the estimated posteriors by the self- supervised approach and posterior inference assuming different priors in Figure C.5 From this figure (especially in Documents and 4), one can qualitatively see that the estimated posterior and the PAM MCMC posteriors are aware of the correlation between topics, while the LDA MCMC posterior fails to take the correlation into consideration and hence more different from the ground truth topic proportions.
Estimated posteriors
PAM MCMC posteriors
Deca
Ground truth topic proportions
MCMC posteriors
Zakmet22
Figure C.5: Comparison of estimated, PAM MCMC, LDA MCMC, and ground truth posterior topic proportions for sample test documents generated by PAM. Overall, across our set of 200 test docu- ments, the estimated posteriors were able to outperform the mis-specified LDA posteriors.
26

---

Under review as a conference paper at ICLR 2022
D REAL-DATA EXPERIMENT ADDITIONAL DETAILS
In this section, we give more details about our real data experiments in Section 6 In Section D.I we describe the data processing. We give more detailed description of baselines and how extracted our representations in Section D.2 In Section D.3 we report more results using different model architectures.
D.1 DATA PROCESSING
Here we detailed our usage of the AG news dataset Zhang et al. (2015). Each category has 30,000 samples in the training set and 19,000 samples in the testing set. We first preprocessed the data by re- moving punctuation and words that occurred in fewer than 10 documents, obtaining in vocabulary of around 16,700 words, in a similar fashion done by Tosh al. (2020).
To split the data set into unsupervised dataset and supervised dataset. For each category of doc- uments, we selected random sample of 1000 documents as labeled supervised dataset while the remaining 116,000 documents fall into unsupervised dataset for representation learning.
D.2 EXTRACTING REPRESENTATION
To give some details about two baseline representations we used, we described in details as follows:
Bag of Words (BOW): For single document, constructs BOW embedding by creating a bag-of-words frequency vector of the dimension of vocabulary size, where each entry i represent the frequency of words with id in that particular document.
Word2vec: To generate word2vec representation fitted the Skip-gram word embedding model on unsupervised dataset (Mikolov et al. (2013)). The implementation done through the Gensim library (Rehurek & Sojka (2011)), where used an emhedding di- mension of 300 and window size 3. Representation is taken as the average of all trained word embeddings in single document.
For our own self-supervised method, to extract a representation, we attempted at 1) softmax+last layer: apply Softmax directly to the last layer output, 2) word2vec+last layer: apply an word2vec embedding matrix to last layer output after Softmax to reduce dimension (the word2vec matrix is trained over unsupervised dataset), and 3) seflmaax+seconlllas layer: apply Softmax function on top of the second-to-las layer (equivalent applying an identity matrix to replace the last layer). The dimension of representation with word2vec is 300 while the softmax + second2 last layer has dimension of 4096. The original last layer representation has dimension of the vocabulary size (around 6, 700), and has shown inferior results than those with dimension reduction techniques.
We included comparison of the two approaches with dimension reduction in Tab from which we observe that Softmax on the second-to- -ocond-to-last layer output performed better across layers of 3,4 and 5. We reported the result using the sofmaxx layer in Figure
Test Accuracy
Method
# layers (residual) word2vec+ layer
to layer
0.8508
0.8714
4
0,8450
0.8621
5
0.8492
0.8689
Table D.I: 1: Test Accuracy for different representation extraction method. We fixed the rest of hyper- parameters be: 5000 embedding dimension, 150 epochs, 0.0002 learning rate, sampling 4 words in labels, weight decay of 0.01 and resample rate 2.
D.3 MODEL ARCHITECTURE
We include here more details model architecture used in real data experiments. We used residual model, and tested two main hyperparameters: number residual blocks and hidden dimen- sion size. We fixed the rest of hyperparameters: used 5000 embedding dimension and trained
27

---

Under review as a conference paper at ICLR 2022
Conpursse
0N
0.00
011
200
(a)
(b)
Naccal Uipel cacqua 428)
6.02
(c)
Figure D.I: RBL representation performance varies with different residual model capacity: the best performing is the one with layers and 4096 hidden dimensions, and it was what we used for Figure In all these runs, use Softmax second-to-last layer to obtain our representations
for 150 epochs, with 0.0002 learning rate, weight decay of 0.01 and resample rate of We sampled 4 words in labels for variance reduction purpose.
Varying Depth and Width We investigated both the effect model depth and width on the perfor- mance of RBL representation. We trained networks of width 2048. 3000, and 4096 nodes respec- tively, where the number of node refers to the number node in the linear layer inside residual block. We varied the depth of the network by choosing place 3, 4 and of such block.
As shown in Figure D.I appears that using wider models in the unsupervised phase leads to better performance when training linear classifier on the learned representations. Number of residual block does not seem to have a clear relationship with test accuracy from limited experiments we ran. The best accuracy is achieved when have residual blocks with dimension of 4096, which is what we used to generate results in Figure 2
28


Reviews:
[{"review_id": "jXDeexGk0C", "reviewer": "Reviewer_b3P7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The main strength of this paper lies in the thorough theoretical analysis and corresponding proofs. The authors not only prove that a new reconstruction-based objective can also extract posterior topic information for a general topic model, but also strengthen the guarantee for contrastive objective in Tosh et al. (2020) by removing some of their assumptions and the necessity of landmark documents. \n\n- While the theoretical analysis is interesting, objective and motivation of this paper is not entirely clear and convincing. It seems there is lack of outlook for future work where this analysis could be useful, and the empirical study on real data is also somewhat weak.  There is no comparison with more recent self-supervised and contextual document representation work, such as BERT mentioned in the beginning of the paper. Actually this is also a common issue in both Arora et al. (2019) and Tosh et al. (2000), which are two most closely related papers.\n    - The LDA “is actually doing very poor on several ‘objectively’ evaluable predictive tasks”, because “it is not designed, nor trained for such tasks, such as classification, there is not warrantee that the estimated topic vector is good at discriminating documents\" (http://www.cs.cmu.edu/~epxing/talks/ACL2.pdf).\n    - BOW, word2vec and NCE are weak base lines far behind more recent self-supervised and contextual representation models for document classification.\n    - The experiments are not conducted on a fair basis. For example, word2vec is a light-weighted neural network with only one layer, but this paper implements the proposed model using some deep neural networks including 8 heavy-weighted Transformer blocks, which still demonstrate no significant difference. \n\n- typos\n    - \"constrastive\" in Page 1\n", "summary_of_the_paper": "The main strength of this paper lies in the thorough theoretical analysis and corresponding proofs. The authors not only prove that a new reconstruction-based objective can also extract posterior topic information for a general topic model, but also strengthen the guarantee for contrastive objective in Tosh et al. (2020) by removing some of their assumptions and the necessity of landmark documents. ", "summary_of_the_review": "The theoretical analysis is interesting, but the motivation of analysis and evaluations are not entirely convincing. I suggest to focus on theoretical analysis and clarify where this analysis could be useful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wa9BYjk2Dkr", "reviewer": "Reviewer_KozZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n---------\n+ Takes on an interesting and timely topic (self-supervised learning for probabilistic models)\n+ Broad scope of objectives, looking at both \"reconstruction\" and \"contrastive\" learning\n+ Broad scope of topic models, considering a \"general\" family that includes LDA, CTM, and Pachinko Allocation as special cases\n+ Seems to extend the \"nearest\" previous work (Tosh et al. 2020) meaningfully, focusing on reconstruction objectives not just contrastive learning\n\nWeaknesses\n----------\n\nI list 5 concerns here, with detailed discussion and questions for the authors below\n\n- W1: While theorems suggest \"existence\" of a linear transformation that will approximate the posterior, the actual construction procedure for the \"recovered topic posterior\" is unclear\n\n- W2: Many steps are difficult to understand / replicate from main paper\n\n- W3: Unclear what theorems can say about finite training sets\n\n- W4: Justification / intuition for Theorems is limited in the main paper\n\nResponses to W1-W3 are most important for the rebuttal.\n\n\n## W1: Actual procedure for constructing the \"recovered topic posterior\" is unclear\n\nIn both synthetic and real experiments, the proposed self-supervised learning (SSL) method is used to produce a \"recovered topic posterior\" p( w | x). However, the procedure used here is unclear... how do we estimate p( w | x) using the learning function f(x)?\n\nThe theorems imply that a linear function *exists* with limited (or zero) approximation error for any chosen scalar summary of the doc-topic weights w. However, how such a linear function is constructed is unclear. The bottom of page four suggests that when t=1 and A is full rank, that \"one can use the pseudoinverse of A to recover the posterior\", however it seems (1) unclear what the procedure is in general and what its assumptions are, and (2) odd that the prior may not needed at all.\n\n*Can the authors clarify how to estimate the recovered topic posterior using the proposed SSL method?*\n\n\n## W2: Many other steps are difficult to understand / replicate from main paper\n\nHere's a quick list of questions on experimental steps I am confused about / would have trouble reproducing\n\nFor the toy experiments in Sec. 5:\n\n- Do you estimate the topic-word parameter A? Or assume the true value is given?\n- What is the format for document x provided as input to the neural networks that define f(x)? The top paragraph of page 7 makes it seem like you provide an ordered list of words. Wouldn't a bag-of-words count vector be a more robust choice?\n- How do you set t=1 (predict one word given others) but somehow also use \"the last 6 words are chosen as the prediction target\"?\n- How do you estimate the \"recovered topic posterior\" for each individual model (LDA, CTM, etc)? Is this also using HMC (which is used to infer the ground-truth posterior)?\n- Why use 2000 documents for the \"pure\" topic model but 500 in test set for other models? Wouldn't more complex models benefit from a larger test set?\n\nFor the real experiments in Sec. 6:\n\n- How many topics were used?\n- How did you get topic-word parameters for this \"real\" dataset? \n- How big is the AG news dataset? Main paper should at least describe how many documents in train/test, and how many vocabulary words.\n\n\n## W3:  Unclear what theorems / methods can say about finite training sets\n\nAll the theorems seem to hold when considering terms that are expectations over a known distribution over observed-data x and missing-data y. However, in practical data analysis we do not know the true data generating distribution, we only have a finite training set.\n\nI am wondering about this method's potential in practice for modest-size datasets. For the synthetic dataset with V=5000 (a modest vocabulary size), the experiments considered 0.72 million to 6 million documents, which seems quite large. \n\n*What practically must be true of the observed dataset for the presented methods to work well?*\n\n\n## W4: Justification / intuition for Theorems is limited in the main paper\n\nAll 3 theorems in the main paper are presented without much intuition or justification about why they should be true, which I think limits their impact on the reader. (I'll try to wade thru the supplement, but did not have time before the review deadline). \n\nTheorem 3 tries to give intuition for the t=1 case, but I think could be stronger: why should f(x) have an optimal form $p( y=v_1 | x)$? Why should \"these probabilities\" have the form $A E[ w | x]$? I know space is limited, but helping your reader figure things out a bit more explicitly will increase the impact.\n\nFurthermore, the reader would benefit from understanding how tight the bounds in Theorem 4 are. Can we compute the bound quality for toy data and understand it more practically? \n\n\nDetailed Feedback on Presentation\n---------------------------------\n\nNo need to reply to these in rebuttal but please do address as you see fit in any revision\n\nPage 3:\n- \"many topic models can be viewed\"... should probably say \"the generative process of many topic models can be viewed...\"\n\n- the definition of A_ij is not quite right. I would not say \"word i \\in topic j\", I would say \"word i | topic j\". A word is not contained in a topic, Each word has a chance of being generated.\n\n- I'd really avoid writing $\\Delta(K)$ and would just use $\\Delta$ throughout .... unclear why this needs to be a function of $K$ but the topic-word parameters (whose size also depends on $K$) does not\n\n- Should we call the reconstruction objective a \"partial reconstruction\" or \"masked reconstruction\"? I'm used to reconstruction in an auto-encoder context, where the usual \"reconstruction\" objective is literally to recover all observed data, not a piece of observed data that we are pretending not to see\n\n- In Eq. 1, are you assuming an ordered or unordered representation of the words in x and y?\n\nPage 4:\n\n- I would not reuse the variable y in both reconstruction and contrastive contexts. Find another variable. Same with theta.\n\nPage 5:\n\n- I would use $f^*$ to denote the exact minimizer, not just $f$\n\n\nFigure 2 caption should clarify:\n\n- what is the takeaway for this figure? Does reader want to see low values? Does this figure suggest the approach is working as expected?\n- what procedure is used for the \"recovered\" posterior? Your proposed SSL method? \n- why does Pure have a non-monotonic trend as alpha gets larger?\n\n", "summary_of_the_paper": "This paper considers the problem of performing posterior inference for probabilistic topic models. A \"general\" topic model consists of two key parts:\n\n- $A$ : the topic-word probability matrix\n- $\\Delta$ : the prior distribution over doc-topic probability vector w\n\nGiven these parts and a new document x, the goal is to infer the posterior over w: $p(w | x)$. \n\nIt is well-known that this posterior can be estimated in several ways using standard approximate Bayesian inference methods: MCMC, variational, etc. The detailed steps of these methods are usually customized to the choice of prior.\n\nThis paper suggests a new possibility: that a learned transformation -- denoted f(x) -- trained using a so-called *reconstruction* loss (see Eq 1), can be then translated into the desired posterior via a linear function. Surprisingly, the paper suggests that learning f(x) does not depend on the prior \\Delta, so the big idea is that this representation is robust to misspecification of the prior.\n\nConcretely, the contributions of the paper seem to be:\n\n1) Theorem 3, which claims that there exist linear weights \\theta that, when applied to the vector produced by the ideal learned transformation f(x) that minimizes the reconstruction objective (Eq 1), can exactly equal the expected value of any polynomial summary function of the doc-topic probability vector w under the posterior.\n\n2) Theorem 4, which claims that even if f is not an exact minimizer, if it is within an additive tolerance \\epsilon of the ideal loss, then there exist again weights \\theta such that the squared error between the true posterior summary and the linear function has bounded error for all documents.\n\n3) Theorem 5, which looks at the contrastive objective in Eq 2, and shows that there exists a linear function that can recover the expected polynomical summaries of random variable w under the posterior.\n\nExperiments in Sec. 5 look at toy data, with results in Fig 2 and Table 1. Essentially, these results show that the proposed SSL method produces lower error posterior estimates (in terms of  total-variation distance from ground truth) than using Bayesian inference with the wrong prior. \n\nExperiments in Sec. 6 look at real data (the AG news dataset), comparing the representations learned by SSL at a downstream classification task to baselines of bag-of-words, word2vec, and the contrastive method of Tosh et al. Fig. 3 suggests that the proposed method offers a few percentage points absolute gain in accuracy over word2vec, and is similar to the previous NCE method (Tosh et al) in accuracy.", "summary_of_the_review": "Overall I think the direction of the paper is interesting, but the main paper at present is missing some key pieces (esp. how the recovered topic posterior is produced once the SSL representation f is learned). In addition, I have concerns about the disconnect in how theorems might apply to finite training sets in practice, and how little intuition was provided in the paper for the theory. \n\nI could be persuaded to change my mind by a strong rebuttal.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GSsuBCZeQlr", "reviewer": "Reviewer_HibN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe main results here seem novel and potentially useful. The proposed method is a new way to perform efficient approximate inference for topic models with some guarantees. \n\nCompared to previous work the authors provide a new self-supervised learning technique for topic models with its own theory.\n\nThe synthetic experiments show that the approach is useful when a misspecified model is used.\n\nThe real data experiments show that this self supervised learning technique is useful for feature extraction.\n\nWeakness:\n\nI feel that some of the results of this work could use further explanation and that some of the experiments are confusing or underwhelming:\n\nI do not feel I was able to devote adequate time to understanding the details of the results shown in section 3 and I’d like to continue looking in to it. For the third paragraph of section 3, could the authors give additional intuition for how this approach gives the correct expectation of the posterior seemingly without knowledge of the prior p(w)?\n\nFigure 1 provides very little information, as it seems to only reflect a single synthetic document. I’m not sure what I’m supposed to take away from it.\n\nFigure 2 seems like a poor way to present this information. Even simple table would probably be more interpretable and precise.\n\nUnder: Robustness of self-supervised learning. What is the “traditional topic inference” used? Simply the sampling approach discussed above? It is also strange that the results are mostly shown comparing across misspecified models. I would expect to see comparisons between other types of inference (MAP, variational) using correctly specified models. Comparing explicitly misspecified models for synthetic data feels a bit contrived. \n\nWhy are the results on real data only about semi-supervised learning? This feels like a bit of a non-sequitur given the rest of the paper. I would expect to see at least qualitative results showing inference with this method on real data.\n\nThere has been quite a bit of prior work on supervised and semi-supervised topic models. I would expect at least some of these methods to be represented in section 6. Some examples being: https://proceedings.neurips.cc/paper/2007/file/d56b9fc4b0f1be8871f5e1c40c0067e7-Paper.pdf, https://www.jmlr.org/papers/volume13/zhu12a/zhu12a.pdf, http://proceedings.mlr.press/v84/hughes18a.html\n\nGenerally the focus of this paper is on inference. Can this approach be applied within a learning algorithm for topic models?", "summary_of_the_paper": "This paper introduces a new method for performing inference on topic models, based on self supervised learning. Their approach is generalizable to multiple types of topic models. Specifically, the authors show that the expectation of some polynomial function of the topic posterior will be a linear function of the output of function that optimizes the self-supervised learning objective. The authors explore 2 types of self-supervised learning objectives, one based on reconstruction and one based on contrastive learning that was explored in prior work. The authors provide proofs for their main result as well a result robust to approximate minimization of the self-supervised objective. In their experiments the authors show that their approach outperforms inference with a misspecified model. The authors also run semi-supervised learning experiments on their approach.\n", "summary_of_the_review": "Overall the results in this work are interesting and possibly useful, but I feel that there are some lingering questions about applicability, particularly as the experimental results are underwhelming.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5ln0Snfk8B", "reviewer": "Reviewer_HZxy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Topic-word weight matrix $A$: It is known that one advantage of topic models is that each latent topic can be represented as a set of words. The topic-word weight matrix is learnt together with the posterior of the topic proportion vector $w$. All the theorems assume that $w$ is a condition, and it seems that $\\theta$ is the left pseudo-inverse of $A$. Looking at existing topic models, either probabilistic or neural, it is not clear to me how $A$ is learned by the network models used in the experiments.\n* Neural topic modelling: How is this work related to neural topic models based on autoencoding variational inference, for instance, by Srivastava and Sutton 2017? It might be good to see how a neural topic model can recover the topic posterior.\n* The synthetic experimental results seem to say that the reconstruction-based objective can recover the posterior of $w$ given a document. What is the implication of this from a topic modelling practitioner's point of view? Can one use the reconstruction-based objective with neural networks in place of topic models? However, it is known that one advantage of topic models, particularly the probabilistic ones, is that they can learn interpretable topics. Besides the approximation of the posterior of $w$, I cannot see how the proposed theorems could benefit the topic modelling community. \n* It is an interesting statement that \"attention-based architecture performs the best for recovering topic posterior distribution for CTM and PAM documents\". I wonder if the authors can discuss this more, like the possible reasons.\n* The experiments on the real dataset seems to show that the self-supervised objectives can learn useful representation for document classification. Again, I would like to see how neural topic models perform in this scenario. Furthermore, compared with the other embedding method, like the language models, what is the advantage of this work? ", "summary_of_the_paper": "This paper shows theoretically and empirically that self-supervised objectives can be used to extract useful information about the posterior of the topic proportion vector given a document, regardless of the underlying models. It extends the findings of Tosh et al.2020 about contrastive objectives to reconstruction-based objectives. It is interesting to see that one simple reconstruction objective recover the posteriors generated from the different probabilistic topic models on the synthetic dataset.", "summary_of_the_review": "Overall, showing that both the reconstruction-based objective and the contrastive object can recover the posterior of the topic proportion vector is interesting. It would be good to demonstrate how this theoretical finds can benefit the wide topic modeling community with more experiments on real-world datasets. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper generated a large amount of discussion.  Three reviewers were marginally above and one marginally below.  The paper presents an intriguing relationship between self-supervised learning and topic model inference that extends earlier work of Tosh.  The result seems to be subtle because there was considerable discussion with the authors wrapping up with a reminder of what the main goal is:  SSL can achieve the state-of-the-art performance for topic inference problem, moreover (main goal) SSL can be oblivious to the specific topic model.  This is indeed intriguing.  But with all the discussion, and one persistent negative reviewer, I feel the paper needs to be polished.  Given the theorem gives a testable statement, I don't see why experimental results cannot be done for 4 different real data sets, to give us more confidence.
Decision: Reject