Understanding The Robustness of Self-supervised Learning Through Topic Modeling
Authors: Zeping Luo
Shiyou Wu
Cindy Weng
Mo Zhou
Rong Ge
Venue: ICLR 2023 poster (2023)
Keywords: deep learning theory
self-supervised learning

Abstract: Self-supervised learning has significantly improved the performance of many NLP tasks. However, how can self-supervised learning discover useful features, and why is it better than traditional approaches such as probabilistic models are still largely unknown. In this paper, we focus on the context of topic modeling and highlight a key advantage of self-supervised learning - when applied to data generated by topic models, self-supervised learning can be oblivious to the specific model, and hence is less susceptible to model misspecification. In particular, we prove that commonly used self-supervised objectives based on reconstruction or contrastive samples can both recover useful posterior information for general topic models. Empirically, we show that the same objectives can perform on par with posterior inference using the correct model, while outperforming posterior inference using misspecified models.

Published as a conference paper at ICLR 2023
UNDERSTANDING THE ROBUSTNESS OF SELF- SUPERVISED LEARNING THROUGH TOPIC MODELING
Zeping Luo', Shiyou Wu", Cindy Weng", Mo Zhou', Rong Ge
Duke University, USA
(zeping. luo, sh you. wu, cindy weng) @duke. edu, (mozhou, rongge Bcs duke.edu
ABSTRACT
Self-super learning has significantly improved the performance of many NLP tasks. However, how can self-supervised learning discover useful representations, and why is it better than traditional approaches such as probabilistic models are still largely unknown. In this paper, focus on the context of topic modeling and highlight a key advantage of self-supervised learning when applied to data generated by topic models, self-supervised learning can be oblivious the specific model, and hence is less susceptible to model misspecification. In particular, we prove that commonly used self-supervised objectives based on reconstruction or contrastive samples can both recover useful posterior information for general topic models. Empirically, show that the same objectives can perform on par with posterior inference using the correct model, while outperforming posterior inference using misspecified models.
INTRODUCTION
Recently researchers have successfully trained large-scale models like BERT (Devlin et al. 2018) and GPT (Radford et al. 2018). which offers extremely powerful representations for many NLP tasks (see e.g.. Liu et al. (2021) Jaiswal et al.) (2021) and references therein). To train these models, often one starts with sentences
large text corpus, mark random words as "unknown" and ask the neural network to predict the unknown words. This approach is known as self-supervised learning (SSL).
Why can self-supervised approaches learn useful representations? To understand this we first need to define what are "useful representations". A recent line of work (Tosh et al., 2021a Wei et al. 202T) studied self-supervised learning the context of probabilistic models: assuming the data generated by probabilistic model (such topic model or Hidden Markov Model), one define representation of observed data as the corresponding hidden variables in the model (such as topic proportions in topic models hidden states in Hidden Markov Model). These works show that self-supervised learning approach is as good as explicitly doing inference using such models.
This approach naturally leads to the next question why can sele-super learning perform better than traditional inferencing based on probabilistic models? In this paper we study this question the context of topic modeling, and highlight one key advantage for self-supervised learning: robustness to model misspecification.
Many different models (such Latent Dirichlet Allocation (LDA) Blli et al. 2003 , Correlated Topic Model (CTM) (Blei & Lafferty| 2007). Pachinko Allocation Model (PAM) (Li & McCallum )006 have been applied in practice. Traditional approaches would require different ways of doing inference depending on which model is used to generate the data. On the other hand, we show that no matter which topic model is used to generate the data, if standard self self-supervised learning objectives such as the aeconstruction-based objective]] or the contrastive objective" can be minimized, then they will generate representations that contain useful information about the topic proportions of a document. Self-supervised learning is oblivious to the choice of the probabilistic model, while the traditional approach of probabilistic modeling depends highly on the specific model. Therefore, one
"quual contribution.
1 See Equation (1. similar to the objective used in Pathak et al. 2016 Devlin et al.
2018
"See Equation this was also used in Tosh (022a))

---

Published as a conference paper at ICLR 2023
would expect self- sepf-supe learning to perform similarly to inferencing with the correct model, and outperforms inferencing with misspecified model.
To verify our theory, we run synthetic experiments to show that self--upe learning indeed outperforms inferencing with misspecified models. Unlike large-scale models, our self-supervised learning is applied in the much simpler context of topic models, but we also demonstrate that even this simple application can improve over simple baselines on real data.
1.1 RELATED WORKS
Sell-Supervised Learning Self-supervised learning recently has been shown be able learn useful representation, which is later used for downstream tasks. See for example Bachman et al. (2019) :
Caron et al. (2020); Chen et al. (2020a c Grill et al. (2020) Chen & He (2021); Tian et al. (2020a) He et al. (2020) and references therein. In particular, Devlin et al. (2018) proposed BERT, which shows that self-supervised learning has the ability to train large-scale language models and could provide powerful representations for downstream natural language processing tasks.
Theoretical Understanding of Self-Supervised Learning Given the recent success of self-
supervise learning, many works have been tried to provide thcoretical understanding on contrastive
learning (Arora et al., 2019; Wang & Isola 20200 Tosh et al., 202 Tian et al.) 2020b HaoChen
et al.
2021: Wen & Lif L022 Zimmermann et al. (021)) ) and Deconstruction-based learning (Lee
et al.
2020: Saunshi et al. [2020] Teng & Huang 2021). Also, several papers considered the problem
from multi-view perspective (Tsai et al.) 2020 Tosh et al.] 2021b), which covers both contrastive
and reconstruction-basse
learning. Moreover, Wei et al. (2020) and Tian et al. (202T) studied the
theoretical properties of self- trainini and the contrastive learning without the negative pairs respec- tively. (Sannshi et al. (2020) investigated the benefits of pre-trained language models for downstream tasks. Most relevant to our paper. Tosh et al. (2021a) considered the contrastive learning in the topic models setting. Our theoretical results extend their theory to reconstruction--asse
objective (while also removing some assumptions for the contrastive objective), and our empirical results show that the reconstruction--aset objective can be effectively minimized.
Theoretical Analysis of Topic Models Many works have proposed provable algorithms to learn topic models, such methods of moment based approaches (Anandkumar et al. 2012 2013 2014 2015) and anchor word based approaches (Papadimitriou et al. 2000 Arora et al. 2012 2016a Gillis & Vavasis 2013 Bittorf et al. 2012), Much less is known about provable inference for topic models. Sontag & Roy (201)) showed that MAP estimation can be NP-hard even for LDA model. Arora et al. (2016b) considered approximate inference algorithms.
1.2 OUTLINE
We first introduce the basic concepts of topic models and our objectives in Section Then in Section prove guarantees for the reconstruction-baat based objective. Section connects the contrastive objective to reconstruction--asee objective which allows us to prove a stronger guarantee for the former. We then demonstrate the ability of self-supervised learning to adapt to different models by synthetic experiments in Section Finally, we also evaluate the reconstruction based objective on real-data to show that despite the simplicity of the topic modeling context, extracts reasonable representations in Section D
2 PRELIMINARIES
In this section we first introduce some general notations. Then we briefly describe the topic models we consider in Section 2.1 Finally define the self-supervised learning objectives in Section 2.3 and give our main results.
Notation We use [n] to denote set (1,2...., n). For vector Rd, denote || as its le norm and ||=]| as its & norm. For matrix A E Rmmx,, we use A; E Rm to denote its i-th column. When matrix A has full column rank, denote its left pseudo-inverse as At = (A A For matrix or general tensor T xddxx--xxx xd: we use vector vec(I") E .d to represent vectorization. Let
S. = {a E R > D4 = 1, 2 0} to denote k - dimensional probability simplex. For two probability vectors p, q, define their total variation (TV) distance TV 9) Ipp all2 /2.
2

---

Published as a conference paper at ICLR 2023
2.1
TOPIC MODELS
Many topic models treat documents as a bag-of-words and use a two-step procedure of generating a document: first each document viewed proportions w), and each topic is viewed as a probability distribution over words. To generate a word, one first selects a topic based on the topic proportions w, and then samples word from that particular topic. Since the topic distributions are shared across the entire corpus, and the topic proportion vector w is specific to documents, it makes sense to define w as the representation for the document.
probability distribution of topics (often called the topic
More precisely, let V be finite vocabulary with size V and K. be set of K topics, where each topic a distribution over V. We denote the topic-word matrix as with A. = P(word i topic j), so that each column in A represents the word distribution of topic. Each document corresponds to a convex combination of different topies with proportions w. For each word the document, first sample a topic according to the topic proportions w, and then sample the word from the corresponding topic (cquivalently, one can also sample a word from distribution Aw). Different topic models differ in how they generate w, which we formulate in the following definition:
Definition 2.1 (General Topic Model), A general topic model specifies distribution A(K) for each number of topies K. Given a topic-word matrix A and A (K), to generate a document, one first sample w ~ A (K) and then sample each word in the document from the distribution Av.
Here A(K) is a prior distribution of w which crucial when trying to infer the true topic proportions given the words in document. Of course, different topic models may also specify different priors for the topic-word matrix A. However, given large number of documents generated from the topic model, in many settings one can hope to learn the topic-word matrix A and the prior distribution A(K) (see e.g., Arora et al. 2012 201 6a)), therefore we consider the following inference problem:
Definition 2.2 (Topic Inference). Given a topic-word matrix A, prior distribution A A(K) for topic pro- portions w, and a document v. the topic inference problem tries to compute the posterior distribution wl( A) given the prior w ~ A(K).
Note that our general topic model can capture many standard topic models, including pure topic model, LDA, CTM and PAM.
2.2 SEMI-SUPERVISER LEARNING SETUP
Our goal is to understand the representation learning aspect of self-supervised learning, where we are given large number of unlabeled documents and small number of labeled documents. In the unsupervised learning stage, the algorithm only has access to the unlabeled documents, and the goal is learn representation (a mapping from documents to feature vector). In the supervised learning stage, the algorithm would use the labeled documents to train a simple classifier that maps the feature vector to the actual label. We often refer to the unsupervised learning stage as "feature learning" and the supervised learning stage as "downstream application".
In the theoretical model, we assume that the label is simple function of the topic proportions The goal is to apply self-supervised learning on the unlabeled documents to get good representation, and then use this representation to predict the label y. Of course, if one knows the actual parameters of the model, the best predictor for y would be to first estimate the posterior distribution w and then apply the function that maps w label y. We will show that for functions that are approximable by low degree polynomials self-supervised learning can always provide a good representation.
2.3 SELF-SUPERVISED LEARNING
In general, self-supervised learning tries to turn unlabeled data into a supervised learning problem by hiding some known information. There are many ways achieve this goal (see c.g., Liu et al. (2021); Jaiswal et al. 22211). In this paper, we focus on two different approaches for self-supervised learning: reconstruction- based objective and contrastive objective. We first formally define the objectives and then discuss our corresponding result.
Reconstruction-Bann Based Objective One common approach of self-supervised learning is to first mask part of the data and then try to find a function ff to reconstruct the missing part given the unmasked part of input. This is commonly used for language modeling (Devlin et al. 2018 Radford et al.
3

---

Published as a conference paper at ICLR 2023
2018). In the context of topic modelling, suppose all the documents generated from underlying topic model with A and ((K). Then for any given document Tunsup- since each word is i.idd sampled (so that the order of words is not important), them as unknown, then we ask the learner to predict these t words given the remaining words in the document. Specifically, we split Tunsup into x: and y where y is the words that we select and let x is the document with these t words removed. We aim to select a predictor f that minimizes the following reconstruction objective:
pick t random words from the document and mark
min Licoons (f) a En (e(((t)) v)l.
where ((U,)) = - -yk log Uk is the cross entropy loss. Here we slightly abuse the notation to use y as an one-hot vector for V classes and f also outputs a probability vector on V². Depending on the context, we will use to denote either the actual next ( words or its corresponding one-hot label. Now we are ready to present our result for the reconstruction based objective:
Theorem 2.3. (Informal) Consider the general topic model setting as Desinition 2.1 suppose suppose,fuction function
f minimizes the reconstruction-ban basoddd objective ( Then, any polynomial P(w) of the posterior w (z, A) with degree at most can be represented by a linear function of fax).
The theorem shows that if we want to get basic information about posterior distribution (such as mean, variance), then it suffices to predict a small constant number of words (1 for mean and 2 for variance). See Theorem 3.1 for the formal statement.
Contrastive Objective Another common approach in self-supervised learning is the contrastive learning (He et al. 2020 Chen et al. 2020a). In contrastive learning, the training data is usually a pair of data x') with label y E {0, where label means (z, x²) ) is a positive sample (r and x' are similar) and label 0 means (2r, a²) is a negative sample (x: and x' are dissimilar). The task is to find function / such that it can distinguish the positive sample and negative sample, i.e., f(r x') = Formally, we want select predictor f such that it minimizes the following contrastive objective:
min Leontrast Egg yy (I x²) a'),
In the context of topic models, following previous work (Tosh et al., 2021a), we generate the data (ar, y) as follows. Suppose all documents generated from underlying topic model with A and A (K). We first generate a document from the word distribution Aw where w is sampled from A (K). Then, (i) with half probability we generate words from the same distribution Aw to form the document x' and set y = 1; (ii) with half probability we generate words from a different word distribution Aw with
A(K) (so that w F w') and set y = We consider the square loss (( v) ( 2 for simplicity.
We now give our informal result on contrastive objective. See Theorem 4.1 for the formal statement. Note that this theorem generalizes Theorem 3 in Tosh et al. (201a))
Theorem 2.4. (Informal) Consider the general topic model setting as Refinition2.1 suppose function functiof minimizes the contrastive objective (2). Then we can use I and enough documents to construct representation g(x) such that any polynomial P(w) of the posterior wxxx,A A) with degree at most t can be represented by a linear function of g(æ).
Advantage of SSL approach Note that neither objectives [ or (2) depend on the prior distribution A (K). so is possible optimize these without specifying specific model; on the other hand, even the definition of topic inference (Definition 2.2 relies heavily on A AKK).
3 GUARANTEES FOR THE DECONSTRUCTTON--ASSI OBJECTIVE
In this section, we consider the reconstruction-basee objective [ and provide theoretical guarantees for its performance. We first show that if such objective with unknown words can be minimized, then any polynomial of topic posterior w|xx,A) A) with degree at most t can be represented by linear function of the learned representation.
Theorem 3.1 (Main Result). Consider the general topic model setting Definition 2.7 suppose topic-word matrix A satisfies rank( A) = K and function f minimizes the reconstruction based objective [1. Then, for any document and its posterior (x. A), the posterior mean of any
4

---

Published as a conference paper at ICLR 2023
polynomial P(w) with degree most is linear in fz)). That is, there exists 8 E RV such that for all documents x
[L[((w) A] = 0 f(x).
To understand this theorem, first consider a warm-up example where = 1 (see details in Section appendix). Intuitively, this case the best way to predict the missing word for a given document is to estimate its topic proportion matrix. Therefore, if the output of self-supervised learning f(x) (which is a V-dimensional vector indexed by words) minimizes the loss, then f(x) must have the form f(z) AE [1|22, AE(w full rank multiplying by the pseudo-invers The proof for the general case of predicting t-words requires more careful characterization of the optimal prediction and its relationship to E., [Pw)) A], which we defer to Section A.2
and then predict the word using Aw where A is the topic-word
A]. When A of recovers the expectation of the posterior w (a, A).
The above discussion also suggests that though the representation dimension may seem to be V its "effective" dimension is in fact K which is much smaller than V See more details in Appendix A.4
Robustness for an approximate minimizer In Theorem 3.1 we focus on the case when function is exactly the minimizer of the reconstruction-basee objective ( i.e., Irecona (f) Lecconst miny Lrecons(()) However, in practice one cannot hope to find such a function exactly. In the following, provide robust version of Theorem such that it allows us find an approximate solution instead of the exact optimal solution.
To present our result, we need first introduce the following notion of condition number, which was used in many previous works, such as collaborative filtering systems (Kleinberg & Sandler (008)))(0008) and topic models (Arora al. (2066b). Intuitively, xs ()) measures how large vector would change after multiplying with B in the l norm sense.
Definition 3.2 (l Condition Number). For matrix B E |R define its & condition number k(B) as x<( () A min || Belll ||=|]] max4e[1] |B:ll1 where B; is the i-th column of B.
Let Woost E R be the topic posterior tensor for unknown words y (VI....,3t) given remaining document x. That is, for each entry [W [Woselle at or.. P(for all i, 2, is the topic of word yılz, A) = [w[w,, ... Wz, Ax and W Wpost = E (w x A Thus, any polynomial P(w) of degree most 1. can be represented as R.,,, [P,]((()) A vec(Wppoi) for some B, where sec(Wpost) E IR K" is the vectorization of Wpose-
We now are ready to present the robust version of Theorem 3.1 It shows if we only find a function I whose loss is at most larger than the optimal loss, then a linear transformation of our learned representation can still give good approximation of the target polynomial within O()) error.
Theorem 3.3 (Robust Version). Consider the general topic model setting Definition 2.7 suppose topic-word matrix A satisfies rank(A) (A) = K and function I satisfies Trecomed (f) S L Freconst + E for some
> O. Then, for any document x and its posterior w/(x, A), the posterior mean of any polynomial
((w) with degree at most is approximately linear in (xx). That is, there exists a O IR V* such that
Ex [Bw[P(w))zz A] f()))) < 1111 10 20 (AT)e,
where [P.))(( A] = 3 vec(W post).
Note that the dependency have if c = 0, i.e., the f we would have in Theorem 3.1 Thus, this quantity should be understood as the complexity of the target function for the downstream task. Empirically we show that s<(At) is small in Section C.6 The proof is deferred to Section A.3
|| (At) is expected, since this is the norm ||012 that would
4 GUARANTEES FOR THE CONTRASTIVE OBJECTIVE
In this section, consider the contrastive objective 2 for self sulf-super learning and provide similar provable guarantees on its performance as the reconstruction-bal based objective.
To simply the notations, WL.GG assume topic set K [K] so that we use topics 2, [K] indices.
5

---

Published as a conference paper at ICLR 2023
We use the same approach as Tosh al. (2021a) to construct representation g(z). Given set of landmark documents {4)== with length [4 as references, the representation is defined as
f(x,2')
g(a, (4) ((s
g(x,x') f(x, 2')
The following theorem gives the theoretical guarantee for this representation. Similar to Theorem 3.1 for reconstruction- based objective, it shows that any polynomial P(w) of degree most can be represented by a linear function of the learned representation. Our proof here relies on the observation that having 9 { 1.)a ==11 for all short landmark documents of length t gives similar information in the reconstruction-basse objective does. This observation allows us to remove the anchor words assumption needed in (Tosh al. 2021
Theorem 4.1. Consider the general topic model setting as Definition 2.1 suppose topic-wont matrix A satisfies rank( A) = K and function minimizes the contrastive objective (2). Assume we randomly sampled m Kt different landmark documents { and construct g(z. (1) )=)) as (3) Then, for any document and its posterior wXx A). the posterior mean of any polynomial P(w) with degree at most is linear in g(z, (4))-=) That is, there exists 0 E ah such that for all documents x
E.[P(w)) Al 00 g(x, (4)
In fact, one can use the same set of documents for both representation and downstream task so that we do not need additional landmark documents.
5 SYNTHETIC EXPERIMENTS
In this section, we optimize the reconstruction-uee based objective ( on data generated by several topic models show that self-supervised learning performs on with inference using correct model, and both of them outperform inference with misspecified models.
5.1 TOPIC MODELS
We consider four types of topic models in our experiments. Our first topic model is the pure- topic model, where each document's topic comes from discrete uniform distribution over the K -ddimensional linear basis, namely {e1, e2..... Our second topic model is the Latent Dirichlet Allocation (LDA) model, where ((K) is a symmetrie Dirichlet distribution Dir(1/K).
We also interested in topic models that involve more subtle topic correlations and similarities between different topic's 's word distributions. To this end, we consider the Correlated Topic Model (CTM) Blei & Lafferty 2007) and the Pachinko Allocation Model (PAM) (Li & McCallum 2006). Our goal is to construct settings where the correlation between topics provide useful information for inference. To achieve this goal, in CTM, we construct groups of 4 topics. Within the group, topic pairs (0,2) and (1,3) are highly correlated (as specified by the prior), while topics 0 and 1 share many words (see Figure [ for an illustration of the setting). In this case, if we observe a document with words that could either belong to topic 0 or 1, but this same document also has words that are associated with topic 2, we can infer that the first set of words are likely from topic 0, not topic 1. We construct such correlations in CTM by setting the diagonal entries of its Gaussian covariance matrix to and the covariance between correlated pairs topics to 0.99 times the diagonal entries, with remaining entries set to zero. We construct similar examples for PAM, see Appendix C for details.
5.2 SIMULATION SETUP
Document generation Our documents are synthetically generated through the following steps: we first construct V K topic-word matrix A. Then, for each document, we determine the document length from Poisson distribution (ois(A), draw a topic distribution w from AKK), and draw n words i..dd from the word distribution given by Aw. In simulation, set K 20, V 5000, À E (30, 60}. Often A will drawn from a Dirichlet distribution Dir(a/ K) (see Appendix C for details), and we take larger a introduces more similarity between topics.
3, 5, 7,9 to vary the difficulty level of the inference problem, where

---

Published as a conference paper at ICLR 2023
CTM kppic Prsportion
03
06-
04-
02-
2
Figure I: One example of group of 4 topics in the Correlated Topic Model. Left: Weight of each topic in document's topic proportion. In this example topics I and 3 have large proportions as they are correlated. Right: Pairwise topic similarity (cosine similarity of the pair's 's word distribution).
Neural nefwork models We consider two different network architectures as f in the experiments: (1) Tully-connected network with residual connection, with bag-of- words document representation as the input since the order does not matter in topic models; (2) Attention-basee architecture. Due the architecture of attention, the input needs to be sequence of words instead of pag-of-wood words vector. Thus, we transform bag-of-words representation into full document by repeating each word by its frequency and concatenating them in random order, and then the full document the input. More specifically, the attention based architecture contains 8 transformer blocks where each transformer block consists of a 768-dimensional attention layer and ferd-forward layer, with residual connection applied around every block. The network's second to last layer averages over the outputs of the last transformer block and the final layer projects it to V-dimensional word distribution. See more details in Section C.5) We find that fully-connected neural network with residual connections performs the best on recovering topic posterior distribution for pure-topic and LDA documents, and attention- based architecture /Vaswani et al. 2017) performs the best for recovering topic posterior distribution for CTM and PAM documents.
Training setup total amount of training data varies from 720K documents to 6M documents. The loss function is the reconstruction-bent word. To reduce the variance during the training. in each training document, we sampled words as the prediction target, hide them from the model and ask the model to predict each word separately. The trained model evaluated on test documents generated from the same topic prior as the training documents. We use 5,000 test documents for the pure-topic prior and 200 test documents for the remaining priors. For each test document, we use MCMC (see Appendix C.9) assuming the document's correct topic prior approximate the ground truth topic posterior. To get the posterior estimations from SSL approach, we multiply the rreudo-inverss output (xx get the estimated posterior mean vector At fxx) Section I. Then, we measure topic posterior recovery loss as the Total Variation (TV) distance between the recovered topic posterior mean vector and the ground truth topic posterior mean vector.
During training, we resample 60,000 new documents after every 2 epochs, and the
objective ①I) with = 1, i.e., we want the model to predict one missing
the topic matrix with the model
explained in Section 3 (and also
5.3 EXPERIMENT RESULTS
Topic posterior recovery As illustrated in Table I after 200 training epochs, model can accurately recover the topic posterior mean vector. Meanwhile, it can be observed that for larger values of the Dirichlet hyperparameter a, the recovery loss gets higher. This is expected because higher a leads to more similar topics, which makes the learning problem more difficult. This effect is also captured by Definition 3.2 and show the computed condition numbers in Section C.6
Major topic recovery We examine the extent our recovered topic posterior captures the major topics in a given document. For pure-topic and LDA documents, we measure major topic recovery accuracy of correctly estimating the topic with largest proportion. Considering that topics in CTM and PAM documents are correlated in pairs, we measure the major topic recovery accuracy for CTM and PAM as the top-2 topic overlap rate on their test documents. Table shows that our algorithm is successful throughout all settings we consider.
Robustness of self-supervised learning We compare the self-supervised approach to traditional topic inference (MCMC with specifed topic prior, the results variational inference are reported
7

---

Published as a conference paper at ICLR 2023
TV Distancs
Decument Type
ML. Apani
Document
Type
Pure
LDA
CTM
PAM
Pure
LDA
CTM
PAM
00148
01757
0,050
034539
L0000
03000
09025
0.020
00003
+0,037
+00025
0.0406
0.0275
+00330
01899
00299
0.3300
01300
4.00076
4 +00053
+0,048
10,0038
4 +0,0000
1.00558
0,0311
400044
000501
0.109]
0.0970
0.0787
1,0000
0,5000
0.8975
03675
0.0000
= 000002
2 +0.0057
0,0034
1 0,0000
4 100041
0,0296
0.0007
0,0291
0,1235
0,107]
0,0800
1,0000
0,9130
0.8615
0.5675
+00071
0.0383
00517
001355
D.TRO
DA077
0.9900
0.9100
013300
4.00045
A 0.0089
& 0.0064
0.00053
A 0.00998
0.0039
0,0330
#00161
Table 1: Self-supervised learning approach performs reasonably on recovering the topic posterior mean. Left: TV distance between recovered topic posterior (self-supervised learning approach) and true topic posterior for different topic models. Right: Major topic(s) recovery accuracy for different topic models. The 95% confidence interval reported in both tables.
TV Distance
Document LDA
Type (a 1)
Major Topic(s) Recuvvrry Method
Document Type (a = 1]
Method
Pure
CTM
PAM
LDA
CTM
PAM
000406
0.1182
0.1218
0.9150
0.835)
0.8500
LDA
LDA
0,000
0,0099
0.0099
0.0387
-00344
0.0360
0,2083
0,2060
0.3154
0,9000
0,9178
0,300
CTM
CTM
0,00038
0.0064
0.0082
0.0416
0,0275
0.0370
0.3782
0.3459
0.3933
0,8750
0.7250
0,9330
PAM
PAM
0,0038
0,0458
0.0399
0,0320
SSL (ours) 0.0148
0,0757
0,0550
0.0489
0.9050
0.9175
0,9022
SSL (ours)
0.0037
0.0025
0.0275
0.0330
Table 2: Robustness of self-supervised learning approach. Left: TV distance between recovered topic posterior and true topie posterior of self-supervised learning approach versus posterior inference via Markov Chain Monte Carlo assuming a specific prior for a = 1. Right: Major topic recovery rate of our approach versus posterior inference via Markov Chain Monte Carlo assuming a specific prior for a 1. In both table, the 95% confidence interval is reported.
in Section Specifically, for each a value, we take 200 test documents from each category of documents, and we run posterior inference assuming specific topic model and calculate the TV distance between this posterior mean vector and the ground truth posterior mcan vector. We exclude assuming pure-topic prior from our comparison because it gives invalid results for documents with mixed topics. For a 1, as shown in Table 2 closer to the ground truth topic posterior than that recovered from misspecified topic prior.
the topic posterior recovered from SSL approach is
We also report the major topic recovery accuracy for SSL approach against topic inference using both correct and incorrect topic model in Table The major topic recovery rate of SSL approach is similar to that of posterior inference assuming the correct prior. For pure topic model, all four methods get 100% major topics recovery rate on pure-topic documents. For LDA, again all four methods perform similarly well since the instance is not difficult. However we observe significant difference for more complicated CTM and PAM models. For these models, the SSL approach performs similarly to posterior inference using the correct model, and both of them inference using misspecified models. The results for a = 3, 5, 7,9 are presented in Section The comparison further reveals the robustness of self-superivised learning.
significantly better than posterior
6 EXPERIMENTS ON REAL DATA
The self- suff-supere learning approach studied in this paper is simplified for theoretical analysis as uses bfg-of-words approach. In this section, we show that although this simplified approach is not state-of-art, it has reasonable performance compared to several baselines.
We use the AG news dataset (Zhang et al. 2015). in which each document has label of one out of four categories: world, sports, business, and sci/tech. Each category has 30,000 samples in the training set and 19,000 samples in the testing set. The number words in the vocabulary is about 16,700. We generally follow the experiment setup done by Tosh et al. (2021a) (see more details in Section D.I. but the representation we used is generated by our reconstruction objective. Experiment results
Experiment Setup
additional datasets are provided in Appendix D
8

---

Published as a conference paper at ICLR 2023
200
13.00
2000
1007
2300
1.0
Figure 2: Performances of RBI., NCE and baselines (BOW, Word2vec, L.A)) real data.
Our training follows the semi-supervised setup that involves two major phases: unsupervised phase and supervised phase. In the unsupervised phase, we train the self-supervised model with reconstruction-baasee objective most of training data. In the supervised phase, using the document representation generated by the SSL model as imput, we train a linear classifier to classify the category of the documents using multi-class logistics regression (by Scikit-learn (Pedregosa et al.) (01T)) with 3-fold cross validation for parameter tuning (12 regularization term and solver). We measure the topic recovery aceuracy on test set compare the performance of different methods.
We choose residual blocks as the basic building block for our neural network architecture, with varying width and depth. The network used for Figure 2 has residual blocks and width of 4096 neurons per layer. Other detailed hyperparametens and ablation studies can be found in Appendix DD3 We train the model using reconstruction based objective technique in synthetic experiments.
( with = 1 using the same variance-reductioon
To extract our representation, we use an identity matrix to replace the last layer. That is, we take a softmax function top of the second-to-last layer of the neural network. This is different from our theory but we found that effectively reduces the high output dimension and improves performance. We include more details in Section i.2 Appendix.
Experiment Results The performance of Deconstruction--assse Learning representation (RBL) is shown in Figure 2 where the test accuracy on representation is plotted against number of training samples used to train the classifier in supervised learning phase. The representation RBL performs better than both bag-of-words (BOW), Word2vec and LDA baselines (see more details about baselines in Section D.2). Notably, Word2vec representation inferior only by a small margin, and works well in general even when limited training samples are provided. On the contrary, although BOW representation has decent test accuracy when training samples are abundant, it performs significantly poorly on smaller training set.
also compare our result to previous noise contrastive estimation (NCE) representation benchmark achieved by Tosh et al. (202 using contrastive objective, where their reported best accuracy is around 87.5% when full 4000 training samples are used, slightly higher than our RBL corresponding test accuracy of 87.1%. In attempt to reproduce their result, parameter tuning yields the best accuracy of 86% when full training samples were used. We plot our own reproduced results on in Figure 2 since parameter tuning does not achieve close benchmark their result.
7 CONCLUSION AND LIMITATIONS
"All models are wrong but some are useful." If one self-supervised objective can capture all models, then it would be able to extract useful information. In this paper, we studied the self-supervised learning in the topic models setup and showed that it can provide useful information about the topic posterior no matter what topic model is used. Our results generalized previous work (Tosh et al.) 2021a) to both contrastive learning and reconstruction-basel learning and our techniques allow us to depend on weaker assumptions. We also empirically showed that the reconstruction-ben performs better than the posterior inference under misspecified models, and it can provide useful representation for the topie inference problem. Our theoretical analysis is limited to bag-of- wo-ds setting, which also limits its empirical performance. Extending the work to more complicated models is an immediate open problem.
learning

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is supported by NSF Award DMS-2031849, CCF-1845171 (CAREER), (Tripods) and Sloan Research Fellowship.
REFERENCES
Anima Anandkumar, Dean Foster, Daniel Hsu, Sham M Kakade, and Yi-Kai Liu. A spectral algorithm for latent dirichlet allocation. Algorithmica, 72(1):193-2144 2015.
Animashree Anandkumar, Daniel Hsu, and Sham M Kakade. A method of moments for mixture models and hidden markov models. In Conference on Learning Theory, pp. 33-1. JMLR Workshop and Conference Proceedings, 2012.
Animashree Anandkumar, Rong Ge, Daniel Hsu, and Sham Kakade. A tensor spectral approach to learning mixed membership community models. In Conference on Learning Theory, pp. 867-881. PMLR, 2013.
Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of machine learning research, 15: 2773-2832, 2014.
Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topic models-going beyond svd. 2012 IEEE 53rd annual sympositmm foundations of computer science, pp. 1-10. IEEE, 2012.
Sanjeev Arora, Rong Ge, Ravi Kannan, and Ankur Moitra. Computing a nonnegative matrix pectorization-provaal SIAM Journal Computing, 1584 2016a.
Sanjeev Arora, Rong Ge, Frederic Koehler, Tengyu Ma, and Ankur Moitra. Provable algorithms for inference in topic models. In International Conference on Machine Learming, pp. 2859-2867,
PMLR, 2016b.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.0922 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint 1906.00990009 2019.
Victor Bittorf, Benjamin Recht, Christopher Re, and Joel A Tropp. Factoring nonnegative matrices with linear programs. arXiv preprint arXiv:/206. 1270, 2012.
David M Blei and John D Lafferty. A correlated topic model of science. The annals of applied statistics, 1(1):17-35, 2007.
David M Blei, Andrew Ng. and Michael Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022, 2003.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882. 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint 20Xv:2006.10029 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the JEEF/CVF Conference Computer Vision and Patern Recognition, pp. 15750-15755 2021.
10

---

Published as a conference paper at ICLR 2023
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020c.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv://100008805 2018.
Nicolas Gillis and Stephen A Vavasis. Fast and robust recursive algorithmsfor separable nonnegative matrix factorization. IEEE transactions on pattern analysis and machine intelligence, 36(4): 698-714, 2013.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-super learning. arXiv preprint 07Xiv:2006.07733, 2020.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. arXiv preprint arXiv:2 156, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedingy of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2020.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 9(1):2, 2021.
Jon Kleinberg and Mark Sandler. Using mixture models for collaborative filtering. Journal of Computer and System Sciences, 74(1):49-69 2008.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprint 11Xi::2008800064, 2020.
Wei Li and Andrew McCallum. Pachinko allocation: Dag-structured mixture models of topic correlations. In Proceedings of the 23rd international conference on Machine learning, pp. 577-584, 2006.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http
// 1/www.aad acw.wacclll. web owwpboorg/eeoo ant
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances neural information processing systems, pp. 3111-3119, 2013.
Christos H Papadimitriou, Prabhakar Raghavan, Hisao Tamaki, and Santosh Vempala. Latent semantic indexing: probabilistic analysis. Journal of Computer and System Sciences, (1 2000.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-288 2011.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
11

---

Published as a conference paper at ICLR 2023
Ali Rahimi and
Benjamin Recht. Random features for large-scale kernel ma- Advances in Neural Information Processing Systems, volume 20.
chines.
In
2007.
URL htpa://proceding 013a006f03dbcc333 592e of Paw.aaaaaaa5--aa..pd....
Radim Rehurek and Petr Sojka. Gensim-pythho framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2), 2011.
John Salvatier, Thomas V Wiecki, and Christopher Fonnesbeck. Probabilistic programming in python using pymc3. PeerJ Computer Science, 2:e55, 2016.
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648, 2020.
David Sontag and Dan Roy. Complexity of inference in latent dirichlet allocation. Advances in neural information processing systems, 24: 14:1008-1016, 2011.
Jiaye Teng and Weiran Huang. Can pretext-based self-supervised learning be boosted by downstream data? a theoretical analysis. arXiv preprint arXiv:2103.03:: 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer ESsionEECVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020a.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00577 2020b.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. arXiv preprint arXiv:2102. 06810, 2021.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic
posterior information to linear models. Journal of Machine Learning Research, 22(281):1-33 2021a. URL http:hht:////tt http::/mmrrl.rrrrro www.rr/pppp//////eeeeeeaas w/2/els///22////e/ http:///mlll..rrr//paass////////
1.099hhmml
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning. multi-view redun- dancy, and linear models. In Algorithmic Learning Theory. pp. 1179-1206. PMLR, 2021b.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self-supervised learning from a multi-view perspective. arXiv preprini arXiv:2006.05576, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 509855998999 6008, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622 2020.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. In Advances in Neural Information Processing Systems, 2021. URL htt.::ttp////ecct. http:::://ppeeevveeeee www./eetfoorna 1d=DDV225Cboox
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. arXiv preprint arXiv:2105.15134 2021.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28:649-657, 2015.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. arXiv preprint 102.08000 2021.
12

---

Published as a conference paper at ICLR 2023
A OMITTED PROOFS IN SECTION 3
In this section, we give the omitted proofs in Section We first give a warmupp example to illustrate our proof idea in Section A.I Then we give the proof of our main result (Theorem 3.] Section A.2 and the proof of robust version (Theorem 3.3 in Section A.3
A.1 WARM-UP EXAMPLE: RECONSTRUCT THE UNKNOWN WORD
In this warm-up example, we consider the simple setting where we try to predict the only one unknown word of the document using reconstruction-basee objective (I. The following result is the special case of Theorem 3.T with 1, which shows that the learned representation able to give any linear function of the topie posterior.
Theorem A.1. Consider the general topic model setting as Definition 2.1 suppose topic-word matrix satisfies rank(A) K and and function J minimizes the reconstruction-buusee objective (T). Then, for any document and its posterior w wxx A), any linear function P(w) is linear in (xx), that is for any P(w) TTw, w, there exists a E RV such that for all documents x
Eu | [Twwor) =
f(x)) f()).
Following the proof idea described in Section we first give a characterization of the optimal function f. The following lemma shows that f()) must be the word posterior vector given the document The proof this lemma is simply based on the property of the cross-entropy loss function and we defer it to Section A.1.1 Recall that our vocabulary set is V fei...., ev and Tumsup (x,y) v) is the given document, where x is unmasked part and is word marked as unknown.
Lemma A.2. f minimizes the reconstruction-bee based objective ( then we have for all document x
f(x) (P = v1/z). =v1=))
P(y = uy [
Based on the above lemma, we are able to prove Theorem A. is easy to see that the word posterior distribution is [w]x we have f(x) AE W z Since the columns of A linearly independent, we know En [||z] = A Af)(()) Thus, for any linear function 3 w. there exists Section
0 TAh 3 such that [3T w|x] 0 f(x). The formal proof of the general case given in
A.1.1 PROOF LEMMA A.2
Instead of focusing on the = 1 case, we directly give the corresponding lemma of Lemma A.2 for general Recall that Tunsup (æ, y) is the given document, æ is the unmasked part, y (y), (y1--... yt) is the unknown words that we want to predict and V (ei....,yy) is the set vocabulary. shows that the optimal f the words posterior distribution. Note that the words posterior distribution is linear in the topic posterior of f-th moment, which is useful for the later analysis.
Lemma A.3 (Lemma A.2 General Case). If minimizes the reconstruction-bent based objective then have for all document x:
f(x) = (P(y (v1, 01 0i. u1).......))))))... u1)l,) = (12, u....., (1),,,,,,)),,..........,,.)).,,, P(y = (vv, ...... uv (z))
Proof. Denote the word posterior distribution p" (x:). We will show f(x) p" (x:). By the law of total expectation, we have
Lievons
E. Eyi ( v)]=]]
We know the probability of y given x is p" (x)) Since f cross-entropy loss, we have
Treconst 2 Ey
Σ
1o" (x))e loglp" (x))&
kE(V*)
where the equality is obtained when f(x) p" (x). Thus, when f minimizes the reconstruction- objective (T). we have ((x) = p² pxr).
13

---

Published as a conference paper at ICLR 2023
A.2 PROOF OF THEOREM 3.1
In this section, give the proof of Theorem 3.1
Theorem A. is covered by Theorem 3.1 the special case of f = 1. The proof follows the same idea as for Theorem A.] We first show that f() the f-words posterior (Lemma A.3). Then just
1 case where can recover topie posterior with A f()), we can also recover topic posterior of t-th moment with A f(x) with some matrix A. The result follows by the observation that for a polynomial P(w) with degree most t, Eu (P()) is a linear function of the topic posterior of t-th moment.
Theorem 3.1 (Main Result). Consider the general topic model setting Definition 2.1 suppose topic-word matrix A satisfies rank( A) = K and function f minimizes the reconstruction sensstructtio-bbeedd objective T Then, for any document x and its posterior w1 A), the posterior mean of any polynomial P(w) with degree at most is linear in f()). That is, there exists a E RRV such that for all documents
[w (r, A] = 0 f(x).
Proof. Since f the minimizer of reconstruction-basee objective (I. by Lemma A.3 we know given an input document
f(x) - (P(y = (wi, Ui...,,. u1)xx, = (02, UI,..., (1)],),,,))),,........ F(y = (vv, .v...., uv) (v)
We will show that f(x) is linear in the topic posterior. In the following, we focus [f(z)] which is word posterior probability of the f. unknown words being "/,-...30 .e E V given document Recall SK be the K dimensional probability simplex. By the law of total probability, we have
(f)))ln.vaa........ wa.......u
Ply1, V2.....uul.2.
[ WESx
P(u,, V2. Vb, w r)
P( M2. ................................ w. x (w(())))
SESS
JuESx
P(y,, U2. Ut. 21, Pu....
[[[]]
Σ
P(a a P().... . ll......... Res w (w)l((
JWESK [[
Note that z; the topic of word yı- Since consider the general topic model as Definition 2.1 know
P((Vi.22,.-. Me 21, Z2,..., Ze; w) PP(11 W2..... Mt [1, 22 m1)
Plus (z))
II A Aw.a
8=1
where is the word matrix. Hence,
Σ
II
Aguzz
P(21, F2; ..., (P)
21,82....2 [[K] i=1
WESK
Σ
II
Aguzz
W2, (P( (w)
21 E(K]
WESK
Σ
I
A
E
I Wx, T
21 2,
14

---

Published as a conference paper at ICLR 2023
Recall that the topie posterior tensor is W post Ev [u Rxx... where each entry [W WWpostls......se (25 is the topic word ys for E [t|l) = Ew (22, Ws, [z]. Therefore,
(f(1))91
E
II
Ag.., a. [W Wwpstlesnea.....
21 [[K] 1=1
f(x) (A 2 A @ & vee poost)
times
where @ is the Kronecker product, A= = A A8 AEEK A and vec(W post) E KKK is the vectorization of W Wpost. Note that At At @ At, so have vee(W A (xx)
Since P(w) is polynomial of degree at most t, we know there exists 3 such that E. P[w)]æ] 3 vec(W Thus, let 0 (At)* B. have Ew[ x B vec(W) 0 f(x).
A.3 PROOF OF THBOREM 3.3
Similar to Lemma A.2 we can show that the representation [ar is close to words posterior. Then, the key lemma to obtain Theorem (.3 the following result. suggests that function f c-close to the optimal function f (word posterior distribution) under cross-entropy loss, then can still recover the topic posterior with proper up to ((vE) error. For example, we will choose B A when t 1. See the proof of Theorem 3.3 for the choice of B in the general case.
Lemma A.4. For cross-entropy loss f and any two probability vectors p.p", if f(p,p') s pin p- p22 {(p. + then for any matrix B we have Bp Bp 1 S x(
Proof. Recall that is cross- loss, we know
((p.p")
P.E.p.=pp2 ((p.p') DRL(P DR S²)
min
By Pinsker's Inequality, we have that
1p-p"'l p'll S /2DKL(p' 11 S V2c.
Then, for any matrix B we have
BBp S ((BB)pP-pp"l S x(B)
where we use the definition of & condition number (Definition 3.2)
We now are ready to give the proof of Theorem 3.3 which is based on Lemma A.4 and the proof of Theorem 3.1
Theorem 3.3 (Robust Version). Consider the general topic model setting as Definition 2.1 suppose topic-word matrix A satisfies rank(A) = K and function f satisfies Lreconss (f) S Licommst + c for some
e > O. Then, for any document x and its posterior w/(x, A). the posterior mean of any polynomial
P(w) with degree at most is approximately linear in f(x). That is, there exists a O E IR. V* such that
Eg
(E., (P()) A] 8T f(x))² s 2 |||1 2 (A')s,
where (Pw))aa,,,, A] = 3T
Proof. Recall from the proof of Theorem
3.1 that E. [P(w)[a] = 3 vec(W = AT f (z).
where At At & ... @ At and x(At) (AA)). Same as in the proof of Theorem 3.1 let
0 (AT)T B. We have
E,
(Em (P())(x AT f(=))
=E, (BT (
<Bn
[1 |lt (x) (1)=)(((x))
< || E. [|Atf (z) -
15

---

Published as a conference paper at ICLR 2023
We going use Lemma A.4 to bound f f(x)l By Lemma A.3 know f*)x)
is the word posterior, so (x) Bylr [2]]]] Since é is cross cross-entropy loss, we know
(+)()) [f(x)]
Σ yk log kEIVI
DKL(f f1)))
x
which implies ((f(z), - min, C(p. (x)) S Therefore, by Lemma A.4 with = AAT, we know
(=) <<(At) x'(At)Vee.
Given the desired bound, we have
Eg (Pw))(() 0- [()) x
A.4 DISCUSSION ON EFFECTIVE DIMENSION
REPRESENTATION
Although the dimension of fx) is Vr, its "effective dimension" is actually K', which is much smaller than V (the number of topic K usually is much smaller than the size of vocabulary V). To see this, let's consider the simple case of / = 1. As we discussed in the paragraph below Theorem 3.1, we have [(x) = ATR. wr xr, A), where A vxx is the topie-word matrix and E[w|a, A] the posterior mean of w E TR: In words, this means that f(x) is completely determined by linear transformation a K -dimensiional vector. Therefore, when we consider the feature of n documents matrix ( .... [x) E Iy the rank of this matrix is at most K and one can get these K KCddimnnsional features by running standard dimension reduction method (e.g.. SVD) on the original feature matrix.
16

---

Published as a conference paper at ICLR 2023
B OMITTED PROOFS IN SECTION 4
In this section, give the omitted proofs in Section Section B.I and proof of Lemma 1 in Section B.2
We give the proof of Theorem in
B. PROOF OF THEOREM 4.1
first give characterization of the representation g( z²). which would be useful the later analysis. Note that this the same as the one shown in Tosh al. (2021a). We provide its proof for completeness in Section B.2
Before presenting the proof of Theorem 4.1
Lemma B.1. f minimizes the contrastive objective
2 then have
f(x,x')
Pyy 1/x,x')
x(x,2') & 1 f(0,x')
Ply 0/z,x")
Now we are ready to proof Theorem
4.1
Theorem 4.1. Consider the general topic model setting as Definition 2.1 suppose topic-word matrix A satisfies rank(A) K and function minimizes the contrastive objective (2) Assume we randomly sampled m K different landmark documents {1, y and construct ( {1, 1 (3). Then, for any document and its posterior wxx A), the posterior mean of any polynomial P(w) with
degree at most L is linear in g {1 m 1=1) That is, there exists 0 E R such that for all documents
Em. A = 0 g(x, (1)
Proof. Since f is the minimizer of contrastive objective by Lemma B.1 we know
Ply 1/2,x') P = 0 x. x')
g(r, x²)
Similar to the proof of Theorem 3.1 we will show that g(x, x²) is linear in the topic posterior when x' is fixed. By Bayes' rule, we have
P(x, x' 1)PP(y = 1),P(a,
P(x, x = =) x ly 0)
g(t, =
P(x, x' /V 0)T = 0)/P(sr
where we P(y - Pyy
1/2.
Denote the words x' as a'..... ......,,, at and their corresponding topics as Æ1,..., 2t. Then, by our way of generating x, x', we know
P(z, = 0) = P(x)F(x'),
P(z, y 1) = / P(ri...., a/l, 2. 2, 1)P(w, xly 1)dw
Σ ................................ w PP (wr,) dr,,
=((K..((((K)
which implies
((x,x')= P(x')
Σ
P( P -/ ((d)))
Note that
P(c4,...,2/1a. ..... 2) I P())(z) II
A
==1
17

---

Published as a conference paper at ICLR 2023
where A is the topic-word matrix. Hence,
g(x,x') F(x')
Σ
II
Aut.2
P(z z4 (w) P( ( ddw
[[KK] i=1
Σ
II
II
((w2x)dw
P(a')
2 [(K] i=1
Σ
II
A Bu
II
x
P(x')
21 S1 [[E] iml
Recall that the topic posterior tensor is W post [W En [2z, W..,,, [z]. Therefore,
B Bww
xx..X where each entry
g(x,x²) P(x')
Σ
II A (W [Wpotlesrea.....
21. E(K)
A|x'] vec (oc((Wpet)
P(x')
where A = A8A8 A A Alx') E RR is the row of A that correspond to x' ........8 2/6. and vec(W post)) I2K the vectorization of Wpost-
Recall we have m randomly sampled different documents {4-)-1 Denote D E R as a diagonal matrix such that D.,, P(4), and Á E gmx K submatrix of A such that each row of Á A[4]. Since 4 are randomly sampled, so Dr O. Thus, we know
glx, (4) D - Avec(W
which implies vec(W = At Dg(x, (4))==)) Note that we need to show At well-defined. Since A has full column rank, we know A also has full column rank. Thus, the submatrix A also has full column rank since
K7. This implies At is well-defined.
Since P(w) is polynomial of degree most t, we know there exists 8 such that E, [2w)] 3 vec p. Thus, let 0 = D(At)T B, we have
By [P()) = T vec(Wppsst) post) = 0 g(r, (4)
B.2 PROOF OF LEMMA B.
Lemma B.1. f minimizes the contrastive objective
2 then we have
f(x,x')
Ply 1|,, x²) Ply 2,,,,'') x²)
g(x, x') 1- f(x,
Proof. Since f is the minimizer of contrastive objective and
Tecontrst (f) = EEz,'' yy [(S x') - y) = E [Eyyee (((xxx²)) y)
is easy to see that f is the Bayes optimal predictor IP(y la,, x'). Therefore, we know
g( = Ply = 1/2,
P 00
18

---

Published as a conference paper at ICLR 2023
B.3 DISCUSSIONS ON THE
The following corollary shows that solving the downstream task is equivalent to solve a kernel regression (in some sense the landmark documents are just random features for this kernel (Rahimi & Recht 2007))
Corollary B.2. Denote {(Iss ]i)];= as the downstream dataset, where is = Ew (P()) is the kargetfor document 2., we set landmarks (4) to be {x; then solving the downstream task is equivalent to a kernel regression, that is
arg min Σ (üx o g (4))1) arg min 11 GØ
where 1j = (Ji,.... Bm) ) and G E RR is a kernel matrix such that Gij g(
As discussed in Corollary B.2 we show that one can use the same set documents for both representation and downstream task. Recall that in Theorem 4.1 we need a large number of landmark documents to generate the representation and then use for the downstream task (c.g.. .. fit polynomial of topic posterior). However, one may not have enough additional documents to be used as landmarks. Therefore, the benefit of the self-referencing is that we do not need additional landmark documents to generate the representation.
The key observation in Corollary B.2 is that the learned representation g x²) can be viewed as a kernel. Suppose the documents for the downstream task are .1,-.., Im. Construct kernel matrix G E Rmxm such that Gij = g (2r xz)- Then, it easy to see
θ g(z,, (2,))==) = G
where G; is the i-th column of G. Thus, 0 can be found by the following kernel regression:
min G0
where ū (.1..... and 5t E., (P())a:))
19

---

Published as a conference paper at ICLR 2023
C SYNTHETIC EXPERIMENT ADDITIONAL DETAILS
In this section, we include more details about our synthetic experiments in Section 5 In Section
we describe the document generation process for PAM model. Additional results on topic posterior recovery loss and major topic recovery are included in Section C.2 and Section C.3
In Section C.4 and Section C we report the results about different training epochs and hyperparameters tuning. & condition number is reported in Section C.6 and give some visualizations for topic correlations for the PAM model Section C.7 Moreover, in Section C.8 we report the performance recovering the topic posterior mean vector when 2. We The confidence interval reported in the paper is calculated as mean + 1.96 * Stestdocs/ / V /Ntestdoss where mean is the mean of TV distance (or accuracy), Stestdocs is the standard deviation of TV distance (or accuracy) and Nestdocs is the number of test documents.
SSL on experiments 20SORTXTI/V TI/IIII//.........
Details of generating topic-word matrix For pure topic and LDA experiments, each column of A (i.e., each topic's word distribution) is generated from Dir Dir(a//K) K) For CTM and PAM, goal is to construct K/4 groups of 4 topics such that within each group, topic O and topic 1 have similar word distribution, and topic and topic 3 have different word distribution (and different from the word distribution of topic 0 and as well). To achieve this, we first sample K/ word distribution vectors from (ir(a/ (a/ K), and for each word distribution vector, we permute its entries to get the word distribution of topic 0, 1,2, within each group and intentionally align the large entries in topic O and to ensure high similarity between them.
C.I PACHINKO ALLOCATION MODEL TOPIC PRIOR GENERATION
The topic proportion the Pachinko Allocation Model (PAM) described Section generated from the following process: for each document, first sample "super-topic" proportion from symmetric Dirichlet distribution Dir(1/K,) and super-topic-to-toppi proportion from a symmetric Dirichlet distribution Dir(30). Then, sample each word by first sample super-topic according to the super-topic proportion and then sample the actual topic from the super-topictto-topi proportion. In our experiment, we set K, 10. Figure C.I offers few examples of PAM topic proportion generated from the PAM model.
Topic Proportions
0.8
0.8
0.8
0.8
06
0.6
06
0.6
02
0.
0.2
02
00
0.
0.
10
20
10
20
20
10
Figure C.I: PAM topic proportion
C.2 MAJOR TOPIC RECOVERY ADDITIONAL RESULTS
In Section 5.3 we measure the major topics recovery of CTM and PAM documents as the top-2 topic overlap rate. Since topics are correlated in pairs in both of these topic models, we are also interested in the top-4 and top-6 topics overlap rate. As shown in Figure C.2 for both CTM and PAM documents, the average top-4 overlap rate is above 66% and the average top-6 overlap rate is above 62% on all a values we test on. These results show that our model can accurately capture more than just the major pair of correlated topics, but also some correlated pairs of less dominant topics.
C.3 SELF-SUPERVISED LEARNING VERSUS ASSUMING SPECIFIC TOPIC PRIOR
We present in Section 5.3 a comparison between the performance of our model and Markov Chain Monte Carlo assuming specific topic model for on the task of TV distance and major topic recovery. In this subsection, we supplement the experiment results for larger values of a along with 1. In particular, consider self-supervised learning with million training documents and
20

---

Published as a conference paper at ICLR 2023
1yrh light Hocurery
liph spet Mouuetyy dicements
16-
66-
Figure C.2: Recovery rate for top-2, top-4, and top-6 topics, measured on CTM (left) and PAM (right) test documents.
Document Type (cr
Method
Pure
LDA
CTM
PAM
TEX
0.1006 0000T
0.1182 000099
0.1218 = 0,0996
CTM
0,2083 0,0088
0,2000 0.0164
0.3154 0.0082
PAM
0,3782 + 0.0038
0,3459 = 0.0128
0,3939 a 0.0096
SSL-6M (ours)
0.0020
0.0037
SSL-120K (ours)
0.0311 1 0.0025
0,0878 + 0.001T
0.0678
000038
0,0600 = 0.0028
Document Type (or
Method
TIA
0.8661 * 0,0030
0.1942 & 0.0156
0.18TT 0.046
0.0048
0.2299
0,3362
PAM
0,4031 0.0046
0.3665 + 0.0117
0,4510
0.0100
SSL-6M (ours)
0.0308 + 0.0070
0.0899 # 0.0033
0.0799 + 0.0048
0.0712 + 0.0038
SSL-120K (ours)
0.0346 0,0007
0.1292 0.0067
0.1062 0.0058
0.0048
Document Type (or
Method
LDA
0.0331 0.0000
0.123 0.0159
0,99 # 0.0144
CTM
0.2655 0.0059
0.2413 + 0.0090
0.3428 + 0.0090
PAM
0,4337 0.0055
0,3608
0.0089
0,4794
0.0097
SSL-6M (ours)
0.0501 0.0030
0,1041
+ 0,0062
0.970 a 0.0037
0,0787 = 0.0043
0.0011
Documeat Type (ar
LDA
CTM
PAM
LDA
0,0883 0.0050
0,2438
0.0169
0,2138 I 0.0188
CTM
0,2840
0.0060
0,2556 & 0.0089
0,3530 & 0.0099
PAM
0,4561
0.0052
0.3707
0,0089
0,4979
0.0T01
SSL (ours)
0.0391 0.0022
0,1233 + 0.0771
0.1071
0.0068
0.0960 + 0J003
SSL-120K (ours)
Document Type (a 9)
Method
Pure
LDA
CTM
PAM
LDA
0,1051
0.0065
0,2559
0.0162
0,2281
0.0152
CTM
0.3129
000079
0.2580 & 0.0092
0.3556 # 0.0084
PAVI
0143055 0.077
0.3776
0.0193
0.3156 0.088
SSL-6M (ours)
0.0317 # 0.0048
0.1358 # 0.0089
0.1101 # 0.0064
0.0971 + 0.0033
(ours)
Table C.I: TV between our recovered topic posterior and the true topic posterior our self-supervised learning approach versus topic inference via Markov Chain Monte Carlo assuming specific prior for a 1,3, 5, 7,9. The 95% confidence interval is reported.
with 120 thousand training documents after 200 training epochs, and we expect that the former can allow our model to approximate the optimal predictor described Theorem 3.1 and the latter may offer some insights on our approach's 's performance in more practical settings, such as on a real world dataset with limited size. Additionally, when training with 120 thousand documents, we randomly shuffle each document and reassign the t target words in every epoch, to make our model learns enough information from the limited amount training data. We report our results the form of 95% confidence interval.
Table CJ shows that our approach with million training documents consistently outperforms misspecified topic priors by yielding much lower TV distance from the true topic posterior. Note that the entries with correct topic model are omitted, because in our experiment the Markov Chain Monte Carlo recovered topic posterior assuming the correct topic model is exactly what as
21

---

Published as a conference paper at ICLR 2023
Document Type (a
Method
Pure
LDA
CTM
PAM
LDA
1.0 0.0
0.9150 = 0.0387
0.8850 = 0.0344
0,8500 = 0,0360
CTM
1.0 = 0.0
0.9000
0.0416
0.9175 2 0.0275
0.8300 0.0370
PAM
1.0 0.0
0,8750 = 0.0458
0.7250 # 0.0397
0.9050 = 0.0320
6M (ours)
1.0 0.0
0.9050
0.0406
0.9175
0.0275
0.9025 * 0.0330
SSL-120K (ours)
1.0 0.0
0.9150 0,0387
0.9100 0.0292
0.9200 0.0289
Document Type (a 3)
Method
Pure
LDA
CTM
PAM
LDA
1.0
0.8900 0.0434
0.8200 # 0.0354
0.8200 0.0404
1,0
0.8800 = 0.0450
0.8900 = 0.0326
0.7650 + 0.0404
PAM
1.0 0.0
0.3550
0.0488
0.6375
0.0392
0.8950
0.0357
SSL-6M (ours)
1,0
0.8750 0.0458
0.8900
0.0311
0.8950
0.0344
SSL-120K (ours)
0.9950 0.0098
0.8700 0.0466
0.8900 0.0312
0,8950 0.0357
Document Type
5)
Method
Pure
LDA
CTM
PAM
LDA
1.0 0.0
0,9050 0.0406
0.8500 0.0339
0.7750 0.0390
CTM
1.0 0.0
0.9050
0.0406
0.8975 = 0.0304
0.6775 + 0.0421
PAM
1.0 0.0
0.8850
0. 0.0442
0.6150 + 0.0371
0.8825 0.0379
SSL.66 (ours)
1.0 0.0
0.9000 # 0.0416
0.8975 + 0.0296
0.8675 a 0.0407
SSL-120K (ours)
0.9900 = 0.0138
0.8800 # 0.0450
0.8900 = 0.0303
0.8750 # 0.0390
Document Type (a
7)
Method
Pure
LDA
CTM
PAM
LDA
1.0 = 0.0
0,8800 = 0.0450
0.7750 = 0.0397
0.7475 = 0.0433
CTM
1.0 = 0.0
0.8650 + 0.0474
0.8825 + 0.0353
0.6250 = 0.0390
PAM
1.0 0.0
0.8450 0.0502
0.5725 = 0.0356
0.8700 = 0.0411
SSL-6M (ours)
1.0 = 0.0
0.9150 0.0387
0.8675
0.0383
0.8675
0.0407
SSL-120K (ours)
0.9650 0.0255
0.9350 0.0342
0.8775
0.0370
0.8725 0.0398
Document Type (a 9)
Method
Pure
LDA
CTM
PAM
LDA
0,8850 0.0442
0.7750 0.0384
0.7350 0.0432
CTM
1.0 0.0
0,8950
0.0425
0,8800
0.0348
0.5925 0.0392
PAM
1.0 0.0
0.8650
0.0474
0.5675
0.0358
0.8600
0.0428
SSL-6M (ours)
0.9950 0.0098
0.9100
0.0397
0.8350
0.0330
0.8325
0.0461
SS0K (ours)
0.9800
0.0194
0.8600 0.0481 0.8800 0.034]
0.8425 0.0430
Table C.2: Major topic recovery rate of our approach versus posterior inference via Markov Chain Monte Carlo assuming specific prior for 1,3, 5, 9. We report the 95% confidence interval.
our ground truth topic posterior. Meanwhile, our approach with 120 thousand training documents outperforms misspecified prior in almost every scenario, except when compared to L.DA prior pure topic documents. This is is likely because the LDA topic prior is relatively concentrated on one specific topic and thus similar to the pure topic prior.
We also present a comparison between our self-supervised learning approach and posterior inference assuming specific topic prior the task of recovering major topics in the topic proportion for = 1, 3, 5, 7, 9, as supplement to Section 5.3 As shown in Table C.2 in almost every scenario we test on, our model can perform competitively against the MCMC-interred posterior assuming the correct topic prior and outperform misspecified topic prior.
To further investigate the performance of self-supervised learning. we compare our approach with Variational Inference assuming a specific topic prior the task of recovering major topics in test document's 's topic proportion. Table C.3 shows that, similar to our findings from Table C.2 our model can perform competitively against the Variational Inference posterior assuming the correct topic prior, particularly for relative small a values, and can outperform misspecified topic prior in almost every test scenario.
C.4 THE EFFECT OF TRAINING EPOCIIS
We measure how the number training epochs may influence topic posterior recovery loss by varying the number of training epochs. The results present in Section based models trained for
22

---

Published as a conference paper at ICLR 2023
Document Type (a
Method
Pure
LDA
CTM
PAM
LDA
1.0 = 0.0
0.8950 = 0.0424
0.8325 = 0.0382
0.8625 + 0,0339
CTM
1.0 = 0.0
0.9050 0.0410
0.9125 + 0.0283
0.8150 0.0354
PAM
1.0 = 0.0
0.8600 = 0.0481
0.7123 = 0.0382
0.9050 + 0.0325
SSL (ours)
1.0 0.0
0.9050
0.0406
0.9175 0.0275
0.9025 0.0330
SSL-120K (ours)
1.0 0.0
0.9150
0.0387
0.9100 = 0.0292
0.9200 0.0289
Document Type (a
3)
Method
Pure
LDA
CTM
PAM
LDA
1.0 = 0.0
0,8750 = 0.0452
0.7325 = 0.0396
0.7875 = 0.0382
CTM
1.0 = 0.0
0.8800 = 0.0450
0.8800 # 0.0325
0.6975 # 0.0410
PAM
1.0 0.0
0.8550 = 0.0488
0.6075 = 0.0354
0.8825
0.0367
SSL-66 (ours)
1.0 = 0.0
0.8750 0.0458
0.8900
0.0311
0.8950
0.0344
SSL-120K (ours)
0.9950 0.0098
0.8700 0.0466
0.8900 0.0312
0.8950 0.0357
Document Type (a =
Method
Pure
LDA 0,8800 0.0452
CTM
PAM
LDA
0.7575 0.0354
0.7375 0.0410
CTM
10 0.0
0,8750 0.0452
0.8975 0.0283
0.5925 0.0396
PAM
1,0 0.0
0,8500 0.0495
0.5675
0.0297
0,8875
0.0368
SSL-6M (ours)
1,0 0.0
0,9000
0.0416
0.8975
0.0296
0.8675
0.0407
SSL- (ours)
0.9900 0.0138
0.8800 0.0450
0.8900 0.0303
0.8750 = 0.0390
Document Type (a LDA
Method
Pure
CTM
PAM
LDA
1.0 = 0.0
0,9250 = 0.0368
0.7175 0.0396
0.6875 0.0410
CTM
1.0 = 0.0
0,9300 = 0.0354
0,8700 = 0.0382
0.5700 = 0.0354
PAM
1.0 = 0.0
0,9050 = 0.0410
0.5400 = 0.0325
0.8750 = 0.0396
SSL-6M (ours)
1.0 = 0.0
0,9150 = 0.0387
0.8675 = 0.0383
0.8675 = 0.0407
SSL-120K (ours)
0,9650 = 0.0255
0.9350 = 0.0342
0.8775 = 0.0370
0.8725 = 0.0398
Document Type (a 9)
Method
Pure
LDA
CTM
PAM
1.0 0.0
0.0550 = 0.0410
0.7250 0.0396
0.6975 0.0410
CTM
1.0 0.0
0.9200 0.0382
0.8725 = 0.0339
0.5425 = 0.0368
PAM
1.0 0.0
0.8750 = 0.0452
0.5300 = 0.0325
0.8425 0.0452
6M (ours)
0.9950 0.0098
0,9100 0.0397
0.8850
0.0330
0.8325 0.0461
SSL- (ours)
0,9800 0.0194
0.8600 0.048T
0,8800 0.0341
0.8425 0.0430
Table C.3: Major topic recovery rate our approach versus posterior inference via Variational Inference assuming specific prior for 1, 5, 7, 9. We report the 95% confidence interval.
200 epochs for all types of documents. Here, we present the topic recovery loss, measured as the Total Variation distance between our recovered topic posterior and the true topic posterior, for epochs=25, 50, 100, 200 using the same model architecture and same sampling scheme as in Section
Table C.4 shows that topic posterior recovery loss steadily decreases as the number of epochs gets larger. Interestingly, even with just training 50 epochs. our model can recover the topic posterior within a Total Variation distance of less than 0.3 off the true topic posterior for documents generated from all four topic models.
C.5 HYPERPARAMETER TUNING EXPERIMENTS
As described in Section 5 the two types of neural network architecture we use are fully-connected neural networks and attention- based neural networks (see Figure C.3). In this section, we present some of our TV results of different combinations of hyperparameters for both types of models. Note that our attention-basee 2017) in that use batch normalization instead of layer normalization in every residual connection. We only use one attention head in each multi-head attention layer. During training, we use AMSGrad optimizer and initial learning rate of 0.0001 or 0.0002. We reduce the learning rate by 50% whenever validation loss does not decrease 10 epochs.
neural network slightly differs from typical transformers (Vaswani al.
The main hyperparameters in fully-con collecconde neural networks include the hidden dimension, the number of layers, and whether we apply residual connections. Table|C.5|shows the TV distance results when vary these hyperparameters for the I.A scenario when 1. However, fully-connected
23

---

Published as a conference paper at ICLR 2023
Dirichlet hyperaaaameter
Document type
Number epochss
7
Pure-topic
25
0,0005
0.11225
0.1756
0.6222
0,8285
0.2873
100
0,0160
0.0363
0.0511
0.0911
0.1257
0.0517
TDA
25
0,1236
0.1583
0.1812
0.2022
0.2218
0.2117
100
0,0805
0.1018
0.1101
0.1361
0.1538
CTM
25
0,1089
0.1599
0.1563
0.1747
0.1971
100
0,0623
0.0948
0.1102
0.1
0.1243
0.1101
PAM
25
0.1072
0.1342
0.1480
0.1688
0.1754
100
0.0543
0.0755
0.0859
0.1087
0.1095
0.0971
Table C.4: Topic posterior recovery loss of our self-supervised learning approach, measured in Total Variation distance, for all four types of documents and a 1, 7, The number of training epochs ranges from 25 to 200, and we sample 60K new training documents in every 2 epochs, which corresponds to 720K 6M training documents.
eborbution
Adt
Embedding
Figure C.3: Our attention based neural network architecture
neural networks perform poorly in CTM and PAM case. For instance, when 1 and hidden dimension = 4096, we find that hyperparameters that work well in the LDA case no longer yield satisfactory results (Table
TV
Hidden dimension
layers (residual)
1024
2048
4096
0.1509
0.1112
0.1095
0,7902
0.7893
0.7901
(residual)
0.0871
0.0867
0.0862
(residual)
0.0822
0.0835
0,0709
Table C.5: Fully-connected neural network's performance the LDA a scenario.
For attention- based neural networks applied to CTM and PAM scenarios, we mainly consider the number of layers and the hidden dimension per attention layer. Table C.7 presents the TV distance between recovered topic posterior and true topic posterior in the CTM setting, with varying number of layers and attention dimension for a 1,3, and using a more coarse estimate the topic posterior than what we use as our final true CTM topic posterior. We find that the model performs the best when attention dimension is 768 or 1024, and when the attention dimension increases 2048 the model gives a higher TV (see Table C.8).
24

---

Published as a conference paper at ICLR 2023
TV
Layer Type
layers
regular
residual
3
0.1824
0.1663
0.1724
0.1656
0.1700
0.1556
Table C.6: Fully-connected neural network's performance in the CTM scenario with 4096 hidden dimensions,
TV
layers
0
Attention dimension
8
0.0902
0.0844
0,0814
0.0851
0.0890
0.0836
768
0.1471
0.1429
0.1398
1024
0.1440
0.1384
0.1383
5
768
0.1794
0.1767
0.1708
1024
0.1878
0.1787
0.1782
Table C.7: Attention breettoo--baed neural network's performance on CTM documents for a 1,3,
TV
Attention dimension
768
1024
2048
3
0.1471
0.1440
0.1670
5
0.1794
0.1878
0.1922
Table C.8: 4-layer attention-base neural network's performance CTM documents when attention
layer's dimension varies, for a = 3, 5.
C.6 TOPIC POSTERIOR RECOVERY LOSS AND CONDITION NUMBER
We have proved in Theorem 3.3 that the upper bound of topic posterior recovery loss depends on the l Condition Number s( Af). In this section, we show that k( s((A) is small for every topic-word matrix in our experiments (Figure C.4). Note that we use the same topic-word matrix A for pure-topic model and the LDA model. For CTM and PAM, of the topics that corresponds with pairwise topic correlations. Therefore, pure-topic model and LDA model share the same k(At), and CTM and PAM share the same n( x((At).
the same topic-word matrix up to reordering
Figure C.4: &, Condition Number for topic-word matrix A
C.7 VISUALIZING TOPIC CORRELATIONS FOR PAM
For documents generated by the PAM model, we plotted the estimated posteriors by the self- supervised approach and posterior inference assuming different priors in Figure C.5 From this figure (especially in Documents and 4), one can qualitatively see that the estimated posterior and the PAM MCMC posteriors aware of the correlation between topies, while the LDA MCMC posterior fails to take the correlation into consideration and hence is more different from the ground truth topic proportions.
C.8 TOPIC POSTERIOR RECOVERY FOR
We have shown by experiment that self-supervised learning accurately recover the topic posterior when f = 1, that is, when the neural network model is trained to predict one missing word from document. In this subsection, using the self-supervised objective to train the neural network model to predict missing word in document. For the sake of reducing computational complexity, change the setting from
experiments for t 2 case in Theorem 3.3 which means
25

---

Published as a conference paper at ICLR 2023
Estimated posteriors
PAM posteriors
Ground truth topic proportions
LDA MCMC posteriors
Figure C.5: Comparison of estimated, PAM MCMC, LDA MCMC, and ground truth posterior topic proportions for sample test documents generated by PAM. Overall, across set of 200 test documents, the estimated posteriors were able to outperform the misspecified LDA posteriors.
V 5000, K 20, K, 10 to V 500, K 8, and K, (where K, a parameter in the PAM model) while keeping document length, resample rate, training epoch, and the number of test documents the same as in Section[5] Once a neural network f trained, from the proof of Theorem 3.1 in Section A.2 we know that the training loss minimizer f satisfies f(z) = (A @ (W post when t = 2, so we obtain vec(W by vec( post) (A @ A)' f(x) where we expect W post to be a symmetric matrix. But we find that in our experiment W Wpost is nearly symmetric but not perfectly symmetric, and amend this, we use 2 (opost + WT as the SSL-recovered posterior tensor.
Document Type
Pure
LDA
CTM
PAM
6
3
8
6
6
5
6
6
8
8
8
10
Table C.9:
The number of attention blocks used in the neural network architecture for / = 2 for all
four types of documents and a = 1, 3, 5, 7,
We observe that the attention-based neural network architecture with 384 hidden dimension and varying number of attention blocks (shown in Table C.9) is performs better on learning the distribution of 2 missing words than the fully-connected neural network for all four types of documents, so all of the following results are based the output of trained attention-based neural network. We use the posterior mean from MCMC assuming the correct prior as the ground-truth topic posterior, and use the TV distance to measure how far the SSL-recovered topic posterior deviates from it, shown in T. In most scenarios, the TV distance is less than 0.15, and TV distance gets slightly larger as a increases, i.e., the word distribution of each topic gets less distinguishable.
TV Distance
Document Type
Pure
LDA
CTM
PAM
0.0081 = 0.0002
0.0636 = 0.0031
0.0748 # 0.0053
0.0748 # 0.0035
3
0.0291 # 0.0003
0.0870 # 0.0039
0.1206 # 0.0069
0.1055 # 0.0047
5
0.0431 = 0.0004
0.1075 + 0.0050
0.1523 = 0.0067
0.1544 = 0.0062
7
0.0578 + 0.00004
0.1258 # 0.0049
0.1627 = 0.0058
0.1693 = 0.0065
9
0.0662 + 0.0006
0.1413 + 0.0052
0.1859 + 0.0061
0.2770 + 0.0152
Table C. 10: Topic posterior recovery loss of our self-supervised learning approach for = 2, measured in Total Variation distance, for all four types of documents and confidence interval.
1. 1,3, 5, 7,9. We report the 95%
We also measure how the largest entries in the SSL-recovered topic posterior overlap with the largest entries in the topic prior. Note that for each document, since the underlying topic of each word is
26

---

Published as a conference paper at ICLR 2023
Document Type
Pure
LDA
CTM
PAM
1
1.0 + 0.0
0.9350 = 0.0342
0.9088 + 0.0286
0.8638 + 0.0402
3
1.0
0.8800 = 0.0450
0.8700 = 0.0363
0.8075 = 0.0476
5
1.0
0.0
0.8950 = 0.0425
0.8050 = 0.0473
0,8012 # 0.0512
7
1.0
0.0
0.9100 = 0.0397
0.8450 = 0.0428
0.8125 = 0.0472
9
+.0
0.8950 = 0.0425
0.7675 = 0.0505
0.6625 = 0.0485
Table C.11: The major topic pair recovery rate of our self-supervised learning approach for 2 for all four types of documents and a 1, 5, 7, 9. We report the 95% confidence interval.
drawn iid, the joint topic prior distribution for the missing words is the Kronecker product of the document's topic prior distribution with itself. For the overlap rate, in the / case we measured the overlap of the top topie for pure topie and LDA documents and the overlap of top 2 topics for CTM and PAM documents, because most pure topic documents and LDA documents are generated by one dominant topic whereas most CTM and PAM documents are generated by two correlated dominant topics. Similarly, in the t = 2 case we measure the overlap of the top entry for pure topic and LDA documents and the overlap of top 2
entries for CTM and PAM documents. We will refer to this overlap rate as the "major topic pair recovery rate". The results are shown in Table C. Overall, the overlap rate is very high, which indicates that for = 2 our self-supervised learning approach can capture the most dominant topic(s) for the 2 missing words.
C.9 DETAILS MCMC NUTS SAMPLING METHOD
In our experiments, we use PyMC3 package Salvatier et al. (2016) to perform NUTS-based MCMC use 3000 tuning iterations tune the sampler's step size that we approximate an acceptance rate of 0.9. After tuning, we draw 2000 samples from the posterior distribution and use the mean of these 2000 samples our best estimate for the topie posterior mean.
posterior sampling. Specifically,
To further check the performance of NUTS, we look into the LDA documents. As shown below, we observe that MCMC posterior assuming LDA model (bottom row) also very similar the ground truth (top row) used to generate the documents.
Desamnet 1
Decumen 2
Doummet
Teac
Teal
Document 1
Document 2
Document
Document
Figure C.6: Comparison of NUTS- MCMC topic posterior assuming correct model and the document-gennerating topic proportions for sample test documents generated by LDA.
27

---

Published as a conference paper at ICLR 2023
D REAL-DATA EXPERIMENT ADDITIONAL DETAILS
In this section, we give more details about our real data experiments in Section 6 describe the data processing. We give a more detailed description of baselines and how we extracted our representations in Section D.2 In Section D33] we report more results using different model architectures. Moreover, in Section D.4 we offer details about experiments other real-world datasets and report the results.
In Section D.1 we
D.1 DATA PROCESSING
Here detailed usage of the AG news dataset by Zhang et al. (2015). Each category has 30,000 samples in the training set and 19,000 samples in the testing set. We first preprocessed the data by removing punctuation and words that occurred in fewer than 10 documents, obtaining in vocabulary of around 16,700 words, in a similar fashion done by Tosh et al. (2021b).
To split the data set into unsupervised dataset and supervised dataset, we selected random sample of 1000 documents as labeled supervised dataset for each category of documents, while the remaining 11,000 documents fall into unsupervised dataset for representation learning.
D.2 EXTRACTING REPRESENTATIONS
To give some details about two baseline representations we used, we described in details as follows:
Bag of Words (BOW): For single document, we constructs BOW embedding by creating a bag-of-words frequency vector of the dimension of vocabulary size, where each entry represent the frequency of words with id in that particular document.
Word2vec: To generate Word2vec representation fitted the Skip-gram word embedding model on unsupervised dataset Mikolov et al. (2013). The implementation done through the Gensim library Rehurek & So1ka/2011), where we used embedding dimension of 200 (after tuning, see D.II and window size of 5. Representation is taken as the average of all trained word embeddings in single document.
LDA: We construct a representation derived from LDA by first fitting Latent Dirichlet Allocation model Blei et al. (2003) on the unsupervised dataset and then use the posterior distribution over topics given a document as representation. We implemented LDA using seikit-learn Pedreeooa et al. (201T) with default parameters except for number of topies, which we use as representation size. We use final dimension of 100 (after tuning, see Figure D.I
LDA Tame Srbedding Compurionn
0.0
6.50
Line bine
9at manmpte
Figure D.I: Test Accuracy for different number of topics used for fitting LDA embedding model on the AG News dataset (Using N=10 for 95% Confidence Interval).
For our own self-supervised method, to extract a representation, we attempted 1) softmaxtlstt layer: apply Softmax directly to the last layer output, 2) Word2vec+Last layer: apply an Word2vec embedding matrix to last layer output after Softmax to reduce dimension (the Word2vec matrix is trained over the unsupervised dataset), and 3) teftmax+ on top of the second-tu tonud-to-lad to-t layer (equivalent to applying an identity matrix to replace the last layer). The dimension of representation with Word2vec is 300 while the softmax + second2 last layer has a dimension of 4096. The original last layer representation has a dimension of the vocabulary size (around 16, 6,700), and has shown inferior results than those with dimension reduction techniques.
layer: apply Softmax function
28

---

Published as a conference paper at ICLR 2023
Embedding dimension
Test Accuracy
200
0.8498 = 0.0007
300
0.8488 # 0.0007
500
0.8484 + 0.0007
700
0.8483 = 0.0006
1000
0,8488 # 0,0009
2000
0.8473 + 0.0006
3000
0.8481 = 0.0001
5000
0.8482 = 0,0006
Table D.1: Test Accuracy for different embedding dimension used for fitting Skip-gram word embedding model on the AG News Dataset (Using N=10 for 95% Confidence Interval). We fixed the window-size to be default of 5 and used all 4000 training examples.
We included comparison of the two approaches with dimension reduction in Table D.2, from which we observe that Softmax on the second-to-last layer output performed better across layers of 3,4 and 5. We reported the result using the softmaxt+ layer in Figure 2
Test Accuracy
Method
# layers (residual) Word2vec+last layer
softnmax+seeoondllass layer
softmax+lastlayer
3
0,8508
0.8714
0,8395
0.8450
0.8621
0.8430
5
0.8492
0.8689
0.8382
Table D.2: Test Accuracy for different representation extraction method. We fixed the rest of hyperparameters to be: 5000 embedding dimension, 150 epochs, 0.0002 learning rate, sampling words in labels, weight decay of 0.01 and resample rate 2.
D.3 MODEL ARCHITECTURE
We include here more details model architecture used in real data experiments. We used residual model, and tested two main hyperparameters: number of residual blocks and hidden dimension size. We fixed the rest of hyperparameters: epochs, with 0.0002 learning rate, weight decay of 0.01 and resample rate of 2. We sampled 4 words in labels for variance reduction purpose.
used 5000 embedding dimension and trained for 150
Varying Depth and Width We investigated both the effect of model depth and width on the perfor- mance of RBL representation. We trained networks of width 2048, 3000, and 4096 nodes respectively, where the number of node refers to the number of node in the linear layer inside a residual block. We varied the depth of the network by choosing to place 3, and of such block.
As shown in Figure D.2 it appears that using wider models in the unsupervised phase leads to better performance when training a linear classifier on the learned representations. Number of residual block does not seem to have a clear relationship with test accuracy from limited experiments we ran. The best accuracy is achieved when we have 3 residual blocks with dimension of 4096, which what we used to generate results in Figure 2
D.4 MORE REAL--DATA EXPERIMENTS
In addition to the experiments on the AG news dataset, we also run experiments on the IMDB movie review sentiment classification dataset by Maas et al. (201T) and the Small DBpedia ontology dataset, which is a subset of the DBpedia ontology by Zhang et al. (015), following the preprocessing by particular, the IMDB movie review dataset contains a unlabeled set 50k
Tosh et al. (2021b). movie reviews, a training set of 12.5k positive movie reviews and 12.5k negative movie reviews, and a test set of 12.5k positive movie reviews and 12.5k negative movie reviews. We randomly selected 1 documents from each class in the training set 3.5k documents in the test set as our test documents, and used the remaining 91k documents as our unsupervised dataset for training SSL model to learn document representation. After filtering,
training document and selected
29

---

Published as a conference paper at ICLR 2023
0N
0.00
081
200
(a)
(b)
Naccal Uipel cacqua 428)
6.02
(c)
Figure D.2: RBL representation performance varies with different residual model capacity: the best performing is the one with 3 layers and 4096 hidden dimensions, and was what we used for Figure In all these runs, Softmax second-to-Be layer obtain representations
0.5
040
030
1803
1.00
(b)
Figure D.3: Performances of RBL and baselines (BOW, Word2vec, LDA) IMDB and SmalIDBpe- dia datasets. RBL generally outperforms all baselines.
the vocabulary size is about 35k. The Small DBpedia dataset consists 4 non-overlapping classes, which are company, artist, athlete, and office holder from the original DBpedia ontology dataset, and each class consists 40k training documents and 5k test documents. We randomly chose 1k training documents each class as labeled training documents and used the rest of training documents as the unsupervised dataset, and used the original test documents as our test dataset. After we filtered rare words in the original DBpedia dataset, the vocabulary size is about 10k.
In our experiments, the setup of the baselines follows Section D.2 We plot the test accuracy of our method against the baselines in Figure D.3 We did not include NCE in representation comparison since limited tuning does not achieve comparable results as in Tosh et al. (202 (tt0oi).Paaamaeee... Parameter tuning result for LDA and Word2vec is documented in Figure D.4 and Table D.3 respectively.
In the left panel of Figure D33 we compare the performance of semi-supervised learning on IMDB dataset of our representation RBL well baselines. We point out that in general, RBL has similar performance to Word2vec, outperforming LDA and BOW. However labeled training data is less abundant, Word2vec and LDA have slightly stronger performance.
30

---

Published as a conference paper at ICLR 2023
6.0
18
030
-1100
Taaiig
(b)
Figure D,4: Test Accuracy for different number of topics used for fitting LDA embedding model on the IMDB and Small DBpedia dataset. We use 100 as final dimension for both datasets based on the tuning above.
Test Accuracy
Datasets
Embedding Dimensions
IMDB
Small DBpedia
0.8423=0.0014
0,94401 0.0007
200
0.8529=0.0012
0.9541=:0.004
500
0.8545=0.0011
0.9568110.0044
T000
0.8544 0.0005
1.956740.00003 0.0003
Table D.3: Test Accuracy for different embedding dimension used for fitting Skip-gram word embedding model on the IMDB and Small DBpedia Dataset (Using N=10 for 95% Confidence Interval), We fixed the window-size to be default 5 and use available training examples. We use dimension of 500 for both datasets for final baseline based on the results above.
In the right panel Figure D33 we perform a similar representation comparison on Small DBpedia dataset. While RBL outperforms all other baselines when there are more than around 1000 training examples, it performs worse than Word2vec and LDA when labeled training data is less abundant.
31


Reviews:
[{"review_id": "kL75U4I-MU", "reviewer": "Reviewer_6Fra", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies self-supervised learning (SSL) for general topic models. The main approaches of SSL include reconstruction-based objective and contrastive objective. The paper investigates that feature representations given by SSL provide basis functions of the higher order posterior mean. Also, the robustness analysis provides the error bound of near-optimal solutions in the reconstruction-based SSL. In numerical experiments, the reconstruction-based SSL is examined and compared to the inference with some topic models. The numerical results indicate that the SSL approach is superior to the statistical inference using misspecified models. ", "summary_of_the_review": "This paper tackles an important problem. However, the novelty compared to existing works is not clearly discussed. ", "strength_and_weaknesses": "Strength\n- This paper tackles an important problem. Numerous theoretical analysis on SSL has been published in recent years. Most of them focus on the generic effectiveness of the feature representation of the SSL for downstream tasks. This paper restricts the problem to the topic model that includes the main applications of SSL. The robustness analysis in Theorem 3.3 is practically important. \n\nWeaknesses\n- This paper includes a generalized result of [Tosh et al.'21]. However, a detailed comparison with [Tosh, et al.'21] is not presented. More concretely, which condition is relaxed compared to [Tosh et al.'21]? How much is the practical usefulness of such a relaxation?\n- The required dimension of the feature representation is relatively high. Is it possible to show how to reduce the dimension of the functions f and g to obtain similar theoretical results? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QXq0mgxPzPj", "reviewer": "Reviewer_KUB2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of representation learning for bag-of-words count data, trying to understand the usefulness of recent self-supervised learning, including reconstruction objectives (Eq 1) and contrastive objectives (Eq 2). The key question is: why might these approaches be better than previous ones? The paper's goal is to develop theorems and experiments that try to compare/relate the representations produced by these objectives to the per-document posterior representations of probabilistic topic models (like Latent Dirichlet Allocation).\n\nThe main claims see to be that SSL representations are\n\n- as good as the true topic model when the data is generated by a true topic model (backed by theorems in Sec 3/4 and experiments)\n- better than a topic model that is *misspecified* for the data (backed by experiments in Sec 5)\n\nTaken together, the paper suggests that SSL is *robust* to model misspecification, and this may help explain why it is a performant choice for real bag-of-words data.\n\nI'll note the first claim seems somewhat surprising, given that a key part of a topic model is its prior over the document's distribution over topics $$w$$, and the SSL does not know about this prior at all, yet recovers moments of the posterior well.\n\nSec. 5 performs experiments on synthetic data where the true topic model is known (several possible models are considered: pure topic model, LDA, Correlated Topic Model). Here, results in Table 1 assess the distance between the SSL recovered posterior and the \"true\" posterior (fit with NUTS).\n\nSec. 6 provides one brief experiment on real data (AG news, where each doc is one of four categories). Here, representations are assessed in terms of their ability to predict the category of documents in the test set. The authors show how reconstruction-based SSL outperforms other baselines as training set size varies from 100-400 documents.\n\nWhile most theoretical results (Sec. 3) and experiments (Sec. 5-6) focus on reconstruction-based SSL, contrastive SSL is the focus of a theorem in Sec. 4.\n", "summary_of_the_review": "Overall, I think this study offers some nice new insights about why self-supervised learning might be effective on data from a wide range of possible models. If my concerns are addressed, I think this could be a valuable paper to accept.", "strength_and_weaknesses": "\n## Strengths\n\n* Paper addresses a timely question (understanding the representational power of self-supervised learning)\n* Theoretical results cover both reconstruction (Sec 3) and contrastive (Sec 4) paradigms of SSL\n* The implications of Theorem 3.1 are reasonably explained to the reader\n* Comparisons to many (4) distinct types of topic models in Sec. 5 help back up claims\n* Real data demonstration in Sec. 5 looks promising\n\n## Weaknesses\n\nHere I list the primary weaknesses as I see them (each is elaborated below under its respective heading of Quality/Clarity/Reproducibility)\n\n* W1: Missing assessment of sensitivity to length of document / assymmetric priors\n* W2: Better to compare predicted distributions over words than topics?\n* W3: Details of NUTS-based posterior sampling missing\n* W4: Clarify how A is handled in experiments\n* W5: Releasing code would significantly help reproducibility\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Hzvxer8JQYU", "reviewer": "Reviewer_DvbM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper demonstrates the point that Self-supervised learning is immune to the choice of the probabilistic model, therefore shows robustness to model misspecification. In contrast, traditional approach of probabilistic modeling hinges on the specific predefined model. Therefore, the advantage of self-supervised learning is that SSL may outperforms traditional model when inferencing with misspecified model. The evaluation parts measures topic posterior recovery loss as the Total Variation (TV) distance between the recovered topic posterior and the ground truth topic posterior mean, supporting the advantage of the SSL methods. ", "summary_of_the_review": "The paper gives a series of theorems justifying the bounded error of topic classification task, when using the features out of the SSL model.  The paper is well written, and evaluation supports the claims. I have some concerns in linking the theorems to the actual classification error of the topics, I expect to hear from the authors in this regard.  ", "strength_and_weaknesses": "Strength:\n\nThe paper demonstrates the ability of self-supervised learning to adapt to different models. Since the prior distribution is not present in the SSL models,  SSL methods therefore avoid the risk mis-specifying the model. This helps the model to better inference the posterior of the topics. In the meanwhile, the paper gives the explicit theoretical guarantees for self-supervised learning with the topic modeling task,  and showed that SSL can provide useful information about the topic posterior without bounded error, even if SSL does not have prior information about the topics. This observation is relatively new and fresh to me. \n\nWeakness:\n\nThe paper is a bit hard to follow in the sense that the proposed theories are very loosely connected to the actual classification error on the downstream task in the form of the polynomials of the w variables. I would appreciate it if the authors might comment on the significance of assuming error bound on the polynomials form of the posterior on w variables.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster