Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper demonstrates the point that Self-supervised learning is immune to the choice of the probabilistic model, therefore shows robustness to model misspecification. In contrast, traditional approach of probabilistic modeling hinges on the specific predefined model. Therefore, the advantage of self-supervised learning is that SSL may outperforms traditional model when inferencing with misspecified model. The evaluation parts measures topic posterior recovery loss as the Total Variation (TV) distance between the recovered topic posterior and the ground truth topic posterior mean, supporting the advantage of the SSL methods. 
strength_and_weaknesses: Strength:

The paper demonstrates the ability of self-supervised learning to adapt to different models. Since the prior distribution is not present in the SSL models,  SSL methods therefore avoid the risk mis-specifying the model. This helps the model to better inference the posterior of the topics. In the meanwhile, the paper gives the explicit theoretical guarantees for self-supervised learning with the topic modeling task,  and showed that SSL can provide useful information about the topic posterior without bounded error, even if SSL does not have prior information about the topics. This observation is relatively new and fresh to me. 

Weakness:

The paper is a bit hard to follow in the sense that the proposed theories are very loosely connected to the actual classification error on the downstream task in the form of the polynomials of the w variables. I would appreciate it if the authors might comment on the significance of assuming error bound on the polynomials form of the posterior on w variables.

clarity,_quality,_novelty_and_reproducibility: The paper lacks a bit of clarity in eventually linking the polynomial form of w with the linear classification error. This helps to verify the significance of the proposed methods. 

The paper is novel in the sense that this is the first paper I came across in attributing the model mis-specification with the actual performance of SSL methods. 
summary_of_the_review: The paper gives a series of theorems justifying the bounded error of topic classification task, when using the features out of the SSL model.  The paper is well written, and evaluation supports the claims. I have some concerns in linking the theorems to the actual classification error of the topics, I expect to hear from the authors in this regard.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies self-supervised learning (SSL) for general topic models. The main approaches of SSL include reconstruction-based objective and contrastive objective. The paper investigates that feature representations given by SSL provide basis functions of the higher order posterior mean. Also, the robustness analysis provides the error bound of near-optimal solutions in the reconstruction-based SSL. In numerical experiments, the reconstruction-based SSL is examined and compared to the inference with some topic models. The numerical results indicate that the SSL approach is superior to the statistical inference using misspecified models. 
strength_and_weaknesses: Strength
- This paper tackles an important problem. Numerous theoretical analysis on SSL has been published in recent years. Most of them focus on the generic effectiveness of the feature representation of the SSL for downstream tasks. This paper restricts the problem to the topic model that includes the main applications of SSL. The robustness analysis in Theorem 3.3 is practically important. 

Weaknesses
- This paper includes a generalized result of [Tosh et al.'21]. However, a detailed comparison with [Tosh, et al.'21] is not presented. More concretely, which condition is relaxed compared to [Tosh et al.'21]? How much is the practical usefulness of such a relaxation?
- The required dimension of the feature representation is relatively high. Is it possible to show how to reduce the dimension of the functions f and g to obtain similar theoretical results? 

clarity,_quality,_novelty_and_reproducibility: Overall this paper is clearly written. However, the novelty is not very clearly mentioned. Showing the details of existing work [Tosh et al.] and clarifying the extension would be necessary. 
summary_of_the_review: This paper tackles an important problem. However, the novelty compared to existing works is not clearly discussed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of representation learning for bag-of-words count data, trying to understand the usefulness of recent self-supervised learning, including reconstruction objectives (Eq 1) and contrastive objectives (Eq 2). The key question is: why might these approaches be better than previous ones? The paper's goal is to develop theorems and experiments that try to compare/relate the representations produced by these objectives to the per-document posterior representations of probabilistic topic models (like Latent Dirichlet Allocation).

The main claims see to be that SSL representations are

- as good as the true topic model when the data is generated by a true topic model (backed by theorems in Sec 3/4 and experiments)
- better than a topic model that is *misspecified* for the data (backed by experiments in Sec 5)

Taken together, the paper suggests that SSL is *robust* to model misspecification, and this may help explain why it is a performant choice for real bag-of-words data.

I'll note the first claim seems somewhat surprising, given that a key part of a topic model is its prior over the document's distribution over topics $$w$$, and the SSL does not know about this prior at all, yet recovers moments of the posterior well.

Sec. 5 performs experiments on synthetic data where the true topic model is known (several possible models are considered: pure topic model, LDA, Correlated Topic Model). Here, results in Table 1 assess the distance between the SSL recovered posterior and the "true" posterior (fit with NUTS).

Sec. 6 provides one brief experiment on real data (AG news, where each doc is one of four categories). Here, representations are assessed in terms of their ability to predict the category of documents in the test set. The authors show how reconstruction-based SSL outperforms other baselines as training set size varies from 100-400 documents.

While most theoretical results (Sec. 3) and experiments (Sec. 5-6) focus on reconstruction-based SSL, contrastive SSL is the focus of a theorem in Sec. 4.

strength_and_weaknesses: 
## Strengths

* Paper addresses a timely question (understanding the representational power of self-supervised learning)
* Theoretical results cover both reconstruction (Sec 3) and contrastive (Sec 4) paradigms of SSL
* The implications of Theorem 3.1 are reasonably explained to the reader
* Comparisons to many (4) distinct types of topic models in Sec. 5 help back up claims
* Real data demonstration in Sec. 5 looks promising

## Weaknesses

Here I list the primary weaknesses as I see them (each is elaborated below under its respective heading of Quality/Clarity/Reproducibility)

* W1: Missing assessment of sensitivity to length of document / assymmetric priors
* W2: Better to compare predicted distributions over words than topics?
* W3: Details of NUTS-based posterior sampling missing
* W4: Clarify how A is handled in experiments
* W5: Releasing code would significantly help reproducibility


clarity,_quality,_novelty_and_reproducibility: 
## Novelty

The paper cites adequate recent work in this space. 
Tosh et al. (2021a) is the most closely related study, which shows "contrastive learning is capable of recovering a representation of documents that reveals their underlying topic posterior information to linear models".

This previous work focuses on contrastive SSL, while the present paper's focus is primarily on reconstruction SSL. The present paper does look at the contrastive setting (see Sec. 4), and the theorem there nicely removes the anchor words assumption needed by Tosh et al.



## Quality

Overall, the study and especially the synthetic experiments seem well designed.

However, there a couple major issues I'd like to hear more about

### W1: Missing assessment of sensitivity to length of document / assymmetric priors

The successful recovery of the posterior despite complete lack of knowledge of the prior is somewhat surprising. I wonder if two things could explain this:

* the likelihood is dominating the per-document posterior in most of the experiments considered, and the specific choice of prior is thus less relevant
* the chosen priors used are fairly default in their assumptions (e.g. LDA uses a symmetric Dir(1/K) prior, so all topics are equally likely to appear in the document) 

I'm curious if the results in Sec 5 would hold even if we had an asymmetric prior (e.g. LDA where some topics more common than others) and if test time documents were significantly shorter (e.g. maybe 2-10 words, much smaller than expts in Sec 5 which seem to use Poisson(30) words). 

Clearly, in the limit where there are *no* observed words in a test document, the posterior must equl the prior, and the SSL doesn't have access to the prior (though I guess it can try to learn it from abundant training data). I'm just hoping to understand the limits of the proposed method for recovering the posteriors of classic probabilistic models. 


### W2: Better to compare predicted distributions over words than topics?

The comparisons in Table 1 all assess estimated per-document distributions over topics, denoted w

I'm wondering if there's a chance that if a (misspecified model) gets this distribution wrong, it could still have a good prediction for the missing word. For example, in a given fixed topic-word matrix A, two different topics can put mass on the same word v. As long as the posterior predictive over words (=Aw) is close to the truth, the model would predict well, even if the model uses a different topic than the "true" one.

Do the authors have thoughts about this? 
Probably the topics are distinct enough that there is one clearly "best" w for each document, but I'd just like to check in on this.


## Clarity / Reproducibility

Overall mostly easy to understand, I appreciated that in a paper with theory-heavy contributions the practical implications of the theorems (e.g. Theorem 3.1) were clarified to the reader in the main text.

Major issues are listed here

### W3: Details of NUTS-based posterior sampling missing

Even though widely used in probabilistic programming packages, I've often found NUTS with default settings may not sample well from some distributions without further tuning. Because NUTS isn't commonly used to do posterior inference for models like LDA (even though of course it seems perfectly suitable), I'm hoping to hear a little more about what diagnostics you've performance to be satisfied your chains are converging to the intended posterior distribution.

For example, for LDA have you compared your NUTS results to classic Gibbs samplers for LDA? Or to the "true" w used to generate a document?

Also, when you know the "true" generative model used to simulate the data (as in Table 1), do you really need to use NUTS at all?

### W4: Clarify how topic-word parameters A are handled in experiments

In Table 1 and 2, can you clarify whether A is held fixed to the true topic-word parameters used to generate all data? And that each method is only solving the posterior estimation of w given observed x and fixed A? That detail is missing in the paper.

Furthermore, the text says "Often A will be drawn from ... Dir(\alpha /K)...." 

Are there cases where you generate A some other way? If so, how?

### W5: Releasing code would significantly help reproducibility

Hopefully this is an easy one to address.
I think the community would benefit from being able to compare to the toy and real datasets studied here under similar settings.


Minor issues:

* Would be useful to be clear (in supplement) how exactly you calculate the 95% confidence intervals in various tables.

summary_of_the_review: Overall, I think this study offers some nice new insights about why self-supervised learning might be effective on data from a wide range of possible models. If my concerns are addressed, I think this could be a valuable paper to accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores the quality of representations learnt by Self-supervised learning(SSL) through Topic models. It argues that Topic models are robust to model mis-specification when learnt either through contrastive or reconstruction based objective functions. 
The main result 
is the paper shows that the obtained representations can represent any polynomial function of the posterior weights given enough data. The line of investigation is similar to earlier work by Tosh( et al'21). However they go beyond Tosh et al., by removing the Anchor Word assumption.

The main strengths of the paper are 
1. Contributes to the understanding of feature representation through SSL. 
2. It provides rigorous statements of the key claims especially the robustness claims

The main weakness of the paper
1. Most of the groundwork is already there in Tosh et al.'21, in particular theorem 3 of the reference. Here the results are extended to Topic models. Here the reasoning remains more or less the same, but the anchor word assumption is relaxed. Appears to be modest contribution wrt to state of the art
2. The technical presentation can significantly improve






justification_for_why_not_higher_score: The improvement over SOTA is not that spectacular. However this is to be expected as the field of Topic models is more than two decades old. Also, the referees did not raise the scores after perusing the rebuttal. This makes it difficult to make a stronger case.
justification_for_why_not_lower_score: Most issues raised by the reviewers have been addressed by the authors.
Decision: Accept: poster