Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: See summary of the review below.
strength_and_weaknesses: See summary of the review below.
clarity,_quality,_novelty_and_reproducibility: See summary of the review below.
summary_of_the_review: I think this is an interesting paper, but I found the motivation for the approach lacking. 

The paper describes various issues in going from MLM parameter estimation objectives, to density estimation over sequences. The paper first describes a number of naive methods (section 3.1.1) that are clearly heuristic, lacking clear guarantees, and which not surprisingly lead to bad performance. The paper then describes prior results from pseudo-likelihood that do lead to well-founded estimators, and give better performance. 

This doesn’t feel like a significant enough contribution for ICLR, for two reasons. 1) it’s not surprising at all that using well-founded methods, known a long way back in the pseudo-likelihood literature, give improvements. 2) density estimation of this type is of limited interest, as auto-regressive models do not have these issues; compelling arguments need to be made for why the MRF-based approach is preferable, and how the method might be scaled from pair-wise interactions.


In addition, I think the motivation/discussion in the introduction needs to be clarified, specifically:

“From a probabilistic perspective, masked language models (MLM) make strong independence assumptions”. There is *a lot* riding on the phrase “from a probabilistic perspective” here. I think what the authors are getting at is going from the MLM training criterion to a full density p(x) over sentences x - and how this may be challenging given the way current MLM objectives are set up. That’s ok, but two comments: 1) the original MLM objective was never motivated for density estimation, i.e., estimation of p(x), it was used for pretraining/representation learning. It was always clear that there might be some connection through pseudo-likelhood to density estimation, but the method never depended on that link being made. The MLM method is also completely sound in estimating a particular conditional distribution that arises from masking the input sentence. 2) the authors themselves do not give a particularly compelling method to density estimation themselves, which leaves the motivation on somewhat shaky ground. 

The introduction continues in this vain: “such model misspecifications arising from incorrect statistical assumptions may not be catastrophic”. Again, the original MLM objective did not try to derive a model for p(x), so the term “model misspecification” is in my opinion highly misleading here. 

“However, we observe that MLMs are increasingly being employed as probabilistic models of language” - presumably by a “probabilistic model of language” you mean a density p(x) over sentences? The authors need to clarify these terms.

I’m concerned that a naive reader will get the impression that MLM methods are fundamentally unsound, and that is just not a correct characterization in my opinion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies how a masked language model (MLM) can be used as a traditional language model. The authors consider the case of two masked tokens and they proposed pairwise Markov random fields (MRFs). They experimentally show that pairwise MRFs are worse probabilistic models of language from a perplexity standpoint. They studied two formal approaches for deriving better language models and one algorithmic solution using a feed-forward neural layer.
strength_and_weaknesses: Pros

* Showing that distributions defined with two masked tokens do not define unary distributions close to that of the original MLM
* Proposal of statistically founded methods to answer this problem
* Proposal of an algorithmic solution to give a proxy

Cons

* The motivations are not clear at least to me
clarity,_quality,_novelty_and_reproducibility: The paper introduces so-called pairwise MRFs in order to define language models for MLMs when pairs of words are masked. From an NLP perspective, I do not see for which applications it could be useful to mask pairs of words. For applications, the more interesting case should be when the two tokens are contiguous, i.e. the case for which the independency condition is false. Moreover, if such applications do exist, why would it be useful to consider them as language models for these applications. Let us suppose that pairwise MRFs are useful, now is the question why the unary distributions they defined should coincide with the distributions of the original MLM.

I think the paper could be made clearer: it is not always easy to see which MLM you are speaking of (pairwise MLM, original MLM, parent MLM, ...); it should also made clear at the beginning that a pairwise MLM is not trained with masked pairs but that you consider "classical" MLMs. This being said, the study is well designed from both a theoretical and experimental perspective. It does not come as a surprise that the AG model performs better given that the model is optimized for a joint distribution which is faithful to the original MLM. The experimental results for the algorithmic approach are promising.
summary_of_the_review: The paper contains a statiscally funded study completed by interesting experiments. The motivations should be made explicit. For now, I am not convinced by the usefulness of the study at least from an NLP perspective.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper examines the perspective of interpreting masked language models (MLMs) as a generative model, where the unmasking procedure corresponds to predicting unary conditionals. The paper notes that the different unmasked tokens are predicted in a conditionally independent way, and focuses previous attempts that instead interpret the MLM as a Markov random field to overcome the conditional independence limitations. 

First the paper shows the limitations of MRF methods by showing that pairwise MRFs (when two tokens are masked) can be worse than the naive conditionally independent model. Second, they propose an iterative optimization problem, with amortization for computational efficiency, that improves MLM for scoring pairwise tokens.
strength_and_weaknesses: Strengths
- the paper identifies that previous attempts at interpreting MLMs as MRFs may not be faithful, in the sense that unary conditionals may not match.
- the paper improves the 'faithfulness' of the joint distribution via a learned interaction layer which allows modeling the joint with just one evaluation pass (by using a mixing function and the independent unary conditionals).

Weaknesses
- the contribution only focus on pairwise MRFs, which seems incremental. Why not generalize to cliques of size k for some small k, or arbitrary MRFs with low width?


Question
- I had a question throughout the paper, not about this particular paper but about the whole general direction of interpreting MLM as MRFs. It seems that we are jumping through a lot of hoops to define a MRF, find joint distributions with faithful conditionals, run expensive inference, etc.... Can't we simply use the MLM unary conditionals directly in an autoregressive way and avoid all of this trouble? E.g. see https://arxiv.org/abs/2110.02037 https://arxiv.org/abs/2205.13554
clarity,_quality,_novelty_and_reproducibility: The writing is clear and well structured. The studied problem is important and relevant to the conference.

summary_of_the_review: The problem of generation with MLMs is important and has been studied before. This paper shows improvements with pairwise MRFs and has good writing and execution. The scope of the improvement however, may not be that large.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper discusses limitations of MLMs, interpreted as MRFs, and proposes better alternatives. It received a lukewarm, although not wholly negative, reaction from reviewers. As it was borderline, we discussed it async, the results of said discussion being summarized below. The consensus shifted towards agreeing on rejection, with the point being made that despite the clarity of the writing, the justification for the premise of the paper was not strong enough. To quote one of the reviewers during discussion: "Autoregressive unrolling methods of masked language models already mitigates the conditionally independent assumption. Moreover, the computational limitation of the methodology to pairwise MRFs makes the scope rather narrow." From reading through the comments and discussion, and the rebuttals, I agree with the consensus that was reached. At very least, a better framing of the necessity for this work is desirable, and I hope that by incorporating the feedback offered by reviewers, the authors will have a better result upon resubmission to another conference.
justification_for_why_not_higher_score: A clear reviewer consensus emerged during discussion.
Decision: Reject