Towards Conditionally Dependent Masked Language Models
Authors: Lucas Torroba Hennigen
Yoon Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Markov random fields
masked language models
compatibility

Abstract: Masked language modeling has proven to be an effective paradigm for learning representations of language. However, when multiple tokens are masked out, the masked language model's (MLM) distribution over the masked positions assumes that the masked tokens are conditionally independent given the unmasked tokens---an assumption that does not hold in practice. Existing work addresses this limitation by interpreting the sum of unary scores (i.e., the logits or the log probabilities of  single tokens when conditioned on all others) as the log potential a Markov random field (MRF). While this new model no longer makes any independence assumptions, it remains unclear whether this approach (i) results in a good probabilistic model of language and further (ii) derives a model that is faithful (i.e., has matching unary distributions) to the original model. This paper studies MRFs derived this way in a controlled setting where only two tokens are masked out at a time, which makes it possible to compute exact distributional properties. We find that such pairwise MRFs are often worse probabilistic models of language from a perplexity standpoint, and moreover have unary distributions that do not match the unary distributions of the original MLM. We then study a statistically-motivated iterative optimization algorithm for deriving joint pairwise distributions that are more compatible with the original unary distributions. While this iterative approach outperforms the MRF approach, the algorithm itself is too expensive to be practical. We thus amortize this optimization process through a parameterized feed-forward layer that learns to modify the original MLM's pairwise distributions to be both non-independent and faithful, and find that this approach outperforms the MLM for scoring pairwise tokens.

Under review as a conference paper at ICLR 2023
TOWARDS CONDITIONALLY DEPENDENT MASKED LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Masked language modeling has proven to be effective paradigm for learning representations of language. However, when multiple tokens are masked out, the masked language model's (MLM) distribution over the masked positions as- sumes that the masked tokens are conditionally independent given the unmasked tokens-ae an assumption that does not hold in practice. Existing work addresses this limitation by interpreting the sum of unary scores (i.e., the logits or the log probabilities of single tokens when conditioned on all others) as the log potential a Markov random field (MRF). While this new model no longer makes any indepen- dence assumptions, it remains unclear whether this approach (i) results in good probabilistic model of language and further (ii) derives a model that is faithful (i.e., has matching unary distributions) to the original model. This paper studies MRFs derived this way in a controlled setting where only two tokens are masked out at a time, which makes it possible to compute exact distributional properties. We find that such pairwise MRFs are often worse probabilistic models of language from a perplexity standpoint, and moreover have unary distributions that do not match the unary distributions of the original MLM. We then study statistically- motivated iterative optimization algorithm for deriving joint pairwise distributions that
compatible with the original unary distributions. While this iterative approach outperforms the MRF approach, the algorithm itself is too expensive to be practical. We thus amortize this optimization process through a parameterized feed-forward layer that learns to modify the original MLM's pairwise distributions be both mon-independent and faithful, and find that this approach outperforms the MLM for scoring pairwise tokens.
INTRODUCTION
Masked language modeling has proven to be an effective paradigm for learning generalizable rep- resentations of language (Devlin et al. 2219 Liu et al. 2019; He et al., 2021) and other structured domains (Rives et al. 2021, Mahmood et al. 2021; He et al, 2022). From a probabilistic per- spective, masked language models (MLM) make strong independence assumptions. When multiple tokens are masked out, MLMs assume that the distributions over the masked tokens are condition- ally independent given the unmasked tokens assumption that clearly does hold for language. For example, consider the sentence: "The [MASK] [MASK] 2 pleasantly surprised by an analysis paper." MLM's assume that the distribution over the two tokens are independent and thus cannot systematically assign higher probability to grammatical subject-verb agreements ("reviewer was" and "reviewers were") than ungrammatical ones (""reviewer were" and ""reviewers was"). These types of statistical dependencies can occur for words that are far apart.
The [MASK] is tired from reading so many papers that focused on performance gains, [MASK] 2 pleasantly surprised by an analysis paper.
Indeed, such long-range dependencies animate much work on hierarchical approaches to language which posit (usually tree-like) structures in which words that are "close" in structure space (but potentially far apart in surface form) have high dependency with one another.
From purely representation learning perspective, such model misspecifications arising from incor- rect statistical assumptions may not be catastrophic. These assumptions can enable scalable training and even aid in learning better representations by serving as statistical bottleneck that forces more

---

Under review as a conference paper at ICLR 2023
information to be captured by the hidden states However, we observe that MLMs are increasingly being employed as probabilistic models of language, for example for scoring (Salazar et al. 2020
Xu et al. 2022) and sampling/decoding (Wang & Cho 2019; Ghazvininejad
al. 2019 Ng al. 2020 Yamakoshi et al. 2022) sentences. Under such probabilistic uses of MLMs, it becomes critical ensure that the underlying statistical assumptions are plausibly grounded in reality.
Existing work has approached this problem by using the conditionals of an MLM to define an alter- native probabilistic model of language that does not make said conditional independence assump- tions. Noting that the unary conditional distributions of an MLM (i.e., the conditional distributions output by the MLM when a single token is masked out) do not make any independence assumptions, Goyal et al. (2022) define a fully connected Markov random field (MRF) language model whose log potential of a sentence is defined to the sum of these unary log probabilities (or logits). This ap- proach, while sensible, raises two questions: (i) is this new model a good probabilistic model of language, and (ii) are the conditionals of the derived model faithful to the original MI.,, i.e., are the unary conditionals of the new model the same as (or similar to) the unary conditionals of the MLM The latter faithfulness question is important because due to the scale at which these models trained, it is not completely outrageous posit that the unary conditionals learned by the MLM are close enough the true unary distributions of language
This paper investigates both questions in controlled pairwise conditional setting where only two tokens are masked out at a time, which makes it possible to compute the MRF's pairwise distribution exactly. Surprisingly, we find that such pairwise MRFs are often worse probabilistic model of language than even the original MLM that assumes independence between the two masked tokens. We moreover find that the MRF's unary distributions do not match the MLM's unary distributions. In light of this result, we study two alternative approaches to deriving non-independent pairwise distributions from the MLM's unary distributions. The first approach exploits the Hammersley- Clifford sufford--Beag theorem (Besag) 1974), which allows one to write down a joint distribution in terms of unary conditionals. The second approach uses an iterative algorithm that finds a joint distribution over two masked positions whose unary conditionals are closest, in the KI. sense, to the unary conditionals of the MLM (Arnold & Gokhale 1998). We find that joint pairwise distributions from the iterative approach have better perplexity than both the MRF and the MLM, and also have unary conditionals that are closer to those of the original MLM's. While effective, the iterative algorithm is too expensive to be practical. We thus propose amortized variant of the iterative approach that can compute non-independent pairwise conditionals using only a single forward pass of the MLM followed by an efficient feed-forward layer, and find that this amortized approach outperforms original MLM when scoring adjacent pairwise tokens. Our code will be made publicly available.
2 BACKGROUND
We begin by introducing notation. Let y be vocabulary of tokens, and T be the text length, and w TT be an input sentence/paragraph. We are particularly interested in the case when a subset S C [T] A {I....,T) of the input w is replaced with the special [MASK] tokens; this case we will use the notation 4- | wg) to denote the output distribution of the MLM at position i E S, where Wg is derived from w by masking out w, for all S. MLMs are trained to maximize the log-likelihood of a set of masked words S in a sentence, More formally, consider an MLM parameterized by vector 0 E θ and some distribution ua-)) over subsets of positions to mask S C [T]. The MLM learning objective can then be written as:
arg max E W-P(-) Smi()))
Σ YES
log 4oss(w, (we W 0)
S
where p(-) denotes the true data distribution. Let (- the data distribution and further let (s)
wg) analogously be the conditionals of
ws) Iles (is(w; wg)) be the joint distribution
Indeed, prior work has found that masking out contiguous words (which on average have higher depen- dency than non-contigumn words;Joshi et al. (020) or employing more aggressive masking rates (Wettig et al.
2022 can improve representation lcarning.
Off course, it possible that the set of unary conditional distributions themselves may he incompatible
Aroold & Press] 1989), i.c., there is no joint distribution whose unary conditionals exactly equal those of the
MLM's. In our empirical study we show that this is indeed the case. w/w.:///sisseeeyetcctreeeee-eeee As noted by https://accinntthougsssss www.pppew..oo//0190000000
2

---

Under review as a conference paper at ICLR 2023
over the masked tokens ws. Then the above be rewritten
arg min E Smu(+) [[S]
DKL DKL (Ps Wg)) |5/3 Wgg 01)
Thus, we can interpret MLMs learning a family of conditionals distributions {4513) Igg(r) that as- sume that the masked words are conditionally independent given the unmasked words.
3 CONDITIONALLY DEPENDENT MASKED LANGUAGE MODELS
While the conditional independence assumption that underpins MLMs enables scalable training while conditioning on bidirectional context invalid for language, i.c., the MLM misspecified. Note that all conditional distributions where we mask two or more words, i.c., when [S] 2 2, are affected by this, For this reason, we will be particularly interested in the unary conditionals of the MLM, which arise when [S]
2 as noted in the introduction this assumption is clearly
we let S = {1} for some I. [T], we will slightly abuse notation and refer to dojz _ as the unary conditional of the MLM at position t. Since (i) unary conditionals do not make any independence assumptions and (ii) the scale at which modern MLMs are trained is such that might reasonably expect DKI. (Pae |I 4qp)) to be small, they become natural objects of study when trying to build conditionally dependent models from MLMs.
3.1 MARKOV RANDOM FIELDS DERIVED FROM MLMs
To address the conditional independence limitation of MLMs, recent work has proposed using the unary conditionals of the MLM define new probabilistic model over larger units of language
such as phrases and sentences (Wang & Cho 2019 Goyal et al. 2022). The idea is to define:
MRF (w) x II 917((0r Wz), q
DE[T]
which can be interpreted as fully connected MRF, whose log potential is given by the sum of the unary log probabilities. One can similarly define variant of this where the log potentials are logits of the unary conditionals of the MIM, i.e., y MMRFL is the logit of w; conditioned on w. These two models correspond to MRF with a single fully connected clique, and thus they do not make any conditional independence assumptions.
I $tp[ws wy) where $tfE(1x wx)
This construction addresses the conditional independence limitation of the original MLMs, but re- sults in a different probabilistic model, which raises two questions. First, it is not immediately clear if this probabilistic model is actually good probabilistic model of language, since while we can approximately sample from this MRF with (for example) MCMC (Goyal et al. 2022). computing perplexity is completely intractable E Second, despite being defined in terms of the unary condi- tionals, it is not clear to what extent this model is faithful to the original MLM, i.e., it is not clear whether the unary conditionals of the MRF are the same as (or close to) the unary conditionals of the parent MLM. This is again important because due to the scale at which these models are trained, the conditionals learned by the MLM may be close enough the true unary distributions of language. Thus, an ideal MRF should have unary distributions that are faithful to the MLM's.
Theoretically, not hard to show that an MRF defined this way can have nonzero KL to the original unary distribution even in the simple two variable case where we assume that the MRF is constructed from true unary distributions, as given by the following preposition.
w2),P2211 (- wi) be the true (i.e., popula- tion) unary conditional distributions. Define an MRF as
Proposition 3.1. Let w1, W2 E V and further let p112(- (
g1,2(W1, W2) x P1jz(w1 w2) Pap ( w1).
and let 9- W2), 4 w1) be the conditionals derived from the MRF. Then there exists P1j2- P211 such that
(pI(1 w2) [1)e( w2)) > 0.
"Though see Yang al. 2019) for an alternttiee approach. 'The partition function of the MRF would require [T) forward passes through the MLM. We also tried estimating the partition through importance sampling with GPT-2 (Radford et al. 2019), but found the estimate be quite poor.
3

---

Under review as a conference paper at ICLR 2023
See App. A for proof. This indicates that even in a simple setting with two variables, MRF-derived distributions may not be faithful. This motivates our pairwise setting below.
3.1.1 PAIRWISE MRFs
Since the partition function of the MRF intractable, we cannot compute the unary conditionals of the MRF We can, however, analyze the where use the same approach to define pairwise MRF over a pair of positions. Given two positions S = fa, b} C [T], and a context Wg (i.c., a setting of all the other positions), define pairwise MRF over those two positions via:
MRF aa,b/3 (Wa Whb Wg) X 4aj5
Wg) (ops(wb wgg)
The definition of 2 wg), which uses the unary conditional logits, is analogous. Doing so requires 2 forward passes through the MLM for each conditional, which expensive but tractable on modern GPUs with large batches. Given this pairwise MRF, we can now compute the perplexities and faithfulness metrics on a dataset ((wb (m) wi W
(n)
) ) of English sentences where the
tokens at positions a and b (i.e., tokens Wa and w¹) have been masked.
Language model performance. evaluate the pairwise MRF as a language model, we compute two measures: unary perplexity and pairwise perplexity. The unary perplexity (U-PPL) over single tokens is given by,
N Σ n=1
(w(")) log 4b,, (
(n) (n) ((()) + log (w) wof") w!"") (m))
(n))
(n),
exp
2.V
An ideal model would obtain unary perplexity that is similar to the MLM's (which uses q instead of MRF in the above expression). The pairwise perplexity (P-PPL) over two tokens is given by,
1
MRE log 4n
(n), (w(")
(n)
(n)
2.
#=1
We would expect good model of language to obtain lower pairwise perplexity than the original MLM which (wrongly) assumes conditional independence.
Faithfulness. We also the derived unary conditionals assess faithfulness to the MLM's unary conditionals by calculating the average conditional KL divergence (A-KL) between the unary conditionals,
ΣΣ R=1 WEV
DKL ws. w (10) l|
MRF Capp.s (- WP, w (n)) )).
ND
If the MRF is completely faithful the MIM, this number should be zero. The above metric aver- ages the KI. across the entire vocabulary V, but practice we may only be interested in assessing closeness only when conditioned on the gold tokens. We thus compute variant of the above metric where we only average over the conditionals for the gold token (G-KL):
1
DKL (Ga)t.s(
wy
(m)) w
MRF
w wo"")
w
n=1
This metric penalizes unfaithfulness in common contexts more than in uncommon contexts. Again, this should be zero if the models are faithful.
Experimental setup. We calculate the above metrics 1000 examples from natural language inference dataset (SNLI, Bowman et al. 2015) and a summarization dataset (XSUM, Narayan et al. 2018). Since dependencies other, we consider two schemes for selecting the two tokens to be masked for each sentence: masks over tokens chosen uniformly tokens (chosen uniformly at random) in a sentence (Contiguous pairs). For exact comparison we make sure the masking is the same for all models. In addition, we consider both BERT BASE and BERT (cased) as the ML from which to obtain the unary conditionals!
more likely emerge when the tokens being masked next each
random (Random pairs), and also
random contiguous
WWe the Huggingface Wolf al. 2020 implementations of these models.
4

---

Under review as a conference paper at ICLR 2023
Random pairs
Contiguous pairs
Dataset Scheme
U-PPL
P-PPL
A-KL
G-KL
U-PPL
P-PPL
A-KL
G-KL
MLM
11.22
19.01
1.080
0.547
13.78
74.68
4.014
1.876
MRF.
13.39
71.44
0.433
0.267
23.45
13568. 17
1.543
0.607
SNLI
MRF
12.30
21.65
0.658
0.179
18.35
126.05
1.967
0.366
HСB
12.51
22.62
0.593
0.168
17.71
589.02
2.099
0,416
AG
10.76
12.68
0.007
0.085
13.26
21.59
0.018
0.181
MLM
4.88
6.12
0.404
0.227
4.910
39.33
4.381
2.128
MRFL
5.17
9.12
0.148
0.085
6.55
2209.94
1.561
0,383
XSUM
MRF
5.00
6.23
0.262
0.049
5.53
47.62
2.242
0.185
HCB
5.08
6.21
0.256
0.052
6.46
174.32
2.681
0.328
AG
5.00
5.29
0.003
0.044
5.27
8.42
0.016
0.143
MLM
9.50
18.57
1.374
0.787
10.42
104.12
4.582
2,463
MRF
11.52
76.23
0.449
0.276
15,43
8536.92
1,470
0.543
SNLI
MRF
10.57
19.54
0.723
0.193
13.07
93.33
1.992
0.359
HCB
10.71
20.70
0.797
0.215
14.43
458.25
2.563
0.552
AG
8.57
10.11 1
0.007
0.097
9.64
15.64
0.019
0.173
MLM
3.80
5.67
0,530
0.413
3.91
103.86
5.046
3.276
MRF.
3.94
7.06
0.156
0.068
4.62
1328.20
1.441
0.290
XSUM
MRF
3.87
4.94
0.322
0.036
4.16
36.66
2.258
0.145
HCB
3.91
5.14
0.346
0.059
5.67
164.15
2.954
0,400
AG
3.88
4.13
0.003
0.042
4.21
6.62
0.016
0.126
Table 1: Comparison of MRF, HCB and AG constructions randomly sampled SNLI (Bowman et al. 2015
sentences and XSUM (Narayan et al. 2018) summaries. We apply the constructions to two MLMs: BERT mast (D) and BERT (L) We consider both masking tokens uniformly at random (Random pairs) and masking adjacent tokens uniformly at random (Contiguous pairs). PPL metrics measure the quality of the models and KL metrics measure their faithfulness to the MLM's unary conditionals. For all metrics, lower is better.
Results. The results are shown in Tab. Comparing the MRF and MRF, (i.e., the MRF using logits), the former consistently outperforms the latter, indicating that using the raw logits generally results in worse language model m Comparing the MRFs to MLM, we see that the unary perplexity (U-PPL) of the MLM lower than those of the MRFs, and that the difference is most pronounced in the contiguous masking case. More surprisingly, we see that the pairwise perplexity (P-PPL) is often (much) higher than the MLM's, even though the MLM makes unrealistic conditional indepen- dence assumptions. These results indicate that the derived MRFs are in general worse probabilistic models of language for unary/pairwise tokens (except with an MRF derived from BERTLARGE in the contiguous pair setting). Finally. also find that the MRFs' unary conditionals are not faithful to those of the MRFs based on the KL measures (A-KL, G-KL).
This example from SNLI qualitatively illustrates case where both the unary and pairwise perplex- ities from the MRF underperforms the MLM: "The [MASK]] [MASK] 2 at the casino", where the tokens "man is" are masked. In this case, both MRFs assign virtually zero probability mass to the correct tokens, while the MLM assigns orders of magnitude more (around 0.2% of the mass of the joint). Upon inspection, this arises because 4211.5 (is man) 0.02 and 412,, (man is) 2x 10-5, which makes the numerator of / (man, is) be RJ 0. The MRF could still assign high probability to this pair the denominator also R 0, but in this case we have 42 (was) man) P3 0.33 and 4,,, (man was)= x 0.03, which makes the denominator well above O. This causes the completion "man is" to have disproportionately little mass in the joint compared other to combinations ("man was") that were ascribed more mass by BERT's unary conditionals.
3.2 TOWARDS FAITHFUL PAIRWISE DISTRIBUTIONS
While the MRF probabilistic model surmounts the conditional independence limitation of MLMs, based on the above results, this comes at the expense of a typically worse probabilistic model overall. Since we ultimately wish to address the conditional independence limitation in order to obtain a bet- ter probabilistic model, this raises the question: Are there other ways of using the unary distributions (which perform well) to construct pairwise distribution that does not make conditional indepen-
This gives additional support to the findings of Goyal et al. (202)). who found that MCMC sampling from MRF resulted in sentences with better perplexity than sentences sampled from the MRF, as measured by GPT2.
5

---

Under review as a conference paper at ICLR 2023
dence assumption and leads to a better probabilistic model of language? We study two alternative approaches for constructing joint distributions from the unary conditionals of an MLM.
construction. The Hammersley- Clifford-Besag theorem (HCB; Besag 1974 provides a way of reconstructing joint distribution p(-) from its unary conditionals. states that given pivot point w' = (w] (wi...., w4-)) E V the probability of some w E V under
Hanmerslle--Cifford--Be
the joint distribution p()) is given by:
Pujaue (ue Wyt, Werr) Pap( (2) WsG. Weer)
p(w) & II
1E
where Wet & (W1,... 11-1),, and similarly w _ (we+1..... WT). Importantly, unlike the MRF approach, if the unary conditionals of the MLM compatible (i.e., they are the unary conditionals of some strictly positive joint), then HCB will recover that joint, irrespective of the choice of pivot.
As with the MRF, computing the HCB-implied joint for MLM would require O((T- 1)) forward passes through the MLM. That said, if we restrict our attention to only two position S {a, b} C [T] as before, we can use eq. to construct a distribution over those two positions in [V] + 1 forward passes:
HCB (w (w: whb ws) x (Wo wh: W's) Uajo,((uri wh, wis) Sola. wa wg)
where (w'y, wx)) 22 is some pivot and Wy is context.
Arnold-Gohkale construction. One way to frame the faithfulness objective is to find joint distribution whose unary conditionals have smallest KL to the unary conditionals of the MLM. Since this is intractable for arbitrary joint constructions, we focus the case of pairwise joints, just as in the MRF and HCB cases. For any pair positions S = {a, b} and a context Wg. we directly optimize for joint distribution whose unary conditional distributions are faithful:
q
AG = arg min
Σ WVEV
DKL (1bla,s, | Wa. wg) 1//0,,0,3( Wa, Wg))
+
Σ
DKL (ajo,s)) wh, ll Pap.ss( ll Wo, w))
WVEV
Arnold & Gokhale (AG; 1998) study this minimization problem, and provide an effective iterative algorithm for it. The algorithm initializes the starting pairwise distribution 00.1s(( Wg)) to be uniform, and makes the following updates until convergence,
Jajo,3(Ws (Wa
Wb; Wg) + Iba,,5(1b Was Wz)
AGiel (wa, Wi wg) x ba
AGU 2215
AGO (ws w3)
(wa
Wz))
+
Obtaining the MIL unary conditionals to be able to run this algorithm requires 2 || forward passes through the MI.M, as in the MRF case.
Remark. The HCB construction implicitly assumes compatibility of unary distributions when de- riving the joint distribution. On the other hand, the AG construction assumes that the unary distribu- tions themselves are incompatible with one another (i.e., there no joint distribution whose set of unary distributions is exactly equal to those of the MLM's: Arnold & Press] 1989). and instead finds a joint distribution that is near faithful. Of course, the exact compatibility of learned unary condi- tionals is unlikely to occur in practice. However, of contemporary MLMs to argue that large-scale training results in learned unary conditionals that close enough to the true unary conditionals (which are by definition compatible). Since both HCB and AG should yield the same exact joint if the unary distributions are compatible, these ex- periments can assess the extent which MLMS actually learn compatible unary distributions.
might again appeal to the rich parameterization
Experimental setup. We evaluate the quality of the HCB and AG constructions using the same experimental setup as in the MRF experiments (i.e., same sentences/masks), For the AG joint, we run 50 steps of the iterative process in eq. (3), which was enough for convergence. For the HCB joint, we pick pivot using the mode of the pairwise joint of the MLM. (We did not find HCB to be too sensitive the pivot in preliminary experiments.)

---

Under review as a conference paper at ICLR 2023
Results. The results shown in Tab. 2 HCB obtains roughly comparable performance MRF in the random pair masking case. In the contiguous pair case, exhibits similar failure modes as the MRF in producing extremely high pairwise perplexity (P-PPL) values. The faithfulness metrics similar to the MRF's. The AG approach, on the other hand, outperforms the MRFL, MRF and HCB approaches for all metrics. This is most evident in the contiguous masking case, where AG attains lower pairwise perplexity than all models, including the MLM itself. In some cases, we find that the AG model even outperforms the MLM in terms of unary perplexity, which suggests its unary conditionals are better than the MLM's unary conditionals. This is remarkable since the unary con- ditionals of the MLM were trained to approximate the unary conditionals of language, and they do not make any conditional independence assumptions. This suggests that the procedure to construct AG may have regularizing effect that occasionally leads to a slightly improved probabilistic model of language. The AG's KI. measures is substantially lower than the other models, though this is not surprising since AG is trained to optimize that objective. More interestingly, we see that AG's gold KL (G-KL) tends to be par with the other models, which suggests that it is still not faithful to the MLM in the contexts that are most likely to arise, and moreover confirms that the MLM's unary conditionals themselves are incompatible (Arnold & Press 1989).
LEARNING TO BE FAITHFUL VIA INTERACTION LAYERS
The effectiveness of the AG approach suggests that modeling the dependencies between masked tokens in a sentence should yield an improved probabilistic model of language. However, the above approaches are expensive to run, requiring ((V))) forward passes of the MLM. The MLM, on the other hand, only requires a single forward pass but makes a conditional independence assumption. Can we derive an efficient approach that models pairwise dependencies but only requires a single MLM pass? In this section we study an approach for learning an efficient feed-forward neural network f. which we call an interaction layer, that learns to adjust the output of the MLM such that the derived model is faithful to the MLM's unary conditionals.
Concretely, suppose we want to build a pairwise (near) faithful model over two positions S {a. 6} given a fixed context Wg. We define a new model that does not make any independence assumptions between the two masked tokens:
2 (Wa, w/| W 0) x (ajs(Wa (gs) (oss(Wn wg)) fo [wa Wb; a, b, ws).
This approach allows the new model to exploit the rich linguistic information encoded by the pairwise conditionals of the MLM. without inheriting their conditional independence assumption. Specifically, our function f computes the adjustment factor as:
So (wa, wob o, b. wg)) - exp (s x (go(a, Wb, h, w "embbwwa) + (b, Wa a, w (mb()))))
(a, Wa, b, wg) Zo (Urepr(( ws) + Vemb + Wpos(b) + +y) +
where a is a PRRLU (He et al. 2015) activation function, emb(w) (w) returns the MLM's (word) embedding of a token W, pos(x) returns the positional embedding of the MLM for position x, repr(æ; Wz) is the MLMs contextualized (i.e, final-layer) representation at position I we input the sequence Wy. and 0 (U, V. W, Z. y, 2, s} are learnable parameters. The intuition behind this design is that the function g(-) takes as input the MLM's ' contextualized representation at the current position (e.g., a) and adjusts it according to what would happen if we knew what the token at the other position (e.g.. b) would be; using these adjusted representations, we then compute an exponen- tiated dot product with the word embeddings, much like how the final-layer softmax of the MLM is an exponentiated dot product between contextualized representations and the word embeddings.
While the above requires only a single pass of the MLM, ((I)22) passes of f are needed to evaluate every element of the joint. However, by the Zipfian nature of language, know this joint should be sparse. For this reason, we propose a training scheme wherein we use f to obtain a restricted joint distribution over K2 elements, treating all elements that fall outside that set having zero probability. In practice, we construct this set of K2 elements by doing a forward pass through the original MLM, obtaining the top-K most likely tokens at each of the two positions, and then taking their Cartesian product.
To train the interaction layer, we minimize the KL divergence between the pairwise conditional implied by AG and the pairwise conditional implied by the interaction layer. Formally, given a (n) dataset of masked sentences D= {(w(")) ug w( w we optimize the following objective
7

---

Under review as a conference paper at ICLR 2023
(the MLM's parameters remain fixed):
arg min
Σ
DEL
DEL DKL Cabbss(s w(") ll 1 (,M5s (:, W
(x))
0)).
0)).
0
Since we do not normalize over the full vocabulary, we can no longer obtain perplexities. Thus for these experiments we instead resort standard ranking-basee measures. Recall at k (R@k) computes how often our interaction layer placed original tokens in the sentence (i.e., the ones that were masked) among the top l elements of the joint, viz.,
Σ (rank(ur))) wh " n=1
Ma,M5( w (n)) 2 k3.
We compute this for k E (1,5, 10), where rank y: returns the rank of (xx,) in the joint μ.. We also compute mean reciprocal rank (MRR):
(ran((w(") (1) [a w((
N #
(*)) are not in the restricted joint (i.e., they were assigned zero probability mass), then we take its reciprocal rank to be zero.
Note that if the correct tokens (wa"), (m))
Experimental setup. We evaluate on the same datasets as before. For our training dataset, we examples from Wikipedia. We train the interaction layers for 50K steps using the Adam op- timizer (Kingma & Ba 2015). with a learning rate of 0.001 and a batch size of 64. We initialize that, initially, the joint is only adjusted in a minor way. In prelim-
the learnable scalar Rs inary experiments, we found that using the pairwise (unnormalized) logits of the MLM instead of the (normalized) pairwise conditionals worked better. We also lightly explored other variants of the architecture described above (e.g., different ways of combining the embeddings), but found that the design given above worked well enough while being efficient. Finally, since previous experiments indicated that the potential for performance gains over the pairwise conditionals of MLM mod- est when masking tokens uniformly at random, we opted to focus solely on the case of masking adjacent tokens (i.c., the contiguous pairs setting).
Results. The results shown in Tab. suggest that the interaction layer (INT) tends to outperform the MLM, but underperform other constructions. That said, considering that the joint induced by the interaction layer is much cheaper to compute (i.e., O(1) MLM forward passes instead of (K²) passes), this positions the interaction layer as viable alternative to using the MLM's pairwise conditionals. Figure shows
shows that optimizing towards the AG joint (top) improves MRR (bottom).
DISCUSSION
Our study illuminates the deficiencies of the MRF approach and applics statistically-motivee ap- proaches to craft more performant probabilistic models. However, it is admittedly not clear how these insights can immediately be applied to improve downstream NLP tasks. We focused on mod- els over pairwise tokens in order to avoid sampling and work with exact distributions for the various approaches (MRF, HCB, AG). However this limits the generality of our approach (e.g., we cannot score full sentences). We nonetheless believe that our empirical study is interesting on its own and further suggests new paths for developing efficient, conditionally dependent, and faithful MLMs. For example. one could also use the interaction layer train against the target gold tokens in- stead of just conditioning on gold tokens to obtain the unary conditionals with, for example, noise contrastive estimation (Gutmann & Hyvärinen) 2010). Such MLMs trained at scale could provide an alternative to autoregressive language models and provide new functionalities (e.g., controllable editing of sentences).
RELATED WORK
Probabilistic interpretations of MLMss In one of the earliest works about sampling from MLMs, Wang & Cho 2019) propose to unary conditionals sample sentences. Recently Yamakoshi ct al. (2022) highlight that, while Wang & Chols (209)) approach only constitutes a Sacudo-Gibbe sampler, the act of re-sampling positions uniformly random guarantees that the resulting Markov
8

---

Under review as a conference paper at ICLR 2023
Contiguous pairs
Dataset
Scheme
R@I
R@S
R@ 10
MRR
MLM
0.117
0.260
0.305
0.184
1.8
MRF.
0.175
0.379
0.458
0.271
Validation loss
MRF
0.140
0.333
0,424
0.235
SNLI
HCB
0.167
0.364
0,451
0.264
AG
0,219
0.416
0.473
0.311
INT
0.145
0.299
0,363
0.216
1.2
MLM
0.183
0.342
0.401
0.259
MRFL
0.339
0.579
0.660
0.446
MRF
0.304
0.523
0.615
0.407
XSUM
HCB
0.317
0.550
0.625
0.425
AG
0.379
0.593
0.670
0.479
INT
0.233
0.400
0.465
0.316
MLM
0.110
0.235
0.288
0.172
0.32
Validation MRR
MRF.
0,231
0.419
0.518
0.324
MRF
0.170
0.374
0.474
0.270
SNLI
HCB
0.204
0.396
0,484
0.298
0.3
AG
0.234
0.442
0.530
0.334
INT
0.127
0.265
0.328
0.196
MLM
0,151
0.288
0.343
0.215
0.28
MRFL
0.399
0.639
0.721
0.508
MRF
0.352
0.592
0.689
0.462
XSUM
HCB
0.360
0.568
0.659
0.459
AG
0.414
0.647
0.723
0.518
10k
20k
30k
40k
50k
INT
0.162
0.324
0.402
0.242
Step
Table 2: Comparison of interaction layer (INT), when
Figure 1: Validation loss for optimizing towards AG joint (top) and mean reciprocal rank (bottom) for BERT interaction layer during training.
applied to both BERT (B) and BERT LARGE (L), to previous joint constructions.
chain has a unique, stationary distribution (Bengio al. 2013 2014). In contrast to sampling directly from BERT, Goyal et al. (2022) propose deriving an MRF from the unary conditionals learned by BERT, and sample from this via Metropolis- Mettopolis-Hastinnss. Ghazvininejad et al. (2019) and Savinov et al. 2022) use conditional ML.s for fast decoding of translation models. Our interaction layer is related to residual energy networks (Deng al. 2020), which also learn modify the output from an existing model.
Compatible distributions. The statistics community has long studied the problem of assessing the compatibility of set of conditionals (Amold & Press 1989 Gelman & Speed 1993 Wang & Kuo 2010; Song et al. 2010). AAmold & Gokhale (998) and Amold et al. (2002) explore algorithms for reconstructing near-compatible joints from incompatible conditionals, which leverage in our work. [esag (1974) also explores this problem, and defines a procedure (viz., eq. (2)) for doing so when the joint distribution strictly positive and the conditionals are compatible. Lowd (2012)
apply a version of HCB to derive Markov networks from incompatible dependency networks (Heck- erman al. 2000). Pseudo-gibbs sampling, which refers to sampling from incompatible (or near compatible) conditionals, has also been studied from both empirical and theoretical perspectives (Chen 2011 Chen & 1p 2015 Kuo & Wang 2019)
7 CONCLUSION
In this paper we studied pairwise MRFs derived from unary conditionals of MLMs and empirically observed them not only be worse language models but also have unary distribution that do not match the original MLM's. We then studied two statistically motivated approaches for deriving more faithful MRFs. and found that the iterative optimization approach, which identifies joint that is more faithful to the unary conditionals of the original MLM, performs well. Finally, we experimented with amortizing the optimization algorithm via learned conditionally dependent language model.
layer to derive

---

Under review as a conference paper at ICLR 2023
REFERENCES
Barry C. Arnold and DV Gokhale. Distributions most nearly compatible with given families of conditional distributions. Test, 7(2):377-390, 1998.
Barry C. Arold and S. James Press. Compatible conditional distributions. Journal of the American Statistical Association, 84(405):152-156, 1989. ISSN 0162-1459, doi: 10,2307/2289858. URL https//www..oooooo/ssbee/225555 Publisher: [American Statistical Association, Taylor & Francis, Ltd.].
Barry C. Amold, Enrique Castillo, and José María Sarabia. Exact and near compatibility of discrete conditional distributions. Computational Statistics & Data Analysis, 40(2):231-252, 2002.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In Proceedings the 26th International Conference Neural Informa- tion Processing Systems Volume 1, NIPS' 13, pp. 899-907, Red Hook, NY, USA, 2013. Curran Associates Inc.
Yoshua Bengio, É ric Thibodeau-Laufer, Guillaume Alain, and Jason Yosinski. Deep generative stochastic networks trainable by backprop. In Proceedings of the 31st International Conference International Conference Machine Learning Volume 32, ICML' 14, pp. I1- 11-226-11-234 JMLR.org, 2014.
Julian Besag. Spatial interaction and the statistical analysis of lattice systems. Journal of the Royal Statistical Society. Series B (Methodological), 36(2):192-2) 226,, 1974. ISSN 0035-9246. URL https//wwww..strrogggaabe//////
Publisher: [Royal Statistical Society, Wiley].
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2015.
Shyh-Huei Chen and Edward H Ip. Behavior of the gibbs sampler when conditional distributions are potentially incompatible. Stat Comput Simul., 85(16), 2015.
Shyh-Huei Chen, Edward H. Ip, and Yuchung J. Wang. Gibbs ensembles for nearly compatible and incompatible conditional models. Comput. Stat. Data Anal., 55(4), 2011.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. In Proceedings of ICLR, 2020.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/v//11-11 1423. URL hhtps:// aclanthology.org/N119. 1423
Andrew Gelman and T. Speed. Characterizing joint probability distribution by conditionals.
Journal of the Royal Statistical Society: Series B (Methodological 55(1), 1993.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-Predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP),, pp. 6112-6121, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DI1 1633. URL hutps://ccaathoolg..o///194
Kartik Goyal, Chris Dyer, and Taylor Berg- Kirkpatrick. Exposing the implicit energy networks behind masked language models via Metsopolis-Haatings.. In Proceedings of ICLR, 2022. arXiv: 21 2106.02736.
10

---

Under review as a conference paper at ICLR 2023
M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnor- malized statistical models. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS), volume 9 of JMLR W&CP, pp. 297-304, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10.1 10///CCV.0015.233..
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of CVPR, 2022.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhance BERT with Disentangled Attention. In Proceedings of ICLR, 2021.
David Heckerman, Max Chickering, Chris Meek, Robert Rounthwaite, and Carl Kadie. Dependency networks for inference, collaborative filtering, and data visualization. Technical Report MSR- TR-2000-16, 2000. Proceedings of the International Conference Dependable Systems and Networks.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the Associ- ation for Computational Linguisties, 8, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL http:p//aaari...oo////////11
Kin-Lin Kuo and Yuchung J. Wang. Pseudo-Gibbs sampler for discrete conditional distri- butions. Annals of the Institute of Statistical Mathematics, 71(1):93-105, February 2019. doi: (007=
URL http:///waa...pp....gg////////ap
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. 2019.
Daniel Lowd. Closed-Form Learning of Markov Networks from Dependency Networks. In Pro- ceedings of UAI, 2012.
Omar Mahmood, Elman Mansimov, Richard Bonneau, and Kyunghyun Cho. Masked graph model- ing for molecule generation. Nature Communications, 12(1), 2021.
Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D133 1206. URL https://cclannholoyyo/////
Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In Proceedings of EMNLP. Association for Computational Linguistics, 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15). 2021. ISSN 0027-8424. doi: 20.1073/pnas.20162555 18...
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics. pp. 2699-277211 Online, July 2020. Association for Computational Linguistics. doi: 10118555/112220.aclla 240. URL JtR//h/pp//aaaabbbog..///0/0...p

---

Under review as a conference paper at ICLR 2023
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron den Oord. Step-unrolled Denoising Autoencoders for Text Generation. In Proceedings of ICLR, 2022.
Chwan-Chin Song, Lung-An Li, Chong-Hong Chen, Thomas J. Jiang, and Kin-Lin Kuo. Compati- bility of finite discrete conditional distributions. Stafisfica Sinica, (20):423-440, 2010.
Alex Wang and Kyunghyun Cho. BERT has mouth, and must speak: BERT as Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 30-36, Minneapolis, Minnesota, June 2019, Asso- ciation for Computational Linguistics. doi: 10. N0.1865311/W1992300 URI https//s////alln oorg/W 19-2304
Yuchung Wang and Kun-Lin Kuo. Compatibility of discrete conditional distributions with struc- tural zeros. J. Multivar. Anal., 101(1):191-199, 2010.
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-at natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45 45.4,, Online, October 2020. Association for Computational Linguistics. doi: orgg2220.emmll-demmos
URL hutps://aclanthology
Liyan Xu, Yile Gu, Jari Kolchmainen, Haidar Khan, Ankur Gandhe, Ariya Rastrow, Andreas Stol- cke, and Ivan Bulyko. RescoreBERT: Discriminative Speech Recognition Rescoring With Bert. In Proceedings of ICASSP, 2022.
Takateru Yamakoshi, Thomas Griffiths, and Robert Hawkins. Probing BERT's priors with serial reproduction chains. In Findings of the Association for Computational Linguistics: ACL 2022.
Association for Computational Linguistics, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems, 2019.
A UNFAITHFUL MRFs
Here we show that even if the unary conditionals used in the MRF construction are compati- ble (Arnold & Press, 1989), the unary conditionals of the probabilistic model implied by the MRF construction deviate (in the KL sense) from the true conditionals. This is important because (i) it suggests that we might do better (at least in terms unary PPL) by simply sticking to the conditionals learned by MLM, and (ii) this not the case for either the HCB the AG models, i.e., if we started with the correct conditionals, HCB and AG's joint would be compatible with the ML..
To this, suppose we defined our MRF using the unary conditionals of the true data distribution, Further, for simplicity, suppose T = 2. Hence, our MRF has the form:
(,,2 w22 x Pij[(w] w2) P211 (wz w1)
Then for some Wy V. we have:
Pila(1) W2) P211 (W2 wi)
4m w2) = E Swe Priz(w' W2) Pat1 (wz w()
12

---

Under review as a conference paper at ICLR 2023
Now, consider the KL between the true unary conditionals and the MRF unary conditionals:
P1)2(W w2)
(R1(1 w2) (1/2 (2)) = Σ
P1|2(W w2) log
0112 (w (02)
WEV
(2)) P2]1 ( w')
Σ P1ja(w) W2) log WEV
P2l1 (w2 w)
log Pa mmg) |22 (w2
w)] Ew (2)))))))))))))) log P2j1 (w2 | w)]
This term is the Jensen gap, and in general it can be non-zero. To see this, suppose V = {aa b] and consider the joint
97
P1,2 W2) =
100
W1, W2 = a
otherwise
100
with corresponding conditionals P2/1 (x: b) P1|2 (ar b)
2 for all
V and
97
(2) a) = (1j2(2 aa) =
98
x = a
1
= b
Now, take W2 = b. We then have
(P1 b) q:)2(- b))
log Ew 2(1b)
[P2n (b
w)] (ww-ppal-1b [lo P211 (b w)]
1
1
log
log
+ log
2
98
2
2
98
1
log
+
log
= 1.27
196
2
196
which demonstrates that the KL can be non-zero.
13


Reviews:
[{"review_id": "89E0ngo98J", "reviewer": "Reviewer_cQUX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "See summary of the review below.", "summary_of_the_review": "I think this is an interesting paper, but I found the motivation for the approach lacking. \n\nThe paper describes various issues in going from MLM parameter estimation objectives, to density estimation over sequences. The paper first describes a number of naive methods (section 3.1.1) that are clearly heuristic, lacking clear guarantees, and which not surprisingly lead to bad performance. The paper then describes prior results from pseudo-likelihood that do lead to well-founded estimators, and give better performance. \n\nThis doesn’t feel like a significant enough contribution for ICLR, for two reasons. 1) it’s not surprising at all that using well-founded methods, known a long way back in the pseudo-likelihood literature, give improvements. 2) density estimation of this type is of limited interest, as auto-regressive models do not have these issues; compelling arguments need to be made for why the MRF-based approach is preferable, and how the method might be scaled from pair-wise interactions.\n\n\nIn addition, I think the motivation/discussion in the introduction needs to be clarified, specifically:\n\n“From a probabilistic perspective, masked language models (MLM) make strong independence assumptions”. There is *a lot* riding on the phrase “from a probabilistic perspective” here. I think what the authors are getting at is going from the MLM training criterion to a full density p(x) over sentences x - and how this may be challenging given the way current MLM objectives are set up. That’s ok, but two comments: 1) the original MLM objective was never motivated for density estimation, i.e., estimation of p(x), it was used for pretraining/representation learning. It was always clear that there might be some connection through pseudo-likelhood to density estimation, but the method never depended on that link being made. The MLM method is also completely sound in estimating a particular conditional distribution that arises from masking the input sentence. 2) the authors themselves do not give a particularly compelling method to density estimation themselves, which leaves the motivation on somewhat shaky ground. \n\nThe introduction continues in this vain: “such model misspecifications arising from incorrect statistical assumptions may not be catastrophic”. Again, the original MLM objective did not try to derive a model for p(x), so the term “model misspecification” is in my opinion highly misleading here. \n\n“However, we observe that MLMs are increasingly being employed as probabilistic models of language” - presumably by a “probabilistic model of language” you mean a density p(x) over sentences? The authors need to clarify these terms.\n\nI’m concerned that a naive reader will get the impression that MLM methods are fundamentally unsound, and that is just not a correct characterization in my opinion.\n", "strength_and_weaknesses": "See summary of the review below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7lmQTdbzzoz", "reviewer": "Reviewer_zRNa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies how a masked language model (MLM) can be used as a traditional language model. The authors consider the case of two masked tokens and they proposed pairwise Markov random fields (MRFs). They experimentally show that pairwise MRFs are worse probabilistic models of language from a perplexity standpoint. They studied two formal approaches for deriving better language models and one algorithmic solution using a feed-forward neural layer.", "summary_of_the_review": "The paper contains a statiscally funded study completed by interesting experiments. The motivations should be made explicit. For now, I am not convinced by the usefulness of the study at least from an NLP perspective.", "strength_and_weaknesses": "Pros\n\n* Showing that distributions defined with two masked tokens do not define unary distributions close to that of the original MLM\n* Proposal of statistically founded methods to answer this problem\n* Proposal of an algorithmic solution to give a proxy\n\nCons\n\n* The motivations are not clear at least to me", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "01poQz9pXG2", "reviewer": "Reviewer_fxig", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper examines the perspective of interpreting masked language models (MLMs) as a generative model, where the unmasking procedure corresponds to predicting unary conditionals. The paper notes that the different unmasked tokens are predicted in a conditionally independent way, and focuses previous attempts that instead interpret the MLM as a Markov random field to overcome the conditional independence limitations. \n\nFirst the paper shows the limitations of MRF methods by showing that pairwise MRFs (when two tokens are masked) can be worse than the naive conditionally independent model. Second, they propose an iterative optimization problem, with amortization for computational efficiency, that improves MLM for scoring pairwise tokens.", "summary_of_the_review": "The problem of generation with MLMs is important and has been studied before. This paper shows improvements with pairwise MRFs and has good writing and execution. The scope of the improvement however, may not be that large.", "strength_and_weaknesses": "Strengths\n- the paper identifies that previous attempts at interpreting MLMs as MRFs may not be faithful, in the sense that unary conditionals may not match.\n- the paper improves the 'faithfulness' of the joint distribution via a learned interaction layer which allows modeling the joint with just one evaluation pass (by using a mixing function and the independent unary conditionals).\n\nWeaknesses\n- the contribution only focus on pairwise MRFs, which seems incremental. Why not generalize to cliques of size k for some small k, or arbitrary MRFs with low width?\n\n\nQuestion\n- I had a question throughout the paper, not about this particular paper but about the whole general direction of interpreting MLM as MRFs. It seems that we are jumping through a lot of hoops to define a MRF, find joint distributions with faithful conditionals, run expensive inference, etc.... Can't we simply use the MLM unary conditionals directly in an autoregressive way and avoid all of this trouble? E.g. see https://arxiv.org/abs/2110.02037 https://arxiv.org/abs/2205.13554", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject