Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work makes a step towards sequence generation from MLMs such as BERT. This is an interesting task itself but also managing to do so will enable more applications about sampling from Energy functions of text, think of combining MLMs scores with sequence level scores obtained by the same MLM and using the resultant energy to obtain samples directly. 

(Wang and Cho 19) proposes an energy parametrization similar to E_raw in this paper for a Gibbs sampling approach under a faulty assumption that the token conditionals are independent from each other this has been clarified by Wang and cho'19 later in an independent post.  In this work, the authors extend on this clarification both theoretically and empirically. in section "conditional distribution under E_raw" They show that MLM conditionals do not correspond to the global sequence distribution specified by E_{raw}. And if one wanted to do Gibbs sampling, they drive the correct conditionals under E_{raw}, however, these are expensive to compute as they will need |V| times fwd passes of the MLM to compute the conditional for each position $t$.

An alternative approach proposed by the authors is to use Metropolis Hasting in brief they use two scoring functions,
 
1) $E_{raw}$ raw scoring (sum of MLM logits) the one proposed by (Wang and Cho 2019) 
2) $E_{norm}$ locally normalized scoring (sum of MLM softmax outputs). 

Both of those parametrizations are not globally normalized and are hard to directly sample from. Therefore they propose to use MH to sample from those scorers using a proposal distribution for the transition probability $q(X',X)$. 

One advantage of MH is that q here could be any proposal distribution authors decide to use here to use the Masked Conditionals as a proposal distribution. Additionally, they try variants of this proposal distribution q,  by varying temp, nucleus sampling, and block MH sampling, the latter is motivated by the low mixing rates of the original proposal q despite its high acceptance rate.  

This work conducts several experiments to show the following: 

1- Comparing MH with E_raw and E_norm vs the faulty Gibbs Sampling approach. Degenerations are obvious in the case of the Gibbs sampling which conforms to the theoretical motivations described above. 

2- Temperature / Nucleus sampling: the main findings here: is that the high rejection rates of MH samplers and that gibbs sampler could be useful in practice with low temperatures.

3- Block MH sampling with the proposal q: showing improvements than the previous proposals in terms of novel transition rates and BLEU scores for NMT experiments. 

4- extra experiments with temperature annealing of the energy function comparing with Mask-predict and autoregressive LM

5 - human evaluation for open-ended generation 

main_review: *Pros*
- The paper tackles an interesting problem of sampling from energy functions with local conditionals. 
- The theoretical contributions of the paper are quite sound. The rationale behind (wang and cho'19) faulty assumption and the use of MH as an alternative. 
- The experiments show interesting trials with different variations and honest explanations of the obtained scores. Overall sampling from MLM is a new task compared to auto-regressive LMs it is natural that the performance obtained is going to be slightly inferior. 

*Areas of enhancement*

- The readability of Section 2 (Energy) could be enhanced, the connections between E_local and E_raw could be elaborated since both are quite similar computationally. The current structure gives the impression that they are completely different formulations. 

- Section 6 (Experiments) could be enhanced by providing a global narrative and coherence between the experiments, It is easy to get lost and compare different tables. For example, it took me some time to notice that the results from Table 4 are superior to those of table 3 and 2 and therefore Block MH sampling is useful. 
   
- For Open-ended text generation experiments the corpus level diversity using (SELF-BLEU) should be reported overall, some generation examples could be displayed in the appendix to judge the overall quality (see questions below).


*Questions / recommendations to authors*

- What are the theoretical/conceptual differences between E_norm and E_raw? I see they have been used both in previous work and empirically they perform differently, are there any conceptual differences that motivated you to use both formulations? If it was merely experimental it is fine just to mention it clearly in the text. 

- Samples from MH are known to have high auto-correlation (i.e. i.i.d), this is due to the nature of the local proposal. For applications like open-ended generation (unlike NMT) where the user expects large corpus diversity, this is a crucial feature to have. Although authors report *Novel* transition rates this could be misleading as it only shows the novel transition while the space of effective X could still be limited. I recommend reporting SELF-BLEU in the open-ended generation experiments to indicate corpus-level diversity. For more details see: https://arxiv.org/pdf/1811.02549.pdf

- section 7: "Although the autoregressive approach is superior (30.18 de-en ..."  it would be great adding those results in table 5 to allow easier comparison even when they are superior to non-autoregressive models (which is understandable). 

- Generation examples: For transparency, one would like to see examples generated from the MH algorithm, perhaps dump a subset of all generations of each method in a table in the appendix. One would like to asses the auto-correlation effect of MH on the diversity of the generated samples. 

- The locally normalized scoring formulation is referred to as  $E_local$ $E_norm$, "norm" and "local" interchangeably this confuses the reader.  

- Equation of $E_{local}$ in the bottom of page 3 has two symbols t and i. it is not clear where the $i$ comes from is this a typo? 
 
- Page 3: "the following free conditionals are inconsistent" in fact this example is not clear, I had to go back and forth many times with the appendix to get it right. It would be great to take the time to explain this example in a self-contained way. 

- Figure 2 is not clear that is the green curve 
summary_of_the_review: Overall the paper is theoretically sound experiments are sufficient in my opinion and shows that the proposed method works.
I would love the authors to enhance the accessibility/readability of the manuscript by unifying mathematical notation, defining the rationale behind using $E_{local}$ and $E_{norm}$ rather than long paragraphs referring to previous work instead of motivation. It would be great to simplify the connections between them in layman terms (sum of scores and sum of softmax outputs).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to use metropolis-hasting Monte-Carlo to draw samples from energy-based distributions over sequences. They construct proposal distributions based on trained masked language models (MLMs). Compared to an alternative that uses Gibbs sampling on MLMs, the proposed method is theoretically sound (i.e., correctly approximating the target distribution) and empirically effective on a couple of sentence generation tasks. 

The proposed method is novel and principled.  

main_review: The strengths of the paper is its technical novelty and rigor as well as its thorough experimental studies. The weakness of this paper is its lack of clarity on a few specifics. 

The proposed method could approximate energy-based sequence models with a good proposer (i.e., MLM), and the paper clearly discusses its general theoretical advantages over a Gibbs sampling alternative. The extensive experiments on machine translation and unconditional generation well support the claims.  

However, I would appreciate more details on the specific target distributions used in the experiments: e.g., I am not familiar with MASK-PREDICT and it is not obvious to me how the general technical remarks apply to this specific case. 

summary_of_the_review: Overall, I think it is a good paper. I believe my concerns are more about presentation instead of the actual methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tries to interpret masked language models as energy-based sequence models and proposes two energy parametrizations derivable from the trained MLMs. The primary contribution in paper is to rectify the incorrect assumption in the prior work and propose Metropolis-Hastings (MH) based sampling algorithms for these energy networks. And another contribution is to design a block-replacement proposal distribution for improve mixing of the Markov chain in our proposed MH sampling framework, which results in faster generation and better samples. The paper justifies their findings on the conditional generation task of neural machine translation (NMT) by the pre-trained models MASK-PREDICT and the task of unconditional generation by Bert.

main_review: strengths：
1. The author introduces energy networks to rectify the incorrect assumption which is existed in previous work [1].  
2. This paper proposed two novel energy parametrizations and introduces Metropolis-Hasting sampling to text generation based on MLM models.
3. All methods are empirically verified in a variety of experiment settings.

weaknesses：
1. This paper lacks the introduces of previous works, such as undirected generation approaches and energy networks, which makes the paper is hard to understand without reading the references.
2. There are many experiments between proposed method with the degenerate Gibbs sampling, but do not compared with other generation models like GPT, BART, etc. The potentiality of  proposed methods can not be estimated.
3. The computational complexity of MH sampler is very high as described in this paper, which may not valuable for real applications. 

questions:
1. Is the methods proposed in this paper applied in inference stage? Which means the model cannot applied in other datasets except the dataset which is used for training the pre-trained model.
	
typos:
In section 4,  "We empirically study the proposed Metropolis Hastings scheme .... and the task of uncondtional generation." -> unconditional 

Reference:
[1] Wang and K. Cho. BERT has a mouth, and it must speak: BERT as a Markov random field language model. arXiv preprint arXiv:1902.04094, 2019.

summary_of_the_review: This paper interprets the masked language model with a novel perspective, and introduces  energy networks and Metropolis-Hasting sampling to text generation based on MLM models. It proves the effectiveness of MLM models on text generation, a good paper to accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a Metropolis-Hastings based sampler, which can be used to draw high-quality samples from non-probabilistic masked language models (MLMs). Two parametrizations for the energy functions from the MLMs are examined - raw scoring and locally normalized scoring. Sampling experiments are conducted for both open-ended unconditional generation and a conditional generation task of machine translation. Variants of proposal distributions are studied, by varying temperature, using nucleus sampling and block MH sampling.
main_review: -Strengths

The authors make good points, pointing out that the prior attempt to interpret a MLM (like BERT) as an MRF is incorrect and proposing to define the proposal distribution by masked conditionals for the MH sampler. A series of empirical experiments on effects of temperature, nucleus sampling and block MH sampling are conducted. Both conditional generation (NMT) and unconditional generation are covered.

-Weaknesses

1. Missing important relevant references.

EBMs (a.k.a. un-normalized models, random fields) have been successfully developed for language modeling in recent years. The sampling methods proposed in this manuscript for the energy-based language model has been studied in [1-5], including Gibbs sampling, MH sampling, block MH sampling, albeit the specific energy functions are different. A recent work in [6] also defines an energy-based language model from MLMs. Connecting and comparing to these previous works are needed.

[1] B. Wang, Z. Ou, and Z. Tan, “Trans-dimensional random fields for language modeling,” ACL, 2015.

[2] B. Wang, Z. Ou, and Z. Tan, “Learning trans-dimensional random fields with applications to language modeling,” IEEE transactions on pattern analysis and machine intelligence, 2018.

[3] B. Wang and Z. Ou, “Language modeling with neural trans-dimensional random fields,” IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2017.

[4] B. Wang and Z. Ou, “Learning neural trans-dimensional random field language models with noise-contrastive estimation,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.

[5] B. Wang and Z. Ou, “Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation,” IEEE Spoken Language Technology Workshop (SLT), 2018.

[6] Clark, Kevin, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. "Pre-training transformers as energy-based cloze models." EMNLP 2020.

2. The theoretical contribution seems weak, given the above prior works.
This manuscript presents extensive empirical results, but such contribution may not be substantial enough to motivate acceptance.

3. When the tasks are text generation (conditional or unconditional), it is expected to compare the proposed method with GPT based generation, in both quality and computational cost. Otherwise, it is not clear how useful the proposed method is. The proposed samplers seem to be computational very expensive, and the authors do not provide any discussions on how to reducing the cost.

Minors: Many notations in Algorithm 1 are not defined in the main text or not consistent when compared to notations in the main text, such as:
f_mlm, \sigma, f_E
summary_of_the_review: see above.
review_title: null

Meta-review: This paper interprets pre-trained masked language models (MLMs) as energy-based sequence models and designs a tractable MCMC sampling algorithm based on Metropolis-Hastings with proposals derived from MLMs themselves. 

The strategy is simple, reasonably elegant, and fixes technical mistakes of prior work. The proposed algorithm addresses intractabilities of some naive MCMC schemes, does not require modifications to MLM training, and makes good use of MLMs themselves as proposals thus being crucially economical about resources. 

We had some concerns about speed of generation and the paper's positioning with regards to existing strategies for sampling from energy-based models (already during parameter estimation). While I understand that for many applications speed of generation is crucial, I think that on its own should not keep this line of research outside our best venues. And I hope steps like this one will lead to faster algorithms in the near future. I do relate to the issue of positioning, and I am glad the authors did not take it lightly. In the rebuttal phase the related work and positioning have been improved, but the authors remarked that the limited space for the camera-ready was preventing them from expanding the discussion. A note to authors: it's not a bad idea to have an expanded related work section in appendix.
Decision: Accept (Poster)