Introduction
BERT BIBREF0 is a recently released sequence model used to achieve state-of-art results on a wide range of natural language understanding tasks, including constituency parsing BIBREF1 and machine translation BIBREF2 . Early work probing BERT's linguistic capabilities has found it surprisingly robust BIBREF3 .
BERT is trained on a masked language modeling objective. Unlike a traditional language modeling objective of predicting the next word in a sequence given the history, masked language modeling predicts a word given its left and right context. Because the model expects context from both directions, it is not immediately obvious how to efficiently evaluate BERT as a language model (i.e., use it to evaluate the probability of a text sequence) or how to sample from it.
We attempt to answer these questions by showing that BERT is a combination of a Markov random field language model BIBREF4 , BIBREF5 with pseudo log-likelihood BIBREF6 training. This formulation automatically leads to a sampling procedure based on Gibbs sampling.
BERT as a Markov Random Field
Let $X=(x_1, \ldots , x_T)$ be a sequence of random variables $x_i$ 's. Each random variable is categorical in that it can take one of $M$ items from a vocabulary $V=\left\lbrace  v_1, \ldots , v_{M} \right\rbrace $ . These random variables form a fully-connected graph with undirected edges, indicating that each variable $x_i$ is dependent on all the other variables.
Using BERT as an MRF-LM
The discussion so far implies that BERT is in fact a Markov random field language model (MRF-LM) and that it learns a distribution over sentences (of some given length.) This suggests that we can use BERT not only as parameter initialization for finetuning but as a generative model of sentences to either score a sentence or sample a sentence.
Experiments
Our experiments demonstrate the potential of using BERT as a standalone language model rather than as a parameter initializer for transfer learning BIBREF0 , BIBREF2 , BIBREF16 . We show that sentences sampled from BERT are well-formed and are assigned high probabilities by an off-the-shelf language model. We take pretrained BERT models trained on a mix of Toronto Book Corpus BIBREF17 and Wikipedia provided by BIBREF0 and its PyTorch implementation provided by HuggingFace.
Evaluation
We consider several evaluation metrics to estimate the quality and diversity of the generations.
We follow BIBREF18 by computing BLEU BIBREF19 between the generations and the original data distributions to measure how similar the generations are. We use a random sample of 5000 sentences from the test set of WikiText-103 BIBREF20 and a random sample of 5000 sentences from TBC as references.
We also evaluate the perplexity of a trained language model on the generations as a rough proxy for fluency. Specifically, we use the Gated Convolutional Language Model BIBREF21 pretrained on WikiText-103.
Following BIBREF22 , we compute self-BLEU: for each generated sentence, we compute BLEU treating the rest of the sentences as references, and average across sentences. Self-BLEU measures how similar each generated sentence is to the other generations; high self-BLEU indicates that the model has low sample diversity.
We also evaluate the percentage of $n$ -grams that are unique, when compared to the original data distribution and within the corpus of generations. We note that this metric is somewhat in opposition to BLEU between generations and data, as fewer unique $n$ -grams implies higher BLEU.
We use the non-sequential sampling scheme, as empirically this led to the most coherent generations. We show generations from the sequential sampler in Table 4 in the appendix. We compare against generations from a high-quality neural language model, the OpenAI Generative Pre-Training Transformer BIBREF23 , which was trained on TBC and has approximately the same number of parameters as the base configuration of BERT. For all models, we generate 1000 uncased sequences of length 40.
Results
We present sample generations, quality results, and diversity results respectively in Tables 1 , 2 , 3 .
We find that, compared to GPT, the BERT generations are of worse quality, but are more diverse. Particularly telling is that the outside language model, which was trained on Wikipedia, is less perplexed by the GPT generations than the BERT generations. GPT was only trained on romance novels, whereas BERT was trained on romance novels and Wikipedia. However, we do see that the perplexity on BERT samples is not absurdly high, and in reading the samples, we find that many are fairly coherent.
We find that BERT generations are more diverse than GPT generations. GPT has high $n$ -gram overlap (smaller percent of unique $n$ -grams) with TBC, but surprisingly also with WikiText-103, despite being trained on different data. BERT has lower $n$ -gram overlap with both corpora, perhaps because of worse quality generations, but also has lower self-BLEU.
Conclusion
We show that BERT is a Markov random field language model. We give a practical algorithm for generating from BERT without any additional training and verify in experiments that the algorithm produces diverse and fairly fluent generations. Further work might explore sampling methods that do not need to run the model over the entire sequence each iteration and that enable conditional generation. To facilitate further investigation, we release our code on GitHub at https://github.com/kyunghyuncho/bert-gen and a demo as a Colab notebook at https://colab.research.google.com/drive/1MxKZGtQ9SSBjTK5ArsZ5LKhkztzg52RV.
Acknowledgements
AW is supported by an NSF Graduate Research Fellowship. KC is partly supported by Samsung Advanced Institute of Technology (Next Generation Deep Learning: from Pattern Recognition to AI) and Samsung Electronics (Improving Deep Learning using Latent Structure).
Other Sampling Strategies
We investigated two other sampling strategies: left-to-right and generating for all positions at each time step. See Section "Using BERT as an MRF-LM" for an explanation of the former. For the latter, we start with an initial sequence of all masks, and at each time step, we would not mask any positions but would generate for all positions. This strategy is designed to save on computation. However, we found that this tended to get stuck in non-fluent sentences that could not be recovered from. We present sample generations for the left-to-right strategy in Table 4 .

Question: Which metrics are used for evaluating the quality?
Answer: BLEU, perplexity,  self-BLEU, percentage of $n$ -grams that are unique

BLEU,  perplexity