SlowGuess/NAACL_2021_OCR huggingface dataset:
Tianyi Zhang
Computer Science Department
Stanford University
tianyizhang@cs.stanford.edu
We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained performance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the masked language model (MLM) objective and existing methods for learning statistical dependencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful inductive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the implied statistical dependence structure outperforms a classic method for unsupervised parsing (58.74 vs. 55.91 UUAS).
Pretrained masked language models (Devlin et al., 2019; Liu et al., 2019b) have benefitted a wide range of natural language processing (NLP) tasks (Liu, 2019; Wadden et al., 2019; Zhu et al., 2020). Despite recent progress in understanding what useful information is captured by MLMs (Liu et al., 2019a; Hewitt and Manning, 2019), it remains a mystery why task-agnostic masking of words can capture linguistic structures and transfer to downstream tasks.
One popular justification of MLMs relies on viewing masking as a form of cloze reduction.
Cloze-like Masking
Figure 1: We study the inductive bias of MLM objectives and show that cloze-like masking (left) does not account for much of the downstream performance gains. Instead, we show that MLM objectives are biased towards extracting both statistical and syntactic dependencies using random masks (right).
Dependency Learning
Cloze reductions reformulate an NLP task into a prompt question and a blank and elicit answers by filling in the blank (Figure 1). When tested by cloze reductions pretrained MLMs and left-to-right language models (LMs) have been shown to possess abundant factual knowledge (Petroni et al., 2019) and display impressive few-shot ability (Brown et al., 2020). This success has inspired recent hypotheses that some word masks are cloze-like and provide indirect supervision to downstream tasks (Saunshi et al., 2020; Lee et al., 2020). For example, a sentiment classification task (Pang et al., 2002) can be reformulated into filling in like or hate in the cloze  $I$  [MASK] this movie. Such cloze-like masks provide a clear way in which an MLM can implicitly learn to perform sentiment classification.
While this hypothesis is appealing, MLMs in practice are trained with uniform masking that does not contain the special structure required by cloze-like masks most of the time. For example, predicting a generic word this in the cloze I like [MASK] movie would not offer task-specific supervision. We quantify the importance of cloze-like and generic masks by explicitly creating cloze-like masks using task-specific lexicons and comparing models pretrained on these masks. These experiments suggest that although cloze-like masks can be helpful, the success of uniform masking cannot
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5131-5146 June 6-11, 2021. ©2021 Association for Computational Linguistics
be explained via cloze-like masks alone. In fact, we demonstrate that uniform masking performs as well as a negative control where we explicitly remove cloze-like masks from the mask distribution.
To address this mismatch between theory and practice, we offer a new hypothesis of how generic masks can help downstream learning. We propose a conceptual model for MLMs by drawing a correspondence between masking and graphical model neighborhood selection (Meinshausen and Buhlmann, 2006). Using this, we show that MLM objectives are designed to recover statistical dependencies in the presence of latent variables and propose an estimator that can recover these learned dependencies from MLMs. We hypothesize that statistical dependencies in the MLM objective capture useful linguistic dependencies and demonstrate this by using recovered statistical dependencies to perform unsupervised parsing, outperforming an actual unsupervised parsing baseline (58.74 vs 55.91 UUAS; Klein and Manning, 2004). We release our implementation on  $\mathbf{G h i t b u^{1}}$
Theories inspired by Cloze Reductions. Cloze reductions are fill-in-the-blank tests that reformulate an NLP task into an LM problem. Existing work demonstrates that such reductions can be highly effective for zero/few-shot prediction (Radford et al., 2019; Brown et al., 2020) as well as relation extraction (Petroni et al., 2019; Jiang et al., 2020).
These fill-in-the-blank tasks provide a clear way by which LMs can obtain supervision about downstream tasks, and recent work demonstrates how such implicit supervision can lead to useful representations (Saunshi et al., 2020). More general arguments by Lee et al. (2020) show these theories hold across a range of self-supervised settings. While these theories provide compelling arguments for the value of pre-training with cloze tasks, they do not provide a clear reason why uniformly random masks such as those used in BERT provide such strong gains. In our work, we quantify this gap using lexicon-based cloze-like masks and show that cloze-like masks alone are unlikely to account for the complete success of MLM since generic and non-cloze masks are responsible for a substantial part of the empirical performance of MLMs.
Theories for vector representations. Our goal of understanding how masking can lead to useful inductive biases and linguistic structures is closely related to that of papers studying the theory of word embedding representations (Mikolov et al., 2013; Pennington et al., 2014; Arora et al., 2015). Existing work has drawn a correspondence between word embeddings and low-rank factorization of a pointwise mutual information (PMI) matrix (Levy and Goldberg, 2014) and others have shown that PMI is highly correlated with human semantic similarity judgements (Hashimoto et al., 2016).
While existing theories for word embeddings cannot be applied to MLMs, we draw inspiration from them and derive an analogous set of results. Our work shows a correspondence between MLM objectives and graphical model learning through conditional mutual information, as well as evidence that the conditional independence structure learned by MLMs is closely related to syntactic structure.
Probing Pretrained Representations. Recent work has applied probing methods (Belinkov and Glass, 2019) to analyze what information is captured in the pretrained representations. This line of work shows that pretrained representations encode a diverse range of knowledge (Peters et al., 2018; Tenney et al., 2019; Liu et al., 2019a; Hewitt and Manning, 2019; Wu et al., 2020). While probing provides intriguing evidence of linguistic structures encoded by MLMs, they do not address the goals of this work, which is how the pretraining objective encourages MLMs to extract such structures.
Masked Language Modeling asks the model to predict a token given its surrounding context. Formally, consider an input sequence  $X$  of  $L$  tokens  $\langle x_1,\ldots ,x_L\rangle$  where each variable takes a value from a vocabulary  $\mathcal{V}$ . Let  $X\sim D$  be the data generating distribution of  $X$ . Let  $x_{i}$  be the  $i$ th token in  $X$ , and let  $X_{\backslash i}$  denote the sequence after replacing the  $i$ th token with a special [MASK] token. In other words,
$$
X _ {\backslash i} := \langle x _ {1}, \ldots , x _ {i - 1}, [ \mathrm {M A S K} ], x _ {i + 1}, \ldots , x _ {L} \rangle .
$$
Similarly, define  $X_{\backslash \{i,j\}}$  as replacing both  $x_{i}$  and  $x_{j}$  with [MASK]. MLM determines what tokens are masked by a mask distribution  $i\sim M$ . The goal of MLM is to learn a probabilistic model  $p_{\theta}$
1https://github.com/tatsu-lab/mlm_inductive.bias
Modified Input: beautiful movie positive
Cloze-like Mask: beautiful movie [MASK]
Generic Mask: beautiful [MASK] positive
that minimizes
$$
L _ {\mathrm {M L M}} = \underset {X \sim D, i \sim M} {\mathbb {E}} - \log p _ {\theta} (x _ {i} | X _ {\backslash i}).
$$
In BERT pretraining, each input token is masked with a fixed, uniform probability, which is a hyperparameter to be chosen. We refer to this strategy as uniform masking.
Finetuning is the canonical method for using pretrained MLMs. Consider a prediction task where  $y \in \mathcal{V}$  is the target variable, e.g., the sentiment label of a review. Finetuning uses gradient descent to modify the pretrained parameters  $\theta$  and learn a new set of parameters  $\phi$  to minimize
$$
L _ {\text {f i n e t u n e}} = \underset {X \sim D ^ {\prime}, y \sim p (y | X)} {\mathbb {E}} - \log p _ {\theta , \phi} (y | X),
$$
where  $p(y|x)$  is the ground-truth distribution and  $D^{\prime}$  is the data distribution of the downstream task.
Our goals. We will study how the mask distribution  $M$  affects downstream performance. We define perfect cloze reductions as some partition of the vocabulary  $\mathcal{V}_y$  such that  $p(x_i\in \mathcal{V}_y|X_{\backslash i})\approx p(y|X)$ . For a distribution  $M$  such that the masks we draw are perfect cloze-reductions, the MLM objective offers direct supervision to finetuning since  $L_{\mathrm{MLM}}\approx L_{\mathrm{finetune}}$ . In contrast to cloze-like masking, in uniform masking we can think of  $p_\theta$  as implicitly learning a generative model of  $\mathbf{X}$  (Wang and Cho, 2019). Therefore, as  $M$  moves away from the ideal distribution and becomes more uniform, we expect  $p_\theta$  to model more of the full data distribution  $D$  instead of focusing on cloze-like supervision for the downstream task. This mismatch between theory and practice raises questions about how MLM with uniform masking can learn useful inductive biases.
When  $L_{\mathrm{MLM}}$  is not  $L_{\mathrm{finetune}}$ , what is  $L_{\mathrm{MLM}}$  learning? We analyze  $L_{\mathrm{MLM}}$  and show that it is similar to a form of conditional mutual information based graphical model structure learning.
3.2 Case Study for Cloze-like Masking
To motivate our subsequent discussions, we perform a controlled study for the case when  $L_{\mathrm{MLM}} \approx$
Figure 2: In our case study, we append the true label to each input and create ideal cloze-like masks. We study how deviations from the ideal mask distribution affect downstream performance by adding in generic masks.
Figure 3: SST-2 development set accuracy. CLOZE- $p\%$  is pretrained on a mixture of masks where  $p\%$  of the masks are Cloze-like. NOPRETRAIN trains a classifier without any pretraining. Even a small modification of the ideal mask distribution degrades performance.
$L_{\mathrm{finetune}}$  and analyze how deviations from the ideal mask distribution affect downstream performance. We perform analysis on the Stanford Sentiment Treebank (SST-2; Socher et al., 2013), which requires models to classify short movie reviews into positive or negative sentiment. We append the ground-truth label (as the word positive or negative) to each movie review (Figure 2). Masking the last word in each review is, by definition, an ideal mask distribution. To study how the deviation from the ideal mask distribution degrades downstream performance, we vary the amount of cloze-like masks during training. We do this by masking out the last word for  $p\%$  of the time and masking out a random word in the movie review for  $(100 - p)\%$  of the time, and choose  $p \in \{0, 20, 40, 60, 80, 100\}$ .
Experimental details. We split the SST-2 training set into two halves, use one for pretraining, and the other for finetuning. For the finetuning data, we do not append the ground-truth label. We pretrain small transformers with  $L_{\mathrm{MLM}}$  using different masking strategies and finetune them along with a baseline that is not pretrained (NOPRETRAIN). Further details are in Appendix A.
Results. We observe that while cloze-like masks can lead to successful transfer, even a small modification of the ideal mask distribution deteriorates performance. Figure 3 shows the development set accuracy of seven model variants averaged across ten random trials. We observe as  $p$  decreases, the performance of CLOZE-  $p\%$  degrades. Notably, CLOZE-80% is already worse than CLOZE-100% and CLOZE-20% does not outperform NOPRE-TRAIN by much. We notice that CLOZE-0% in fact degrades finetuning performance, potentially because the pretrained model is over-specialized to the language modeling task (Zhang et al., 2020; Tamkin et al., 2020). While this is a toy example, we observe similar results for actual MLM models
Figure 4: Our conceptual framework of MLM. All coordinates of  $X$  are dependent on the latent variable  $\mathbf{Z}$  while there is only sparse dependency among  $X$ .
across three tasks (Section 5.1), and this motivates us to look for a framework that explains the success of generic masks in practice.
In the previous section, we saw that cloze-like masks do not necessarily explain the empirical success of MLMs with uniform masking strategies. Understanding uniform masking seems challenging at first, as uniform-mask MLMs seem to lack task-specific supervision and is distinct from existing unsupervised learning methods such as word embeddings (which rely upon linear dimensionality reduction) and autoencoders (which rely upon denoising). However, we show in this section that there is a correspondence between MLM objectives and classic methods for graphical model structure learning. As a consequence, we demonstrate that MLMs are implicitly trained to recover statistical dependencies among observed tokens.
4.1 Intuition and Theoretical Analysis
Our starting point is the observation that predicting a single feature  $(x_{i})$  from all others  $(X_{\backslash i})$  is the core subroutine in the classic Gaussian graphical model structure learning algorithm of Meinshausen and Buhlmann (2006). In this approach,  $L$  different Lasso regression models are trained (Tibshirani, 1996) with each model predicting  $x_{i}$  from  $X_{\backslash i}$ , and the nonzero coefficients of this regression correspond to the conditional dependence structure of the graphical model.
The MLM objective can be interpreted as a nonlinear extension of this approach, much like a classical algorithm that uses conditional mutual information (MI) estimators to recover a graphical model (Anandkumar et al., 2012). Despite the similarity, real world texts are better viewed as models with latent variables (e.g. topics; Blei et al., 2003) and many dependencies across tokens arise due to latent variables, which makes learning the direct dependencies difficult. We show that MLMs
implicitly recover the latent variables and can capture the direct dependencies while accounting for the effect of latent variables. Finally, MLMs are only approximations to the true distribution and we show that the MLM objective can induce high-quality approximations of conditional MI.
Analysis setup. To better understand MLMs as a way to recover graphical model structures, we show mask-based models can recover latent variables and the direct dependencies among variables in the Gaussian graphical model setting of Meinshausen and Buhlmann (2006). Let  $\mathbf{X} = [\mathbf{x}_1,\dots ,\mathbf{x}_L]\in \mathbb{R}^L$  represent an input sequence where each of its coordinates  $\mathbf{x}_i$  represents a token, and  $\mathbf{Z}\in \mathbb{R}^{k}$  be a latent variable that controls the sequence generation process. We assume that all coordinates of  $\mathbf{X}$  are dependent on the latent variable  $\mathbf{Z}$ , and there are sparse dependencies among the observed variables (Figure 4). In other words, we can write  $\mathbf{Z}\sim \mathsf{N}(0,\Sigma_{\mathbf{ZZ}})$  and  $\mathbf{X}\sim \mathsf{N}(A\mathbf{Z},\Sigma_{\mathbf{XX}})$ . Intuitively, we can imagine that  $\mathbf{Z}$  represents shared semantic information, e.g. a topic, and  $\Sigma_{\mathbf{XX}}$  represents the syntactic dependencies. In this Gaussian graphical model, the MLM is analogous to regressing each coordinate of  $\mathbf{X}$  from all other coordinates, which we refer to as masked regression.
MLM representations can recover latent variable. We now study the behavior of masked regression through the representation  $\mathbf{x}_{\mathrm{mask},i}$  that is obtained by applying masked regression on the  $i$ th coordinate of  $\mathbf{X}$  and using the predicted values. Our result shows that masked regression is similar to the two-step process of first recovering the latent variable  $\mathbf{Z}$  from  $\mathbf{X}_{\backslash i}$  and then predicting  $x_{i}$  from  $\mathbf{Z}$ .
Let  $\Sigma_{\mathbf{X}\mathbf{X},\backslash i,i}\in \mathbb{R}^{d - 1}$  be the vector formed by dropping the  $i$ th row and taking the  $i$ th column of  $\Sigma_{\mathbf{X}\mathbf{X}}$  and  $\beta_{2\mathrm{SLS},i}$  be the linear map resulting from the two-stage regression  $\mathbf{X}_{\backslash i}\rightarrow \mathbf{Z}\rightarrow x_i$
Proposition 1. Assuming that  $\Sigma_{\mathbf{XX}}$  is full rank,
$$
\mathbf {x} _ {\text {m a s k}, i} = \beta_ {2 S L S, i} \mathbf {X} _ {\backslash i} + O (\left\| \Sigma_ {\mathbf {X X}, \backslash i, i} \right\| _ {2}),
$$
In other words, masked regression implicitly recovers the subspace that we would get if we first explicitly recovered the latent variables  $(\beta_{2\mathrm{SLS},i})$  with an error term that scales with the off-diagonal terms in  $\Sigma_{XX}$ . The proof is presented in Appendix C.
To give additional context for this result, let us consider the behavior of a different representation learning algorithm: PCA. It is well-known that PCA can recover the latent variables as long as
the  $\Sigma_{\mathbf{ZZ}}$  dominates the covariance  $Cov(\mathbf{X})$ . We state this result in terms of  $\mathbf{X}_{\mathrm{PCA}}$ , the observed data projected to the first  $k$  components of PCA.
Proposition 2. Let  $\lambda_{k}$  be the  $k$ th eigenvalue of  $A\Sigma_{\mathbf{Z}\mathbf{Z}}A^{\top}$  and  $\lambda_{\mathbf{X}\mathbf{X},k + 1}$  be the  $k + 1$ th eigenvalue of  $\Sigma_{\mathbf{X}\mathbf{X}}$  and  $V$  be the first  $k$  eigenvectors of  $\operatorname{Cov}(X)$ . Assuming  $\lambda_{k} > \lambda_{\mathbf{X}\mathbf{X},k + 1}$ , we have
$$
\mathbb {E} _ {\mathbf {X}} \left\| A \mathbf {Z} - \mathbf {X} _ {P C A} \right\| _ {2} \leq
$$
$$
\frac {\sqrt {2} \left\| \Sigma_ {\mathbf {X X}} \right\| _ {o p}}{\lambda_ {k} - \lambda_ {\mathbf {X X} , k + 1}} \left(\left\| A \mathbf {Z} \right\| _ {2} + \sqrt {\operatorname {t r} \left(\Sigma_ {\mathbf {X X}}\right)}\right) + \left\| A A ^ {\top} \right\| _ {o p} \sqrt {\operatorname {t r} \left(\Sigma_ {\mathbf {X X}}\right)},
$$
where  $\| \cdot \|_{op}$  is the operator norm and  $\operatorname{tr}(\cdot)$  is the trace.
This shows that whenever  $\Sigma_{XX}$  is sufficiently small and  $\lambda_{k}$  is large (i.e., the covariance is dominated by  $Z$ ), then PCA recovers the latent information in  $\mathbf{Z}$ . The proof is based on the Davis-Kahan theorem (Stewart and Sun, 1990) and is presented in Appendix C.
Comparing the bound of PCA and masked regression, both bounds have errors that scales with  $\Sigma_{\mathbf{XX}}$ , but the key difference in the error bound is that the error term for masked regression does not scale with the per-coordinate noise (diag  $(\Sigma_{\mathbf{XX}})$ ) and thus can be thought of as focusing exclusively on interactions within  $\mathbf{X}$ . Analyzing this more carefully, we find that  $\Sigma_{\mathbf{XX},\backslash i,i}$  corresponds to the statistical dependencies between  $x_{i}$  and  $\mathbf{X}_{\backslash i}$ , which we might hope captures useful, task-agnostic structures such as syntactic dependencies.
MLM log-probabilities can recover direct dependencies. Another effect of latent variables is that many tokens have indirect dependencies through the latent variables, which poses a challenge to recovering the direct dependencies among tokens. We now show that the MLMs can account for the effect of latent variable.
In the case where there are no latent variables, we can identify the direct dependencies via conditional MI (Anandkumar et al., 2012) because any  $x_{i}$  and  $x_{j}$  that are disconnected in the graphical model will have zero conditional MI, i.e.,  $I(x_{i};x_{j}|X_{\backslash \{i,j\}}) = 0$ . One valuable aspect of MLM is that we can identify direct dependencies even in the presence of latent variables.
If we naively measure statistical dependency by mutual information, the coordinates of  $\mathbf{X}$  would appear dependent on each other because they are all connected with  $\mathbf{Z}$ . However, the MLM objective resolves this issue by conditioning on  $X_{\backslash \{i,j\}}$ . We show that latent variables (such as topics) that are
easy to predict from  $X_{\backslash \{i,j\}}$  can be ignored when considering conditional MI.
Proposition 3. The gap between conditional MI with and without latent variables is bounded by the conditional entropy  $H(\mathbf{Z}|X_{\backslash \{i,j\}})$ ,
$$
\begin{array}{l} I \left(x _ {i}; x _ {j} \mid X _ {\backslash \{i, j \}}\right) - I \left(x _ {i}; x _ {j} \mid \mathbf {Z}, X _ {\backslash \{i, j \}}\right) \\ \leq 2 H (\mathbf {Z} | X _ {\backslash \{i, j \}}). \\ \end{array}
$$
This suggests that when the context  $X_{\backslash \{i,j\}}$  captures enough of the latent information, conditional MI can remove the confounding effect of the shared topic  $\mathbf{Z}$  and extract the direct and sparse dependencies within  $\mathbf{X}$  (see Appendix C for the proof).
MLM objective encourages capturing conditional MI. We have now shown that conditional MI captures direct dependencies among tokens, even in the presence of latent variables. Next, we will show that the MLM objective ensures that a LM with low log-loss accurately captures the conditional MI. We now show that learning the MLM objective implies high-quality estimation of conditional MI. Denote  $X(i,v)$  as substituting  $x_{i}$  with a new token  $v$ ,
$$
X (i, v) = \langle x _ {1}, \dots , x _ {i - 1}, v, x _ {i + 1}, \dots , x _ {L} \rangle .
$$
Conditional MI is defined as the expected pointwise mutual information (PMI) conditioned on the rest of the tokens,
$$
I _ {p} = \underset {x _ {i}, x _ {j}} {\mathbb {E}} [ \log p (x _ {i} | X _ {\backslash i} (j, x _ {j})) - \log \underset {x _ {j} | x _ {i}} {\mathbb {E}} p (x _ {i} | X _ {\backslash i} (j, x _ {j})) ]
$$
where  $I_{p}$  is the abbreviation of  $I_{p}(x_{i};x_{j}|X_{\backslash \{i,j\}})$ . Our main result is that the log-loss MLM objective directly bounds the gap between the true conditional mutual information from the data distribution and an estimator that uses the log-probabilities from the model. More formally,
Proposition 4. Let
$$
\hat {I} _ {p _ {\theta}} = \mathbb {E} _ {x _ {i}, x _ {j}} [ \log p _ {\theta} (x _ {i} | X _ {\backslash i} (j, x _ {j})) - \log \mathbb {E} _ {x _ {j} | x _ {i}} p _ {\theta} (x _ {i} | X _ {\backslash i} (j, x _ {j})) ]
$$
be an estimator constructed by the model distribution  $p_{\theta}$ . Then we can show,
$$
\left| \hat {I} _ {p _ {\theta}} - I _ {p} \right| \leq \mathop {\mathbb {E}} _ {x _ {j}} D _ {\mathrm {k l}} \left(p (x _ {i} | X _ {\backslash i} (j, x _ {j})) \| p _ {\theta} (x _ {i} | X _ {\backslash i} (j, x _ {j}))\right),
$$
where  $D_{\mathrm{kl}}$  represents the KL-divergence.