SlowGuess/NAACL_2021_OCR huggingface dataset:
Here, the KL-divergence corresponds to the  $L_{\mathrm{MLM}}$  objective, up to a constant entropy term that depends on  $p$ . We present the proof in Appendix C. In other words, the MLM objective is implicitly encouraging the model to match its implied conditional MI to that of the data. We now use this result to create an estimator that extracts the conditional independence structures implied by MLM.
4.2 Extracting statistical dependencies implied by MLMs
Our earlier analysis in Proposition 4 suggests that an MLM with low loss has an accurate approximation of conditional mutual information. Using this result, we will now propose a procedure which estimates  $\hat{I}_{p_{\theta}}$ . The definition of  $\hat{I}_{p_{\theta}}$  shows that if we can access samples of  $x_{i}$  and  $x_{j}$  from the true distribution  $p$ , then we can directly estimate the conditional mutual information by using the log probabilities from the MLM. Unfortunately, we cannot draw new samples of  $x_{j} \mid X_{\{i,j\}}$ , leading us to approximate this distribution using Gibbs sampling on the MLM distribution.
Our Gibbs sampling procedure is similar to the one proposed in Wang and Cho (2019). We start with  $X^0 = X_{\backslash \{i,j\}}$ . For the  $t$ th iteration, we draw a sample  $x_i^t$  from  $p_\theta(x_i|X_{\backslash i}^{t-1})$ , and update by  $X^t = X^{t-1}(i,x_i^t)$ . Then, we draw a sample  $x_j^t$  from  $p_\theta(x_j|X_{\backslash j}^t)$  and set  $X^t = X^t(j,x_j^t)$ . We repeat and use the samples  $(x_i^1,x_j^1),\ldots,(x_i^t,x_j^t)$  to compute the expectations for conditional MI.
This procedure relies upon an additional assumption that samples drawn from the MLM are faithful approximations of the data generating distribution. However, we show empirically that even this approximation is sufficient to test the hypothesis that the conditional independences learned by an MLM capture syntactic dependencies (Section 5.2).
We now test two predictions from our analyses. First, similar to our observation in the case study, we show that cloze-like masks do not explain the success of uniform masks on three real-world datasets. Second, our alternative view of relating MLM to graphical models suggests that statistical dependencies learned by MLMs may capture linguistic structures useful for downstream tasks. We demonstrate this by showing that MLMs' statistical dependencies reflect syntactic dependencies.
5.1 Uniform vs Cloze-like Masking
Setup. We now demonstrate that real-world tasks and MLMs show a gap between task-specific cloze masks and random masks. We compare the MLM with random masking to two different control groups. In the positive control (CLOZE), we pretrain with only cloze-like masks and in the negative control (NOCLOZE), we pretrain by explicitly excluding cloze-like masks. If the success of MLM
can be mostly explained by implicit cloze reductions, then we should expect CLOZE to have strong downstream performance while NOCLOZE leads to a minimal performance gain. We compare pretraining with the uniform masking strategy used in BERT (UNIFORM) to these two control groups. If UNIFORM performs worse than the positive control and more similar to the negative control, then we know that uniform masking does not leverage cloze-like masks effectively.
Simulating Pretraining. Given computational constraints, we cannot retrain BERT from scratch. Instead, we approximate the pretraining process by continuing to update BERT with MLM (Gururan-gan et al., 2020), which we refer to as second-stage pretraining. Although this is an approximation to the actual pretraining process, the second-stage pretraining shares the same fundamental problem for pretraining: how can unsupervised training lead to downstream performance gains?
We study the effectiveness of different masking strategies by comparing to a BERT model without second-stage pretraining (VANILLA). We experiment with three text classification datasets: SST-2 (Socher et al., 2013), Hyperpartisan (Kiesel et al., 2019), and AGNews (Zhang et al., 2015). SST-2 classifies movie reviews by binary sentiment; Hyperpartisan is a binary classification task on whether a news article takes an extreme partisan standpoint; and AGNews classifies news articles into four different topics. On SST-2 and AGNews, we perform the second-stage pretraining on the training inputs (not using the labels). On Hyperpartisan, we use 100k unlabeled news articles that are released with the dataset. For SST-2 and AGNews, we study a low-resource setting and set the number of finetuning examples to be 20. For Hyperpartisan, we use the training set, which has 515 labeled examples. All evaluations are performed by fine-tuning a bert-base-uncased model (See Appendix A for full details).
Approximating Cloze-like Masking. We cannot identify the optimal set of cloze-like masks for an arbitrary downstream task, but these three tasks have associated lexicons which we can use to approximate the cloze-like masks. For SST-2, we take the sentiment lexicon selected by Hu and Liu (2004); for Hyperpartisan, we take the NRC word-emotion association lexicon (Mohammad and Turney, 2013); and for AGNews, we extract topic words by training a logistic regression classifier and
Figure 5: Finetuning performance with different masking strategies averaged across twenty random trials and error bars showing  $95\%$  confidence intervals. VANILLA represents a BERT model without any second-stage pretraining. CLOZE and NoCLOZE represent models train with or without cloze-like masks, respectively. UNIFORM uses the uniform random masking strategy proposed in Devlin et al. (2019) for second-stage pretraining.
taking the top 1k features to be cloze-like masks.
Results. Figure 5 plots the finetuning performance of different masking strategies. We observe that UNIFORM outperforms VANILLA, which indicates that second-stage pretraining is extracting useful information and our experiment setup is useful for studying how MLM leads to performance gains. As expected, CLOZE achieves the best accuracy, which confirms that cloze-like masks can be helpful and validates our cloze approximations.
The UNIFORM mask is much closer to NO-CLOZE than CLOZE. This suggests that uniform masking does not leverage cloze-like masks well and cloze reductions alone cannot account for the success of MLM. This view is further supported by the observation that NO-CLOZE outperforms VANILLA suggesting that generic masks that are not cloze-like still contain useful inductive biases.
Our results support our earlier view that there may be an alternative mechanism that allows generic masks that are not cloze-like to benefit downstream learning. Next, we will empirically examine BERT's learned conditional independence structure among tokens and show that the statistical dependencies relate to syntactic dependencies.
5.2 Analysis: Unsupervised Parsing
Our analysis in section 4.1 shows that conditional MI (which is optimized by the MLM objective) can extract conditional independences. We will show that statistical dependencies estimated by conditional MI are related to syntactic dependencies by using conditional MI for unsupervised parsing.
Background. One might expect that the statistical dependencies among words are correlated with syntactic dependencies. Indeed, Futrell et al. (2019) show that heads and dependents in dependency parse trees have high pointwise mutual information (PMI) on average. However, previous at
tempts (Carroll and Charniak, 1992; Paskin, 2002) show that unsupervised parsing approaches based on PMI achieve close to random accuracy. Our analysis suggests that MLMs extract a more fine-grained notion of statistical dependence (conditional MI) which does not suffer from the existence of latent variables (Proposition 3). We now show that the conditional MI captured by MLMs achieves far better performance, on par with classic unsupervised parsing baselines.
Baselines. We compare conditional MI to PMI as well as conditional PMI, an ablation in which we do not take expectation over possible words. For all statistical dependency based methods (cond. MI, PMI, and cond. PMI), we compute pairwise dependence for each word pair in a sentence and construct a minimum spanning tree on the negative values to generate parse trees. To contextualize our results, we compare against three simple baselines: RANDOM which draws a random tree on the input sentence, LINEARChain which links adjacent words in a sentence, and a classic unsupervised parsing method (Klein and Manning, 2004).
Experimental Setup. We conduct experiments on the English Penn Treebank using the WSJ corpus and convert the annotated constituency parses to Stanford Dependency Formalism (de Marneffe et al., 2006). Following Yang et al. (2020), we evaluate on sentences of length  $\leq 10$  in the test split, which contains 389 sentences (Appendix B.1 describes the same experiment on longer sentences, which have similar results). We experiment with the bert-base-cased model (more details in Appendix A) and evaluate by the undirected unlabeled attachment score (UUAS).
Results. Table 1 shows a much stronger-than-random association between conditional MI and dependency grammar. In fact, the parses extracted
Figure 6: An example parse extracted from conditional MI. The black parse tree above the sentence represents the ground-truth parse and the red parse below is extracted from conditional MI. The correctly predicted edges are labeled with the annotated relations, and the incorrect ones are labeled as wrong.
Method	UUAS
RANDOM	28.50 ± 0.73
LINEARCHAIN	54.13
Klein and Manning (2004)	55.91 ± 0.68
PMI	33.94
CONDITIONAL PMI	52.44 ± 0.19
CONDITIONAL MI	58.74 ± 0.22
from conditional MI has better quality than LINEARCHAIN and the classic method (Klein and Manning, 2004). Unlike conditional MI, PMI only has a close-to-random performance, which is consistent with prior work. We also see that conditional MI outperforms conditional PMI, which is consistent with our theoretical framework that suggests that conditional MI (and not PMI) recovers the graphical model structure.
We also perform a fine-grained analysis by investigating relations where conditional MI differs from LINEARCHAIN. Because the test split is small and conditional MI does not involve any training, we perform this analysis on 5,000 sentences from the training split. Table 2 presents the results and shows that conditional MI does not simply recover the linear chain bias. Meanwhile, we also observe a deviation between conditional MI and dependency grammar on relations like number and cc. This is reasonable because certain aspects of dependency grammar depend on human conventions that do not necessarily have a consensus (Popel et al., 2013).
Figure 6 illustrates with an example parse extracted from conditional MI. We observe that conditional MI correctly captures obj and conj. Knowing the verb, e.g. represents, limits the range of objects that can appear in a sentence so intuitively we expect a high conditional MI between the direct object and the verb. Similarly for phrases like "A and B", we would expect A and B to be statistically dependent. However, conditional MI fails
Table 1: Unlabeled Undirected Attachment Score on WSJ10 test split (section 23). Error bars show standard deviation across three random seeds.
Relation	Conditional MI	Linear Chain
xcomp	48.18	9.93
conj	43.36	7.58
dobj	58.96	30.33
number	50.55	92.62
quantmod	56.82	72.73
cc	31.39	41.10
Table 2: Six relations on which conditional MI disagrees with LINEARCHAIN under log odds ratio test with  $p = 0.05$ . A comprehensive list is in Appendix A.
to capture  $\mathrm{cc}$  (between apathy and or). Instead, it links or with either which certainly has statistical dependence. This once again suggests that the 'errors' incurred by the conditional PMI method are not simply failures to estimate dependence but natural differences in the definition of dependence.
We study how MLM with uniform masking can learn useful linguistic structures and inductive biases for downstream tasks. Our work demonstrates that a substantial part of the performance gains of MLM pretraining cannot be attributed to task-specific, cloze-like masks. Instead, learning with task-agnostic, generic masks encourages the model to capture direct statistical dependencies among tokens, and we show through unsupervised parsing evaluations that this has a close correspondence to syntactic structures. Existing work has suggested that statistical and syntactic dependencies are fundamentally different, with unsupervised parsing based on PMI achieving close-to-random performance. Our work demonstrates that this is not necessarily the case, and better measures of statistical dependence (such as those learned by MLMs) can serve as implicit supervision for learning syntactic structures. Our findings open new space for future works on how syntax can be learned in an emergent way and on how to design masking strategies that further improve dependency learning.
We thank Rishi Bommasani, Lisa Li, Kawin Ethayarajh, the anonymous reviewers and the Stanford NLP group for their helpful feedback and discussions.
A. Anandkumar, V. Y. F. Tan, F. Huang, and A. S. Willsky. 2012. High-dimensional structure estimation in ising models: Local separation criterion. Annals of Statistics, 40(3):1346-1375.
S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski. 2015. Random walks on context spaces: Towards an explanation of the mysteries of semantic word embeddings. arXiv preprint arXiv:1502.03520.
Y. Belinkov and J. Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics (TACL), 7:49-72.
D. Blei, A. Ng, and M. I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research (JMLR), 3:993-1022.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
G. Carroll and E. Charniak. 1992. Two experiments on learning probabilistic dependency grammars from corpora. In AAAI Conference on Artificial Intelligence.
M. de Marneffe, B. MacCartney, and C. D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In LREC.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Association for Computational Linguistics (ACL), pages 4171-4186.
R. Futrell, P. Qian, E. Gibson, E. Fedorenko, and I. Blank. 2019. Syntactic dependencies correspond to word pairs with high mutual information. In Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019), pages 3-13, Paris, France. Association for Computational Linguistics.
S. Gururangan, A. Marasovic, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. 2020. Don't stop pretraining: Adapt language models to
domains and tasks. In Association for Computational Linguistics (ACL), pages 8342-8360, Online. Association for Computational Linguistics.
T. B. Hashimoto, D. Alvarez-Melis, and T. S. Jaakkola. 2016. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics (TACL), 4:273-286.
J. He, G. Neubig, and T. Berg-Kirkpatrick. 2018. Unsupervised learning of syntactic structure with invertible neural projections. In Empirical Methods in Natural Language Processing, pages 1292-1302, Brussels, Belgium. Association for Computational Linguistics.
J. Hewitt and C.D. Manning. 2019. A structural probe for finding syntax in word representations. In *North American Association for Computational Linguistics (NAACL)*, pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
M. Hu and B. Liu. 2004. Mining and summarizing customer reviews. In International Conference on Knowledge Discovery and Data Mining (KDD), KDD '04, page 168-177, New York, NY, USA. Association for Computing Machinery.
Z. Jiang, F. F. Xu, J. Araki, and G. Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics (TACL), 8:423-438.
J. Kiesel, M. Mestre, R. Shukla, E. Vincent, P. Adineh, D. Corney, B. Stein, and M. Potthast. 2019. SemEval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829-839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
D. Klein and C.D. Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Association for Computational Linguistics (ACL), pages 478-485, Barcelona, Spain.
J. D. Lee, Q. Lei, N. Saunshi, and J. Zhuo. 2020. Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008.01064.
O. Levy and Y. Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems, volume 27, pages 2177-2185. Curran Associates, Inc.
N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In *North American Association for Computational Linguistics (NAACL)*, pages 1073–1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Y. Liu. 2019. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky. 2014. The stanford coreNLP natural language processing toolkit. In ACL system demonstrations.
N. Meinshausen and P. Buhlmann. 2006. High-dimensional graphs and variable selection with the lasso. Annals of Statistics, 34(3):1436-1462.
T. Mikolov, K. Chen, G. Corrado, and Jeffrey. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
S. M. Mohammad and P. D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436-465.
B. Pang, L. Lee, and S. Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Empirical Methods in Natural Language Processing, pages 79-86. Association for Computational Linguistics.
M. A. Paskin. 2002. Grammatical bigrams. In Advances in Neural Information Processing Systems (NeurIPS).
J. Pennington, R. Socher, and C. D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
M. Peters, M. Neumann, L. Zettlemoyer, and W. Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. In Empirical Methods in Natural Language Processing, pages 1499-1509, Brussels, Belgium. Association for Computational Linguistics.
F. Petroni, T. Roktaschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller. 2019. Language models as knowledge bases? In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
M. Popel, D. Mareček, J. Štepanek, D. Zeman, and Z. Žabokrtský. 2013. Coordination structures in dependency treebanks. In Association for Computational Linguistics (ACL), pages 517-527, Sofia, Bulgaria. Association for Computational Linguistics.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
N. Saunshi, S. Malladi, and S. Arora. 2020. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648.
R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP).
G. W. Stewart and J. Sun. 1990. Matrix Perturbation Theory. Academic Press.
Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. 2020. Investigating transferability in pretrained language models. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1393-1401. Association for Computational Linguistics.
I. Tenney, P. Xia, B. Chen, A. Wang, A. Poliak, R. T. McCoy, N. Kim, B. Van Durme, S. Bowman, D. Das, and E. Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations (ICLR).
R. Tibshirani. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288.
D. Wadden, U Wennberg, Y. Luan, and H. Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5784-5789, Hong Kong, China. Association for Computational Linguistics.
A. Wang and K. Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. arXiv preprint arXiv:1902.04094.
Z. Wu, Y. Chen, B. Kao, and Q. Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Association for Computational Linguistics (ACL), pages 4166-4176, Online. Association for Computational Linguistics.
S. Yang, Y. Jiang, W. Han, and K. Tu. 2020. Second-order unsupervised neural dependency parsing. arXiv preprint arXiv:2010.14720.
T. Zhang, F. Wu, A. Katiyar, K. Q. Weinberger, and Y. Artzi. 2020. Revisiting few-sample bert fin-tuning. arXiv preprint arXiv:2006.05987.
X. Zhang, J. Zhao, and Y. LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28, pages 649-657. Curran Associates, Inc.
J. Zhu, Y. Xia, L. Wu, D. He, T. Qin, W. Zhou, H. Li, and T. Liu. 2020. Incorporating BERT into neural machine translation. In International Conference on Learning Representations (ICLR).
Dataset	# Classes	# Pretrain	# Finetune	# Test
SST-2	2	67k	20	1.8k
Hyperpartisan	2	100k	515	130
AGNews	4	113k	20	6.7k
Table 3: Specifications of datasets. For AGNews, we put away  $6.7\mathrm{k}$  as a development set.
Experimental details for Section 3.2 Our transformers have 2 layers and for each transformer block, the hidden size and the intermediate size are both 64. We finetune the models for 10 epochs and apply early stopping based on validation accuracy. We use Adam (Kingma and Ba, 2014) for optimization, using a learning rate of  $1e^{-3}$  for pretraining and  $1e^{-4}$  for finetuning.
Experimental details for Section 5.1 Table 3 summarizes the dataset statistics of three real-world datasets we studied. For second stage pretraining, we update the BERT model for 10 epochs. Following the suggestion in Zhang et al. (2020), we finetune the pretrained BERT models for 400 steps, using a batch size of 16 and a learning rate of  $1e^{-5}$ . We apply linear learning rate warmup for the first  $10\%$  of finetuning and linear learning rate decay for the rest. For SST-2 and AGNews, we average the results over 20 random trials. For Hyperpartisan, because the test set is small and the variation is larger, we average the results over 50 random trials and evaluate on the union the development set and the test set for more stable results.