SlowGuess/ACL_2020_OCR huggingface dataset:
Zhiyong Wu $^{1}$ , Yun Chen $^{2}$ , Ben Kao $^{1}$ , Qun Liu $^{3}$
<sup>1</sup>The University of Hong Kong, Hong Kong, China
$^{2}$ Shanghai University of Finance and Economics, Shanghai, China
<sup>3</sup>Huawei Noah's Ark Lab, Hong Kong, China
{zywu,kao}@cs.hku.hk, yunchen@sufe.edu.cn, qun.liu@huawei.com
By introducing a small set of additional parameters, a probe learns to solve specific linguistic tasks (e.g., dependency parsing) in a supervised manner using feature representations (e.g., contextualized embeddings). The effectiveness of such probing tasks is taken as evidence that the pre-trained model encodes linguistic knowledge. However, this approach of evaluating a language model is undermined by the uncertainty of the amount of knowledge that is learned by the probe itself. Complementary to those works, we propose a parameter-free probing technique for analyzing pre-trained language models (e.g., BERT). Our method does not require direct supervision from the probing tasks, nor do we introduce additional parameters to the probing process. Our experiments on BERT show that syntactic trees recovered from BERT using our method are significantly better than linguistically-uninformed baselines. We further feed the empirically induced dependency structures into a downstream sentiment classification task and find its improvement compatible with or even superior to a human-designed dependency schema.
Recent prevalent pre-trained language models such as ELMo (Peters et al., 2018b), BERT (Devlin et al., 2018), and XLNet (Yang et al., 2019) achieve state-of-the-art performance for a diverse array of downstream NLP tasks. An interesting area of research is to investigate the interpretability of these pre-trained models (i.e., the linguistic properties they capture). Most recent approaches are built upon the idea of probing classifiers (Shi et al., 2016; Adi et al., 2017; Conneau et al., 2018; Peters et al., 2018a; Hewitt and Manning, 2019; Clark et al., 2019; Tenney et al., 2019b; Jawahar et al., 2019). A probe is a simple neural net
work (with a small additional set of parameters) that uses the feature representations generated by a pre-trained model (e.g., hidden state activations, attention weights) and is trained to perform a supervised task (e.g., dependency labeling). The performance of a probe is used to measure the quality of the generated representations with the assumption that the measured quality is mostly attributable to the pre-trained language model.
One downside of such approach, as pointed out in (Hewitt and Liang, 2019), is that a probe introduces a new set of additional parameters, which makes the results difficult to interpret. Is it the pretrained model that captures the linguistic information, or is it the probe that learns the downstream task itself and thus encodes the information in its additional parameter space?
In this paper we propose a parameter-free probing technique called Perturbed Masking to analyze and interpret pre-trained models. The main idea is to introduce the Perturbed Masking technique into the masked language modeling (MLM) objective to measure the impact a word  $x_{j}$  has on predicting another word  $x_{i}$  (Sec 2.2) and then induce the global linguistic properties (e.g., dependency trees) from this inter-word information.
- We introduce a new parameter-free probing technique, Perturbed Masking, to estimate inter-word correlations. Our technique enables global syntactic information extraction.
- We evaluate the effectiveness of our probe over a number of linguistic driven tasks (e.g., syntactic parsing, discourse dependency parsing). Our results reinforce the claims of recent probing works, and further complement them by quantitatively evaluating the validity of their claims.
- We feed the empirically induced dependency structures into a downstream task to make a comparison with a parser-provided, linguist-designed
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166-4176 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
dependency schema and find that our structures perform on-par or even better (Sec 6) than the parser created one. This offers an insight into the remarkable success of BERT on downstream tasks.
2 Perturbed Masking
We propose the perturbed masking technique to assess the impact one word has on the prediction of another in MLM. The inter-word information derived serves as the basis for our later analysis.
2.1 Background: BERT
BERT $^{1}$  (Devlin et al., 2018) is a large Transformer network that is pre-trained on 3.3 billion tokens of English text. It performs two tasks: (1) Masked Language Modeling (MLM): randomly select and mask  $15\%$  of all tokens in each given sequence, and then predict those masked tokens. In masking, a token is (a) replaced by the special token [MASK], (b) replaced by a random token, or (c) kept unchanged. These replacements are chosen  $80\%$ ,  $10\%$ , and  $10\%$  of the time, respectively. (2) Next Sentence Prediction: given a pair of sentences, predict whether the second sentence follows the first in an original document or is taken from another random document.
2.2 Token Perturbation
Given a sentence as a list of tokens  $\mathbf{x} = [x_1, \ldots, x_T]$ , BERT maps each  $x_i$  into a contextualized representation  $H_{\theta}(\mathbf{x})_i$ , where  $\theta$  represents the network's parameters. Our goal is to derive a function  $f(x_i, x_j)$  that captures the impact a context word  $x_j$  has on the prediction of another word  $x_i$ .
We propose a two-stage approach to achieve our goal. First, we replace  $x_{i}$  with the [MASK] token and feed the new sequence  $\mathbf{x} \backslash \{x_{i}\}$  into BERT. We use  $H_{\theta}(\mathbf{x} \backslash \{x_{i}\})_{i}$  to denote the representation of  $x_{i}$ . To calculate the impact  $x_{j} \in \mathbf{x} \backslash \{x_{i}\}$  has on  $H_{\theta}(\mathbf{x} \backslash \{x_{i}\})_{i}$ , we further mask out  $x_{j}$  to obtain the second corrupted sequence  $\mathbf{x} \backslash \{x_{i}, x_{j}\}$ . Similarly,  $H_{\theta}(\mathbf{x} \backslash \{x_{i}, x_{j}\})_{i}$  denotes the new representation of token  $x_{i}$ .
We define  $f(x_{i},x_{j})$  as:
$$
f (x _ {i}, x _ {j}) = d \left(H _ {\theta} (\mathbf {x} \backslash \{x _ {i} \}) _ {i}, H _ {\theta} (\mathbf {x} \backslash \{x _ {i}, x _ {j} \}) _ {i}\right)
$$
where  $d(\mathbf{x},\mathbf{y})$  is the distance metric that captures the difference between two vectors. We experimented with two options for  $d(\mathbf{x},\mathbf{y})$ :
- Dist: Euclidean distance between  $\mathbf{x}$  and  $\mathbf{y}$
- Prob:  $d(\mathbf{x}, \mathbf{y}) = a(\mathbf{x})_{x_i} - a(\mathbf{y})_{x_i}$ ,
where  $a(\cdot)$  maps a vector into a probability distribution among the words in the vocabulary.  $a(\mathbf{x})_{x_i}$  represents the probability of predicting token  $x_{i}$  base on  $\mathbf{x}$ .
By repeating the two-stage perturbation on each pair of tokens  $x_{i}, x_{j} \in \mathbf{x}$  and calculating  $f(x_{i}, x_{j})$ , we obtain an impact matrix  $\mathcal{F}$ , where  $\mathcal{F}_{ij} \in \mathbb{R}^{T \times T}$ . Now, we can derive algorithms to extract syntactic trees from  $\mathcal{F}$  and compare them with ground-truth trees that are obtained from benchmarks. Note that BERT uses byte-pair encoding (Sennrich et al., 2016) and may split a word into multiple tokens(subwords). To evaluate our approach on word-level tasks, we make the following changes to obtain inter-word impact matrices. In each perturbation, we mask all tokens of a split-up word. The impact on a split-up word is obtained by averaging<sup>2</sup> the impacts over the split-up word's tokens. To measure the impact exerted by a split-up word, we assume the impacts given by its tokens are the same; We use the impact given by the first token for convenience.
2.3 Span Perturbation
Given the token-level perturbation above, it is straightforward to extend it to span-level perturbation. We investigate how BERT models the relations between spans, which can be phrases, clauses, or paragraphs. As a preliminary study, we investigate how well BERT captures document structures.
We model a document  $D$  as  $\mathbf{N}$  non-overlapping text spans  $D = [e_1, e_2, \ldots, e_N]$ , where each span  $e_i$  contains a sequence of tokens  $e_i = [x_1^i, x_2^i, \ldots, x_M^i]$ .
For span-level perturbation, instead of masking one token at a time, we mask an array of tokens in a span simultaneously. We obtain the span representation by averaging the representations of all the tokens the span contains. Similarly, we calculate the impact  $e_j$  has on  $e_i$  by:
$$
f (e _ {i}, e _ {j}) = d (H _ {\theta} (D \backslash \{e _ {i} \}) _ {i}, H _ {\theta} (D \backslash \{e _ {i}, e _ {j} \}) _ {i})
$$
where  $d$  is the Dist function.
In our experiments, we use the base, uncased version from (Wolf et al., 2019).
2We also experimented with other alternatives, but observe no significant difference.
Figure 1: Heatmap of the impact matrix for the sentence "For those who follow social media transitions on Capitol Hill, this will be a little different."
3 Visualization with Impact Maps
Before we discuss specific syntactic phenomena, let us first analyze some example impact matrices derived from sample sentences. We visualize an impact matrix of a sentence by displaying a heatmap. We use the term "impact map" to refer to a heatmap of an impact matrix.
Setup. We extract impact matrices by feeding BERT with 1,000 sentences from the English Parallel Universal Dependencies (PUD) treebank of the CoNLL 2017 Shared Task (Zeman et al., 2017). We follow the setup and pre-processing steps employed in pre-training BERT. An example impact map is shown in Figure 1.
Dependency. We notice that the impact map contains many stripes, which are short series of vertical/horizontal cells, typically located along the diagonal. Take the word "different" as an example (which is illustrated by the second-to-last column in the impact matrix). We observe a clear vertical stripe above the main diagonal. The interpretation is that this particular occurrence of the word "different" strongly affects the occurrences of those words before it. These strong influences are shown by the darker-colored pixels seen in the second last column of the impact map. This observation agrees with the ground-truth dependency tree, which selects "different" as the head of all remaining words in the phrase "this will be a little different." We also observe similar patterns on "transitions" and "Hill". Such correlations lead us to explore the idea of extracting dependency trees from the matrices (see Section 4.1).
Figure 2: Part of the constituency tree.
Constituency. Figure 2 shows part of the constituency tree of our example sentence generated by Stanford CoreNLP (Manning et al., 2014). In this sentence, "media" and "on" are two words that are adjacent to "transitions". From the tree, however, we see that "media" is closer to "transitions" than "on" is in terms of syntactic distance. If a model is syntactically uninformed, we would expect "media" and "on" to have comparable impacts on the prediction of "transitions", and vice versa. However, we observe a far greater impact (darker color) between "media" and "transitions" than that between "on" and "transitions". We will further support this observation with empirical experiments in Section 4.2.
Other Structures. Along the diagonal of the impact map, we see that words are grouped into four contiguous chunks that have specific intents (e.g., a noun phrase - on Capitol Hill). We also observe that the two middle chunks have relatively strong inter-chunk word impacts and thus a bonding that groups them together, forming a larger verb phrase. This observation suggests that BERT may capture the compositionality of the language.
In the following sections we quantitatively evaluate these observations.
4 Syntactic Probe
We start with two syntactic probes - dependency probe and constituency probe.
4.1 Dependency Probe
With the goal of exploring the extent dependency relations are captured in BERT, we set out to answer the following question: Can BERT outperform linguistically uninformed baselines in unsupervised dependency parsing? If so, to what extent?
We begin by using the token-level perturbed masking technique to extract an impact matrix  $\mathcal{F}$  for each sentence. We then utilize graph-based algorithms to induce a dependency tree from  $\mathcal{F}$ , and compare it against ground-truth whose annotations
are linguistically motivated.
Experiment Setup. We evaluate the induced trees on two benchmarks: (1) the PUD treebank described in Section 3. (2) the WSJ10 treebank, which contains 7,422 sentences (all less than 10 words after punctuation removal) from the Penn Treebank (PTB) (Marcus et al., 1993). Note that the original PTB does not contain dependency annotations. Thus, we convert them into Universal Dependencies using Stanford CoreNLP. We denote this set as WSJ10-U.
Next, two parsing algorithms, namely, the Eisner algorithm (1996) and Chu-Liu/Edmonds (CLE) algorithm (1965; 1967), are utilized to extract the projective and non-projective unlabeled dependency trees, respectively. Given that our impact matrices have no knowledge about the dependency root of the sentence, we use the gold root in our analysis. Introducing the gold root may artificially improve our results slightly. We thus apply this bias evenly across all baselines to ensure a fair comparison, as done in (Raganato and Tiedemann, 2018; Htut et al., 2019).
We compared our approach against the following baselines: (1) right-(left-) chain baseline, which always selects the next(previous) word as dependency head. (2) A random BERT baseline, with which we randomly initialize weights of the BERT model (Htut et al., 2019), then use our methods to induce dependency trees.
We measure model performance using Unlabeled Attachment Score (UAS). We note that UAS has been shown to be highly sensitive to annotation variations (Schwartz et al., 2011; Tsarfaty et al., 2011; Kübler et al., 2009). Therefore, it may not be a fair evaluation metric for analyzing and interpreting BERT. To reflect the real quality of the dependency structures that are retained in BERT, we also report Undirected UAS (UUAS) (Klein and Manning, 2004) and the Neutral Edge Direction (NED) scores (Schwartz et al., 2011).
Results. Tables 1 and 2 show the results of our dependency probes. From Table 1, we see that although BERT is trained without any explicit supervision from syntactic dependencies, to some extent the syntax-aware representation already exists in it. The best UAS scores it achieves (Eisner+Dist) are substantially higher than that of the random BERT baseline with respect to both WSJ10-U(+41.7) and PUD(+31.5). Moreover, the Dist method significantly outperforms the Prob
Model	Parsing UAS
WSJ10-U	PUD
Right-chain	49.5	35.0
Left-chain	20.6	10.7
Random BERT	16.9	10.2
Eisner+Dist	58.6	41.7
Eisner+Prob	52.7	34.1
CLE+Dist	51.5	33.2
Table 1: UAS results of BERT on unsupervised dependency parsing.
Model	UAS	UUAS	NED
Eisner+Dist	41.7	52.1	69.6
Right-chain	35.0	39.9	41.2
Table 2: Performance on PUD when evaluated using UAS, UUAS, and NED.
method on both datasets we evaluated. We thus use Dist as the default distance function in our later discussion. We also note that the Eisner algorithm shows a clear advantage over CLE since English sentences are mostly projective. However, our best performing method does not go much beyond the strong right-chain baseline (with gold root modified), showing that the dependency relations learned are mostly those simple and local ones.
For reference, the famous unsupervised parser - DMV (Klein and Manning, 2004) achieves a 43.2 UAS on WSJ10 with Collins (1999) conventions. Note that the DMV parser utilizes POS tags for training while ours start with the gold root. The results are therefore not directly comparable. By putting them together, however, we see potential room for improvement for current neural unsupervised dependency parsing systems in the BERT era.
From Table 2, we see that although BERT only outperforms the right-chain baseline modestly in terms of UAS, it shows significant improvements on UUAS (+12.2) and NED (+28.4). We also make similar observation with WSJ10-U. This suggests that BERT does capture interword dependencies despite that it may not totally agree with one specific human-designed governor-dependent schema. We manually inspect those discrepancies and observe that they can also be syntactically valid. For instance, consider the sen
tence "It closed on Sunday". For the phrase "on Sunday", our method selects the functional word "on" as the head while the gold-standard annotation uses a lexical head ("Sunday") $^3$ .
The above findings prove that BERT has learned its own syntax as a by-product of self-supervised training, not by directly copying any human design. However, giving the superior performance of BERT on downstream tasks, it is natural to ask if BERT is learning an empirically useful structure of language. We investigate this question in Sec 6.
4.2 Constituency Probe
We now examine the extent BERT learns about the constituent structure of sentences. We first present the algorithm for unsupervised constituent parsing, which executes in a top-down manner by recursively splitting larger constituents into smaller ones.
Top-Down Parsing. Given a sentence as a sequence of tokens  $\mathbf{x} = [x_1, \ldots, x_T]$  and the corresponding impact matrix  $\mathcal{F}$ . We start by finding the best splitting position  $k$  that will separate the sentence into constituents  $((\mathbf{x}_{< k}),(x_k,(\mathbf{x}_{>k})))$ , where  $\mathbf{x}_{< k} = [x_1, \ldots, x_{k - 1}]$ . The best splitting position ensures that each constituent has a large average impact between words within it (thus those words more likely to form a constituent) while at the same time the impact between words of different constituents are kept as small as possible (thus they are unlikely to be in the same constituent). Mathematically, we decide the best  $k$  for the constituent  $\mathbf{x} = [x_i, x_{i + 1}, \ldots, x_j]$  by the following optimization:
$$
\begin{array}{l} \underset {k} {\arg \max } \quad \mathcal {F} _ {i, \dots , k} ^ {i, \dots , k} + \mathcal {F} _ {k + 1, \dots , j} ^ {k + 1, \dots , j} \tag {1} \\ - \mathcal {F} _ {i, \dots , k} ^ {k + 1, \dots , j} - \mathcal {F} _ {k + 1, \dots , j} ^ {i, \dots , k} \\ \end{array}
$$
where  $\mathcal{F}_{i,\dots,k}^{i,\dots,k} = \frac{\sum_{a=i}^{k}\sum_{b=i}^{k}f(x_{a},x_{b})}{2(k-i)}$ . We recursively split  $(\mathbf{x}_{< k})$  and  $(\mathbf{x}_{>k})$  until only single words remain. Note that this top-down strategy is similar to that of ON-LSTM (Shen et al., 2019) and PRPN (Shen et al., 2018), but differs from them in that ON-LSTM and PRPN decide the splitting position based on a "syntactic distance vector" which is explicitly modeled by a special network component. To distinguish our approach from the others, we denote our parser as MART (MAtRix-based Top-down parser)
Experiment Setup. We follow the experiment setting in Shen et al (2019; 2018) and evaluate our method on the 7,422 sentences in WSJ10 dataset and the PTB23 dataset (the traditional PTB test set for constituency parsing).
Results. Table 3 shows the results of our constituency probes. From the table, we see that BERT outperforms most baselines on PTB23, except for the second layer of ON-LSTM. Note that all these baselines have specifically-designed architectures for the unsupervised parsing task, while BERT's knowledge about constituent formalism emerges purely from self-supervised training on unlabeled text.
It is also worth noting that recent results (Dyer et al., 2019; Li et al., 2019a) have suggested that the parsing algorithm used by ON-LSTM (PRPN) is biased towards the right-branching trees of English, leading to inflated F1 compared to unbiased parsers. To ensure a fair comparison with them, we also introduced this right-branching bias. However, our results show that our method is also robust without this bias (e.g., only 0.9 F1 drops on PTB23).
To further understand the strengths and weaknesses of each system, we analyze their accuracies by constituent tags. In Table 3, we show the accuracies of five most common tags in PTB23. We find that the success of PRPN and ON-LSTM mainly comes from the accurate identification of NP (noun phrase), which accounts for  $38.5\%$  of all constituents. For other phrase-level tags like VP (verb phrase) and PP (prepositional phrase), the accuracies of BERT are competitive. Moreover, for clause level tags, BERT significantly outplays ON-LSTM. Take SBAR (clause introduced by a subordinating conjunction) for example, BERT achieves an accuracy of  $51.9\%$ , which is about 3.4 times higher than that of ON-LSTM. One possible interpretation is that BERT is pre-trained on long contiguous sequences extracted from a document-level corpus. And the masking strategy (randomly mask  $15\%$  tokens) utilized may allow BERT to learn to model a sequence of words (might form a clause).
5 Discourse Probe
Having shown that clause-level structures are well-captured in BERT using the constituency probe, we now explore a more challenging probe - probing BERT's knowledge about the struc
<sup>3</sup>This specific choice is actually agreed with the YM (Ya-mada and Matsumoto, 2003) schema.
4170
Model	Parsing F1	Accuracy on PTB23 by Tag
WSJ10	PTB23	NP	VP	PP	S	SBAR
PRPN-LM	70.5	37.4	63.9	-	24.4	-	-
ON-LSTM 1st-layer	42.8	24.0	23.8	15.6	18.3	48.1	16.3
ON-LSTM 2nd-layer	66.8	49.4	61.4	51.9	55.4	54.2	15.4
ON-LSTM 3rd-layer	57.6	40.4	57.5	13.5	47.2	48.6	10.4
300D ST-Gumbel w/o Leaf GRU	-	25.0	18.8	-	9.9	-	-
300D RL-SPINN w/o Leaf GRU	-	13.2	24.1	-	14.2	-	-
MART	58.0	42.1	44.6	47.0	50.6	66.1	51.9
Right-Branching	56.7	39.8	25.0	71.8	42.4	74.2	68.8
Left-Branching	19.6	9.0	11.3	0.8	5.0	44.1	5.5
Table 3: Unlabeled parsing F1 results evaluated on WSJ10 and PTB23.
Model	UAS	Accuracy by distance
0	1	2	5
Right-chain	10.7	20.5	-	-	-
Left-chain	41.5	79.5	-	-	-
Random BERT	6.3	20.4	7.5	3.5	0.0
Eisner+Dist	34.2	61.6	7.3	7.6	12.8
CLE+Dist	34.4	63.8	3.3	3.5	2.6
Table 4: Performance of different discourse parser. The distance is defined as the number of EDUs between head and dependent.
ture of a document. A document contains a series of coherent text spans, which are named Elementary Discourse Units (EDUs) (Yang and Li, 2018; Polanyi, 1988). EDUs are connected to each other by discourse relations to form a document. We devise a discourse probe to investigate how well BERT captures structural correlations between EDUs. As the foundation of the probe, we extract an EDU-EDU impact matrix for each document using span-level perturbation.
Setup. We evaluate our probe on the discourse dependency corpus SciDTB (Yang and Li, 2018). We do not use the popular discourse corpora RST-DT (Carlson et al., 2003) and PDTB (Prasad et al.) because PDTB focuses on local discourse relations but ignores the whole document structure, while RST-DT introduces intermediate nodes and does not cover non-projective structures. We follow the same baseline settings and evaluation procedure in Sec 4.1, except that we remove gold root from our evaluation since we want to compare the accuracy by syntactic distances.
Results. Table 4 shows the performance of our discourse probes. We find that both Eisner and CLE achieve significantly higher UAS (+28) than the random BERT baseline. This suggests that BERT is aware of the structure of the document it is given. In particular, we observe a decent accuracy in identifying discourse relations between adjacent EDUs, perhaps due to the "next sentence prediction" task in pre-training, as pointed out in (Shi and Demberg, 2019). However, our probes fall behind the left-chain baseline, which benefits from its strong structural prior<sup>4</sup> (principal clause mostly in front of its subordinate clause). Our finding sheds some lights on BERT's success in downstream tasks that have paragraphs as input (e.g., Question Answering).
6 BERT-based Trees VS Parser-provided Trees