SlowGuess/ACL_2020_OCR huggingface dataset:
Ethan A. Chi, John Hewitt, and Christopher D. Manning
{ethanchi,johnhew,manning}@cs.stanford.edu
Recent work has found evidence that Multilingual BERT (mBERT), a transformer-based multilingual masked language model, is capable of zero-shot cross-lingual transfer, suggesting that some aspects of its representations are shared cross-lingually. To better understand this overlap, we extend recent work on finding syntactic trees in neural networks' internal representations to the multilingual setting. We show that subspaces of mBERT representations recover syntactic tree distances in languages other than English, and that these subspaces are approximately shared across languages. Motivated by these results, we present an unsupervised analysis method that provides evidence mBERT learns representations of syntactic dependency labels, in the form of clusters which largely agree with the Universal Dependencies taxonomy. This evidence suggests that even without explicit supervision, multilingual masked language models learn certain linguistic universals.
Past work (Liu et al., 2019; Tenney et al., 2019a,b) has found that masked language models such as BERT (Devlin et al., 2019) learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. Recently, large multilingual masked language models such as Multilingual BERT (mBERT) and XLM (Conneau and Lample, 2019; Conneau et al., 2019) have shown strong cross-lingual performance on tasks like XNLI (Lample and Conneau, 2019; Williams et al., 2018) and dependency parsing (Wu and Dredze, 2019). Much previous analysis has been motivated by a desire to explain why BERT-like models perform so well on downstream applications in the monolingual setting, which begs the question: what properties of these models make them so cross-lingually effective?
Figure 1: t-SNE visualization of head-dependent dependency pairs belonging to selected dependencies in English and French, projected into a syntactic subspace of Multilingual BERT, as learned on English syntax trees. Colors correspond to gold UD dependency type labels. Although neither mBERT nor our probe was ever trained on UD dependency labels, English and French dependencies exhibit cross-lingual clustering that largely agrees with UD dependency labels.
In this paper, we examine the extent to which Multilingual BERT learns a cross-lingual representation of syntactic structure. We extend probing methodology, in which a simple supervised model is used to predict linguistic properties from a model's representations. In a key departure from past work, we not only evaluate a probe's performance (on recreating dependency tree structure), but also use the probe as a window into understanding aspects of the representation that the probe was not trained on (i.e. dependency labels; Figure 1). In particular, we use the structural probing method of Hewitt and Manning (2019), which probes for syntactic trees by finding a linear transformation under which two words' distance in their dependency parse is approximated by the squared
5564
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5564-5577 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
distance between their model representation vectors under a linear transformation. After evaluating whether such transformations recover syntactic tree distances across languages in mBERT, we turn to analyzing the transformed vector representations themselves.
We interpret the linear transformation of the structural probe as defining a syntactic subspace (Figure 2), which intuitively may focus on syntactic aspects of the mBERT representations. Since the subspace is optimized to recreate syntactic tree distances, it has no supervision about edge labels (such as adjectival modifier or noun subject). This allows us to unsupervisedly analyze how representations of head-dependent pairs in syntactic trees cluster and qualitatively discuss how these clusters relate to linguistic notions of grammatical relations.
- We find that structural probes extract considerably more syntax from mBERT than baselines in 10 languages, extending the structural probe result to a multilingual setting.
- We demonstrate that mBERT represents some syntactic features in syntactic subspaces that overlap between languages. We find that structural probes trained on one language can recover syntax in other languages (zero-shot), demonstrating that the syntactic subspace found for each language picks up on features that BERT uses across languages.
- Representing a dependency by the difference of the head and dependent vectors in the syntactic space, we show that mBERT represents dependency clusters that largely overlap with the dependency taxonomy of Universal Dependencies (UD) (Nivre et al., 2020); see Figure 1. Our method allows for fine-grained analysis of the distinctions made by mBERT that disagree with UD, one way of moving past probing's limitation of detecting only linguistic properties we have training data for rather than properties inherent to the model.
Our analysis sheds light on the cross-lingual properties of Multilingual BERT, through both zero-shot cross-lingual structural probe experiments and novel unsupervised dependency label discovery experiments which treat the probe's syntactic subspace as an object of study. We find evidence that mBERT induces universal grammatical relations without any explicit supervision, which largely
Figure 2: The structural probe recovers syntax by finding a syntactic subspace in which all syntactic trees' distances are approximately encoded as squared  $L_{2}$  distance (Hewitt and Manning, 2019).
agree with the dependency labels of Universal Dependencies.
We present a brief overview of Hewitt and Manning (2019)'s structural probe, closely following their derivation. The method represents each dependency tree  $T$  as a distance metric where the distance between two words  $d_{T}(w_{i},w_{j})$  is the number of edges in the path between them in  $T$ . It attempts to find a single linear transformation of the model's word representation vector space under which squared distance recreates tree distance in any sentence. Formally, let  $\mathbf{h}_{1:n}^{\ell}$  be a sequence of  $n$  representations produced by a model from a sequence of  $n$  words  $w_{1:n}^{\ell}$  composing sentence  $\ell$ . Given a matrix  $B\in \mathbb{R}^{k\times m}$  which specifies the probe parameters, we define a squared distance metric  $d_B$  as the squared  $L_{2}$  distance after transformation by  $B$ :
$$
d _ {B} (\mathbf {h} _ {i} ^ {\ell}, \mathbf {h} _ {j} ^ {\ell}) = | | B \mathbf {h} _ {i} ^ {\ell} - B \mathbf {h} _ {j} ^ {\ell} | | _ {2} ^ {2}
$$
We optimize to find a  $B$  that recreates the tree distance  $d_{T^{\ell}}$  between all pairs of words  $(w_i^\ell, w_j^\ell)$  in all sentences  $s^\ell$  in the training set of a parsed corpus. Specifically, we optimize by gradient descent:
$$
\arg \min _ {B} \sum_ {\ell} \frac {1}{| s ^ {\ell} | ^ {2}} \sum_ {i, j} | d _ {T ^ {\ell}} (w _ {i} ^ {\ell}, w _ {j} ^ {\ell}) - d _ {B} (\mathbf {h} _ {i} ^ {\ell}, \mathbf {h} _ {j} ^ {\ell}) |
$$
For more details, see Hewitt and Manning (2019).
Departing from prior work, we view the probe-transformed word vectors  $B\mathbf{h}$  themselves—not just the distances between them—as objects of study.
$^{1}$ Code for reproducing our experiments is available here: https://github.com/ethanachi/multilingual-probing-visualization
The rows of  $B$  are a basis that defines a subspace of  $\mathbb{R}^m$ , which we call the syntactic subspace, and may focus only on parts of the original BERT representations. A vector  $B\mathbf{h}$  corresponds to a point in that space; the value of each dimension equals the dot product of  $\mathbf{h}$  with one of the basis vectors.
2.1 Experimental Settings
These settings apply to all experiments using the structural probe throughout this paper.
Data Multilingual BERT is pretrained on corpora in 104 languages; however, we probe the performance of the model in 11 languages (Arabic, Chinese, Czech, English, Farsi, Finnish, French, German, Indonesian, Latvian, and Spanish).<sup>3,4</sup> Specifically, we probe the model on trees encoded in the Universal Dependencies v2 formalism (Nivre et al., 2020).
Model In all our experiments, we investigate the 110M-parameter pre-trained weights of the BERT-Base, Multilingual Cased model.
- MBERTRAND: A model with the same parametrization as mBERT but no training. Specifically, all of the contextual attention layers are reinitialized from a normal distribution with the same mean and variance as the original parameters. However, the subword embeddings and positional encoding layers remain unchanged. As randomly initialized ELMo layers are a surprisingly competitive baseline for syntactic parsing (Conneau et al., 2018), we also expect this to be the case for BERT. In our experiments, we find that this baseline performs approximately equally across layers, so we draw always from Layer 7.
- LINEAR: All sentences are given an exclusively left-to-right chain dependency analysis.
Evaluation To evaluate transfer accuracy, we use both of the evaluation metrics of Hewitt and Manning (2019). That is, we report the Spearman correlation between predicted and true word pair distances (DSpr.). We also construct an undirected minimum spanning tree from said distances, and evaluate this tree on undirected, unlabeled attachment score (UUAS), the percentage of undirected edges placed correctly when compared to the gold tree.
3 Does mBERT Build a Syntactic Subspace for Each Language?
We first investigate whether mBERT builds syntactic subspaces, potentially private to each language, for a subset of the languages it was trained on; this is a prerequisite for the existence of a shared, cross-lingual syntactic subspace.
Specifically, we train the structural probe to recover tree distances in each of our eleven languages. We experiment with training syntactic probes of various ranks, as well as on embeddings from all 12 layers of mBERT.
We find that the syntactic probe recovers syntactic trees across all the languages we investigate, achieving on average an improvement of 22 points UUAS and 0.175 DSpr. over both baselines (Table 1, section IN-LANGUAGE).<sup>8</sup>
Additionally, the probe achieves significantly higher UUAS (on average, 9.3 points better on absolute performance and 6.7 points better on improvement over baseline) on Western European languages. Such languages have been shown to have better performance on recent shared task results on multilingual parsing (e.g. Zeman et al., 2018). However, we do not find a large improvement when evaluated on DSpr. (0.041 DSpr. absolute, -0.013 relative).
We find that across all languages we examine, the structural probe most effectively recovers tree structure from the 7th or 8th mBERT layer (Figure 4). Furthermore, increasing the probe maximum rank beyond approximately 64 or 128 gives
For ease of notation, we will discuss vectors  $B\mathbf{h}$  as being in the syntactic subspace, despite being in  $\mathbb{R}^k$
3When we refer to all languages, we refer to all languages in this set, not all languages that mBERT trains on.
4This list is not typologically representative of all human languages. However, we are constrained by the languages for which both large UD datasets and mBERT's pretraining are available. Nevertheless, we try to achieve a reasonable spread over language families, while also having some pairs of close languages for comparison.
<sup>6</sup>We omit a baseline that uses uncontextualized word embeddings because Hewitt and Manning (2019) found it to be a weak baseline compared to the two we use.
7Following Hewitt and Manning (2019), we evaluate only sentences of lengths 5 to 50, first average correlations for word pairs in sentences of a specific length, and then average across sentence lengths.
8 Throughout this paper, we report improvement over the stronger of our two baselines per-language.
9Here, we define Western European as Czech, English, French, German, and Spanish.
Structural Probe Results: Undirected Unlabeled Attachment Score (UUAS)
Arabic	Czech	German	English	Spanish	Farsi	Finnish	French	Indonesian	Latvian	Chinese	Average
LINEAR	57.1	45.4	42.8	41.5	44.6	52.6	50.1	46.4	55.2	47.0	44.2	47.9
MBERTRAND	49.8	57.3	55.2	57.4	55.3	43.2	54.9	61.2	53.2	53.0	41.1	52.9
IN-LANG	72.8	83.7	83.4	80.1	79.4	70.7	76.3	81.3	74.4	77.1	66.3	76.8
ΔBASELINE	15.7	26.4	28.1	22.6	24.1	18.0	21.4	20.1	19.1	24.1	22.1	22.0
SINGLETRAN	68.6	74.7	70.8	65.4	75.8	61.3	69.8	74.3	69.0	73.2	51.1	68.5
ΔBASELINE	11.5	17.4	15.6	8.0	20.4	8.7	14.9	13.1	13.8	20.2	6.9	13.7
HOLDOUT	70.4	77.8	75.1	68.9	75.5	63.3	70.7	76.4	70.8	73.7	51.3	70.4
ΔBASELINE	13.3	20.5	19.8	11.5	20.1	10.7	15.8	15.2	15.6	20.7	7.1	15.5
ALLLANS	72.0	82.5	79.6	75.9	77.6	68.2	73.0	80.3	73.1	75.1	57.8	74.1
ΔBASELINE	14.9	25.2	24.4	18.5	22.2	15.6	18.1	19.1	17.9	22.1	13.7	19.2
Structural Probe Results: Distance Spearman Correlation (DSpr.)
LINEAR	.573	.570	.533	.567	.589	.489	.564	.598	.578	.543	.493	.554
MBERTRAND	.657	.658	.672	.659	.693	.611	.621	.710	.656	.608	.590	.649
IN-LANG	.822	.845	.846	.817	.859	.813	.812	.864	.807	.798	.777	.824
ΔBASELINE	.165	.187	.174	.158	.166	.202	.191	.154	.151	.190	.187	.175
SINGLETRAN	.774	.801	.807	.773	.838	.732	.787	.836	.772	.771	.655	.777
ΔBASELINE	.117	.143	.135	.114	.145	.121	.166	.126	.117	.163	.064	.128
HOLDOUT	.779	.821	.824	.788	.838	.744	.792	.840	.776	.775	.664	.786
ΔBASELINE	.122	.163	.152	.129	.146	.133	.171	.130	.121	.166	.074	.137
ALLLANGS	.795	.839	.836	.806	.848	.777	.802	.853	.789	.783	.717	.804
ΔBASELINE	.138	.181	.165	.147	.155	.165	.181	.143	.134	.174	.127	.156
Table 1: Performance (in UUAS and DSpr.) of the structural probe trained on the following cross-lingual sources of data: the evaluation language (IN-LANG); the single other best language (SINGLETRAN); all other languages (HOLDOUT); and all languages, including the evaluation language (ALLLANGS). Note that all improvements over baseline ( $\Delta_{\text{BASELINE}}$ ) are reported against the stronger of our two baselines per-language.
Figure 3: Parse distance tree reconstruction accuracy (UUAS) for selected languages at layer 7 when the linear transformation is constrained to varying maximum dimensionality.
no further gains, implying that the syntactic subspace is a small part of the overall mBERT representation, which has dimension 768 (Figure 3).
These results closely correspond to the results found by Hewitt and Manning (2019) for an equivalently sized monolingual English model trained and evaluated on the Penn Treebank (Marcus et al., 1993), suggesting that mBERT behaves similarly to monolingual BERT in representing syntax.
Figure 4: Parse distance tree reconstruction accuracy (UUAS) on layers 1-12 for selected languages, with probe maximum rank 128.
4 Cross-Linguual Probing
4.1 Transfer Experiments
We now evaluate the extent to which Multilingual BERT's syntactic subspaces are similar across languages. To do this, we evaluate the performance of a structural probe when evaluated on a language unseen at training time. If a probe trained to predict syntax from representations in language  $i$  also predicts syntax in language  $j$ , this is evidence that mBERT's syntactic subspace for language  $i$  also encodes syntax in language  $j$ , and thus that syntax
is encoded similarly between the two languages.
Specifically, we evaluate the performance of the structural probe in the following contexts:
- Direct transfer, where we train on language  $i$  and evaluate on language  $j$ .
- Hold-one-out transfer, where we train on all languages other than  $j$  and evaluate on language  $j$ .
4.2 Joint Syntactic Subspace
Building off these cross-lingual transfer experiments, we investigate whether there exists a single joint syntactic subspace that encodes syntax in all languages, and if so, the degree to which it does so. To do so, we train a probe on the concatenation of data from all languages, evaluating it on the concatenation of validation data from all languages.
We find that mBERT's syntactic subspaces are transferable across all of the languages we examine. Specifically, transfer from the best source language (chosen post hoc per-language) achieves on average an improvement of 14 points UUAS and 0.128 DSpr. over the best baseline (Table 1, section SINGLETRAN). Additionally, our results demonstrate the existence of a cross-lingual syntactic subspace; on average, a holdout subspace trained on all languages but the evaluation language achieves an improvement of 16 points UUAS and 0.137 DSpr. over baseline, while a joint ALLLANGS subspace trained on a concatenation of data from all source languages achieves an improvement of 19 points UUAS and 0.156 DSpr. (Table 1, section HOLDOUT, ALLLANGS).
Furthermore, for most languages, syntactic information embedded in the post hoc best crosslingual subspace accounts for  $62.3\%$  of the total possible improvement in UUAS  $(73.1\%)$  DSpr.) in recovering syntactic trees over the baseline (as represented by in-language supervision). Holdout transfer represents on average  $70.5\%$  of improvement in UUAS  $(79\%)$  DSpr.) over the best baseline, while evaluating on a joint syntactic subspace accounts for  $88\%$  of improvement in UUAS  $(89\%)$  DSpr.). These results demonstrate the degree to which the cross-lingual syntactic space represents syntax cross-lingually.
4.4 Subspace Similarity
Our experiments attempt to evaluate syntactic overlap through zero-shot evaluation of structural probes. In an effort to measure more directly the degree to which the syntactic subspaces of mBERT overlap, we calculate the average principal angle between the subspaces parametrized by each language we evaluate, to test the hypothesis that syntactic subspaces which are closer in angle have closer syntactic properties (Table 4).
We evaluate this hypothesis by asking whether closer subspaces (as measured by lower average principal angle) correlate with better cross-lingual transfer performance. For each language  $i$ , we first compute an ordering of all other languages  $j$  by increasing probing transfer performance trained on  $j$  and evaluated on  $i$ . We then compute the Spearman correlation between this ordering and the ordering given by decreasing subspace angle. Averaged across all languages, the Spearman correlation is 0.78 with UUAS, and 0.82 with DSpr., showing that transfer probe performance is substantially correlated with subspace similarity.
4.5 Extrapolation Testing
To get a finer-grained understanding of how syntax is shared cross-lingually, we aim to understand whether less common syntactic features are embedded in the same cross-lingual space as syntactic features common to all languages. To this end, we examine two syntactic relations—pronominal and postnominal adjectives—which appear in some of our languages but not others. We train syntactic probes to learn a subspace on languages that primarily only use one ordering (i.e. majority class is greater than  $95\%$  of all adjectives), then evaluate their UUAS score solely on adjectives of the other ordering. Specifically, we evaluate on French, which has a mix  $(69.8\%$  prenominal) of both orderings, in the hope that evaluating both orderings in the same language may help correct for biases in pairwise language similarity. Since the evaluation ordering is out-of-domain for the probe, predicting evaluation-order dependencies successfully suggests that the learned subspace is capable of generalizing between both kinds of adjectives.
We find that for both categories of languages, accuracy does not differ significantly on either prenominal or postnominal adjectives. Specifi
10For full results, consult Appendix Table 1.
<sup>11</sup>https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.subspaceAngles.html
5568
Language	Prenom.	Postnom.	% data prenom.
de	0.932	0.900	100.0%
zh	0.801	0.826	100.0%
lv	0.752	0.811	99.7%
en	0.906	0.898	99.1%
fi	0.834	0.840	98.5%
cz	0.830	0.894	95.4%
fa	0.873	0.882	9.6%
id	0.891	0.893	4.9%
ar	0.834	0.870	0.1%
Average pre:	0.843	0.862
Average post:	0.866	0.881
Table 2: Performance of syntactic spaces trained on various languages on recovering prenominal and postnominal French noun-adjective edges.
cally, for both primarily-prenominal and primarily-postnominal training languages, postnominal adjectives score on average approximately 2 points better than prenominal adjectives (Table 2).
5 mBERT Dependency Clusters Capture Universal Grammatical Relations
Given the previous evidence that mBERT shares syntactic representations cross-lingually, we aim to more qualitatively examine the nature of syntactic dependencies in syntactic subspaces. Let  $\mathcal{D}$  be a dataset of parsed sentences, and the linear transformation  $B\in \mathbb{R}^{k\times m}$  define a  $k$  -dimensional syntactic subspace. For every non-root word and hence syntactic dependency in  $\mathcal{D}$  (since every word is a dependent of some other word or an added ROOT symbol), we calculate the  $k$  -dimensional head-dependent vector between the head and the dependent after projection by  $B$ . Specifically, for all head-dependent pairs  $(w_{\mathrm{head}},w_{\mathrm{dep}})$ , we compute  $v_{\mathrm{diff}} = B(\mathbf{h}_{\mathrm{head}} - \mathbf{h}_{\mathrm{dep}})$ . We then visualize all differences over all sentences in two dimensions using t-SNE (van der Maaten and Hinton, 2008).
As with multilingual probing, one can visualize head-dependent vectors in several ways; we present the following experiments:
- dependencies from one language, projected into a different language's space (Figure 1)
- dependencies from one language, projected into a holdout syntactic space trained on all other languages (Figure 5)
Figure 5: t-SNE visualization of syntactic differences in Spanish projected into a holdout subspace (learned by a probe trained to recover syntax trees in languages other than Spanish). Despite never seeing a Spanish sentence during probe training, the subspace captures a surprisingly fine-grained view of Spanish dependencies.
- dependencies from all languages, projected into a joint syntactic space trained on all languages (Figure 6)
For all these experiments, we project into 32-dimensional syntactic spaces. $^{12}$  Additionally, we expose a web interface for visualization in our GitHub repository. $^{13}$
When projected into a syntactic subspace determined by a structural probe, we find that difference vectors separate into clusters reflecting linguistic characteristics of the dependencies. The cluster identities largely overlap with (but do not exactly agree with) dependency labels as defined by Universal Dependencies (Figure 6). Additionally, the clusters found by mBERT are highly multilingual. When dependencies from several languages are projected into the same syntactic subspace, whether trained monolingually or cross-lingually, we find that dependencies of the same label share the same cluster (e.g. Figure 1, which presents both English
<sup>12</sup>We reduce the dimensionality of the subspaces here as compared to our previous experiments to match t-SNE suggestions and more aggressively filter non-syntactic information.
$^{13}$ https://github.com/ethanachi/multilingual-probing-visualization/blob/master/visualization.md
5569
Figure 6: t-SNE visualization of 100,000 syntactic difference vectors projected into the cross-lingual syntactic subspace of Multilingual BERT. We exclude punct and visualize the top 11 dependencies remaining, which are collectively responsible for  $79.36\%$  of the dependencies in our dataset. Clusters of interest highlighted in yellow; linguistically interesting clusters labeled.
Example sentences (trimmed for clarity). Heads in **bold**; dependents in **bold italic**.
(b) Postnominal adjectives	fr id fa	Le gaz développpe ses applications domestiques. Film lain yang menerima penghargaan istimewa.
(c) Genitives	en es lv	The assortment of customers adds entertainment. Con la recuperación de la democracia y las libertades Svešiniece piecēlās, atvadijās no vecā vīra
(j) Definite articles	en fr de	The value of the highest bid Merak est une ville d'Indonésie sur la côte occidentale. Selfst mitten in der Woche war das Lokal gut beschucht.