SlowGuess/ACL_2020_OCR huggingface dataset:
and French syntactic difference vectors projected into an English subspace).
5.4 Finer-Grained Analysis
Visualizing syntactic differences in the syntactic space provides a surprisingly nuanced view of the native distinctions made by mBERT. In Figure 6, these differences are colored by gold UD dependency labels. A brief summary is as follows:
Adjectives Universal Dependencies categorizes all adjectival noun modifiers under the amod relation. However, we find that mBERT splits adjectives into two groups: prenominal adjectives in cluster (b) (e.g., Chinese 獨特的地理) and postnominal adjectives in cluster (u) (e.g., French applications domestiques).
Nominal arguments mBERT maintains the UD distinction between subject (nsubj) and object (obj). Indirect objects (iobj) cluster with direct objects. Interestingly, mBERT generally groups adjunct arguments (obl) with nsubj if near the beginning of a sentence and obj otherwise.
Relative clauses In the languages in our dataset, there are two major ways of forming relative clauses. Relative pronouns (e.g., English the man who is hungry are classed by Universal Dependencies as being an nsubj dependent, while subordinating markers (e.g., English I know that she saw me) are classed as the dependent of a mark relation. However, mBERT groups both of these relations together, clustering them distinctly from most nsubj and mark relations.
Negatives Negative adverbial modifiers (English not, Farsi  $\mathcal{L}$  Chinese) are not clustered with other adverbial syntactic relations (advmod), but form their own group (h).14
Determiners The linguistic category of determiners (det) is split into definite articles (i), indefinite articles (e), possessives (f), and demonstratives (g). Sentence-initial definite articles (k) cluster separately from other definite articles (j).
Expletive subjects Just as in UD, with the separate relation expl, expletive subjects, or third-person pronouns with no syntactic meaning (e.g. English It is cold, French Il baudrait, Indonesian Yang.), cluster separately (k) from other nsubj relations (small cluster in the bottom left).
Overall, mBERT draws slightly different distinctions from Universal Dependencies. Although some are more fine-grained than UD, others appear to be more influenced by word order, separating relations that most linguists would group together. Still others are valid linguistic distinctions not distinguished by the UD standard.
5.5 Discussion
Previous work has found that it is possible to recover dependency labels from mBERT embeddings, in the form of very high accuracy on dependency label probes (Liu et al., 2019; Tenney et al., 2019b). However, although we know that dependency label probes are able to use supervision to map from mBERT's representations to UD dependency labels, this does not provide full insight into the nature of (or existence of) latent dependency label structure in mBERT. By contrast, in the structural probe,  $B$  is optimized such that  $\| v_{\mathrm{diff}} \|_2 \approx 1$ , but no supervision as to dependency label is given. The contribution of our method is thus to provide a view into mBERT's "own" dependency label representation. In Appendix A, Figure 8, we provide a similar visualization as applied to MBERTRAND, finding much less cluster coherence.
5.6 Probing as a window into representations
Our head-dependent vector visualization uses a supervised probe, but its objects of study are properties of the representation other than those relating to the probe supervision signal. Because the probe
never sees supervision on the task we visualize for, the visualized behavior cannot be the result of the probe memorizing the task, a problem in probing methodology (Hewitt and Liang, 2019). Instead, it is an example of using probe supervision to focus in on aspects that may be drowned out in the original representation. However, the probe's linear transformation may not pick up on aspects that are of causal influence to the model.
Cross-lingual embedding alignment Lample et al. (2018) find that independently trained monolingual word embedding spaces in ELMo are isometric under rotation. Similarly, Schuster et al. (2019) and Wang et al. (2019) geometrically align contextualized word embeddings trained independently. Wu et al. (2019) find that cross-lingual transfer in mBERT is possible even without shared vocabulary tokens, which they attribute to this isometricity. In concurrent work, Cao et al. (2020) demonstrate that mBERT embeddings of similar words in similar sentences across languages are approximately aligned already, suggesting that mBERT also aligns semantics across languages. K et al. (2020) demonstrate that strong cross-lingual transfer is possible without any word piece overlap at all.
Analysis with the structural probe In a monolingual study, Reif et al. (2019) also use the structural probe of Hewitt and Manning (2019) as a tool for understanding the syntax of BERT. They plot the words of individual sentences in a 2-dimensional PCA projection of the structural probe distances, for a geometric visualization of individual syntax trees. Further, they find that distances in the mBERT space separate clusters of word senses for the same word type.
Understanding representations Pires et al. (2019) find that cross-lingual BERT representations share a common subspace representing useful linguistic information. Libovický et al. (2019) find that mBERT representations are composed of a language-specific component and a language-neutral component. Both Libovický et al. (2019) and Kudugunta et al. (2019) perform SVCCA on LM representations extracted from mBERT and a massively multilingual transformer-based NMT model, finding language family-like clusters.
14Stanford Dependencies and Universal Dependencies v1 had a separate neg dependency, but it was eliminated in UDv2.
5571
Li and Eisner (2019) present a study in syntactically motivated dimensionality reduction; they find that after being passed through an information bottleneck and dimensionality reduction via t-SNE, ELMo representations cluster naturally by UD part of speech tags. Unlike our syntactic dimensionality reduction process, the information bottleneck is directly supervised on POS tags, whereas our process receives no linguistic supervision other than unlabeled tree structure. In addition, the reduction process, a feed-forward neural network, is more complex than our linear transformation.
Singh et al. (2019) evaluate the similarity of mBERT representations using Canonical Correlation Analysis (CCA), finding that overlap among subword tokens accounts for much of the representational similarity of mBERT. However, they analyze cross-lingual overlap across all components of the mBERT representation, whereas we evaluate solely the overlap of syntactic subspaces. Since syntactic subspaces are at most a small part of the total BERT space, these are not necessarily mutually contradictory with our results. In concurrent work, Michael et al. (2020) also extend probing methodology, extracting latent ontologies from contextual representations without direct supervision.
Language models trained on large amounts of text have been shown to develop surprising emergent properties; of particular interest is the emergence of non-trivial, easily accessible linguistic properties seemingly far removed from the training objective. For example, it would be a reasonable strategy for mBERT to share little representation space between languages, effectively learning a private model for each language and avoiding destructive interference. Instead, our transfer experiments provide evidence that at a syntactic level, mBERT shares portions of its representation space between languages. Perhaps more surprisingly, we find evidence for fine-grained, cross-lingual syntactic distinctions in these representations. Even though our method for identifying these distinctions lacks dependency label supervision, we still identify that mBERT has a cross-linguistic clustering of grammatical relations that qualitatively overlaps considerably with the Universal Dependencies formalism.
The UUAS metric We note that the UUAS metric alone is insufficient for evaluating the accuracy of the structural probe. While the probe is opti
mized to directly recreate parse distances, (that is,  $d_B(\mathbf{h}_i^\ell ,\mathbf{h}_j^\ell)\approx d_T^\ell (w_i^\ell ,w_j^\ell))$  a perfect UUAS score under the minimum spanning tree construction can be achieved by ensuring that  $d_B(\mathbf{h}_i^\ell ,\mathbf{h}_j^\ell)$  is small if there is an edge between  $w_{i}^{\ell}$  and  $w_{j}^{\ell}$ , and large otherwise, instead of accurately recreating distances between words connected by longer paths. By evaluating Spearman correlation between all pairs of words, one directly evaluates the extent to which the ordering of words  $j$  by distance to each word  $i$  is correctly predicted, a key notion of the geometric interpretation of the structural probe. See Maudslay et al. (2020) for further discussion.
Limitations Our methods are unable to tease apart, for all pairs of languages, whether transfer performance is caused by subword overlap (Singh et al., 2019) or by a more fundamental sharing of parameters, though we do note that language pairs with minimal subword overlap do exhibit nonzero transfer, both in our experiments and in others (K et al., 2020). Moreover, while we quantitatively evaluate cross-lingual transfer in recovering dependency distances, we only conduct a qualitative study in the unsupervised emergence of dependency labels via t-SNE. Future work could extend this analysis to include quantitative results on the extent of agreement with UD. We acknowledge as well issues in interpreting t-SNE plots (Wattenberg et al., 2016), and include multiple plots with various hyperparameter settings to hedge against this confounder in Figure 11.
Future work should explore other multilingual models like XLM and XLM-RoBERTa (Lample and Conneau, 2019) and attempt to come to an understanding of the extent to which the properties we've discovered have causal implications for the decisions made by the model, a claim our methods cannot support.
We would like to thank Erik Jones, Sebastian Schuster, and Chris Donahue for helpful feedback and suggestions. We would also like to thank the anonymous reviewers and area chair Adina Williams for their helpful comments on our draft.
5572
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In International Conference on Learning Representations.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2126-2136.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7059–7069. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan First. 2019. Investigating multilingual NMT representations at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291.
Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In International Conference on Learning Representations.
Xiang Lisa Li and Jason Eisner. 2019. Specializing word embeddings (for parsing) by information bottleneck. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Jindrich Libovický, Rudolf Rosa, and Alexander Fraser. 2019. How language-neutral is multilingual bert? arXiv preprint arXiv:1911.03310.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell. 2020. A tale of a probe and a parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Julian Michael, Jan A. Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies in contextual representations. arXiv preprint arXiv:2004.14513.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth International Conference on Language Resources and Evaluation.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8592-8600.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. arXiv preprint arXiv:1902.09492.
5573
Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019).
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
L.J.P. van der Maaten and G.E. Hinton. 2008. Visualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9:2579-2605.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Martin Wattenberg, Fernanda Viégas, and Ian Johnson. 2016. How to use t-sne effectively. Distill, 1(10):e2.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112-1122. Association for Computational Linguistics.
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Emerging cross-lingual structure in pretrained language models. arXiv preprint arXiv:1911.01464.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844.
Daniel Zeman, Jan Hajic, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1-21.
A Additional Syntactic Difference Visualizations
A.1 Visualization of All Relations
In our t-SNE visualization of syntactic difference vectors projected into the cross-lingual syntactic subspace of Multilingual BERT (Figure 6), we only visualize the top 11 relations, excluding punct. This represents  $79.36\%$  of the dependencies in our dataset. In Figure 7, we visualize all 36 relations in the dataset.
Figure 7: t-SNE visualization of dependency head-dependent pairs projected into the cross-lingual syntactic subspace of Multilingual BERT. Colors correspond to gold UD dependency type labels, which are unlabeled given that there are 43 in this visualization.
A.2 Visualization with Randomly- Initialized Baseline
In Figure 8, we present a visualization akin to Figure 1; however, both the head-dependency representations, as well as the syntactic subspace, are derived from MBERTRAND. Clusters around the edges of the figure are primarily type-based (e.g. one cluster for the word for and another for pour), and there is insignificant overlap between clusters with parallel syntactic functions from different languages.
B Alternative Dimensionality Reduction Strategies
In an effort to confirm the level of clarity of the clusters of dependency types which emerge from
5574
Figure 8: t-SNE visualization of head-dependent dependency pairs belonging to selected dependencies in English and French, projected into a syntactic subspace of MBERTRAND, as learned on English syntax trees. Colors correspond to gold UD dependency type labels.
syntactic difference vectors, we examine simpler strategies for dimensionality reduction.
B.1 PCA for Visualization Reduction
We project difference vectors as previously into a 32-dimensional syntactic subspace. However, we visualize in 2 dimensions using PCA instead of t-SNE. There are no significant trends evident.
Figure 9: Syntactic difference vectors visualized after dimensionality reduction with PCA, instead of t-SNE, colored by UD dependency types. There are no significant trends evident.
B.2 PCA for Dimensionality Reduction
Instead of projecting difference vectors into our syntactic subspace, we first reduce them to a 32-
dimensional representation using PCA, $^{15}$  then reduce to 2 dimensions using t-SNE as previously.
We find that projected under PCA, syntactic difference vectors still cluster into major groups, and major trends are still evident (Figure 10). In addition, many finer-grained distinctions are still apparent (e.g. the division between common nouns and pronouns). However, in some cases, the clusters are motivated less by syntax and more by semantics or language identities. For example:
- The nsubj and obj clusters overlap, unlike our syntactically-projected visualization, where there is clearer separation.
- Postnominal adjectives, which form a single coherent cluster under our original visualization scheme, are split into several different clusters, each primarily composed of words from one specific language.
- There are several small monolingual clusters without any common syntactic meaning, mainly composed of languages parsed more poorly by BERT (i.e. Chinese, Arabic, Farsi, Indonesian).
Figure 10: t-SNE visualization of syntactic differences in all languages we study, projected to 32 dimensions using PCA.
C Additional Experiment Settings
C.1 Pairwise Transfer
We present full pairwise transfer results in Table 3. Each experiment was run 3 times with different random seeds; experiment settings with range in
This is of equal dimensionality to our syntactic subspace.
Structural Probe Results: Undirected Unlabeled Attachment Score (UUAS)
Tgt \Src	ar	cz	de	en	es	fa	fi	fr	id	lv	zh	linear	rand	holdout	all
ar	72.7	68.6	66.6	65.3	67.5	64.0	60.8	68.1	65.3	60.1	53.4	57.1	49.8	70.4	72.0
cz	57.5*	83.6	74.7	72.6	71.1	63.5	68.9	71.5	62.4	71.0	58.0	45.4	57.3	77.8	82.5
de	49.3	70.2	83.5	70.8	68.2	58.7	61.1	70.6	56.9*	62.0	52.0*	42.8	55.2	75.1	79.6
en	47.2	61.2	65.0	79.8	63.9	50.8	55.3	65.4	54.5	54.0	50.5	41.5	57.4	68.9	75.9
es	52.0	67.2	69.8	69.4	79.7	56.9	56.8	75.8	61.0	55.6	49.2	44.6	55.3	75.5	77.6
fa	51.7	61.3	60.3	57.0	57.8	70.8	53.7	59.7	56.5	53.1	49.7	52.6	43.2	63.3	68.2
fi	55.5	69.8	68.4	66.6	66.0	60.2	76.5	66.0	61.2	68.2	59.2	50.1	54.9	70.7	73.0
fr	50.8*	67.8	73.0	70.0	74.3	56.9	55.9	84.0	60.9	55.1	49.6	46.4	61.2	76.4	80.3
id	57.1	66.3	67.4	63.6	67.0	61.0	59.2	69.0	74.8	57.5	54.6	55.2	53.2	70.8	73.1
lv	56.9*	73.2	69.2	69.1	67.0	61.5	70.8	66.7	61.1	77.0	60.7	47.0	53.0	73.7	75.1
zh	41.2*	49.7	49.6	51.1	47.3	42.7*	48.1	47.9	44.5*	47.2	65.7	44.2	41.1	51.3	57.8
Structural Probe Results: Distance Spearman Correlation (DSpr.)
Tgt \\Src	ar	cz	de	en	es	fa	fi	fr	id	lv	zh	linear	rand	holdout	all
ar	.822	.772	.746	.744	.774	.730	.723	.770	.750	.722	.640	.573	.657	.779	.795
cz	.730	.845	.799	.781	.801	.741	.782	.796	.745	.791	.656	.570	.658	.821	.839
de	.690	.807	.846	.792	.792	.736	.767	.796	.723	.765	.652*	.533	.672	.824	.836
en	.687	.765	.764	.817	.770	.696	.732	.773	.720	.725	.655	.567	.659	.788	.806
es	.745	.821	.812	.806	.859	.741	.775	.838	.777	.774	.669	.589	.693	.838	.848
fa	.661	.732	.724	.706	.705	.813	.683	.714	.686	.684	.629	.489	.611	.744	.777
fi	.682*	.787	.771	.756	.764	.712	.812	.762	.715	.781	.658	.564	.621	.792	.802
fr	.731*	.810	.816	.806	.836	.738	.767	.864	.776	.760	.674	.598	.710	.840	.853
id	.715	.757	.752	.739	.765	.718	.714	.772	.807	.704	.657	.578	.656	.776	.789
lv	.681	.771	.746	.737	.745	.699	.763	.740	.698	.798	.644	.543	.608	.775	.783
zh	.538*	.655	.644	.644	.633	.593*	.652	.638	.584*	.639	.777	.493	.590	.664	.717