SlowGuess/NAACL_2022_OCR huggingface dataset:
$\Vdash$  Sort by Structure: Language Model Ranking as Dependency Probing
Max Müller-Eberstein $^{\bullet}$  and Rob van der Goot $^{\bullet}$  and Barbara Plank $^{\bullet\Delta}$
$\bullet$  Department of Computer Science, IT University of Copenhagen, Denmark
$\triangle$  Center for Information and Language Processing (CIS), LMU Munich, Germany
many@itu.dk, robv@itu.dk, bplank@cis.lmu.de
Making an informed choice of pre-trained language model (LM) is critical for performance, yet environmentally costly, and as such widely underexplored. The field of Computer Vision has begun to tackle encoder ranking, with promising forays into Natural Language Processing, however they lack coverage of linguistic tasks such as structured prediction. We propose probing to rank LMs, specifically for parsing dependencies in a given language, by measuring the degree to which labeled trees are recoverable from an LM's contextualized embeddings. Across 46 typologically and architecturally diverse LM-language pairs, our probing approach predicts the best LM choice  $79\%$  of the time using orders of magnitude less compute than training a full parser. Within this study, we identify and analyze one recently proposed decoupled LM—RemBERT—and find it strikingly contains less inherent dependency information, but often yields the best parser after full fine-tuning. Without this outlier our approach identifies the best LM in  $89\%$  of cases.
With the advent of massively pre-trained language models (LMs) in Natural Language Processing (NLP), it has become crucial for practitioners to choose the best LM encoder for their given task early on, regardless of the rest of their proposed model architecture. The greatest variation of LMs lies in the language or domain-specificity of the unlabelled data used during pre-training (with architectures often staying identical).
Typically, better expressivity is expected from language/domain-specific LMs (Gururangan et al., 2020; Dai et al., 2020) while open-domain settings necessitate high-capacity models with access to as much pre-training data as possible. This tradeoff is difficult to navigate, and given that multiple specialized LMs (or none at all) are available, practitioners often resort to an ad-hoc choice. In absence of im
mediate performance indicators, the most accurate choice could be made by training the full model using each LM candidate, however this is often infeasible and wasteful (Strubell et al., 2019).
Recently, the field of Computer Vision (CV) has attempted to tackle this problem by quantifying useful information in pre-trained image encoders as measured directly on labeled target data without fine-tuning (Nguyen et al., 2020; You et al., 2021). While first forays for applying these methods to NLP are promising, some linguistic tasks differ substantially: Structured prediction, such as parsing syntactic dependencies, is a fundamental NLP task not covered by prior encoder ranking methods due to its graphical output. Simultaneously, performance prediction in NLP has so far been studied as a function of dataset and model characteristics (Xia et al., 2020; Ye et al., 2021) and has yet to examine how to rank large pools of pre-trained LMs.
Given the closely related field of probing, in which lightweight models quantify task-specific information in pre-trained LMs, we recast its objective in the context of performance prediction and ask: How predictive is lightweight probing at choosing the best performing LM for dependency parsing? To answer this question, we contribute:
- An efficient encoder ranking method for structured prediction using dependency probing (Müller-Eberstein et al., 2022; DEPPROBE) to quantify latent syntax (Section 2).
- Experiments across 46 typologically and architecturally diverse LM + target language combinations (Section 3).<sup>1</sup>
- An in-depth analysis of the surprisingly low inherent dependency information in RemBERT (Chung et al., 2021) compared to its high fine-tuned performance (Section 4).
$^{1}$ Code at https://personads.me/x/naacl-2022-code.
Human Language Technologies, pages 1296 - 1307
Figure 1: Visualization of DEPPROBE. Relational and structural subspaces  $L$  and  $B$  are combined to extract labeled, directed trees from embeddings.
Probing pre-trained LMs is highly related to encoder ranking in CV where the ease of recoverability of class-differentiating information is key (Nguyen et al., 2020; You et al., 2021). This approach is more immediate than existing NLP performance prediction methods which rely on featurized representations of source and target data without actively ranking encoders (Xia et al., 2020; Ye et al., 2021). As most experiments in NLP are conducted using a limited set of LMs—often a single model—without strong prior motivations, we see LM ranking as a critical task on its own.
While probes for LMs come in many forms, they are generally characterized as lightweight, minimal architectures intended to solve a particular task (Hall Maudslay et al., 2020). While non-linear models such as small multi-layer perceptrons are often used (Tenney et al., 2019), there have been criticisms given that their performance highly depends on the complexity of their architecture (Hewitt and Liang, 2019; Voita and Titov, 2020). As such, we rely on linear probes alone, which have the benefit of being extremely lightweight, closely resembling existing performance prediction methods (You et al., 2021), and allow for statements about linear subspaces contained in LM latent spaces.
DEPPROBE (Müller-Eberstein et al., 2022; visualized in Figure 1) is a linear formulation for extracting fully labeled dependency trees based on the structural probe by Hewitt and Manning (2019). Given contextualized embeddings of dimensionality  $d$ , a linear transformation  $B \in \mathbb{R}^{b \times d}$  with  $b \ll d$  (typically  $b = 128$ ) maps them into a subspace in which the Euclidean distance between embeddings corresponds to the number of edges between the respective words in the gold dependency graph.
In our formulation, we supplement a linear transformation  $L \in \mathbb{R}^{l \times d}$  (with  $l =$  number of dependency relations) which maps each embedding to a
subspace in which the magnitude of each dimension corresponds to the likelihood of a word and its head being governed by a certain relation.
By computing the minimum spanning tree in  $B$  and then finding the word with the highest root likelihood in  $L$ , we can determine the directionality of all edges as pointing away from the root. All remaining edges are labeled according to the most likely non-root class in  $L$ , resulting in a fully directed and labeled dependency tree.
Note that this approach differs substantially from prior approaches which yield undirected and/or unlabeled trees (Hewitt and Manning, 2019; Kulmizev et al., 2020) or use pre-computed edges and non-linear classifiers (Tenney et al., 2019). DEPPROBE efficiently computes the full target metric (i.e. labeled attachment scores) instead of approximate alternatives (e.g. undirected, unlabeled attachment scores or tree depth correlation).
Setup We investigate the ability of DEPPROBE to select the best performing LM for dependency parsing across nine linguistically diverse treebanks from Universal Dependencies (Zeman et al., 2021; UD) which were previously chosen by Smith et al. (2018) to reflect diverse writing systems and morphological complexity (see Appendix A).
For each target language, we employ three multilingual LMs—mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), RemBERT (Chung et al., 2021)—as well as 1–3 language-specific LMs retrieved by popularity from HuggingFace's Model Hub (Wolf et al., 2020), resulting in a total of 46 LM-target pair setups (see Appendix C).
For each combination, we train a DEPPROBE to compute labeled attachment scores (LAS), hypothesizing that LMs from which trees are most accurately recoverable also perform better in a fully tuned parser. To evaluate the true downstream performance of a fully-tuned model, we further train a deep biaffine attention parser (BAP; Dozat and Manning, 2017) on each LM-target combination. Compared to full fine-tuning, DEPPROBE only optimizes the matrices  $B$  and  $L$ , resulting in the extraction of labeled trees with as few as 190k instead of 583M trainable parameters for the largest REmBERT model (details in Appendix B).
We measure the predictive power of probing for fully fine-tuned model performance using the Pearson correlation coefficient  $\rho$  as well as the weighted
Figure 2: LAS of DEPPROBE in relation to full BAP across nine language targets (dev) using language-specific and multilingual LM encoders of different architecture types (exact scores in Appendix C).
Kendall's  $\tau_w$  (Vigna, 2015). The latter metric corresponds to a correlation coefficient in  $[-1,1]$  and simultaneously defines the probability of choosing the better LM given a pair as  $\frac{\tau_w + 1}{2}$ , allowing us to quantify the overall quality of a ranking.
Results Comparing the LAS of DEPPROBE's lightweight predictions against full BAP finetuning in Figure 2, we see a clear correlation as the probe correctly predicts the difficulty of parsing languages relative to each other and also ranks models within languages closely according to their final performance. With a  $\tau_w$  of .58 between scores  $(p < 0.001)$ , this works out to DEPPROBE selecting the better performing final model given any two models  $79\%$  of the time. Additionally, LAS is slightly more predictive of final performance than unlabeled, undirected attachment scores (UUAS) with  $\tau_w = .57$  to which prior probing approaches are restricted (see Appendix C).
Given a modest  $\rho$  of .32 ( $p < 0.05$ ), we surprisingly also observe a single strong outlier to this pattern, namely the multilingual RemBERT (Chung et al., 2021) decoupled LM architecture. While DEPPROBE consistently ranks it low as it cannot extract dependency parse trees as accurately as from the BERT and RoBERTa-based architectures, RemBERT actually performs best on four out of the nine targets when fully fine-tuned in BAP. Excluding monolingual LMs, it further outperforms the other multilingual LMs in seven out of nine cases. As it is a more recent and distinctive architecture with many differences to the most commonly-used contemporary LMs, we analyze potential reasons for this discrepancy in Section 4. Excluding RemBERT as an outlier, we find sub
Figure 3: Dependency Information per RemBERT Layer via DEPPROBE's structural, relational and parsing accuracy (UUAS, RelAcc, LAS) on EN-EWT (dev).
stantially higher correlation among all other models:  $\rho = .78$  and  $\tau_w = .78$  ( $p < 0.001$ ). This means that among these models, fully fine-tuning the LM for which DEPPROBE extracts the highest scores, yields the better final performance  $89\%$  of the time.
In practice, learning DEPPROBE's linear transformations while keeping the LM frozen is multiple orders of magnitude more efficient than fully training a complex parser plus the LM's parameters. As such, linear probing offers a viable method for selecting the best encoder in absence of qualitative heuristics or intuitions. This predictive performance is furthermore achievable in minutes compared to hours and at a far lower energy budget (see Appendices B and C).
4 Probing Decoupled LMs
Considering DEPPROBE's high predictive performance across LMs with varying architecture types, languages/domains and pre-training procedures, we next investigate its limitations: Specifically, which differences in RemBERT (Chung et al., 2021) lead to it being measured as an outlier with seemingly low amounts of latent dependency information despite reaching some of the highest scores after full fine-tuning. The architecture has 32 layers and embeddings with  $d = 1152$ , compared to most models' 12 layers and  $d = 768$ . It accommodates these size and depth increases within a manageable parameter envelope by using smaller input embeddings with  $d_{\mathrm{in}} = 256$ . While choosing different  $d$  for the input and output embeddings is not possible in most prior models due to both embedding matrices being coupled, RemBERT decouples them, leading to a larger parameter budget and less overfitting on the masked language modeling pre-training task (Chung et al., 2021).
Figure 4: Per-language  $\alpha$  of RemBERT Layers for DEPPROBE across all layer weights (dark > light).
MODEL	AR	EN	FI	GRC	HE	KO	RU	SV	ZH
mBERT	65	74	65	46	69	58	68	65	58
±.08	±.09	±.35	±.14	±.23	±.18	±.31	±.12	±.17
XLM-R	60	70	66	53	60	49	57	51	51
±.14	±.08	±.18	±.19	±.20	±.08	±.34	±.24	±.53
RemBERT	58	56	52	54	52	46	49	43	39
±.12	±.22	±.15	±.18	±.05	±.14	±.04	±.08	±.24
Table 1: LAS of BAP Trained on Frozen LMs. A biaffine attention parsing head is trained on top of frozen mBERT, XLM-R and RemBERT for each of the nine target languages (± standard deviation).
Layer-wise Probing Prior probing studies have found dependency information to be concentrated around the middle layers of an LM (Hewitt and Manning, 2019; Tenney et al., 2019; Fayyaz et al., 2021). Using EN-EWT (Silveira et al., 2014), we evaluate whether this holds for RemBERT's new architecture. Figure 3 confirms that both dependency structural and relational information are most prominent around layer 17 of 32 as indicated by UUAS and relation classification accuracy (RelAcc) respectively. Combining the structural and relational information in DEPPROBE similarly leads to a peak of the LAS at the same layer while decreasing with further distance from the center.
Across all target languages, we next investigate whether probing a sum over the embeddings of all layers weighted by  $\alpha \in \mathbb{R}^{32}$  can boost extraction performance in RemBERT. The heavier weighting of middle layers by  $\alpha$ , visible in Figure 4, reaffirms a concentration of dependency information in the center. Contrasting probing work on prior models (Tenney et al., 2019; Kulmizev et al., 2020), using all layers does not increase the retrievable dependencies, with LAS differences  $\pm 1$  point. This further confirms that there is not a lack of dependency information in any specific layer, but that there is less within the encoder as a whole.
Frozen Parsing Our probing results show that linear subspaces in RemBERT contain less dependency information than prior LMs. However, DEPPROBE's parametrization is kept intentionally sim
ple and may therefore not be capturing non-linearly represented information that is useful during later fine-tuning. To evaluate this hypothesis, we train a full biaffine attention parsing head, but keep the underlying LM encoder frozen. This allows us to quantify the performance gains which come from inherent dependency information versus later task-specific fine-tuning.
Table 1 confirms our findings from DEPPROBE and shows that despite RemBERT outperforming mBERT and XLM-R when fully fine-tuned, it has substantially lower LAS across almost all languages when no full model fine-tuning is applied. This leads us to conclude that there indeed is less inherent dependency information in the newer model and that most performance gains must be occurring during task-specific full fine-tuning.
Given that DEPPROBE extracts dependency structures reliably from LM architectures with different depths and embedding dimensionalities (e.g.  $\mathrm{RoBERT}_{\mathrm{large}}$  with 24 layers and  $d = 1024$  versus  $\mathrm{RuBERT}_{\mathrm{tiny}}$  with 3 layers and  $d = 312$ ) as well as varying tokenization, optimization and pre-training data, the key difference in RemBERT appears to be embedding decoupling. The probe's linear formulation is not the limiting factor as the non-linear, biaffine attention head also produces less accurate parses when the LM's weights are frozen. Our analyses thus suggest that RemBERT's decoupled architecture contains less dependency information out-of-the-box, but follows prior patterns such as consolidating dependency information towards its middle layers and serving as strong initialization for parser training.
Lastly, RemBERT's larger number of tunable parameters compared to all other LM candidates may provide it further capacity, especially after full fine-tuning. As our probing methods are deliberately applied to the frozen representations of the encoder, it becomes especially important to consider the degree to which these embeddings may change after updating large parts of the model. Taking these limitations into account, the high correlations with respect to encoder ranking nonetheless enable a much more informed selection of LMs from a larger pool than was previously possible.
To guide practitioners in their choice of LM encoder for the structured prediction task of dependency parsing, we leveraged a lightweight, linear
DEPPROBE to quantify the latent syntactic information via the labeled attachment score. Evaluating 46 pairs of multilingual/language-specific LMs and nine typologically diverse target treebanks, we found DEPPROBE to not only be efficient in its predictions, with orders of magnitude fewer trainable parameters, but to also be accurate  $79 - 89\%$  of the time in predicting which LM will outperform another when used in a fully tuned parser. This allows for a substantially faster iteration over potential LM candidates, saving hours worth of compute in practice (Section 3).
Our experiments further revealed surprising insights on the newly proposed RemBERT architecture: While particularly effective for multilingual dependency parsing when fully fine-tuned, it contains substantially less latent dependency information relative to prior widely-used models such as mBERT and XLM-R. Among its architectural differences, we identified embedding decoupling to be the most likely contributor, while added model capacity during fine-tuning may also improve final performance. Our analyses showed that despite containing less dependency information overall, RemBERT follows prior findings such as structure and syntactic relations being consolidated towards the middle layers. Given these consistencies, performance differences between decoupled LMs may be predictable using probes, but in absence of similar multilingual LMs using decoupled embeddings this effect remains to be studied (Section 4).
Overall, the high efficiency and predictive power of ranking LM encoders via linear probing as well as the ease with which they can be analyzed—even when they encounter their limitations—offers immediate benefits to practitioners who have so far had to rely on their own intuitions when making a selection. This opens up avenues for future research by extending these methods to more tasks and LM architectures in order to enable better informed modeling decisions.
We would like to thank the NLPnorth group for insightful discussions on this work, in particular Elisa Bassignana and Mike Zhang. Thanks also to ITU's High-performance Computing team. Finally, we thank the anonymous reviewers for their helpful feedback. This research is supported by the Independent Research Fund Denmark (Danmarks Frie Forskningsfond; DFF) grant number 9063-00077B.
Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.
Pavel Blinov. 2021. RoBERTa-base Russian. https://huggingface.co/blinoff/ roberta-base-russian-v0. Accessed 4th January, 2022.
Jayeol Chun, Na-Rae Han, Jena D. Hwang, and Jinho D. Choi. 2018. Building Universal Dependency treebanks in Korean. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for Chinese BERT. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504-3514.
Xiang Dai, SarvNaz Karimi, Ben Hachey, and Cecile Paris. 2020. Cost-effective selection of pretraining data: A case study of pretraining BERT on social media. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1675-1681, Online. Association for Computational Linguistics.
David Dale. 2021. RuBERT-tiny: A small and fast BERT for Russian. https://habr.com/ru/post/562064/. Accessed 4th January, 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Hanne Eckhoff, Kristin Bech, Gerlof Bouma, Kristine Eide, Dag Haug, Odd Einar Haugen, and Marius Jøhndal. 2018. The PROIEL treebank family: a standard for early attestations of Indo-European languages. Language Resources and Evaluation, 52(1):29-65.
Mohsen Fayyaz, Ehsan Aghazadeh, Ali Modarressi, Ho-sein Mohebbi, and Mohammad Taher Pilehvar. 2021. Not all models localize linguistic knowledge in the same place: A layer-wise probing on BERToids' representations. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 375-388, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Jan Hajic, Otakar Smrz, Petr Zemánek, Petr Pajas, Jan Šnaidauf, Emanuel Beška, Jakub Krácmar, and Kamila Hassanová. 2009. Prague Arabic dependency treebank 1.0.
Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell. 2020. A tale of a probe and a parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7389-7395, Online. Association for Computational Linguistics.
Charles R. Harris, K. Jarrod Millman, Stefan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournaepau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernandez del Rio, Mark Wiebe, Pearu Peterson, Pierre Gerard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. 2020. Array programming with NumPy. Nature, 585(7825):357-362.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.