What does Parameter-free Probing Really Uncover? (2024)

Abstract: Supervised approaches to probing large language models (LLMs) have been criticized of using pre-defined theory-laden target labels. As an alternative, parameter-free probing constructs structural representations bottom-up via information derived from the LLM alone. This has been suggested to capture a genuine “LLMinternal grammar”. However, its relation to familiar linguistic formalisms remains unclear. I extend prior work on a parameter-free probing technique called perturbed masking applied to BERT, by comparing its results to the Universal Dependencies (UD) formalism for English. The results highlight several major discrepancies between BERT and UD, which lack correlates in linguistic theory. This raises the question of whether human grammar is the correct analogy to interpret BERT in the first place.

heading: 1 Introduction
text: Probing large language models (LLMs) consists in mapping their internal states to linguistic classes or relations (Rogers et al., 2020; Belinkov, 2022). Most methods use supervised learning for training a probe to predict pre-determined labels (Hewitt and Manning, 2019; Tenney et al., 2019; Kuznetsov and Gurevych, 2020; Manning et al., 2020; Lasri et al., 2022). However, critics have deemed this insufficient for determining whether LLMs actually represent linguistic structures (Kulmizev and Nivre, 2022; Buder-Gröndahl, 2023). For representation proper, the labels should not only be predictable from the LLM; they should somehow capture its internal architecture on a high level of abstraction.
A possible way forward is to use parameter-free probing, which shuns separate probing classifiers by extracting structural information directly from the LLM (Clark et al., 2019; Mareček and Rosa, 2019; Wu et al., 2020). As a bottom-up approach, this has been interpreted as uncovering the grammar intrinsic to the LLM without relying on a priori presumptions derived from linguistic theory.
In this paper, I focus on a parameter-free probe called perturbed masking, originally presented and applied to BERT by Wu et al. (2020). While it has received criticism for underwhelming results compared to gold-standard parses (Niu et al., 2022), this overlooks its main goal of uncovering BERT’s inherent syntax – which may well deviate from linguistic theory (Wu et al., 2020, 4173). Such deviations do not call for discarding it; instead, they provide insight into how BERT’s architecture can differ from common linguistic assumptions.
I compare dependency graphs derived from BERT to the Universal Dependencies (UD) annotation for English, and uncover major discrepancies related to verbal argument structure, noun phrase structure, modifiers, and prepositions. In particular, BERT treats the root (in UD’s annotation) as a head far more often than UD. This effect of being “attracted by the root” is especially strong in recursive embeddings, but also extends beyond these.
Moreover, BERT’s behavior tends to resist linguistic explanation. For example, despite major disagreements within linguistic theory, argument structure is ubiquitously treated as clause-bound: no feasible analysis assimilates embedded clause arguments to main clause arguments. Yet, the BERT-parse regularly does exactly this. Indeed, the only cases where BERT’s deviations from UD have a salient linguistic interpretation concern prepositions and some possessive constructions, where dependent-head relations are flipped.
The results thus point to the same direction as critiques of supervised probing: the assumption that BERT represents grammar in line with familiar linguistic formalisms lacks proper support. When this is not built directly into the experiment design (via pre-determined target labels), probing reveals fundamental disparities between BERT and commonly accepted syntactic principles. We are thus prompted to question whether human grammar is an appropriate analogy for BERT after all.
327

heading: 2 Methodology
text: I describe the parameter-free probing technique investigated (Section 2.1), the dataset (Section 2.2), and the experiment pipeline (Section 2.3).

heading: 2.1 Perturbed masking
text: Parameter-free probing aims to construct linguistic information directly from the LLM without separate training. Wu et al. (2020) present a prominent techique called perturbed masking, with which they aim to find “the ‘natural’ syntax inherent in BERT” (p. 4173) by utilizing an independently motivated relation of impact between tokens. I replicated their original setup,1 which uses the bert-base-uncased model presented in Wolf et al. (2020).
As input, BERT takes a sequence of tokens x = [x1, ..., xn]. It maps each token xi to a contextual representation Hθ(x)i, where the influence of each token xj ∈ x arises via Transformer attention (Vaswani et al., 2017) based on model parameters θ. For perturbed masking, Wu et al. (2020) first mask token xi, giving x\{xi}. They then also mask token xj , giving x\{xi, xj}. The impact of xj to the representation of xi is now measured as follows, where d is Euclidean distance:2
f(xi, xj) = d(Hθ(x\{xi})i, Hθ(x\{xi, xj})i) Impacts between all token pairs are collected into an impact matrix, which is given as input to an algorithm that constructs a directed dependency graph using the Eisner algorithm (Eisner, 1996).3 The intuitive idea is that heads have the highest impact on their dependents in the matrix.

heading: 2.2 Data
text: Following Wu et al. (2020), I used the English Parallel Universal Dependencies (PUD) dataset (Zeman et al., 2017). Consisting of 1000 sentences of which I discarded seven (see Appendix A), it covers 21047 UD-annotated tokens.

heading: 2.3 Experiments
text: UD assigns each word a head and a dependency relation type (deprel), as exemplified below:4
1https://github.com/LividWo/ Perturbed-Masking#dependency
2Wu et al. (2020) report superior performance to Euclidean distance compared to the difference between probability distributions across targets.
3Wu et al. (2020) also experimented with phrase-structures, but the present setup requires dependency graphs to obtain deprel labels (Section 2.3). See Niu et al. (2022) on phrasestructures generated via perturbed masking.
4All examples are taken from the PUD dataset (shortened).
(1)
then the commercial ends
advmod
det nsubj
The arrow is read as marking a head-dependent relation (in this direction). The root is its own head, and is typically the main verb. The BERT-parse of the same sentence maps all tokens to the root ends:
(2)
then the commercial ends
Here, UD and BERT differ in which head they assign to the determiner the. I denote this by marking the UD-assigned head-dependent relation above and the BERT-assigned relation below:
(3)
then the commercial ends BERT
UD
The challenge in interpreting BERT-parses is that they only give head-dependent relations, not deprels. We thus need external deprels as the theoretical basis of comparing BERT and UD. For this, I use UD-annotations as follows:
Dep(x): deprel assigned to x by UD HeadUD(x): head assigned to x by UD HeadBERT (x): head assigned to x by BERT HU (x) = Dep(HeadUD(x)) HB(x) = Dep(HeadBERT (x))
That is, I compare UD- and BERT-assigned heads in terms of their UD-deprels. These values for the determiner in the example above are:
Dep(the) = det HeadUD(the) = commercial HeadBERT (the) = ends HU (the) = Dep(commercial) = nsubj HB(the) = Dep(ends) = root
Note that, since Dep is derived from UD, HB should not be read as directly describing how BERT treats the head. Instead, it describes how UD would treat the head assigned by BERT.
By classifying discrepancies between BERT and UD, I assess their prevalence and nature in the PUD data. I focus on four phenomena: argument structure, noun phrase (NP) structure, adjective/adverb modifiers, and prepositional phrases (PPs). Sourcecode for the experiments is openly available.5
5https://github.com/tombgro/ parameter-free-probing

heading: 3 Results
text: I replicated the original results of Wu et al. (2020) with their best setup on the PUD data,6 and investigated shifts between BERT and UD in terms of Dep, HU , and HB . Section 3.1 presents general findings, Sections 3.2–3.5 cover linguistic details, and Appendix B provides the raw data.

heading: 3.1 General findings
text: Of all 21047 tokens, 58% were subject to a headdependent shift between UD and BERT. Nearly all Dep-types were involved here, and a clear majority (74%) had a shift ratio over 50%. Clearly the most common HB was root; i.e. shifts typically involved BERT assigning a head which was the root in the UD-parse. This covered 35% of all shifts.
Wu et al. (2020, 4169) suggest that BERT mostly learns local dependencies. To assess this, we calculated dependent-head distances from both parses, and obtained contrasting results: the average is higher in BERT (3.66) than in UD (3.52). Locality thus does not explain the discrepancies. A likely explanation for the increased average dependent-head distance in BERT is its tendency to over-assign the root as a head. As covered in upcoming sections, this can lead to longer dependent-head distances in cases like embedded clauses, where the original UD-head is closer to its dependent than the root.

heading: 3.2 Argument structure
text: Table 1 collects shifts per Dep–HU pair for active and passive clause subjects (nsubj, nsubj:pass) and direct objects (obj).7
In arguments of the root, BERT and UD mostly overlap with shift ratios of 15% − 29%. However, with embedded clauses (ccomp, xcomp, conj, acl:relcl), BERT regularly continues to assign arguments to the root, with far higher shift ratios (64% − 94%) and root as the most common HB . An example is shown below, where BERT assigns the main verb as the head of an embedded subject:
(4)
that ’s not what we need BERT
UD
The BERT-parse thus seems to shun recursion, preferring the root even for embedded arguments.
6 This gives the Unlabeled Attachment Score (UAS) of 41.7, the Undirected UAS (UUAS) of 52.1, and the Neutral Edge Direction (NED) score of 69.6.
7Tables 1–4 contain shifts with the minimum count of 20. “Ratio” denotes the frequency of shifts for each Dep–HU pair.
The same pattern also repeats for objects:
(5) projects include extending the district
BERT
UD
While the explanation of this behavior is not fully clear, in general it shows that the root has an especially high impact for determining the contextual embeddings of other words. One salient possibility is that this arises because the root is usually a main clause verb, which has central influence on both grammatical matters (such as inflection or valency) and semantic matters (such as the possible semantic classes of arguments). Hence, when BERT is pre-trained via masked-token prediction (Devlin et al., 2019), attending to the main clause verb is likely to give useful information pertaining to many masked tokens. A general high impact for the root would follow, in line with these findings.

heading: 3.3 Noun phrase structure
text: Table 2 lists NP-related shifts for three variants of Dep: determiners (det), possessors (nmod:poss), and numerals (nummod). Some of these shifts are grammatically salient: for instance, UD treats the possessor as headed by the possessed noun, but BERT often takes it to be headed by the clitic ’s:
(6)
Clinton ’s large bank account
UD
BERT
However, many cases are linguistically incoherent. For example, BERT sometimes takes possessors to modify a preposition rather than a noun:
(7)
for his review BERT
UD
As usual, BERT also regularly assigns the root as the head, as for the determiner (a) shown here:
(8)
there was a time BERT
UD
In principle, the DP-analysis in formal linguistics treats determiners as noun phrase heads (Abney, 1987), and might initially justify taking the determiner to head the object (a time). However, this would require the noun (time) to be headed by the determiner, but instead it is headed by the root as well. BERT thus does not implement the DP-analysis; the determiner is simply attracted by the root. The same occurs for numeral modifiers:
(9)
300,000 people (...) were involved BERT
UD
Since possessors, determiners, and numerals are the sine qua non of NP-arguments/modifiers, these results illustrate a drastic shift between BERT and widely shared syntactic assumptions about NPs.

heading: 3.4 Adjective and adverb modifiers
text: Table 3 shows shifts related to adjectives (amod), adverbs (advmod), and nominal modifiers (nmod).
The root is a prominent HB in embedded clauses as well as nested modifiers, indicating that BERT does not reliably treat modifiers recursively. For example, embedded wh-adverbs such as why are often assigned as dependents of the main verb:
(10)
I don’t know why I chose her BERT
UD
However, the lack of recursion is insufficient to explain all modifier-related shifts. In particular, adjectives of even non-embedded noun phrases are regularly treated as dependents of the root:
(11)
this will put new limits on (...) BERT
UD
This behavior resists interpretation in all prominent syntactic frameworks on adjectives, which ubiquitously treat them as modifiers of nouns or NPs (c.f. Baker 2003; Dixon 2004; Hofherr and Matushansky (ed.) 2010).

heading: 3.5 Prepositional phrases
text: Table 4 collects shifts related to prepositions or clitics (case) and their complements (obl). BERT regularly treats prepositions as dependents of the token modified by the PP, while UD takes them to be headed by the complement noun:
(12)
plenty of other military hardware BERT
UD
BERT also regularly treats the complement as the preposition’s dependent, in contrast to UD linking it directly to the token modified by the PP:
(13)
browse at the morning market BERT
UD
This is especially interesting since here UD prefers the root as opposed to BERT, unlike in our other findings. It thus looks like a genuine syntactic difference. However, the pattern is no longer reliable when the PP modifies a non-root, as shown by the high shift ratios with embedded clauses as HU . The most prominent HB here was again root.

heading: 3.6 Summary
text: I draw four take-home messages:
1. The root is treated as a head far more by BERT than by UD, even across phrase boundaries. 2. BERT’s overlap with UD drastically decreases in embeddings, displaying a lack of recursion. 3. Headedness in PPs is systematically flipped between UD and BERT. 4. Overall, BERT-parses commonly lack a coherent linguistic interpretation.

heading: 4 Discussion
text: The results are not easily explained by some trivial non-linguistic property. Locality does not account for BERT’s deviations from UD, since the average head-dependent distance is actually higher in BERT-parses (Section 3.1). Another initial possibility could be that BERT mimics naive right-chain performance.8 However, most examples in Sections 3.2–3.5 involve BERT assigning the head leftward (i.e. the dependent rightward). Sometimes this even goes directly against right-chain-like annotation in UD, as in example (11) (Section 3.4).
It is also worth raising the controversial status of the UD format itself (c.f. Rehbein et al. 2017; Osborne and Gerdes 2019). The central issue here concerns function words, which UD treats as dependents of content words – going against alternative formats such as Surface-syntactic Universal Dependencies (SUD) (Gerdes et al., 2018) where these relations are reversed. The corresponding distinction appears in our results as well, with respect to prepositions and NPs (Section 3.5). BERT’s performance might thus accord better alternative formats to UD, such as SUD.
That said, most discrepancies discussed in Section 3 are not specific only to UD. All mainstream syntactic frameworks distinguish between arguments/modifiers of main and embedded clauses (Sections 3.2, 3.4), and treat possessors, determiners, numerals, or adjectives as modifying nouns rather than verbs (Sections 3.3, 3.4). With the possible exception of (root-modifying) PPs (Section 3.5), the shifts are not made linguistically coherent by minor changes to the syntactic formalism.

heading: 5 Conclusions and future work
text: This study uncovered several discrepancies between BERT and UD. While some were syntactically interpretable, BERT’s prevailing tendency to treat the root as a head across phrase boundaries lacks a clear linguistic analogy. This puts to question the idea that BERT should be interpreted in line with traditional grammatical formalisms. Instead, it highlights the need to explain LLMs in their own terms – avoiding reliance on a priori linguistic assumptions not motivated by LLMs themselves.
8Wu et al. (2020) report a 35.0 UAS for the naive rightchain baseline in comparison to the 41.7 UAS for BERT. A related issue concerns the comparison between BERT-derived phrase-structures and a naive right-branching baseline, the similarity between which is covered by Niu et al. (2022).
Limitations
This short paper focused on one model architecture (BERT), one parameter-free probing technique (perturbed masking), and one English dataset (PUD). Extending the work to cover multiple variants of each is an important future prospect. I would especially highlight the importance of inter-lingual comparison, as well as more careful attention to assumptions behind the linguistic formalism.
Methodologically, this study combined quantitative and qualitative analysis, both of which have limitations. Numerical information alone (in Tables 1–4) is insufficient for yielding thorough syntactic details on dependent-head shifts. For obtaining such further analyses, specific parse-pairs between BERT and UD need to be assessed, which is how the example cases were attained. But – as manual work – this is bound to have a smaller coverage. Without seeing any easy way out of this trade-off, I emphasize the need for further work extending both quantitative and qualitative coverage of related phenomena. I hope to have provided a fruitful starting-point for this line of research.
Ethics Statement
Prior source code and data used in the experiments is available as open-source, and the link is given in the paper (Section 2.1). No privacy-sensitive or otherwise harmful data was used, and no experiments on humans or non-human animals were conducted. The source code of the experiments is made available as open-source (Section 2.3).

heading: Acknowledgements
text: I thank Jörg Tiedemann and Timothee Mickus for helpful discussions related to the paper. This project was funded by the Academy of Finland (decision number 350775).

heading: A Appendix: Discarded data
text: The algorithm for generating a dependency graph – obtained from Wu et al. (2020) – assumes that token IDs are unique and match positions in the sentence. However, in some coordinated sentences, the UD parse has the same ID appearing in two consecutive tokens. The BERT-parse, in turn, treats the repeated tokens as having separate IDs, which creates a disparity. Table 5 shows an example:
Here, the verb (acts) is repeated since it serves a double role as the root and a conjunct. UD assigns the same ID (2) to both instances, but BERT uses an increasing counter of IDs. Hence, after the repetition, the respective token IDs between UD and BERT no longer match. Since dependent-head pairs are encoded in terms of IDs, this results in artificial disparities between the parses.
Because the number of such sentences in the PUD data was marginal (7), I discarded them in the experiments to avoid this problem. However, the original UAS, UUAS, and NED scores – obtained via replicating Wu et al. (2020) – are calculated from the full PUD data containing these sentences (see Footnote 6).

heading: B Appendix: complete results
text: Table 6 displays each Dep that was subject to a dependent-head shift between BERT and UD. Tables 7–8 show the same per HU and HB , respectively. Table 9 lists all shifts that appeared at least 20 times in the format Dep–HU–HB . This comprises the data discussed in the main paper, from which Tables 1–4 are derived.

heading: Future Work
text: Extending the work to cover multiple variants of each is an important future prospect. I would especially highlight the importance of inter-lingual comparison, as well as more careful attention to assumptions behind the linguistic formalism. Methodologically, this study combined quantitative and qualitative analysis, both of which have limitations. Numerical information alone (in Tables 1–4) is insufficient for yielding thorough syntactic details on dependent-head shifts. For obtaining such further analyses, specific parse-pairs between BERT and UD need to be assessed, which is how the example cases were attained. But – as manual work – this is bound to have a smaller coverage. Without seeing any easy way out of this trade-off, I emphasize the need for further work extending both quantitative and qualitative coverage of related phenomena. I hope to have provided a fruitful starting-point for this line of research