Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Models
Authors: You Li
Jinhui Yin
Yuming Lin
Venue: EMNLP 2023 Findings (2023)
Keywords: Metrics
Probe
Mechanisms
Pretrained Language Models
TL;DR: This paper presents a technique that improves the efficiency of metrics used for model understanding, along with intriguing experimental results and theoretical analyses.

Abstract: Pretrained language models are expected to effectively map input text to a set of vectors while preserving the inherent relationships within the text. Consequently, designing a white-box model to compute metrics that reflect the presence of specific internal relations in these vectors has become a common approach for post-hoc interpretability analysis of pretrained language models. However, achieving interpretability in white-box models and ensuring the rigor of metric computation becomes challenging when the source model lacks inherent interpretability. Therefore, in this paper, we discuss striking a balance in this trade-off and propose a novel line to constructing metrics for understanding the mechanisms of pretrained language models. We have specifically designed a family of metrics along this line of investigation, and the model used to compute these metrics is referred to as the tree topological probe. We conducted measurements on BERT-large by using these metrics. Based on the experimental results, we propose a speculation regarding the working mechanism of BERT-like pretrained language models, as well as a strategy for enhancing fine-tuning performance by leveraging the topological probe to improve specific submodules.

Rethinking the Construction of Effective Metries for Understanding the Mechanisms of Pretrained Language Models
You Li Jinhui Yin Yuming Lin
Guangxi Key Laboratory of Trusted Software, Guilin University of Electronic Technology liyouêguet.c edu. cn 550giinmii55@gaaiai com ymlineguet edu. cn
Abstract
some extent the existence of operations extracting specific information from the vectors, a common approach for post-hoc interpretability analysis such models (Maudslay et al., 2020; Limisiewicz and Marecek, 2021; Chen et al., 2021; White et al., 2021; Immer et al., 2022). However, even though we may desire strong interpretability from white- box model and metrics computed by it that rigor- ously reflect the ability to extract specific informa- tion from given set of vectors, it can be challeng- ing to achieve both of these aspects simultaneously when the source model lacks inherent interpretabil- ity. Therefore, making implicit assumptions during metric computation is common (Korbblihh et al., 2019; Wang et al., 2022). A simple example is the cosine similarity of contextual embeddings. This metric is straightforward and has intuitive ge- ometric interpretation, making it easy explain, but tends to underestimate the similarity of high- frequency words (Zhou et al., 2022).
Pretrained language models expected ef- fectively map input text to set of vectors while preserving inherent relationships within the text. Consequently, designing white- box model to compute metrics that reflect the presence of specific internal relations in these vectors has become common approach for post-hoc interpretability analysis of pretrained language models. However, achieving inter- pretability in white-bor box models and ensuring the rigor of metric computation becomes chal- lenging when the source model lacks inherent interpretability. Therefore, in this paper, we discuss striking balance in this trade-off and propose novel line to constructing metrics for understanding the mechanisms pretrained language models. We have specifically de- signed a family of metrics along this line of investigation, and the model used compute these metrics is referred ical prohe. We conducted measurements on BERT-large by using these metrics. Based on the experimental results, propose spec- ulation regarding the working mechanism of BERT-like pretrained language models, as well strategy for enhancing fine-tuning perfor- mance by leveraging the topological probe to improve specific submodules."
the topolog-
On the other hand, due to the intuition that a white-box model cannot distinguish embeddings that exhibit practical differences (such as context embeddings and static embeddings), it should be considered ineffective, experimental validation of white-box model's ability to effectively distin- guish between embeddings with evident practical distinctions is common practice in research. Fur- thermore, if the magnitude of metrics computed by a white- box model strongly correlates with the qual- ity of different embeddings in practical settings, re- searchers usually trust its effectiveness. Therefore, in practice, traditional white-box models actually classify sets of vectors from different sources.
1 Introduction
Pretrained language models consisting of stacked transformer blocks (Vaswani et al., 2017) are com- monly expected to map input text to set of vec- tors, such that any relationship in the text corre- sponds to some algebraic operation on these vec- tors. However, it is generally unknown whether such operations exist. Therefore, designing a white- box model that computes a metric for a given set of vectors corresponding to a text, which reflects to
Taking the structural probe proposed by Hewitt and Manning transformation on the embedding of each complete word in the text and use the square of the L2 norm of the transformed vector as a prediction for the depth of the corresponding word in the dependency tree (Hewitt and Manning, 2019). 19). In this way, the
example, they perform linear
*Equal contribution. TCCorreponding Author.
IOur code is available at https://github.cr cttpc/wlc.ccccccc////o/o/cc feerris

---

linear transformation matrix serves as learning parameter, and the minimum risk loss between the predicted and true depths used as a metric. In- tuitively, the smaller the metric is, the more likely the embedding contains complete syntax relations. The experimental results indeed align with this in- tuition, showing that contextual embeddings (such those generated by BERT (Devlin et al., 2019)) outperform static embeddings. However, due to the unknown nature of the true deep distribution, is challenging to deduce which geometric features within the representations influence the magnitude structural probe measurements from the setup of structural probe. Overall, while the results of the structural probe provide an intuition that contex- tual embeddings, such as those generated by BERT, capture richer syntactic relations than those of the traditional embeddings, is currently impossible to know what the geometric structure of "good" em- bedding is for the metric defined by the structural probe.
representation.
Therefore, we propose new line that takes tra-
ditional supervised probes as the structure of the white-box model and then self-supervises it, trying to preserve both of the abovementioned proper- ties as much as possible. The motivation behind this idea is that any feature that is beneficial for interpretability has internal constraints. If cer- tain feature has no internal constraints, it must be represented by vector set without geometric con- straints, which does not contain any interpretable factors. Therefore, what is important for inter- pretability is the correspondence between the inter- nal constraints of the probed features and the vector set, which can describe the geometric structure of the vector set to some extent. In the case where the internal constraints of the probed features are well defined, a probe that detects these features can naturally induce a probe that detects the internal constraints, which is self-supervised. In summary, the contributions of this work in- clude:
In addition, to enhance the interpretability and flexibility of white-box models, it is common to include assumptions that are challenging to empiri- cally validate. For example, Ethayarajh proposed to use anisotropy- self-similarity to measure the context-specificit of embeddings (Ethayarajh, 2019). Since the computation of this metric doesn't require the introduction of additional human la- bels, it is theoretically possible to conduct further analysis, such as examining how fundamental ge- ometric features in the representation (c.g., rank) self-similarity, or simply
We propose a novel self-supervised probe, referred to as the tree topological probe. to probe the hierarchical structure of sen- tence representations learned by pretrained language models like BERT.
2. We discuss the theoretical relationship be- tween the tree topological probe and the struc- tural probe, with the former bounding the lat- ter.
affect mnisotropy-adjusee consider this metric defining new geometric feature. Overall, this is metric that can be dis- cussed purely the mathematical level. However, verifying whether the measured sppeext-specificity in this metric aligns well with context-specificit in linguistics, without the use of, or with only limited additional human labels, may be challenging. Ad- ditionally, confirming whether the model leverages the properties of anisotropy- adjjssed self-similarin during actual inference tasks might also be chal- lenging.
3.
We measure the metrics constructed based on the tree topological probe on BERT-large. According to the experimental results, we propose speculation regarding the working mechanism of a BERT-like pretrained lan- guage model.
4. We utilize metrics constructed by the tree topo- logical probe enhance BERT's submodules during fine-tuning and observe that enhancing certain modules can improve the fine-tuning performance. We also propose a strategy for selecting submodules.
There appears to be trade-off here between two types of metrics:
Related Work
1. Metrics that are constrained by supervised signals with ground truth labels, which provide more practical intuition.
The probe is the most common approach for as- sociating neural network representations with lin- guistic properties (Voita and Titov, 2020). This approach is widely used to explore part of speech
2. Metrics that reflect the geometric properties of the vector set itself, which provide a more formal

---

knowledge (Belinkov and Glass, 2019; Voita and Titov, 2020; Pimentel al., 2020; Hewitt al., 2021) and for sentence and dependency structures (Hewitt and Manning, 2019; Maudslay et al., 2020;
we can determine the correspondence between M and R. The probe is model that investigates the relationship between the geometric constraints of the vector set and Y. is composed of a function set F and a metric Fy defined on Y. The input of a function in F is the representation vector of a language object, and the output is the predicted Y feature of the input language object. The distance
White et al., 2021; Limisiewicz and Marecek, 2021; Chen et al., 2021). These studies demonstrate many important aspects of the linguistic information are encoded in pretrained representations. However, in some probe experiments, researchers have found that the probe precision obtained by both random representation and pretrained representation were quite close (Zhang and Bowman, 2018; Hewitt and Liang, 2019). This demonstrates that it is not suffi- cient to use the probe precision to measure whether the representations contain specific language infor- mation. To improve the reliability of probes, some researchers have proposed the use of control tasks in probe experiments (Hewitt and Liang. 2019). In recent rescarch, Lovering
between the predicted feature and the true feature is calculated by using the metric Ey, and a function f in F that minimizes the distance determined. Here, F limits the range of geometric constraints, and Ey limits the selection of a "good" geometry. Notably, this definition seems very similar to that of learning. Therefore, the larger the scope of F is, the harder it is to discern the form of the geometric constraints, especially when F is neural network (Pimentel et al., 2020; White al., 2021). How- ever, the purpose of the probe is different from that of learning. The goal of learning is to construct a model M (usually a black box), which may have multiple construction methods, while the purpose of the probe is to analyze the relationship between M and R.
realized that in- ductive bias can be used to describe the ease of extracting relevant features from representations. Immer et al. further proposed a Bayesian frame- work for quantifying inductive bias with probes, and they used the Model Evidence Maximum in- stead of trivial precision.
3.2 The Design Scheme for the Topological Probe
3 Methodology
One of the goals of topology is to find homcomor- phic or homotopic invariants (including invariant quantities, algebraic structures, functors, etc.) and then to characterize the intrinsic structure of a topo- logical space with these invariants. Analogously, we can view R as a geometric object and Y as its topology. Can we then define a concept similar to topological invariants with respect to Y??
As the foundation of the white-box model proposed in this paper is built upon the traditional probe, will begin by providing a general description of the probe based on the definition presented in (Ivanova et al., 2021). Additionally, we will introduce some relevant notation for better understanding.
3.1 General Form of the Probe
We define feature invariant for Y as set of conditions Cy such that any element in Y satis- fies Cy. Cy reflects the internal constraints of the probed feature, as well as a part of the logical con- straints of R. Furthermore, Cy is well defined, it induces a set Xcy consisting of all objects satisfy- ing Cy, which naturally extends the metric defined on Y to Xcy-
Given a character set, in formal language, the gen- eration rules uniquely determine the properties of the language. We assume that there also exists set of generation rules R implicitly in natural language, and the language objects derived from these rules exhibit series of features. Among these features, a subset Y is selected as the probed feature for which the properties represent the logical constraints of the generation rule set. Assuming there is another model M that can assign a suitable representation vector to the generated language objects, the prop- erties of Y are then represented by the intrinsic geo- metric constraints of the vector set. By studying the geometric constraints that are implicit in the vector set and that correspond to Y, especially when Y is expanded to all features of the language object,
Furthermore, just the distance measure be- tween two points induce distance measure between a point and a plane, the distance measure between the predicted feature pz and Xcy can also be induced by Ey (denoted as Ecy)::
Ecy (px, Xoy) = min Ey (pz,x) (1) acXoy
It can be easily verified that if Ey is a

---

well-defined distance metric Y. then Eor should also be well-defined distance metric on
De R. visS LH
pa. Once we have Ecy, the supervised probe (F, Ey, Y) can naturally induce self-supervised probe (F, FirrCCy) We refer to (F, F Cy) the self-supervisee version of (F, Ey, Y), also known as the topological probe.
pdep(h:) Iff h,ll?
(2)
where pdep(h:) is the prediction tree depth of ws in Tw and f is real matrix of size m Because Vp there is tree that cannot be embedded as above (Reif al., 2019), so p is usually taken as pdep(hı), pdep(h2) ... pdep(h)u) form a sequence denoted as pdepH-
Notably, the prerequisite for obtaining (F, Ecy, Cy) is that Cy must be well-defined, so Cy should not be a black box. Figure shows an intuitive illustration.
Moreover, we denote the true depth of wi dep(w::. Hence, dep(wi): dep(w22 also form sequence denoted as depw. The metric E in the structural probe is defined as follows:
dep(wr.n
E E(pdepH, deew)
Xcy
Xy D
LR
1
Σ (pdep(hi) dep(w:)) LH
(3)
E,(D, =
|OD]
Therefore, the structural probe is defined as (lf Fr dep).
Ec,18, =0 Ec,
Now provide the constraints Cdep for dep. An important limitation of depw is that it is an integer sequence. Based on the characteristics of the tree structure, it is naturally determined that depw must satisfy the following two conditions:
Figure I: The relationship between the distance from the predicted feature Xoy and the distance from to Xy.
(Boundary condition). If LE > 1, there is exactly one minimum element in depw, and it is equal 1; if I, 2, at least one element in depw equal to 2.
Next, we present specific topological probe that is based on the previously outlined design scheme and serves as a self-supervised variant of the struc- tural probe.
(Recursion condition). we sort depw in as- cending order to obtain the sequence asdepw then
3.3 The Self-supervised Tree Topological Probe
V1 visis LH
Given a sentence W, it is represented by model M as set (or sequence) vectors, denoted as = (W). The number of vectors in H de- noted as LH. and we assign an index 2I. to each vector in H so that the order of the indices matches the order of the corresponding tokens in the sentence. Additionally, sion of the vectors as n. For each W. there exists syntax tree Tw, where each complete word in W corresponds to a node in Tw-
asdep(w:+1) = asdep(wy)
or
asdep(we:1) asdep(us) +
We denote the set of all sequences that conform to Cdep as XCa.p From equation 1, we can induce a metric Ecaer
denote the dimen-
Ecaup (pdepn,XCaan
min xEX
E(pdepn x))
The probed feature Y that the structural probe defines is the depth of the nodes corresponding to complete words. Following the work in (Hewitt and Manning. 2019), we the parameter space of for the structural probe to all real matrices of size m * n, where m < n. The specific form for Given
(4)
construct explicit sequence minsw such that:
Assuming
Ly (pdep(h:) x(w)))² (5)
minsw = arg min
predicting the depth is as follows:
xEX Caep t=1

---

bias; 1 1 otherwise, bias; 1 1 -.. We can obtain an analytical expression for ECaey follows:
(Alignment). Let aj(1 I S (.H) denote the index of apdep; pdepH. Then, let
ECdep (pdepH, XCeep) E(pdepH, minsw)
(6)
pesu( pre(w;)
(8)
Consider the following two examples:
It can be shown that pesuw constructed in the above manner satisfies the following theorem:
1. When pdeps 0.8, 1.5, 1.8, 2.1, then minsw 2,,4.
Theorem 1. Vi - 1, .. Lu - apdep.+1
2. When pdepı = 0.8, 1.8, 1.8, 2.4, then minsw = 4,
apdep; <= 1, then
E(pdepn: pesuw) E Eppdepn, minsw)
can be observed that the predicted depths for nodes further down the hierarchy can also in- fluence the corresponding values of minsw for nodes higher up in the hierarchy. the examples provided, due to the change from 4.5 to 7.5, 1.8 changes from 2 to 3 at the corresponding minsw. Therefore, using straightforward local greedy ap- proach may not yield accurate calculation of minsw, and if simple enumeration method is employed, the computational complexity will be- come exponential.
Therefore, pesuw can be considered an approxi- mation to minsw. Appendix A contains the proof of this theorem. In the subsequent sections this paper, we replace FCcepp (pdepH, XCuup) with E (pdepH- pesuw).
Additionally, an important consideration is de- termining the appropriate value of the minimum element for depw in the boundary condition. In the preceding contents, we assumed root depth of for the syntactic tree. However, traditional struc- tural probe (Hewitt and Manning, 2019; Maudslay et al., 2020; Limisiewicz and Marecek, 2021; Chen et al., 2021; White et al., 2021), the root depth is typically assigned as O due to the annotation con- ventions of syntactic tree datasets. From logical perspective, these two choices may appear indistin- guishable.
However, while a local greedy upproach may not always provide an exact computation of minsw. it can still maintain a certain degree of accuracy for reasonable results of pdepи- This is because cases like the jump from 2.4 to 7.5 should be infrequent in well-trained probe's 's computed sequence of predicted depths, unless the probed representation does not encode the tree structure well and exhibits a disruption in the middle.
However, in Appendix B, we demonstrate that the choice whether the root depth is has a significant impact on the geometry defined by the tree topological probe. Furthermore, we can prove that as long as the assigned root depth is greater than O, the optimal geometry defined by the tree topological probe remains the same certain extent. Therefore, in the subsequent sections of this paper, we adopt the setting where the value of the minimum element of depw is 1.
Before delving into that, we first introduce some notations:
apdepy denote the sequence obtained by sort- ing pdepa in ascending order.
apdep: represents the i-th element of apdeps.
prew be a sequence in XCae
Here, introduce simple method for con- structing minsw from a local greedy perspective. (Initialization). If LH > 1, let pre(w1) = 1; La 2 2, let pre(w2) = (Recurrence). If Lu 2 3 and 3 < Lys let
3.4 Enhancements to the Tree Topological Probe
Let the of all language objects generated by rule R be denoted as XR. and the cardinality of X'R be denoted as |AR]. The structural probe induces a metric that describes the relationship between model M and dep:
pre(w:) p) + bias,-1 where the values of bias;-1 and apdepH are re- lated
(7)
(M) = min
Σ WEXR
E(pdepM(w) depw)
JEF LR
(pre(w;-1) +)+1 -apdep; S (p) - -)pdddepp)
(9)

---

The tree topological probe can also induce a similar
sampling W from Xn. We suppose the density function of this distribution denoted Pa. and the expectation with respect to 0 denoted as Ep,- Then we can approximate & {)((M) as follows:
metric:
Xaspl M) =
1
min JEF
Σ
(eddepprrw):mmnsse (10)
X's (M) (er
ssp(M)
RR
WEAK
(14)
On the other hand, let
While the analysis Pa is not the primary focus of this paper, in the absence of any other constraints or biases on model M, we conjecture that the dis- tribution curve of & may resemble uniform bell curve. Hence, we consider the following distribu- tion approximation:
l (pdep(hs) - x(w)))²
moxsw arg max Ze
(11)
similar to minsw, and mazsw, inducing the fol- lowing metrics:
(a(x) - 6(xr - x²) x x [011]
Xessp(M) (M)
At this point:
1 Kul
-min JEF
Σ
E(pdepM(w); marss)
Xsp( = (Nessp((M) + Xsssp(M)) (15)
WEXR
(12)
Therefore, utilizing a self-supervised metric can approximate the unbiased optimal geometry de- fined by the structural probe:
Since depw E XC when f is given, we have:
E(pdepM(w): depw) < max =EXc
((
Mg arg min ( = (sssss(( (16) M
Furthermore, as Xsp((M) and Xessp(() share the same set of probing functions F, we have:
Moreover, Mg is an analytically tractable ob- ject, implying that the metrics induced by the tree topological probe preserve to a certain extent the two metric properties discussed in the introduction. However, there is a crucial issuc that remains unre- solved. Can we explicitly construct moxsw? ? Cur- rently, have not found a straightforward method similar to constructing pesuw for approximating marsw. However, based on the sorting inequal- ity, we can construct a sequence that approximates marsw based on prew. Let d,(1 < i S Lи)
(M) S (M)
Therefore, Xessp(M) provides with an upper bound for the structural probe metric. Similarly, for X sss,(MM, we also have:
Xxsp (M) & (M)
Therefore, (M) provides us with lower bound for the structural probe metric. In summary, we have the following:
denote LH - i 1. Then, let
(xpes((wa)) = pre(wa,)
(17)
(M) s (M) S (M)
In our subsequent experiments, we approximate E E(pdepH: mazsw) with E(pdepи, xpesuw).
Xs) Xesss(M),, then there is no difference between the tree topological probe and the structural probe. On the other hand, if it is believed that smaller {Mp is desir- able, then estimating XMp(MM within the range (. (M) becomes an interesting problem. We consider the following: Ow-
Experiments
In this section, delve into a range of experiments conducted on the tree topological probe, along with the underlying motivations behind them. To accom- modate space limitations, we include many specific details of the experimental settings in Appendices C and D. Morcover, we focus our experiments on BERT-large and its submodules. Moreover, con- ducting similar experiments on other models is also straightforward (refer Appendix for supple- mentary results of experiments conducted using RoBERTa-large)
((pdep depw) - 000--E-pdppppee E(pdepM(w) minss)
E( mazsw) - E(pdepM(w) minsw) (13)
This leads to an intriguing linguistic distribution, the distribution of Bw E [0 when uniformly

---

Table 1: Grouping M, based on ssp.. Mr-M, de- notes Mr, M1+1. M+ My. For example, the first row of the table indicates that the exact values s'ssp 4.1 Measuring X'ssp and Xessp on BERT
We denote the model consisting of the input layer and the first i transformer blocks of BERR-llarge as MI < i < 24). Since the imput of M; consists of tokenized units, including special tokens [CLS],
Xaupp Nog
Kyg
-log
[SEP], [PAD], and [MASK], we conduct at least four types of measurement experiments:
true
unbizsed
cl. Measurement of the vector set formed by to- ken embedding and special token embedding.
3.223444 01001122101111111100000002222220
e2. Measurement of the vector set formed solely by token embedding.
M1
Figure 2: Negative logarithm of Xepp Keesps unbiased Xsp and true Xap across M₁-
e3. Measurement of the vector set formed by es- timated embedding of complete words using token embedding and special token embed- ding.
fl. Assp and Xessp indeed bound the actual X' sp, and for M14 to Mis, their true X' are very close to their X'ssp.
e4. Measurement of the vector set formed solely by estimated embedding of complete words using token embedding.
12.
Mo serves as a good bascline model. Further- more, using A'sssp and unbiased A'sp allows for effective differentiation between embed- dings generated by models consisting solely of the regular input layer and those generated by models incorporating transformer blocks.
Similarly, due space constraints, focus on discussing el in this paper. The measurement results are shown in Tables 1 and 2. The precise measurement values can be found in Appendix Furthermore, as shown in Figure 2, we present the negative logarithm curves of three measurement values as function of M, variation.
f3. For M1 to Mg. their true X'sp are very close their unbiased X'sp.
f4. Both the curve of (log(X'sss))) and the curve of the true (X''sX's follow an ascending- then-descending pattern. However, the mod- els corresponding their highest points are different, namely, Mg and M16. respectively.
X'ssp
M
0.01~0.05
Mr~
0.05~0.1
M21
0.1~0.15
M23~~Ma
f5.
For the curve of (log(s)))))) its overall trend also shows tern but with some fluctuations in the range of M3 to Mg. However, the model corre- sponding to its highest point consistent with -log (Xessp), which Mg.
sacending-then-descee
pat-
for Mo, Mi, M2, ..,, Mi fall within the range 0.01 to 0.05.
X'essp
M M1~M12 M1a
f6. The true X'sp does not effectively distinguish between My and M1.
0.3~0.4
M4~
0.4~0.5
0.5~1.0
Mr~M2
Ms
M1 M24
Based on the above findings, we can confidently draw the following rigorous conclusions:
1.0~2.0
2 4.0
Mo
cl. Based fl, we can almost infer that depw arg min ΣLn (pdep(h.) =(w:))² for M14 =EXcar
Table 2: Grouping M; hased on Xeessp. Similar to the explanation in the caption of Table 1.
Mi8. This implies that they memorize the preferences of the real data and mini- mize as much as possible to approach the
By examining the experimental results presented above, we can ascertain the following findings:

---

theoretical boundary. Building upon 13, we can further conclude that the cost of memoriz- ing depw an increase in A'sp,, which leads to a decrease in the accuracy of the embed- ding's s linear encoding for tree structures.
to consider language information various levels, but its memory capacity is limited. Therefore, it relies on preceding submodules to filter the infor- mation. The earlier submodules in the model en- code the most generic (unbiased) structures present in the language features. As the model advances, the intermediate submodules start incorporating preferences for general structures based actual data. Once certain stage reached, the later submodules in the model start to loosen their en- coding of generic structures. However, due to the preference information passed from the interme- diate submodules, the later submodules can still outperform the earlier submodules in encoding real structures, rather than generic ones.
c2. Based on fl, we can conclude that there exists model M where the true X' ()) aligns with the (M) determined by Cdep. This indicates that Ceep serves as a sufficiently tight condition.
c3. Based on f3, we can infer that Mi to Mg may not capture the distributional information of the actual syntactic trees, resulting in their generated embeddings considering only the most general case for linear encoding of tree structures. This implies that the distribution curve of their Ow parameters uniformly bell- shaped.
Based c3 and c6, it appears that true A'sp < unbiased X'p Neessp This suggests that for BERT, unbiased Xspp serves as a tighter upper bound for Xsp, and there exists a submodule that achieves this upper bound. Now, the question arises: this also the case for general pretrained models? If so, what are the underlying reasons?
c4. Based on 12 and 16, we can conclude that the tree topological probe provides a more fine- grained evaluation of the ability to linearly encode tree structures embedding vectors compared to the structural probe.
4.2
Using Assp and X' Aessp as Regularization Loss in Fine-tuning BERT
Let us denote the downstream task loss as TM24). Taking A'ssp as an example, using X ssp as a regu- larizing loss during fine-tuning refers to replacing the task loss with:
c5. Based on f3, f4 and f5, we can conclude that in BERT-large, embedding generated by Ms and its neighboring models exhibit the strongest ability to linearly encode tree struc- tures. Moreover, they gradually start to con- sider the distribution of real dependency trees, resulting in the true spM) approaching s'sp( M) until reaching M16-
T(M24) + > X'ss((M) <1< i 24)
where À is regularization parameter. The pur- pose this approach to explore the potential for enhancing the fine-tuning performance by im- proving the submodules of BERT in their ability to linearly encode tree structures. If there exists a submodule that achieves both enhancement in lin- encoding capabilities and improved fine-tuning performance, it implies that the parameter space of this submodule, which has better linear encod- ing abilities, overlaps with the optimization space of fine-tuning. This intersection smaller than the optimization space direct fine-tume fnn-tuning, reduc- ing susceptibility to local optima and leading to improved fine-tuning results.
c6. Based on f4 and f5, can conclude that start- ing from M16. the embeddings generated by Mj gradually lose their ability to linearly en- code tree structures. The values of Xssp and Xesspp for these models are generally larger compared to models before M16- However, they still retain some distributional informa- tion about the depth of dependency trees. This means that despite having higher unbiased X's, their true X s'sp is still smaller than that of Mj before Mg.
From the above conclusions, can further spec- ulate about the workings of pretrained language models such as BERT, and we identify some re- lated open problems.
Conversely, if enhancing certain submodules hin- ders fine-tuning or even leads its failure, it sug- gests that the submodule's parameter space, which has better linear encoding abilities, does not over- lap with the optimization space of fine-tuning. This indicates that the submodule has already attained
Based on c5 and c6, we can speculate that the final layer of pretrained language model needs

---

the smallest X'ssp value that greatly benefits the BERT's performance.
meaning. If there were models like BERT GPT in the "Vzigs" planet, perhaps called "YVJIG" and "TLG," " would the pretraining process of "YVJIG" on "Vmiprhs" be the same as BERT's pretraining on "English"?
Based on fl, we can infer that M11 to Mis are not suitable as enhanced submodules. According to c5, the submodules most likely to improve fine- tuning performance after enhancement should be near Mg. We conducted experiments on a single- sentence task called the Corpus of Linguistic Ac- ceptability (CoLA) (Warstadt et al., 2019), which is part of The General Language Understanding Eval- uation (GLUE) benchmark (Wang al., 2019).
In theory, there should be means to differen- tiate between these two pretraining processes. For blank model (without any training). extracting useful information from "Vmtprhs" and "English" would pose the same level of difficulty. However, it is true that "Vmtprhs" and "English" are distinct, with the letters of Vmmpphs" possibly having dif- ferent shapes or being the reverse order of the "En- glish" alphabet. Therefore, we can say that they have different letter features, although this feature seems be a mere coincidence. In natural lan- guage, there are many such features created by historical contingencies, such as slang or gram- matical exceptions. Hence, when we aim to inter- pret the mechanisms of these black-box models by studying how language models represent language- specific features, must consider which features are ad vdnatageous for interpretation and what we ultimately hope gain from this research.
The test results are shown in Table 3. As pre- dicted earlier, enhancing the submodules around M14 to M1s (now expanded to M12 to M19) proves to be detrimental to fine-tuning, resulting in failed performance. However, we did observe an im- provement in fine-tuming performance for the sub- module Mi0 near Ms after enhancement. This gives us an intuition that we have additional topo- logical probes and similar metrics to X'ssp and Yss,, we can explore enhancing submodules that are in the rising phase of true sp, away from the boundary of unbiased s'sp and X' A'ssp, in an attempt to improve fine-tuning outcomes.
This paper presents a thorough exploration of a key issue, specifically examining the articula- tion of internal feature constraints. By enveloping the original feature within a feature space that ad- heres to such constraints, it is possible effectively eliminate any unintended or accidental components. Within this explicitly defined feature space, met- rics such as Xssp and X'essp can be defined. We can subsequently examine the evolution of these metrics within the model to gain a deeper under- standing of the encoding strategies employed by the model for the original feature, as described in the experimental section of this paper. Once un- derstand the encoding strategies employed by the model, can investigate the reasons behind their formation and the benefits they bring to the model. By conducting studies on multiple similar features, we can gain a comprehensive understanding of the inner workings of the black box.
Method
mean
std
max
DF
63.34
1.71
66.54
EH M3
63.90
2.66
68.73
EH Ms
63.90
1.36
66.04
EH M10
64.87
2.07
68.47
EH M
0.00
0.00
0.00
EH M20
5.48
16.46
54.87
EH M24
40.43
26.60
62.52
Table 3: Direct fine-tuning and sub-module enhance- ment test scores. Here, "DF" denotes direct fine-tuning, while "EH M." represents the fine-tuning with the en- hancement of M; based on X'ssp. The evaluation metric used in CoLA is the Matthew coefficient, where higher value indicates better performance.
5 Conclusion
Consider a thought experiment where there is a planet in a parallel universe called "Vzjgs" with a language called "Vmiprhs". Like "English", "Vmt- prhs" comprises 26 letters as basic units, and there is one-to-one correspondence between the letters of "V "Vmtprhs" and "English". Moreover, these two languages are isomorphic under letter permutation operations. In other words, sentences in "English" can be rearranged so that they are equivalent to sentences in "Vmtprhs", while preserving the same
Limitations
The main limitation of this research lies in the approximate construction of minsw and marsw. which leads true (log(X'sp)) surpass- ing log(XXss ssp) near M16 to some extent. How- ever, this may also be due to their proximity, re- sulting in fluctuations within the training error. On

---

the other hand, the proposed construction scheme for the topological probe discussed in this paper lacks sufficient mathematical formalization. One possible approach is to restate using the language of category theory.
John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. In Proceedings of the 2019 Conference Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- 2019, pages 2733-2743.
Acknowledgements
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HITT 2019, Volume I (Long and Short Papers), pages 4129-4138.
We thank the anonymous reviewers for their help- ful comments and suggestions. This work was supported by National Natural Science Founda- tion of China (Nos. 62362015, 62062027 and U22A2099) and the project of Guangxi Key Labo- ratory of Trusted Software.
Hang Hua, Xingjian Li, Dejing Dou, Cheng-Zhone Xu, and Jiebo Luo. 2021. Noise stability regularization for improving BERT fine-tuning In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLI 2021, pages 3229-322220 -2411.
References
Yonatan Belinkov and James R. Glass. 2019. Analysis methods in neural language processing: A survey. Trans. Assoc. Compul. Linguistics, 7:49-72.
A Alexander Immer, Lucas Torroba Hennigen, Vincent Fortuin, and Ryan Cotterell. 2022. Probing quan- tifying inductive bias. Proceedings of 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, pages 1839-1851.
Daniel M. Cer, Mona T. Diab, Encko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval- 2017 task 1: Semantic textual similarity multilin- gual and cross-lingual focused evaluation. CoRR, abs/1708.00055.
Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. 2021. Probing BERT in hyperbolic spaces. In 9th Inter- national Conference Learning Representations, ICLR 2021.
Anna A. Ivanova, John Hewitt, and Noga Zaslavsky. 2021. Probing artificial neural networks: insights from neuroscience. CoRR, abs/2104.08199
Simon Kornblith, Mohammad Norouzi. Honglak Lee, and Geoffrey E. Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 3519-3529.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Volume (Long and Short Papers), pages 4171-4186
Tomasz Limisiewicz and David Marecek. 2021. Intro- ducing orthogonal constraint in structural probes. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNPP 2021, Volume 1 Long Pa- pers, pages 428-442.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLA 2005.
Charles Lovering. Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre- trained models. In 9th International Conference on Learning Representations, ICLR 2021.
Kawin Ethayarajh. 2019. How contextual are contex- tualized word representations? comparing the ge- ometry of bert, elmo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNI.P-IJCNIPP 2079, pages 55-65.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Mareinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Comput. Lin- guistics, 19(2)333--330.
John Hewitt, Kawin Ethayarajh, Percy Liang, and Christopher Manning. 2021. Conditional prob- ing: measuring usable information beyond a haseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, pages 1626-1639.
Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel. Adina Williams, and Ryan Cotterell. 2020. A tale of a probe and parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 7389-7395.

---

Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theone theoretic probing for linguistic structure. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. ACI. 2020, pages 4609-4622.
task analysis. In Proceedings of the Workshop: An- alyzing and Interpreting Neural Networks for NLP, BlackboxNI.P P. EMNLP 2018, pages 359-361.
Kaidyn Zhou, Kawin Ethayarajh, Dallas Card, and Dan Jurafsky. 2022. Problems with cosine as a measure of embedding similarity for high frequency words. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 2: Short Papers), ACI. 2022, pages 401-423.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viégas, Andy Coenen, Adam Pearce. and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Process- ing Systems 32: Annual Conference on Neural In- formation Processing Systems 2019, NeurlPS 2019, pages 8592-8600.
A Proof of Theorem 1
Proof. For any sequence x E XCuep that is in the same order as pdepH, according the inequality of rankings, for any permutation Ty of a, we have:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 5998- 6008
LR
To (w:)) pdee(1:) S
r(we)) * pdep(h:)
Therefore,
Elena Voita and Ivan Titov. 2020. Information-theorrii probing with minimum description length. In Pro- ceedings of 2020 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2020, pages 183-196.
L (wn(ws) pdep(hs)))
L (x(w)) pdep(h:))2
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations, ICLR 2019.
Since pesuw and pdepu are the same order, we just need to prove that any sequence æ E XC and the same order as pdepy satisfies
Bin Wang, C.-C. Jay Kuo, and Haizhou Li. 2022. Just rank: Rethinking evaluation with word and sentence similarities. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, pages 6060- 6077.
L (x(w)) pdep(h.))²
LH (pesu(w:) - pdep(h.))²
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Trans. Assoc. Comput. Linguistics, 7:625 641.
The theorem is automatically established. Because
La (pesu(ws)) - pdep(h:))2
Jennifer C. White, Tiago Pimentel, Naomi Saphra, and Ryan Cotterell, 2021. A non-linear structural probe. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. NAACL-HLT 2021, pages 138-138.
Lu = (pre(w;; - apdep,))
(18)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's 's transformers: State-of-the -he natural language processing. CoRR, abs/19 ths/1910..3771.
without loss of generality, we can assume pesuw and æ to be ascending sequences and not equal and exist & such that when 1 < i < -
pesu(ws) = x(w))
(19)
Kelly W. Zhang and Samuel R. Bowman. 2018. Lan- guage modeling teaches you more than translation does: Lessons learned through auxiliary syntactic
and
pesu(wk) + x(ww)
(20)

---

Based the recursive condition, can infer that Ipesu(urg x² (wxx) =1
Since
EP E Rm Vi(i = 1,2, LH)
(21)
pesu(w;) Va < h p" Phg < pesu(us)) + vo
Combined with the value condition of bi we further find that
E < pesu p
Let P'
pesu' (wr)
P and when i = 11 .. LH
[pe)u(ws)) apdepal $ [x([ux) apdepsl
Vpesu(ur)
pesu (ux)) fec
The inductive hypothesis when m is
4 pesu(w;)
(pesu(wm) apdepml S ( apdepml
then
pesu (wi)
Due to the condition apdepm+1 apdepm S and the value condition of biasm. still holds when = m + that
"pesulux)2 pesu (w;)²
pesu w4))
After calculation,
vi(i = 1,2, Ln)
(pesu(um+l) apdepm+al
pesu (wi) - V
S (la(wm+1)) apdepm+il
Thus, when i k
< pesu (mx)) + V
(x(ws) pdep(h:))) (pesu(wx) pdep(h:))
Therefore, is well defined, and iH, H3 Spesuw when H4 + Hj s (H.) + (HH)
B Analysis of Tree Depth Minimum
Therefore, o is also an injective function. is easy to prove that the inverse map Φ-1 of & is also an injective function and satisfies the above conditions.
The minimum of pesuw is denoted as depminn Fix- ing pesuw, we let all sets (or sequences) of vectors satisfying the following conditions compose a set denoted by Spesuw.
n
The proof of the theorem above does not apply to the cases where pesu(w) ) = 0 pesu (wi) 0. If depmin greater than then the results of the tree topological probe do not necessarily depend on the selection of depminn and we may set it as 1. However, we have not further explored whether Theorem 2 is necessarily invalid. Nevertheless, we can examine the drawbacks that arise from setting depmin to 0 from another perspective.
]P E Rm Vi(i 1.2
pesu(w:) Va < 1o PT Ph+ < pesu(wr) + Vc, Here, & < p Let pesu(wi) (w)) be depmin and pesu(w;) S pesu(w; (1 = 1,2, L. 1) without loss of generality, and the following theorem can be obtained.
Theorem 2. For any two different sequences pesuw and pestiw, if pesu(w1) > O and pesu (w)) > 0. there is one-to-one mapping & between Spesuw and Q peealy"
When > 2, h, is projected by P near the (m)) dimensional sphere with radius of / /pesu(we),
Vi = 1,2 ..... 1...... LH
1n PT Ph: - pesu(wi)l < Es
Proof. We construct & such that
If depmin = O, then the topology of the geo- metric space composed of all vectors Phj sat- isfying |h| p" Ph depminl < is homeo- morphic to an m-dimensional open ball. This may result in probes exhibiting different prefer- ences for the root and other nodes. However, if depmin > 0, the topology of the geometric space is an m-dimensiional annulus, which the same for all nodes, thus avoiding the issue of preference.
HH Sppssu
((H) = (hi, h2g bin H' E St Peessly Here, hi h1, and when i = 2.
k1 v pesul (wi) pesu(w1) h4 (pesu(ws) * pesu' (wi)

---

C
Data for Training and Evaluating Probes
E Supplementary Chart Materials
Table lists the exact measurements of A'ssp Ressp, and true Xsp for BERT-Large.
To ensure the reliability and diversity of data (ap- propriate sentences) sources, we separated the sen- tences participating in the probe experiment from the training, verification and test data sets of some tasks of The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019), We selected four small sample text classifica- tion tasks in GLUE with reference to (Hua al., 2021), namely, the Corpus of Linguistic Accept- ability (CoLA) (Warstadt et al., 2019), Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005), Recognizing Textual Entailment (RTE) (Wang al., 2019) and Semantic Textual Similarity Benchmark (STS-B) (Cer et al., 2017), which cover the three major task types of SINGLE- SENTENCE, SIMILARITY AND PARAPHRASE and INFERENCE GLUE MRPC, RTE and STS- B are all double sentence tasks, and the experiment needs only BERT to represent a single sentence; thus, we consider two sentences that belong to the same group of data independently, not spliced.
M
X Kss
essp
Pisp
Mo
0.039
5.382
0.3084
M1
0.017
0.536
0.2644
M2
0.017
0.526
0.244
M3
0.018
0.348
0.2016
M4 0.033
0.351
0.1701
Ms 0.025
0.52
0.1622
Me 0.023
0.52
0.1559
M7 0.013
0.345
0.14
Ms
0.01
0.347
0,1424
M9
0.011
0.352
0.1577
M10
0.013
0.359
0.1415
M11
0.021
0.375
0.1128
M12
0.054
0.391
0.0975
M13
0.076
0.42
0.0764
M14
0.084
0.467
0.0651
M15
0.088
0.525
0.0616
M16
0.09
0.663
0.0656
M17
0.086
0.785
0.0808
After the data sets of the four tasks are processed above, the remaining statements are merged into a raw text data set rtdmiz, which contains 47136 sentences. This close to the size of the Pennsyl- vania tree database (Marcus et al., 1993) used by the structural probe (Hewitt and Manning, 2019); short and long sentences are evenly distributed.
Mi8
0.09
0.883
0.1155
M19
0.09
0.999
0,1416
M20
0.092
1.045
0.1615
M21
0.094
1.447
0.2468
M22
0.102
1.715
0.28634
M23
0.107
1.709
0.3171
M24
0.113
1.837
0.328
D Experimental Setup for Training Probes and Fine-tuning
Table 4: Exact values of X,sp- Keasps and true X.p for M,
We use the BERT implementation of Wolf al. and set the rank of the probe matrix to be half the embedding dimension. The probe matrix is ran- domly initialized following a uniform distribution U(- ((00,55,0 0.05).
F Experimental data for RoBERTa-large
Figure 3 shows the negative logarithm curves of three measurement values as a function variation in Mi for RoBERTa-Large. Table 5 lists the exact measurements of X'ssp, Xessp, and true Xep for RoBERTa-Large.
We employ the AdamW optimizer with the warmup technique, where the initial learning rate is set to 2c-5 and the epsilon value is set to 1c-8. The training stops after 10 epochs. The training setup for fine-tuning experiments is similar that of training probes. One notable difference is the regularization coefficient X, which is dynamically determined after one epoch of training, ensuring that (M.)) 0.1, without any manual tuning. T(M24) We conduct experiments on each fine-tuning method by using 10 different random seeds, and we compute the mean, the standard deviation (std), and the maximum values.
From the experimental data, it is evident that the overall pattern of evolution in the graphs for RoBERTa-Large and BERT-Large is consistent. There's s slight initial increase followed by de- cline, but the boundaries for X'sp in the case of RoBERTa-Larg are much tighter, especially in the earlier modules.

---

(log(ip))
-log(Metric)
M;
Figure 3: Negative logarithm of Xas-- Kesaps unbiased Xap and true Aap across M,.
M
Xssp
Xessp
Kisp
Mo
0.008
3.532
0.991
M1
0.145
0.515
0.243
M
0.137
0.469
0.446
M3
0.139
0.470
0.331
M
0.131
0.493
0.257
M
0.132
0.500
0.199
M
0.123
0.494
0.153
Mr
0.117
0.491
0.110
Ms
0.117
0.542
0.109
M
0.114
0.491
0.087
M10
0.113
0.555
0.091
Mn
0.113
0.567
0.091
M
0.110
0.598
0.094
M18
0.114
0.667
0.101
M14
0.113
0.675
0.102
M15
0.124
0.738
120
M16
0.133
0.797
0.136
M17
0.136
0.789
0.131
Mis
0.137
0.807
0.136
M19
0.136
0.831
0.135
M
0.136
0.880
0.145
M21
0.150
1.086
0.169
M22
0.153
1.277
0.176
M2a
0.190
2,006
0.197
M24
0.117
0.711
0.201
Table Exact values of Xasp, Kessps and true Xxp for
M.


Reviews:
[{"review_id": "etu0CTBIde", "reviewer": "Reviewer_QnWV", "summary": "", "questions": "- Is your method potentially applicable to LLMs?\n- Will the distribution approximation at L383 significantly affect the probing results?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer QnWV，\n\nThanks again for your affirmation of our research.\n\nWe have included additional experiments and responses to questions 1 and 2 in the official comments. The experimental results align with the findings and conclusions stated in our paper, and by examining the results of the supplementary experiments, we can observe other interesting phenomena. Unfortunately, experiments on Albert-large have not been completed at this time.\n\nHowever, despite the further experimental validation, our intention was not to make strong assertions that the findings and conclusions in the paper hold true for all PLMs. These are ultimately intuitions and not rigorously substantiated. The core of this paper lies in the analysis from sections 3.1 to 3.4 and the ideas we aim to convey in the conclusion. These aspects are rigorous.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation is great, existing probing methods usually can only provide results like \"A is better than B\", and the results are often predictable. This paper proposes a probing method that can know \"Is A or B good enough?\", which can provide additional insights into the model's capability.\n- The paper provides a detailed theoretical analysis and a \"Rethinking process\" on how to induce the self-supervised probe.\n- They proposed the tree-topological probe that is \"upgraded\" from the structural probe, and the new probing method can provide an upper bound and has the potential to be a guideline for \"upgrading\" other probing methods.\n- Authors provide extensive analysis of the probing results and produce findings and speculations.\n- Authors verify their findings by a finetuning experiment, showing that their probing method is useful to identify the capability of each submodule, and enhancing these submodules by hierarchical information is useful. Which makes the whole paper self-consistent.", "reasons_to_reject": "- Would be better by more experiments (as you mentioned, e2,3,4, and other PLMs)\n", "questions_to_authors": "- Is your method potentially applicable to LLMs?\n- Will the distribution approximation at L383 significantly affect the probing results?", "paper_topic_and_main_contributions": "This paper rethinks the probing of PLM representations. \nBased on the motivation that the current probing methods lack an upper bound to help us understand the model capability more clearly, the authors propose a novel self-supervised probe: a tree-topological probe to probe the hierarchical structure learned by BERT. They theoretically justify the bounding relationship of the tree-topological probe and the structural probe and apply the probing method on BERT-large. Based on their probing results, they provide several findings and speculation, and the probing method can help find the submodules that need to be enhanced. They verify these findings by finetuning experiments, their probing method indeed can identify the submodules that need to (or cannot) be enhanced (by structural knowledge).", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L274, given should be in \\text{given}, not in the math format"}, "extra_text": {}}, {"review_id": "AFtSRJJ6dY", "reviewer": "Reviewer_RAmS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer QnWV\n\nThank you for providing many specific suggestions on improving paper writing. We will consider how to make the paper more reader-friendly in terms of readability.\n\nIn response to the technical issues raised in your review comments, we have addressed and included supplementary experiments in the official comment. Lastly, concerning the writing issues, we would like to make some subtle points of defense.\n\nFirstly, it's undeniable that this paper does not present any ethical concerns in terms of theory and methodology. The discussion in the \"Ethics Statement\" section is based on the notion of \"potential societal impact.\" While it's possible that the actual content of this discussion may not perfectly align with the Ethics Statement section, it is not disjointed from the rest of the paper's content. Currently, there are indeed numerous voices expressing concerns about the LLMs. Therefore, it's reasonable to \"inspire further reflections on the advantages of human intelligence compared to the intelligence of these models,\" as stated in the paper. According to the language used in this paper, if the models excel over humans in the sp aspect, could they potentially outperform humans in the essp or ssp aspects as well? Furthermore, what are the distinguishing factors between possessing essp or ssp capabilities and having sp capabilities?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I like the general intuitions you have on the probe as detailed in section 3.1. The work has theoretical contributions and the framework overall could be beneficial to the wider NLP interpretability community. \n\nThe findings in sections 4.1 are interesting and reveal insights previously not available in the structural probe alone. However, it's really difficult to interpret how these findings could apply to the broader community as it is only performed on BERT-large. The subsequent results of fine-tuning layers using insights from 4.1 are convincing.", "reasons_to_reject": "My main worries for the paper are:\n1. The limited empirical evidence of the work given that the main experiments were only conducted on a single model. I think having two or three additional analyses of related models could really strengthen your work.\n\n2. There are a number of places where perturbations/sensitivity analysis could be done to reveal the robustness of the method/results. Calculating error bounds in Table 3, using multiple different datasets, different ways of approximating max(s_w). \n\n3. The writing could benefit from including more motivation throughout the introduction and transition paragraphs. Why is it important study these phenomena? How would this benefit the larger community? I think there are compelling reasons here and the authors should seek to articulate them. \n\n4. The ethics statement feels detached/removed from the rest of the paper. It's important for the ethics section to engage with the ethical harms and considerations of the presented theories and conducted experiments.\n\nDetails:\nThere are a number of places where the authors are making assumptions without citations or sufficient motivation and reasoning.\ne.g., lines 052-059 --- it's not immediately clear to readers why the distinction between contextual and static embeddings is sufficient for researchers to trust a model's effectiveness. \"Researchers' intuitions\" is not defined/a poor motivation.\n\nThere is motivation provided in the introduction that is not answered throughout the paper e.g., lines 075-080. These are not central questions to the paper and would recommend cutting for space and overall tightening the writing of the paper.\n\nThere are also some claims that could be reworded with more hedges/citations as they are not obviously true statements e.g., lines 100-102. There is also a volume of follow-up work to Hewitt and Ethayarajh which should be cited here as well.\n\nThey key intuition in the self-supervised structural probe is a little hidden in the introduction: line 125 \"In the case where the internal constraints of the probed features are well defined, a probe that detects these features can naturally induce a probe that detects the internal constraints, which is self-supervised.\" This should be highlighted earlier in the introduction.\n\nA key challenge which you hint on line 227 is that there are multiple ways of constructing model M. It would be nice to see more evaluation here (ablation studies) to understand how sensitive the probe is to these chosen parameters.\n\nWriting suggestions: \nIt would be nice if you could define min(s_w) and max(s_w) at the same place in section 3.3 rather than defining max(s_w) when you start talking about enhancements. It would also be nice to intuitively describe what each of these sequences represent. From there, you can easily present equations (0), (10), and (12). \n\nHaving language like, \"since\" and \"similarly\" is not sufficient language to transition between your equations line 366. Why is it important to present the following equations? Why do we care about the properties of these sets? The motivation here is lost to the reader and could really strengthen your work.\n\nIn the case where Xssp(M) = Xessp(M), it would be interesting to know how often this happens. You could put a nice metric to how often the structural probe would miss insights only your probe can interpret. Could also consider moving content from lines 389-403 together with this initial definition (could also go in the appendix).\n\nLine 414, need some justification on why you are focused on BERT-large. Is it the most representative model? Currently as it reads, it's difficult to know if this only works on BERT-large or if there are unique properties to BERT-large that make this analysis rich. Do you have insights on what woudl happen with BERT-base/Roberta?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is focused on designing a suite of metrics called the tree topological probe which is used for interpretability of pre-trained language models. Specifically, the paper examines BERT-large and uses their findings to enhance fine-tuning performance. The main contribution of the work is to provide the interpretability community with a self-supervised probe to examine (pre-trained) language models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "This is a nit but equation (1) is really critical to the understanding of the intuition of your paper. It would be helpful to try and be very clear with the language leading up to it, sections 3.1 and 3.2. A simple figure could also help.\n\nLines 307 - 323 are dense to read. It might help to define and set up all the variables first, before jumping into equations. They currently read as sentences which is hard to parse.\n\nLine 382 could be moved to the appendix. Might be nice to have 367 as a separate section that is comparing and contrasting to the structural probe."}}, {"review_id": "A0DhPAObTg", "reviewer": "Reviewer_cpmL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer cpmL,\n\nThank you once again for your comments and the questions you've raised. We have included the supplementary experiments in the official comment.\n\nRegarding the issue you mentioned about comparing probes, it's possible that there might be a misunderstanding about the problem being discussed in the paper and the proposed methodology. This paper doesn't aim to introduce a probe that possesses the same functionality as the structural probe but is superior, but it presents the concept of reasonably constructing upper and lower bounds for metrics. The ssp and essp introduced in the paper are simply instances where we apply this concept to the structural probe, demonstrating that this idea is not only theoretical but also actionable. \n\nThere isn't a direct correspondence between a probe and its upper and lower bound probes in terms of labels. There might be some relationship between them, but it's not yet clear. Therefore, comparing mins_{W} with the actual depth of the dependency tree currently lacks significance. The primary purpose of providing upper and lower bounds for metrics is to further enhance the interpretability of the metrics and to construct theoretical analytical models on metrics that require the use of additional labels for computation. This aspect can be observed from the section 3.4 and Experiment 1.  Moreover, metrics that require the use of additional labels for computation cannot serve as universal regularized loss functions.\n\nRegarding the questions you raised in the \"Typos Grammar Style And Presentation Improvements\" section, we have provided some responses in the official comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides an interesting formal explanation of probing, and specifically the purpose of unsupervised probing\n- The paper proposes a new unsupervised structural probe\n", "reasons_to_reject": "- The paper’s experimental setup is lacking comparison with previous work, and evaluation of the constructed probe\n- The paper’s writing can be greatly improved\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about the interpretability of NLP models, specifically in the domain of probing classifiers for linguistic properties.\nThe authors propose a new approach for designing an unsupervised probe, reminiscent of the probe proposed by Hewitt and Manning for recovering syntactic structure, but unsupervised.\nThe authors define such a probe (again, similar to the design of Hewitt and Manning, by trying to replicate a syntactic tree structure, defined by the depths of the nodes (words/tokens) in the graph) quite thoroughly, discussing the different properties and constraints on such tree, and how to achieve (approximate) it.\n\n*Main Contributions*\n\n- The authors propose a new unsupervised structural probe for analyzing the internal representations of neural language models\n- Using a metric defined for such a probe, the authors propose a method to improve fine-tuning.\n\n*Methodology*\n\nWhile my topology background is not very strong, I did my best to follow the different metrics, steps, and logical steps made in the paper, and it sounds reasonable (although I cannot guarantee that).\n\n*Evaluation*\n\nThe paper conducts two main experiments: \n1. Measuring their probe using BERT-large on different sentences (from different GLUE datasets)\n2. Using their derived metric as an additional loss function during fine-tuning of the model on the linguistic acceptability dataset (CoLA).\n\nThe first experiment was unclear to me. I could not understand the results or the claims that the authors made.\nThe second experiment shows a minor improvement in performance and is not convincing. Moreover, the authors did not perform any statistical significance tests, which makes it hard to take the 1.5 difference very seriously.\nBesides these two experiments, there were no others.\nI don’t understand this lack of comparison, given the wide range of relevant papers in the field (which they also cite).\nHow does their probe compare to others? E.g. Hewitt and Manning\nHow does their probe compare to the “real” syntactic tree of a sentence (using an annotated treebank)\n\n*Writing*\n\nThe paper was very hard to understand.\nIt makes very strong claims, which I often couldn’t understand or interpret. It is very opaque at times. The results and their interpretation as a list of findings (written as fN.) and conclusions (written as cN.), open problems (written as sN.), and preferences (written as pN.) are also nonstandard. Such writing is not problematic per se, but it breaks the flow of the paper and reads as a list of findings, rather than a story, which is much harder to understand. Specific details are given in the “presentation improvements” section.\nOverall recommendation: while I am generally excited about the idea and direction of the paper, I think the current state of writing (and experiments, to some degree) can be greatly improved. I strongly encourage the authors to resubmit the paper in an improved state.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Unsupervised Probing (also with syntactic structure) \n- https://aclanthology.org/2020.acl-main.383.pdf\n\nCausal Probing\n- https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00359/98091/Amnesic-Probing-Behavioral-Explanation-with\n- https://direct.mit.edu/coli/article/47/2/333/98518\n", "Typos_Grammar_Style_and_Presentation_Improvements": "- L.1 LMs are *not* expected to effectively map input text to vectors, this is perhaps an implicit requirement, but their main objective is to be able to predict the next word in a sequence\n- L.3 what are inherent relationships?\n- L.6 what are internal relations?\n- L.10 what do you mean by white-box?\n- L.12 what do you mean by inherent interpretability?\n- L.15 what do you mean by a novel line?\n- L.19 “line of investigation” sounds peculiar in this context\n- L.21 “conducted measurements” sounds peculiar\n- L.23 “propose a speculation”\n- L.24 what is that “working mechanism”?\n- L.28 what are submodules?\n- L.34 “algebraic operation” is very vague\n- L.37 what do you mean by “what-box model” in this context?\n- L.45 “metric computation process usually requires interpretability” - strange phrasing, I cannot make sense of it\n- L.47 “inherent interpretability of the source model is missing” - strange phrasing, I cannot make sense of it\n- L.50 what do you mean by semiquantitative?\n- L.51 what are artificial hypotheses?\n- L.54 what’s the connection here between contextual embeddings and static embeddings?\n- L.56 what does intuition have to do with interpretability?\n- L.62 what do you mean by “complete word”?\n- L.71 what are “complete syntax relations”?\n- L.76 “exist other embeddings” - I don’t see the connection\n- L.79 why do we care about an upper bound?\n- L.117 how is your approach “beneficial for interpretability”?\n- L.148 what are submodules?\n- L.181 what is “Model Evidence Maximum”? \n- The presentation of the equations can also be improved to be more friendly (e.g. pdep -> p-dep)\n- Tables 1+2: they are not clear, and their captions are also very opaque.\n- L.425-434 (e1-e4) unclear experiments\n- L.444 strange formatting (f1.)\n- L.542-546 I don’t understand this claim and why would we want these properties to be linearly encoded in the representations\n- L.607-625. This paragraph is very opaque\n"}}]

Author rebuttal:

Meta-review: This paper proposes a new method (a tree topological probe) to probe syntactic/hierarchical structure learned by BERT in an unsupervised manner. The authors use their findings to also enhance fine-tuning performance of BERT.

Pros: 
- Reviewers found the paper has strong potential and addresses an important problem in the probing area.
- They found the proposed method interesting and appreciated the theoretical contributions.
- They did not have any critique of the method itself, though this may be due to a lack of understanding (see below).

Cons: 
- Multiple reviewers felt that only experimenting on BERT-Large was a limitation. During the author response, the authors provided more results on RoBERTa-Large and Albert-Large to alleviate this concern.
- [**important**] Multiple reviewers highlighted that the paper was lacking clarity & very difficult to read. They felt the paper was missing a core narrative structure to motivate the work, and that the necessary changes would take more than a minor revision to fix.

Reviewers had some disagreement over scores. Taking all of the above into account, I have adjudicated in favor of the 2 reviewers in agreement (but with overall lower scores). I think the paper has some very interesting ideas and seems generally sound, but certain parts are confusing & may need more clarification.
Decision: Accept-Findings