Probing LLMs for Joint Encoding of Linguistic Categories
Authors: Giulio Starace
Konstantinos Papakostas
Rochelle Choenni
Apostolos Panagiotopoulos
Matteo Rosati
Alina Leidinger
Ekaterina Shutova
Venue: EMNLP 2023 Findings (2023)
Keywords: language model
transformer
interpretability
probing
syntax
pos
dependency
multilingual
TL;DR: We find evidence that LLMs jointly encode linguistic categories such as POS tags and dependency relations, across tasks and languages.

Abstract: Large Language Models (LLMs) exhibit impressive performance on a range of NLP tasks, due to the general-purpose linguistic knowledge acquired during pretraining. Existing model interpretability research (Tenney et al., 2019) suggests that a linguistic hierarchy emerges in the LLM layers, with lower layers better suited to solving syntactic tasks and higher layers employed for semantic processing. Yet, little is known about how encodings of different linguistic phenomena interact within the models and to what extent processing of linguistically-related categories relies on the same, shared model representations. In this paper, we propose a framework for testing the joint encoding of linguistic categories in LLMs. Focusing on syntax, we find evidence of joint encoding both at the same (related part-of-speech (POS) classes) and different (POS classes and related syntactic dependency relations) levels of linguistic hierarchy. Our cross-lingual experiments show that the same patterns hold across languages in multilingual LLMs.

Probing LLMs for Joint Encoding of Linguistic Categories
Giulio Starace* Konstantinos Papakostas', Rochelle Choenni',
Apostolos Panagiotopoulos Matteo Rosati', Alina Leidinger' and Ekaterina Shutova' 'Inttitte for Logic. Language and Computation, University of Amsterdam
Abstract
Existing research probed LLMs for their en- coding of various linguistic properties such as agrecment (Jawahar et al., 2019), word order and sentence structure (Tenney et al., 2018; Hewitt and Manning, 2019), co-reference (Tenney et al.. 2019), semantics (Ettinger, 2020) and multilin- guality (Ravishankar et al., 2019; Libovický et al., 2020). Taking step further, Tenney et al. (2019) and Clark al. (2019) studied where linguistic in- formation is encoded in LLMs by probing different layers. Their results demonstrated that a linguistic hierarchy emerges in BERT representations, with lower layers capturing local syntax and higher lay- ers being employed in higher-level semantic and discourse tasks. However, we do not yet understand how encodings of different linguistic phenomena interact within the models and what extent pro- cessing of linguistically-relate categories relies on the same, shared model representations.
Large Language Models (LLMs) exhibit im- pressive performance on a range NLP tasks, due to the general-purpose linguistic knowledge acquired during pretraining. Ex- isting model interpretability research (Tenney et al., 2019) suggests that linguistic hierar- chy emerges in the LLM layers, with lower layers better suited to solving syntactic tasks and higher layers employed for semantic pro- cessing. Yet, little is known about how encod- ings of different linguistic phenomena interact within the models and to what extent process- categories relies
ing of linguistically-relan on the same, shared model representations. In this paper. we propose a framework for testing the joint encoding of linguistic categories in LLMs. Focusing syntax, we find evidence jointenn encoding both at the same (related part- of-speech (POS) classes) and different (POS classes and related syntactic dependency rela- tions) levels of linguistic hierarchy. Our cross- lingual experiments show that the same patterns hold across languages in multilingual LLMs.
There are many dependencies between process- ing different linguistic phenomena: for instance, in- formation about a word's part of speech is likely to be employed when disambiguating its word sense. Alternatively, lower-level syntax is an important first step for semantic composition and natural lan- guage understanding tasks. In this work, we inves- tigate how the (hierarchical) dependencies between different linguistic categories are encoded LLMs. focusing on syntax. We ask set novel questions: (1) how related syntactic categories (e.g. different parts of speech (POS), such as noun or verb) are encoded within the models; (2) how syntactic cate- gories at different levels of the linguistic hicrarchy (e.g. POS classes and syntactic dependency rela- tions) interact within the model; and (3) whether the observed patterns hold across languages.
1 Introduction
Recent advancements in natural language process- ing (NLP) can be attributed the development and pretraining of large language models (LLMs) such as BERT, GPT-3, and many others (Devlin et al., 2019; Brown et al., 2020; Touvron et al., 2023). For their intended use of providing general-purpose lan- guage representations suitable for many NLP tasks, these models must efficiently capture wide range of linguistic features within their finite capacity. Despite their success, little is known about the way in which different types of linguistic information are organized in these models. Systematically un- derstanding how these models represent linguistic phenomena and their interaction crucial for the development of more effective NLP methods.
To answer these questions, we propose a frame- work for testing the joint encoding of linguistic categories in LLMs. Specifically, we investigate to what extent the information about distinct lin- guistic categories is shared in the parameters of the
Corresponding author: giulin @maill.omm..mmmmmmmmmmmmm

---

Slap Representation dissification Classify representations from encoder using probing network
Step
Step Croesneuralicatoon
Group representations and aggregate into class- representative centroids. The oud bepws to.." trooft Thath cat
Subtract centroids from incoming centroids before re-classification. aewer
quio aopuntoc...
Enodeer
Extract repr.s from ah
While validating, keep track of which repe predict which classses.
A "neutralizer" centroid is sublracted from incoming repr. is before downstream classification.
layer of a pretrained
Layer Liog
encoder.
Probiin Natworkk
Repeat classification an the "neutralized" repris probing network.
Classity repr.s using the probing network.
Group repr.s by their classs predictions.
VERa
NGUN ADU
Vine
VERG
Drops in porf. in a particular target class suggest that the neutralizer target are jointly encoded.
Compute class- representative centraids by averaging repr.s.
Measure accuracy on test sel.
V210
V1-0
Figure A diagram illustrating the three steps of our method: 1) Representation classification. 2) Centroid estimation. 3) Cross-neutralizztion. The complete methodology is outlined in Section 3.
model. Our approach (see Fig. 1) is inspired by the work of Choenni and Shutova (2022), who studied how LLMs share information across languages. We
tasks different levels of the linguistic hierarchy.
Background and related work
employ their nrossnneutralization method, but ex- tend to study how information is encoded across linguistic categories in two syntactic -part of- of-speech (POS) tagging and dependency (DEP) labeling. In short, we test whether removing infor- mation syntactic category results in failure process another, related one. For instance, would removing a representation of the verb class hurt the model's s ability to identify the verbs" syntactic de- pendencies, while still succeeding in this task on other categories, e.g. nouns? This provides insight into whether these categories are jointly encoded.
A wide range of methods have emerged to study the inner workings of neural networks (Belinkov and Glass, 2019; Madsen et al., 2022). Our approach is situated within the field of probing (Conneau et al., 2018), which typically involves the use of a simple auxiliary "probing" network trained to per- form a specific task on the representations from a pretrained LLM. By keeping the probing network shallow and the pretrained model frozen, the pre- dictions can be used to identify which information was already captured by the pretrained LLM.
We study joint encoding patterns within both monolingual and multilingual LLM, namely RoBERTa (Liu et al., 2019b) and XLM-R (Con- neau et al., 2020). For the latter, we focus our analysis on English, Greek, and Italian, investi- gating to what extent multilingual models encode information for POS tags and DEP relations a language-agnostic manner. We find that POS tags that are linguistically related are indeed jointly en- coded by both the monolingual and multilingual models, and observe similar joint encoding pat- terns across all three languages. Moreover, we obtain further evidence of both kunguage-agnostic and language-specific encoding within multilingual models, given that representations specific to POS tags and DEP relations can be approximately trans- ferred across languages without a substantial im- pact on performance. Lastly, we find evidence of joint encoding between related POS tags and DEP relations, suggesting information sharing across
Previous work on probing Shi et al. (2016) are the first to introduce the idea of probing, training a logistic regression classifier on top of the em- beddings from two neural machine translation en- coders to study the syntactic information learned by them. Similarly, Adi et al. (2017) probe sen- tence representations for sentence length and word order by training auxiliary task classifiers pre- dicting these exact properties from the represen- tations. Continuing on this work, Conneau et al. (2018) introduce suite of probing tasks. Ten- ney et al. (2019) apply these to BERT's hidden states, quantifying where linguistic information is captured within the network. They find that the model follows the hierarchy of the traditional NLP pipeline, with lever-level, syntactic features ap- pearing earlier than more complex semantic roles and structures. Ravishankar al. (2019) bring the same analysis to the multilingual setting. Dalvi et al. (2021) find more supporting evidence of this

---

through an unsupervised approach. Manning et al. (2020) probe the attention mechanism in BERT for correspondence between linguistic phenomena such as syntactic dependencies and coreference. They also make use of structural probes (Hewitt and Manning, 2019), finding that the representa- tions from BERT capture parse tree structure.
RoBERTa
XLM-R
en_gum
en_gum
vLv
el gd
POS DEP
95.6%
95.5%
97.9% 97.4%
90.9%
91.4%
93.9%
94.8%
Table Classification accuracy of our probing classi- fiers on English, Italian and Greek datasets for part-of- speech tagging and dependency labeling.
Syntactic knowledge in BERT Various ap- proaches have been used to study what knowl- edge LLMs capture about syntax (Rogers et al., 2021). Previous probing studies have shown that BERT embeddings encode information about parts of speech, syntactic chunks, constituent and de- pendency labeling (Tenney al., 2018; 8; Liu et al., 2019a) as well as broader set of syntactic fea- tures, such tree depth and tense (Conneau et al.. 2018). Yet, while has become evident that syntax is captured to some extent, less is still known about where and how this information is acquired. Some works suggest that syntactic information is encoded in the token representations as they can be used to successfully reconstruct syntactic trees (Vilares et al., 2020; Kim et al., 2019; Hewitt and Manning, 2019). Others have instead studied syntactic knowl- edge the level of attention heads, and show that particular heads specialize to specific aspects of syntax (Htut et al., 2019; Clark et al., 2019). How- ever, Htut et al. (2019) find that these heads can not recover syntactic trees, suggesting that atten- tion heads do not reflect the full extent of syntactic knowledge that these models learn.
Shutova (2022), who probe for joint encoding of typological features across different languages. In particular, they probe LLMs for typological lan- guage properties and test whether subtracting lan- guage "centroids" from model representations neg- atively affects performance in typologically-related languages. While they focus on representations of specific languages, we target the representations of linguistic categories (c.g. nouns), and test whether these are jointly encoded across classes and tasks.
POS tagging and dependency labeling We study the joint encoding of POS categories and syntactic dependencies. Given a sentence, POS tagging is the task of mapping each word to the appropriate part of speech. For instance, in "The sailor dogs the hatch.". "dogs" is verb, while in "He chases the dogs", "dogs" is noun. Depen- dency labeling is the higher-level task of labeling the dependency relation between "head" word and a "dependent" (or "parent" and "child"). For instance, in "That black car.", "black" an adjectival modifier of "car".
Information sharing in LLMs Multiple works study information sharing in LLMs (Blevins et al., 2018; Şahin et al., 2020), with most focusing on cross-lingual sharing (Chi et al., 2020; Shapiro et al., 2021; Stanczak et al., 2022). Libovický et al. (2020) propose simple method that removes language-specifie information from model repre- sentations by capturing it through the mean of set of representations from the respective language. Ravfogel et al. (2020) iteratively remove gender information in word embeddings through projec- tion onto the nullspace (INLP) in order to mitigate bias in biography classification (De-Arteaga et al., 2019). Elazar et al. (2021) use INLP to build coun- terfactual representations for "amnesic probing". Here, the utility of property for given task is es- timated by measuring the influence of removing the property via INLP, treating the removal as a causal intervention. Our work inspired by Choenni and
3
Methodology
3.1 Probing
To study the representations from a given encoder network, we train shallow classifier, or "probing classifier" on probing task, using the representa- tions from the encoder as input. By keeping the encoder frozen, the relatively inexpressive probing classifier, allow- ing us to probe whether the representations from the encoder contain the information necessary to solve the task at hand.
ascribe all the learning
3.1.1 Probing tasks
We focus on POS tagging and DEP labeling as our probing tasks. For both cases, our encoders take sentences input, producing token-level em- beddings each layer. We extract the token-level embeddings from a given layer and pool them into word embeddings (detailed in Section 3.5). For

---

POS
DEP
768, with total parameter count of 125 million".
Encoder /t ttrebaan
Layer
Aggr.
Layer
Aggr.
Probing classifier For our probing classifier, we use a multilayer perceptron (MLP) consisting of two linear layers with tanh activation function in between. We train our probing classifier using the AdamW optimizer (Loshchilov and Hutter, 2019) with learning rate of 10 and weight decay of 10-2 and employ an early stopping criterion based on the validation set. We report the classification accuracy for the best configurations in Table
RoBERTa/ gun
3
XLM-R en_gum
9
first
XLM-R vit
9
first
mean
XLM-R / cl gdt
12
Table 2: Optimal combination of embedding layer and subword pooling function for each encoder (RoBERTa & XI., task (POS & DEP) and language (English, Italian& Greek) combination, chosen outlined in Sec- tion 3.5 and used throughout our experiments. When neutralizing across languages (Section 5) and across tasks (Section 6), we use the configuration of the neu- tralizer for both neutralizer and target.
3.2 Probing Classifier Selectivity Baseline
A sufficiently expressive probing classifier may be capable of learning any task given representations as input. This would render its probing functional- ity obsolete since the information localized by the probing classifier would no longer be necessarily attributed solely to the input representations. To ensure that our probing elassifier is not overly ex- pressive, we conduct a baseline check as outlined by Hewitt and Liang (2019). Here, we construct an analogous control task for POS tagging, where we randomly assign each word in the training set to one of N arbitrary labels, where N is the number of POS tag labels (i.e. 17). A good probe should have high selectivity, computed as the difference between the accuracy on the probing task and the accuracy on the control task. The more selective the probe, more likely it is that the information it accesses is specific to the input representations. If the probe performs equally well on the control task, it suggests that the probe may be leveraging some inherent properties of the model architecture or the data distribution, rather than specifically extracting useful information from the representations. We train a new checkpoint of our probing classifier on the control task, ending training after the same number of update steps performed when training on the POS tagging task, and report the outcome in Section 4.
POS tagging, the probing classifier receives the word embeddings and is trained to classify them across 17 categories. For DEP labeling, we aim to label the dependency between child and parent. Thus, we pair each word in the sentence with its head as labeled in the dataset and concatenate their representations'. The probing classifier takes this concatenation input and is trained to classify the dependency relation between the corresponding words across 36 categories.
3.1.2 Datasets
We use the Universal Dependencies treebanks (Nivre et al., 2020), manually annotated for POS tagging and DEP labeling. We choose the GUM (Zeldes, 2017), VIT (Delmonte et al., 2017), and the GDT (Prokopidis and Papagcorgiou, 2017) datasets for English, Italian, and Greek respectively. All datasets contain word-level annotations with total of 17 POS tags and 36 DEP relations shared across languages?.
3.1.3 Models
Encoders We use RoBERTa (Liu et al., 2019b) and XLM-R (Conncau et al., 2020) as the encoders we probe. RoBERTa is an optimized version of the encoder-only transformer BERT (Devlin et al., 2019), and XLM-R is its multilingual variant, sup- porting 100 languages. Unlike BERT, both models are trained exclusively on the masked language modeling (MLM) objective. We use the "base" version of each model, comprising of 12 encoder layers, 12 attention heads and an embedding size of
3.3 Centroid estimation
To study the joint encoding of linguistic categories (e.g. nouns, verbs, etc.) in our encoders, we use the probing classifier to localize the subspace of the en- coder that corresponds to each of these categories by obtaining their mean vector representation, or "centroid"" similarly to the language centroids of
'Different methods such as adding the mean vector or absolute difference of the pair resulted in similar performance. "We summarize the split sizes and an overview of the pre- processing pipeline for all three languages in Appendix A.
We report the results of brief parameter scaling experi- ment in Appendix B.

---

Libovický al. (2020). The intuition is that repre- sentations that repeatedly result in predictions of a particular class will encode information specific to that class, which can be captured through ag- gregation such as averaging. For POS tagging, the centroid uy of each target POS class t is defined as
suggesting that some features of the encoder play a role in encoding both categories. As baseline, we additionally experiment with subtracting random vectors (rather than centroids) from the encoder representations. the centroids are indeed respon- sible for the drop in performance, then we should not observe similar performance drops when sub- tracting random vectors. We report the outcome of this experiment in Section 4.
(POS) u
1
Σ v VEV.
(1)
3.5 Choosing a configuration for probing
where V/ is the set of the representations of the words in our validation set that were predicted as tag t when probing the /th layer of our the encoder. For DEP labeling, each prediction depends on both the representation of the head h and the child с of the DEP relation. The centroid of each target DEP relation is computed as:
Understanding where to best localize the task- specific information within the models is not trivial. For instance, while Tenney et al. (2019) show that syntax information is more localized in lower lay- ers, de Vries et al. (2020) demonstrate that such findings do not automatically port to multilingual models. Moreover, Del and Fishel (2022) show that for multilingual models, the pooling method used can have important effects on the knowledge that is captured. Thus, we first study which layer and pooling function maximizes the amount of information that is captured in our centroids. As such, we employ the notion of self-neutralizaatin, the case in which the same linguistic category is both the target and the neutralizer.
(DEP) u1
Eh: hd (b.d)c/?
(2)
IP/|
where P/ is the set of (head, dependent) represen- tation pairs that were predicted DEP relation t when probing the 1-th layer of our encoder, and h :h:d d] is the concatenation of the representations.
3.4 Cross-neutralization
We hypothesize that a large drop in accuracy after self-neutralization is indicative of high amount of relevant information being captured by the centroid. This drop should be relative to a high accuracy. We thus select the top quartile of our configurations in terms of original-encoder accuracy, and from this subset use the configu- ration with the highest relative drop due to self- neutralization. We consider representations from layers (1, 6, 9, 12) and pool subwords by either taking the first token in word, or by max- pooling or mean-pooling over the token represen- tations. For each encoder, language and task, we separately find the optimal probing configuration and report them in Table 2. We make use of the validation set for this portion of the methodology.
To study whether different linguistic categories are jointly encoded within LLMs,, we use a cross- neutralization method. We first evaluate the class- specific accuracy of our probing classifier on the original representations of our encoder. We then take a neutralizer centroid estimated outlined in Section 3.3, and subtract it from the encoder representations corresponding to some target cat- egory. For instance, taking verbs as neutralizer and nouns as target, we subtract all verb information from the noun representations. We then repeat prob- ing classification on these "ncutralized" encoder representations. The intuition behind this is that if the encoders were to represent linguistic categories in independent ways, we expect the performance deteriorate only for the linguistic category that we use for computing the centroids, i.e. remov- ing verb information only negatively affects the representations of verbs. However, in the case of joint encoding, we expect to see substantial perfor- mance drops for other target categories as well,
4 Joint encoding of POS tags
We begin by examining whether representations of different POS tags share information. For each POS tag, we compute its centroid and re-classify representations neutralized by the centroid. With regard to our basclines, we find that subtracting random vectors leaves the performance relatively unchanged, with evidence of systematic drops.
*We test the quality of the predicted centroids by comput- ing gold centroids based on the gold labels, and computing cosine similarity between the two. We find that they are near identical (~1 similarity), see Appendix C.

---

hypothesis that joint encoding for closed-class neu- tralizers may be explained by their functional role, which often not reciprocal (verbs will generally not modify auxiliaries).
Target
NOUN
VERB
PRON
DET
ADV
AUX
NOUN
-30
-21
-10
-13
-8
-32
-32
-15
While we observe further evidence of linguis- tically related categories being jointly encoded (e.g. VERB and ADV), this is not ubiquitous. For instance, joint encoding is not found when neutral- izing nouns (NOUN) with determiners (DET). Here, the relative percentage decrease only 2%, despite determiners acting as modifiers for nouns. In gen- eral, we do not find distinct patterns of joint encod- ing unique to closed-class and open-class words. Our hypothesis of symmetry due to co-occurrence is at best supported by the ADV-ADJ pairing, with relative drops of -21% % and -29%. In most other pairings information sharing appears to be asym- metric, suggesting functional dependence or an- other root cause as explanation.
-17
VERB
-6
-14
-80
-5
-3
-15
-19
-11
Neutralizer
PRON
3
-1
-6
2
-9
DET
-12
-21
NUM
6
-10
-3
-8
-3
-94
-13
-1
ADV
2
-14
AUX
3
-12
-53
-2
6
-11
-89
Figure 2: Relative change in accuracy when cross-
neutralizing POS tags (RoBERTa).
For our selectivity baseline, we find that our prob- ing classifier achieves an average accuracy of 96% on the POS tagging task and 63% on the control task, giving selectivity value of 33%. This is in line with probes for similar tasks accepted for their validity in the literature (Hewitt and Liang, 2019). To facilitate visualization and discussion, we select an illustrative subset of POS tags that cov- ers both open- and closed-class words, as we ex- pect potentially different patterns between these two groups. For instance, we expect neutraliza- tion to be symmetric (neutralizer and target can be swapped for similar results) in the case of open- class words as we hypothesize that joint encoding here is dictated by co-occurrence patterns. Con- versely, we expect asymmetric neutralization in the case of close-class neutralizers, supposing that joint encoding here may be dictated by functional dependencies, which may not necessarily be recip- rocal.
Moreover, we note that open-class words tend to be "adept" at cross- proossneutrallzing. sharing informa- tion as neutralizers with many other tags as shown by the NOUN, ADJ and VERB neutralizers rows suc- cessfully neutralizing several columns. However, note similar patterns when neutralizing with the closed-class DET. Overall, we find evidence of joint encoding of POS tags in RoBERTa, but further work necessary to establish which mechanisms lead to specific pairs sharing information or not.
4.1 Joint encoding in multilingual models
We now investigate how our findings transfer to other languages. We examine to what extent mul- tilingual models demonstrate language-specific or language-agnostic information-ssaring behavior. Additionally, we verify the consistency of our method across models as we can directly compare the results between RoBERTa and XLM-R in En- glish, and also across languages, as we can compare the effects of cross-neutralization on English, Ital- ian, and Greek. Our setup is the same as in Section 4, replacing RoBERTa with XLM-R and repeating the experiment in the three languages.
Results We find that linguistically related cat- egories generally tend to be jointly encoded in RoBERTa (see Fig. 2). For instance, the relative decrease of VERB classification accuracy by 53% when neutralizing using auxiliaries (AUX) suggests information sharing. This may be explained by the fact that auxiliaries themselves are verbs (e.g. "has done"), and that they functionally modify verbs. We note that the information flow in this pairing is asymmetric, as noted by the much smaller 11 11% drop in the reciprocal case. This aligns with our
Monolingual
multilingual model When comparing RoBERTa and XLM-R on the English dataset, we observe similar patterns across POS tags (see Fig. and the left-most column of Fig. 3). For instance, for both models, we see that nouns have a neutralization effect on adjectives, numerals, and adverbs. This suggests that given language, multilingual models jointly encode POS tags simi-
'See Appendix for results between all classes.

---

Target
Target
Target
NOUN
ADJ
VERB
PRON
DET
NUM
ADV
AUX
NOUN
ADJ
VERB
PRON
DET
NUM
ADV
AUX
NOUN
ADJ
VERB
PRON
DET
NUM
ADV
AUX
NOUN
-92
29
-15
-13
-12
-39
-25
-20
-08
-25
-11
-16
-3
-39
22
-20
-69
-34
-5
-16
-1
-88
-0
-18
91
Neutralizer
VERB
-10
-25
.94
-7
-33
-17
-13
-33
43
-66
-9
-2
43
20
-10
-18
-18
90
-9
41
-9
-12
PRON
.1
-15
DET
-1
-8
-18
-93
-19
-15
-6
-7
-22
-94
-35
-8
-23
-88
-27
-66
-48
-18
-16
NUM
-13
-18
-8
-84
-33
-4
47
-12
-18
-95
-27
-25
-17
-23
-0
-01
-10
ADV
22
-2
-7
97
-7
-7
-7
94
-8
-11
0
-44
AUX
-12
:59
-3
-22
-9
.94
-18
.80
-7
-2
-18
-5
-66
-3
-2
46
-5
0
-28
-16
-97
English
Italian
Greek
Figure Relative change in accuracy when cross-neutralizing POS tags using embeddings from XLM-R in (left-to-right respectively) English, Italian and Greek, labeled by their universal dependencies treebank.
larly their monolingual counterparts.
languages, this is evidence that the models learn at least partially language-independent representa- tions for these categories. We test this hypothesis by cross-neutralizing every linguistic category in language A with another category from a lan- guage B, e.g. neutralizing all Italian POS tags with English nouns. Aside from probing for language- agnosticism, this allows for further exploration of joint encoding of linguistic categories. To enable consistent neutralization, we use the same layer and pooling function configuration (see Section 3.5) for both neutralizer and target. We do this by utilizing the neutralizing language configuration for both neutralizer and target in cases where they differ.
Joint encoding within languages We find that XLM-R exhibits similar sharing patterns within different languages (see Fig. 3). For instance, we see that adjectives consistently neutralize adverbs across all languages (-31%, -44% and -40% in En- glish, Italian and Greek). This suggests that the rep- resentations responsible for predicting these tags contain some language-agnostic information. However, we also observe some language- specific behavior. For instance, VERB categories cross-neur adjectives (ADJ) more prominently in Italian (-63%) than in English (-26%) and Greek (-18%). This may be explained by the postnominal use of adjectives in Italian which may overlap more closely with the positioning of verbs, particularly in the past participle tense (Cinque, 2010) (in con- trast to the strictly prenominal use in the other lan- guages). The results suggest that XLM-R discovers and leverages language-agnose information when possible, while also learning language-speciin in- formation when necessary.
Results Fig. shows results of cross-neutralizing for Italian POS tags when using the corresponding English centroids from each linguistic category. On the diagonal, neutralizer and targets correspond to the same linguistic category, but for different lan- guages. Accuracy drops substantially here (~ 70% on avg.). This is evidence of language-agnoste en- coding of these linguistic categories, centroids in language neutralizing targets in another lan- guage points information in the representations being shared across both languages.
5 Information sharing across languages
Our findings for individual languages on XLM-R raise the question of whether information about linguistic categories from two different languages can also be jointly encoded. To further probe for language-agnosee joint encoding of linguistic cat- egories, extend our experiment test whether information between two linguistic categories from two different languages is jointly encoded. We posit that categories share information across
We also observe joint encoding different linguistic categories, for instance with English NOUNs neutralizing Italian numerals (NUM) (-48%), or English adjectives (ADJ) neutralizing Italian ad-
"Note that because the linguistic categories are now dif- ferent between neutralizer and target, the diagonal no longer represents self neffrneutallitiion and is simply an artifact of the ordering of the categories to favor legibility.

---

DEP Target
Italian Target
ADVMOD
NOUN
ADJ
VERS
PRON
DET
NUM
ADV
AUX
AMOD
CASE
ADVCL
NSUBJ
DET
CC
COP
OBJ
NOUN
-79
-23
-18
-13
-2
-48
-23
-29
ADV
-97
-26
2
-1
1
-7
-2
-14
2
-
Engilsh Neutralizer
VERB
-22
-67
-5
-2
41
-13
-18
AUX
-10
-3
93
-1
-1
-29
-2
+7
-8
-11
POS Neutralizer
PRON
8
-3
-73
-8
24
-7
-12
-2
4
ADP
-19
-3
4
-1
-86
24
-3
-9
3
-2
-35
VERB
9
-7
0
-3
-6
NUM
-18
-26
-10
-11
-1
-78
-25
-23
CCONJ
-15
-6
-0
-7
-1
-23
99
22
-12
NOUN
-14
-1
-1
-0
4
-13
-3
-33
-10
-6
AUX
-10
-32
-5
-1
-18
-8
-52
PRON
-0
SCONJ
28
-5
-6
2
23
-17
-3
-2
-0
-0
Figure 5: Relative change in accuracy when cross- neutralizing XLM-R embeddings in Italian using cen- troids from English.
POS Target
VERB
CCONU
NOUN
PRON
SCONJ
ADV
ADJ
AUX
DET
ADP
ADVMOD
-2
to draw solid conclusions. For more detailed plots refer the reader Appendix F, where we present results from all combinations of neutralizer and target across the three languages.
AMOD
288
-85
-11
33
-
11
-5
-7
-10
21
cross-lingual cross-neutralizatien
-13
-3
-1
-2
DEP
-44
-16
-5
-88
-6
-1
-0
-2
-12
-19
CASE
-25
-7
-44
-3
-7
38
ADVCL
-16
-13
-13
-4
62
-4
-5
-17
Information sharing across tasks
DEP
CC
-8
-4
-0
-3
-3
-2
-92
-1
20
To study whether information is shared across the linguistic hierarchy, we cross-ncutralize between tasks using POS neutralizers and dependency rela- tion targets. Since given parent may have several dependents while the child will only have one par- ent, we posit that most of the information for the child-pan pareent dependency is captured by the child. We therefore subtract the POS centroids from the child representations in the child-parent concatena- tions for dependency labeling. We also complete the complementary experiment of subtracting the child portion of the dependency relation centroids from the POS embeddings for POS tagging, to test whether joint encoding happens in both directions. As in Section 5, we use the neutralizer configura- tion both neutralizer and target for consistency.
COP
-18
-14
65
-5
-64
-0
-2
OBJ
-11
-15 -11
-3
-3
-7
-3
-96
-4
-11
NSUBJ
Figure 4: Relative change in accuracy cross-neutralizing RoBERTa DEP representations using RoBERTa POS centroids (top) and vice versa (bottom)
verbs (ADV) (-47%). An alternative explanation to language-agnostic representation learning XLMM R may be that instances of code-switching or lan- guage borrowing in the pretraining data may have encouraged XLM-R to jointly encode different lin- guistic categories across languages. Future work may investigate why joint encoding occurs across different categories of different languages.
Results Fig. 4 (top) shows the results when neu- tralizing RoBERTa representations for DEP labels using POS centroids. We find further evidence of linguistically related units being jointly encoded. For instance, adverb POS tags (ADV) neutralize ad- verbial modifier (ADVMOD) dependency labels, can be seen by the 97% relative drop accuracy. The fact that POS tag representations jointly con- tain information that crucial for encoding DEP
We repeat the experiment using English neu- tralizers on the Greek corpus, and observe sim- ilar but milder trends. This may be due to En- glish being phylogenetically closer to Italian than Greek (Chang et al., 2015), leading to higher de- gree of information sharing between the former two languages, but further work in determining which language pairs share more information is needed

---

DEP Target
DEP Target
DEP Target
ADVMOD
ADVMOD
ADVMOD
AMOD
AUX
DET
CASE
ADVCL
CC
COP
OBJ
NSUBJ
AMOD
AUX
DET
CASE
ADVCL
CC
COP
OBJ
NSUBJ
AMOD
AUX
DET
CASE
ADVCL
CC
COP
08J
NSUBJ
ADV
.22
-19
2
3
2
-91
8
4
-18
-
4
-89
7
4
0
11
-2
1
0
ADJ
16
16
4
POS Neutralizer
AUX
-0
-
81
2
57
-7
9
98
53
73
-88
5
81
31
-8
38
DET
-9
-6
2
92
-1
-1
-7
-8
-7
3
2
83
34
0
-1
-14 -7
-3
88
92
21
-1
0
-9
ADP
-16
-3
-1
-74
3
-4
-11
-2 -6
-9
99
15
-0
72
20
-2
-3
-7
-30
33
-1
-9
-9
-2
-3
-7
VERB
10
2
40
2
-1
-5
-19
2
90
-3
2
-3
5
2
63
88
-11
CCONJ
95
NOUN
-16
-8
-12
PRON
-2
-
4
20
-1
-0
22
-66
-0
-6
2
36
-0
-6
0
-1
2
10
-5
SCONJ
-23
2
4 -0
-1
4
2
-0
-8
-11
-10
-3
-56
-7
3
88
-9
-39
2
-1
4
2
-1
-2
0
-0
-2
English
Italian
Greek
Figure 6: Relative change in accuracy sample of dependency relations when rrutsrneutaliizing XLM-R DEP embeddings using XLM-R POS centroids in (left-to-right respectively) English, Italian and Greek.
labels shows that LLMs learn hierarchically, remi- niscent of the classical NLP pipeline.
monolingual (RoBERTa) and multilingual (XLM- R) case. Lastly, we cross-neutralize between POS tagging and DEP labeling, and find evidence information sharing across the linguistic hierarchy. We test specifically for joint encoding of different syntactic categories that rely on the same linguistic concept, such ing some relationship with the adjective and the noun POS tags. However, our method could be extended to test for joint encoding of categories other tasks that can be expected to share informa- tion. This in turn could be informative on whether an LLM indeed captures this shared information between different tasks. A more complete map of this knowledge could help develop better mod- els in many application scenarios: for instance, in multitask learning setting where negative inter- ference between parameter updates from different tasks is known to hamper performance (Zhao et al., 2018) or a lifelong learning setting where learning new task often Icads to (catastrophic) forgetting of the previously learned ones (Biesialska et al., 2020). Future work may additionally seek to ex- plore more languages, particularly low-resource and typologically distant languages, or consider including higher-level semantic tasks from the clas- sical NLP pipeline, such as semantic role labeling. word sense disambiguation and coreference resolu- tion. Further work is also necessary for understand- ing why certain representations share information while others do not.
To study whether our findings generalize beyond English and RoBERTa, we repeat the experiment on XLM-R (see Fig. 6). Large drops in accuracy on the diagonal suggest that many information- sharing patterns hold across languages and models, consolidating our findings from sections 4.1 and 5. We also find taggaaee-speciffe results: pronouns (PRON) neutralize nominal subjects (NSUBJ) only in English (-56%), having surprisingly little effect on the other languages (-5% and -2%). On occasion, we find evidence of bidirectional sharing between different levels of the linguistic hierarchy. In Fig. 4 (bottom), note that certain pairs highlighted by the diagonals such adpositions (POS: ADP) and case relations (DEP: CASE) present evidence of joint encoding both when neutralizing DEP with POS centroids (-55%) and when neutralizing POS with DEP centroids (-84%). This seems to suggest that the hicrarchical nature of the representations learned by these LLMs is not necessarily unidi- rectional: information appears to be shared both upwards and downwards in the linguistic hierarchy.
"amod" dependency imply-
7 Conclusion
We study information sharing between linguistic categories in LLMs, finding evidence of joint en- coding between pairs of related POS tag classes. By applying method to XLM-R, idence of joint encoding in XLM-R across lan- guages, showing that our findings hold for both the
find ev-

---

Limitations
Will Chang, David Hall, Chundra Cathcart, and Andrew Garrett, 2015. Ancestry-Constrainee Phylogenetic Analysis Supports the esis. Language, 91(1):194-244.
In this work, we limit our experiments to an encoder-only architecture. Further research could be carried out on encoder-decoder or decoder-only architectures.
Steppe Hypoth-
Ethan Chi, John Hewitt, and Christopher D. Manning. 2020. Finding Universal Grammatical Relations in Multilingual BERT. ACI.
Additionally, our experiments only examine three languages. Future work may therefore aim to extend the experiments include more languages, particularly low-resource languages. Similarly, our discussion of the linguistic hierarchy is limited to two tasks from the early stages of the classical NLP pipeline. As mentioned, further work may extend the experiments to higher-level semantic tasks. Lastly, the lack of cross-neutralizzatinn effects between some linguistic categories does not neces- sarily indicate the absence of joint encoding, but merely that our representations were not sufficient to prove its existence.
Rochelle Choenni and Ekaterina Shutova. 2022. In- vestigating Language Relationships Multilingual Sentence Encoders Through the Lens of Linguistic Typology. Computational Linguistics, pages 1-38.
Guglielmo Cinque. 2010. The Syntax of Adjectives: A Comparative Study. The MIT Press.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019, What Does BERT Look at? An Analysis of BERT's Attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Com- putational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised Cross lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
References
Yossi Adi, Einat Kermany. Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In International Conference on Learning Rep- resentations.
Yonatan Belinkov and James Glass. 2019, Analysis Methods in Neural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into single S&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 2 2126-2136. Melbourne, Aus- tralia. Association for Computational Linguistics.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual Lifelong Learning in Natural Language Processing: A Sur- vey. In Proceedings of the 28th International Con- ference on Computational Linguistics, pages 6523- 6541, Barcelona, Spain (Online). International Com- mittee on Computational Linguisties.
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Dur- rani, Jía Xu, and Hassan Sajjad. 2021. Discovering Latent Concepts Learned in BERT. In International Conference on Learning Representations.
Terra Blevins, Omer Levy, and Luke Zeutlemoye. 2018. Deep RNNs Encode Soft Hierarchical Syntax. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume Short Papers), pages 14-19, Melbourne, Australia. Association for Computational Linguistics.
Maria De-Arteaga, Alexey Romanov, Hanna Wal- lach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fair- ness. Accountability, and Transparency, EAT* '19, page 120-128, New York, NY, USA. Association for Computing Machinery.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005. 14165 [cs].
Wietse de Vries, Andreas van Cranenburgh, and Malv- ina Nissim. 2020. What's so special about bert's layers? a closer look at the nlp pipeline in mono- lingual and multilingual models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4339-4350

---

Maksym Del and Mark Fishel. 2022. Similarity of sentence representations in multilingual Ims: Resolv- ing conflicting literature and case study of baltie languages. Baltic Journal of Modern Computing, 10(3):337-347.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguisties.
Rodolfo Delmonte, Antonella Bristot, and Sara Tonelli. 2017. VIT Venice Italian Treebank: Syntactic and Quantitative Features.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagu- nas, Alexander M. Rush, and Thomas Wolf, 2021. Datasets: A Community Library for Natural Lan- guage Processing.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing; Behavioral expla- nation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:1 175.
Jindřich Libovický, Rudolf Rosa, and Alexander Fraser, 2020. On the Language Neutrality of Pre-trained Multilingual Representations. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020, pages 1663-1674, Online. Association for Computational Linguistics.
Allyson Ettinger. 2020. What BERT Is Not: Lessons from New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics, 8:34-48 48.
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah Smith. 2019a. Lin- guistic Knowledge and Transferability of Contex- tual Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 1073- 1094..
William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning.
John Hewitt and Percy Liang. 2019. Designing and In- terpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP", pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b, RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. arXiv: 1907. 1692 [cs].
John Hewitt and Christopher D. Manning. 2019. A Structural Probe for Finding Syntax in Word Repre-
sentations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4129 4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoc Interpretability for Neural NI.P: A Survey. ACM Computing Surveys, 55(8):155:1-1553
Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman, 2019. Do Attention Heads in BERT Track Syntactic Dependencies?
Christopher D. Manning, Kevin Clark, John Hewitt. Urvashi Khandelwal, and Omer Levy. 2020. Emer- gent linguistic structure in artificial neural networks trained by self- supervision. Proceedings of the Na- rional Academy of Sciences, 4444430330443300544
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics.
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang- goo Lee. 2019. Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction. In International Conference on Learning Representations.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection.

---

In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4044, Marseille, France. European Language Resources Association.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisticss pages 4593- 4601, Florence, Italy. Association for Computational Linguisties.
Prokopis Prokopidis and Haris Papagcorgiou. 2017. Universal Dependencies for Greek. In Proceedings of the NoDal 2017 Workshop on Universal Depen- dencies (UDW 2017), pages 102-106, Gothenburg. Sweden. Association for Computational Linguistics.
Jan Tenney. Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2018. What do you learn from context? Probing for sentence structure in con- textualized word representations. In International Conference on Learning Representations.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guard- ing protected attributes by iterative nullspace projec- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.
Vinit Ravishankar, Memduh Gökırmak, Lilja Øvrelid, and Erik Velldal. 2019. Multilingual Probing of Deep Pre-Trained Contextual Encoders. In Proceedings of the First NLPL. Workshop on Deep Learning for Natural Language Processing, pages 37-47, Turku, Finland. Linköping University Electronic Press.
David Vilares, Michalina Strzyz, Anders Søgaard, and 2020. Parsing as Pretrain- ing. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 34(05):91 14-9121.
Carlos
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A Primer in BERTology: What We Know About How BERT Works. Transactions the Asso- ciation for Computational Linguistics, 8:842-866.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Hug- gingFace's Transformers: Sfate-of-the-ar Natural Language Processing.
Gözde Gül Şahin, Clara Vania, Tha Kuznetsov, and Iryna Gurevych. 2020. LINSPECTOR: Multilingual Prob- ing Tasks for Word Representations. Computational Linguistics, 46(2):335-335
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation Rare Words with Subword Units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguistics.
Amir Zeldes. 2017. The GUM corpus: Creating mul- tilayer resources in the classroom. Language Re- sources and Evaluation, 51(3):581-612 -12.
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. 2018. A Modulation Module for Multi-task Learning with Applications in Image Retrieval. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pages 401-416.
Naomi Shapiro, Amandalynne Paullada, and Shane Steinert- Seannert-Thhlllee Threlkeld. 2021. A multilabel approach to morphosyntactic probing. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 4486 4524, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does String- Based Neural MT Learn Source Syntax? In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing. pages 1526-1534, Austin, Texas. Association for Computa- tional Linguistics.
Karolina Stanezak, Edoardo Ponti, Lucas Torroba Hen- nigen, Ryan Cotterell, and Isabelle Augenstein. 2022. Same neurons, different languages: Probing mor- phosyntax in multilingual pre-trained models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1589-1598, Seattle, United States. Association for Computational Linguistics.

---

A Dataset pre-processing
C
Golden and Predicted Centroid Similarity
Table 3 presents the train, validation and test split sizes (in terms of number of sentences) for each language considered in our work.
We present the cosine similarity between golden and predicted centroids for the POS tagging setup in table 4, as mentioned in Section 3.3.
The sentences in the corpora from the Universal Dependencies framework are already tokenized to the word level and stored as lists of words in the tokens field. However, since we use sub-word to- kenizers, namely the Byte Pair Encoding (Semnrich et al., 2016) and SentencePiece (Kudo and Richard- son, 2018) tokenizer, we further split the words into their sub-word tokens. Depending on the task, we also include either the upos field, which is list of integers corresponding to one of the 17 uni- versal POS tags available, the head and deprel fields which contain the head and one of the 36 dependence relations for dependency labeling'. should be noted that we only keep the language- independent relations, as some of them appear only with a language-specife modifier, and including them would make comparisons across languages less straightforwardd
D Self-neutralization visualization
Figs. 8 and 7 showcase the decrease in ac- curacy when self-ncutralizing in the POS tag- ging/dependeney labeling task for RoBERTa and XLM-R accordingly.
E Engineering Logisties
We rely on PyTorch Lightning (Falcon and The PyTorch Lighnining team, 2019) and Hugging Face Datasets (Lhoest et al., 2021) and Trans- fomers (Wolf et al., 2020) for our implementa- tion. We run experiments A100 GPU, with each training taking approx- imately three minutes. Our code is available giittul.comllssssffaalllliiiirr
an NVIDIA
Furthermore, upon inspecting the datasets observed that the annotators had split contractions into their parts and included them next the orig- inal contraction for the Italian and Greek corpora. However, ground-trutt truuh labels were only provided for the sub-words, with the compound words an- notated as special class "_". Hence, we filtered out the compound words from these datasets and retained their sub-parts. In addition to that, for de- pendency labeling, ignored words with the root dependency label, as they have no head and their prediction trivial.
F Detailed Cross-Neutralizing Results
We display the complete results for our cross- neutralization experiments in Figs. 9 9 18. More specifically, Fig. 9 corresponds the monolingual setting with RoBERTa English, and Fig. 11 to the multilingual setting with XLM-R on En- glish, Italian and Greek. Figs. 12 14 show the cross-lingust setting for every possible pairing of our three languages with XLM-R. Finally Figs. 15 and 16 show the monolingual cross-task setting in RoBERTa in POS-DEP and DEP-POS directions while Figs. 17 and 18 show the same but in the multilingual setting with XLM-R.
B Scaling to larger models
We repeat the experiments from Sections 4 & 5 to verify whether they hold across different model sizes. More specifically, we scale the English POS tag cross-neutrallization from RoBERTA-base RoBERTA-large (Fig. 10a), and from XLM-R- base to XLM-R-large (Fig. 10b) and XLM-R-XL (Fig. 10c). We notice that similar patterns occur across all tested model sizes (e.g. NOUN neutraliz- ing ADV). The effect appears to be generally less pronounced for larger models, although we also did find instances where the drop in accuracy is the same or higher.
Train
Validation
Test
English (GUM)
4287
784
890
Italian (VTT)
8277
743
1067
Greek (GDT)
1662
403
456
Table Sentence count in each split of each of the datasets we consider, averaging 24 words per sentence.
AA full list of POS tags and dependency relations be found the Universal Dependencies website.

---

100
100.0
DEP
87.5
Accuracy Decreasee (%)
Appuracy Decrease (%)
85.0
Pabber
4,31
La
82.5
80.3
12
12
Probing Layer
Probing Layer
(a) POS tagging Accuracy Decrease using different WordPicce (h) Dependency laheling Accuracy Decrease using different pooling methods ("aggregation" in the plot).
WordPiece pooling methods ("aggregation" in the plot).
POS tagging (left) and dependency labeling (right) tasks using embeddings extracted from different layers and setup configurations, for each of for English (en gum), Italian fit_vit) and Greek (el gdt) treebanks. For dependency labeling, we use the best child-head concatenation mode (ONLY) based on the results we aquired with RoBERTa, as shown Fig. 8b.
Figure 7:
Decrease in performance for XLMMR when self-neutralizing
POS
7687
Accuracy Decrease (%)
90
Table 4: Cosine similarities between golden and pre- dicted centroids for the RoBERTa POS tagging setup.
POS Tag
Cosine Similarity
80
NOUN
1,000
ADP
1.000
Probing Layer
ADJ
1.000
(a) Accuracy decrease for POS tagging when using different
PUNCT
1,000
WordPiece poolings.
AUX
1,000
DEP
100.0
VERB
1,000
97.5
DET
1,000
Accuracy Decrease (%))
CCONJ
1.000
95.0
NUM
1.000
92.5
ADV
0.9999
90.0
SCONJ
0.9998
87.5
ONLY
PRON
1.000
PART
1,000
82.5
SYM
0.9951
80.0
rean
PROPN
0.9999
Probing Layer
X
0.9737
INTJ
0.9980
(b) Accuracy decrease for dependency labeling when using different WordPiece pooling methods ("aggregation" in the plot) and child-head concatenation configurations.
Figure 8: Decrease in performance for RoBERTa when self-neutralizing in the POS tagging (top) and depen- dency labeling (bottom) tasks using embeddings ex- tracted from different layers and setup configurations.

---

Target
ADJ
ADP
ADV
AUX
CCONJ
DET
NOUN
NUM
PART
PRON
PUNCT
SCONJ
VERB
BvB
ADJ
-4
-12
-6
-8
-11
-0
-18
-10
ADP
-7
-89
-24
-8
-3
-2
0
-23
-27
-8
0
-39
-1
-8
ADV
AUX
-12
-4
-11
-89
-2
-3
-5
-9
-2
0
-11
-53
99
CCONJ
-3
-94
2
DET
-16
-6
-24
-5
-0
-89
-2
-21
-8
-12
0
-19
-0
-8
Neutralizer
NOUN
-21
-4
-15
-8
-90
-22
-13
-0
-30
-10
-22
NUM
-1
-13
-5
-4
0
PART
-2
-6
-13
-6
-0
0
-96
-2
0
-19
-4
33
PRON
-3
-10
PUNCT
-7
-6
-31
-10
-10
-5
-5
-5
-5
-62
-20
-5
-12
-6
VERB
-14
-5
-19
-11
-4
-3
-6
-15
-11
-5
0
-21
-90
-11
Figure 9: Relative change accuracy when cross-neutralizing POS tags using centroids from RoBERTa in English.
Target
Target
Thrget
MOOM
MIX
ADV
NUM
XDV
NOUN
-4
NOVN
40
-3
NOVN
10
+11
AD3
-10
-35
-1
4
-15
12F
-15
ADU
-37
-2
3
27
30
-12
ADU
-15
200
-7
-3
47
-34
VERB
12
455
3
-5
-17
VER3
-2
45
21
43
-15
VIRB
-20
-2
-0
-5
-15
PHON
.2
PRON
-5
PHON
42
-11
AUX
(a) RoBERTa-large
(b) M-L-Relarree
(c) XLM-R-XL
Figure 10: Change in accuracy when crutrainu English POS tags with centroids from different model sizes.

---

Target
Target
ADJ
ADP
ADV
AUX
OCONU
NOUN
NUM
PART
PRON
PUNCT
SCONJ
VERB
ADJ
ADP
ADV
AUX
CCONJ
DET
NOUN
NUM
PRON
PROPN
PUNCT
VERB
avg
91
ADJ
-18
-13
21
-17
45
114
ADP
-11
34
20
11
3
3
22
24
9
-0
37
3
-10
ADP
97
16
17
2
10
31
23
15
ADV
22
97
3
2
27
2
-
5
2
ADV
27
-44
-8
29
3
8
27
8
36
15
-7
-11
AUX
12
9
94
3
22
-0
14
59
AUX
46
-2
COONJ
-1
21
97
2
4
14
-2
-0
-1
-0
CCONJ
4
-1
28
88
97
-2
26
4
-6
114 -6
-5
Neutralizer
NOUN
29
25
20
6
12
92
39
31
13
-1
39
15
-24
NOUN
25 -
22
20
10
86
39
11
-6
33
-11
NUM
25
18
89
18
52
-12
-17
PART
3
114
2
2
-5
48
-3
PRON
14
11
93
23
24
-9
PRON
3
8
3
13
91
22
33
PROPN
-17 -1
-18
24
4
43
18
17
96
12
-12
15
PUNCT
10
SCONJ
$
-11
40
3
2
-5
18
11
-1
-0
91
22
-
SCONJ
-8
-1
24
8
19
3
6
36
23
18
98
22
-
VERB
-10
-33
-19
Target
ADJ
ADP
ADV
AUX
CCONU
DET
NOUN
NUM
PART
PRON
PROPN
PUNCT
SCONJ
avg
X
ADJ
83
40
18-25
16
27 6 36-16
ADP
AUX
- 4 97
3
28
23
CCONU
11
21
32
26 8
DET
28 -6
18-16
83
18
26-2733
24 -1 49-20
Neutralizer
NOUN
34 -6
-20-13-10
88
28-19-1--
45-24
NUM
23 -0
10
-1-81
77
-1
27
488
PART
12
PRON
28
PROPN
21
PUNCT
10 5
28-31
0 16
18
48-17
17
SCONU
. 11
14-18
2 10
82
30
VERB
18 4 12
18-29
11
9
30
4
90
38-14
X
88
2 9
12-18
-1
9 29
96
Figure 11: Relative change in accuracy when cross-neutralizing POS tags using centroids from XLM-R. In English (top, left), Italian (top right), and Greek (bottom).

---

Italian Target
Greek Target
ADV
ADP
ADV
CCONJ
DET
NOUN
NUM
PRON
PROPN
PUNCT
SCONJ
VERB
ADU
ADP
ADV
AUX
CCONJ
DET
NOUN
NUM
PART
PRON
PROPN
PUNCT
SCONJ
VERS
avg
X
ADJ
70
47
29
-12
-14
ADP
-8
43
12
13
5
2
44
12
27
-10
ADP
-3 23
9
-3
41
2
2
2
13
24
ADV
-5 -1
75
-3
2
4
4
5
10
AUX
-3 -1 -2
2
0
1
5
0
3
40
11-18
Englishh Neutrallzzrr
AUX
-10
-0
8
52
6
11
18
-5
8
10 32
CCONJ
-0 -1 -10
3
-15
COONJ
-2
-33
-1
-77
-2
11
-7
DET
14 -1
22
DET
4
5
33
35
-35
17
12
-1
NOUN
NOUN
-23
23
29
-5
-2
-19
448
-13-19 -9
-33
-17
19
English
PART
-1 -1
-7
3
0
1
2
17 -1
PRON
.6
-1
7
3
8
24
73
4
42
3
-5
PRON
-1 0
8
40
0
37
2
-7
3 -1
PROPN
-5 -0
9
2
0
11
8
22
1
2
37
22
110
PUNCT
-1 +0 -11
8
17
4
10
46
57
PUNCT
-5
22
-3
15
10
-14
-14
SCONJ
4
9
38
SCONJ
.6
35
11
3
45
13
10
42
-1
VERB
10
23
24-12
Figure 12: Relative change in accuracy when cross-neutralizing Italian (left) and Greek (right) POS tag embeddings using English centroids, using XLM-R representations.
English Target
Greek Target
ADJ
ADV
AUX
CCONJ
DET
NOUN
NUM
PRON
PUNCT
SCONJ
VERB
avg
ADJ
ADP
AUX
CCONJ
NOUN
NUM
PRON
PROPN
PUNCT
SCONJ
VERB
awg
.63
-11
-19
4
-33
48
12-14
ADP
-6
11
11
0
10
4
12
12
-0
36
ADV
11
85
0
2
48
7
21
-7
43
AUX
-5
-9
-0
88
42
Italian Neutralizer
Italian Neutralizer
CCONU
-1
-2
3
0
5
12
11
9
-1
-11 -3
CCONJ
2
25
78
-5
2
0
18
DET
20
-5
12
49
43
22
14
41
-9
-17 -11
DET
2
-9
-1
-
-7
-0
-18
NOUN
32
NOUN
21
4
25
14
9
-
79
30
13
-1
46
14
22
NUM
10
-
88
3
8
55
84
2
6
-11
-15
NUM
12
-7
23
5
-3
-
30
61
9
-0
-62
10
-13
PRON
5
11
2
34
17
12
19 -2
PRON
2
2
-2
-1
77
2
-2
-0
-17
-5
-3
PUNCT
-6
334
74
33
-1
-0
14
26
77
14
100
10
73
PUNCT
13
35
12
-3
144
3
-10-61
-8
.22
SCONJ
3
88
-
5
17
48
SCONJ
32
3
-
-7
-20
-8
-11
3
-28
X
8 -0
5
0
1
16
7
27
2
88
Figure Relative change in accuracy when cross-neutralizing English (left) and Greek (right) POS tag embeddings using Italian centroids, using XLM-R representations.

---

English Target
Italian Target
ADJ
ADP
AUX
OCONU
NOUN
NUM
PART
PRON
PUNCT
SCONJ
VERB
avg
ADJ
ADV
AUX
CCONJ
DET
NOUN
NUM
PRON
PROPN
PUNCT
VERB
avg
ADJ
-73
-20
-1
21
-35
188
48
-15
-11
ADP
400
16
-
3
2
-1
8
12
3
4
40
11
4
ADP
-9
-79
-16
-1
42
3
20
30
-12
ADV
78
13
5
3
26
2
4
85
6
ADV
15
95
14
14
-1
-5
35
4
21
9
-8
AUX
-0
16
-
3
-1
4
-1
AUX
Greek Neutraliieer
CCONJ
-3
16
-3
50
2
4
2
-1
40
-9
-2
-3
Greek Neutralizer
CCONJ
4
-0
18
7
40
-1
34
2
0
19
-8
44
DET
2
-33
NOUN
29
-2
28
-5
10
15
77
35
32
21
40
33
21
-23
NOUN
26
0
23
-8
-2
89
45
16
13
45
-19
NUM
30
4
23
-1
39
2
-8
NUM
-38
-0
23
-21
-2
19
444
-2
48
-11
-9
35
PRON
74
PRON
-0
5
-6
-1
-1
-6
2
-1
26
40
15
-3
4
PROPN
26 -0
24 -3
-3
-68
335
27
-55
47 -15 -19
PUNCT
-2
22
77
8
12
29
20
-
91
-6
-17
-1
SCONJ
2
32
4
4
-0
SCONJ
-8
4
36
-7
30
-3
-11
54
18
29
-11
-15
VERB
29
22
-1
-6
114
-10
35
-7
-0
21
-94
-14
VERB
31
6
11
20
-17
14
Figure Relative change in accuracy when cross-neutralizing English (left) and Italian (right) POS tag embeddings using Greek centroids, using XLM-R representations.

---

POS Target
CCONJ
PUNCT
SCONJ
ADJ
ADP
ADV
AUX
DET
NOUN
NUM
PART
PRON
VERB
2vg
ACL
ADVCL
ADVMOD
AMOD
AUXDEP
CASE
CC
CCOMP
COMPOUND
CONJ
COP
CSUBJ
DEP Neutralizer
DETDEP
DISCOURSE
EXPL
FIXED
FLAT
IOBJ
LIST
MARK
NMOD
NSUBJ
NUMMOD
OBJ
OBL
PARATAXIS
PUNCTDEP
XCOMP
Figure 15: Relative change in accuracy when cross-neutralizing RoBERTa POS embeddings using RoBERTa DEP centroids.

---

DEP Target
COMPOUND
PARATAXIS
ADVMOD
ACL
ADVCL
AMOD
AUX
CASE
CC
CCOMP
CONJ
COP
CSUBJ
DET
MARK
NMOD
NSUBJ
OBJ
OBL
PUNCT
XCOMP
avg
ADJ
ADP
ADV
AUX
CCONJ
DET
INTJ
POS Neutralizer
NOUN
NUM
PART
PRON
PROPN
PUNCT
SCONJ
SYM
VERB
X
Figure 16: Relative change in accuracy when cross-neutralizing RoBERTa DEP embeddings using RoBERTa POS centroids.

---

Figure Relative change accuracy when noos-neutraalzzing M-R POS embeddings using XLM-R DEP centroids In Enngishh Italian and Greek, Some languages have less columns rows than others because the particular category for the missing row/columm is not in their dataset.
Neutralizer
DISCOURSE
VOCATIVE
PARATAXIS
NUMMOD
COMPOUND
AD ADWMCO
XCOMP
PUNCT
OBL
OBJ
NSUBJ
NMOD
MARK
UST
J0aJ
PLAY
FXXD
EXPL
DET
CSURJ
COP
CONJ
CCOMP
CC
CASE
AUX
AMCD
NDVCL
KGL
AD
ADS
ADV
AUA
DET
vouN N
POS Target
Engilsh
NUN
PAR
PART
PRO
PRO
FUNCI
S000
38
REPARANDUM
ORPHAN
OBL
NUMMOD
NSUBJ
NMOD
MARK
GOESWITH
FLAT
FXED
EXPL
DET
csuaJ
COP
CONJ
COMPOUND
CCOMP
cc
CASE
AUX
AMOD
ADVMOD
ADVCL
AGL
A
AM
A0
AUX
NOON
DEL
NOUN
Italfin
NUM
PRON
PROPP
PUNCT
KERR
AD
PARATAXIS
COESNTH
ADVICC
PUNCT
GBL
0a.
NSUBJ
NMCO
MARK
LIST
108,1
FXED
EXPL
DET
csuaJ
CONU
COOMP
AUX
AMCO
ADVCL
ACL
C2
AD1
ADB
ADV
AUX
DET
MOUN
POS
Greek
NUM
PRON
PUNK
PINC
ACON
YER
K

---

DEP Target
COMPOUND
PARATAXIS
ADVCL
ADVMOD
AMOD
CASE
CCOMP
CONU
MARK
NMOD
NSUBJ
PUNCT
XCOMP
ACL
AUX
COP
DET
OBJ
OBL
avg
ADJ
ADV
CCONJ
POS Neutralizer
J
English
NUM
PART
PRON
PROPN
PUNCT
SCONJ
SYM
VERB
PARATAXIS
ADVMOD
NUMMOD
ADVCL
AMOD
AUX
CASE
CC
CCOMP
CONJ
COP
DET
EXPL
FIXED
FLAT
MARK
NMOD
NSUBJ
OBJ
OBL
PUNCT
XCOMP
awg
ADJ
ADP
ADV
AUX
POS Neutralizer
CCONJ
NOUN
NUM
PRON
PROPN
PUNCT
SCONJ
VERB
X
ACL
ADVCL
ADVMOD
AMOD
AUX
CASE
CC
CCOMP
CONJ
COP
DET
FLAT
MARK
NMOD
NSUBJ
NUMMOD
OBJ
OBL
PUNCT
XCOMP
avg
ADJ
ADP
ADV
AUX
POS Neutralizer
CCONJ
DET
NOUN
Greek
NUM
PART
PRON
PROPN
PUNCT
SCONJ
VERB
Figure 18: Relative change in accuracy when cross-neutralizing XLM-R DEP embeddings using XLM-R POS centroids. In English, Italian and Greck. Some languages have less columns rows than others because the particular category for the missing row/column is not in their dataset.


Reviews:
[{"review_id": "zFsW5dUqCu", "reviewer": "Reviewer_neqX", "summary": "", "questions": "Question A: You mention that you finetune a probing classifier. What was the probe trained on in the first place?\n\nQuestion B: Why didn’t you report the selectivity of the trained probe?\n\nQuestion C: You mentioned in the introduction that one of the research questions that the paper addresses are “how syntactic categories” are stored within the models. Why didn’t you address this question? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and helpful feedback! \n\nRegarding question A, we apologize for the unclear language here. The probing classifier itself is not pretrained on anything, and rather is randomly initialized. So in a certain sense saying we fine-tune the probing classifier is incorrect, and we understand your confusion. Instead, we should have said we _train_ the probing classifier on the representations incoming from the frozen pre-trained encoder. We will make sure to correct this in the camera-ready version.\n\nRegarding the selectivity baseline: Thank you for this suggestion! Following your advice, we now ran a selectivity baseline on the POS tagging task which is easier to memorize, following Hewitt and Liang (2019). We train our probing classifier on RoBERTa-base representations on the true POS tagging task, and then train a new checkpoint for the same number of steps on the control task. Under this setup, we find that our probing classifier achieves an average accuracy of 96% on the POS tagging task and 63% on the control task, giving a selectivity of 33%. We believe this should be sufficient to indicate our results are indeed valid. We will make sure to include this baseline in the camera-ready version.\n\nRegarding question C, the “syntactic categories” we refer to in the introduction are precisely the POS tags and dependency labels we study. For instance, we may refer to the NOUN POS tag as a syntactic category. We envision our results, showing that many of these categories are jointly encoded in different ways (e.g. across tasks and/or languages), shed some light on how these categories are encoded in the models, one of the questions we pose in the introduction.\nWe will make sure to clarify this in the camera-ready version.\n\nLast but not least, thank you for the feedback on the figure. We will make sure to increase the font size.\n\n## References\n\n1. Hewitt J, Liang P. Designing and Interpreting Probes with Control Tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) [Internet]. Hong Kong, China: Association for Computational Linguistics; 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Approach suggested helps in understanding how linguistic categories are encoded within language models, how they are shared between tasks, and how they are shared between languages.\n", "reasons_to_reject": "- Only reported classification accuracy of the probes trained without reporting selectivity. The probe trained might be memorizing the data rather than learning to predict the linguistic categories.", "questions_to_authors": "Question A: You mention that you finetune a probing classifier. What was the probe trained on in the first place?\n\nQuestion B: Why didn’t you report the selectivity of the trained probe?\n\nQuestion C: You mentioned in the introduction that one of the research questions that the paper addresses are “how syntactic categories” are stored within the models. Why didn’t you address this question? \n", "paper_topic_and_main_contributions": "The authors proposed a framework for testing joint encodings of linguistic categories in Large Language Models. They extended the idea of \"cross-neutrilization\" to study how information is encoded across different linguistic categories in two syntactic tasks (Part-of-Speech tagging and dependency labeling). The contribution of the paper are summarized below: \n- Proposed a framework for analysis of joint encodings in language models.\n- Provided evidence for joint encoding between different POS tags. \n- Provided evidence for joint encoding across languages. \n- Provided evidence for shared knowledge across tasks. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Text in figure 1 is small. Please consider increasing font size"}, "extra_text": {"Missing_References": "Some previous work on how syntactic information is stored in Transformer based models and how it evolves within the network are missing from the background section. Specifically:\n\n- @misc{dalvi2022discovering,\n      title={Discovering Latent Concepts Learned in BERT}, \n      author={Fahim Dalvi and Abdul Rafae Khan and Firoj Alam and Nadir Durrani and Jia Xu and Hassan Sajjad},\n      year={2022},\n      eprint={2205.07237},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n\n\n\n"}}, {"review_id": "TU1kmQBCpX", "reviewer": "Reviewer_HBxT", "summary": "", "questions": "- Why did you choose to use a two-layer MLP for your probes (as opposed to a simple linear model)?  Did you also look at selectivity (performance on a control task) to make sure that the probing model itself is not too powerful?\n- Is there more theoretical justification for subtracting-the-centroid as removing the information?  I'd be curious to hear more about a comparison with amnesic probing / iterative nullspace projection, which removes the linear information of an attribute via projection onto the nullspace.  Would that be an alternative to your method?\n- Why dependency label prediction as a task, and not as well whether or not there is a dependence relation?\n- Why were English, Italian, and Greek the only three languages studied?  How were they chosen?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and helpful suggestions!\n\n## Baselines\n\nAs you suggested, we tested two additional baselines to provide a more comprehensive evaluation. We ran a selectivity baseline on the POS tagging task which is easier to memorize, following Hewitt and Liang (2019). We train our probing classifier on RoBERTa-base representations on the true POS tagging task, and then train a new checkpoint for the same number of steps on the control task. Under this setup, we find that our probing classifier achieves an average accuracy of 96% on the POS tagging task and 63 % on the control task, giving a selectivity of 33%. We believe this should be sufficient to indicate our results are indeed valid. We will make sure to include this baseline in the camera-ready version.\n\nIn the second baseline, we subtracted random vectors (rather than centroids) from the model representations. We found that this left the performance relatively unchanged, with no evidence of the systematic drops observed when subtracting centroids. Both additional experiments strengthen our results, and we are very grateful to you and your fellow reviewers for the suggestions! We will make sure to include both of these experiments in the camera-ready version of the paper. \n\n## Other Questions\n\nWe’d like to thank you for pointing us to the related work on INLP and amnesic probing which would pose an interesting alternative to our proposed method. Indeed these would appear to be valid substitutes for subtraction. We agree that our paper would benefit from a more thorough comparison between INLP and our method for completeness. We will make sure to include this in the camera-ready version of the paper. \n\nRegarding your first question, we decided to use a two-layer MLP following Choenni and Shutova (2022), Şahin et al. (2020) and Tenney et al. (2019) who do the same. Note that with “two-layer MLP” we mean that our classifier consists of two linear layers with a non-linear activation in between. We will clarify this in the camera-ready version. \n\nAs for your third question, we chose to focus on dependency label prediction rather than dependency parsing mainly for simplicity. Because we were mainly concerned with identifying representations tied to specific dependency labels, we envisioned the labeling task as sufficient for this. Full dependency parsing, including identifying dependency arcs, may have introduced an additional capability requirement to our probing setup and would have made it more difficult to disentangle the task of assigning dependency types from the task of identifying the dependency arcs.\n\nWith regard to the final question, these three specific languages were chosen to facilitate our analysis and interpretation of the results, as at least one author was familiar with each. \n\n\n## References\n\n 1. Şahin GG, Vania C, Kuznetsov I, Gurevych I. LINSPECTOR: Multilingual Probing Tasks for Word Representations. Computational Linguistics. 2020 Jun;46(2):335–85. \n2. Choenni R, Shutova E. Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology. Computational Linguistics. 2022 Sep;48(3):635–72. \n3. Hewitt J, Liang P. Designing and Interpreting Probes with Control Tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) [Internet]. Hong Kong, China: Association for Computational Linguistics; 2019\n4. Tenney I, Xia P, Chen B, Wang A, Poliak A, McCoy RT, et al. What do you learn from context? Probing for sentence structure in contextualized word representations. In 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting question about the similarity of representations across tasks and across languages.\n- Analyzes both mono- and multi-lingual models.", "reasons_to_reject": "- Lack of baselines, both for the probing models (e.g. selectivity), and for the subtraction method.  It would be good to subtract a random vector, or the centroid of all word representations (or some such) to see how much worse the given subtractions are.\n- Could benefit from more thorough discussion of alternative attribute-removal methods (see questions / references) and other studies looking at crosslingual joint encoding.\n- Some analyses are merely speculative, e.g. for across-POS differences.", "questions_to_authors": "- Why did you choose to use a two-layer MLP for your probes (as opposed to a simple linear model)?  Did you also look at selectivity (performance on a control task) to make sure that the probing model itself is not too powerful?\n- Is there more theoretical justification for subtracting-the-centroid as removing the information?  I'd be curious to hear more about a comparison with amnesic probing / iterative nullspace projection, which removes the linear information of an attribute via projection onto the nullspace.  Would that be an alternative to your method?\n- Why dependency label prediction as a task, and not as well whether or not there is a dependence relation?\n- Why were English, Italian, and Greek the only three languages studied?  How were they chosen?", "paper_topic_and_main_contributions": "This paper explores \"joint encoding\" of linguistic properties in both mono and multilingual language models.  Two properties are \"jointly encoded\" if they are encoded in similar portions of the vector space; the authors measure this by taking the centroid of the representations for one property (class, e.g. POS tag) and subtracting it from the representations and seeing if it effects performance on another category (e.g. another POS, dependency labeling, or same tag in another language).  The results are suggestive: many properties do exhibit joint encoding, and there are also similarities in the effects across languages within one multilingual model.  The paper presents interesting experiments and results, but could use more motivation for the approach and more detailed comparison with other methods.\n\nAFTER REBUTTAL EDIT: given the news about the baselines, and the enriched discussion of INLP, I am happy to upgrade my score from 3 to 4.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Amensic probing / INLP: https://aclanthology.org/2020.acl-main.647/ , https://aclanthology.org/2021.tacl-1.10\n- Crosslingual morphosyntactic probing, with an eye to joint-encoding-like phenomena: https://aclanthology.org/2021.findings-emnlp.382/ , https://aclanthology.org/2022.naacl-main.114/"}}, {"review_id": "bpYCyxULEs", "reviewer": "Reviewer_ZArK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and helpful feedback! We are glad to hear that you enjoyed reading our paper, and found the approach interesting and analyses insightful.\n\nWe agree that experiments on the effect of model size would certainly add value, particularly given recent research on scaling laws. Following your suggestion, we repeated our proposed methodology for POS tag cross-neutralization on RoBERTa-large. Compared to RoBERTa-base, We notice that similar patterns occur across model sizes (e.g. NOUN neutralizing ADV). The effect appears to be generally less pronounced for larger models, although we also did find instances where the drop in accuracy is the same or higher. We repeated the same experiment with XLM-R-base, XLM-R-large and XLM-R-XL and observed the same patterns and trends. We will include these results in the camera-ready version.\n\nAs for our treatment of the transformer layers, we devoted attention to this in the work because we expected potentially significant differences in how joint encoding occurs depending on the layer given the work of Tenney et al. (2019). However, we later found this not to be the case, hence the lack of results in this theme. This is perhaps due to the relatively low-level nature of the tasks considered (in terms of the linguistic hierarchy), and may change for higher-level NLP tasks (e.g. more complex NLU tasks). We note that this can be partially inferred from Fig. 7, in the appendix, where the self-neutralizing accuracy drop is rather significant (between 80% and 95%) regardless of layer configuration, and no strong trend can be identified. We do realize that this is not immediately clear in the main text, and we will make sure to further elaborate on this in the camera-ready version.\n\nAs for our choice of probing classifier, we agree that considerations are speaking for and against both the linear and the non-linear probes. To further validate the results of our probes, based on the suggestions of the reviewers, we ran a selectivity analysis on the POS tagging task, as proposed by Hewitt and Liang (2019). We train our probing classifier on RoBERTa-base representations on the true POS tagging task, and then train a new checkpoint for the same number of steps on the control task. Under this setup, we find that our probing classifier achieves an average accuracy of 96% on the POS tagging task and 63% on the control task, giving a selectivity of 33%. We believe this should be sufficient to indicate our results are indeed valid. We will make sure to include this baseline in the camera-ready version.\n\nWe believe that our method and findings are important as they shed light on the ways in which LLMs generalize linguistic information. In this paper, we tested specifically for joint encoding of different syntactic categories that rely on the same linguistic concept (e.g. an “amod” dependency implies some relationship with the adjective and the noun POS tags). However, the approach can be extended to test for joint encoding of classes/labels/categories in other tasks that can be expected to share information. This in turn would inform us on whether an LM indeed captures this shared information between different tasks or if does not. Knowing this can help us develop better models in many application scenarios: for instance, in a multitask learning setting (where negative interference between parameter updates from different tasks is known to hamper performance) or a lifelong learning setting (where learning a new task often leads to (catastrophic) forgetting of the previously learned ones). We will add a discussion of this in the final version of the paper.\n\nWe would like to end with a quote from John Hewitt: \"Gaining insights into the natures of NLP’s unsupervised representations may help us to understand why our models succeed and fail, what they’ve learned, and what we yet need to teach them”. This is something we very much agree on.\n\n## References\n\n1. Hewitt J, Liang P. Designing and Interpreting Probes with Control Tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) [Internet]. Hong Kong, China: Association for Computational Linguistics; 2019\n2. Tenney I, Das D, Pavlick E. BERT Rediscovers the Classical NLP Pipeline. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics [Internet]. Florence, Italy: Association for Computational Linguistics; 2019. p. 4593–601.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- The paper is well-written and interesting to read.\n\n-- The approach devised to understand how LLMs jointly encode linguistic properties is very interesting. By investigating the joint encoding of linguistic categories, the paper sheds light on the model's ability to learn diverse properties simultaneously. This aligns with the existing line of research and contributes to our understanding of the capabilities of these language models.\n\n-- The discussion of the results is insightful and adds value to the findings. The paper not only presents empirical evidence but also provides an analysis and interpretation of the observed patterns, giving the reader a deeper understanding of the implications of the research.", "reasons_to_reject": "-- Despite the interesting approach and the insightful discussion of the results, the contribution of the paper appears somewhat limited. The research could have been better presented as a short paper. Additionally, more experiments would have been beneficial for better appreciating the findings, such as testing different models at various sizes. Moreover, the authors experimented only using representations from layers 1, 3, 6, 9 and 12 and subsequently selecting the best configuration for each task, language and model. It would have been also very interesting to see how the ability of the model to jointly encode linguistic properites changes across layers. \n\n-- The author tested a two-layer MLP model as probing model. However, in the last few years, there has been a long debate about the validity of probing models and how to choose the best model configuration (e.g. very simple linear models vs. more complex NN models). I think that this issue should have been taken into account to give more validity to the findings of this work.\n\n-- The introduction suggests that understanding how LLMs represent linguistic phenomena is crucial for developing more effective NLP methods. However, the paper lacks a discussion on the possible implications of the study's findings. Although I think that not all research needs to have direct application purposes (especially in the context of interpretability studies), it would have been interesting to see a discussion on the potential implications and applications of these results for building more efficient NLP systems, as mentioned in the introduction. Such insights would add further value to the research and its relevance to the NLP community.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the ability of Large Language Models (LLMs) to joint encoding linguistic categories. In particular, it builds on previous research suggesting that LLMs exhibit a linguistic hierarchy in their layers, with lower layers being more suitable for syntactic tasks and higher layers for semantic processing. The authors propose a framework for testing the joint encoding of linguistic categories in LLMs, focusing on syntax and, in particular, on POS tagging and DEP labelling. They find evidence of joint encoding at both the same level, such as related part-of-speech (POS) classes, and different levels, such as POS classes and related syntactic dependency relations. Furthermore, their cross-lingual experiments demonstrate that these patterns hold across languages in multilingual LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework for analyzing joint encodings in LMs, and evidence for the existing of these for certain labels and tasks. The reviewers identified missing baselines as well as additional experiments (e.g. different model sizes); but these have all been provided by the authors in the rebuttal. It is well written, and has enough significance (especially after the rebuttal). The leftover cons would be the question of validity of a 2-layer probe (this is a larger discussion within probing), the lack of discussion of implications, and that some analyses are merely speculative. I would say the first one is beyond the scope of this paper, the discussion of implications could be added, and speculative analyses could be seen as interesting avenues for future work.
Decision: Accept-Findings