Representational correlates of hierarchical phrase structure in deep language models
Authors: Matteo Alleman
Jonathan Mamou
Miguel A Del Rio
Hanlin Tang
Yoon Kim
SueYeon Chung
Venue:  (2021)
Keywords: bertology
interpretability
computational neuroscience
population coding

Abstract: While contextual representations from Transformer-based architectures have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings. In particular, it is not entirely clear what aspects of sentence-level syntax are captured by these representations, nor how (if at all) they are built along the stacked layers of the network. In this paper, we aim to address such questions with a general class of input perturbation-based analyses of representations from Transformer networks pretrained on self-supervised objectives. Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of Transformer representations to several kinds of structure in sentences. Each probe involves swapping words in a sentence and comparing the representations from perturbed sentences against the original. We experiment with three different perturbations: (1) random permutations of n-grams of varying width, to test the scale at which a representation is sensitive to word position; (2) swapping of two spans which do or do not form a syntactic phrase, to test sensitivity to global phrase structure; and (3) swapping of two adjacent words which do or do not break apart a syntactic phrase, to test sensitivity to local phrase structure. We also connect our probe results to the Transformer architecture by relating the attention mechanism to syntactic distance between two words. Results from the three probes collectively suggest that Transformers build sensitivity to larger parts of the sentence along their layers, and that hierarchical phrase structure plays a role in this process. In particular, sensitivity to local phrase structure increases along deeper layers. Based on our analysis of attention, we show that this is at least partly explained by generally larger attention weights between syntactically distant words.

Under review as a conference paper at ICLR 2021
REPRESENTATIONAL CORRELATES OF HIERARCHICAL PHRASE STRUCTURE IN DEEP LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
While contextual representations from pretrained Transformer models have set a new standard for many NLP tasks, there is not yet complete accounting of their inner workings. In particular, it is not entirely clear what aspects of sentence-level syntax are captured by these representations, nor how (if at all) they are built along the stacked layers of the network. In this paper, we aim to address such questions with a general class of interventional, input perturbation--asee of representations from Transformers networks pretrained with self-supervision. Importing from computational and cognitive neuroscience the notion of represen- tational invariance, we perform series of probes designed to test the sensitivity Transformer representations to several kinds of structure in sentences. Each probe involves swapping words in sentence and comparing the representations from perturbed sentences against the original. We experiment with three different perturbations: (1) random permutations of n-grams of varying width, to test the scale at which a representation is sensitive to word position; (2) swapping of two spans which do do not form syntactic phrase, to test sensitivity to global phrase structure; and (3) swapping of two adjacent words which do or do not break apart a syntactic phrase, to test sensitivity local phrase structure. We also connect our probe results to the Transformer architecture by relating the attention mechanism to syntactic distance between two words. Results from the three probes collectively suggest that Transformers build sensitivity to larger parts of the sentence along their layers, and that hierarchical phrase structure plays a role in this process. In particular, sensitivity to local phrase structure increases along deeper layers. Based on our analysis of attention, we show that this is at Ieast partly explained by generally larger attention weights between syntactically distant words
analyses
INTRODUCTION AND RELATED WORK
is still unknown how distributed information processing systems encode and exploit complex relational structures in data. The fields of deep learning (Saxe et al. 2013 Hewitt & Manning 2009)). neuroscience (Sarafyazd & Jazayeri 2019 Stachenfeld et al. 2017). ). and cognitive science (Elman 1991. Kemp & Tenenbaum 2008 Tervo al.) 2016) have given great attention to this question, including a productive focus on the potential models and their implementaions of hierarchical tasks, such predictive maps and graphs.
Natural (human) language provides a rich domain for studying how complex hierarchical structures are encoded information processing systems. More so than other domains, human language is unique in that its underlying hierarchy has been extensively studied and theorized in linguistics, which provides source of "ground truth" structures for stimulus data. Much prior work on characterizing the types of linguistic information encoded in computational models of language such as neural networks has focused on supervised readout probes, which train a classifier on top pretrained models to predict a particular linguistic label (Belinkov & Glass] 2017 Liu et al., 2019a Tenney et al. 2019). In particular, (Hewitt& & Manning (209)) apply probes to discover linear subspaces that encode tree-distances distances in the representational subspace, and Kim et al. (2020) show that these distances can be used even without any labeled information to induce hicrarchical structure. However, recent work has highlighted issues with correlating supervised probe performance with the amount
Datasets, extracted features and code will he publicly available upon publication.

---

Under review as a conference paper at ICLR 2021
of language structure encoded in such representations (Hewitt & Liang 2019) Another popular approach to analyzing deep models is through the lens of geometry (Reif et 2019; Gigante et al. 2019). While geometric interpretations provide significant insights. they present another challenge in summarizing the structure in a quantifiable way. More recent techniques such as replica- ppsee mean field manifold analysis method (Chung al., 2018 Cohen et al., 2019 Mamou et al.. 2020) connects representation geometry with linear classification performance, but the method is limited to categorization tasks.
In this work, we make use of an experimental framework from cognitive science and neuroscience to probe for hierarchical structure in contextual representations from pretrained Transformer models (i.e..
BERT (Devlin et al.) (018) and its variants). A popular technique in neuroscience involves measuring change the population activity response to controlled, input perturbations (Mollica et al. 2020
Ding et al. 2016). We apply this approach to test the characteristic scale and the complexity (Fig. of hterarchical phrase structure encoded deep contextual representations, and present several key findings:
1. Representations are distorted by shuffling small n- n-grams in early layers, while the distortion caused by shuffling large n-grams starts to occur later layers, implying the scale of characteristic word length increases from input downstream layers.
2. Representational distortion caused by swapping two constituent phrases is smaller than when the control sequences of the same length are swapped, indicating that the BERT representations sensitive to hierarchical phrase structure.
3. Representational distortion caused by swapping adjacent words across phrasal boundary is larger than when the swap is within a phrasal boundary; furthermore, the amount of distortion increases with the syntactic distance between the swapped words. The correlation between distortion and tree distance increases across the layers, suggesting that the characteristic complexity of phrasal subtrees increases across the layers.
4. Early layers pay more attention between syntactically closer adjacent pairs and deeper layers pay more attention between syntactically distant adjacent pairs. The attention paid in each layer can explain some of the emergent sensitivity to phrasal structure across layers.
Our work demonstrates that interventional tools such as controlled input perturbations can be useful for analyzing deep networks, adding to the growing, interdisciplinary body of work which profitably adapt experimental techniques from cognitive neuroscience and psycholinguistics to analyze computational
models of language (Futrell et al. 2018 Wilcox et al., 2019 Futrell et al. 2019
Ettinger 2020).
2
METHODS
Eliciting changes in behavioral and neural responses through controlled input perturbations is a common experimental technique in cognitive neuroseience and psycholinguisties (T'ao & Livingstone et al., (020). Inspired by these approaches, we perturb input sentences and measure the discrepancy between the resulting, perturbed representation against the original. While conceptually simple, this approach allows for a targeted analysis of internal representations obtained from different layers of deep models, and can suggest partial mechanisms by which such models are able encode linguistic structure. We note that sentence perturbations have been primarily utilized in NLP for representation learning (Hill et al. 2016 Artetxe et al.) 2018 Lample et al. 2018), data augmentation (Wang et al. 2018 Andreas 2020). and testing for model robustness (e.g.. against adversarial examples) (Jia & Liang 2017 Belinkov & Bisk, 2018). A methodological contribution of our work is to show that input perturbations can serve useful tool for analyzing representations learned by deep networks.
2.1 SENTENCE PERTURBATIONS
In this work we consider three different types of sentence perturbations designed to probe for different phenomena.
n-gram shuffling In the n-gram shuffling experiments, we randomly shuffle the words of a sentence in units of n-grams, with varying from (i.e., individual words) 7 (see Fig. 2h for example).
2

---

Under review as a conference paper at ICLR 2021
Large The
owiS
ane
not
what they
seem
Irout
Attendio-> > MLP
"recestive
fields"?
*6 The
KJowls 1 ag are > xyl not
kywwhrt Xithey?
syseem
Aftermin MIP
Small
The
owls
are
not
what
they
soom
Iepal lever
rrecestive
Altemtion MLP
nolds"?
oft The ((08)) sfore of not
sythoyy)
e/pcem
Layer
Fight tchen
The
The owls not what they seem
rosoed
alword original)
alword permuted)
comnect
Complee
conteet:
Simple
?
not what owls they are soom The
deeper baans
eady Siyers
Figure 1: Do Transformers build complexity along their layers? (a) The representation a word is function of its context, and this cartoon illustrates an hypothesis that deeper representations use larger contexts. (b) An example parse tree, illustrating our notion of phrase complexity. (c) Cartoon of the distortion metric, where vectors are the z-scored feature vectors and color map vectors words.
While the number of words which change absolute position is similar for different n, larger n will better preserve the local context (i.e.. relative position) more words. Thus, we reason that n-gram swaps affect the representations selective to the context with size n or higher within the sentence, and that lower n will result in greater distortion in sentence representations.
Phrase swaps The R--raam shuffling experiments probe for sensitivity of representations to local context without taking into account syntactic structure. In the phrase swap experiments, we perturb a sentence by swapping two randomly chosen spans. We explore two ways of swapping spans. In the first setting, the spans are chosen such that they are valid phrases according to its parse tree In the second setting, the spans are chosen that they are invalid phrases. Importantly, in the second, control setting, we fix the length of the spans such that the lengths of spans that are chosen to be swapped are the same as in the first setting (see Fig. (3 for an example). We hypothesize that swapping invalid phrases will result in more distortion than swapping valid phrases, since invalid swaps will result in greater denigration of syntactic structure.
Adjacent word swaps In the adjacent word swapping experiments, we swap two adjacent words in sentence. We again experiment with two settings in the first setting, the swapped words stay within the phrase boundary (i.e., the words share the same parent), while in the second setting, the swapped words cross phrase boundaries. We also perform a more fine- finegrained analysis where we condition the swaps based on the "syntactic distance" between the swapped words, where syntactic distance is defined as the distance between the two words in the parse tree (see Fig. 4b). Since phrase corresponds
subtree the parse tree, this distance also quantifies the number of nested phrase boundaries between two adjacent words. Here, we expect the amount of distortion to be positively correlated with the syntactic distance of the words that are swapped.
2.2 CONTEXTUAL REPRESENTATION FROM TRANSFORMERS
For our sentence representation, we focus on the Transformer-famile of models pretrained on large-scale language datasets (BERT and its variants). Given an input word embedding matrix X E 7xd for sentence of length T, the Transformer applies self attention over the previous layer's representation to produce new representation,
X: = (H H.H)),
H = AI, V
(X
(X,
A = softmax
/dz
We constituency parse trees from the English Penn Treehank Marcus al.
1994.
3

---

Under review as a conference paper at ICLR 2021
where fi is an MLP layer, H is the number of heads, dæ i and Q K,,,, V1, E TRdxd are respectively the learned query, key, and value projection matrices layer / for head i. The MLP layer consists of residual layer followed by layer normalization and nonlinearity. The O-th layer representation Xo is obtained by adding the position embeddings and the segment embeddings to the input token embeddings X, and passing it through normalization layer"
d is the head embedding dimension,
In this paper, we conduct our distortion analysis mainly on the intermediate Transformer representa- tions X; = [x.......... x1,r), where X1,: E IR is the contextualized representation for word at layer We analyze the trend in distortion as function of layer depth / for the different perturbations. We also explore the different attention heads HI,; E ! and the associated attention matrix A,, E RTxx to inspect whether certain attention heads specialize at encoding syntactic information.
2.3 DISTORTION METRIC
Our input manipulations allow us to specify the distortion at the input level, and we wish to measure the corresponding distortion in the representation space (Fig. [ Due the attention mechanism, a single vector in an intermediate layer is a function of the representations of (potentially all) the other tokens in the sentence. Therefore, the information about a particular word might be distributed among the many feature vectors of a sentence, and we wish to consider all feature vectors together as a single sentence-level representation.
We thus represent each sentence as a matrix and use the distance induced by matrix 2-norm. Specif- ically, let P E {0. TxT be the binary matrix representation of a permutation that perturbs the input sentence, i.e., X = PX. Further let X, and X, be the corresponding sentence representations for the I-th layer for the perturbed and original sentences. To ignore uniform shifting and scaling, we also z-score each feature dimension of each layer (by subtracting the mean and dividing by the standard deviation where these statistics are obtained from the full Penn Treebank corpus) to give Zi and Z₁ Our distortion metric for layer is then defined as IZ P 'Zill /TT//d where || is the matrix 2-norm (i.e., Frobenius norm) 5 Importantly, we invert the permutation of the perturbed representation to preserve the original ordering, which allows us to measure the distortion due to structural change, rather than distortion due simple differences in surface form. We divide by <Td to make the metric comparable between sentences (with different T') and networks (with different d).
Intuitively, our metric is the scaled Euclidean distance between the z-scored, flattened sentence representations, E T.dd Because each dimension is independently centered and standardized, the maximally unstructured distribution of z; an isotropic T ddddmensioonal Gaussian. The expected distance between two such vectors is approximately v2Td. Therefore, we can interpret distortion value approaching /2 as comparable if we had randomly redrawn the perturbed representation.
3 EXPERIMENTAL SETUP
We apply our perturbation- analysis on sentences from the English Penn Treebank (Marcus et al., 1994), where we average the distortion metric across randomly chosen sentences (see Sec. | for the exact details). We analyze the distortion, as measured by length-normalize the perturbed and original representations, as a function of layer depth. Layers that experience large distortion when the syntactic structure is disrupted from the perturbation can be interpreted as being more sensitive to hierarchical syntactic structure.
Frobenius norm between
As we found the trend be largely similar across the different models, in the following section, we primarily discuss results from BERT
which has 12 layers and hidden size of 768 (Devlin et al., 2018). We show results from other pretrained and randomly-initialinn Transformer-based models, including RoBERTa (Liu et al. 2019b), ALBERT (Lan et al. 2019). DistilBERT (Sanh et al. 2019). and XLNet (Yang al. 2019). in the appendix (Sec. A.2).
'However, the exact specification for the MLP and Xo may vary across different pretrained models. 'BERT uses BPE tokenization (Sennrich et al. 2015). which means that some words are split into multiple tokens. Since we wish to evaluate representations word-level. if a word is split into multiple tokens, its word representation computed the average of all its token representations.
'There are many possible ways of measuring distortion, such as the average cosine similarity or distance between corresponding feature vectors. as well as different matrix norms. We observed the results to be qualitatively similar for different measures, and hence we focus on the Frobenius norm in our main results. We show the results from additional distortion metries in Sec. A.3
4

---

Under review as a conference paper at ICLR 2021
Drighs
kome the
Kandiv sifllie
Sgram
your LSC ede 1.12
2:g0e
your
3gan rdd 1,04 The lutar de catee you
Cortedlonp
1ape
1380
Figure 2: Swapping n-grams and phrases. (a) Examples of basic n-gram shuffles, where colors indicate the units of shuffling. (b) Distortion metric computed at cach layer, conditioned on n-gram size. Error bars hereafter represent standard error across 400 examples. (c) example parse tree, with phrase boundaries shown as grey brackets, and two low-order phrases marked; and examples a phrasal and control swap, with colors corresponding to the phrases marked above. (d) Distortion, computed at each layer, using either the full sentence, the subsentence of unswapped words, or the subsentence of swapped words, conditioned on swap type.
4 RESULTS
We summarize our findings for the different perturbations below. While not shown in the main results, we note that randomly-initiaiizz (i.e. untrained) models (somewhat unsuprisingly) exhibit flat distortion trend for all perturbations (see Sec. A.2). This indicates that the patterns observed here are due to the model's structural knowledge acquired through training, and not simply due to the underlying architecture.
4.1 CHARACTERISTIC SCALE INCREASES ALONG BERT LAYERS
When we shuffle in units of larger n--rrmms it only introduces distortions in the deeper BERT layers compared to smaller n-gram shuffles The n-gram sized shuffles break contexts larger than n, while preserving contexts of size n smaller. Interestingly, smaller n-gram shuffles diverge from the original sentence in the early layers (Fig. [b,, top curve), implying that only in early layers are representations built from short-range contexts. Larger n-gram shuffles remain minimally distorted for 'longer' (Fig. P bottom curve), implying that long-range contexts play a larger role deeper layer representations.
Phrasal boundaries matter Since BERT seems to build larger contexts along its layers, we now ask whether those contexts are structures of some grammatical significance. A basic and important syntactic feature is the constituent phrase, which BERT has previously been shown to represented in some fashion (Goldberg] [019 Kim et al. 2020). We applied two targeted probes of phrase structure in the BERT representation, and found that phrasal boundaries are indeed influential.
If we swap just two n-grams, the BERT representations are less affected when phrases are kept intact. We show this by swapping only two n-grams per sentence and comparing the distortion when those n n-grams are phrases to when they cross phrase boundaries (Fig. Bl), where we control for the length of n-grams that are swapped in both settings. There is less distortion when respecting phrase boundaries. Furthermore, the distortion is evident among all feature vectors, including those in the position of words which did not get swapped (Fig. (dd). The global contextual information, distributed across the sentence, is affected by the phrase boundary.
4.2 PIRASE IIERARCIY MATTERS
Having seen that representations are sensitive to phrase boundaries, we next explore whether that sensitivity is proportional to the number of phrase boundaries that are broken, which is quantity related to the phrase hierarchy. Instead of swapping entire phrases, we swap two adjacent words and analyze the distortion based on how far apart the two words are in the constituency tree (Fig. 3 This analysis varies the distance in the deeper tree structure while keeping the distance in surface form constant (since we always swap adjacent words).
If the hierarchical representations are indeed being gradually built up along the layers of these pretrained models, we expect distortion to be greater for word swaps that are further apart in tree distance. We indeed find that there is a larger distortion when swapping syntactically distant words (Fig.Bp). This distortion grows from earlier to later BERT layers. Furthermore, when looking at the per-head representations of each layer, we see that in deeper layers there are more heads showing
'Nte that for adjacent words, the numher of broken phrase boundaries equals the tree distance minus two.
5

---

Under review as a conference paper at ICLR 2021
a
Dietert on from aduoent word rwase in BEXT
Pank cont., distettion tree disiancs
danollaa
(pea
x))
-11
Inphrase swaps
Out Wwee:
The monil har dats not bei
LAge7
Hand
d
oxtiRarian, phase
Ou grane
P 5Y50
Leyer
Figure 3: Syntactic distance affects representational distortion. (a) An example of adjacent swaps which do and do not cross a phrase boundary, with low-order phrases colored. Phrase boundaries are drawn in red. (b) Distortion in each layer, but conditioned on the tree distance. (c) For each head (column) of cach layer (row), the (Spearman) rank correlation between distortion and tree distance of the swapped words. Colors are such that red is positive, blue negative. (d) Histogram of PMI values. for pairs in the same phrase and not. (e) Similar to b, but averaging all out-of-phrase swaps, and separating pairs above ('high') below ('low") the median PMI.
positive rank correlation between distortion and tree distance (Fig. 30). In addition to sensitivity to phrase boundaries, deeper BERT layers develop a sensitivity to the number of boundaries that are broken.
Controlling for co-occurrence Since words in the same phrase may tend to occur together more often, co-occurrence is potential confound when assessing the effects of adjacent word swaps. Co-occurrence is a simple statistic which does not require any notion of grammar to compute indeed is used train many non-contextual word embeddings (e.g., word2vec Mikolov et al. (2013). GloVe (ePennnggnoon et al. (2014)). So it is natural to ask whether BERT's resilience to syntactically closer swaps goes beyond simple co-occurrence statistics. For simplicity, let focus on whether swap occurs within phrase (tree distance 2) or not.
As an estimate co-oceurrence, we used the pointwise mutual information (PMI). Specifically, for p(w.r) two words w and v the PMI is log p(w)p(v)- which is estimated from the empirical probabilities. We confirm that adjacent words in the same phrase do indeed have a second mode at high PMI (Fig, (3)). Dividing the swaps into those whose words have high PMI (above the marginal median) and low PMI (below it), we can see visually that the difference between within swaps persists in both groups (Fig.Be) For more careful statistical test, the appendix we show results from running linear regression between distortion and the phrase boundary which accounts for dependency on any smooth function of PMI (see details A.4). Even when accounting for the effect of PMI, there is significant correlation between the breaking of a phrase and the subsequent distortion. This indicates that the greater distortion for word swaps which cross phrase boundaries is not simply due surface co-occurrence statistics.
swaps and out-of-phraa
Effects on linguistic information Do our input perturbations, and the resulting the distortions, reflect changes in the encoding of important linguistic information? One way to address this question, which is popular in computational ncuroscience (DiCarlo & Cox (207)) and more recently BERTology (Liu et al. 2019a Tenney et al. 2019), to see how well linear classifier trained on a linguistic task generalize lizerallizeeiie from the (representations of the) unperturbed sentences the perturbed ones. With supervised probes, we can see how much the representations change with respect to the subspaces that encode specific linguistic information.
Specifically, we relate representational distortion to three common linguistic tasks of increasing complexity: part of speech (POS) classification; grandparent tag (GP) classification (Tenney et al.

---

Under review as a conference paper at ICLR 2021
POS classification
Grandparent classification
Canstituency distance probe
12
0.35
10
Average distortion
0.30
Leyer 12
0.25
Tree distance
Free distance
Tree distance
0.20
0.15
0.10
Layer
0.00
0.02
0.04 0.06 0.08
0.00
0.05
0.10
0.00
0.01
0.02
0.03 0.04
Average decrease log-likelhhood
Average decrease in log-likelihood
Average decrease in log nog-likelihaa
Figure 4: Distortion and inference impairment for increasing linguistic complexity. In each plot, a point is the average (distortion, 'impairment') for given layer and given class of word swap distance. Points connected by lines according to their swap type (i.e. tree distance). The circles are colored according to layer (see right for a legend). Averages are taken over 600 test sentences, with one of each swap type per sentence, and both distortion and log-lik log-likelihood are computed for every word in the sentence.
2019); and a parse tree distance reconstruction (Hewitt & Manning 2019 The probe trained on each of these tasks is a generalized linear model, linearly mapping a datapoint (i.e. BERT representations from different layers) to a conditional distribution of the labels, for more details). Thus a ready measure of the effect of each type of swap. for single sentence. is log p(y(((x.)) - Averaging this quantity over all datapoints gives measure of content-specific distortion within representation, which we will call "inference impairment".
(see A.5
log p(y)0(X;)) (x:)), where x; is same datum as x; in the perturbed representation"
Based on the three linguistic tasks, the distortion we measure from the adjacent word swaps is more strongly related to more complex information. The inverted L shape of Fig. 4a suggests that increasing distortion is only weakly related to impairment of POS inference, which is perhaps unsurprising given that POS tags can be readily predicted from local context. A deeper syntactic probe, the GP classifier ( does show a consistent positive relationship, but only for swaps which break a phrase boundary (i.e. distance >2). Meanwhile, impairment of the distance probe (4p), which reconstructs the full parse tree, has a consistently positive relationship with distortion, whose slope proportionate the tree distance of the swap. Thus, when specifically perturbing the phrasal boundary, the representational distortion is related to relatively more complex linguistic information.
4.3 POSSIBLE MECHANISTIC EXPLANATION THROUGH ATTENTION
In the transformer architecture, contexts are built with the attention mechanism. Recall that attention mechanism for allowing input vectors to interact when forming the output, and the ultimate output for a given token is a convex combination of the features of all tokens (Eq. While any interactions between inputs must be mediated by attention, it is not obvious how the contextual information of particular layer is captured by attention in that layer. It has been shown qualitatively that, within a layer, BERT allocates attention preferentially to words in the same phrase (Kim et al. 2020). Our next suite of experiments asks whether this might explain the observed relationship between tree distance and distortion.
We find that in many Transformer heads, the attention- much like distortion is proportional to the syntactic distance between two words. Fig. 5P summarizes this trend by showing the Spearman rank correlation between the parse tree distance of adjacent word pairs, and the attention paid between those words. Different attention heads within a layer range from correlated to anti-correlated and with slightly more positively correlated heads in deeper layers. However, there is great variability in this, suggesting that only certain heads learn to specialize to syntactic phenomena.
While the original paper predicted dependency tree distance, in this paper instead predict the constituency distance.
'POS- and GP-tag prediction outputs a sequence of labels for each sentence, while the distance probe outputs the constituency tree distance between each pair of words. Then log p(y)p(()))))))) (xx)) ) is simply the log probability of individual label.
7

---

Under review as a conference paper at ICLR 2021
a
Attention all heads BERT layer
(The luxury maker) (last year) sold (1,214 (the U.S.)
Altended
Rank corr., attention & tree distance
03
0.4
2000 wil sterrii covarote
62
0.1
0.0
0.1
-02
-03
11 -0.4
2.4
-32
0.4
Hoad
1968 COTT Intterdinn treu dalanoo
praluc witout asteniin coveriate
Figure 5: Attention provides a possible explanation for the trends we observe distortion. (a) An example of the attention matrices for all heads in single layer (layer 8), given the above sentence as input. Phrases in the sentence drawn blocks in the matrix. (b) The rank correlation of attention
tree distance for all the hcads/layers. (c) The rank correlation coefficients of distortion (y-axis) and attention (x-axis) against tree distance, colored by layer. Marker size is proportional
minus p-value of the distortion/distance correlation. (d) A comparison of p-values on the distortion vs. tree distance correlation without (x-axis) and with attention splines as covariates (y-axis).
We observe that at least some of the observed correlation between swap-induced distortion and parse distance can be accounted for by attention. Of course, all interactions between inputs are mediated by attention, but it is not certain that the contextual information in particular layer comes from the attention that layer. To test whether the correlation between
distance and distortion persists when accounting for attention, we used linear regression with any smooth function of attention as covariate (see A.4). We observe larger p-values in the controlled regression, indicating that the correlations become less significant when accounting for attention (Fig. (dd). This suggests that the attention in each layer helps build sensitivity to syntactic distance.
DISCUSSION
In this paper, we used the representational change response to perturbed input in order to study the encoding of hierarchical phrasal structure in deep language models. We also identify link between the perturbation-induced distortion to the magnitude of attention paid to words within and out of phrase boundaries as a potential mechanistic explanation. Across different models, find that the word-level contexts used to represent a sentence grow in size and complexity along the model layers, similar the inereasing size of receptive fields found in sensory systems.
In neuroscience, it is well accepted that small receptive fields tuned to simple stimuli (i.c., edges) are combined to form larger receptive fields tuned more complex stimuli (i.e., objects) (Riesenhuber & Poggio 1999). In language, a phrase within sentence can serve as a conceptual unit, much like an object a visual scene, thus motivating our perturbation-bast sensitivity of phrases. We showed that BERT and its variants are indeed sensitive to the phrasal unit, as demonstrated by greater invariance to perturbations preserving phrasal boundaries compared to control perturbations which break the phrasal boundaries (Fig. for BERT, SM for other models).
probe for object-like representational
Our method and results suggest many interesting future directions. We hope that this work will motivate: (1) formal theory of efficient hierarchical data representations distributed features; (2) search for the causal connection between attention structure, the representational geometry, and the model performance; (3) potential applications in network pruning studies; (4) an extension of the current work hypothesis generator in neuroscience to understand how neural populations implement tasks with an underlying compositional structure.
8

---

Under review as a conference paper at ICLR 2021
R2
Baseline for rank correlations
nggameuullin (rubword distortio))
Adjacent swaps (sulyard distotionn
Park coreention wtth ditance
DCRT model
9gmm c3n
tranee
2.3
Untrained
0.5
0.7
0.2
0.4
10
Layer
Change in swapped word attention (dansp)
rank corrtdayep treedist
Consitional correlations
6.20
12:
b
pieistorton, www.treeee de
10
0.15
10
0.4
-.2
Layer
0.05
0.00
0.0
-0.05
-..2
-0.15
-0.4
-03
-0.20
8.2
0.0
622
9A
Head
pldistortion, free disti
classification
Grandparent classification
Constituercy tree reconstruction
-0.30
-222
Ikslhood
Weerage Beaitaodd
-225
-0.35
ditance Unswasped
-0.7
-0.8
-2.40
-0.55
-1.
-2.45
10
Layer
R3
Simulation scaledOtn layer)
Actual BERT vectors (1-gram shuffle)
Different normalizations
1.4
Raw
2-score
Dstortion
1.0
1.0
Layer var.
nomm
0.8
0.5
Naive
Z-scored
0.2
0.2
Variance layer
10
5.0 Trivial scale
10.0
Simulated "Layer"
R4
R5
Distartion phrase type swap size
Distortion ngram size swap size
Phrase type NP
Coefiicient (90% cont. Incervl)
BERT model
Coefficient (90% cont interval)
Pre traiteddd
0.00
0.5
0.03
Untroined
Distortion
0.4
0.02
-0.02
Untrained
Pre trained
model
0.01
0.3
0.00
-0.06
10
10
-0.08
10
Layer
Layer
Layer
Figure Additional experiments and clarifications (by reviewer).

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jacob Andreas. Good-Enough Compositional Data Augmentation. In Proceedings ACL, 2020.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In Proceedings of ICLR, 2018.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine transla- tion. In Proceedings of ICLR, 2018.
Yonatan Belinkov and James Glass. Analyzing hidden representations in end-to-end automatic speech recognition systems. In Advances in Neural Information Processing Systems, pp. 2441-2451, 2017.
Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.
Uri Cohen, Sue Ycon Chung, Daniel Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. bioRxiv, pp. 644658, 2019.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 118)3333--44, 2007.
Nai Ding, Lucia Melloni, Hang Zhang, Xing Tian, and David Poeppel. Cortical tracking hierarchi- cal linguistie structures in connected speech. Nature neuroscience, 19(1):158, 2016.
Jeffrey L Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2-3):195-21) 225, 1991.
Allyson Ettinger. What bert is not: Lessons from new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34 48, 2020. doi: 16. a\ L002998
Richard Futrell, Ethan Wilcox, Takashi Morit, and Roger Levy. RNNs psycholinguistic subjects: Syntactic state and grammatical dependency. 1rXiv:1809.01333 2018.
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pp. 32-42, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/N19-1004 URL https: www a.c.elllwee wtt.://www.......atttooooooo///a
Scott Gigante. Adam S Charles, Smita Krishnaswamy, and Gal Mishne. Visualizing the phate of neural networks. Advances in Neural Information Processing Systems, pp. 1840-1851, 2019.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint arxi::
Bruce Hansen. Econometrics.
2000.
cronometrics.
John Hewitt and Percy Liang. Designing and Interpreting Probes with Control Tasks. In Proceedings of EMNLP, 2019.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4129-41335 2019.
10

---

Under review as a conference paper at ICLR 2021
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1367-1377, San Diego, Californa,, June 2016. Association for Computational Linguistics. doi: 10. 1865/vvvv
N16-1162. URL URL htt.h:tts://// /wwp://wwww.. www.wwwww..oooooo www.w//wwww..w..a..e...ee..e.eee wwt.://ooaaanoogooggooooooo /Ni 61662
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of EMNLP, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
Charles Kemp and Joshua B Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences, 105(31):10687-1062) 2008
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction. In Proceedings of ICLR, 2020.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In Proceedings of ICLR, 2018.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2019.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transfcrability of contextual representations. CoRR, hts/1993.08855, 2019a. URL
http://arxiv/ wttp:/ogg///..//// 08855
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019b.
Jonathan Mamou, Hang Le, Miguel A Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Yeon Chung. Emergence of separable manifolds in deep language representations. arXiv preprint arXiv:2006.0100 2020.
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Fer- guson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, HLT '94, pp. I 114-119, Stroudsburg, PA, USA, 1994, Association for Computational Linguistics. ISBN 1-55860-357-3. doi:
URL / org/ 3111//17111111111111111111111111 http://b....rr111...1111111/11.. 1075835
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111-3119, 2013.
Francis Mollica, Matthew Siegelman, Evgeniia Diachek, Steven T Piantadosi, Zachary Mineroff, Richard Futrell, Hope Kean, Peng Qian, and Evelina Fedorenko. Composition is the core driver of the language-selective network. Neurobiology of Language, 1(1):104-134 2020.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 5732-5741. Curran Associates, Inc., 2018
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1543,,
2014. URL TRRLhttp: wwww ac web. .ow.b.orrg//oo http://w...b..or////aaooooo///// wttp://ahhhooooyy/1111111
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha DicksDickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6076-6085. Curran Associates, Inc., 2017.
11

---

Under review as a conference paper at ICLR 2021
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and Measuring the Geometry BERT. In Advances in Neural Information Processing Systems, pp. 8592 8600, 2019.
Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(11):019--1025, 1999.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter, 2019.
Morteza Sarafyazd and Mehrdad Jazayeri. Hierarchical reasoning by neural circuits in the frontal cortex. Science, 364(6441) :eaavv91)) 2019.
Andrew Saxe, James McClellans, and Surya Ganguli. Learning hierarchical categories in deep neural networks. Proceedings of the Annual Meeting of the Cognitive Science Society, volume 35, 2013.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07900 2015.
Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel J Gershman. The hippocampus as predictive map. Nature neuroscience, 20(1 1):1643, 2017.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Proceedings of ACI., 2019.
Gowanlock R Tervo, Joshua B Tenenbaum, and Samuel Gershman. Toward the neural implemen- tation of structure learning. Current opinion in neurobiology, 37:99-105, 2016.
Doris Y Tsao and Margaret S Livingstone. Mechanisms of face perception. Annu. Rev, Neurosci, 31: 41 417-437, 2008.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In Proceedings of EMNLP, 2018.
Ethan Wilcox, Roger Levy, and Richard Futrell. Hierarchical representation in neural language models: Suppression and recovery of expectations. arXiv preprint wxXiv:1906.04066 2019.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Lc.
XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of NeurlPS, 2019.
A SUPPLEMENTARY MATERIAL (SM)
A.1 ADDITIONAL DETAILS ON THE DATASET
In this section, we describe additional details of the manipulations done on the datasets.
n-gram shuffling For given a sentence. we split it into sequential non-overlapping n-gram'ss 's from left to right; if the length of the sentence is a multiple of n, the remaining words form an additional m-gram, m < n. The list of the n-gram's 's is randomly shuffled. Note that the I-gram case is equivalent to a random shuffling of the words. In our analysis, we consider n-grams, with varying from (i.e., individual words) to 7 and all the sentences have at least 10 words.
We provide here an example of n-gram shuffling.
Original: The market pessimism reflects the gloomy outlook in Detroit
1-gram market pessimism the 's Detroit in The gloomy reflects outlook
2-gram: 's pessimism in Detroit The market reflects the gloomy outlook
3-gram The market 's gloomy outlook in pessimism reflects the Detroit
12

---

Under review as a conference paper at ICLR 2021
4 gram: in Detroit The market 's pessimism reflects the gloomy outlook
S-gram the gloomy outlook in Detroit The market 's pessimism reflects
6-gram: outlook in Detroit The market 's pessimism reflects the gloomy
7-gram: in Detroit The market 's pessimism reflects the gloomy outlook
Phrase swaps Using constituency trees from the Penn Treebank et al. 1994). we define phrases constituents which don't contain any others within them. (See Fig. 2 Fig. E in the main text.) Phrase swaps thus consist of swapping one phrase with another, and leaving other words intact.
To provide an appropriate control perturbation, we swap two disjoint n-grams, which are the same size true phrases but cross phrase boundaries.
Adjacent word swaps To better isolate the effect of broken phrase boundaries, we used adjacent word swaps. Adjacent words were chosen randomly, and one swap was performed per sentence.
A.2
ADDITIONAL MODELS
A.2.1
PRE-TRAINED MODELS
We present briefly the pre-trained models that we used for the experiments
BERT bert-base paase-cased 12-layer, 768-hidden, 12-heads, 10M parameters.
RoBERTa roberta -base. 12-layer, 768-hidden, 12-heads, 125M parameters.
ALBERT a albert-baseeeee
repeating layers, 128 embedding, 768-hidden, 12-heads,
11M parameters.
DistilBERT disti bbert beer--uccaeeedd 6-layer, 768-hidden, 12-heads, 66M parameters.
The model distilled from the BERT model bert-base-uncase checkpoint.
XLNet xlnet -baeet-base-casee 12-layer, 768-hidden, 12-heads, 110M parameters.
Note that the hidden size is 768 across all the models. For each pre-trained model, input text is tokenized using its default tokenizer and features are extracted at token level.
A.2.2 UNTRAINED MODELS
To control for properties which come purely from the architecture, we also compute with randomly- initialized (untrained) models. All model weights are random number. Note that this random initialization has also an impact on the embedding layer.
Here, provide side-by-side comparison of results on trained untrained model from each model class (n-gram: Fig. 7 adjacent: Fig. 8 Across different model classes and tasks, none of our results were replicated in the untrained models. Thus the pattern of invariance we report cannot be explained by architecture alone.
A.3 ADDITIONAL DISTORTION METRICS
In addition to the scaled Frobenius distance, we considered other ways of measuring distortion in the representation. Here we report results for two different metrics canonical correlation analysis and shifted cosine similarity.
CCA Canonical correlations analysis (CCA) Raghu et al. (2017) measures the similarity of two sets of variables using many samples from each. Given two of random variables (I1, (2, an)) and y (y1 ....... ym ), CCA finds linear weights a E T and b E Th which maximise cov(a- x, b b. In our context, treat the representation of the original sentence x, and the representation of the perturbed sentence as and the resulting correlation as a similarity measure.
We the httpa///wwt/attaoooooooooooonnta
from httpthttt///ttt
13

---

Under review as a conference paper at ICLR 2021
bert basee-assed
n-gram
Presroined Untraine
Lotte
Gntel
unch
cotss
cases
Loter
unise
Geseez
albert-base-vi
roberta-base
11
08
31
96
35
24
Lecooo
Lertt
Larzu
Liees
Laran
carssa
caress
ceroto
83
L40000
Lock
Lortoo
carcse
Lanse
wortee
cacor
Laraae
Leraaro
Leveas
distibbert-baseuncaee
xinet-base-cased
00
600
LArEKT
LeND3
LATES
LANESS
Lonoee
Lanto
Lonto
Latdnd
Larae
Lonts
Lasser
Lamae
Letter
lamaoo leseess COOEE
Figure 7: Distortion per layer in r-gram shuffling experiments, using different transformer architectures. Lines are the mean Frobenius distance, and the shading is 1 standard error the mean.
14

---

Under review as a conference paper at ICLR 2021
bert base-base-dded
Distertion matble
high bigram PMI
pretrained, phrase pretrained, phrase untrained, phrase untrained, out phrase
LAISS Jank
Lateer
canse
Lamer
Lema
Laiaee
Larees cinen Lemac
albert-base-vl
roberta-base
6a5
Diporrton
625
655
L01800
(Ar50)
LAnto
Lerees
Lartee
Lerass
caree
Larzor
Lerees
Leraia
LANBo
(4180)
cinto
larees
Lerees
carto
Lareae
Lerzo
Lares))
distibber-tass--uccsedd
iinet-bese-cesed
Distortion metric
13
100001
LA1DO
LATOM
LASSO
LASSE5
LEVER11
Figure 8: Distortion per layer in the adjacent word swapping experiments, using different transformer architec- Lines the Frobenius distance, and the shading +1 standard error of the
15

---

Under review as a conference paper at ICLR 2021
n-gram shuffling
Phrase swap
CCA
Ful sentetic
Dity swassel
Pntruiaal phrad Pect rsl centol
Larees
122
PWCCA
0.0
0.0
08
080
Lotta 503 100
10000
69
Cosine
03
gues
Figure 9: Results from the pretrained BERT model on the n-gram shuffling and phrase swap experiments, using alternative distortion metrics described in the text.
Since CCA requires many samples, we use the set of all word-level representations across all per- turbed sentences, For example, to construct the samples x from S perturbed sentences, we get use |X||X2|..x [.s).. where each X; E 768x1; Unless specified otherwise, S = 400. For good estimates, CCA requires many samples (on the order of at least the number of dimensions), and we facilitate this by first reducing the dimension of the matrices using PCA. Using 400 components pre- serves ~ 90% of the variance. Thus, while CCA gives a good principled measure of representational similarity, its hunger for samples makes unsuitable
per-sentence metric.
We also measured distortion using Projection Weighted Canonical Correlation Analysis (PWCCA). an improved version of CCA estimate the true correlation between tensors Morcos al. 22118) 10
As reported in Figure 9 we did not find any qualitative differences between PWCCA and CCA in our experiments.
Cosine A similarity measure defined individual sentences is the cosine between the sentence- level representations. By sentence-level representation, we mean the concatenation of the word- level vectors into single vector N (where N is the dimension of each feature vector). Treating each dimension of the vector a sample, we can then define the following metric: corr (s Toriginal s
swappedd This equivalent to computing the cosine of the vectors after subtracting the (scalar) mean across dimensions, hence we will refer "cosine'.
A.4 PARTIAL LINEAR REGRESSION
order to control for uninteresting explanations our results, we often make use of simple method for regressing out confounds. Generally, want assess the linear relationship between X and Y.
Foro both CCA and PWCCA, we use the implementation from htcps.///iithub.commeeee
svcca
16

---

Under review as a conference paper at ICLR 2021
Without PMI
With PMI
Layer
Coeff. x10
p-value
Coeff.
x x10
p-value
Emb.
-0.21
5.6 x 10-5
0.11
9.4 X 10
0.11
3.4 x 10-2
0.05
4.2 x 10-1
2
-0.74
< 10-16
0..3
2.12 10
3
-1.6
< 10016
-1.3
2.2 10
4
-2.0
< 10-16
-1.4
x.4 10-16
-2.1
< 10-16
-1.5
8.8 10-16
6
224
10-16
-1.7
< 10-16
7
-2.6
< 10-16
-1.7
1.6 x 10-15
-3.4
< 10-16
-2.3
< 10-16
9
-388
10-16
-27
10-16
10
-4.1
< 10-16
-3.0
< 10
33.8
< 10 16
-28
10-16
12
-42
10-16
-3.1
< 10-16
Table 1:
Coefficients and p-values of the regular ('without PMI)) and controlled ('with PMI') regressions of distortion against phrase houndary.
when accounting for the (potentially non- non-linear) effect of another variable Z. In our experiments, X is always the swap-induced distortion and Y is the swap type, like integer-valued tree distance or binary-valued in/out phrase. We wish to allow E[Y| Z] and E|X|Z] which is achieved by the least-squares solution to the following partially linear model:
] to any smooth function Z,
Y 3,X + 82 f(Z) = (Y ( ~ 8x - 2[X
where f(z) is a vector of several (we use 10) basis functions (we used cubic splines with knots at 10 quantiles) of Z. Both regressions have the optimal 8z. but the
the left is computationally simpler (Hansen, 2000). The standard confidence intervals on 83 apply.
Intuitively, the 3. obtained by the partially linear regression above related to the conditional correlation of X and Y given Z: p(X, Y Z| Like an unconditonal correlation, it will be zero if X and Y are conditionally independent given Z, but not necessarily vice versa (both X and Y must be Gaussian for the other direction to be true). To compute conditional rank correlations (which assess monotonic relationship between X and Y), we rank-transform X and (this changes the confidence interval calculations).
We apply this method to attentions in Fig. In these supplemental materials, we will also report the results when X is the binary in/out phrase variable, and Z is PMI. The full p-values and coefficients of the uncontrolled and controlled regressions can be found in Table where we observe that past layer 2, the p-value phrase boundary is very significant (p < 10 10-12).
A.5 SUPERVISED PROBES
In this section, we describe the experiments based on the three linguistic tasks: parts of Speech (POS); grandparent tags (GP); and constituency tree distance.
The POS and GP classifiers were multinomial logistic regressions trained to classify each word's s POS tag (e.g. 'NNP', 'VB') and the tag of its grandparent the constituency tree, respectively. If word has no grandparent, its label is the root token 'S'. The probes were optimized with standard stochastic gradient descent, 50 sentences from the PTB per mini-batch. epochs, 10-3 learning rate, were sufficient to reach convergence.
The distance probe is a linear map B applied to each word-vector w in the sentence, and trained such that, for all word pairs j. TreeDist(i,) j) matches (ww. w as closely possible. Unlike the classifiers, there is freedom in the output dimension of B; we used 100, although performance and results from Hewwit Manning (2019) in two ways: (1) we use constituency trees, instead of dependency trees, and (2) instead of an 1.1 loss function, use the Poisson (negative) log-likelihood the loss
empirically the same for any choice greater than ~ 64. Our probes are different
17

---

Under review as a conference paper at ICLR 2021
function. That is, if A | wy)ll. and yi. TreeDist((..))
4 = Yo.j log \io A log Vi.j!
Otherwise, the probes are trained exactly as in Hewitt & Manning 2019). Specifically, we used standard SGD with 20 sentences from the PTB in each mini-batch, for 40 epochs.
Evaluation A linear model is fit to maximize px)]0(x)), with probability function (multinomial for classifiers, Poisson for distance), and x coming from the unperturbed transformer representation. We evaluate the model on X, which are the representations of the data when generated from a perturbed sentence. We take the average log p(y)((x,)) - log p(y (K:)) over all the data i all sentences. For example, all words for the classifiers, and all pairs of words for the distance probe. Concretely, we are just measuring the difference in validation loss the same probe on the x data and the data. But because the loss is appropriate probability function, can interpret the same quantity as a difference in Log-likelihooe between the distribution conditioned on the regular representation and that conditioned on the perturbed representation. Distortion is similarly computed using the full sentence, providing a number for each swap in each sentence.
18


Reviews:
[{"review_id": "x-MlO8prs--", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThis paper addresses how pre-trained Transformer models build their contextual representations along with the layers by measuring changes in the outputs on a series of probes. Specifically, a probe involves swapping words in a sentence and measuring the distortion in the representations. The series of probes are designed to test how the models respond to different syntactic-related swaps including n-gram size and syntactic phrase boundary and distance. The results mainly focus on BERT (base-cased), but some results of the other variants have the same trend. The author first confirms that the observed distortions are the learned behavior by showing a \"flat\" distortion trend of untrained models. When subjecting to n-gram swaps, the results show that smaller n-grams have a larger distortion. We can also see that later layers are affected more than earlier layers (for all kinds of swaps). Regarding hierarchical phrase structure, the results show that the phrasal boundaries are important, even accounted for PMI of swapped words. The rank correlations between distortion and the syntactic distance are higher in the deeper layers across all attention heads, but the correlations are somewhat weak (around 0.3). In addition, distortions have a larger impact on a relatively more complex task (parse tree distance reconstruction > POS tagging).  Finally, an analysis of the attention shows that the attention layers contribute to the observed distortion. The main conclusion is that BERT (and its variants) builds its contextual representation by increasingly incorporating more phrasal units along with the layers. \n\n### Recommendation:\nOverall, I would like to recommend this paper for the conference. The paper presents a novel approach to study the behavior of Transformer models and the findings are quite interesting. My main concerns are that some of the implications of the results are quite hard to follow, and I am not sure how they might impact future downstream researches.\n\n#### Pros:\n- I would like to commend the author for clear writing. The paper was very well written and I enjoyed reading the paper.\n- The main goal of the paper is to understand (or at least shade some light on) how BERT works. I think it is equally important as the interpretability of the models (i.e., explain the prediction).\n- The perturbation-based probes are easily related to language features than other methods such as attention visualization or a salience-based approach. Although there exists work that uses the same approach, I think this paper presents novels finding on how the representations of the pre-trained models \"distort\" given different perturbations.\n- The paper provides comprehensive experiments including many carefully designed experiments to reveal the key insights.\n\n\n#### Cons:\n\n- Some of the key findings, such as \"long-range contexts play a larger role deeper layer representations\" are very hard to follow. \n- The differences between the distortions among tree distance and the rank correlations are somewhat weak, or rather we do not have a baseline to compare against. I am not sure how strong finding 3 (in the introduction) is supported by the results. Perhaps, to improve the experiment further, the author could include the results of different model architectures (RNN-based models) to compare and contrast.\n- I think the experiment on the attention weights is not helpful. Since attention layers are the main mechanism in the Transformer models, it is not surprising that it plays an important role. In the end, it is still inconclusive how attention captures the contextual information. The author could analyze the changes or distortion of attention weights.\n\n### Questions:\n1. It is very important how the author handles subwords. The explanation should not be a footnote and there should be more detail on when the \"average\" happens (i.e., from input to BERT or only distortion computation).\n1. I can see why the author would like to aggregate subwords into words when probing. How much do the averaging embeddings change the distortion? Does it affect the findings? Wouldn't it be more helpful to include a probe regarding subwords?\n1. What is the corpus that PMI is computed on?\n1. What are the base likelihoods (without the perturbation)? Could we imply that the average decrease in log-likelihood is explained by the base likelihoods?\n\n### Comments:\n- A.4 is not quite related to the attention covariate, though the intention is understandable.\n- Fig. 5e does not exist. I think you mean Fig. 5d (which is quite hard to interpret).\n- I think a more careful analysis of the distortion vectors (Fig 1c) should be done (i.e., why is it desirable that the models should be robust to some perturbations? And what is the trend of the distortions across different types of perturbations?). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "vEP3cI5QCTx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the ability of BERT model to learn good representations of sentences, through a purely empirical study, there is no other contribution of a new model or a new analysis technique. In the experimental analysis, phrases or words are swapped and the corresponding changes in the sentence representations are analyzed from all the hidden layers. I didn't see any surprising outcomes from the analysis, so not sure how impactful this work is. I would rather see this kind of work in a workshop track for ICLR.\n\nA minor comment: only one dataset is considered in analysis. I would like to see the analysis sentences from the training set as well as many test sets.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WDOZdzxq6xj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Overall ---\n\nThis paper provides some insights into the relation of BERT's output w.r.t. the parser tree (in terms of constituent phrases) of the input sentence. As some previous work has pointed out, BERT model contains the parsing information (e.g., Hewitt & Manning NAACL’19)). This work can be regarded as a moderate reification and improvement of that thought (but it is still limited to existing scopes and methodologies). \n\nThe merits of this paper: (1) the paper reveals some interesting facts such as BERT are sensitive to phrasal hierarchy and there are behavioural discrepancies between different layer; (2) the experiments are comprehensive, including both the distortion analysis and conventional probe approaches.\n\nIn my point of view, the main issue of this paper is, like many other works, there is no strong theoretical explanation for the phenomenon being investigated. In this sense, the novelty of this paper is not so strong.\n\n--- Major comments ---\n\n1.\tIn experiments, it is not clear whether the randomness of BERT itself has been deducted. The randomness could be caused by the dropout operation which may lead to the discrepancy on output even using the same sentence.\n2.\tIn the future version, I recommend to further provide two tests: (1) in current settings, the distortions are showed in sentence-level (e.g., by summing up all distortions within a sentence?). I would like to see a finer-grained test, i.e., whether the most distortion parts are produced by the swapping part; (2) The genre of the Penn treebank dataset is limited to general texts or articles that may be seen in the training corpus of BERT. I would recommend testing on other domains (e.g., biomedical or academic) that BERT never saw before (or structure-less data that do not present a syntactic structure).\n3.\tDo X~ and X in section 2.3 use the same mean and deviation?\n4.\tIt seems that Fig. 2(c) takes account into both NP and VP, what if we only constrain the phrases to be only NP? Will the distortion becomes large since swapping subject and object will lead to totally different meanings?\n5.\tThere is a lack of explanation about \"all words\", \"swapped\", and \"unswapped\" settings of Fig. 2(d).\n6.\tIs there any intuition for the step back on the layers 11 and 12 in Fig. 4(b) and 4(c)?\n\n--- Minor comments ---\n\n1.\tStrictly speaking, the terminology “gram” in Fig.2(a) should be called \"chunk\", as “grams” are usually related with sliding window thus overlap with each other.\n2.\t“Fig.3(a)” in the paragraph just above Section 4.2 should be “Fig.2(c)”.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uoi5qYXu1X", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the sensitivity of BERT representations to different kinds of permutations in the input sentence. These transformations include n-gram permutation, span swaps (with or without crossing syntactic phrase boundaries), adjacent token swaps (with different syntactic distance). The authors measure the l2 distance between representations coming from original and perturbed input.\n\nOverall, the results suggest that BERT is sensitive to hierarchical phrase structure.\n\n------------------------------------------------------------------------------------------------------\nStrengths\n\nThe idea of measuring a network’s response to input transformations is nice and potentially could be used to test different kinds of hypotheses.\n\n------------------------------------------------------------------------------------------------------\nWeaknesses\n\nMy main concern is the measure used for a distance between representations. Namely, this is the l2 distance, which accounts for distance in neurons. Therefore, it does not tell us to what extent representations encode different things but rather how different are their individual neurons. For example, different phenomena can be either focused in a network (encoded by only a few neurons), or distributed (see, e.g., the paper [1] or a more recent [2]). \n\nTherefore,\n1) the phenomena that suffer from perturbations have different impact on the l2 distance because they affect different number of neurons\n2) as a consequence of the above, it is not clear what to conclude from the proposed results: they are likely to be not because of the high-level differences in what original/perturbed representations encode, but rather in\nhow these underlying phenomena affect individual neurons. \n\n(To measure differences in what representations encode you can use, for example, PWCCA (NeurIPS 2019 “Insights on representational similarity in neural networks with canonical correlation”) or other related measures.)\n\n[1] AAAI 2019 “What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models”\n\n[2] EMNLP 2020 “Analyzing Individual Neurons in Pre-trained Language Models”\n\nOther concerns.\nIf we put aside the validity of the measure, it is still not clear what to take out of these results: they rather show that the method passes sanity checks rather than tell us something we didn’t know before. For example, there’s a huge line of works showing that BERT “understands” syntax and phrase composition (looking at representations, geometry, attention heads, etc). Hence the results stated in contributions 2, 3, 4 are not surprising. Contribution 1 is also more of a sanity check: of course, shuffling smaller parts has to cause more distortion than shuffling longer phrases. \n\n------------------------------------------------------------------------------------------------------\nQuestions\n\nSection 4.1, paragraph 1: when referring to figure 2b, you say “When we shuffle in units of larger n-grams, it only introduces distortions in the deeper BERT layers compared to smaller n-gram shuffles.”\nI have trouble seeing this from the figure. For all lines, the distortion goes up almost linearly from layer 2 to layers 9-10. Yes, shuffling larger ngrams causes lower distortion, but this is expected. Am I missing something?\n\n------------------------------------------------------------------------------------------------------\nMissing references (in addition to mentioned above)\n\n1) When hypothesizing about heads specializing in syntax (section 4.3), none of the relevant previous work is mentioned. For example,\n[3] - for NMT Transformer, showed that the important heads are specialized, and many of them are syntactic.\n[4] - repeated the previous syntax-heads evaluation for BERT.\n[5] - also mention that some heads track syntax.\n\n2) [6] - looks relevant: it also measures changes in representations across models, layers, for different tokens, etc. Maybe the most relevant to your work is the part showing how different words influence each other (e.g., rare words influence others more than frequent ones; the same analysis for POS). Note also which measure they use - pwcca, which is invariant to linear transformations of representations.\n\n[3] ACL 2019 “Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned”\n\n[4] “Do Attention Heads in BERT Track Syntactic Dependencies?”\n\n[5] BlackBoxNLP 2019 “What Does BERT Look At? An Analysis of BERT's Attention”\n\n[6] EMNLP 2019 “The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives”\n\n------------------------------------------------------------------------------------------------------\nPresentation\n\nThe paper is overall fairly clear, but figures 2d and 3e are not readable. Namely, (even with a maximum zoom) it is very hard to distinguish between solid, dashed and dash-dot lines of the same color.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "K1aY-JyCqjY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the extent to which pre-trained transformer models successfully capture linguistic structure. The approach taken is to present the model with carefully constructed pairs of linguistic probes and then measure the difference in response to a naturally occurring sentence versus one the has been mutated using one of three different strategies. The first is to permute n-grams of a predetermined order. The next is to swap phrases within a parse tree with the results being contrasted with swapping n-grams than don't correspond to phrase boundaries. Finally, the authors explore swapping words that cross varying amounts of syntactic structure.\n\nThe part of the paper that I liked best was the introduction of the distortion metric in section 2.3. This seems like it could be a generally useful means for probing networks both for NLP as well as possible for other domains (e.g., CV). The paper would have benefited from spending more time developing and motivating the metric. The construction looks well thought out, however it would have been good to include at least some ablation experiments to show that z-score normalization and including the perturbation matrix in the formulation is necessary for the types of experiments performed later in the paper.  I imagine the z-score normalization is helpful, but I wonder if the probe would also work without the perturbation matrix. If this is the case, it would allow the metric to use to explore manipulations that go beyond word re-orderings.\n\nThe experiments in the paper are interesting in that they support the case that large pre-trained transformer models do capture linguistic structure. However, this is also a complementary weakness in that the paper doesn't add anything particularly surprising given prior work in this area.\n\nThe paper would be strengthen by introducing additional probes in order to more deeply understand what is an isn't captured by existing pretrained models. I would have also liked to see some treatment of other pretrained models including near BERT variants such as RoBERTa or Albert as well as other models with more distinct architectures and training objectives (e.g., T5 or a GPT model). Alternatively, the paper could have take the approach of developing the best overall probe to detect what is captured by a model. The latter would involve running experiments with different probes and in the best case on different models to discover which method is the most discriminating. \n\nAs a minor nite, the paper attempts to make a connection to neuro-science. This would be better done if there was more of a clear and explicit connection between the techniques explored in the paper and existing neuro-science work beyond just the fact the model is using probes and measuring network activity.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a significant body of seemingly solid work, but its contribution nevertheless feels limited: It evaluates a single MLM on a single dataset, and results are largely unsurprising. Note: The authors added experiments on other LMs in the rebuttal. The idea of using perturbations is related in spirit to many interpretability methods and adversarial techniques, and using higher-order correlations for interpreting neural networks is, for example, at the heart of relational similarity analysis. A few suggestions to make the work more relevant to a wider audience: Compare with several probing techniques - e.g., in a tree-decoding set-up - or contrast results across domains (using OntoNotes), or across languages (using OntoNotes and other PTB-style treebanks). Also: While results were added for multiple LMs, differences were not analysed in detail. 
Decision: Reject