Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper investigates the extent to which pre-trained transformer models successfully capture linguistic structure. The approach taken is to present the model with carefully constructed pairs of linguistic probes and then measure the difference in response to a naturally occurring sentence versus one the has been mutated using one of three different strategies. The first is to permute n-grams of a predetermined order. The next is to swap phrases within a parse tree with the results being contrasted with swapping n-grams than don't correspond to phrase boundaries. Finally, the authors explore swapping words that cross varying amounts of syntactic structure.

The part of the paper that I liked best was the introduction of the distortion metric in section 2.3. This seems like it could be a generally useful means for probing networks both for NLP as well as possible for other domains (e.g., CV). The paper would have benefited from spending more time developing and motivating the metric. The construction looks well thought out, however it would have been good to include at least some ablation experiments to show that z-score normalization and including the perturbation matrix in the formulation is necessary for the types of experiments performed later in the paper.  I imagine the z-score normalization is helpful, but I wonder if the probe would also work without the perturbation matrix. If this is the case, it would allow the metric to use to explore manipulations that go beyond word re-orderings.

The experiments in the paper are interesting in that they support the case that large pre-trained transformer models do capture linguistic structure. However, this is also a complementary weakness in that the paper doesn't add anything particularly surprising given prior work in this area.

The paper would be strengthen by introducing additional probes in order to more deeply understand what is an isn't captured by existing pretrained models. I would have also liked to see some treatment of other pretrained models including near BERT variants such as RoBERTa or Albert as well as other models with more distinct architectures and training objectives (e.g., T5 or a GPT model). Alternatively, the paper could have take the approach of developing the best overall probe to detect what is captured by a model. The latter would involve running experiments with different probes and in the best case on different models to discover which method is the most discriminating. 

As a minor nite, the paper attempts to make a connection to neuro-science. This would be better done if there was more of a clear and explicit connection between the techniques explored in the paper and existing neuro-science work beyond just the fact the model is using probes and measuring network activity.
review_title: Measuring network activations in response to linguistic probes yields expected results

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper investigates the sensitivity of BERT representations to different kinds of permutations in the input sentence. These transformations include n-gram permutation, span swaps (with or without crossing syntactic phrase boundaries), adjacent token swaps (with different syntactic distance). The authors measure the l2 distance between representations coming from original and perturbed input.

Overall, the results suggest that BERT is sensitive to hierarchical phrase structure.

------------------------------------------------------------------------------------------------------
Strengths

The idea of measuring a network’s response to input transformations is nice and potentially could be used to test different kinds of hypotheses.

------------------------------------------------------------------------------------------------------
Weaknesses

My main concern is the measure used for a distance between representations. Namely, this is the l2 distance, which accounts for distance in neurons. Therefore, it does not tell us to what extent representations encode different things but rather how different are their individual neurons. For example, different phenomena can be either focused in a network (encoded by only a few neurons), or distributed (see, e.g., the paper [1] or a more recent [2]). 

Therefore,
1) the phenomena that suffer from perturbations have different impact on the l2 distance because they affect different number of neurons
2) as a consequence of the above, it is not clear what to conclude from the proposed results: they are likely to be not because of the high-level differences in what original/perturbed representations encode, but rather in
how these underlying phenomena affect individual neurons. 

(To measure differences in what representations encode you can use, for example, PWCCA (NeurIPS 2019 “Insights on representational similarity in neural networks with canonical correlation”) or other related measures.)

[1] AAAI 2019 “What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models”

[2] EMNLP 2020 “Analyzing Individual Neurons in Pre-trained Language Models”

Other concerns.
If we put aside the validity of the measure, it is still not clear what to take out of these results: they rather show that the method passes sanity checks rather than tell us something we didn’t know before. For example, there’s a huge line of works showing that BERT “understands” syntax and phrase composition (looking at representations, geometry, attention heads, etc). Hence the results stated in contributions 2, 3, 4 are not surprising. Contribution 1 is also more of a sanity check: of course, shuffling smaller parts has to cause more distortion than shuffling longer phrases. 

------------------------------------------------------------------------------------------------------
Questions

Section 4.1, paragraph 1: when referring to figure 2b, you say “When we shuffle in units of larger n-grams, it only introduces distortions in the deeper BERT layers compared to smaller n-gram shuffles.”
I have trouble seeing this from the figure. For all lines, the distortion goes up almost linearly from layer 2 to layers 9-10. Yes, shuffling larger ngrams causes lower distortion, but this is expected. Am I missing something?

------------------------------------------------------------------------------------------------------
Missing references (in addition to mentioned above)

1) When hypothesizing about heads specializing in syntax (section 4.3), none of the relevant previous work is mentioned. For example,
[3] - for NMT Transformer, showed that the important heads are specialized, and many of them are syntactic.
[4] - repeated the previous syntax-heads evaluation for BERT.
[5] - also mention that some heads track syntax.

2) [6] - looks relevant: it also measures changes in representations across models, layers, for different tokens, etc. Maybe the most relevant to your work is the part showing how different words influence each other (e.g., rare words influence others more than frequent ones; the same analysis for POS). Note also which measure they use - pwcca, which is invariant to linear transformations of representations.

[3] ACL 2019 “Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned”

[4] “Do Attention Heads in BERT Track Syntactic Dependencies?”

[5] BlackBoxNLP 2019 “What Does BERT Look At? An Analysis of BERT's Attention”

[6] EMNLP 2019 “The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives”

------------------------------------------------------------------------------------------------------
Presentation

The paper is overall fairly clear, but figures 2d and 3e are not readable. Namely, (even with a maximum zoom) it is very hard to distinguish between solid, dashed and dash-dot lines of the same color.

review_title: Overall nice, but (i) the distortion measure is questionable and (ii) not clear what to take out of the results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper analyzes the ability of BERT model to learn good representations of sentences, through a purely empirical study, there is no other contribution of a new model or a new analysis technique. In the experimental analysis, phrases or words are swapped and the corresponding changes in the sentence representations are analyzed from all the hidden layers. I didn't see any surprising outcomes from the analysis, so not sure how impactful this work is. I would rather see this kind of work in a workshop track for ICLR.

A minor comment: only one dataset is considered in analysis. I would like to see the analysis sentences from the training set as well as many test sets.

review_title: Empirical analysis of BERT through swapping words or phrases, no surprising insights 

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary:
This paper addresses how pre-trained Transformer models build their contextual representations along with the layers by measuring changes in the outputs on a series of probes. Specifically, a probe involves swapping words in a sentence and measuring the distortion in the representations. The series of probes are designed to test how the models respond to different syntactic-related swaps including n-gram size and syntactic phrase boundary and distance. The results mainly focus on BERT (base-cased), but some results of the other variants have the same trend. The author first confirms that the observed distortions are the learned behavior by showing a "flat" distortion trend of untrained models. When subjecting to n-gram swaps, the results show that smaller n-grams have a larger distortion. We can also see that later layers are affected more than earlier layers (for all kinds of swaps). Regarding hierarchical phrase structure, the results show that the phrasal boundaries are important, even accounted for PMI of swapped words. The rank correlations between distortion and the syntactic distance are higher in the deeper layers across all attention heads, but the correlations are somewhat weak (around 0.3). In addition, distortions have a larger impact on a relatively more complex task (parse tree distance reconstruction > POS tagging).  Finally, an analysis of the attention shows that the attention layers contribute to the observed distortion. The main conclusion is that BERT (and its variants) builds its contextual representation by increasingly incorporating more phrasal units along with the layers. 

### Recommendation:
Overall, I would like to recommend this paper for the conference. The paper presents a novel approach to study the behavior of Transformer models and the findings are quite interesting. My main concerns are that some of the implications of the results are quite hard to follow, and I am not sure how they might impact future downstream researches.

#### Pros:
- I would like to commend the author for clear writing. The paper was very well written and I enjoyed reading the paper.
- The main goal of the paper is to understand (or at least shade some light on) how BERT works. I think it is equally important as the interpretability of the models (i.e., explain the prediction).
- The perturbation-based probes are easily related to language features than other methods such as attention visualization or a salience-based approach. Although there exists work that uses the same approach, I think this paper presents novels finding on how the representations of the pre-trained models "distort" given different perturbations.
- The paper provides comprehensive experiments including many carefully designed experiments to reveal the key insights.


#### Cons:

- Some of the key findings, such as "long-range contexts play a larger role deeper layer representations" are very hard to follow. 
- The differences between the distortions among tree distance and the rank correlations are somewhat weak, or rather we do not have a baseline to compare against. I am not sure how strong finding 3 (in the introduction) is supported by the results. Perhaps, to improve the experiment further, the author could include the results of different model architectures (RNN-based models) to compare and contrast.
- I think the experiment on the attention weights is not helpful. Since attention layers are the main mechanism in the Transformer models, it is not surprising that it plays an important role. In the end, it is still inconclusive how attention captures the contextual information. The author could analyze the changes or distortion of attention weights.

### Questions:
1. It is very important how the author handles subwords. The explanation should not be a footnote and there should be more detail on when the "average" happens (i.e., from input to BERT or only distortion computation).
1. I can see why the author would like to aggregate subwords into words when probing. How much do the averaging embeddings change the distortion? Does it affect the findings? Wouldn't it be more helpful to include a probe regarding subwords?
1. What is the corpus that PMI is computed on?
1. What are the base likelihoods (without the perturbation)? Could we imply that the average decrease in log-likelihood is explained by the base likelihoods?

### Comments:
- A.4 is not quite related to the attention covariate, though the intention is understandable.
- Fig. 5e does not exist. I think you mean Fig. 5d (which is quite hard to interpret).
- I think a more careful analysis of the distortion vectors (Fig 1c) should be done (i.e., why is it desirable that the models should be robust to some perturbations? And what is the trend of the distortions across different types of perturbations?). 
review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: --- Overall ---

This paper provides some insights into the relation of BERT's output w.r.t. the parser tree (in terms of constituent phrases) of the input sentence. As some previous work has pointed out, BERT model contains the parsing information (e.g., Hewitt & Manning NAACL’19)). This work can be regarded as a moderate reification and improvement of that thought (but it is still limited to existing scopes and methodologies). 

The merits of this paper: (1) the paper reveals some interesting facts such as BERT are sensitive to phrasal hierarchy and there are behavioural discrepancies between different layer; (2) the experiments are comprehensive, including both the distortion analysis and conventional probe approaches.

In my point of view, the main issue of this paper is, like many other works, there is no strong theoretical explanation for the phenomenon being investigated. In this sense, the novelty of this paper is not so strong.

--- Major comments ---

1.	In experiments, it is not clear whether the randomness of BERT itself has been deducted. The randomness could be caused by the dropout operation which may lead to the discrepancy on output even using the same sentence.
2.	In the future version, I recommend to further provide two tests: (1) in current settings, the distortions are showed in sentence-level (e.g., by summing up all distortions within a sentence?). I would like to see a finer-grained test, i.e., whether the most distortion parts are produced by the swapping part; (2) The genre of the Penn treebank dataset is limited to general texts or articles that may be seen in the training corpus of BERT. I would recommend testing on other domains (e.g., biomedical or academic) that BERT never saw before (or structure-less data that do not present a syntactic structure).
3.	Do X~ and X in section 2.3 use the same mean and deviation?
4.	It seems that Fig. 2(c) takes account into both NP and VP, what if we only constrain the phrases to be only NP? Will the distortion becomes large since swapping subject and object will lead to totally different meanings?
5.	There is a lack of explanation about "all words", "swapped", and "unswapped" settings of Fig. 2(d).
6.	Is there any intuition for the step back on the layers 11 and 12 in Fig. 4(b) and 4(c)?

--- Minor comments ---

1.	Strictly speaking, the terminology “gram” in Fig.2(a) should be called "chunk", as “grams” are usually related with sliding window thus overlap with each other.
2.	“Fig.3(a)” in the paragraph just above Section 4.2 should be “Fig.2(c)”.

review_title: Reveal the correlations of hierarchical phrase structure and BERT's output, but lack theoretical explanation.

Meta-review: The paper presents a significant body of seemingly solid work, but its contribution nevertheless feels limited: It evaluates a single MLM on a single dataset, and results are largely unsurprising. Note: The authors added experiments on other LMs in the rebuttal. The idea of using perturbations is related in spirit to many interpretability methods and adversarial techniques, and using higher-order correlations for interpreting neural networks is, for example, at the heart of relational similarity analysis. A few suggestions to make the work more relevant to a wider audience: Compare with several probing techniques - e.g., in a tree-decoding set-up - or contrast results across domains (using OntoNotes), or across languages (using OntoNotes and other PTB-style treebanks). Also: While results were added for multiple LMs, differences were not analysed in detail. 
Decision: Reject