Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number
Authors: Sophie Hao
Tal Linzen
Venue: EMNLP 2023 Findings (2023)
Keywords: interpretability
analysis
representations
hidden vectors
syntax
subject-verb agreement
transformers
pre-trained models
language models
bert
causal analysis
causality
causal intervention
inlp
TL;DR: We show through causal intervention that Transformer language models conjugate verbs using an interpretable linear representation of subject number in hidden vectors.

Abstract: Deep architectures such as Transformers are sometimes criticized for having uninterpretable "black-box" representations. We use causal intervention analysis to show that, in fact, some linguistic features are represented in a linear, interpretable format. Specifically, we show that BERT's ability to conjugate verbs relies on a linear encoding of subject number that can be manipulated with predictable effects on conjugation accuracy. This encoding is found in the subject position at the first layer and the verb position at the last layer, but distributed across positions at middle layers, particularly when there are multiple cues to subject number.

Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number
Sophie Hao
Tal Linzen
New York University sophie. hww.nyu.oeeecoooo edu
New York University 1 linzenênyu. edu
Abstract
predict that intervening on the hidden space by reflecting hidden vectors to the opposite side of the number subspace will cause BERT to gener-
Deep architectures such Transformers sometimes criticized for having uninterpretable "black-box" representations. We use causal in- tervention analysis to show that, in fact, some linguistic features are represented a linear, interpretable format. Specifically, BERT's ability to conjugate verbs relies on linear encoding subject number that can be manipulated with predictable effects con- jugation accuracy. This encoding is found in the subject position at the first layer and the verb position at the last layer, but is distributed across positions at middle layers, particularly when there are multiple to subject number.
ate plural conjugations for singular subjects, and vice versa. Our experiment confirms this predic- tion dramatically: BERT's s verb conjugations are 91 % correct before the intervention, and up to 85% incorrect after the intervention.
show that
In addition to these findings, our experiment makes observations regarding the location of sub- ject number encodings across token positions, and how it changes throughout BERT's forward com- putation. We find that subject number encodings originate in the position of the subject at the embed- ding layer, and move to the position of the inflected verb at the final layer. When the sentence contains additional cues to subject number beyond the sub- ject itself, such an embedded verb that agrees with the subject, subject number encodings propa- gate to other positions the input at middle layers. Unlike our study, prior counterfactual interven- tion studies have not been able to consistently pro- duce the expected changes in LM behavior. In Fin- layson et al. (2021) and Ravfogel et al. (2021), for example, interventions only cause slight degrada- tions in performance, leaving LM behavior mostly unchanged. These numerically weaker results show that LM behavior is influnced by linear feature encodings, but is ultimately driven by other repre- sentations, which may have a non-linear structure. In contrast, our results show that the linear encod- ing of subject number determines BERT's ability to conjugate verbs. The mechanism behind verb conjugation is therefore linear and interpretable, far from being a black box."
1 Introduction
Although neural network language models (LMs) are sometimes viewed uninterpretable "black boxes," substantial progress has been made towards understanding which linguistic regularities LMs are sensitive and how they represent those regulari- ties, in particular in the case of syntactic constraints such as subject-verb agreement. This progress in- cludes not only the discovery that LM predictions adhere to such constraints (c.g., Linzen et al.. 2016), but also the development of tools that have revealed encodings of syntactic features in hidden represen- tations (Adi et al., 2017; Giulianelli et al., 2018, among others).
Most prior work on LMs' internal vector repre- sentations has demonstrated the existence of syn- tactic information in those vectors, but has not de- scribed how LMs use this information. This paper addresses the latter question using causal interven- tion paradigm proposed by Ravfogel et al. (2021), We first hypothesize that at least one hidden layer of BERT (Devlin et al., 2019) encodes the gram- matical number of third-person subjects and verbs in a low-dimensional number subspace the hid- den representation space, where singular number is linearly separable from plural number. We then
Background and Related Work
This study contributes to a rich literature on the representation of natural language syntax in LMs.
'Code for our experiment available at: https:// gi thub. hamm/yyidnghaoccusasll -conua--oonnaaation

---

We briefly review this literature in this section; more comprchensive overview is offered by Lasri et al. (2022).
Counterfactual Intervention: Let X1, Au,..., Au be the coordimates f.00 along the number subspace. mod- ify h as shown below. If a 2 2, then the modified vec- lor should encode the opposite subject number. If 1, then the modified vector should contain information about subject number.
LMs and Syntax. A popular approach the study of syntax in LMs is through the use of behav- ioral experiments. An influential example Linzen et al. (2016), who evaluate English LSTM LMS on their ability to conjugate third-person present-tense verbs. Since verb conjugation depends on syntac- tic structure in theory, this study can be viewed as an indirect evaluation of the LM's knowledge of natural language syntax. Linzen et al. methodol- ogy for evaluating verb conjugation probability scores assigned different verb forms, testing whether an LM is more likely to generate correctly conjugated verbs than incorrectly conju- gated verbs. Follow-up studies such as Marvin and Linzen (2018), Warstadt et al. (2019), and Gauthier et al. (2020) have refined the behavioral approach by designing challenge benchmarks with experi- mental controls the structure of example texts, which allow for fine-gras trae-ggaind evaluations of specific linguistic abilities.
author
(1.)
novel
teachers
movies'
h = 0.0
Xyb
compare
Verb Conjugation: We predict that intervention with a: 2 2 will cause BERT to conjugate verbs incorrectly.
Before Intervention: P Piss] >> IP IP[arc]
After Intervention: Pis) < P(are]
BERT
The author that the teachers admire [MASK] happy.
Figure Illustration of our counterfactual intervention (above) and our verb conjugation test (below).
Probing and LM Representations. Another ap- proach to syntax in LMs is the use of probing clas- Belinkov al., 2017: Hup- kes and Zuidema, 2017: Hupkes et al., 2018). By contrast with behavioral studics, probing studies analyze what information is encoded in LM repre- sentations. A typical analysis attempts to train the probing classifier to decode the value of a syntactic feature from hidden vectors generated by an LMM If this is successful, then the study concludes that the hidden space contains an encoding of the rele- vant information about the syntactic feature. When the probing classifier is linear, the study can ad- ditionally conclude that the encoding has a linear structure. An overview of probing results for BERT is provided by Rogers et al. (2020).
conjugation and other linguistic abilities. The goal of this paper is to present an instance where lin- ear encodings fully determine the verb conjugation behavior of an LM.
sifiers (Adi al., 2017;
3 Methodology
Let hl1,)) E 76 be the hidden vector from layer I of BERT for position i. Our hypothesis is that there is an orthonormal basis B = (b(*), ((2), (768)) such that for some & < 768, the first h basis vectors span a number subspace that lin- early separates hidden vectors for singular-subject sentences from hidden vectors for plural-subject sentences. Our prediction that the counterfactual intervention illustrated Figure where hidden vectors are reflected to the opposite side of the number subspace, will reverse the subject number encoded in the vectors when applied with sufficient intensity (as determined by the hyperparameter oi), causing BERT to conjugate the main verb of sen- tence if its subject had the opposite number. This section describes (1) how our counterfactual inter- vention is defined, (2) how we find the basis vectors for the number subspace, and (3) how we measure the effect of this intervention on verb conjugation.
Counterfactual in- tervention enhances the results probing study by determining whether a feature encoding discov- ered by a linear probe is actually used by the LMM, or whether the probe has detected spurious pattern that does not impact model behavior. Early stud- ies such as Giulianelli et al. (2018), Lakretz al. (2019), Tucker et al. (2021), Tucker al. (2022), and Ravfogel al. (2021) provide evidence that manually manipulating representations of subject number can result causal effects on LM verb
Counterfactual Intervention.

---

Counterfactual Intervention. Suppose that the hidden vector ((.i) is computed from an input con- sisting of a single sentence. The goal of our coun- terfactual intervention is to transform (1:3) into a vector h that BERT will interpret as a hidden vec- tor representing the same sentence, but with the opposite subject number. To do so, we first assume that h (1.)) is written in terms of the basis IB:
Measuring the Effect of Intervention. We eval- uate BERT's verb conjugation abilities using a paradigm based on Goldberg (2019), where masked language modeling is performed on sentences with a third-person subject where the main verb, is or
are, is masked out. We calculate conjugation ac- curucy by interpreting BERT's output as binary classification, where the predicted label is "singu- lar" if IP[is] > IP |P|are] and "plural" otherwise. To test our prediction about the causal effect of number encoding on verb conjugation, we measure conju- gation accuracy before and after intervention with a 2 2. If intervention causes conjugation accuracy to drop from s 1000% to =0%, then we conclude that we have successfully encoded the opposite sub- ject number into the hidden vectors. If conjugation accuracy drops to =50%, then number information has been erased, but we cannot conclude that the opposite subject number has been encoded.
768
(1.3) Σ 2,b6)
where for each j. the coordinate Xj is the scalar projection of h (1.1) onto the unit vector b0);
> (n (1))))) bG)
Next, we assume that the coordinates of h (1.4) along the number subspace, λ1, A2...., AL. collectively encode the input sentence's subject number, and that -A1; -2....... dE encode the opposite sub- ject number. We compute h by simply moving these coordinates of h (L.i) towards the opposite sub- ject number:
Experiment
We test our prediction by performing an experiment using the bert-base-uncased instance of BERT. For each layer, we apply counterfactual interven- tion and measure its effect on conjugation accuracy. We perform two versions of our experiment: one where intervention is applied to all hidden vectors ("global intervention"), and one where intervention is only applied to hidden vectors in the subject po- sition ("local intervention"). We repeat our experi- ment five times, with each trial using linear probes trained on a freshly sampled, balanced dataset 4,000 hidden vectors.
h h-
The variable a is a hyperparameter that determines the intensity of the counterfactual intervention. When = 1, the coordinates along the number subspace set to 0; h is then interpreted as vector that encodes no information about subject number. If our hypothesis about the number sub- space is correct, then counterfactual intervention with a 2 2 should result in a vector that encodes the opposite subject number.
Data. We use data from Ravfogel et al. (2021), which consist of sentences with relative clause intervening between the main subject and the main verb (e.g., The author that the teacher admires is happy). We sample the INLP training vectors from their training split, and we use their testing split to measure conjugation accuracy.
Finding the Number Subspace. We use the iter- alive nullspace projection (INLP, Ravfogel et al., 2020; Dufter and Schütze, 2019) method to calcu- late the basis for the number subspace. We begin by using BERT to encode collection of sentences and extracting the hidden vectors (f.i) the positions of main subjects. We then train a linear probe to de- tect whether these hidden vectors came from sin- gular subject or plural subject, and take (1) to be the probe's 's weight vector, normalized to unit length. To obtain 60) for > 1, we use the same proce- dure, but preprocess the data by applying counter- factual intervention with 1 and = j- This erases the subject number information captured by previously calculated basis vectors, ensuring that b(7) is orthogonal to (()) (22) ((2)...., b(i-1)
We tune the hyperparameters a (intensity of intervention) and & (dimensionality of the number subspace) using grid search over the range E (2,3,5) 5) and k {2, 8)..
Hyperparameters.
Main Results. Figure shows our results. The values of a and k do not affect our results quali- tatively, but they do exhibit direct relationship with the magnitude of the effect of intervention on conjugation accuracy. We focus on the results for

---

Main Experiment Results
INLP Applied Subject
INLP Applied to Verb
alpha 5
100
Accuracy
Accuracy
50
50
3
6
12
3
6
9
12
0
2
11
Layer Modified
Layer Modified
Layer Modified
Local
Global Chance
Global
No Intervantion
No Intervention
Chance
Figure 4: Conjugation accuracy drops to 7.6% when intervening layer 12 with INLP training vectors tracted from the verb position (right) instead of the sub- ject position (left). Error bands represent 95% con- fidence intervals obtained from 5 samplings of INLP training vectors.
Figure 2: The effect of local and global intervention on conjugation accuracy. Error bands represent 95% confidence intervals obtained from samplings of INLP training vectors.
Redundant Cues:
The author that admires teachers happy.
Redundant Cues to Number. Some sentences our training and testing data contain an embedded verb that agrees with the main subject. For example, in the sentence The author that admires the teacher is happy, the singular verb admires agrees with the subject author. Since can deduce the number of the subject from the number of this embedded verb, even in the absence of any direct access representation of the subject, in these sentences the embedded verb serves as a redundant cue to subject number.
No Redundant Cues:
The author that the teachers admire happy.
Redundant Cues
Redundant
100
Accuracy
50
0
0
12
6
Layer Modified
Layer Modified
Local
Global
Subj. Verb
No Intervention
Chence
The linear encoding of subject number spreads to positions other than the subject when there are redun- dant to subject number, such embedded verb. In the "subj. + verb" condition, intervention is applied to the subject and embedded verb positions. Error bands represent 95% confidence intervals obtained from 5 sam- plings of INLP training vectors.
Figure 3:
Figure 3 shows the effects of intervention broken down by the presence of redundancy. When there is no redundancy, near-zero conjugation curacy observed after both local and global in- tervention applied to layers 0-6. This shows that when the subject is the only word that conveys subject number, verb conjugation depends solely on the hidden vector in the subject position. By contrast, local intervention has no effect conju- gation accuracy in the presence of redundant cues, and neither does intervention in the positions of the subject and the embedded verb (the "subj. verb" condition). This shows that the presence of a redundant cue to subject number causes BERT to distribute the encoding of subject number to multi- ple positions.
a - 5 and & = 8, which exhibit the greatest im- pact of intervention on conjugation accuracy. The full hyperparameter tuning results can be found in Appendix A.
Our prediction is confirmed when global inter- vention, where hidden vectors across all positions are modified, is applied to layer 8. Verb conjuga- tions are 91.7% correct before intervention, but 84.6% incorrect after intervention. Local interven- tion on layer 8, where only the hidden vector in the subject position is modified, has much weaker effect, only causing conjugation accuracy to drop to 57.5% (42.5% incorrect). These results show that BERT indeed uses linear encoding of subject number to comply with subject-verb agreement. The location of this linear encoding is not con- fined to the position the subject, but is rather distributed across multiple positions.
Upper Layers. In layers 10-12, neither local nor global intervention has any effect on conjugation accuracy. We hypothesize that this is because, at these layers, the INLP linear probe cannot iden- tify the number subspace using training vectors extracted from the subject position of sentences.
To test this hypothesis, we extract INLP training vectors from the position of the main verb instead of the subject as before, and apply local interven-

---

MuhtiBRTs Results
Side Effects
BERT Layers. Probing studies have found that lower layers of BERT encode lexical features, while middle layers encode high-level syntactic features and upper layers encode task-specific fea- tures (Hewitt and Manning, 2019; Jawahar et al., 2019; Kovaleva et al., 2019; Liu et al., 2019; Ten- ney et al., 2019). Our results confirm this in the case of cue redundancy: at layer 8, the representa- tion of subject number is not tied any position; while layer 12, it is tied to the [MASK] posi- tion, where it is most relevant for masked language modeling. When there is no cue redundancy, how- ever, subject number is tied to the subject position until layer 9, suggesting that subject number is treated lexical feature of the subject rather than sentence-level syntactic feature.
100
10
Accuracy
Perplextyy
10
50
10"
0
3
9
12
Layer Modified
Layer er Modified
Subspaca
Local
Random Subspacee
Global
Intervention
Intervention
Chance
Figure 5: Left: Similar results are obtained when re- peating the experiment using the MultiBERT models. Right: Intervention adverse effect on perplexity. Error bands represent 95% confidence intervals obtained from samplings of INLP training vectors.
number-neutral words has no
tion to the position of the masked-out main verb. Supporting our hypothesis, both local and global intervention result in near-zero conjugation accu- racy (Figure 4, right), showing that at upper layers. only the position of the main verb is used by BERT for conjugation.
Effect Size. Prior counterfactual intervention studies only report marginal changes in perfor- mance after intervention (e.g., Kim et al., 2018; Dalvi et al., 2019; Lakretz et al., 2019; Finlayson et al.. 2021; Ravfogel et al., 2021). For example, the largest effect size reported by Ravfogel et al. (2021) is no more than 35 percentage points. These results suggest that the linear encoding is only a relatively small part of the model's 's representation of the feature. Our results improve upon prior work by identifying an aspect of LM behavior that is entirely driven by linear feature encodings.
Robustness. To verify that our results are robust to differences in model instance, we repeat our experiment using the MulúBERTSs (Sellam et al., 2022), a collection of 25 BERT models pre- trained from different random initializations. As shown in the left side of Figure 5, we obtain similar results to Figure 2, indicating that our findings are not specific to pert-base-uncased
Conclusion
Using causal intervention analysis. this paper has revealed strong evidence that BERT hidden representations contain lincar encoding of main subject number that is used for verb conjugation during masked language modeling. This encoding originates from the word embeddings of the main subject and possible redundant cues, propagates to other positions at the middle layers, and migrates to the position of the masked-ou verb at the upper lay- ers. The structure of this encoding is interpretable, such that manipulating hidden vectors along this en- coding results in predictable effects on conjugation accuracy.
Side Effects. Does the number subspace encode information beyond number? To test this, we apply intervention to number-neutral words (i.e., all words other than nouns and verbs) along the number subspace. We find that this has no ef- fect on masked language modeling perplexity for those words (Figure 5). contrast, interven- tion on number-neutral words along a random 8- dimensional representation subspace increases per- plexity by a factor of 52.8 on average. This shows that the number space selectively encodes number, such that manipulating hidden vectors along the number subspace does not affect predictions unre- lated number.
Our clean and interpretable results offer subject number an example of feature that large language model might encode using a straight- forwardly linear-structured representation scheme. For future work, we pose the question of what kinds of features may admit similarly strong results from a causal intervention study like this one.
5 Discussion
this section, we discuss our results in relation to our current knowledge about línear representations.

---

Limitations
In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 861-872, Vancouver, Canada. Association for Computational Linguisties.
Below we identify possible limitations of our ap- proach.
Experimental Control. By utilizing Ravfogel et al.'s 's (2021) dataset, where sentences adhere to a uniform syntactic template, we have exerted tight experimental control over the structure of our test examples. This control has allowed us, for instance. to identify the qualitatively distinct results from Figure 3 between inputs with and without redun- dant cue to subject number. In more naturalistic setting, is possible that verb conjugation may be conditioned by factors other than linear encoding of subject number, such as semantic collocations or discourse context.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Be- linkov, Anthony Bau, and James Glass. 2019. What Is One Grain of Sand in the Desert? Analyzing Indi- vidual Neurons in Deep NLP Models. Proceedings of the AAAI Conference on Artificial Intelligence, (3(01))6339-6311
Jacob Devlin. Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, volume (Long and Short Papers), pages 4171-411 Minneapolis, MN, USA. Association for Computational Linguistics.
Asymmetry of Findings. Although have shown that BERT uses a linear encoding of subject number to conjugate verbs, we can never prove us- ing our approach that BERT does not use a linear encoding of a feature to some end. the instances where we are unable to encode the opposite sub- ject number, we cannot rule out the possibility that BERT uses linear encoding of subject number that cannot be detected using INLP.
Philipp Dufter and Hinrich Schütze. 2019. Analytical Methods for Interpretable Ultradense Word Emhed- dings. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNL) pages 1185-119 1191, Hong Kong, China. Association for Com- putational Linguistics.
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1828-1843, Online. Association for Computational Linguistics.
Ethical Considerations
We do not foresee any ethical concerns arising from our work.
Acknowledgments
We thank the EMNLP 2023 reviewers and area chairs well as members of the New York Uni- versity (NYU) Computation and Psycholinguistics Lab for their feedback on this paper.
Jon Gauthier, Jeunifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An Online Plat- form for Targeted Evaluation of Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations pages 70-76, Online. Association for Computational Linguisties.
This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise. This material is based upon work supported by the National Science Foun- dation (NSF) under Grant No. BCS-21 14505.
Mario Giulianelli. Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Un- der the Hood: Using Diagnostic Classifiers In- vestigate and Improve how Language Models Track Agreement Information. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and In- terpreting Neural Networks for NLP. pages 240-248. Brussels, Belgium. Association for Computational Linguisties.
References
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi,
and Yoav Goldberg, 2017. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In ICLR 2017 Conference Track, Toulon, France. arXiv.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What Neural Ma- chine Translation Models Learn about Morphology?
Yoav Goldberg. 2019. Assessing BERT's Syntac- Abilities. Computing Research Repository. arXiv:1901.05288

---

John Hewitt and Christopher D. Manning. 2019,
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4(0):521- 535.
Structural Probe for Finding Syntax in Word Repre- sentations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4129-4138. Minneapolis, Minnesota. Association for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov,
Matthew E. Peters, and Noah A. Smith. 2019. Lin- guistic Knowledge and Transferability of Contex- tual Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1073 10944, Minneapolis, MN, USA. Association for Computational Linguistics.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and 'Diagnostic Classifiers' Re- veal How Recurrent and Recursive Neural Networks Process Hierarchical Structure. Journal of Artificial Intelligence Research, 61:907-926.
Dieuwke Hupkes and Willem Zuidema. 2017. Diag- nostic classification and symbolic guidance to un- derstand and improve recurrent neural networks. In Interpreting, Explaining and Visualizing Deep Learn- ing ... now What? (NIPS 2017 Workshop), Long Beach, CA, USA.
Rebecca Marvin and Tal Linzen. 2018. Targeted Syn- tactic Evaluation of Language Models. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What Does BERT Learn about the Structure of Language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy, Association for Computational Linguistics.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computa- tional I. Linguistics.
Been Kim, Martin Wattenberg, Justin Gilmer, Car- rie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability Beyond Feature Attri- bution: Quantitative Testing with Concept Activation Vectors (TCAV), In International Conference on Machine Learning, 10-15 July 2018, Stockholmsmüs- san, Stockholm Sweden, volume 80 of Proceedings of Machine Learning Research, pages 2668-2677, Stockholm, Sweden. PMLR.
Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual Interventions Re- veal the Causal Effect of Relative Clause Represen- tations on Agreement Prediction. In Proceedings of the 25th Conference Computational Natural Lan- guage Learning, pages 194 209, Online. Association for Computational Linguistics.
Olga Kovaleva, Alcxey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the Dark Secrets of BERT. In Proceedingy of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP),, pages 4365-4374, Hong Kong, China. Association for Com- putational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A Primer in BERTology: What We Know About How BERT Works. Transactions of the Asso- ciation for Computational Linguistics, 8:842-866.
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' Amour, Tal Linzen, Jasmijn Bastings, Tulia Raluca Ture, Jacob Fisenstein, Dipanjan Das, and Ellie Pavlick. 2022. The MultiB- ERTs: BERT Reproductions for Robustness Analysis. In ICLR 2022 Conference Track, Online.
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Ba- roni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, volume (Long and Short Papers), pages 1--00, Minneapolis, MN. Asso- ciation for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das. and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In ICLR 2019 Conference Track, New Orleans, LA, USA. arXiv.
Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the Usage of Grammatical Number. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 8818-88 88881, Dublin, Ireland. Association for Computational Linguistics.
Mycal Tucker, Tiwalayo Eisape. Peng Qian, Roger Levy, and Julie Shah. 2022. When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human

---

Language Technologies, pages 5393-5408, Seattle, WA, USA. Association for Computational Linguis- tics.
Mycal Tucker, Peng Qian, and Roger Levy. 2021. What This Modified That? Syntactic Interventions with Counterfactual Embeddings. In Findings of the Asso- ciation for Computational Linguistics: ACI-IICNI.P 2021. pages 862 875, Online. Association for Com- putational Linguistics.
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Ha- gen Blix, Yining Nic, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang. Jason Phang, Anhad Mohananey, Phu Mon Huut, Paloma Jeretic, and Samuel R. Bowman. 2019. Investigat- ing BERT's Knowledge of Language: Five Anal- ysis Methods with NPIs. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 2877-2887, Hong Kong, China. Association for Computational Linguistics.
Hyperparameter Tuning Results
Our full hyperparameter tuning results are shown in Figure 6.

---

alpha 2 1k=2 2
alpha 2 2|k=4
alpha 2| 2 8
100
Accuracy
alpha k
alpha
alpha
Accuracy
50
alpha 5 2
alpha = 5 k
alpha = k=
100
Accuracy
50
2
3
10 11 12
7 8
9
10
11
3 4
6
12
Layer Modified
Layer Modified
Layer Modified
Local
Global
No Intervention
Chance
Figure 6: Hyperparameter tuning results for a (intensity of counterfactual intervention) & (dimensionality of the number subspace). Error bands represent 95% confidence intervals obtained from samplings of INLP training vectors.


Reviews:
[{"review_id": "ntqrMrBHyb", "reviewer": "Reviewer_cZfP", "summary": "", "questions": "Question A: Will subjects happen to have sub-tokens? If so, how do you deal with it when extracting corresponding hidden vectors?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We would like to discuss the novelty of our paper, and how our results relate to previous ones.\n\nThe novelty of our paper is that our conclusions are much more justified than in previous papers. While prior studies have found _some_ evidence in favor of linearly encoded syntactic features, our paper is the first to provide _conclusive_ evidence for it. \n\nIn Ravfogel et al. (2021), for example, causal intervention only had a slight impact on the model’s behavior. While this does show that their target feature (membership in a relative clause) _has_ a linear encoding, the weakness of their effect size also shows that this linear encoding is not particularly important for the model. On the other hand, our experiments show that linear encodings _determine_ model behavior in the case of verb conjugation: manipulating the linear encoding predictably alters model behavior almost 100% of the time. \n\nThis finding is novel, and we argue that it is in fact surprising. In past work we have only seen linear encodings that _influence_ model behavior; our paper is the first to present a linear encoding that _determines_ model behavior. We believe it is quite surprising that the linear encoding of subject number can override all other variables that might drive model behavior in the case of verb conjugation.\n\nIn addition to this finding, we also expand upon previous work by pinpointing the location of our linear encodings, both across layers of the Transformer and across positions of the input. This kind of analysis has not been previously done in causal intervention studies, and we believe our results do provide a novel understanding of Transformer representations. \n\nAs for Question A: the training data has exactly one subject that spans two tokens, and the testing data only has single-token subjects. In the case of this one subject, we treat both tokens as “subjects,” in the sense that hidden representations for both positions are included as possible SVM training vectors. Because this particular subject is rare, and because only 2,000 randomly sampled vectors are included in the SVM training data, we do not believe that this substantially affects our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and clearly structured.\n\nThis paper has conducted some interesting experiments to analyse different aspects, such as the robustness, side effects. ", "reasons_to_reject": "The study is focused on one type of model and one type of linguistic feature. The title of the paper would be better supported with experiments in more settings (e.g., different transformer-based models).\n\nGiven previous work like Finlayson et al. (2021) and Ravfogel et al. (2021), I do not think this paper is novel enough or provides novel enough conclusions to publish at EMNLP. The methods are not new and the results are not surprising. ", "questions_to_authors": "Question A: Will subjects happen to have sub-tokens? If so, how do you deal with it when extracting corresponding hidden vectors?\n", "paper_topic_and_main_contributions": "This paper investigates whether the subject number is linearly encoded in BERT's hidden states, and shows that the representation can be manipulated to encode the opposite number information with linear transformations. A great number of experiments are conducted to analyze the effect of different clues, the robustness and side effects. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "jZc9sOBAuB", "reviewer": "Reviewer_SDLf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\nWe do in fact believe that our paper presents new knowledge about Transformers—namely, as you describe, that some high-level syntactic features are encoded _almost exclusively linearly_ by Transformers, particularly in the middle layers. While we agree that the existence of linear encodings in general is not a new finding, our paper does present novel findings on:\n- which features have a linear encoding, \n- where (across layers and positions) this linear encoding is found, and \n- how important this linear encoding is compared to other drivers of model behavior. \n\nMost importantly, whereas previous papers have shown that linear encodings _influence_ model behavior, our paper is the first to present a syntactic feature whose linear encoding _determines_ model behavior. Crucially, the latter conclusion is only possible if the effect size of intervention exceeds 50 percentage points and approaches 100 percentage points, which is what our paper shows. Therefore, the contribution of our paper is not merely that we obtain better numerical results; it's that these improved results allow us to reach a conclusion that previous papers have not been able to reach.\n\nIn order to make these arguments more clear, we are happy to change the title to: “Verb Conjugation in Transformers is Determined by Linear Encodings of Subject Number.”\n\nWe are happy to incorporate a discussion of linear vs. non-linear encodings, as well as the distinctions between diagnostic, behavioral, and causal probing (in the terminology of Lasri et al., 2023, linked in the review). As for the calculation of $\\lambda$s, they are simply the scalar projections of the hidden vectors onto the (unit-length) basis vectors, given by the following formula: \n$$\\lambda_j = \\left( \\boldsymbol{h}^{(l, i)} \\right)^\\top \\boldsymbol{b}^{(j)}$$\nThis is a standard formula in linear algebra, and we are happy to explain this explicitly in the final version if this paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a clear link between a well-defined manipulation of the vector space of token embeddings and an important linguistic property. It focuses on a single language and model type, but inside these constraints it does hyperparameter checking and repeats the experiment with different models from the same class (BERT base uncased).", "reasons_to_reject": "Even though the object of study and the results are clearly presented, the actual contributions of the paper are a bit muddled. The fact that \"Transformers Represent Some Linguistic Features Linearly\" is not, strictly speaking, a new finding: the whole line of work on iterative nullspace projection is predicated on this idea, as it tries to erase/manipulate linearly seaparable properties inside Transformer representations. \n\nThe authors say that previous works \"report effect sizes smaller than 50 percentage points... [and] do not show that the opposite value of the feature can be encoded\" (lines 69-73). This is true, but some of the previous work seems to have been solving a slightly different task using a very similar method, and in the end it remains unclear if (i) we are presented with numerically better results that are due to an improved methodology or (ii) the authors claim that an improved methodology coupled with a change in the task definition leads to new knowledge about Transformers in general, i.e. that they encode some linguistic features _almost exlusively_ linearly. \n\nMoreover, \"encoding\" here is equated with \"represented in such a way that their effect on the behaviour of the model can be undone using linear methods\". As we know, Transformers can also encode information that they do not use for predictions and that can be still recovered with non-linear classifiers.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper argues that there is a subpace in the vector space of BERT token representations that encodes grammatical number, i.e. the information necessary for correctly predicting the form of the verb as regards the correct number agreement with the subject. The authors show that it is possible to find an orthonormal lower-dimensional basis for this subspace and then use the elements of this basis to erase or invert the information about grammatical number contained in token representations. The authors use iterative nullspace projection by Ravfogel et al. (2020) to compute the basis and measure the effect of their interventions using the dataset on number agreement compiled by Goldberg (2019). They show that their method is largely robust to the choice of hyperparameters (dimensionality of the subspace and the \"intensity of intervention\", essentially a general weight for the sum of weighted basis vectors), but that it is important to carefully choose the target layer and token depending on the structure of the sentence. The authors repeat their experiments on the MultiBERTs showing that their results are not limited to the uncased version of BERT base.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "A short discussion of the notion of \"encoding\" in the sense of what model representations encode vs. what the models actually uses for predictions would be in order. https://arxiv.org/pdf/2204.08831.pdf provides an overview of relevant literature.", "Typos_Grammar_Style_and_Presentation_Improvements": "The presentation of the methodology is a bit surprising: the key formula is presented inside a figure and its elements are not discussed at all in the main text. Nothing is said at all about how to calculate $\\lambda$'s, which is probably recoverable but should be more explicit.\n\n\"Some\" in the title is replaced with a single feature, grammatical number, already in the abstract. Unless the authors want to add number to other features discussed in previous works, with arguably weaker results, this looks like overselling."}}, {"review_id": "smQiPvhsXG", "reviewer": "Reviewer_vo72", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. \n\nThe points about testing a wider range of model types is fair and well-taken. We would like to mention that we do consider a number of different syntactic structures in our testing data, even if those structures are subject to experimental control, as we describe in our Limitations section. And while we have only considered BERT-base models, we have repeated our experiments on a number of other BERT-base models (i.e., the MultiBERTs) trained from different random initializations.\n\nAs for the title, we are happy to change it in order to better convey the content of the paper, perhaps to: “Verb Conjugation in Transformers is Determined by Linear Encodings of Subject Number.”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It's a well-written paper. The experiment setup is neat and the result analysis is clear.\n- This paper proposes an interesting finding that the BERT model has linear encoded representation for the verb conjugation task. They show strong evidences and robust analysis to support the finding under their specific experiment setup.\n- The paper might be able to inspire future work to investigate other linguistic features embedded in the BERT using the similar paradigm", "reasons_to_reject": "This paper focuses on examining whether the transformer models represent some linguistic features linearly by applying the causal intervention analysis. They did the analysis on a specific setup, e.g, a single model type. I think the analysis can be better supported and more useful to the future work by having more variances added to the experiment setup:\n- BERT models come with difference sizes (base, large, etc.)\n- other up-to-date transformer models or LLMs\n- texts with difference syntactic / semantic structures (authors were aware of this in the limitation section)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an extension of the \"BERTology\". By probing the vectors from different layers of the BERT model using the causal internvention analysis, this paper shows that the BERT model contains a linear encoding of the representations for the task of verb conjugation. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "This is not a weakness so I put it here. The title of this paper I think is kind of \"exaggerating\" in a way that it's so general that readers might not be able to get a gist on what specific linguistic feature is being discussed in the paper. There is no harm I think putting words relating to verb conjugation in the title."}}]

Author rebuttal:

Meta-review: The paper argues that there is a subpace in the vector space of BERT token representations that encodes grammatical number, i.e. the information necessary for correctly predicting the form of the verb as regards the correct number agreement with the subject. The authors show that it is possible to find an orthonormal lower-dimensional basis for this subspace and then use the elements of this basis to erase or invert the information about grammatical number contained in token representations. The authors use iterative nullspace projection by Ravfogel et al. (2020) to compute the basis and measure the effect of their interventions using the dataset on number agreement compiled by Goldberg (2019). They show that their method is largely robust to the choice of hyperparameters (dimensionality of the subspace and the "intensity of intervention", essentially a general weight for the sum of weighted basis vectors), but that it is important to carefully choose the target layer and token depending on the structure of the sentence. The authors repeat their experiments on the MultiBERTs showing that their results are not limited to the uncased version of BERT base.

While the paper is interesting and well written it seems that the novelty and experimental setup are somewhat limited (i.e. the excitement is somewhat limited).
Decision: Accept-Findings