Outlier Dimensions Encode Task Specific Knowledge
Authors: William Rudman
Catherine Chen
Carsten Eickhoff
Venue: EMNLP 2023 Main (2023)
Keywords: outlier dimensions
LLMs
fine-tuning
interpretability
TL;DR: Our results suggest that outlier dimensions can encode crucial task-specific knowledge and that the value of a representation in a single outlier dimension drives downstream model decisions.

Abstract: Representations from large language models (LLMs) are known to be dominated by a small subset of dimensions with exceedingly high variance. Previous works have argued that although ablating these outlier dimensions in LLM representations hurts downstream performance, outlier dimensions are detrimental to the representational quality of embeddings. In this study, we investigate how fine-tuning impacts outlier dimensions and show that 1) outlier dimensions that occur in pre-training persist in fine-tuned models and 2) a single outlier dimension can complete downstream tasks with a minimal error rate. Our results suggest that outlier dimensions can encode crucial task-specific knowledge and that the value of a representation in a single outlier dimension drives downstream model decisions.

Outlier Dimensions Encode Task-Specific Knowledge
William Rudman', Catherine Chen', Carsten Eickhoff
Brown University' University of Tübingen?
@aiilliam_rudaaa catherine carsten. (aatherrine_sscheeeeerrrer
chenceernwwwweeee edu
Abstract
There currently is little understanding of how fine-tuning impacts outlier dimensions and why ab- lating outlier dimensions is harmful to downstream performance. We address this gap in the litera- ture by investigating 1) how fine-tuning changes the structure of outlier dimensions and 2) testing the hypothesis that outlier dimensions contain task- specific knowledge. This study makes the follow- ing novel contributions:
Representations from large language models (LLMs) are known to be dominated by small subset of dimensions with exceedingly high variance. Previous works have argued that although ablating these outlier dimensions in L.M representations hurts downstream perfor- mance, outlier dimensions are detrimental to the representational quality of embeddings. In this study, we investigate how fine-tuning im- pacts outlier dimensions and show that 1) out- lier dimensions that occur in pre-training per- sist fine-tuned models and 2) a single out- lier dimension complete downstream tasks with a minimal error rate. Our results suggest outlier dimensions can encode crucial task- specific knowledee ledge and that the value of rep- resentation in a single outlier dimension drives downstream model decisions.
1.
We find that outlier dimensions present in pre- training remain outlier dimensions after fine- tuning, regardless of the given downstream task or random seed.
2. We demonstrate that outlier dimensions in ALBERT, GPT-2. Pythia-160M, and Pythia- 410M encode task-specific knowledge and show that it is feasible to accomplish down- stream tasks by applying a linear threshold to a single outlier dimension with only a marginal performance decline.
1 Introduction
Large Language Models (LLMs) highly over- parameterized with LLM representations utilizing only a small portion of the available embedding space uniformly (Gordon et al., 2020; Prasanna et al., 2020; Rudman et al., 2022). Representations of transformer-based L.MSs are dominated by a few outlier dimensions whose variance and mag- nitude are significantly larger than the rest of the model's representations (Timkey and van Schijn- del, 2021; Kovaleva al., 2021). Previous studies devoted to the formation of outlier dimensions in pre-trained L.LMs suggest that imbalanced token frequency causes an uneven distribution variance in model representations (Gao et al., 2019; Puccetti et al., 2022). Although many argue that outlier di- mensions "disrupt" model representations, making them less interpretable and hindering model perfor- mance, ablating outlier dimensions has been shown cause downstream performance decrease dra- matically (Kovaleva et al., 2021; Puccetti et al., 2022).
Related Works
Two seminal works discovered the presence of "out- lier" (Kovaleva et al., 2021) or "rogue" (Timkey and van Schijndel, 2021) dimensions in pre-trained LLMs. Following Kovaleva et al. (2021) and Puc- cetti et al. (2022), we define outlier dimensions as dimensions in LLM representations whose vari- ance at least 5x larger than the average variance in the global vector space. The formation outlier dimensions is caused by a token imbalance in the pre-training data with more common tokens hav- ing much higher norms in the outlier dimensions compared to rare tokens (Gao et al., 2019; Puccetti et al., 2022). Although the community agrees the origin of outlier dimensions, their impact on the representational quality of pre-trained LLMs has been widely contested.
The concept of isotropy (i.c., the uniformity variance in a distribution) closely related to out-
'Code: http://shhiu..cmmmwwwaaatttttttr

---

GPT2 Pre-Trsined
BERT Pre-Trained
ALBERT Pre- Pre-Treneed
DistiBBERT Pre-Trsined
200.0
3.0
Value Por Dincusloo
200.3
100
10
4.0
104.3
20
60
0.0
100.3
60
00.3
-10
-00
-0.
300
30)
9A
103 76
38J
20
aU
D03
790
100
230
aG
621
TED
530
188 200 800 Dimesston Space
331 26
ds
Space
Space
Dimeusion
Space
GPT2 Fine-Trned
BERT Fine-Tunet
ALBERT Fine-TTie Tuned
DistilEERT Fine- Tuned
Valae Pre Dimunzoon
40
20
Negalio
9:-
-2.
00.0
+0.
20.0
00
6.0
-20
Paatte
20.3
Negaiie
Begalee
Dimstsion Vactor Space
Dicamniion Vector Sjomm
Dineniion Vector Sprce
Dimession Vactor Space
Figure 1: Average activation diagrams of sentence embeddings on the SST-2 validation dataset. The x-axis represents the index of the dimension, and the y-axis is the corresponding magnitude in that given dimension. Top: pre-trained models where fine-tuning occurs. Bottom: models fine-tuna tne-tuned to complete SST-2.
lier dimensions. Namely, the presence of outlier di- mensions causes model representations to be highly anisotropic (Rudman and Lickhoff, 2023). Many previous works have argued that mitigating the im- pact of outlier dimensions by forcing LLM rep- resentations be isotropic improves model inter- pretability and performance (Rajace and Pilehvar, 2021: Liang et al., 2021; Mu et al., 2017; Zhou et al.. 2020; Gao et al., 2019). Further, Timkey and van Schijndel (2021) claim that outlier dimen- sions do not meaningfully contribute to the model decision- decision-making process and that removing outlier dimensions aligns LLM embeddings more closely to human similarity judgments. Although the no- tion that isotropy is beneficial to model represen- tations has been widely adopted in the literature. recent studies have shown that many tools used to measure isotropy and the impact of outlier dimen- sions in NLP are fundamentally flawed (Rudman et al., 2022).
stream tasks. this study, we demonstrate that certain LLMs store specifcc knowledge in a I-dimensional subspace and provide evidence sup- porting claims that outlier dimensions are benefi- cial to model performance.
3 Experiments
Training Details We fine-tune 4 transformer coder LLMs: BERT (Devlin et al., 2018), ALBERT (Lan et al., 2020), DistilBERT' (Sanh et al., 2020). RoBERTa (Liu et al., 2019) and 4 transformer de- coder LLMs: GPT-2 (Radford et al., 2019), Pythia- 70M, Pythia-160M, Pythia-4 40M (Biderman et al., 2023). We fine-tune our models on 5 binary clas- sification tasks contained in the GLUE benchmark (Wang et al., 2018): SST-2 (Socher et al., 2013), QNLI (Rajpurkar et al., 2016), RTE (Dagan et al., 2005), MRPC (Dolan and Brockett, 2005), QQP. A detailed description of each task is available Sec- tion A of the Appendix. A detailed description of our hyperpurameter search, exact hyperparameters, and random seeds is given in Section B. We follow the common practice of reporting results on GLUE tasks using the hidden validation dataset.
There is growing body of literature arguing that anisotropy is a natural consequence of stochas- tie gradient descent and that compressing repre- sentations into low-dimensional manifold corre- lates with improved downstream performance (Zhu et al., 2018; Ansuini et al., 2019; Recanatesi et al., 2019; Rudman and Eickhoff, 2023). Recent works in NLP suggest that L.Mss store linguistic and task-specific information in low-dimensional sub- space (Coenen et al., 2019; Hernandez and An- dreas, 2021: Zhang al., 2023). Further, Rud- man and Eickhoff (2023) argue that encouraging the formation of outlier dimensions in LLM repre- sentations improves model performance on down-
3.1 Persistence of Outlier Dimensions
Methods After training the model, we calculate the variance of sentence embeddings on the valida- tion data on each task and each random seed and count the number of times given dimension has variance 5x larger than the overall average. We visualize outlier dimensions by creating "activation diagrams" where the x x-axis is the index of a given dimension, and the y-axis is the magnitude of a

---

RoBERTs
BERT
ALBERT
DiCTBERT
088
567
216
58
133
120
05
001
Dinension
933
500
450
881
746
27h
10055
Froquency
Frequenccy
Frequency
Dimensson
Proquenccy
GPT2
Pythia-70MM
Pythia- Pythla-160M
Pythie-4
136
314
50
274
119
529
110
316
772
706
21
199
710
09
26%
75%
100%
08
2035
30%
TXX
10075
055
20.5
70%
100%
05
25%
2005
70%
1007
Frequency am Outlee Dimesiion
Prequency on Outlier Dimension
Frequency am Outlier Dimension
Frequetic am Outller Dimension
Figure 2: Frequency of the most commonly occurring outlier dimensions across all fine-tuning tasks and all random seeds. The x-axis plots the dimension frequency, and the y-axis plots the dimension index.
sentence embedding in that dimension. We report the average sentence embeddings across random seeds for all activation diagrams.
an occurrence rate higher than 70%.
Not only do the outlier dimensions in Pythia models have low occurrence rates, but the Pythia models have far fewer outlier dimensions present in the embedding space compared BERT, AL- BERT, DistilBERT, and RoBERTa. In general, far more outlier dimensions emerge in the encoder models considered in this study compared to the decoder models. In particular, GPT-2 and Pythia- 70M only have and unique outlier dimensions that appear across all fine-tuning tasks and random seeds compared to 62, 60, 24, and 64 for BERT, ALBERT, DistilBERT, and RoBERTa, respectively, Interestingly, Figure 2 shows that the most com- mon outlier dimensions in BERT and DistilBERT are the same, indicating that outlier dimensions persist even when distilling larger models. Further discussion the persistence of outlier dimensions is available in Section C.
Results Figure demonstrates how fine-tuning impacts model representations of sentence embed- dings by plotting activation diagrams from BERT, ALBERT, DistilBERT, and GPT-2 on the SST-2 validation data before (top row) and after (bottom row) fine-tuning. Activation diagrams for the re- maining four models available in Section E in the Appendix. The magnitudes of outlier dimen- sions in GPT-2 are far larger than any of the mod- els considered in this study. The outlier dimension with the largest variance in GPT-2 has an average variance value 351 1.82 compared to 4.87, 9.30, and 4.68 for BERT, ALBERT, and DistilBERT (see Section C for full results). For GPT-2, fine-tuning exacerbates the variance of existing outlier dimen- sions but decreases the mean value of outlier di- mensions. Notably, in GPT-2, the exact set of top 3 outlier dimensions in pre-training persist when fine-tuning models to complete downstream tasks. Figure 2 demonstrates that a small subset of outlier dimensions emerge for given model regardless of the downstream classification tasks or the random seed. In particular, in GPT-2 and RoBERTa, there are dimensions that qualify as outlier dimensions for every fine-tuning task and random seed. Outlier dimensions in the Pythia models have far lower occurrence rate than any of the models in the paper. This finding is especially pronounced Pythia- 70M and Pythia- Aithia-160M. 160M, where no dimensions have
3.2 Testing Outlier Dimensions for Task-Specifie Knowledge
Methods order to test the hypothesis that out- lier dimensions contain task-specific knowledge, we attempt to complete an inference task using only the outlier dimension in the fine-tuned model with the highest variance. For the remainder of this paper, refer to the outlier dimension with the highest variance as the principal outlier dimen- sion, which we denote as p. After fine-tuning our model, we use a simple brute-force algorithm find a linear decision rule to complete the down- stream GLUE task using only the principal outlier

---

Task
ALBERT
DiGHBERT
Pythis-70M
Pythis 100M
Pythin-4I0M
313677558
04.50R
0177/91899
87.307262 2168
8912508664
ONLI
50.1199699 522.66
91.2888.16 33.42
$6.48866.12 322.54
92.78/9909
35.67
87 37.89935.90
322.16
50.1485..55 522.47
85.4182.57 $2.97
91,4191411 25000
RIE
51.0155.00 238.7
66,7UK3.CM 25.41
55.1415244 23490
78,50/0.314 & 8.16
61,64/99.4 5.M0
55.14/53997 & 2.12
61,8200000 A 19.12
TLS8 47.062235.9
MRPC
118807600 310.3
87.00/99 72 38. 18
#1 5675006 37.97
15/90.00 70 6.33
78 78.9074.08
66.1
70,71/8875
4.12
Ti4 AAAAs 26 A 8.76
78 65 20
00P
27
90.9981
89.38/8991
$6.38/70:0
89.12/85.77
85,04150477
Avg
K5, SPOUTST 13
RO ITIV
15.177E2 1922
81,9079966 3233
16
M0000/4:11
Table 1: Comparing the performance of the fully fine-tuned model to our brute-force algorithm on the principal outlier dimension, p (Equation 1). We compute the percent decrease between the full model performance and the 1-D performance using Equation as full-performance minus ID- ID-performance divided by full- performancc. The reported performance is an average over 4 random seeds.
dimension. We first collect small sample of 500 sentence embeddings from the training data to find P and calculate its mean value, which we denote as Ap- Equation| describes the classification decision rule for an input sentence using only p.
respectively. Additionally, the average percent de- crease in performance is significantly lower for GPT-2 (2.85%), ALBERT (6. 12%), Pythia- 160M (7.48%) compared to BERT (16.33%) DistilBERT (19.12%) and RoBERTa (19.23%).
10 x S Mp + Flabel (1 if x, > Mp te
3.3 Variance vs. ID-Performance
(1)
Methods In this section, we extend our experi- ment Section 3.2 by testing each 1-D subspace in model activations for specifci knowledge. Namely, we use our brute-force algorithm Equa- tion I to learn a linear threshold for cach dimension of the model's sentence embeddings.
where Tp denotes the principal outlier di- mension for an input sentence I, E {-50, -4.0,49.5..... ... 49.5,50), Let Xtahel denote the training accuracy of Equation 1. If Xiahel > Xlabel we flip the inequalities in Equation 1:
Results Figure 3 shows multiple 1-D subspaces in model sentence embeddings that contain task- specific information. Importantly, the downstream performance of single dimension is strongly cor- related with the variance in that given dimension. Even in cases where the principal outlier dimension does not contain enough task-specific information to complete the downstream task, there are several non-principal outlier dimensions that are capable of completing the downstream task with a high degree of accuracy. For instance, applying Equa- tion to the largest outlier dimension in BERT on QNLI results in 22.26% decrease in performance, whereas using Equation 1 on the 5th largest out- lier dimension only leads 0.56% reduction in performance. We even find a few cases where ap- plying Equation results in improved performance over feeding the full representations into a classifi- cation head. For full results and further discussion, see Table 3 and Section D in the Appendix.
if up 2 14 t So
Glabel
xp < Up t
After finding both the value of e that maximizes accuracy on the training data and the correct direc- tion of the incquality, we measure the ability of p complete downstream tasks using Equation on the hidden validation data.
Results In GPT-2, using the principal outlier di- mension results in only 3% performance drop compared to using the full model representations for most tasks (Table 1). Further, there are several tasks in ALBERT, Pythia-160M, and Pythia-410M where there is little no change in performance when using only the principal outlier dimension. Although outlier dimensions encode task-specific knowledge in some models, outlier dimensions in BERT, DistilBERT, RoBERTa, and Pythia-70M are insufficient for completing most downstream tasks. In particular, for QNLI, using our brute- force algorithm on single outlier dimension in GPT-2, ALBERT, Pythia- 160M, and Pythia-410M only results in a 2.16%, 3.42%, 2.97%, and 0% per- formance decrease, where performance on QNLI drops by 22.66%, 23.54%, 33.67% and 21.47% for BERT, DistilBERT, RoBERTa, and Pythia-70M
Discussion
Previous studies investigating the role of outlier di- mensions tend to only focus on BERT RoBERTa, yet make broad claims about the role of outlier di- mensions for L.MSs as a whole (Kovaleva et al., 2021; Puccetti et al., 2022). Our results demon- strate that outlier dimensions have different func-

---

BERT QNLI
ALBERT QNLI
DiSTIBERT QNLI
RoBERTs QNLI
10
10.0
05
6/
D6
066
0.4
Performancs
Paffemuuee
Performaan
GPT-2 QNLI
Pythis-70M QNLI
Pythin-100M QNLI
Prthin-410M QNLI
134
D0
3X00
330
03
15
105 0.50
0.55
0,90
10:5
165
01
0.4
0.5
155
455
Figure 3: Comparing the downstream performance of all I-D subspaces of sentence embedding activations on QNLI against the variance in that given dimension. Results for tasks are available in Section D in the Appendix, The red dashed line indicates the threshold for whether dimension qualifies an "outlier dimension" (i.e., 5x the average variance in vector space).
tions for different models and tasks. In particular, outlier dimensions contain task-specific knowledge that can complete downstream fine-tuning tasks for some models (GPT-2, ALBERT, Pythia-160M, and Pythia-410M) but not for others (BERT, Distil- BERT, RoBERTa, and Pythia-70M). Future work should avoid generalizing results from one obser- vation to all other models.
low-dimensional subspace (Hernandez and An- dreas, 2021; Coenen et al., 2019; Zhang al., 2023). The persistence of the same small set of outlier dimensions in pre-training and fine- tuning across various classification tasks and ran- dom seeds provides strong evidence that the low- dimensional subspaces learned for different tasks are highly similar. Namely, when fine-tuning, cer- tain LLMs adapt the same small set of outlier di- mensions to store task knowledge.
Although numerous studies have argued that out- lier dimensions are harmful to model representa- tions (Gao et al., 2019; Timkey and van Schijndel, 2021; Cai et al., 2021), we quantitatively show that the single principal outlier dimension can store enough saek-fpe knowledge in GPT-2, AL- BERT, P and Pythia-410M to complete downstream tasks. In cases where the principal outlier dimension does not contain sufficient task- specific knowledge complete downstream tasks, we find that there often non-principal outlier di- mensions that retain high 1-D performance. par- ticular, Figure 3 shows that there are con-principal outlier dimensions in BERT and RoBERTa that can complete QNLI using Equation with only a 0.56% and 1.01 101% % performance decrease compared to us- ing full model representations and classification head. These findings help explain recent results showing that encouraging even larger variance in outlier dimensions is beneficial LLM fine-tuning performance (Rudman and Eickhoff. 2023).
Conclusions & Future Works
This study challenges the dominant belicf in the literature that outlier dimensions are detrimental to model performance by demonstrating that 1) the exact outlier dimensions that emerge in pre- training persist when fine-tuning models regardless of the classification task or random seed and 2) in some LLMs, outlier dimensions contain enough task-specific knowledge linearly separate points by class label. However, it is still unclear why the principal outlier dimension contains task-specific knowledge in some models and not others. Future work should investigate the specifics of these oc- currences and how this finding is affected by model scale, architectural choices, and training objectives. Ultimately, understanding the mechanisms and im- plications of outlier dimensions in LLMs can con- tribute to advancements transfer learning, model interpretability, and optimizing performance in sev- eral NLP tasks.
Additionally, our finding that 1-D subspaces con- tain specifce knowledge strengthens the argu- ments that LLMs store linguistic knowledge in

---

6 Limitations
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Representation degeneration prob- lem in training natural language generation models. CoRR, abx/1907.12009
Our study follows the common practice of only con- sidering binary classification tasks. Studies have yet to investigate how changing outlier dimensions behave when fine-tuning for alternative tasks such as question-answe
Jonas Geiping and Tom Goldstein. 2022. Cramming: Training a language model on single gpu in one day.
or generative tasks. We limit our analysis to smaller models that are easy to fine- tunc. We do not consider how model size impacts the presence of outlier dimensions and whether outlier dimensions store task-specific information with very large LLMs. However, outlier dimen- sions will likely continue to play a role in larger models, given that outlier dimensions are persistent in GPT-2 and that most of the largest models in NLP are transformer decoders.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 143-155, Online. Association for Com- putational Linguistics.
Evan Hernandez and Jacob Andreas. 2021. The low- dimensional linear geometry of contextualized word representations. CoRR, abs/2105.07109.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. BERT busters: Outlier dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 3392-3405, Online. Association for Computational Linguistics.
References
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan, 2019. Intrinsic Dimension of Data Representations in Deep Neural Networks. Curran Associates Inc., Red Hook, NY, USA.
Zhenzhong Lan, Mingda Chen, Sehastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
Yuxin Liang, Rui Cao, Jie Zheng, Jie Ren, and Ling Gao. 2021. Learning to remove: Towards isotropic pre- trained BERT embedding. CoRR, abs/2104.05274,
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692
Xingyu Cai, Jijji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Con- ference on Learning Representations.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry of bert.
Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2017. All-but-the-top: Simple and effective postprocessing for word representations. CoRR, abs/1702.01117
Sai Prasanna. Anna Rogers, and Anna Rumshisky. 2020. When BERT Plays the Lottery, All Tickets Are Win- ning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 3208-3229, Online. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proceedings of the First Inter- national Conference on Muchine Learning Chal- lenges: Evaluating Predictive Uncertainty Visual Ob- ject Classification, and Recognizing Textual Entail- ment, MLCW'05, page 177-190, Berlin, Heidelberg. Springer-Verth Verlag.
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice OrlettO. 2022. Outlier dimensions that disrupt transformers are driven by frequency. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 1286-1304, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99

---

Sara Rajaee and Mohammad Taher Pilehvar. 2021.
Zhong Zhang, Bang Liu, and Junming Shao. 2023. Fine- tuning happens in tiny subspaces: Exploring intrinsic task-specific subspaces of pre-trained language mod- els.
cluster-based approach for improving isotropy in con- textual embedding space. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence Natural Language Processing (Volume 2: Short Papers), pages 575-584, Online. Association for Computational Linguistics.
Wenxuan Zhou, Bill Yuchen Lin, and Xiang Ren. 2020. Isobn: Fine-tuning BERT with isotropic batch nor- malization. CoRR, abs/2005.02178.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehenston of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu-
Jinwen Ma. 2018. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects.
ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguisties.
A Dataset Details
Stanford Sentiment Treebank with 2 classes (SST- 2) is a binary classification task where models must determine whether a short movie review is posi- tive or negative in sentiment (Socher et al., 2013). Question-ansswrring Natural Language Inference (QNLI) is binary natural language inference task where models must decide whether or not a given answer is entailed from a specified question (Ra- jpurkar et al., 2016). Recognizing Textual Entail- ment (RTE) is a binary classification task where a model must determine a given sentence logically follows preceding sentence. The Microsoft Re- search Paraphrase Corpus (MRPC) tasks models with determining if a pair of sentences para- phrases of each other (i.e., semantically equivalent). The Quora Question Pairs (QQP) dataset consists of question pairs from Quora. Models must deter- mine if the sentence pairs are semantically equiva- lent. Note that all tasks are datasets in the GLUE benchmark (Wang et al., 2018).
Stefano Recanatesi, Matthew Farrell, Madhu Ad- vani, Timothy Moore, Guillaume Lajoie, and Eric Shea-Brown. 2019. Dimensionality compression and expansion in deep neural networks. CoRR, abs/1906.00443.
William Rudman and Carsten Eickhoff. 2023. Stable anisotropic regularization.
William Rudman, Nate Gillman, Taylor Rayne, and Carsten Eickhoff. 2022. IsoScore: Measuring the uniformity of embedding space utilization. In Find- ings of the Association for Computational Linguis- tics: ACI. 2022, pages 3325-3339, Dublin, Ireland. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642. Seattle, Washington, USA. Association for Computational I Linguistics.
Model
L.R.
Batch Size
Epoch
BERT
3e-5
32
2
ALBERT
le-5
32
3
DistilBERT
le-5
64
5
William Timkey and Marten van Schijndel. 2021. bark and no bite: Rogue dimensions in transformer language models obscure representational quality. CoRR, abs/2109.0e 0440044
RoBERTa
le-5
64
3
GPT-2
le-5
32
3
Pythia-70M
le-5
32
4
Pythia-160M
le-5
32
4
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A muuli--tas task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Pythia-410M
le-5
32
4
Table 2: Detailed model hyperparameters.
B Hyperparameter Details
Following Geiping and Goldstein (2022), we hyperparameter-ttne each model on a single task to learn a set of global hyperparameters. We then use the set of global hyperparameters to fine-tune each
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 2625-641.

---

BERT
ALBERT
DISHIBERT
RoBERTO
160M
Pytiia 410M
AAC
50093483
62.973465
740859917
75.0634.89
68.7833.64
63,4428366
7031411:76
O0F
AVg.
Table 3: Maximum value of applying our brute force algorithm to all ID subspaces in the last layer sentence embedding to the full model performance. A represents the percent change between the maximum ID value and the full model performance. Note that bold entries are those where the best ID subspace performs better than the full model representations. Our brute force algorithm on the best ID subspace. Variance % is the percentile of the variance of the dimension that corresponds to the maximum brute-force classification accuracy. The reported performance is an average over 4 random seeds.
model on the remaining tasks. For this study, we learn global hyperparameters from QNLI and train our models using random seeds. Note that we exclude COLA (Warstadt et al., 2019) from anal- ysis as Geiping and Goldstein (2022) find COLA is highly sensitive to hyperparameter tuning and performs poorly with global set of parameters. For each model, search for the optimal com- bination of learning rate (le-5, 3e-5, 5e-5, le-4), batch size {16, 32, 64) and the number of train- ing epochs (1,2,3,4,5). To correctly perform the hyperparameter tuning, we randomly remove 5% of the training data and use it as hyperparameter- tuning evaluation data since we report performance results on the GLUE validation data. Table 2 de- tails the complete set of hyperparameters used for each task in this paper. Note: we train all of our models using mixed-precision training, except for Pythia-70M, and Pythia- 160M. For Pythia-70M and Pythia- 160M, we use full precision since we re- ceived NaN values in our loss function when using mixed-point precision training. Once we learn the global hyperparameters set in Table 2, we fine-tune the models on random seeds 1,2.3,4.
Fine- Tuned
Model
Nom Outliers
Avg. Var(o)
Num Outliers
Arg. Vartel
BERT
0.10
62
4.87
ALBERT
3
3.74
60
9.30
DistiBERT
0.04
25
4,68
RoBERTa
0.02
54
13:45
GPT-2
6
619.18
8
3511.82
Pythia-70M
2
13.49
4
61,54
Pythis-16UM
35
32.71
27
9.85
Pythia-110N
11
12.15
129
6,48
Table 4: Counts of the number of outlier dimensions that appear across all tasks and the average variance value of the principal outline dimensions,
D All 1-D Results
Variance vs. Performance In this Section, we provide full results for the experiment Sec- tion 3.3. Namely, we apply Equation to all 1- dimensional subspaces in the sentence embeddings of each model every task. For nearly every model and every task, we find the same strong cor- relation between dimension variance and the abil- ity to encode task-specific information. There are very few exceptions to this trend. However, there are two tasks (RTE& MRPC) where there is no strong correlation between performance and vari- ance for some of the models considered in this pa- per. Note that in MRPC, correlation between high variance value and performance does not emerge as Pythia-70M and Pythia-160M barely perform above predicting the class majority label of 68.38%. In fact, no 1-D subspace of P00h fine-tuned on MRPC performs above "random" performance of 68.38%. For RTE, there is only a mild correla- tion between variance and performance for BERT, ALBERT, and RoBERTa. For DistilBERT and the Pythia models, the correlation between variance and performance degrades even further. We hy- pothesize this is in part due to RTE being a difficult task where models tend to perform poorly. An in- teresting direction of future work would be to see if there are types of tasks, such as question-answers
C Persistence of Outlier Dimensions Continued
First, note that the dimension of Pothi acti- vation space is 512, and the dimension of Pythia- 410M's activation space is 1028. All other models have activation spaces with 768 dimensions. Table 4 lists the number of outlier dimensions and the aver- age maximum variance value of all models on all tasks. Fine-tuning models increases both the num- ber of outlier dimensions present in embedding space as well as the average maximum variance value. This trend the strongest for the encoder models considered in this paper as well as GPT-2 and 70M.

---

RoBERTTa Pre-Treee Trsined
PYTHIA-TUM Pre- Trained
PYTHIA-1GUM Pre-Trxined
PYTHIA-TIM Pre- Trained
10.0
600
2010
2200
8.0
0.0
0.0
10.0
-10.
-4.0
-0000
-3000
-2500
120 300 260 10 206 69 70
580
100
7-0
200
200
200
103
300
263
890 260
630 200 569
100 30 30 80 60 aE WOHDES
Dimomssio in Vector Space
Dimeusion Vectio Spacce
Dimnmttton in Vector Space
Dimettsio in Vector Space
ROBERTA A Fine- Tuned
PYTHIA 70M Fine Tuned
PYTHIA 160M Fine- Tuned
PYTHIA 410M Fine Tuned
Value Per Limesion
2000
Pontiee Naatie
I0.
Negiae
43.0
2001
TILD
2031
9.0
ILD
-2001
Nogattin
-200
Nagalion
Pesition
130 20 360 RU 566 59 76
503
I1.
2411
201
200 Sjarco
303 20) 3E3
23)
E30 800
180 20 as ak S0P AADLIS
Diaoonsion Space
Dimcusion
Vertor Space
Spaze
Figure Average activation diagrams of sentence embeddings on the SST-2 validation dataset. The x-axis represents the index of the dimension, and the y-axis is the corresponding magnitude in that given dimension. Top: pre-trained models where no fine-tuning occurs. Bottom: models fine-tuned to complete SST-2.
textual entailment, that impact an outlier dimen- sion's i ability to store task-specific information.
different trends. In Pythia-160M and Pythia-410M. the average variance in the principal outlier dimen- sion decreases after fine-tuning. Table shows that the average max variance decreases from 32.71 9.85 in 1ythia-160M and decreases from 12.15 6.48 Pythia-410M. Interestingly, in Pythia-700 70M and Pythia-160M, the embedding dimensions are much further from the origin compared to every other model considered in this paper. Our results highlight how, even for models with similar archi- tectures (Pythia models and GPT-2), the structure of embedding space can be very dissimilar. Further research needed to understand how model archi- tecture and training objectives impact the structure of model embeddings.
Maximal 1-D Subspaces Figures 3 and demon- strate that oftentimes the principal outlier dimen- sion is not the best performing 1-D subspace. In Table 3, we report the maximum performance of I-D subspace after applying Equation I complete the downstream task along with the percentile of the variance of the maximal 1-D subspace. Trends Table provide further evidence for our find- ing that the variance of the activation value corre- lates with 1-D performance. With the exception of POth the average performance decrease between the maximum 1 I-D performance and the full model performance is less than <5% for all models considered in this paper. Surprisingly, find cases where the maximum 1-D subspace per- forms better than feeding the full representations into a classification head. This finding is the most pronounced on RoBERTa-RTE RTE, Pythia-410M M00iaa-40--QQO0 QQQ,, and Pythia-410M-QNLI where the best 1-D sub- space improves upon full model performance by 4.09%, 4.54%, and 2.56%, respectively.
E All Activation Diagrams
In this section, we report the remaining models' (RoBERTa, Pythia-70M, Pythia 1ythia-160M, and Pythia- 410M) activation diagrams SST-2. Trends SST-2 are representative of all of the tasks consid- ered in this paper. We report similar phenomenon of variance drastically increasing RoBERTa and Pythia-70M after fine-tuning, particularly in outlier dimensions. The Pythia models, however, exhibit

---

THST SSTM
ATZERTT S9T9
DESDERR SST9
RoCIERTa SET7
GIPT-22 5812
Pethia-703 701 5812
Pettis 183
Lylho-110M S812
300
BERT RIE
ALBERTI STE
I's LIBERT RTZ
BoBERTs RTE
Deffecaco
Perberance
GPT2 RTT
Pythts RTY
Pytho 18M RTT
Pertia 1IMM
EIE
IE MHRCC
ALTORT MERC
Dretittin MIPP?
Heat MIDK!
Pc.
500
Pobensnce
CPT-2 MRRCC
Pyllii-TIG MBPC
Pythis- 1000 MRPC
Py, Pydlis-tiM MRPC
Peckmuaa
Pechenance
BERT
ALBERT
Dis IERT QQP
BoBERTI
300
678
679
13:
GPT QOP
Perbo T0M 00p
Pyclio 1800 OOP
110M QOP
2000
Figure 5: Comparing the downstream performance of all I-D subspace of sentence embedding activations on SST-2, RTE, MRPC, and QQP against the variance in that given dimension. Results for all tasks are available in Section D in the Appendix. The red dashed line indicates the threshold for whether a dimension qualifies as an "outlier dimension" (L.e. 5x the average variance in vector space).


Reviews:
[{"review_id": "3jG79k0E8Z", "reviewer": "Reviewer_WcPR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. Our study shows that outlier dimensions are beneficial specifically for some models (GPT-2 and ALBERT) and not others (BERT and DistilBERT), which is different from previous claims that outlier dimensions are harmful to LLMs in general (Kovaleva et al. 2021, Puccetti et al. 2022). Given the similarities between ALBERT and BERT/DistilBERT, such as the same vocabulary size, pre-training task, GELU activation functions, and transformer encoder architecture, it is difficult to determine the exact factors that cause a reliance on outlier dimensions. There are, however, a few notable architectural differences between ALBERT and BERT/DistilBERT, such as the use of parameter sharing in ALBERT. To determine which architectural features allow a model to store class information in outlier dimensions, we are actively expanding our results to another prominent decoder model in the literature: RoBERTa (Liu et al. 2019). GPT-2 is the only decoder model and, thus, was the only model trained with a language modeling objective. We are currently running experiments using a selection Pythia (Biderman et al. 2023) transformer decoder models to test if our findings hold for a wider variety of transformer decoder models. Preliminary findings show that outlier dimensions in Pythia-70M, Pythia-160M, and Pythia-410M all encode task-specific information similarly to GPT-2, indicating that model architecture and pre-training objectives are highly important in a model’s ability to store task-specific information into a single dimension. \n\nOur additional experiments will allow us to make more generalizable conclusions about the role of outlier dimensions in fine-tuning and will help provide more evidence as to what specific architectural or pre-training task differences allow our results to generalize to other models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper explores a novel aspect of language models by focusing on outlier dimensions and their impact on both pre-trained and fine-tuned models. This unique perspective adds to the understanding of how language models operate and influence downstream tasks.\n\nThe study finds that in certain LLMs (GPT-2 and ALBERT), a single outlier dimension alone can perform well in completing downstream tasks, achieving a minimal error rate. This implies that a single dimension can drive the decision-making of the downstream models.", "reasons_to_reject": "The study shows that the beneficial influence of outlier dimensions might be specific to the tested models (GPT-2 and ALBERT). It's essential to address why these findings cannot be generalized to other models and architectures.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the presence and impact of outlier dimensions in representations from language models  during fine-tuning for downstream tasks. \n\nThe main contributions of the paper are as follows:\n\nThe paper explores the phenomenon of outlier dimensions in representations from LLMs. It highlights that these outlier dimensions dominate the representation space and are characterized by high variance.\n\nThe paper demonstrates that outlier dimensions are crucial for downstream tasks, as ablating or removing them hurts the downstream task performance. This suggests that outlier dimensions encode task-specific knowledge that is important for the success of the fine-tuned models.\n\n\n\n\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g6Y4ol7uHH", "reviewer": "Reviewer_Qjdk", "summary": "", "questions": "What do you think these results can derive, in terms of future sentence embedding pre-training design? The paper will also be strengthen by adding this discussion.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions and your thorough review of our paper!\n\nOur results provide some evidence that an individual dimension's ability to complete a downstream task correlates with the magnitude of the individual dimension. Namely, the fact that the average variance value of the principal outlier dimensions in GPT-2 (3511.82) and ALBERT (9.30) are significantly larger than the principal outlier dimensions in BERT (4.87) and DistilBERT (4.68) is likely related to the ability of that dimension to store task-specific information. After running additional experiments, we find a strong correlation between variance and 1D performance. We notice that for BERT and DistilBERT, it is not always the single largest outlier dimension that encodes most task information. However, high variance dimensions are much more likely to cover critical task knowledge. Applying Equation 1, to learn a linear threshold on the 6th largest outlier dimension in BERT fine-tuned on SST-2 yields a classification accuracy of 90.73% compared to a classification accuracy of 91.86%  on the full model representations. The revised version of the paper will include detailed graphs and a dedicated discussion of this relationship.\n\n We cannot draw definitive conclusions as to whether differences in training objectives and model architecture play a role in determining whether the principal outlier dimension contains class-specific information. However, it is interesting to note that the variance of GPT2’s principal outlier dimension is much greater than those of its BERT-based counterparts. We are currently running experiments with an additional transformer encoder architecture: RoBERTa (Liu et al. 2019) and several transformer decoder architectures contained in the Pythia  (Biderman et al. 2023) suite of models to better understand how different model architectures and pretraining objectives impact a model’s ability to store class-specific information into outlier dimensions and to test further the hypothesis that the variance in a given outlier dimension correlates to improved downstream performance. Preliminary findings show that outlier dimensions in Pythia-70M, Pythia-160M, and Pythia-410M all encode task-specific information similarly to GPT-2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Very interesting results which are different from previous common wisdom that outlier dimensions are harmful.\n* Insightful observation that outlier dimensions encode task-specific tasks by a simple yet intriguing linear classification experiment.\n* The paper is easy-to-follow and well-written.", "reasons_to_reject": "* Section 3.1 studied the maginitude of outlier dimensions; however, only shallow observations were offered, without further analyses or insights. \n* In experiments, the authors compared different models, and observed a few interesting differences between different architectures. However, the paper will be stronger if it can connect these models’ training objectives/architecture choices with these differences. Despite this, I understand that this can be very difficult and hard to elaborate on in a short paper. This is more like a wish.", "questions_to_authors": "What do you think these results can derive, in terms of future sentence embedding pre-training design? The paper will also be strengthen by adding this discussion.", "paper_topic_and_main_contributions": "This paper studied the role of outlier dimensions (dimensions of much larger magnitude) in sentence embeddings (obtained from pre-trained language models). Despite previous studies argue that outlier dimensions hurt the quality of sentence embeddings, the authors observed that these outlier dimensions encoded important task-specific information, because a single outlier dimension sometimes (i.e., on some models/tasks) can perform downstream tasks with little performance loss. Moreover, the authors obtained a few interesting observations, including 1) outlier dimensions from pre-training phrase presisted after fine-tuning, and 2) outlier dimensions are shared before and after distillation (by comparing BERT and DistillBERT).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KGJ1lJiE3Y", "reviewer": "Reviewer_BqrZ", "summary": "", "questions": "Question 1: L 165: is 3511.82 correct here? It looks as it would be 300 or something?\n\nQuestion 2: Can we draw any conclusion about language characteristics from the fact that only a couple of outlier dimension are sufficient for downstream tasks using LLMs? That would be really interesting.\n\nQuestion 3: Spelling out the task abbreviations would be helpful.\n\nQuestion 4: A couple of more words on what outlier dimensions are would suffice.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful questions. \n\n$\\textbf{Question 1:}$  We apologize for the lack of clarity. Figure 1 reports the average sentence embeddings across different random seeds. Note that the average variance across random seeds will not, in general, equal the variance of the average of sentence embeddings. In contrast to Figure 1, the values we report in L 165 are the average of the variance, not the variance of the average. Further, the results reported in L165 are an average of all tasks and all random seeds compared to Figure 1. which visualizes the average sentence embeddings across four random seeds on SST-2.  \n\n$\\textbf{Question 2:}$  That is a great question! It is also a hard one to answer satisfactorily. Previous works (Gao et al. 2019, Pucceti et al. 2022) have connected outlier dimensions to a  token imbalance in the pre-training corpus, with Gao et al. 2019 suggesting that the Zipf distribution of word frequencies causes outlier dimensions. Upon receiving reviews, we again examined the token distribution frequency in the fine-tuning datasets. For SST-2, we find that certain words that are highly correlated with a given label of positive or negative sentiment, such as,  “good”,  “funny”, “bad,” and “dull”, occur with very high frequency in the SST-2 training data. Although this is beyond the scope of our study, a fruitful direction for future work would be to examine words in the training data with the highest mutual information and frequency with a given class label and examine the impact on 1D performance and the embedding norm of sentence embedding by swapping these words with less frequent synonyms that should not alter the model’s prediction. Concretely, we could replace the word “funny” in a sentence in SST-2 with a less common synonym, such as “chucklesome,” and measure the impact on outlier dimension variance and 1D performance. \n\n$\\textbf{Question 3:}$ Apologies for this oversight. We will provide the full names of the datasets in the paper and briefly describe each task in the appendix.\n\n$\\textbf{Question 4:}$ To address this, we will amend the description of “outlier dimensions” in our paper: Outlier dimensions are dimensions in model representations where the variance and magnitude of the dimension are significantly larger than the rest of the embedding space. Following Kovaleva et al. 2021 and Pucetti et al. 2022, a given dimension is an outlier dimension if the variance of the dimension is at least 5x larger than the average variance in the embedding space. Namely, given a set of sentence embeddings $X \\in R^{n}$, with covariance matrix $\\Sigma$ a dimension $i \\in \\{1,..,n\\}$ is an outlier dimension if $\\Sigma_{i,i} \\geq 5 \\cdot \\frac{1}{n} \\sum_{k=1}^{n} \\Sigma_{k,k}$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Positive: The paper shows that a linear threshold on only a few outlier dimensions decreases performance in downstream tasks for large language models only a little bit. it is written clearly and points out important characteristics for outlier dimensions.", "reasons_to_reject": "see Questions", "questions_to_authors": "Question 1: L 165: is 3511.82 correct here? It looks as it would be 300 or something?\n\nQuestion 2: Can we draw any conclusion about language characteristics from the fact that only a couple of outlier dimension are sufficient for downstream tasks using LLMs? That would be really interesting.\n\nQuestion 3: Spelling out the task abbreviations would be helpful.\n\nQuestion 4: A couple of more words on what outlier dimensions are would suffice.", "paper_topic_and_main_contributions": "Content: The paper presents a study on how outlier dimensions of word representation are preserved in fine-tuning, and how efficient they are in down-stream tasks using LLMs compared to BERTs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the impact of fine-tuning on outlier dimensions providing two insights. First, outlier dimensions in pretraining persist in fine-tuned models and second, a single outlier dimension can provide reasonable performance on a downstream task. The paper provides a nice novel insight that is interesting and well-presented.

Proposed points of improvement are (1) that the aspect that results may not generalise should be discussed: the rebuttal provides additional experiments and mentions similarities between models. (2) Analyses remain shallow and (3) address what conclusions can be drawn from the observation that a couple of outlier dimensions are sufficient. 

It can not be expected of a short paper, where an interesting observation that leads to new research directions is sufficient, to address all of the above. Given sufficient space, it would indeed be important to discuss what can be expected in terms of generalization and what remains for future research (if it only applies to some LMs, it's interesting as well) and what it means that even one outlier dimension can be fine-tuned for a task: should that even be the case?
Decision: Accept-Main