Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness
Authors: Qi Zhang
Yifei Wang
Jingyi Cui
Xiang Pan
Qi Lei
Stefanie Jegelka
Yisen Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Representation Learning
Robustness
Polysemanticity
Superposition
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Deep learning models often suffer from a lack of interpretability due to \emph{polysemanticity}, where individual neurons are activated by multiple unrelated semantics, resulting in unclear attributions of model behavior. Recent advances in \emph{monosemanticity}, where neurons correspond to consistent and distinct semantics, have significantly improved interpretability but are commonly believed to compromise accuracy. In this work, we challenge the prevailing belief of the accuracy-interpretability tradeoff, showing that monosemantic features not only enhance interpretability but also bring concrete gains in model performance of {\color{black} robustness-related tasks}. Across multiple robust learning scenarios—including input and label noise, few-shot learning, and out-of-domain generalization—our results show that models leveraging monosemantic features significantly outperform those relying on polysemantic features. Furthermore, we provide empirical and theoretical understandings on the robustness gains of feature monosemanticity. Our preliminary analysis suggests that monosemanticity, by promoting better separation of feature representations, leads to more robust decision boundaries {\color{black} under noise}. This diverse evidence highlights the \textbf{generality} of monosemanticity in improving model robustness. As a first step in this new direction, we embark on exploring the learning benefits of monosemanticity beyond interpretability, supporting the long-standing hypothesis of linking interpretability and robustness. Code is available at \url{https://github.com/PKU-ML/Monosemanticity-Robustness}.

Published as a conference paper at ICLR 2025
BEYOND INTERPRETABILITY. THE GAINS OF FEATURE MONOSEMANTICITY ON MODEL ROBUSTNESS
Qi Zhang! Yifei Wang²*
Jingyi Cui"
Yisen Wang Xiang Pan
Qi Lei'
Stefanie Jegelka
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
2 MIT CSAIL
3 New York University
TUM CIT, MCML, MDSI
5 MIT EECS, CSAIL
6 Institute for Artificial Intelligence, Peking University
ABSTRACT
Deep learning models often suffer from lack of interpretability due to polyse- manticity, where individual neurons are activated by multiple unrelated semanties, resulting in unclear attributions of model behavior. Recent advances in monose- manticity, where neurons correspond consistent and distinct semantics, have significantly improved interpretability but are commonly believed to compro- mise accuracy. In this work, challenge the prevailing belief of the accuracy- interpretabilityy tradeoff, showing that monosemantic features not only enhance interpretability but also bring concrete gains in model performance of robustness- related tasks. Across multiple robust learning scenarios including input and label noise, few-shot learning, and oft-of-domain generalization-- results show that models leveraging monosemantic features significantly outperform those relying polysemantic features. Furthermore, provide empirical and theoretical un- derstandings on the robustness gains of feature monosemanticity. Our preliminary analysis suggests that monosemanticity. by promoting better separation of feature representations, leads to more robust decision boundaries under noise. This di- verse evidence highlights the generality of monosemanticity in improving model robustness. As first step in this new direction, we embark on exploring the learning benefits of monosemanticity beyond interpretability, supporting the long- standing hypothesis of linking interpretability and robustness. Code is available https: //te::///iiee thub. PKEPPKUMMMMM MMOO-MMMMMMo.oommatiiiio Robusstess...
INTRODUCTION
A long-standing problem of deep learning is the so-called "black-box" nature. People find that an important factor for its lack of interpretability is feature polysemanticityy where a single neuron (a dimension of feature maps) is activated by multiple irrelevant semantics (Arora et al., 2018; Olah al., 2020), preventing clear attributions of neural behaviors. Following this understanding, recent research has made breakthroughs towards attaining monosemanticity, i.c., neurons corresponding to consistent semantics (monosemantic), which dramatically improves model interpretability; see an illustrative example on polysemanticity and monosemanticity in Figure 1(a). They achieve this through architectural designs (Elhage et al., 2022a; Wang et al., 2024) post-training explanation modules (Cunningham et al., 2024), and have successfully sealed to visual backbones (e.g.. ResNet) and large language models (LLMs) (e.g.. Claude, GPT, and Gemma), discovering many intriguing phenomena and applications (Templeton, 2024; Gao et al., 2024; Lieberum et al., 2024).
However, these works on monosemanticity suggest an inevitable "accuracy-interpretaalll tradeoff: monosemantic features, although more interpretable, come at the sacrifice of expressive power and
"Equal Contribution.
I "orresponning Author: Yisen Wang (yisen.wang
wang @ www.edu..n............

---

Published as a conference paper at ICLR 2025
66.866 266.7
Paly (SAF)
Valldation Accuracy
Mo no (NCU)
Messeraar
48.1
45.9
40
34.4
7 ,006
30
20
Clean
Label Noise
Input Noise
(a) Illustration of activated samples Polysemantic (Left) and (b) Test accuracy (%) of classifiers learned Monomsemantic (Right) dimension
upon Polysemantic and Monosemantic fea- tures on different scenarios
Figure 1: A comparison between Polysemantic (Contrastive Learning (CL)) and Monosemantic features (Non-negative Contrastive Learning (NCL), Sparse Autoencoder (SAE)) pretrained on ImageNet- 100. We consider noisy labels (90% noise rate) and Gaussian input noise (0.6 stdev); see more details in Appendix A.4.
underperform polysemantic features at prediction accuracy. For example, Gao et al. (2024) observe that the sparse autoencoders significantly enhance the monosemanticity of language models while obtaining inferior performance in downstream prediction tasks. Elhage et al. (2022b) show that monosemantic representations indicate lower model performance in reconstruction tasks with a toy model. Cunningham et al. (2024) exhibit negative correlation between the expressive power and monosemanticity. This widely accepted belief limits the applications of monosemanticity techniques to only enterepretability-eelae
domains. In this paper, we aim to push this boundary one step forward by demonstrating that monosemanticity can also bring significant gains on practical model performance beyond interpretabilityy
In particular, we discover widely appearing phenomenon, that monosemantic features are much more robust compared to polysemantic features, across multiple scenarios related to "robustness". One such scenario is learning with noise. Real-world data often imperfect with low-quality input and mislabeling, manifested in the form of various data noises and distribution shifts. We find that under either input label noises, learning classifier upon (pretrained) monosemantic features attain much higher accuracy (e.g., +13.7% top-1 accuracy under 90% label noise) than polysemantic features, shown in Figure 1(b). This feature-centric result also offers new perspective noisy learning where existing studies primarily focus on robust learning objectives (Wang et al., 2019a; Song et al., 2020; Ma et al., 2020).
The second secenario is few-shot finetuning for downstream classification. Today's large visual backbones often need to be finetuned on a small amount of downstream labeled data, where models easily overfit and deteriorate. We find that monosemantic finetuning. i.e., preserving the monose- manticity of representations during finetuning (with a technique from Wang et al. (2024)), can attain much higher accuracy under few-shot data compared vanilla polysemantic finetuning (e.g., +3.9% top-1 accuracy with 10% samples). The same method also works for finetuning with noisy data or training from scratch.
With these benefits in mind, we further explore a third scenario, LLM finetuning, which receives wide applications these days (Minaee et al., 2024). Pretrained LLMs need to be carefully finetuned on small-scale language data for different purposes, e.g., instruction following and certain abilities (e.g., reasoning), while avoiding conflicting and forgetting. Since LLMs do not have natural representation space like visual models, we devise a simple sparse variant of LoRA (an efficient tuning method), named MonoLoRA. to encourage the monosemanticity of the updates of all features. We show preliminary evidence that when finetuning an aligned LLM (Llama-2-7b-on (a classification task) and Dolly (instruction following task), MonoLoRA better preserves model alignment while improving task performance.
on SST-2
At last, we provide a deeper understanding of the robustness gains of monosemanticity. Empirically, compare the salient features of different classifiers, observing that the more robust classifiers tend to depend on more monosemantic features. Theoretically. as a preliminary step, we compare polysemantic and monosemantic features under toy model proposed in Elhage al. (2022b). Our
2

---

Published as a conference paper at ICLR 2025
analysis suggests that monosemantic features, due to their better separation, are less susceptible to overfitting to noise, resulting in more robust decision boundaries than polysemantic features.
In summary, this work challenges the common "accurayy-inerrretabiltt"" tradeoff by demonstrating the potential of feature monosemanticity to bring clear gains in model accuracy of robustness- tasks. These gains manifest themselves in various aspects of "learning robustness" that we can think of: input noise, label noise, out-of-domain data, few-shot image data, and few-shot language data, The diverse set of evidence strongly indicates that feature monosemanticity provides general sense of robustness compared polysemantic features, echoing with the long-lasting hypothesis on the relationship between better feature interpretability and better robustness (e.g., human decisions are both interpretable and robust) (Bengio et al., 2013; 2019). As a first step in this direction, we believe that it will embark on more intriguing discoveries and understandings on the learning benefits of monosemanticity beyond interpretability.
2 PRELIMINARY & RELATED WORK
Polysemanticity and Superposition Hypothesis. Across various domains, many previous studies (Nguyen et al., 2016; Mu & Andreas, 2020; Olah et al., 2020) have consistently observed that a feature dimension in neural networks is usually activated with multiple unrelated semantics. Researchers define this phenomenon as the feature polysemanticity. In contrast, when each dimension is activated with a single latent natural concept, the features are denoted as monosemantic features. A popular explanation of the feature polysemanticity is the superposition hypothesis (Arora et al., 2018; Olah et al., 2020), which states that each polysemantic dimension is an approximately linear combination of multiple natural concepts. To verify that, Elhage et al. (2022b) propose a toy model that obtains polysemantic features with the superposition hypothesis. Comparing polysemantic and monosemantic features, there exists a common belief that monosemantic features exhibit better interpretability the cost of downstream performance (Cunningham et al., 2024; Elhage et al., 2022b). However, in this paper, we challenge this trade-off, finding that monosemantie features also show superiority when the performance is evaluated on robustness tasks.
Methods to Attain Feature Monosemanticity. To enhance the feature interpretabilityy researchers propose several methods to obtain monosemantic features. For example, Variational Autoencoder (VAE) (Kingma, 2013) and its variants (Higgins et al., 2017; Chen et al., 2018) have been used to find the disentangled features with monosemantcity. However, the performance of these methods in real-world tasks like image classification and natural language understanding is quite unsatisfactory. Recently, researchers have tried to attain monosemanticity with minimal influence on performance. The approaches can be majorly divided into two categories (Bereska & Gavves, 2024): intrinsic and post-hoc methods. The intrinsic methods, represented by non-negative contrastive learning (Wang et al., 2024), focus on adjusting the pretraining algorithms. While the post-hoc methods apply downstream modifications on learned features. For example, the sparse autoencoder, which reconstructs the features from a sparse bottleneck layer, has recently shown impressive monoseman- ticity in various models (Ng et al., 2011; Gao al., 2024). Different from existing works, our paper 1) analyzes the monosemanticity from a totally new perspective, i.e., we observe the relationship between monosemanticity and robustness; 2) challenges the widely-believed interpretability-an accuuacy trade-off in realistic and widely appearing scenarios; 3) introduces a new theoretical perspective to analyze the monosemanticity; 4) proposes new implementations to enhance monomsenaticity in large language models.
Robustness Learning. In the development of deep learning models, robustness is critical measure for evaluating the quality of features (Wang et al., 2021; Xu et al., 2021). The evaluation of robustness involves various task scenarios. Common conditions include assessing the robustness of features against noisy labels (Song et al., 2022; Ma et al., 2018: Wang al., 2018), distribution shifts (Yang et al., 2024; Wang et al., 2022; Xin et al., 2023), overfitting (Ying, 2019; Wang et al., 2019b), etc. In this paper, we analyze the robustness from a new perspective, i.e., we evaluate the influence of monosemanticity in different robustness tasks. For learning with noisy labels, we apply the symmetric label noise to the training samples, i.e., with a probability 17 (noise rate), the labels of samples are uniformly flipped to the other classes. For robustness against distribution shifts, we apply various shifts, such as Gaussian noise, uniform noise, and real-world distribution shifts (Wang et al., 2019a; Geirhos et al., 2018) to the validation samples. For robustness against overfitting, we finetune the vision and language models with fewer samples and evaluate the validation performance.
3

---

Published as a conference paper at ICLR 2025
3 THE ROBUSTNESS GAINS OF MONOSEMANTICITY
In this section, we compare polysemantic with monosemantic features across three different robust learning scenarios commonly encoutered in the foundation model regime: first, noisy linear probing on pretrained features (either polysemantic or monosemantic); second, noisy and few-shot finetuning from pretrained weights; third, finetuning LLMs on small-scale supervised data.
3.1 MONOSEMANTIC FEATURES ARE ROBUST UNDER LINEAR PROBING
Foundation models typically have two training phases: 1) self-supervised learning (SSL.) on massive unlabeled data, and 2) supervised finettninn on small numan-labeled data (classification, instruction following, or specific tasks). In fact, since SSL-pretrained features contain rich semantics, learning a simpler linear classifier on top, known as linear probing (LP), can often attain competitive performance fully supervised ones (Chen et al., 2020). Therefore, start with this simplest setting for comparing the robustness of polysemantic and monosemantic pretrained features. Specifically, we consider standard linear probing setting, where we first pretrain features on unlabeled data and then learn a linear classifier on top with noisy labeled data.
3.1.1 METHODS FOR FEATURE MONOSEMANTICIT
Among existing interpretability research, there ticity: 1) intrinsic methods, where pretrained features are intrinsically monosemantic; 2) post-hoc methods, where we apply additional techniques decode (polysemantic) pretrained features to monomsemantic ones. Here, we consider two representative methods for each paradigm.
categories of methods attain monoseman-
Intrinsic Monosemanticity with NCL. Many previous works have tried to train interpretable features by adding sparsity regularization (Tibshirani, 1996) or identifiability constraints (Zhang et al., 2024); but they hardly scale to large-scale data with competitve performance. A recent work, NCL (non- negative contrastive learning) (Wang et al., 2024), as a modern counterpart to NMF (non-negative matrix factorization) (Lee & Seung, 1999). attains high sparsity and monosemanticity while having minimal influence on final performance. Specifically, NCL adopts the following InfoNCE loss (Oord et al., 2018) with non-negative feature outputs:
exp(f+ f
(xC = EEr,z1 log
(I)
(xp(f+(=)TT++=))))
1 M EM (x](+(()TTf(=))
where (x, ) xxxx") are the positive and negative pairs in contrastive learning, fx = (f))) a is an activation function and f is the original neural network. With the non-negative constraints, the activations of learned representations become sparse and each dimension is almost only activated with samples from the same class (Wang et al., 2024).
Post-hoc Monosemanticity with SAE. Another approach is to apply downstream modification on pretrained neural networks. Sparse autoencoders (SAEs) (Ng et al., 2011) find wide success in attaining monosemanticity in language models (Templeton, 2024; Gao et al., 2024; Lieberum et al., 2024). SAFS reconstruct the original outputs of pretrained networks from a sparse bottleneck layer. To be specific, the encoder and decoder are defined as:
2(x) = topK((W (f - bpre)),
(2)
f(x) (x) + bprer
where f(x) is the representation of input .; Wene. Wdee- bpre and bene are the parameters of SAE: topK is a sparse activation function proposed by Gao et al. (2024) that only preserves the top K elements; and the SAE training loss is the reconstruction of original features, i.e., LSAF (z)lf(z)) 12 As a result, the sparse latent feature 2(x) has much better monosemanticity than the original feature fxx)
Evaluation of Monosemanticity. We evaluate the monosemanticity of different models with current metrics. Specifically, we follow Wang et al. (2024) and adopt semantic consistency as the metric. The semantic consistency calculates the proportion of activated samples that belong to their most frequent class along each dimension. We evaluate the monosemanticity of different models, including original models, models trained by NCL, and models with SAES. As shown in Table 1, both NCL and SAE significantly improve the semantic consistency of trained models.
4

---

Published as a conference paper at ICLR 2025
Table I: Semantic consistency of different models. The semantic consistency calculates the proportion of activated samples that belong to their most frequent class along each dimension.
Models
Poly (CL)
Mono (NCL)
Mono (SAE)
Linear Probing on CIFAR-100 Linear Probing on Imagenet- 100
1.0
8.2
1.0
12.3
7.2
Poly (CE)
Mono (NCH)
Fine-tuning with 20% Samples Fine-tuning with 20% Noisy Lables
2.1
18.4
3.4
20.6
Table Linear probing accuracy and gain (%) of polysemantic and monosemantic representations on ImageNet- 100 and CIFAR-100 under different rates (%) of label noise (0 (clean label) to 90).
Dataset
Features
60
90
Poly
5..6.10.2
53.2:10.2
52.11:.4
49.5
0..
Mono(SAE)
54.5:10.1
53.9:1.22
53.0+10.3
50.611.4
38.0=
CIFAR-100
-0.1
+0.7
+0.9
+1.1
+2.6
Mono(NCL))
52.8+0044
54.2++0.3
52.7+0.2
51.5++0.2
45.0=
-1.8
+1.0
+0.6
+2.0
+9.6
Poly
66.810.22
63.310.2
60.1
54.910.4
3..
Mono(SAE)
66.1
65.2
60.7+0.2
58.610.11
45.7=0.2
ImageNet- 100
-0.7
+1.9
+0.6
+3.7
+11.2
Mono(NCL))
66.8+0.2
65.5+0.11
63.8+0.22
60.8+0.2
48.1
+0.0
+2.2
+3.7
+5.9
+13.6
3.1.2 EXPERIMENTS
Setup. For the baseline, pretrain ResNet-18 18 (He al., 2016) backbone with the widely contrastive framework SimCLR (Chen et al., 2020) on CIFAR-100 and ImageNet- 100. In comparison, use Non-negative Contrastive Learning (Wang et al., 2024) and Sparse Autoencoder (Gao et al., 2024) to represent two primary strategies for obtaining monosemantic features, i.c.. improve the pretraining algorithm and apply downstream modification. For Non-negative Contrastive Learning (NCL),, we follow the default SimCLR settings, with the addition of a non-negative constraint using the ReLU function. For the Sparse Autoencoder (SAE), we apply it following the pretrained backbone as Equation (2), and then we train the linear classifier on the frozen latent representation of SAE. More details can be found in Appendix A.1.
Robustness Against Label Noise. When evaluating the robustness against label noise, train linear classifier following the frozen pretrained encoders, where the labels are uniformly flipped to the other classes with a probability 7/ (noise rate). As shown in Table 2, when the linear classifiers trained on the samples with clean labels, monosemantic and polysemantic features exhibit comparable performance. However, in the presence of label noise, both NCL and SAE significantly outperform across various datasets. Especially, when the noise rates are aggressive, the improvements are substantial, with NCL showing 13.7% improvement on ImageNet- 100 and 9.2% improvement CIFAR- 100 under 90% noisy labels. The results are consistent with the results in toy models and further verify that monosemnatic features obtain stronger robustness against label noise.
Robustness Against Distribution Shifts. For evaluating the resilience features to distribution shifts, we evaluate three types of shifts, including random input noise, random Gaussian noise, and real-world distribution shifts (Wang al., 2019a; Geirhos et al., 2018; Hendrycks & Dietterich, 2019) on ImageNet- 100 datasets. The models and classifiers are trained on the clean ImageNet- 100 dataset while their classification performance is evaluated with noisy samples. As shown in Figure 2(a), 2(b), and 2(c), both the pretraining constraints and downstream modifications that enhance feature monosemanticity improve classification accuracy under noisy samples, and the benefits rise with the increase of noise strength. The results suggest that the monosemantic features can also enhance the robustness against various noises applied in inputs.
5

---

Published as a conference paper at ICLR 2025
6D
Paly (CL)
Mono Mono (MCL)
Velidatoon Acturacc
66
Poly (CL)
(NCL)
(SAE)
veldation Kcrra
22.5
35.0
30.0
Poly (CL)
Mono (SAD)
$0
85
27.5
Velidation
64
25.0
8.
22.5
62
20.0
0.2 Uniform Noise Strungth
Skotch
Stylacd Dataset
Comrptionn
Caussian Strength
(a) Gaussian Input Noise
(b) Uniform Input Noise
(c) Real-world Distribution Shift
Figure 2: The evaluation of robustness against input distribution shifts on ImageNet- 100. Monose- mantic representations (SAE, NCL) exhibit improvements in the robustness against different kinds of distribution shifts.
3.2 MONOSEMANTIC FEATURES ARE ROBUST UNDER FEW-SHOT AND NOISY FINETUNING
In practice, fully finetuning a large pretrained model on downstream labeled data can often achieve better performance than linear probing, but it also easily overfits if there is only few amount of labeled data. Here, we compare standard finetuning (polysemantic) to monosemantic finetuning.
3.2.1 METHOOSS FOR MONOSEMANTIC FINETUNING
Standard Finetuning. For the baseline, we consider a common finetuning setting, i.c., we pretrain the encoders with contrastive learning unlabeled mag and then learn a linear classifier on
exp(f((a) my) labeled Imagenet-100 100 with the cross-entropy loss: CCE(f) = Ex.y log 5e exp(f(x)T where f is the encoder network and w, is the linear classifier weight of the related label. Unlike linear probing, we train classifiers on the pretrained representations without clipping the gradient of encoders.
Non-negative Tuning. According to NCL (Wang al., 2024), replacing the original cross-entropy (CE) loss used in the supervised learning process with the non-negative cross-entropy (NCE) loss will maintain monosemanticity during supervised learning. Thus, strategy. To be specific, NCE applies non-negative transformation to the representations fxx), i.e.,
it as monosemantic finetuning
(NCE(f) Ez,,y log exp(f+(x) Twy)
Eo=1 exp(f+(x)Tuu) w.)
where /+(x) (f(x)) with non-negative activation function, e.g., ReLU. By respectively finetuning contrastive pretrained models with CE and NCE objectives, we compare the robustness of polysemantic and monosemantic features across two different tasks: few-shot finetuning and noisy label finetuning.
Evaluation of Monosemanticity. We still measure the monosemanticity of different models by the semantic consistency (Wang et al., 2024). We evaluate the models different cases, including models fine-tuned with label noise and few-shot samples. As shown in Table 1, non-negative tuning significantly improves the semantic consistency of trained models.
3.2.2 EXPERIMENTS
Few-shot Finetuning. As the finetuning process usually involves fewer training samples, a crucial challenge for feature robustness is preventing overfitting small training datasets. To evaluate the performance polysemantic and monosemantic features during few-shot finetuning, we respectively use 10%, 20%, 50% and the entire training set of ImageNet- 100 to finetune the pretrained repre- sentations with CE and NCE objectives. As shown Figure 3(a), 3(b), the monosemantic features exhibit lower training accuracy but higher validation accuracy in few-shot finetuning, and the advantages grow when the training set becomes smaller, which implies that the monosemanticity helps representations to be less likely to overfit the training set in the downstream task.
Noisy-label Finetuning. We also evaluate robustness against label noise in finetuning tasks on ImageNet- 100. During the finetuning process, the labels of training samples are uniformly flipped to the other classes with probability 7) (noise rate). As shown in Figure 3(c), non-negative finetuning

---

Published as a conference paper at ICLR 2025
26.3
92
Mono
(C1)
Mono (NCE)
72.2
71.
156.661
Mallaaai Accuraay
13.4
2AD
63.7
#5
60
8B
0R.5
04
60
$2.5
500
163
80
$1.5
40
Poly (CE)
10 20
50
100
10
20
50
1000
20
40
60
Percentage Training Samples (%)
Percentage
Training Samples (%)
Noiee Rate
(a) Training Accuracy of Few-shot (b) Validation Accuracy of Few-
(c) Noisy Label Finetuning
Finetuning
shot Finetuning
Figure 3: The robustness of the models finetuned with polysemanticity (CE) and monosemanticity (NCE) under different noises on ImageNet- 100. Attaining monosemanticity during the finetuning process enhances the robustness across various tasks.
leads to significant gains under label noise that keep growing with the increase of the noise rate. Notably, monosemantic features exhibit at most 11.9% improvement under large noise rate.
These empirical results indicate that maintaining feature monosemanticity during the finetuning process can bring better learning robustness against overfitting and label noise.
3.3 MONOSEMANTIC LORA FOR LARGE LANGUAGE MODELS
When evaluating, consider common scenario related to robustness in large language model fine-tuning. Specifically, during the fine-tuning process, the large language models often compromise the already learned alignment, which leads security risk (Qi et al., 2023; Li et al., 2025). In practice, we use the Llama-2-7B-Cnn
(Touvron al., 2023) as the aligned model and further finetune it with SST2 (Socher et al., 2013) as the review sentiment classification task and Dolly (Conover et al., 2023) datasets as the dialogue generation task. To evaluate the security and alignment performance, we use the ShieldGemma- 9B (Zeng et al., 2024) and Beavertails- (Ji et al., 2024) models to evaluate the alignment of model responses based on the response on Beavertails datasets (Ji et al., 2024). More details of the parameters and the metrics can be found in Appendix A.3.
Table 3: Evaluation of LoRA and MonoLoRA with (-ama-2-7BBCLL fine-tuned SST2 and Dolly datasets. Task performance measured by accuracy (SST2) and Rougel. score (Dolly) (Lin, 2004). Alignment performance is assessed based responses the Beavertails dataset.
Dataset
Model
ShieldGemma Alignment Scores (4)
Aligament
Task
Beavertails Scores
Task Perf.
Spursity
Sparsity
(4)
(t)
Danger.
Harass.
Hate.
Sex.
Avg
Base
7.66
2,88
6.14
2.64
4.83
20.90
88.65
SST2
LoRA
8.48
6.91
6.77
92.78
MonoLoRA
5.37
2.23
4.63
1,88
3.53
45.54
36.71
20.00
94.84
Base
7.66
2.88
6.14
2.64
4,83
20.90
10.21
Dolly
LoRA
10.54
3.53
7.53
2.86
6.12
23,80
14.08
MonoLoRA
10.49
3.56
7.40
2.70
6.04
38.69
40.00
22.60
14.48
As shown in Table 3, MonoLoRA models exhibit greater alignment resilience compared to standard LoRA models while maintaining comparable fine-tuning performance. To assess monosemanticity; we adopt sparsity (zero-value ratio) of intermediate activations as a surrogate metric (Gurnee et al., 2023) and evaluate it the Beavertails (Alignment Sparsity), SST2 (Task Sparsity), and Dolly (Task Sparsity) datasets. Higher sparsity indicates stronger monosemanticity. Our results suggest that enhancing monosemanticity at the neuron level improves the robustness of LLMs, mitigating the risk of compromising learned capabilities when fine-tumed with downstream data.
4 UNDERSTANDING THE ROBUSTNESS GAINS OF MONOSEMANTICITY
In Section 3, we provide comprehensive evaluation of the robustness gains feature monoseman- ticity across multiple scenarios. Yet, do not have fully clear understanding of why monosemantic
7

---

Published as a conference paper at ICLR 2025
22.48
Monos Momosemantican Score of the Most Salunt Frstuire
Cisaal Sumpla
16. 32
14.90
10
CIFAR-100
ImageNet 100
Method
(a) The most salient feature of the lowest-accuracy class is polyse- mantic.
(b) The most salient feature of the highest-accuracy class is monose- mantic.
(c) Correctly classified samples have more monosemantic features.
Figure 4: Influence of feature monosemanticity on classification performance, where the classifier is applied after frozen contrastive encoder and trained with 90% noisy labels. (a), (b) respectively draw the activated samples on the dimensions with the largest clssifier weight of the lowest-accuracy and highest-accurace classes ImageNet 100. (c) demonstrates the monosemanticity scores (Wang et al., 2024) of wrongly and correctly classified samples.
features are more robust. As preliminary step demystify this phenomenon, in this section, investigate the influence of monosemanticity on learned classifiers from both empirical (Section 4. and theoretical (Sections 4.2 & 4.3) perspectives. For simplicity, focus the label noise scenario.
4.1 NOISY CLASSIFIERS PREFER MONOSEMANTIC FEATURES IN PRACTICE
To further understand the robustness improvements brought by monosemanticity, we investigate the difference in the salient features of the robust and non-robust classifiers under noisy conditions. Taking the linear classifier trained on ImageNet- 100 with 90% noisy labels (Section 3.1)) as an example, start with respectively visualizing the dominant features for classes with the highest and lowest accuracy. For each class, we find the feature dimension with the largest classifier weight for the ground-truth label and visualize the top-activated samples along the dimension. As shown in Figure 4(a), 4(b), we observe clear difference: samples activated in the dimension related to the lowest accuracy class (jeans) belong different classes while samples activated in the dimension related to the highest accuracy class (boathouse) share the same label, i.e., the classifier with higher performance under label noise relies more monosemantic dimension.
We then validate this observation with the semantic consistency (Wang et al., 2024) as the quantitative monosemanticity score. The semantic consistency calculates the proportion of activated samples that belong to their most frequent class along a dimension. With larger semantic consistency, the dimension is more likely to be activated by the samples from the same class, i.c., the feature is more monosemantic, To compare the robust and non-robust classifiers, we respectively draw the samples that are wrongly and correctly classified by the classifiers learned on ImageNet- 100 with 90% noisy labels. For the embedding of each sample, we draw the dimension with the largest activation value and calculate the semantic consistency. As shown in Figure 4(c), we observe that the semantic consistency of the most salient features in correctly classified samples is much higher than that of misclassified samples. The results further indicate that the classifiers with superior performance under noise tend to depend on monosemantic features.
4.2 REPLICATING MONOSEMANTICITY GAINS WITH THE SUPERPOSITION MODEL
To further establish a theoretical understanding of the robustness benefits brought by monosemanticity. we introduce a toy model proposed by Elhage et al. (2022b) for the simplicity of analysis. The toy model constructs polysemantic representations with the superposition hypothesis (Arora et al., 2018), a widely-used explanation of feature polysemanticity. The hypothesis states that a polysemantic feature is an approximately linear combination of multiple latent semantics while a monosemantic feature is the reconstruction of single natural concept. With the hypothesis, the toy model enables researchers to replicate the polysemanticity phenomenon and theoretically analyze the properties of polysemantic and monosemantic features, e.g., occurrence conditions, learning dynamics, and
8

---

Published as a conference paper at ICLR 2025
Monosermantic Representation
-1.0
Polysemantic Representation
-0.3
0.6
0.58
Mono
0.50
Poly
Motatto Accurso
36322282420
36322822220 363222824200
D.31
0.3
0:26
-0.1
0.2
0.19
0.16
-=1.0
--0.3
omo RARNNNA mm
Ocan
Label Noise
Input Noiso
(a) Polysemantic dimensions correspond multiple latent semantics.
(b) Polysemantic features have worse performance under noisy data.
Figure The comparison between polysemantic and monosemantic features on the toy model introduced by Elhage et al. (2022b) (n = 40, m 20, S = 0.2). (a) demonstrates the Parameters (WTW) of monosemantic (Left) and polysemantic features (Right) on the Toy Model. (b) evaluates the classification performance of features against different noises. The label noise denotes applying 90% noisy labels the training samples and input noise denotes applying Gaussian noise to the validation samples.
geometric structures (Lecomte al., 2024; Marshall Kirchner, 2024; Chen et al., 2023). In this section, we start by introducing the setups and observing the robustness of different features on the toy model.
Toy Model Setups. In practice, we follow the settings proposed by Elhage et al. (2022b) and evaluate the robustness of polysemantic features the toy model. Specifically, assume each sample has n dimensions and each dimension represents natural concept. As the features real-world datasets usually sparsely activated (Olshausen & Field, 1997). we assume each dimension of with probability S. If not zero, we let each
sample æ has an associated sparsity and let æ; dimension be uniformly distributed between [0, 1]. When evaluating the performance,
consider the classification tasks of natural concepts, i.c., the labels satisfy that y(x) = arg max Z; For the
encoding network. we consider a linear model = Wa, where W E R with m < n, i... the hidden dimension is smaller than the input dimension. In practice, we use the reconstruction of as the training objective and obtain two kinds of learned features. As shown in Figure 5(a), when the superposition does not occur, we observe that W W is diagonal and has only non-zero elements, which means the model only captures m concepts and each dimension is monosemantic, In contrast, when superposition happens, features obtain more concepts than the model dimensions and different concepts are projected into the same dimension.
Noisy Learning Settings. To evaluate the robustness of features, we respectively add noise to the labels and samples. For training with noisy labels, denote the noise as 7), where each label uniformly switched to one of the other an aggressive noise rate (90%). With labeled samples, train linear classifier following the frozen features and evaluate the classification accuracy on validation dataset without noisy labels. For noisy sample validation, train linear classifier the clean dataset and add the Gaussian noise to the validation set samples.
labels with probability 1). In experiments, we selected
As shown in Figure 5(b), in the absence of noise, polysemantic features exhibit better performance, which expected as the superposition enables features capture more concepts. However, when there exists noise in the labels and samples, the situation changes significantly. The feature without superposition shows improvements over that with superposition under both label noise and input noise. The empirical results replicate the phenomenon where the monosemantic features are more robust than polysemantic features.
Empirical Results.
4.3 THEORETICAL ANALYSES WITH THE SUPERPOSITION MODEL
After replicating the robustness gains monosemanticity on the toy model, we then establish theoretical comparison between polysemantic and monosemantic features. For ease of theoretical analysis, consider binary classification case in the toy model (n 2, 1, 0.2). To be specific, a sample x has two latent features I1, D2, and the model parameter W E IR 1x2 When we obtain the monosemantic features, the model output is Vnonno = Ø1 In contrast, when obtaining

---

Published as a conference paper at ICLR 2025
polysemantic features, the model keeps more natural concepts than the representation dimension. According to Elhage et al. (2022b), one common geometric structure of polysemantic features is antipodal pairs formed by two concepts. Therefore, we assume the learned polysemantic feature to be Uppoly = 11 2-2-
For conciseness of expressions, we introduce the following notations on mean and variance for a given feature representation v. For a clean distribution without label noise, we denote the conditional means and variances by (v) == E(vly = i), (v) := [((2 (o())) i), = 0, 1. For distinction, for a noisy distribution, we use já and Borrowing the concept from linear discriminant analysis (LDA) (Fisher, 1936), we deem that good linearly discriminative representation should have a large distance between different classes whereas maintaining the intra-class variance as small as possible, i.e. maximize Aμ(v)) = [P)()) μi (v) whereas minimizing (v) and of (v). Therefore, to quantitatively compare polysemantic and monosemantic representations,
use the criterion J(V) = Ap(r)/(oo(P)os (v)). A larger value of (2) indicates better linear separability.
Theorem 4.1 (Conditional means and variances of monosemantic & polysemantic features). Let Vmmano = and Vplly Iι - I2. For conditional means, we have Ho(Vpoly) < (o(umana)
and Mu < f1 (Vicooo)), yet Ap(Vpoly) > Ap((moono). For conditional variances, we have (P = (M and of (M > og ("mono)) Overall, we have J [Vpoly) > J (Pmono)-
According to the LDA criterion, the polysemantic feature with a larger J(v) is more linearly separable. Intuitively, because the polysemantic embedding encodes information of both æ1 and T2, it can do better classification w.r.t. the labels depending on both features. However, when there exits label noise, we observe a different situation.
Theorem 4.2 (Influence of label noise linear seprarability). We denote the linear separability criterion under noise as J(v) ( For noise rate 1) [0, 0.5),
](pooly) [[vrooo)
S (Vmono) S 1.
(4)
J [Vpolt)
Meanwhile, we obtain J(ppoty) S (Vmoona) when 17 E [0.25, 0.5).
As shown in Theorem 4.2, with the increase of noise rate, the linear separability (J()) of both polysemantic and monosemantic features becomes worse. However, J (Vmmono) decreases more slowly. As a result, when the noise rate is aggressive enough (17 2 0.25), the monosemantic feature exhibits better linear seperability than the polysemantic one. Moreover, in Appendix B.3, we show that input noise has similar influence on linear separability. The theoretical results reveal that the linear separability of monosemantic features is more robust than polysemantic ones in the toy model settings, which leads better performance in tasks under noise. As a preliminary step, we believe that the examples can inspire further theoretical analysis of the relationship between monosemanticity and model robustness in more realistic situations, such as fine-tuning scenarios.
CONCLUDING REMARKS
Recent work has made significant strides in enhancing model interpretability by promoting feature monosemanticity through various techniques. However, prevailing belief in the literature posits an accuracy- interecctittrrretabbltty tradeoff, suggesting that achieving monosemantic features for better inter- pretability necessarily compromises prediction accuracy. In this study, we have challenged this notion by demonstrating the advantages of monosemanticity beyond interpretability alone. Specifically, we found that monosemantic features are significantly more robust to various types of distribution shifts, including input noise, label noise, and real-world out-of-domain inputs. Additionally, we have shown that maintaining feature monosemanticity during fine-tuning serves as an effective regularizer, reducing model overfitting in few-shot settings, noisy environments, and during large language model (LLM) fine-tuning. We also provide an in-depth analysis of the benefits of monosemantic features from both theoretical and empirical aspects. These diverse sources of learning robustness collectively indicate that monosemantic features have general sense of robustness, resonating with its benefits in interpretability. Therefore, rather than viewing monosemanticity as a necessary cost for interpretability, we advocate for embracing and exploring the multiple learning advantages it offers. We believe our work, as a pioneering effort in this direction, will inspire future research to investigate these possibilities further.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we elaborate on the details of our experiments and theoretical analysis in the main paper and the appendix. In Section 3.1, 3.2, 3.3 of the main paper, we respectively introduce the methods for capturing polysemantic and monosemantic features in linear probing, finetuning vision models and finetuning LLMs. Furthermore, in Appendix A, we introduce the hyperparameters and implementation details of adopted methods, and the detailed settings of the robustness evaluation, including input and label noise, few-shot learning, and out-of-domain generalization. For theoretical results, we introduce the toy models Section 4.3 of the main paper and provide detailed proofs and explanations for the theoretical comparison in Appendix B.
ACKNOWLEDGEMENT
Yisen Wang was supported by National Key R&D Program China (2022ZD0160300), National Nat- ural Science Foundation of China (92370129, 62376010), and Beijing Nova Program (20230484344, 20240484642). Yifei Wang and Stefanie Jegelka were supported part by the NSF AI Institute TILOS, and an Alexander von Humboldt Professorship.
REFERENCES
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pultern Analysis and Machine Intelligence, 35(8):1798-1822 2013.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv: 10912, 2019.
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety-a review. arXiv preprint arXiv:2404.1 14082, 2024.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, 2019.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NeurIPS, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet. Dynamical versus bayesian phase transitions in toy model of superposition. arXiv preprint arXiv:2310.06301, 2023.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tune 1lm. Company Blog of Databricks, 2023.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. ICLR, 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer EIShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022a.
11

---

Published as a conference paper at ICLR 2025
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatiield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprini arXiv:2209. 10652, 2022b.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7 (2):179-188, 1936.
Lco Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint S5t::2406.04093, 2024.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 1.12311, 2018.
Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In AAAI, 2017.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.1 12.611,,,,21 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Jammnn Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Cc Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of IIm via human- preann-reference dataset. In NeurlPS, 2024.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.00
Victor Lecomte, Kushal Thaman, Rylan Schaeffer, Naomi Bashkansky, Trevor Chow, and Sanmi Koyejo. What causes polysemanticity? an alternative origin story of mixed selectivity from incidental causes. In ICLR 2024 Workshop on Representational Aligument, 2024.
Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factoriza- tion. Nature, 401(6755):788-7999 791, 1999. doi: 10.1038/44565.
Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. SaloRA: Safety-alignment preserved low-rank adaptation. In ICLR, 2025.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once gemma 2. arXiv preprint arXiv:2408.05147. 2024.
Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, 2004.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-driven learning with noisy labels. In ICML, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In ICMI., 2020.
12

---

Published as a conference paper at ICLR 2025
Simon C Marshall and Jan H Kirchner. Understanding polysemanticity in neural networks through coding theory. arXiv preprint arXiv:2401. 17775, 2024.
Shervin Minace, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, arXiv preprint arXiv:2402.06196 2024.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In NeurIPS, 2020.
Andrew Ng al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19, 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv:1602.03666. 2016.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 5()):000224000,, 2020.
Bruno A Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research, 37(23):3311-322 1997.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv; 1807.03780 2018.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.0366 2023.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6653 2014.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A thcoretical analysis of contrastive unsupervised representation learning. In (CML,2019
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. arXiv preprint arXiv: 2007.08199, 2020.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, (4(11):8335-81) 2022.
Adly Templeton. Scaling monosemanticity: Extracting interpretuble feature from claude 3 sonnet. Anthropic, 2024.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In NeurlPS, 2015.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurlPS, 2019a.
Qixun Wang, Yifei Wang, Hong Zhu, and Yisen Wang. Improving out-of-distributiin generalization by adversarial training with structured priors. In NeurlPS, 2022.
13

---

Published as a conference paper at ICLR 2025
Xuezhi Wang, Haohan Wang, and Diyi Yang. Measure and improve robustness in nlp models: A survey, arXiv preprint 13X8::211...8813, 2021.
Yifei Wang, Qi Zhang, Yaoyu Guo, and Yisen Wang. Non-negative contrastive learning. arXiv preprint arX1v:2403.12459, 2024.
Yisen Wang, Weiyang Liu, Xingjun Ma. James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, 2018.
Yisen Wang, Xingjun Ma. Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019b.
Shiji Xin, Yifei Wang. Jingtong Su, and Yisen Wang. On the connection between invariant learning and adversarial training for distof-dist generalization. In AAAI, 2023.
Jiarong Xu, Junru Chen, Siqi You, Zhiqing Xiao, Yang Yang, and Jiangang Lu. Robustness of deep learning models graphs: A survey. Al Open, 2:69-78, 2021.
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized duittf-distributiin detection: A survey. International Journal of Computer Vision, pp. 1-28, 2024.
Xue Ying. An overview overfitting and its solutions. In Journal of physics: Conference series, volume 1168, pp. 022022. IOP Publishing, 2019.
Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, et al. Shieldgemma: Generative ai content moderation based gemma. arXiv preprint arXiv:2407.2177, 2024.
Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable contrastive learning with automatic feature importance discovery. NeurIPS, 2024.
14

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
A. EXPERIMENT DETAILS FOR NOISY LINEAR PROBING
During the pretraining process, utilize ResNet- 18 (He et al., 2016) the backhone and train the models on CIFAR- 100 and ImageNet-100. We pretrain the model for 200 epochs. The projector is a two-layer MLP with hidden dimension 16384 and an output dimension 2048. We train the models with batch size 256 and weight decay 0.0001. When implementing NCL and SAE, we follow the default settings of SimCLR. For NCL, we adopt ReLU as the activation function For SAE, the encoder and decoder are linear layers with 2048 input and output dimensions, and the number of activated features in the hidden layer is 256.
During the linear evaluation, we train a classifier following the frozen backbone pretrained by different methods for 50 epochs. For noisy label probing, we apply symmetric label noise when training the linear classifiers, ie., the labels are uniformly flipped the other classes with the noisy rate. And for random imput noise, we train the linear classifiers on clean datasets, while applying different scales of uniform noise and Gaussian noise the validation sample. For real-world out-of-domain distribution shifts, we use ImageNet-sketch and ImageNet-stylized datasets (Geirhos et al., 2018; Wang et al., 2019a). As we pretrain the network on ImageNet-100 100, we select the samples of the corresponding 100 classes from these out-of-distribution datasets and evaluate the accuracy.
A.2 EXPERIMENT DETAILS FOR FEW-SHOT AND NOISY FINETUNING FROM PRERTRAINED FEATURES
During the pretraining process, we utilize ResNet- 18 (He et al., 2016) as the backbone and train the models ImageNet-100) We pretrain the model for 200 epochs. We projector which is two-layer MLP with hidden dimension 16384 and output dimension 2048. We pretrain the models with batch size 256 and weight decay 0.0001. During the finetuning process, train classifier following the backbone for 100 epochs respectively with standard and non-negative tuning, following the default settings of finetuning. When implementing the onn-eiaaevve tuning, we select the ReLU function as the non-negative operator. For few-shot finetuning, we respectively random draw 10%, 20%, 50%, and 100$ training samples from the original ImageNet- 100 training set. For noisy label fine-tuning, we still apply symmetric label noise with different noise rates to the training samples.
A.3 EXPERIMENT DETAILS FOR MONOSEMANTIC LLM FINETUNING
Hlper-pprametern We finetune the Llama-2-7b-Chat model in SST2 with 20 epochs. batch size 16 and learning rate le-4. we Lora with rank r = 8, scaling factor a = 4, and dropout rate 0.1 as default. For Dolly, we finetune it epoch (by common practice), with batch size 4. The LoRA module is added to every query and value mapping module the base model. For finetuning we use 5000 samples from the dataset. For inference, we use 1000 samples.
Security Scores:
We use the ShieldGemma-99 (Zeng et al., 2024) and Beavertails-7B (Ji et al., 2024) models to evaluate the alignment of model responses based on the response on Beavertails datasets (Ji al., 2024). The Bevertails-7B and ShieldGemma-9D are sheild models which can detect the unsafe behaviors of models.
Prompt Template
Listing 1: SST-2 Prompt
Below is an instruction that describes task. Write response that appropri stely completes the reguest.
244 Instruction
Analyze the sentiment of the input, and respond only 'positive' or "negstive",
144 Inpul: [sentenco]
*tt Response:
15

---

Published as a conference paper at ICLR 2025
Listing 2: ShieldGemma Guideline
contain or genaration DE content that harning and/or others Io.g., secessing or building tirearns and explosive cevices, pronotion of tercorisn, instroctions for suicidel.
bangereas ConcentV*: The promot
vho Harasement (t; The prompt shall contair seek generation of content that is nal melicious, Largeling amother individual (e.g.r physical of tragic events, disparaging victins of violancel.
Intinidat ing, builying,
abusive
TH Bate Speccalf; The prompt shall not contain or seek goneration of centent that carestine ideatity and /er protected atcributes (o.g., rscizl slaze, praneticn et discrinination. colls to violence against wttp://litttiiii/n///nn.
protected
the gender, gender identity, other protessed status, He allew content that includes reference to protected individosl group in a poaitive nannez.
ethnicity, religton, any
dissbility, age, nacionality, veterar status,
\ho Saxually Explicit inforatiinn"; The prompt shall not contain or seek generacion of content that ceatains refarenees sexuti
descriptions, simed st causing arousall. Medical scienticic terns related to tunan anatony sex education are pernitted.
sexually
A.4 EXPERIMENTS DETAILS FOR FIGURE 1
For Figure I(a), respectively draw random dimension from the models trained by CL and NCL, and then draw the top-activated samples along two dimensions. We utilize ResNet-18 (He et al., 2016) as the backbone and train the models on ImageNet- for 200 epochs.
For Figure 1(b). we evaluate the performance in linear probing with noise. During the linear evaluation, we train a classifier following the frozen backbone pretrained by different methods for 50 epochs. For noisy label probing, we apply 90% symmetric label noise when training the linear classifiers. For random input noise, we train the linear classifiers on clean datasets, while applying Gaussian noise with 0.6 standard variation to the validation sample.
B PROOFS
B. PROOFS RELATED TO THEBREM 4.1
B.1.1 MONOSEMANTIC REPRESENTATIONS
In the monosemantic case, we assume the learned representation only keeps the most important dimension T1-
Theorem B.1 (Conditional mean and variance of monosemantic representations). The conditional means and variances of Vionn I are
(1 S)²
12 S
Ho (Ymono)
and 11 (1
(5)
3 1 52
31+ S
1 ( S)2
13+5
of (2 (mooo)
*M (Pmono)))) and (f(oooo))
1/1 (MMooo)))
(6)
6 1
6 61+
Proof of Theorem B.I. We first calculate the conditional probability density functions.
P( 5 x/y= = 0)
P(z1 S 20,01 x2)
P(r: S x2)
P(zi < &2, 54 < xx) +P) < 02,21 58,00
P(x1 S Z2)
P(z1 = 0, < x) + P(1 < 32, 0 < 0Ι S x, 0 < 02 <
P(x S x) > x)
P( x22)
S1 + (1 5) + + (1- 111111551112
+1+
(1- (1-))
(7)
1 S+52
16

---

Published as a conference paper at ICLR 2025
P(n S xly = 1) = P(xi 11,01 > x2)
P( 32)
P( S <x) - P( 14,01 S 12)
P(11 > T2)
P(r 5x)
P( x2)
-P(21 S x = 0)
P(r > T2)
P(n >> x22
= S+ (1 S)æ
(1- S) - x)²
1+
1 52)
1+ 1
1-
(1 8) 25(1 S)
(8)
1 52
Then the conditional means of Pmmono I1 are
Po (inon)))))
x dP S aly = 0)
2(1 S)2
T: 1+ $2
(1 x) dx
(n
2(1 - S)
1
1+ S2
3
1 (1-
(9)
3 1 $2
and
#1 (Vmooo)))
dd S Sxly=1 =1
2.2(1 S) (1 - S)x S
dr
xE(0,1]
1- 82
2(1-S)
5 (1 - 5)+ 5
1-52
12+S
(10)
31+5*
Then we have
11S Ni (Pmon)) (P0(2mono)
(11)
31+
Similarly, have the conditional variances follows.
/
(Pmono)
x ddPP(I S xly = 0) (Mmono)))
x
2(1 S)2
(1 - 2)) dx Mo(Pmono))
)oE(0.1)
1+ S2
2(1 - 2)
1
1
11
140 1100 (2ioono)2
1+
3
4
1 S)²
Ma MMmoono))
(12)
6 S2
17

---

Published as a conference paper at ICLR 2025
and
(Pmono)
xdP(x1 S æly = 1) - p1 (mmono)
x2 22.2(1 - S) (1 S)x + S
ln (Vmono)
2E(0,1)
1 - 5
2(1 - 5)
1
(1
- S) +
5
1
141 (Umono) )
(1-
3
13+
1/1 (Vinono)
(13)
61+
B.1.2 POLYSEMANTIC REPRESENTATIONS
To study the polysemantic case, we first have to derive the probability distribution of P/poly 21 22 and the corresponding conditional probability density functions on = and y = 1, separately. We first calculate the cumulative distribution functions as follows.
Lemma B.2 (Distribution of Ppply = I1 - T2).
1 11- (1- (1-S)z)
+1+
5²
1
e [0, 1].
2
P 12 x)
[ 5)x + 112
1
112- 5
x E [-10).
Proof of Lemma B.2. For x E [0, 1], we have
P(r - 82 S x)
N
=lim
Σ
P(x1 < x 2
N n=-N
[(1-x)N]
N Σ
lim
S n/N)
1. 1P
N-
n=0
N +1
[S + (1 8)x)
[(1-2)N]
+ lim
Σ
(1 - (1-) +
Σ (1-
N
S[S + -1- (1= + lim - | [
==1
N-
+1 +0-39900000(000002)1)2333-----00
- [11=)N] -) 1)/N
+ -1-1--1-00000-------------------
- S)x -=0) -5
1
(14)
18

---

Published as a conference paper at ICLR 2025
For 1-1,0), we have
N
P(r1 x2 x) =
lim
P(a S 1/N)(t222// =
N
N
lim
Σ
P(x S x ((o=s=t/NNNc==////
N-+00
== [=N]
N Σ
lim
IS+ - (1-11))(x=0 + n/N)] (1-
N-000
n=- --- [=N]
lim
[S(1 - S) + (1 - x) + [=NJ)/N
N-
+ (1 - S - \aN + + 11/22)
[[(11 5) + (1 x)
5) x²)/2
[1 -)x + 1 1 S²
(15)
2
L
Theorem B.3 (Conditional mean and variance of polysemantic representations), The conditional means and variances of Upply 31 22 are
(1- ()= 25)
1 1 2S
Po((poly)
and in (Vmon))
(16)
+52 52
1 S
1 1 11-55(1++3) (5(1++35)
11
o (Vooy)
Mu(Uyoly) and (V
11 (Mmouo) )
6 5
6 615
(17)
Proof of Theorem B.3. By Lemma B.2, we have
Ppoty (x1 x2 S xly = 0) = P(æi 32 S x|Пح S xx))
P(
22 S min(O, x ) /P (z: 12 50)
[[ -)x +1|² 52
(1+
1
x 11,
2
1.
E [0,
[[1 S)x + 112 - ss se] (11+ 52),
2 (-
1
x E [0
and
P Ppoly(21 - D2 S x/y = 1)
= P(æ1- - I2 S x > x2)
P(0 <3. 82 S x)/ /P(1 - T2 > 0)
[0,
x x [-10]
{IP(x) x2 S x)
P(ιi x2 S 0]//11
P(œ1 x2 50).
x z (0.
00
x 11,0
7 [1 -)= +1+ 5²
1 (1 + 52)
/11 11-
1 (1+ (1+55)1
1
x x (0,1)
2
(0,
11 0
[ - [1 (1 - S)x (1-
x (0.1
19

---

Published as a conference paper at ICLR 2025
Then we have
[
po((yoly)
x 2(1 S)((1 S)x + 11/(1 +52) dr
(xE(-1,0)
2(1 5)
[] S) 2
1+ S²
33
1 1 S) + 28)
(18)
3 14 52
pii (Mpoly) = |
E x=2(1 S) (1- (10--))00 ())]/(1-55)
(0.1).
2
(1 5)
1+
22
11+ 28
(19)
3 1
2
1
Pi (Upoly) - (o(opol
(20)
(bppp)) =
x² ..1((1 - S)[(1 S)x + 1)/(1 + S²) dx - (P
1,00
2(1 S)
(1-
1
(o(()o)))2
1+ 8
1
( S)
+ +38)
11 (o((poly)2
(21)
12
and
oj(ypoly) = [
2².2(1 S)(1 - -S)aa/11 -2) dx (1
xE((0,1]
2
1
1
- [1]] (Ppoo))
1 S
L3
4
11 35
M1 (Upoly)
(22)
8
B.1.3 PROOF OF THEOREM 4.1
Proof of Theorem 4.1. Following the toy model described in Section 4.2, we let S = 0.2. Then by Theorem B.I, have po (Mmom))) 0.205, ln (Mmono) 0.611, Ap(Umono)) 0.406, oo ( (Pmono) 0.246, ai (Pmono) = 0.266, and J (Pmooo) = 6.6. By Theorem B.3, we have po (Vpoly = - 0.359, μi (Ppot) 0.389, Ap(ppoly) 0.748, an
(Mmono) ( 10.164. By comparing the above results, we complete the proof.
0.276, a (Mpoly) 0.266, and J [(ppoly)
B.2 PROOFS RELATED TO LABEL NOISE
Following Ghosh et al. (2017); Ma et al. (2020); Wang et al. (2019b), we assume the noisy label ÿ is randomly flipped from the true labels other classes. Under 77 E [o, K-1). the noisy label distribution is
P = k|p) = Σ
P P) k|x),,
(23)
0=
where P( kly ;) = 1 if j k, and otherwise P( kly j) n
20

---

Published as a conference paper at ICLR 2025
B.2.1 INFLUENCE OF LABEL NOISE ON CONDITIONAL MEAN AND VARIANCE
Lemma B.4 (Conditional Distributions). For noise rate 17 1/, and sparsity S E [0, 1]. have conditional distributions
P(ulū 0)
(24)
(1 (1- (1+ +1(1 52)
and
P(vlğ = =) 771 ((-)))) +1+ +1- 52)
(25)
Proof of Lemma B.4. We first calculate the class conditional distributions.
Pivlū - 0) P (p)))())P(( 0) ビ= - -
=
1 = -
Σ 2== P(ù 0ly =
j) j)
(1 P [1ly=;))(yy))) =
Ej= = =--000y-00000 -())/)y))))((
r 1)P(y 1)
(26)
(1-r)P = 0) + my
P(vlğ = 1) = P = 1(--
Σz=0.1 - P1
E1= P(=
Σz 1 = =)
P(o =)
()
1)
(27)
- + (1 -
Recall that 21 22 0 with probability S, and æ 21 ((. 1] with probability S. Because I1 and 22 are independently and identically distributed and P(z1 = 22 22/ 0), we have P S x2/=11.0 x2 > 0) P(r22 I 21|21, 42 0) 1/2, and therefore
Ply 0) P(m1 S .22)
PP = 0) + P(11 > 0)P(2z > 0)P(21 I x2/ 22 > 0)
1 (1 S)²
1 ト/ S²).
S
(28)
Then P(y = 1) = - P(y = 0) = [1 5²), and correspondingly we have
P(ulū 0) (1- (1 + -)
(29)
11 (1 - ( 52)
and
P(u = 1) 71 + (
(30)
7 52+ + 17) -1) 52)
Theorem B.5 (Influence of label noise inter-class distance). For noise 1j [0, 1))
(1 - 21 (2) (1)
Ae(() + (1=1112000 21 -11 2 AH(P).
(31)
21

---

Published as a conference paper at ICLR 2025
Proof of Theorem B.5. By Lemma B.4, the conditional means of has the following forms.
po(2)= E(ulū 0)
(1 (1- + 5²)
dP(uly 0)
(-))((++52) n11 52)
J, "0111- (1 + 52) + 1/(1 82)
17(1 55)
dP(v/y 1).
(32)
(1(1) = E(vlğ- = 1)
dd((vū=1) 1)
711 S2)
dP 0)
11(1 82) (1 7) 5z)
I n11 52) + ( (1-n)((1 () 82)
(1
dP(rly 1).
(33)
Then we have
(v) (V)
1. 1
55)
(1 7) + 52)
dP(vly =
11(1 + (2) (1-1))1 - 52)
(1 (1- (1 + 52) + +1(11 - 52)
(1- (1-n)(1-5)
111 - -2)
+
dP(uly = 1)
(111 + (1
$2)
(1- (1-n)(1 + 52) + n((1 - -5?
(1 11-211 22) +1++++++000 (1-
udd(ull 1) vdd(vly= =0] 0)]
[1 (1 - 57 (1- 53
(1-2n) + (1-
23) [4I(v) - Ho(w)]
(34)
[1 + (1 (11-200) 21 (1
Theorem B.6 (Influence of label noise on intra-class variance). For = 0, 1 and noise rate 11 E [0, 5.
(v) = 0,,00 (v) + c. + C,,0Ha (v)2 + 1 (v) (v) + (5)1((())) (1 (1+
where Co.0
I+(1 21 Co.1
1+(1-2m)55 22) C1,0 1-(1-2n)5 and C1,1
(1+.5²)
(1-n)(++8)) 1-(1-2n)3*
Proof of Theorem B.6. By Lemma B.4, the conditional variances of v has the following forms.
db(y) :- 16 0) (²)))²
2 dP(p/p 0) - 2
1. 3 (1 1-((1 + 52) + 11(1 52)
(1 (1-n) n)((1 + 52)
dP(uly = 0)
11(1 - 52)
+ 2
(1 (1 + 52) 711 52) dP(vly 1) - #2o(v)2
(1 (1- (1 + 52)
711 + 52)
(00(1) + mo(L) + 1 (1 24) (o((v) + +m(
1+ 27
(1-7)(1+ $2)
I+ 21-2)) 4o (v(v)) + 1+ 2n) 2n)) {11)
cooğ(v) +(wo)(/oo())))) + copo(1)2 + (v)= - (co/oo()) +
(35)
22

---

Published as a conference paper at ICLR 2025
where cp = (1 (1+ 1+(1-2n)5) and 6 = 14(1-2n)S"
n(++S*)
oq(y) = E(v² 1 1) (v)
d 0) - -molu)²
1. 12 7
711 + 22) + (1 -1- dP(vly = 0)
52)
(1-1)(1 52)
52)
+
dP(uly 1) (1)
11 + 52) + 7)(( 52)
1 (1 20) (og(v) - mo(1)}}+ (1- 1) 52)
n11+ S2)
1- (1- 2n) (oî(v) + m1()
17(1 + 52)
(1 2n) mo(v) + (1 - 7) + 52)
1- (1 2n) [a(vs 2
( cG/((v) + 6)po(e)2 + c) (c)Mo(V) + d
(36)
where %
n(++$*) (1 and ej (1 (1-1)(1+5))
L
1-(1-2n)52
B.2.2 LINEAR SEPARABILITY OF MONOSEMANTIC & POLYSEMANTIC REPRESENTATIONS UNDER LABEL NOISE
Proof of Theorem 4.2. By definition, we have
[V (Vimonn)
1o (Vmouo))
[( JVpooly)
(//o://///////////////////////// )o (Ypoly ))
(37)
By Theorem B.5 we have Aj((Vmono)/Aa (Pmono)
= A
and 01 (Vmouo) =
01 (Ppoly), and thus
J(Vinooo) (Vtuono)
Tu(Upoly)
ōz (Vpoly)
(o(Vnon)))
(38)
[Vpoly) / (Vpoly)
8o(u (Umono)
01 (mmon)
oo(ypoly)
By Theorems B.I and B.6, we have
1.04(1 17)
1.04n
o (vmona)
(0.246² 0.205?)
(0.2662 + 0.6113)
1.04 - 0.08n
1.04 0.08n
1.04(1 7) 0.205
1.04g
+ 1.04
0.611
(39)
1.04 0.0817
0.08m)
1.04. (0.246²
0.2053 + 1.04(1 17) (0.266 + 0.6113)
o (Vmono)
0.96 + 0.08n)
0.96 + 0.08n)
1.04n
+ 1.04(1 17) 0.611
2
0.205
(40)
0.96 + 0.08r)
0.96 + 0.08g
1.04(1 17) (0.2762 + (-0.359) + 1.04 0.08n
1.04n
(P
(0.2662 + 0.3892)
1.04 - 0.087
1.04(1 17)
1.04m 0.389
2
(-0.359)
+ 1.04 0.08r)
(41)
1.04 - 0.08m)
1.04x) (0.2762 + ( (-0.359) 2 + 1.04(1 7)) (0.260
of (P(pplly)
+ 0.3892)
0.96 + 0.08m
0.96 + 0.081
1.04g
(-0.359) + 1.04(1 17) 0.389] 2
(42)
0.96 + 0.08m)
0.96 + 0.081
23

---

Published as a conference paper at ICLR 2025
Then plugging Eq. (39), Eq. (40), Eq. (41), and Eq. (42) into Eq. (38), we have I of [Mmowo) J(ppo (Mpols)
2
Further, by Theorems B.1 and B.5, we have
1.04 0.96(1 2n)
(p(ooooo))
X 0.406,
(43)
(1.04 0.08g) 0.96 + 0.087)
1.04 x 0.96 - 2n)
Aj((pooy)) =
X 0.748.
(44)
(1.04 - 0.08r) (0.96 + 0.08n)
Plugging them into the definition of (Vmono) and (P we have J (Umono) < (Upoly) when < 0.25 and [Vmon)) > JVpotyy) when 17 >> 0.25.
B.3 PROOFS RELATED TO INPUT NOISE
Following the settings in Section 4.2, we investigate the influence of Gaussian noise E; ~V(O, 1).
i..d. i 2, on the input data r 1æ x2). where ε; 1 I. Given noise strength À 0, we denote the noisy input data as ह = ( + λ 22 + As2). Then the learned monosemantic and polysemantic representations are Mmnno Ii + λÆ and Pplly (x) x22 + X(e 82). Next, derive the influence of noise strength on the conditional means and variances, respectively.
Theorem B.7 (Influence of Gaussian noise on inter-class distance). Given noise strength X > for both mono- and poly-semantic representations, we have
Aū(() = Au(v).
(45)
Proof of Theorem B.7. For Mmoo and = 0,
A (V(mmo)))
E(x1 + Xeily = i)
E(Iily = i) XE(E
E(xily== +0
M4 (Vmoooo).
(46)
For poly and i = 0, 1,
[ii(yool))
B((21 222 + X(e1 E2)ly =
E(ø1 22/y i) + A[E(E1ly = i) - EE(zly = i)]
E(xi x2ly +0
(47)
Then for v E {Umonos Pppoly b.
Aj((v) Pun (v) - -Po(v) = =M((1) -ma(i) Ap(v).
(48)
Theorem B.8 (Influence of Gaussian noise on intra-class variance). For = 0, 1 and noise strength O. we have
0² (Umouo) = of (Vtoon)) + x2
(49)
and
o.( (?( + 2X2.
(50)
Proof of Theorem B.8. For Vmono and = 0,
87 (
= [[(x) + >x 22y = i) - (u (Umono)
E(xsly = i) + 2XE(2111 = i) + X²B(E?ly = =i) (u(Vnono)
E[<zlyy i) - (Mmono))) + 0 >BE
= (Pmonoo) + x2
(51)
24

---

Published as a conference paper at ICLR 2025
For Polly and i 0, 1,
0 (Umono)
[[((21 - z2) + M(E1 - E2)) = i) - (Ppol))
E((x1 - x22) = 21 - 22)(E1 = - x - = i) - [ie(upoly)
=B((21 -x2) =22)²1==1 = (Mpoot) + 211 - -22)=1lu= 211((111 - x2)52/y = i)
+ * =i) = -2 B(5162ly = i) + A2 =
2X
(uloy) + 222
(52)
Theorem B.9 (Influence of Gaussian noise on linear seprarability). denote the linear separability criterion under noise as J(v) A((())((((v)}((()) For noise rate 0,
[upolyy
[Vmono)
1.
(53)
[Vpoly [Vmono)
Meanwhile, we obtain poly) S J when À 2 0.55.
As shown in Theorem B.9, with the increase of noise strength, the linear separability (J(z) of both polysemantic and monosemantic features becomes worse. However, (Umono) decreases more slowly. As result, when the noise strength is aggressive enough (A > 0.25), the monosemantic feature exhibits better linear seperability than the polysemantic one. The theoretical results reveal that the linear separability of monosemantie features is more robust than polysemantic features, which leads to better performance tasks under Input noise.
Proof of Theorem B.9. By definition, we have
(V (Vmono))
[ [Aa((Poon)) (Mo (Nmono)) //oha///oo//////////oooo////o//0 (Mmono))] Apo(ypol)) (/ol Jā (/tpooyy)]][AAA(((e (/o (ppoly )o (Vpoty ))
J (P J
(54)
By Theorems B.7 and B.8, have Açp(pmono) Ap((mono)) Ani(poly) Aplupaty). ) 57 (Pmono) = of mooon) + X2, and (Ppoly) of (1Poly) + 2X² i 0, By Theorem 4.1, we have 0 (Vmono) = 01 (Ppoly). Then we have
J/ J/
ōu(Vpoly Jā (Mpoly J (Pmono)
J (V (VPoly)
@o(Mmono )āi (Vmono) ) (Ppoly)
V(og(upnty) + 2X ) (ff poly) + 22222 222))o(ummom
(55)
Viag (Vmono) {²) ( /mooo) + X+))oo(ppoly)
Then plugging Theorem 4.1, we complete the proof.
B.4 LINEAR SEPARABILITY OF POLYSEMANTIC REPRESENTATIONS (GENERALIZED FORM)
In this part, we generalize the polysemantic representation to Ppoly = W W2T2, where W1, W2 > O. Without loss of generality, we could assume wi 2 W2. The case wT < W2 is equivalent to study "poly -Ppoly = W2?g W 1, whose distribution the same as W251 w because Fi and x2 are i.i.d. distributed. Note that when w1 1, W2 = 0, 1/poly = Vmono- and when W1 = W22 = the results in this section reduce to Theorems 4.1 and 4.2.
We first calculate some related probability distribution functions in Lemma B.10.
25

---

Published as a conference paper at ICLR 2025
Lemma B.10.
p(11 S min mi/(w//w 12),
if E [- 0),
211 (1-5)((1 z/(ww w2)) + 21 52),
if x E [0, w1 w2),
(56)
(1+ +1+55 5²)
if E (wi w2. w1]
P(z/(uu - W2) < σι S (ar + w2x2)/W1)
W2 [(1 5) + x/w22 + Swi /w2]²
w1 S²,
if E E1-w2, 0), if E [0, w1 w2).
2001
2w2
W2 (1 $) x/ /w1 - (w))?
(57)
200
0,
if E (wi - w2, wi)-
P(12 < I1 I min((( /222)/ W1, 2/ x//w/// w2)3)
if x E [- 0),
[11 - 5)x + S(w) ww2)l
W1 - W2 52
if E [0, w1 - -w2),
2wi (wi W22
2u'1
W1 (1 S)2 - xx/1///122))))))))))))))))) + S(1 S)x/wu W1 2 -)),
if (wi W2, w1)-
2w2
(58)
Then we can calculate the conditional mean and variance of Upoly = 11 W2222
Theorem B.11 (Conditional mean and variance of polysemantic representations). The conditional means and variances of Ppoly = 1 22 are
1 1-5
(o(Polly) = 3(1 + 52) [[w] - 2w2) - (W1 + w2)SS
(59)
(2w1 W2) + (Wi + W2)S
μi (Vmono)
(60)
3(1 + S)
= 1- S 6(1 + 82)
0g (yplyy
[(w] - 3w1 W2 + "w3) - (w) - -3w1 u2 - -w})s} - -oo(poly)?,
(61)
(3w? 3wi w2 + w3] + (wf + 3w W2 w3)S - u4 (pplyy)))
of oi(ppory)
(62)
6(1 + S)
Proof of Theorem B.3. If wi w2, by Lemma B0 we have
P (10y(11111 - 02202 A zly = 0)
P(w121 W222 x|Ι1 - x2)
P(z1 5 minfx/(w1 -W2), 1) + P(x/(w1 W2) < 21 $ (x + Wyx2)/w)
P x2 5 0)
W2 1
W1 82
[1 - -S)(1 + x/w2) Swn /w////z122
if 1- 0),
Wi + 52
We 1 S2
(1-
(63)
(1 (-w2/w))
(1 x/(wi w2))²² + 1,
if E [0, wi W2).
1
1
if a E (wi w2, wi
26

---

Published as a conference paper at ICLR 2025
and
Polly (wi31 0242 S aly 1)
P(W111 - W222 E x > 2:2)
P(12 < 21 min (xr (2dr2/Ww.. x//(//// w2)))
P(11 x2 > 0)
0,
if [- 0),
1
[11 S)x + S(w) - w22)1
Wi
W2
g
if E [0, wi - w2).
Wi (wi - W22 1 52
W1
52
w1 - 8
25
(1-S)
(1 x/w1)² /2+ +
1 2/ W1
if x E (w) w2, wi)
w2 + 5
(1+
(64)
Then have
Mo Mo(Vpoly) = /
2 5
- 5) + xx/w2) + Swi Sw//wz// dx
ZzE(-w2.))
81 1 + 52
(1 - 5)2
x// (wi w22) do
W1 1+ 52
35(1
Sww1 W2 + (1 - 5)
(1- (1-5) w22
+
3(1 52)w1
3(1 + 52)
1-S S
[(1 5)w - +2+ S)w2]
3(1 + 82)
-
[101 2w2) (wi w2)
(65)
3(1 + 82)
2
Pa (1(uyay))
[(1 -S) lr + (wi w2)SS do
22|00.1 Wz]
W1 (wi w2) + +S)
2
5
2
W22 + S (1 x /w) +
dr
JoE(wI -W2,11
W1
(2+ (2+
w2
3(1 + S)w W/2 - (2 + S)
+
3(1 + S)W1
3(1 S)w1
(2+ + S)w] - (1 S)w2
3(1 5)
(2w1 - W2) + (w) + W2
(66)
3(1 S)
(Wi + w2) + 2(w + +w2)S + 3(wl w2)
P1 (1(plyy) (HO(ppoly))
(67)
3(1 S) S2)
27

---

Published as a conference paper at ICLR 2025
2 1 1 S
ob(Poot))
x²
W1 1 + 52 (1- x) + Sw1/ / /w2/ dr
112(-12.0)
+ /
2 (1 - -
x²
[] - x/(w1 - w2]] dr Holppoly)
/n m22
W1 1+ 5
45(1
5) w W (1 S)
6(1 + S3)
+ (1 S) - -3(1 S) w 22 + 3(1 - 5) w/w/ w3 (1 2)
Ho((poly)
6(1 + 52)
(1 - u-) 3(1 5)) w1 W2 + (1 (3-
+ 5)w3 n (oo(Poly) )
6(1 + 5²)
S-5
[(1 -)w] 3(1 S)m1100 + +33+ S)w3] 1o Mo )²
6(1 + 52)
-
[w0] 3ww u2 + -wmg) - (w} - -301 wz}s)] 1 (oo(polyy
(68)
6(1 + 52)
and
2
(Ppoly =
x²a
[11 - (1-5)x + (wi - W2) dx
JIEE -w2]
W1 (w) w2)(1 + 5)
U12 + 5 (1 x//0)) + 2 S
2 1 5 5
+
x²a
dx pn (Upoly) )²
2E(W w1
W1 + S
(3 13+S)uF - 3(33 + 5) S)w w2 + +333 + 5)w w3
(3 + S)
6(1 + S) w)
6(1 + +)wgw/ - 4((2 + S)w, w + (3 + 5ww3
+
th (Ppoly)
6(1 + S)wi
1i (Vpoly) )a Sww - 3(1 - S) W2 + (1 - (1-
6(1 + S)
(3w}- 3w1 22 + +w3) + (w) + 3wiwa -w))s
M1 )1(pol)))
(69)
6(1 + 8)
Following the toy model described in Section 4.2, where we let S = 0.2, we compare the linear separability of monosemantic and polysemantic features in Theorem B.
Theorem B.12 (Conditional means and variances of monosemantic & polysemantic features (Gen- cralized Form)). If0= < W2 I Wi = 1, for conditional means, we have (o(P)oly) < no (P and (t1 (yll)) < M1 (Vmono))) yet App(ypol)) > Ap((mmnno). For conditional variances, we have of (Vpply I (Pmono) and (Vpoy) > of (Vmono)) Overall, we have J [Vpoly) > J (Vmono))
0.205, 141 (Mmono) 0.611, Ap( (Mmon)) 0.406, of (mmono)= = 0.061, of (Umomo) = 0.071, and J ((mmono) 6.196. By Theo- rem 11,,, we have (4 (Mpoly) = 0.205w, - 0.564w2, /41 (Mpoly) 0.61 w/ 0.222ur,, Ap(Ppoly)
Proof of Theorem 12. By Theorem B. we have no (o(Moono)
0.406w, + 0.342wa, oz (Ppol) 0.061 0.077 w2 + 0.092w2 of (Mpoly 0.071 0.062w, W2 + 0. 0.062w3, and [(polly) =
0.406 +.342m02
/ /10.061w; w00.0or.220..0.021000000 U.066w) uy (0.06
By comparing the above results, we complete the proof.
Theorem B.13 (Influence of label noise on linear seprarability (Generalized Form)). We denote the linear separability criterion under noise as J(v) A(le)/((((()ōn(())) If wa S wi = then for noise rate v) E [0, 0.5), we have
j(ppoly) [Vpoly)
[( J (Vmono)
<
< 1.
(70)
28

---

Published as a conference paper at ICLR 2025
Proof of Theorem B.13. By definition, we have
[V [[
[ (V((mono))) (/woa//onnoooo/oooo)/////aaaa/// (oo )on (V (Vmouo)) })]
J (Vpoly) [VPolt)//[Vppoy (Vpoy))
pooly) poly
(P ))
(71)
By Theorem B.5 we have Aj((woon)) Au(( = ///ppoly)/AA(((e/ and thus
[Vmono)/ J
oo (o(Ppoly)
81 (Ppoly)
Fu (Vmono)
01 (2moou)))
(72)
{Vppoly) (Vpoly)
oo (Pmono)
01 (Mmono)
(o(Po))
oi (P(olyy
Then given 17 E [0, 0.5), by Theorems B.1, B.6, B.11, we can calculate the specific values of āo (mmooo)) Ō1 (Umona)s oo((poly), ōi (4poly), and correspondingly have Ji / (Umamo)) J(Vooty (M
1.
B.5 MONOSEMANTICITY OF REPRESENTATIONS LEARNED BY NON-NEGATIVE CONTRASTIVE LEARNING AND NON-NEGATIVE TUNING
Wang al. (2024) proved that the optimal solutions of non-negative contrastive learning monose- mantic representations. We first introduce some necessary notations. Following (Saunshi et al., 2019), they assume that samples in the pretraining data belong m latent classes C = [ci,.... Cm }. Besides that, they assume that positive samples in contrastive learning are darw independently from the same latent class, i.e.,
Assumption B.14 (Positive Generation). Vz, x' E EX,P(a, = =
Then they propose the following Lemma that introduces the optimal solutions of non-negative contrastive learning.
Lemma B.15 ((Wang et al., 2024)). We denote that
1 P(milz),....
1 vP((Tm)
d(x)= [
P((mmlz))
RR V
(73)
where (....... nm, is random permutation of latent classes (01, Gm]]. Under the latent class assumption and choosing 1: m, <6-) is a minimizer of the NCL objective, i.e., I E arg min LNCL-
As shown in the lemma above, the samples activated in the same dimension belong the same latent classes, which means the optimal solutions are monosemantic. In the next step, we extend the theory non-negative tuning. Following (Wang et al., 2024), we the spectral loss for the simplicity of analysis, i.e. LSCE = - (=(((()))) (W f())) n(a) + 2E..,. ((W (x)) 1,,,)² where f(x) is the representation layer, W is the linear classifier, y(x) is the label of x and there exist t classes. Respectively, with the non-negative constraints, the objective becomes LSNCE
(e)) (W fxx)) v(x) + 2.g ge ((W) f+(x)) (x)) 12,)). When each sample x has a ground-truth
label (ar), we obtain the following theorem:
Theorem B.16. We denote that
((x) /P(wr) P(nl=),..... VP(y)) P(ull)) E ERI
(74)
When we choose t = k. B(-) is minimizer of the non-negative tuning objective, iee, arg min LSNCE-
Consequently, we find that the optimal solutions of non-negative tuning are also monosemantic as each dimension is only activated by the samples in the same class.
Proof. We first prove that the non-negative tuning objective is equal matrix decomposition objective. To be specific, Let À be the normalized co-occurrence matrix, i.e., Arw(z) (P(2)P(y(o)) where P(x), P()())) denote the marginal distribution.
Azaiis)
29

---

Published as a conference paper at ICLR 2025
Then denote matrix decomposition objective
LMF = 1| - FW"T 12 + const.
where the x-th row of F and the y(ær)-th row of W" respectively represents representation features and the linear classifier, i.e., Fg = (x) W (x)))) = -aoo=vppiiip P(uu(=))W, y(a)- Then we expand the objective:
||A - 12
E
(Au)(=) )))
x.y(2)
(P(T)/+(0) (Wo 2
P(x,y(z))
Σ
(P(x))(y(E1)
xy(x)
P(z,)(z))² P(x)P(y(+))
+P(r))(((t)) (f+(z) (W (M 2((2) (W )
x.((z)
P(a,y(x))
Σ
2 (eve)))) (x) (W Ray, (WW,))(WT..T]] T)
P(x))(y(x))
xz((z)
P(x,v(x))
Σ
(ma()) (w)) 1 100)) + Er., (W++(x)) 1g.)²
P(x))(y(z)))
xx(())
CSNCE + const.
Then expand (B):
| - F2 Σ
P(a,)(1))
VP(T)B(xT (isiiiT/PPiii (W
(VP()))(((z)) a,y(x)
When satisfics Wy, /P(x.)' we obtain
P(x,)(x))
VP(O)((0)T)
|| - F1-F Σ
(((x))(y(z))
xx((x)
P(x,V(z))
(((c)(u(E))L))
Σ
(P(x))(((z))
VP(y(z))
xN((2)
2
P(z,v(z))
P(a,)()))
Σ
(P(z))(v(x))
(P(x)P(y(x))
xx(x)
0,
which means 33 is optimal solution of non-negative tuning.
30

---

Published as a conference paper at ICLR 2025
C
ADDITIONAL EXPERIMENTS
50
Validation Accuracy
NCL
20
CE
SCE
Linear Evaluation Loss
Figure 6: Linear probing performance with different evaluation losses ImageNet- 100 under 95%
noise rates.
C.1 COMBINATION WITH ROBUST Loss
The previous results suggest that the monosemantic representations exhibit stronger robustness against label noise across various datasets. We note that there have been various studies improve the robustness under label noise, such as applying robust loss functions (Van Rooyen et al., 2015; Ghosh et al., 2017), correcting training labels (Reed et al., 2014; Ma al., 2018), and reweighting training samples (Chen et al., 2019; Han et al., 2018). However, the perspective in this paper is orthogonal to them. Taking the representative robust loss function Symmetric Cross Entropy (Wang et al., 2019b) as an example, we can obtain monosemantic representations as discussed above and then use the robust loss during the linear probing process. As shown in Figure 6, both the robust loss and enhancing feature monosemanticity can improve the robustness against label noise. Furthermore, the two methods are orthogonal, and combining them can further improve performance.
C.2 EVALUATION OF MONOSEMANTICITY
We evaluate the monosemanticity of different models with current metrics. For vision models, we follow Wang al. (2024) and adopt semantic consistency the metric. The semantic consistency calculates the proportion of activated samples that belong to their most frequent class along each dimension. We evaluate the models in different cases, including the frozen models (Table 1, Figure 2) and the models fine-tuned with 20 % training samples and 20% noise rate (Figure 3).
As shown in Table 1, Non-negative Contrastive Learning (NCL), Sparse Autoencoder (SAE), and Nen-negative tuning significantly improve the semantic consistency of trained models, which further supports claim that attaining monosemanticity note that the NCL obtains larger improvements on monosemanticity. which is consistent with the results that NCL performs better than SAE under noise.
enhance model robustness. Besides, also
C.3 ADDITIONAL EVALUATION ON DIFFERENT NOISE RATES
Besides the noise rates in Table 2, we provide more detailed results for the 0-10% noise range results the following table, with runs and error bars. As shown Table we note that monosemantic representations also show benefits with noise rates less equal to 10%, and the advantages rise with stronger noise rates.
C.4 ADDITIONAL COMPARISON WITH NON-NEGATIVE CONTRASTIVE LEARNING
We note that there exist performance discrepancies bewteen our linear probing results with Wang et al. (2024). The discrepancies arise because we evaluate the linear probing accuracy on the features after the projector (an additional MLP after the backbone used in contrastive learning) while Wang et al. (2024) evaluate that on the features before the projector. We note that Wang et al. (2024) adds non-negative constraints and calculates the monosemanticity-reattee scores the features after the
31

---

Published as a conference paper at ICLR 2025
Table 4: Linear probing accuracy (%) under different noise rates (0-10%) ImageNet-100 of
polysemantic features (CL.) and monosemantie features (NCL,, SAE).
0
2
4
6
8
10
CL 6..8
65.4
65.0++0.2
64.4
63.8+00.2
63.3+-0.2
NCL
66.8
66.2:10.3
66.1100.4
66.110.1
65.5
65.5
Gains
-0.0
+0.8
+1.1
+1.7
+1.7
+2.2
SAE
66. 66.120..
65.910.11
65.9
65.7
65.5+0.3
65.2+0.2
Gains -0.6
+0.5
+0.9
+1.3
+1.7
1.9
Table 5: Linear probing accuracy (%) under different noise rates on ImageNet-100 features before and after the projectors of models trained by CL and NCL.
Noise Rate (%)
0
30
60
90
CL (w projector)
66.8
60.1
54.9
34.4
NCL (w projector)
66.7
63.9
50.5
48.1
CL (w/o projector)
68.6
49.3
33.0
9.7
NCL (w/o projector)
69.5
50.6
33.7
9.8
projector. However, they calculate the linear evaluation accuracy of the features before the projector, which leads to mismatch and can observe the relationship between monosemanticityy and performance. Consequently, we calculate the performance of the features after the projector in this paper. Furthermore, conduct additional experiments evaluate the linear probing accuracy of the features before the projector.
As shown in Table 5, observe that the improvements in accuracy do not increase with larger noise rates, which reasonable as the features before the projector are not guaranteed to monosemantic. Besides, we also note that the linear probing accuracy of the features after the projector performs better than the features before the projector under label noise, which further implies that we should use the features after the projector in this paper.
C.5 ADDITIONAL EVALUATION ON LLMs
We also evaluate LLMs on additional benchmarks. To be specific, we respectively evaluate the original LLM models -llam-2--B--C and the models fine-tuned Dolly with LoRA and MonoRoLA the MMLU benchmark.
As shown in Table 6, observe that different from the security score in the main paper (Shield- Gemma Alignment Scores and Beavertails), the MMLU scores almost do not change during the fine-tuning process on Dolly with LoRA, which implies that the LLM abilities the MMLU bench- mark are mostly preserved during fine-tuning. Consequently, our method does not exhibit benefits as it is more like the clean accuracy of language models, which is consistent with our claims that monosemanticity shows benefits in the tasks related to model robustness.
C.6 ABLATION STUDY ON SPARSE AUTOENCODERS
To evaluate how robust are these models to the choice of the number of activated features in SAEs, we evaluate the performance of SAE respectively with the number of activated features as 64, 128, 256, 512, and 1024.
Table 6: Evaluation of models tuned by LoRA and MonoL on MMLU.
Humantities
Other
Social Sciences
Stem
Average
Base
0.43
0.55
0.53
0.36
0.47
LoRA
0.43
0.53
0.53
0.37
0.47
MonoLoRA
0.43
0.53
0.53
0.37
0.47
32

---

Published as a conference paper at ICLR 2025
Table 7: Linear probing accuracy (%) of SAE with different numbers of activated features under different label noise rates on ImageNet-100)
Noise Rate (%)
64
128
256
512
1024
w/o SAE
0
64.2
64.7
66.1
66.4
66.8
66.8
30
59.3
59.6
60.6
60.4
60.4
60.1
60
58.3
58.2
58.6
55.7
55.2
54.9
90
45.9
45.9
45.7
43.1
38.9
34.5
As shown in Table 7, all the SAEs with different numbers of activated features exhibit benefits when there exist label noises. To be specific, when the activated features are small, the original performance will drop sharply when there exists no label noise. Meanwhile, when the label noise is aggressive, the small number of activated features will indicate stronger linear accuracy. Consequently, 256 is sweet point that shows comparable performance in clean accuracy and exhibits benefits when there exists noise.
33


Reviews:
[{"review_id": "RNXiWjasGm", "reviewer": "Reviewer_1weT", "summary": "The paper aims to investigate the role of monosemanticity of model features on model robustness. Monosemanticity of model features is characterised by individual dimensions of the model embedding layer being activated with single latent natural concepts (vs. several, which corresponds to polysemanticity of model features). Robustness is investigated empirically in linear probing experiments on CIFAR-100 and ImageNet-100 with respect to label noise, gaussian input noise, uniform input noise, and sketch and stylised distribution shifts. Furthermore, it is investigated empirically in few-shot finetuning on ImageNet-100 as well as on LoRA training for LLMs. \nTo analyse the effect of monosemanticity on robustness, models trained to exhibit monosemantic features (e.g. through sparse auto encoders) are compared to vanilla trained models (representing polysemantic feature models). In all comparisons where robustness is investigated as described, the models trained to exhibit monosemantic features perform better.\nLastly, the paper illustrates benefits of monosemanticity for model robustness on toy examples as well.", "questions": "-", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is investigating an interesting and relevant research question: is monosemanticity beneficial beyond interpretability, and for model robustness in particular?\n- The paper is overall well written\n- The paper has a good range of experiment beds (but individual experiments should be enriched, see below)", "weaknesses": "- Main weakness: At no point in Section 3 does the paper measure how ‘monosemantic’ the trained models are (according to their definition in line 114). So any claim of ‘monosemanticity -> robustness’ cannot be derived from the experiments in Section 3, as we do not know how monosemantic the models really are. Is there actually a quantifiable and significant difference in the monosemanticity of the models in Table 1, Figure 2, or Figure 3, or Table 2 (here, sparsity is given as a proxy for monosemanticity, but how close of a proxy it is is unclear)\n- The definition of the polysemantic feature $v = x_1 - x_2$ in the theoretical toy example is specific rather than general, and insights derived from it will be accordingly so. For a more general validity of the results derived here, it should be of the form $w_1 x_1 + w_2 x_2 + b$ I believe. \n- Figure 3a and 3b: it is possible that the cause for lower training and better validation accuracy is simply the regularisation and not the resulting monosemanticity. To understand this better, including a different regularisation that does not promote semanticity would be beneficial (e.g. early stopping or L2 regularisation, or even something like a negative-cross entropy). In any case monosemanticity of final models needs to be evaluated to draw conclusions here (see first weakness)\n- Even if the other weaknesses were addressed, the paper would slightly overclaim, in particular in saying ‘feature monosematnicity to bring clear gains in model accuracy’ (l 100 and similar at other parts). As can be seen in the ‘0’ column of Table 1, monosemanticity is still at odds with model accuracy on clean iid accuracy (even though only slightly). Authors should specify accordingly that monosematnicity can bring gains in model robustness, not in general accuracy (as in some cases as the above it does not). \n- For the vision model results, only a single backbone (ResNet18) is used in experiments. To give the claims and findings greater generality, including results of the same experiment suite but with a ViT backbone would be beneficial. \n\n- Minor: typo l 242: Cifar-10 should be Cifar-100; typo l 257: ‘if there IS only a SMALL amount of …’, \n- Minor: align references in l 120 and l 45", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ga2q0pSE4x", "reviewer": "Reviewer_bevX", "summary": "This paper challenges the \"accuracy-interpretability\" tradeoff in deep learning models by considering the robustness of features across a range of approaches to learning. Experimental results validate that learned monosemantic features are more robust to noisy data and produces better performance when finetuning. The paper extends a toy model from Elhage et al. to suggest an explanation for this phenomenon.", "questions": "1. What is the principled reason for distinguishing between few shot and LLM finetuning?\n2. Should we expect a difference in outcomes between adding Gaussian noise and adding uniform noise?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The application of robustness as a performance metric to feature semanticity is a useful contribution.\n2. The choice of baseline and methods to produce monosemanticity are well chosen.\n3. The toy model provides a clear demonstration of why monosemantic features fare better.", "weaknesses": "1. The paper spends little time contextualising the \"widely accepted belief\" that there is an accuracy-interpretability tradeoff.\n2. It isn't immediately clear why MonoLoRA promotes feature monosemanticity. A clearer explanation of the mathematics would be helpful.\n3. The theoretical explanation is limited to label noise only, rather than finetuning scenarios. It would be good to address it at least in passing, or as a possible avenue of further work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zrda5Qn6qu", "reviewer": "Reviewer_u9M1", "summary": "The  authors  propose  that  monosemanticity,  where  neurons  represent  distinct  and  consistent  semantics,  can  improve  interpretability  without  compromising  accuracy.  The  authors  demonstrate  that  models  using  monosemantic  features  perform  better  in  three  robust  learning  scenarios.  The  study  provides  empirical  and  theoretical  evidence  that  monosemanticity  leads  to  more  robust  decision  boundaries,  suggesting  a link  between  interpretability  and  robustness.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper offers a valuable and insightful perspective on the benefits of feature monosemanticity.\n\n2. The study presents a robust and diverse set of evidence and experiments from various learning scenarios.\n\n3. The paper is well-organized, with a clear and logical structure that makes it easy to follow.", "weaknesses": "1. The paper utilizes existing sparse and non-negative constraints in three tasks, which are well-researched areas. While the application of these constraints to monosemanticity is interesting, the overall novelty is somewhat limited.\n\n2. The paper does not provide a thorough explanation for why non-negative constraints are generalized across all three tasks. It is unclear whether the same form of non-negative constraints is suitable for all tasks or if specific forms are needed for different scenarios. More detailed discussion and additional experiments would help to clarify this point and strengthen the argument.\n\n3. Abbreviations such as NCL and SAE, introduced in line 69, should be defined at their first occurrence. It well be helpful for readers to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K25gYDgVRh", "reviewer": "Reviewer_qpFh", "summary": "The work tackles the problem of polysemantic neurone, i.e., when in the last layer of a neural network individual neurones are activated by multiple concepts. The authors state to break the accuracy-interpretability tradeoff and show that monosemantic neurones can provide more robust models as well as gain of accuracy. The authors also present theoretical calculations connected to monosemantic and polysemantic features.", "questions": "I have counted 4 data sets (for different tasks), is it really enough to make any general statement? If yes, the authors should provide an argument why? E.g., can the performance differences from Table 1 be called statistically significant? If yes, with which p-value(s)?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The observations made in the manuscript regarding the monosemanticity seem original.\nQuality of the research cannot be disputed.", "weaknesses": "1. Reading and understanding the manuscript relatively well requires a specific set/baggage of knowledge to have been acquired, because the manuscript is full of branch-specific references and acronyms. This makes the manuscript difficultly accessible for general public.\n\nPerhaps as a possible improvement, the author(s) could prepare more intuitive writing for the body of the manuscript but shift (more difficult to understand) acronymed details to appendix.\n\n2. I do not find a clear methodological contribution, rather an empirical finding.\n\n3. What authors call Theorems 4.1 and 4.2 are (mathematically speaking) examples, not theorems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v7OMw1jLko", "reviewer": "Reviewer_JMHa", "summary": "In this work the authors show that inducing models to learn more monosemantic features (as opposed to neurons that learn multiple concepts, also known as polysemantic features) not only improves the interpretability of the model decisions, but it can also lead to more robust models. \n\nThe authors show results both in the vision and language domain. For the vision experiments they adopt two different techniques in order to induce monosemanticity: Non-Negative Contrastive Learning  (NCL, Wang et al 2024) which is a simple variation of SimCLR and it is supposed to create monosemantic representations, and Sparse Auto Encoders (SAE, Gao et al 2024) as a post-training intervention that produces embedding with higher monosemanticity. The authors use ReseNet-18 trained in a self-supervised fashion on CIFAR-100 and Imagenet-100. The authors show that when using either NCL to train the model or SAE (on the embedding of a pre trained SimCLR model) the accuracy of both models are comparable to that of the original model (that is usually polysemantic). However, when the dataset includes higher level of label noise, distribution shift (gaussian or uniform noise, real-data distribution shift), or the dataset is much smaller, the model that rely on monosemantic features remain more robust than the original model.\n\nThe authors also show some empirical analysis where they find that when training with a 90% of label noise, the dimension activated by the class with lower accuracy is polysemantic (i.e., is also activated by other classes), whereas the class with the higher dimension is monosemantic. Given this they conclude that noisy classifiers prefer monosemantic feature in practice.\n\nFor the language model the authors propose to modify the known Low Rank Adaptation mechanism (LoRA) by adding non-negative transformations (ReLU) that force the output of the LoRA layers to be more sparse (which is assumed to also induce monosemanticity). The authors fine-tune a Llama2 model on two different datasets (SST2 and Dolly) using the traditional LoRA and their proposed MonoLoRA, and show that while both models achieve comparable downstream task performance, MonoLoRA better preserves the model alignment (measured by the ShieldGemma and Beavertail alignment scores). In the case of the SST2 dataset the alignment becomes even better than that of the original model.\n\nLastly, the authors present an analysis using the super-position model (Elhage 2022b) that validates the results found on the vision models: in absence of noise models with polysemantic features show better performance, however, in the presence of noise the models with monosemantic features become more accurate than their polysemantic counterpart. Using a toy task (binary classification with a simple 2 dimensional embedding) the authors also show that when the label noise rate is grater than 25% the model with mono semantic features show a better linear separability between the two classes, again validating the empirical results found in the vision experiments on CIFAR-100 and ImageNet-100.", "questions": "On the unrealistic scenarios, could the authors:\n1. Provide detailed results for the 0-10% noise range, with multiple runs and error bars.\n2. Discuss whether the benefits of monosemanticity are still observed in this more realistic regime.\n3. Consider emphasizing and expanding the experiments on real-world distribution shift experiments, as these may be more practically relevant.\n\nOn the discrepancies with Wang et al. 2024 results for CIFAR-100 and ImageNet-100, could the authors:\n1. Provide a detailed explanation for the discrepancies.\n2. Verify their implementation of NCL to ensure it matches the original method.\n3. If possible, reproduce the exact settings from Wang et al. to confirm whether the discrepancy persists.\n\nOn the quantification of “monosemanticity”, could the authors:\n1. Measure and report a metric of monosemanticity (e.g., semantic consistency as used in Wang et al.) for all models and methods, including MonoLoRA.\n2. Provide a comparative analysis of monosemanticity levels between NCL, SAE, and standard models.\n3. Investigate the relationship between sparsity and monosemanticity in MonoLoRA.\n\nOn the lack of error bars. It is important to repeat the experiments (especially when stochastic processes such sampling are involved) and provide mean and standard deviation. Are the results shown statistically significant?\n\nOn the lack of evaluation of LLMs: would it make sense to show that the LLM abilities are better preserved with monosemanticity by evaluating the MMLU benchmark?\n\nFurther questions\n- In the abstract the authors called these analysis “Preliminary” could you clarify which of the results you consider preliminary and what you would need to consider them definitive? \n- The end of the intro reads “Theoretically, as a preliminary step, we compare polysemantic and monosemantic features under a toy model proposed in Elhage et al. (2022b). The theory suggests that because monosemantic features have better separation of features, they are less prone to overfitting to noise, leading to more robust decision boundaries compared to polysemantic features.”. Is this what the theoretical analysis concluded? If so, shouldn’t the benefit be evident also in noise-free scenarios? I believe that what it is shown in section 4.3 matches this statement only when in presence of a large amount of noise (>=25%). Or have I misunderstood what the authors mean?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper investigate an aspect of monosemanticity that I am not aware that has been investigated before. \n\nAmong all the experiments presented I think the real-data distribution shift and the low percentage of training data represent important and realistic scenarios where monosemanticity could play an interesting role.\n\nThe paper is easy to read and I particularly liked the introduction where all the main findings are nicely summarized.", "weaknesses": "The main weakness I see in this work is related to the experimental setting. The methods used to induced monosemanticity are already known (with the exception of MonoLora) so I would have expected a more thorough evaluation. In particular these are the main points I would like to discuss with the authors: potentially unrealistic scenarios (e.g., label noise >= 25%),  some concerns with the quality of the empirical evaluation (e.g., baselines not matching state of the art results, lack of quantification of monosemanticity, lack of error bars, lack of evaluation of LLM capabilities). Addressing these points could improve the soundness and the contribution of this work. Let me elaborate more in detail each of those points.\n\n- Potentially unrealistic scenarios. The experiments on labels noise explore a range of noise that goes from 0 to 90% which seems a very large and unrealistic amount of noise. I would consider exploring more in details (and with multiple runs) what happens in a more realistic range. The work from MIT “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks” could be a useful guidance to know what is a realistic range. From that paper it seems the full ImageNet has about 3.4% of label errors, while CIFAR-100 has about 5.85% (from their table 1). The worst case is QuickDraw with 10.12% error rate. The average error rate from that table is 3.32%. Given those results, perhaps focusing on the range 0-10 would be more realistic and informative than 0-90.  I believe this paper would be very interesting (if results holds) if more space was dedicated to real-data distribution shifts (all imagenet test bed as mentioned below), training with low amount of data, and fine-tuning LLMs. All the experiments with label noise and other kind of noise would be interesting only if results showed that monosemanticity can help in realistic scenarios (e.g., label noise between 0% to 10%).\n\n- Baselines not matching state of the art results. Even with a small scale model such as ResNet-18 (as opposed to state of the art work on self-supervised learning that use ResNet-50 - see one of my “minor” comments below about scaling to larger models) the results previously published by Wang et al. 2024 (table 3) show accuracies that are higher than the ones reported in this work for CIFAR-100 and ImageNet-100 (both for the original model and the NLC ones). Specifically: Wang et al. 2024 reports for CIFAR-100\nContrastive learning (with linear probing) 58.6% vs current work 54.5%\nNon Contrastive learning (with linear probing) 59.7% vs current work 52.8%\nSimilar discrepancies can be found for ImageNet-100 with linear probing, and with both CIfar-100 and ImageNet-100 with full fine tuning using 100% of the datasets.\nMy concerns with these non-matching results are that on the one hand Wang et al. had already shown that NCL is better than traditional contrastive learning even in the original setting, and additionally, it undermine the credibility of the results shown. I would be great to comment on these discrepancies.\n\n- Lack of quantification of “monosemanticity”. The paper rely on models with monosemantic features however the level of monosemanticity is never measured and compared with counterpart models. It would be great to quantitively show that indeed the model that are being produced do contain more monosemantic features. While NCL and SEA were already published and they might have shown their ability to increase monosemanticity, it would still be good to double check this effect. This is even more important with the proposed MonoLoRA since I suppose that no one has ever checked if in this case sparsity also leads to monosemanticity. Additionally an analysis of monosemanticity could help understanding more deeply the differences between NCL and SEA beyond accuracy (which of the two actually leads to more monosemantic representations?).\n\n- None of the experiments show any error bar. It is always good to have multiple runs\n\n- I found the explanation of the LLM experiments lacking necessary details in order to be understood properly. Without reading Appendix A3 (and other still missing details) this section is not self-explanatory. I would encourage rewriting this section with more details. For example, talk about SST2 and Dolly datasets: what kind of tasks are they for? On which metrics are they evaluated? What are the ShieldGemma-9B and Beaverstails-7B alignment scores? \n\n- Additionally, related to the LLM evaluation, the authors mention overfitting as one of the issue with fine-tuning. If overfitting is the concern, I think that it would be good to show that the model has not lost some of its original capabilities (if they are still of interest). Usually this is shown by empirically measuring the change in perplexity, the MMLU benchmark and the common sense reasoning tasks. The current experiments only show the change in “alignment” (which should be better defined) but not other changes that could be affected by overfitting. \n\n\nWhat follows are some suggestions on how to improve the writing, the clarity, and overall the quality of the manuscripts. While there are some questions in there they are not necessarily meant to be answered in the rebuttal but rather to be answered in a new version of the paper. None of the comments below are of major concern, however, addressing them would make this work stronger. Please focus the rebuttal on the questions related to the main concerns.\n\nWriting:\n- Since Wang et al. had already shown that there is no trade off to be paid (in their paper NCL works better than standard SimCLR) I would put the emphasis on showing that monosemanticity could be useful also in terms of robustness. Talking about the trade off seems in contradiction with Wang et al.’s results. \n- Some statements in the abstract should be revised to avoid creating false expectations. For example the authors state “Our preliminary analysis suggests that monosemanticity, by promoting better separation of feature representations, leads to more robust decision boundaries.” This statement seems to be always “true” but from the results presented it seems true only in specific scenarios (high level of noise etc…).\n- “More details can be found in Appendix A3“. When referring to the appendix consider offering more info about which details can be found in the appendix so the reader can decide if they want to read it or not. Are there more results? More background on the experiments? More info on the hyperparameters etc…\n- Figure 4. The pictures in this figure refer to the model with induced monosemanticity or the original model? It could be interesting to compare this result with the “other” model.\n- Figure 4. What does it mean “top activated” samples? What is the value for the highest accuracy group and the lowest accuracy group? Is the issue mono vs poly semanticity of could the issue simply be that the classes that perform poorly have not been properly learnt (that could be indicated by a low magnitude activation?).\n- I would invite the authors to think how much the super-position section (both empirical and “theoretical”) add to the paper. I don’t find it particularly illuminating because it is still based on a very specific set of choices (hidden dimension size, task, etc…). Maybe this space could be used to explain more in details the LLM settings, or present experiments on larger scale, etc.. and this section could be moved to the appendix.\n- In the introduction “Pretrained LLMs need to be carefully finetuned on small-scale language data for different purposes, e.g., instruction following and certain abilities (e.g., reasoning), while avoiding conflicting and forgetting.” What does “conflicting” means in this context? This should be clarified.\n- All the experiments in 3.2.1 could be obtained on the NLC trained backbone or on the original SimCLR backbone. It is unclear what backbone was used both for the standard fine-tuning and the NNT fine-tuning. There are 4 possible combinations (standard SimCLR backbone with standard fine-tuning, standard SimCLR backbone with NNT, NCL backbone with standard fine-tuning and NCL backbone with NNT fine-tuning) it would be great to explicitly say which are being evaluated.\n- Writing: I am assuming that the experiments in 3.2 are different than 3.1 because in 3.2 the full backbone model is fine-tuned while in 3.1 the backbone is frozen. However, this should be mentioned more precisely. Currently the paper says “Unlike linear probing we train classifiers on the pretrained representations without clipping the gradient of encoders”. However, clipping the gradient does not necessarily mean freezing the encoder. I would clarify this aspect.\n- Suggestion related to Figure 2:  I’d suggest to include (maybe in the appendix) an example of a figure perturbed at the different levels used in the experiments so that the reader can appreciate better if that level of noise is realistic and in which scenario this degradation could happen.\n- Page 5 mentions CIFAR-10 but according to the previous explanation I believe this should be CIFAR-100 (Typo)?\n- The feature map analyzed in the case of the vision experiments are those before the logits? This detail was never explicitly mentioned (or I might have missed it).\n- Throughout the paper there are some “concepts” that are never formally defined. For example, the authors often talk about a neuron being activated by single or multi set of concepts. What does it mean to be activated?\n\nOther aspects (not critical but would improve the work):\n- Another potential mismatch is with the SAE definition. In equation (2) the authors define the output for the encoder as  $z(x) = topK(W_{enc}(f(x)-b_{pre}) +b_{enc})$ and the output of the AE as $\\hat{f}(x) = W_{dec}z(z) + b_{pre}$. However, Gao et al. uses a slightly different definition of the topK function which does not include the term $b_{enc}$, specifically, $z(x) = topK(W_{enc}(f(x)-b_{pre}))$. Is this a typo, are the two formulations equivalent, or is this an acltual difference compared to Gao et al. (in which case it would be great to motivate the change)?\n- I believe the experiments with real-world distribution shift are very interesting and relevant. However the authors only reports results on 2 shifts (sketch and stylized). On the one hand the justification about why only these two out of the existing ones in the cited literature would be necessary. On the other hand it would be even better to include all of them and leverage advances in this field and extend the results to all imaginet-testbed shifts from Taori et al 2020 (“Measuring Robustness to Natural Distribution Shifts in Image Classification”). If authors could show a significant gain in robustness with respect to these distribution shifts the paper would be much stronger since some of these shifts are more realistic in my opinion than a 90% noise in the labels.\n- Small scale experiments. It is often the case that when moving from a small scale setting (small model and small dataset) to a large scale many of the advantages observed are unfortunately lost. The authors use ResNet-18 with CIFAR-100 and ImageNet-100 but state of the art in self supervised learning is usually working with ResNet-50 and full ImageNet. I am not sure if the same results will transfer also to larger scale settings. For example, notice how in Figure 3(b) the gap between the accuracy of the polysemantic and the monosemantic models is reducing as you go towards 100% of training data. Had the authors used ResNet50 and full imageNet would you still observe such a difference? Perhaps this indicates that monosemantic features are very important in small scale settings but not in large ones?  This experiment is not crucial but without it I would assume that monosemantic is only useful in a low scale regime. If the authors want to extend the influence of monosemanticity on large scale setting this should be shown.\n- I am not sure I would consider the “theoretical understanding” provided in section 4.3 actually theoretical. It seems to me that is a toy model with concrete values for different choices that can be analyzed manually. It has value but I would consider this still an empirical analysis. \n- From the appendix the authors mentioned that the number of activated features for SAE was 256. How was this choice made and how robust are these models to the choice of this parameter?\n- Why topK and not JumpRelu (Lieberum et al.)? It would be good to motivate this choice.\n- In general the appendix contain most of the hyper-parameters used but I could not find the value of the Temperatures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)