SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Empirically, we find that  $A_{k}$  is well correlated with the downstream performance of  $\mathcal{Z}$ . Specifically, we perform a synthetic experiment to simulate concept erasure and assess the efficacy of  $A_{k}$  in capturing the alignment of  $\mathcal{Z}$ , where we sample a representation set  $(x\sim \mathbb{R}^{100})$  and their corresponding labels. We remove information from these representations by projecting them onto nullspaces of its dominant eigenvectors (details provided in Algorithm 1). During this process, we measure the prediction accuracy for the original labels
and  $A_{k}$  scores (shown in Figure 2). We find the  $A_{k}$  is highly correlated with the prediction accuracy achieving Pearson correlation scores  $\sim 0.99$  (the average correlation over multiple runs). We also compare  $A_{k}$  with a few different alignment measures (including MI estimates) and find that our method outperforms others (more details in Appendix B).
Lemma 2 (Alignment for random representations). Expected alignment score achieved by a concept erasure framework  $f$  that generates random representations is  $\mathbb{E}[A_k(f)] = k / n$ .
The proof is provided in Appendix A.2. This result shows the importance of choosing  $k$ . If  $k$  is too small, the  $A_{k}(f)$  scores may be low for many concept erasure functions. Conversely, if  $k \approx n$ , then the  $A_{k}(f)$  scores will almost always be close to 1. In our experiments (Figure 2), we find that  $A_{k}$ 's correlation is maximized when  $k = 0.5n$ .
Figure 2: Correlation of alignment score,  $A_{k}(f)$ , with accuracy for a synthetic dataset. For different  $k$  values,  $A_{k}(f)$  achieve high Pearson correlation  $\sim 0.99$ .
In this section, we provide the specifics of the experimental setup and evaluation results for concept erasure using KRaM across various datasets. The implementation of KRaM is publicly available at https://github.com/brcsomnath/KRaM.
Setup. In all settings, we follow the same routine for concept erasure: (a) we obtain representations either directly from the dataset or an encoder (e.g., BERT, GPT-3), which are kept frozen; (b) we perform concept erasure in a post-hoc manner to obtain representations  $f(x) \in \mathcal{Z}$ , where  $f$  is a non-linear neural network; and (c) we use  $f(x)$  on downstream tasks and report the metrics.
Datasets. We assess the effectiveness of KRaM in erasing 3 types of concept variables: (a) categorical concepts - we apply KRaM to erase binary gender variables from GloVe embeddings and race from BERT embeddings for tweets in the DIAL dataset [7]; (b) continuous concepts - we evaluate KRaM on a synthetic dataset, generated using a continuous latent variable, and UCI Crimes [36]. For these datasets, we treat one of the latent continuous variables and African American (AAE) population ratio as the concepts to be erased, respectively; (c) vector-valued concepts - we evaluate on Jigsaw toxicity detection dataset [1], where we consider religion and gender (which are vector-valued variables) as the concepts to be erased from GPT-3.5 [15] embeddings. We present additional details in Appendix C.1.
Baselines. We compare KRaM with the following baselines: (a) INLP [49] (linear) iteratively projects representations onto the nullspace of optimal separating linear subspaces; (b) RLACE [50] (linear) is an extension of INLP that performs concept erasure by solving minimax game; (c) KCE [51] (non-linear) presents a kernelized version of the minimax game introduced in RLACE; (d) FaRM [18] (non-linear) employs rate-distortion maximization for erasing categorical concepts ; (e) KRaMlinear uses KRaM's with a linear erasure function  $f$ . To the best of our knowledge, there are no existing methods for continuous or vector-valued concept erasure. For continuous concepts, we normalize the labels and quantize them into  $n_b$  bins (a hyperparameter). We denote a concept erasure method as MethodQ whenever quantization is used. For vector-valued concepts, we extend nullspace projection-based
Figure 3: Vector-valued concept (religion) erasure performance using KRaM on Jigsaw toxicity dataset. KRaM achieves better performance ( $\uparrow$  MSE &  $\downarrow$  ΔGDP) than baseline approaches in most settings.
erasure techniques by quantizing each dimension and projecting onto a series of nullspaces. It is unclear how to utilize non-projection-based methods for vector-valued concept erasure.
Metrics. Following previous work [18, 50], we assess concept erasure quality using the following:
Probing representations. We use a scikit-learn MLP classifier (non-linear) [45] to probe  $f(x)$ , report classification accuracy for categorical attributes, MSE for continuous and vector-valued attributes (in a dimension-wise manner). A better concept erasure method would have low accuracy and a high MSE for the deleted concept.
Downstream metrics. For datasets with a downstream task, we evaluate if the deleted concept still affects the task by measuring statistical parity. We report DP for categorical concepts,  $\Delta$ GDP [33] (see Appendix C.2) for continuous concepts, and  $\Delta$ GDP values for each dimension of vector-valued concepts. Lower statistical parity scores are expected after concept erasure. However, we note that concept erasure does not necessarily guarantee fairness [16]. Applying these methods for fairness would require a more application-specific analysis of the risks and biases.
Alignment. We report the alignment scores ( $A_{k}$  for  $k = 0.5n$ , therefore  $A_{k} \in [0.5,1]$ ) wherever a downstream task is absent. Higher  $A_{k}$  scores are expected. If a downstream task is available, we probe  $f(x)$  for that task, where we expect high accuracy or low MSE scores.
In this section, we report the performance of KRaM in erasing different types of concepts: categorical, continuous, and vector-valued variables. Note that the primary objective of concept erasure is to robustly erase the concept variable (by achieving lower probing and fairness metrics) even if that reduces the utility of the representations to some extent. This is different from adversarial learning where we try to achieve a balance between fairness and utility. In our experiments, we show KRaM is able to robustly erase concepts while retaining a significant amount of original information.
Vector-valued Concept Erasure. Erasure of vector-valued concepts is useful when the attribute to be deleted is not available in the form of a categorical or normalized continuous variable. We evaluate KRaM on erasing information about religion, a vector-valued concept, in the Jigsaw toxicity dataset [1], where the downstream task involves detecting whether an online comment is toxic. Each text is annotated with a vector-valued concept: religion with scores over the categories (\{'buddhist', 'christian', 'hindu', 'jewish', ' muslim', 'others'\}). We obtain text representations from GPT-3.5 API and use a RBF kernel with a cosine distance function. In Figure 3, we report the MSE and  $\Delta$  GDP scores for KRaM along with other baselines. We observe that KRaM performs the best achieving low  $\Delta$  GDP with high MSE scores across most concept labels. We also observe that the change in toxicity classification accuracy (93.2% → 92.1%) is minimal during concept erasure. This showcases the efficacy of KRaM in erasing vector-valued attributes as it achieves up to 76% MSE gains over the best baselines. We also report the results for vector-valued gender erasure in Appendix D.
Continuous Concept Erasure. We evaluate the efficacy of KRaM in erasing continuous concepts on a synthetic dataset and UCI Crimes. We compare with baseline approaches that use quantized concept labels. In Table 1 (left), we report the results on the synthetic dataset and observe that KRaM performs the best in preventing leakage of  $a$  (high MSE scores) while achieving considerable alignment score,  $A_{k}$ . While both FaRM and KRaM utilize non-linear erasure functions, a necessity for robust concept erasure, they tend to achieve relatively lower  $A_{k}$  scores. Despite this, it is important to note that significant information can still be preserved through non-linear warping, which we show
Method	Synthetic	UCI Crimes
MSE (a) ↑	Ak ↑	Rank ↑	MSE (y) ↓	MSE (a) ↑	ΔGDP ↓	Ak ↑
Original	0.006	1.0	100	0.046	0.030	0.058	1.0
Random	0.174	0.50	100	0.211	0.251	0.006	0.50
INLPQ [49]	0.084	0.85	100	0.055	0.056	0.0	0.90
RLACEQ [50]	0.021	0.87	100	0.038	0.022	0.051	0.81
FaRMQ [18]	0.068	0.74	100	0.050	0.064	0.013	0.62
KRaM	0.109	0.67	100	0.069	0.104	0.001	0.59
KRaMlinear	0.083	0.75	100	0.067	0.082	0.022	0.69
Table 1: Continuous concept erasure: We evaluate on the synthetic and UCI Crimes. Post concept erasure using KRaM, we observe a significant increase in MSE  $(a)$  combined with a drop in  $\Delta$  GDP.
Method	DIAL	Glove
Acc. (y) ↑	Acc. (a) ↓	DP ↓	Acc. (a) ↓	Ak ↑	Rank ↑
Original	75.5	87.7	0.26	100.0	1.0	300
Random	50.8	50.5	0.01	50.2	0.50	300
INLP [49]	75.1	69.5	0.16	86.3	0.85	210
RLACE [50]	75.5	82.1	0.18	95.5	0.93	300
KCE [51]	75.0	80.1	0.12	63.5	0.62	100
FaRM [18]	74.8	54.2	0.09	53.9	0.65	247
KRaM	72.4	54.0	0.08	52.6	0.65	246
KRaMlinear	75.4	67.5	0.18	67.0	0.73	130
Table 2: Categorical concept erasure: We assess binary gender and race erasure from GloVe and BERT representations (from DIAL) respectively. We denote the top 3 results for any metric using  $\mathbb{P}$ , and  $\mathbb{P}$  respectively. Desired trends for all metrics are shown using  $\uparrow$  or  $\downarrow$ .
in the categorical experiments. Note that linear erasure functions are able to retain nearest neighbour structures better, thereby achieving higher  $A_{k}$  scores. In Figure 5, we visualize the UMAP projection of synthetic data, where the representations' position is indicative of the latent continuous concept attribute prior to concept erasure (left). Post concept erasure (right), we observe no such discernible correlation. For UCI Crimes, we perform erasure for the African-American (AAE) population ratio, and use the generated representations to predict the normalized number of crimes per capita ( $y$ ). Table 1 (right) shows that KRaM generates representations with minimal information about AAE ratio (high MSE ( $a$ ) and low ΔGDP scores ( $\Delta GDP \sim 0$ ). These experiments showcase KRaM's efficacy in erasing continuous attributes and minimizing their impact on downstream tasks (low ΔGDP scores).
Categorical Concept Erasure. In Table 2, we evaluate categorical concept erasure on DIAL tweet classification (race) and GloVe (gender) datasets. For DIAL dataset, we obtain BERT [34] representations of tweets, perform concept erasure for race (binary) attribute, and use the generated representations  $f(x)$  for sentiment classification. We report the accuracy of predicting sentiment  $(y)$ , race  $(a)$ , and demographic parity (DP) of the predictions in Table 2 (left). We observe that KRaM performs the best in erasing race attribute (evident from high Acc.  $(a)$ ) and demographic parity while attaining comparable accuracy on sentiment classification (Acc.  $(y)$ ). For GloVe embeddings (Table 2 (right)), we observe that KRaM achieves the state-of-the-art result in suppressing the gender leakage (probing accuracy for predicting binary gender attribute). We observe that linear techniques INLP and RLACE, obtain high alignment scores,  $A_{k}$ , but their representations still contain significant gender information (indicated by high Acc.  $(a)$ ). This shows a trade-off between information alignment and concept erasure, where robustly erasing a concept may also result in the removal of other information. Categorical concept erasure has been extensively studied, and the fact that a general erasure framework, KRaM, can perform on par with state-of-the-art methods underscores its efficacy.
In this section, we perform several analysis experiments to understand the functioning of KRaM.
Comparison with MI estimation techniques. In this experiment, we compare our concept erasure framework with a few state-of-the-art mutual information (MI) estimation approaches. Essentially,
Dataset	Acc. (a) ↓	Ak↑	WS-353 ↑
GloVe	100.0	1.0	0.70
InfoNCE [43]	64.0	0.55	0.19
MINE [9]	76.8	0.54	0.03
CLUB [17]	98.1	0.55	0.06
KNIFE [47]	50.2	0.53	0.10
KRaM	52.6	0.65	0.48
Figure 5: Visualization of UMAP projection of representations obtained from the synthetic dataset before and after concept erasure. Concept erasure makes the positions of representations uncorrelated with their continuous concept labels (denoted by their color).
the task of concept erasure can be formalized as maximizing the objective:  $I(\mathcal{Z},\mathcal{X}) - I(\mathcal{Z},\mathcal{A})$ , where  $I(\cdot ,\cdot)$  denotes the mutual information between two sets. We optimize this objective function with the following MI estimates: InfoNCE [43], MINE [9], CLUB [17], and KNIFE [47]. In Table 4, we report the gender accuracy (Acc.  $(a)$ ),  $A_{k}$ , and performance on WordSim-353 benchmark [2], which is also reflective of the alignment, on GloVe dataset. We observe that MI techniques lose significant information from the original representations achieving near-random  $A_{k}$  and WS-353 scores. We believe this happens because MI approaches optimize their estimation parameters along with the erasure function  $f$ , which is difficult.
We observe a unique scenario for the CLUB method, where the Acc.  $(a)$  is high and  $A_{k}$  is low. This implies that the clusters (related to different genders) may be retained but the nearest neighbour structure within the clusters is perturbed significantly. Probing for the downstream task (Acc.  $(a)$ ) may give you the impression that information from the original space is retained, while  $A_{k}$  provides a more fine-grained view contradicting such an incorrect conclusion. In general, we find that  $A_{k}$  values are relatively well correlated with WS-353 scores, which is a good measure of the information retained in the representation space after erasure. However, computing WS-353 requires additional annotation that may not be feasible for representation sets other than word embeddings.
We also compare KRaM with several other mutual information estimation based baselines that use a unique objective function for controlling information in representations. Specifically, we compare with MIFR [55], CCL [56], and ICVAE [42] and report the results on DIAL dataset in Table 3. Note that all of these methods work for categorical concepts only, and ICVAE requires access to concept labels for test instances as well, a limitation compared to KRaM and the other methods. In Table 3, we observe MIFR is unable to erase concepts robustly (based on Acc.  $(a)$  and DP scores). ICVAE and CCL are able to delete concepts but at the significant cost of deleting a lot of original
Figure 4: Comparison of KRaM with state-of-the-art mutual information (MI) estimation methods. KRaM achieves a balance between good concept erasure nearly with high  $A_{k}$ .
Method	Acc (y) ↑	Acc (a) ↓	DP ↓
Original	75.5	87.7	0.26
MIFR [55]	75.4	68.7	0.21
CCL [56]	50.7	52.6	0.01
ICVAE [42]	66.5	53.3	0.10
KRaM	72.4	54.0	0.08
Table 3: Comparison of KRaM with mutual information based baseline approaches on DIAL dataset. We observe that KRaM achieves a fine balance between concept erasure and retaining task performance.
information. Loss of information from the original space (low Acc.  $(y)$ ) using ICVAE and CCL is quite similar to other MI methods reported in Figure 4. Compared to these methods, KRaM is able to get similar concept erasure performance while achieving much higher Acc.  $(y)$  scores. We report additional results using these baselines on the GloVe dataset in Appendix D.
Image-based datasets. We perform experiments on image-based datasets to evaluate the efficacy of KRaM on different domains. We consider two different setups using CelebA [37] and Colored MNIST [6] datasets. In CelebA, we consider the binary variable attractiveness as the target attribute and whether the face has makeup applied (binary variable) as the protected attribute. With concept erasure using KRaM, the accuracy of predicting the makeup attribute dropped from  $85.7\%$  to  $69.6\%$ . The demographic parity of the predictions (for attractiveness) also improved from 0.94 to 0.54. This shows the effectiveness of KRaM in removing the makeup concept.
For Colored MNIST, we follow the setup of [6] to create a biased version of the MNIST dataset [22]. The background color of the digits is made to be well correlated with the digit in the training set. However, no such correlation exists in the test set. We use a 0.8 correlation between the background
Figure 6: Evolution of  $R(\mathcal{Z})$  and  $R(\mathcal{Z}|\mathbf{K})$  under different setups. We monitor the impact on the volume of the overall feature space in different settings of the objective function.
color and the digit label. We treat background color as the concept to be erased. Naively training a classifier on the images will perform poorly on the test set as the classifier can easily overfit the background color. The digit information could be predicted with an accuracy of  $79.9\%$ . Post-concept erasure we observe that the classifier's performance improves from  $79.9\%$  to  $87.1\%$ . This shows the effectiveness of KRaM to remove background information and help the classifier generalize better.
Evolution of loss functions. In this experiment, we investigate the evolution of loss terms  $R(\mathcal{Z})$  and  $R(\mathcal{Z}|\mathbf{K})$  under various settings of the objective function (described in Figure 1) on the synthetic dataset. In Figure 6 (a), we observe that indeed maximizing the kernelized rate-distortion function also leads to an increase in  $R(\mathcal{Z})$ , which can result in the concept variable not being completely erased. In Figure 6 (b), we examine the scenario where  $R(\mathcal{Z}|\mathbf{K}) - R(\mathcal{Z})$  is maximized, where we observe indicate a substantial drop in  $R(\mathcal{Z})$ . This reduction often implies a low intrinsic dimension and a substantial information loss. In Figure 6 (c), we present the evolution of loss terms for KRaM's objective. We notice that  $R(\mathcal{Z})$  aligns closely with the initial number of bits, denoted as  $b$ . Meanwhile, the  $R(\mathcal{Z}|\mathbf{K})$  is maximized to its full extent (similar to Figure 6 (a)). Through these experiments, we provide empirical verification for the insights discussed in relation to Figure 1.
We conduct additional ablation experiments that involve varying hyperparameters and kernel functions. The results of these experiments are reported in Appendix D.
In this paper, we proposed KRaM, a novel framework to robustly perform concept erasure from a representation set. KRaM uses a kernelized formulation of the rate-distortion function, where the kernel is created using concept labels. This approach ensures that instances with similar concept labels become dissimilar in the representation space, which ultimately results in the erasure of the concept variable. KRaM is a versatile method capable of erasing a wide range of concepts, including categorical, continuous, and vector-valued variables. We theoretically analyze several properties of the proposed KRaM objective. Empirical evaluation shows the efficacy of KRaM on a wide range of setups ranging from gender erasure from GloVe embeddings to vector-valued concept erasure from GPT-3.5 embeddings. We also propose a heuristic-based measure to capture the information alignment of the erasure function  $f$  by analyzing the  $k$ -nearest neighbours of the representations. While KRaM effectively erases concepts, it does result in the loss of some information from the original space, as evidenced by the alignment scores. Determining the minimum amount of information that must be distorted to fully erase a concept remains an open question. Future research could concentrate on gaining a deeper understanding of this issue and developing techniques that can erase concepts from representations while having minimal impact on alignment with original representations.
The authors are thankful to Alex Beutel (OpenAI previously Google Research) and Kelsey Allen (Google DeepMind) for helpful feedback on an earlier version of this paper. The work of Somnath Basu Roy Chowdhury and Snigdha Chaturvedi was supported in part by Amazon Research Awards.
[1] Jigsaw unintended bias in toxicity classification. 2019.
[50] Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning, pages 18400-18421. PMLR, 2022.
[51] Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6034-6055, 2022.
[52] Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Kernelized concept erasure. In EMNLP 2022. arXiv, 2022.
[53] James R Schott. Matrix analysis for statistics. John Wiley & Sons, 2016.
[54] Jssai Schur. Bemerkungen zur theorie der beschränkten bilinearformen mit unendlich vielen veränderlichen. 1911.
[55] Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2164-2173. PMLR, 2019.
[56] Yao-Hung Hubert Tsai, Martin Q Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning: Removing undesirable information in self-supervised representations. arXiv e-prints, pages arXiv-2106, 2021.
[57] Laura Wendlandt, Jonathan K. Kummerfeld, and Rada Mihalcea. Factors influencing the surprising instability of word embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2092-2102, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[58] Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In NIPS, 2017.
[59] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9434, 2020.
[60] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning, pages 325-333. PMLR, 2013.
We prove the general bounds for  $R(\mathcal{Z}|\mathbf{K})$  by independently proving the lower and upper bound using the following intermediate results.
Lemma 3 (Lower bound for  $R(\mathcal{Z}|\mathbf{K})$ ). For any  $\mathcal{Z} \in \mathbb{R}^{n \times d}$ , a kernel matrix  $\mathbf{K} \in \mathbb{R}^{n \times n}$  using a kernel function satisfying  $k(x, x) = 1$  and  $\epsilon > 0$ , it holds that:
$$
R (\mathcal {Z} | \mathbf {K}) \geq R (\mathcal {Z}), \tag {7}
$$
where the equality is satisfied only when  $\mathbf{K} = \mathbf{1}\mathbf{1}^T$ .
Proof. We start off by writing down the expanded form of  $R(\mathcal{Z}|\mathbf{K})$  as:
$$
R (\mathcal {Z} | \mathbf {K}) = \frac {1}{2} \log_ {2} \det  \left(I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K}\right). \tag {8}
$$
In the above equation 8, we first note that both  $\mathcal{Z}\mathcal{Z}^T$  and  $\mathbf{K}$  matrices are positive-semi definite symmetric matrices. Using Schur product theorem [54], we can show that their hadamard product  $\mathcal{Z}\mathcal{Z}^T\odot \mathbf{K}$  is also positive semi-definite (for  $d > 1$ ). Next, we utilize the following property for Hadamard products:
Theorem 7.25 [53]. Given two positive semi-definite square matrices  $A$  and  $B$  of dimension  $m$ . Then, the following property holds:  $\det(A \odot B) \geq \det(A) \prod_{i=1}^{m} b_{ii}$
Applying this property to  $\mathcal{Z}\mathcal{Z}^T\odot \mathbf{K}$ , we get the following result:
$$
\det  \left(\mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K}\right) \geq \det  \left(\mathcal {Z} \mathcal {Z} ^ {T}\right), \tag {9}
$$