SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathbf{K}_{ii} = 1, \forall i$ . Now, since  $\mathcal{Z}\mathcal{Z}^T \odot \mathbf{K}$  and  $\mathcal{Z}\mathcal{Z}^T$  are positive semi-definite, their corresponding eigenvalues are non-negative,  $\lambda_i(\mathcal{Z}\mathcal{Z}^T \odot \mathbf{K}) \geq 0$  and  $\lambda_i(\mathcal{Z}\mathcal{Z}^T) \geq 0$ . Since the eigenvalues are non-negative, we can extend Equation 4 as follows:
$$
\prod_ {i} \lambda_ {i} (\mathcal {Z Z} ^ {T} \odot \mathbf {K}) \geq \prod_ {i} \lambda_ {i} (\mathcal {Z Z} ^ {T})
$$
$$
\prod_ {i} \left(1 + \frac {d}{n \epsilon^ {2}} \lambda_ {i} (\mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K})\right) \geq \prod_ {i} \left(1 + \frac {d}{n \epsilon^ {2}} \lambda_ {i} (\mathcal {Z} \mathcal {Z} ^ {T})\right) \tag {10}
$$
$$
\det  \left(I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K}\right) \geq \det  \left(I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} \mathcal {Z} ^ {T}\right)
$$
$$
R (\mathcal {Z} | \mathbf {K}) \geq R (\mathcal {Z}),
$$
where the second inequality holds because the affine transform of positive variables preserves inequalities. The equality is satisfied when  $\mathbf{K} = \mathbf{1}\mathbf{1}^T$ .
Lemma 4 (Upper bound for  $R(\mathcal{Z}|\mathbf{K})$ ). For any  $\mathcal{Z} \in \mathbb{R}^{n \times d}$ , any kernel matrix  $\mathbf{K} \in \mathbb{R}^{n \times n}$  using a kernel function satisfying  $k(x, x) = 1$  and  $\epsilon > 0$ , it holds that:
$$
R (\mathcal {Z} | \mathbf {K}) \leq \frac {n}{2} \log_ {2} \left(1 + d / n \epsilon^ {2}\right). \tag {11}
$$
Proof. We start by noting that the Hadamard product of two positive semi-definite matrices  $\mathcal{Z}\mathcal{Z}^T\odot$ $\mathbf{K}\in \mathbb{R}^{n\times n}$  is positive semi-definite (using the Schur product theorem). We also assume that the representations  $z_{i}\in \mathcal{Z}$  are unit normalized, thereby the diagonal entries of  $(\mathcal{Z}\mathcal{Z}^T)_{ii} = 1$ . The diagonal entries  $\mathbf{K}_{ii} = 1$ , which implies  $(\mathcal{Z}\mathcal{Z}^T\odot \mathbf{K})_{ii} = 1$ . Given these facts, we can write the following properties of about the eigenvalues of  $\mathcal{Z}\mathcal{Z}^T\odot \mathbf{K}$ :
$$
\lambda_ {i} \left(\mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K}\right) \geq 0, \sum_ {i} \lambda_ {i} \left(\mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K}\right) = n, \tag {12}
$$
where the second property follows from the fact that  $\mathrm{tr}(\mathcal{Z}\mathcal{Z}^T\odot \mathbf{K}) = n$ . We are interested in finding the maximum value of  $R(\mathcal{Z}|\mathbf{K})$  that can be written as:
$$
R (\mathcal {Z} | \mathbf {K}) = \frac {1}{2} \log_ {2} \prod_ {i = 1} ^ {n} \left(1 + \frac {d}{n \epsilon^ {2}} \lambda_ {i} \left(\mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K}\right)\right). \tag {13}
$$
To maximize  $R(\mathcal{Z}|\mathbf{K})$ , we need to maximize the product within the logarithm. Each term within the product  $1 + \frac{d}{n\epsilon^2}\lambda_i(\mathcal{Z}\mathcal{Z}^T\odot \mathbf{K})\geq 0$  (eigenvalues of a PSD matrix). Using the AM-GM inequality, the product is maximized when all the individual terms are equal,
$$
\lambda_ {i} \left(\mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K}\right) = n / n = 1. \tag {14}
$$
Substituting this result in Equation 13, we obtain the following upper bound:
$$
R (\mathcal {Z} | \mathbf {K}) \leq \frac {n}{2} \log_ {2} (1 + d / n \epsilon^ {2}), \tag {15}
$$
where the equality is achieved when  $\mathcal{Z}\mathcal{Z}^T = I$  when all the representations are orthogonal. Note that this is only possible when  $d\geq n$ .
Proof of Lemma 1. By combining the results of Lemma 3 & 4, we get the following:
$$
R (\mathcal {Z}) \leq R (\mathcal {Z} | \mathbf {K}) \leq \frac {n}{2} \log_ {2} (1 + d / n \epsilon^ {2}). \tag {16}
$$
Lemma 2 (Alignment for random representations). Expected  $A_{k}(f)$  score achieved by a concept erasure framework  $f$  that generates random representations is  $\mathbb{E}[A_k(f)] = k / n$ .
Proof. To prove this, we first assume two randomly generated  $k$ -nearest neighbour graphs (since the original representation is uncorrelated with the randomly generated one we can consider it as random). As it is a  $k$ NN graph, for each node has an expected degree  $\mathbb{E}[d] \approx k$ , where  $d$  is the degree of the node. Now, let's consider the probability of a node  $x_{i}$  being part of node  $x_{j}$ :
$$
p \left(x _ {i} \in \operatorname {k n n} \left(x _ {j}\right)\right) = \frac {d _ {i}}{n} \tag {17}
$$
$$
\mathbb {E} [ p (x _ {i} \in \mathrm {k n n} (x _ {j})) ] = \frac {\mathbb {E} [ d _ {i} ]}{n} = \frac {k}{n},
$$
where  $d_{i}$  is the degree of node  $i$  and  $n$  is the total number of representations. Since computing the exact probability requires knowledge of the degree of the node, we compute the expectation of the same. Next, we compute the probability that node  $i$  is present in both  $k$  NN sets (before and after debiasing) of node  $j$ :
$$
\begin{array}{l} \mathbb {E} [ \operatorname {k n n} (x) \cap \operatorname {k n n} (z) ] = \mathbb {E} \left[ \sum_ {j} p \left(x _ {i} \in \operatorname {k n n} \left(x _ {j}\right) \wedge z _ {i} \in \operatorname {k n n} \left(z _ {j}\right)\right) \right] \\ = \sum_ {j} \mathbb {E} [ p (x _ {i} \in \operatorname {k n n} (x _ {j})) p (z _ {i} \in \operatorname {k n n} (z _ {j})) ] \tag {18} \\ = \sum_ {j} \mathbb {E} \left[ p \left(x _ {i} \in \operatorname {k n n} \left(x _ {j}\right)\right) \right] \mathbb {E} \left[ p \left(z _ {i} \in \operatorname {k n n} \left(z _ {j}\right)\right) \right] \\ = \sum_ {j = 1} ^ {n} k ^ {2} / n ^ {2} = k ^ {2} / n, \\ \end{array}
$$
where the first step utilizes linearity of expectation, and the second step follows from the fact that the degree of distribution of  $\mathcal{X}$  and  $\mathcal{Z}$  are independent. Replacing the result from Eqn 18 in Eqn 6, we get  $\mathbb{E}[A_k(f)] = k / n$ .
A.3 Additional Theoretical Results
Lemma 3 (Upper Bound of  $R(\mathcal{Z}|\mathbf{K})$  for categorical concepts). For categorical concept variables with the kernel values  $\mathbf{K}_{ij} \in \{0,1\}$ ,  $R(\mathcal{Z}|\mathbf{K})$  is bounded by the sum of rate-distortion functions of representation set from individual classes  $\mathcal{Z}_j$
$$
R (\mathcal {Z} | \mathbf {K}) = \sum_ {j = 1} ^ {m} \frac {1}{2} \log_ {2} \det  \left(I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} _ {j} \mathcal {Z} _ {j} ^ {T}\right) \leq \sum_ {j = 1} ^ {m} R (\mathcal {Z} _ {j}), \tag {19}
$$
where the equality holds only when  $\mathcal{Z}_j\mathcal{Z}_j^T = 0, \forall j$  and  $m$  is the number of classes.
Proof. For categorical variables, the kernel function takes the following form:
$$
k (i, j) = \left\{ \begin{array}{l l} 1, & \text {i f} a _ {i} = a _ {j} \\ 0, & \text {i f} a _ {i} \neq a _ {j} \end{array} . \right. \tag {20}
$$
If the kernel function  $k(\cdot ,\cdot)$  is of the above form. Using the corresponding kernel matrix  $\mathbf{K}$  we get,
$$
\mathbf {M} = I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} \mathcal {Z} ^ {T} \odot \mathbf {K} = I + \frac {d}{n \epsilon^ {2}} \left[ \begin{array}{c c c c} \mathcal {Z} _ {1} \mathcal {Z} _ {1} ^ {T} & 0 & \dots & 0 \\ 0 & \mathcal {Z} _ {2} \mathcal {Z} _ {2} ^ {T} & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & \mathcal {Z} _ {k} \mathcal {Z} _ {k} ^ {T} \end{array} \right], \tag {21}
$$
where  $\mathbf{M}$  becomes a block diagonal matrix and  $Z_{i}$ 's are representations belonging to class  $i$ . Using the determinant property of block diagonal matrices, we have:
$$
\log_ {2} \det (\mathbf {M}) = \sum_ {j = 1} ^ {m} \log_ {2} \det \left(I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} _ {j} \mathcal {Z} _ {j} ^ {T}\right) \tag {22}
$$
$$
R (\mathcal {Z} | \mathbf {K}) = \sum_ {j = 1} ^ {m} \frac {1}{2} \log_ {2} \det \left(I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} _ {j} \mathcal {Z} _ {j} ^ {T}\right).
$$
The individual terms in the above summation are closely related to the rate-distortion function of representation belonging to each class,  $j$ , as shown below:
$$
R \left(\mathcal {Z} _ {j}\right) = \frac {1}{2} \log_ {2} \det  \left(I + \frac {d}{n _ {j} \epsilon^ {2}} \mathcal {Z} _ {j} \mathcal {Z} _ {j} ^ {T}\right), \tag {23}
$$
where  $n_j$  is the number of representations in class  $j$ . Note,  $n_j < n$ , where  $n$  is the total number of representations. Using the property that multiplying a matrix with a scalar is equivalent to multiplying its eigenvalues with the same scale, and that  $\mathcal{Z}_j\mathcal{Z}_j^T$  is a PSD matrix. We can show:
$$
R (\mathcal {Z} | \mathbf {K}) \leq \sum_ {j = 1} ^ {m} \frac {1}{2} \log_ {2} \det  \left(I + \frac {d}{n \epsilon^ {2}} \mathcal {Z} _ {j} \mathcal {Z} _ {j} ^ {T}\right) \tag {24}
$$
$$
R (\mathcal {Z} | \mathbf {K}) \leq \sum_ {j = 1} ^ {m} R (\mathcal {Z} _ {j}).
$$
Discussion. Notice that this is closely related to the MCR² objective, which tries to learn discriminative subspaces for individual classes. For concept erasure, we aim for the opposite effect by making instances from the same class dissimilar by maximizing their rate-distortion function.
Algorithm 1 Correlation Computation Routine
1: Input: Input representation set  $\mathcal{X} \in \mathbb{R}^{n \times d}$   
2:  $\mathcal{Y} = \operatorname{sgn}(\mathcal{X}W_1W_2)$  generate labels using random weights  $W_2 \in \mathbb{R}^{d \times m}$ ,  $W_1 \in \mathbb{R}^{m \times 1}$   
3:  $\mathbf{U}, \Sigma, \mathbf{V} = \mathrm{svd}(\mathcal{X})$   
4:  $\mathcal{Z}_0 = \mathcal{X}$  Initialize the representations  
5:  $A = \{\}$ , scores = {} accuracy and alignment sets  
6: for  $i \in \{1, \dots, d\}$  do  
7:  $\mathbf{u} = \frac{\mathbf{V}^T(i)}{\|\mathbf{V}^T(i)\|}$  access the  $i$ -th column of  $\mathbf{V}$   
8:  $\mathbf{P}_i = \mathbf{I}_d - \mathbf{u}\mathbf{u}^T$  null space projection matrix  
9:  $\mathcal{Z}_i = \mathcal{Z}_{i-1}\mathbf{P}_i$   
10:  $A = A \cup \operatorname{acc}(\mathcal{Z}_i, \mathcal{Y})$  compute accuracy  
11: scores = scores ∪  $A_k(\prod_{i=1}^{k} P_i)$  compute alignment scores  
12: end for  
13:  $r = \text{Pearson}(A, \text{scores})$  compute the Pearson correlation  
14: return  $r$
B Alignment Scoring
In this section, we present several measures to capture information alignment and compare them with our proposed metric (in Section 4).
KSG MI estimator [35]. The Kraskov-Stogbauer-Grassberger (KSG) estimator uses the nearest neighbour information in the joint and marginal space to obtain a mutual information estimate. Specifically, it computes the number of neighbours around a point within a hypercube in the marginal spaces. The length of the hypercube is set based on the max-norm distance to the  $k$ -th neighbour in the joint space. The KSG MI estimate between two sets  $\mathcal{X}$  and  $\mathcal{Z}$  can be shown as follows:
$$
I _ {\mathrm {K S G}} (\mathcal {X}, \mathcal {Z}) = \psi (k) - 1 / k - \mathbb {E} [ \psi \left(n _ {x}\right) + \psi \left(n _ {z}\right) ] + \psi (N), \tag {25}
$$
where  $\psi (\cdot)$  is the digamma function,  $n_x$  and  $n_z$  are the number of points in the hypercube of the respective marginal spaces. In our experiment, we use the KSG MI estimator to evaluate the alignment between representation sets before and after concept erasure.
Degree distribution. In a  $k$ -nearest neighbour graph, some nodes are more connected to others (hub nodes) while others are sparsely connected. Building on our intuition of alignment  $A_{k}$  using the nearest neighbour graphs of representations, we can consider changes in its degree distribution,  $D(\mathcal{X})$ , during concept erasure to gauge how the underlying structure of the representation set has changed. We quantify the change using either L1-norm, L2-norm, or KL-divergence between the normalized degree distributions  $D(\mathcal{X})$  and  $D(\mathcal{Z})$ .
Experiments. We perform experiments in a controlled setup to evaluate the efficacy of the proposed alignment measures.
(a) Simulated Erasure. In this experiment, we simulate knowledge erasure from a set of synthetic representations and observe how the alignment scores correlate with the downstream accuracy. Algorithm 1 shows the details for this process. First, we sample a set of representations from a uniform distribution  $\mathcal{X} \sim \mathbb{R}^{n \times d}$  from a uniform distribution and construct a label set  $\mathcal{V}$  (using randomly sampled weights  $W_{1}, W_{2}$ ). In a way, the label set retains some information about the original representations that we will probe as erasure happens. Then, we gradually remove information from representations  $\mathcal{Z}$  by projecting them onto the nullspace  $\mathbf{P}$  formed using the eigenvectors  $\mathbf{u}$ . After each iteration of projection, we compute the accuracy of predicting  $\mathcal{V}$  and alignment score,  $A_{k}$ . We report the Pearson correlation between the accuracies and information alignment in Table 4 (left side), along with the hyperparameter  $k$  used for each measure. We observe that  $A_{k}$  outperforms other approaches achieving better correlation, which showcases the efficacy of our approach.
(a) Correlated Gaussians. In this experiment, we sample two sets of Gaussians (zero mean) with a fixed covariance  $\sigma$ . In this setup, the mutual information has a closed-form solution:
$$
I (\mathcal {X}, \mathcal {Z}) = - \frac {1}{2} \log \left(1 - \sigma^ {2}\right). \tag {26}
$$
We use the samples to compute the different alignment measures and investigate if they're correlated with the actual mutual information (Equation 26). Note that there does not exist an explicit mapping
Metric	Simulated Erasure	Correlated Gaussian
k/n (%)	Pearson (r) ↑	k/n (%)	Pearson (r) ↑
KSG	10	0.965	0.02	0.989
KL-div (degree)	0.1	0.874	0.2	0.490
L2-norm (degree)	0.1	0.865	0.2	0.458
L1-norm (degree)	0.1	0.905	0.2	0.564
Alignment: Ak	50	0.994	50	0.969
Table 4: Comparison of  $A_{k}$  with other alignment measures on synthetic datasets. We observe that  $A_{k}$  achieves the best Pearson correlation scores with downstream accuracy on simulated concept erasure experiments due to the presence of a mapping function  $f$ . In a separate experiment, the KSG estimator shows the highest correlation with MI.  $A_{k}$  also achieves a high correlation score, while the degree distribution-based measures perform poorly due to the lack of a mapping function.
between these samples. In Table 4 (right side), we report the Pearson correlation scores for different measures. We find that the KSG MI estimator outperforms others, with  $A_{k}$  coming in as a close second. This is because our alignment scores assume a 1-to-1 mapping between the sets, which is absent in this case. The degree-distribution-based scores suffer even more as their measure is even more strongly reliant on the mapping. These results show that the alignment score  $A_{k}$  leverages the bijective mapping to generate scores that are well correlated with the mutual information but can be inaccurate in cases where the mapping function is absent.
In this section, we provide various implementation details about our experimental setup. Specifically, we describe the details of the dataset, metrics, and hyperparameters utilized.
In this section, we describe the details of the datasets that were used in the experimental section.
GloVe embeddings [46]. We revisit the problem of deleting gender information (binary attribute) from word embeddings [13]. Specifically, we consider the GloVe embeddings of the 150k most frequently occurring words. For training KRaM, we follow the setup of [49, 18] to select the 7500 most male-biased, female-biased, and neutral words determined by the magnitude of the word vector's projection onto the gender direction (the largest principal component of the space of vectors formed using the difference gendered word vector pairs).
DIAL [12] is a Twitter-based sentiment classification dataset, where each tweet is associated with sentiment labels and "race" information (binary concept label) of the author. The sentiment concept labels are "happy" or "sad" and the binary race concept labels are "African-American English" (AAE) or "Standard American English" (SAE).
Synthetic dataset. We create a dataset where the representations are generated using a continuous latent variable,  $a$ , which serves as our concept label. During data generation, we first sample the latent variable  $a \sim \mathrm{Uni}(0,1)$ , and then sample the high-dimensional representation  $x \sim \mathcal{N}(a\mathbf{1}_d, aI_d)$  where  $\mathbf{1}_d$  is a vector of ones and  $I_d$  is the identity matrix. For this dataset, we set the dimension of the representations to be  $d = 100$ . In this setup, we observe that the latent concept label,  $a$ , is being used to scale the underlying isotropic Gaussian distribution. Therefore, post-concept erasure the representation space should appear like an isotropic Gaussian distribution, which is indeed the case as shown in Figure 5.
UCI Crimes [36]. This dataset<sup>1</sup> contains information about US communities in 1990 from various surveys. The dataset provides 128 attributes (both categorical and continuous variables) from 1,994 different US communities. We concatenate individual attributes of a community to obtain its representation. The regression task involves predicting the number of violent crimes per capita. We consider the ratio of African-American (AAE) people (continuous attribute) in a community as the concept to be erased.
Jigsaw Toxicity Classification [1]. This dataset contains online comments and the binary classification task involves detecting whether a comment is toxic or not. In this dataset, we consider two different concepts: religion and race. We consider a vector-valued protected attribute for this dataset. For the religion concept, we consider an unnormalized vector over the following categories: {'buddhist', 'christian', 'hindu', 'jewish', ' muslim', 'other religion'} . Similarly, for the gender we consider the following categories: {'bisexual', 'female', 'heterosexual', 'homosexual, gay, or lesbian', 'male', 'other gender', 'other sexual orientation', 'transgender'} . During concept erasure of either concept, we only consider instances where at least one of the concept categories has a non-zero value and reserved  $20\%$  of the instances as the test set. This resulted in a dataset with a train/test split of (72k, 18k) for the religion concept and (106k, 26k) for the gender concept. We retrieve text representations for the comments from GPT-3.5 [15] and perform concept erasure on them.
C.2 Metrics
In this section, we present the details of the fairness metrics used in our experiments.
Demographic Parity (DP). Demographic Parity measures the difference in the probability of a prediction w.r.t to the protected attribute  $\mathcal{A}$ . Formally, it is defined as:
$$
\mathrm {D P} = \sum_ {y \in \mathcal {Y}} | p (\hat {y} = y | \mathcal {A} = a) - p (\hat {y} = y | \mathcal {A} = \bar {a}) |, \tag {27}
$$
where  $a, \bar{a}$  are possible values of the binary concept and  $\mathcal{V}$  is the set of possible target attribute labels.
<sup>1</sup>https://archive.ics.uci.edu/ml/datasets/Communities+and+Crime
Generalized Demographic Parity (△GDP). Most of the literature on fairness metrics has focused on categorical variables. We use Generalized Demographic Parity (GDP) [33], which measures the discrepancy in outcome with respect to a continuous variable. GDP measure extends Demographic Parity for continuous protected attributes. It is defined as follows:
$$
\Delta \mathrm {G D P} = \int_ {0} ^ {1} | m (a) - m _ {\text {a v g}} | P (\mathcal {A} = a) d a, \tag {28}
$$
where  $m(a) = \mathbb{E}[\hat{y}|\mathcal{A} = a]$  is expected prediction of the model when protected attribute  $\mathcal{A} = a$ ,  $m_{\mathrm{avg}} = \mathbb{E}[\hat{y}]$  is overall expected prediction, and  $P(\mathcal{A} = a)$  is the probability that the protected attribute takes value  $a$ . The probability density  $P(\cdot)$  can be measured using a histogram or kernel method. We used a kernel function to evaluate the probability density.
In our experiments, we primarily deal with two hyperparameters: regularization constant,  $\lambda$  (in Equation 4), and  $\sigma$ , associated with the standard deviation of a Gaussian kernel  $(k(x,y) = e^{-\|x - y\| / \sigma^2})$ . We set these parameters by performing a grid search on the development set using Weights & Biases [11]. We use a multi-layer neural network with ReLU non-linearity as the erasure function  $f$ . We further perform ablation experiments to understand the impact of these parameters on concept erasure performance (shown in Figure 8). All networks were trained using a single 22GB NVIDIA Quadro RTX 6000 GPU and experiments were executed in PyTorch [44] framework.
In this section, we present additional experiments to analyze KRaM's concept erasure performance.
Vector-valued Concept Erasure. In this section, we present the results of vector-valued gender concept removal from GPT-3.5 text embeddings from the Jigsaw Toxicity classification dataset. We report the MSE and  $\Delta$  GDP results in Figure 7. We observe that KRaM is able to significantly increase the gender MSE while simultaneously reducing the  $\Delta$  GDP scores. During the debiasing process, we observe that there is minimal impact on the toxicity classification accuracy ( $91.9\% \rightarrow 90.1\%$ ).
Figure 7: Vector-valued concept erasure performance using KRaM on Jigsaw toxicity classification dataset (gender concept). We observe a significant reduction in  $\Delta$  GDP scores post erasure of vector-valued gender concept with negligible impact on toxicity classification performance.
Ablation with different kernels. We perform ablation experiments with different kernel functions used to define the  $\mathbf{K}$  and observe its impact on the concept erasure performance. In Table 5, we report the results for erasing the continuous concept on the synthetic dataset. Apart from the kernel function, we use the same hyperparameters in all setups. We observe that KRaM achieves similar concept erasure performance using different kernel functions. We observe that using the Gaussian kernel function in KRaM yields the best erasure performance and alignment score  $A_{k}$  improves when we use a linear erasure function  $f$ .
Method	MSE (a) ↑	Ak ↑
Original	0.006	1.0
KRaM (Laplace)	0.083	0.68
KRaM (Cauchy)	0.092	0.63
KRaMlinear (Gaussian)	0.083	0.75
KRaM (Gaussian)	0.109	0.67
Controlled experiments. In order to better understand concept erasure performance, we evaluate KRaM on axis-aligned data in a controlled setup. We generate a synthetic two-dimensional dataset involving two Gaussians centered at (0, 2) and (0, -2) respectively, and identity covariance. We consider the  $y$ -axis as the continuous concept to be deleted. The desired outcome of deleting an axis-aligned concept in two-dimensional data is that points should lie on a 1D line and the KRaM output should be least correlated with the axis concept that was deleted. That is a reduction in dimension such that most of the data variation should be constrained along a single dimension for this case. We looked at the eigenvalues of both the original data and the concept deleted (by KRaM) output. The fraction of eigenvalue masses in the original data  $(65\%, 35\%)$  and in the KRaM concept deleted output the fraction of masses is  $(99.998\%, 0.002\%)$ . This shows that KRaM is able to effectively delete the target concept resulting in a drop in intrinsic data dimension.
Comparison with MI baselines. We present the results on mutual information (MI) estimation-based baselines on the GloVe dataset. Consistent with the results in Table 3, we observe that other MI estimation baselines are either unable to erase concepts robustly (MIFR) or end up erasing a significant amount of information along with the concept (CCL and ICVAE) in Table 6. In contrast to these approaches, we observe that our rate-distortion based framework, KRaM, is able to achieve a good balance between concept erasure and retaining original information (indicated by  $A_{k}$ ).