Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure
Authors: Ramin Akbari
Milad Afshari
Vishnu Boddeti
Venue: NeurIPS 2025 poster ()
Keywords: Concept Erasure
kernel methods
adversarial training
model bias/unfairness mitigation
representation learning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We introduce Obliviator, a nonlinear concept erasure method that guards against nonlinear adversaries. For any level of unwanted attribute protection, our method achieves higher task performance, revealing an empirical upper bound on this trade-off.

Abstract: Concept erasure aims to remove unwanted attributes, such as social or demographic factors, from learned representations, while preserving their task-relevant utility. While the goal of concept erasure is protection against all adversaries, existing methods remain vulnerable to nonlinear ones. This vulnerability arises from their failure to fully capture the complex, nonlinear statistical dependencies between learned representations and unwanted attributes. Moreover, although the existence of a trade-off between utility and erasure is expected, its progression during the erasure process, i.e., the cost of erasure, remains unstudied. In this work, we introduce Obliviator, a post-hoc erasure method designed to fully capture nonlinear statistical dependencies. We formulate erasure from a functional perspective, leading to an optimization problem involving a composition of kernels that lacks a closed-form solution. Instead of solving this problem in a single shot, we adopt an iterative approach that gradually morphs the feature space to achieve a more utility-preserving erasure. Unlike prior methods, Obliviator guards unwanted attribute against nonlinear adversaries. Our gradual approach quantifies the cost of nonlinear guardedness and reveals the dynamics between attribute protection and utility-preservation over the course of erasure. The utility-erasure trade-off curves obtained by Obliviator outperform the baselines and demonstrate its strong generalizability: its erasure becomes more utility-preserving when applied to the better-disentangled representations learned by more capable models.

Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure
Ramin Akbari'
Milad Afshari"
Vishnu Naresh Boddeti
Michigan State University {akbarigh, afsharim, vishnu)@msu.edt edu
Abstract
Concept erasure aims to remove unwanted attributes, such as social demographic factors, from learned representations, while preserving their task-relevant utility. While the goal of concept erasure is protection against all adversaries, existing methods remain vulnerable to nonlinear ones. This vulnerability arises from their failure to fully capture the complex, nonlinear statistical dependencies between learned representations and unwanted attributes. Moreover, although the existence of trade-off between utility and erasure is expected, its progression during the erasure process, i.e., the cost erasure, remains unstudied. In this work, we introduce Obliviator, a post-hoc erasure method designed to fully capture nonlinear statistical dependencies. We formulate erasure from a functional perspective, leading optimization problem involving a composition of kernels that lacks closed-form solution. Instead of solving this problem a single shot, we adopt iterative approach that gradually morphs the feature space achieve hüility-resseving erasure. Unlike prior methods, Obliviator guards unwanted attribute against nonlinear adversaries. Our gradual approach quantifies the cost of nonlinear guardedness and reveals the dynamics between attribute protection and priityy-reeervtionn over the course of erasure. The utility-erasume trade-off curves obtained by Obliviator outperform the baselines and demonstrate its strong generalizability: its erasure becomes more better-disentangled representations learned by more capable models.
when applied to the
1 Introduction
Pretrained Language Models (PLMs) have become central Natural Language Processing (NLP), achieving remarkable performance across various tasks. However, they often encode unwanted concepts, such as demographic or social attributes, leading to biases and potentially unfair predictions [31, 26). Concept erasure aims remove such undesirable concepts from the learned representa- tions [27]. either supervised, leveraging both task and sensitive labels or unsupervised, using only sensitive labels.
Current concept erasure methods fall short of fully removing undesired attributes. The task involves balancing two often competing objectives: (1) removing unwanted concepts, and (2) preserving as much information as possible. Early methods relied on linear projection techniques [5, 15, 26, 27] applied in post-hoe manner, i.e.. modify learned representations without needing direct access to the PLMs, offering simplicity and efficiency. Later methods shifted to addressing concept erasure in nonlinear manner [28, 30, 6, 2]: however, they remain limited in capturing
'Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Female
Male
FaRM
Ads
kSAL
Obliviator
Professor
Physician
Figure 1: Erasure of Gender from Representation on BIAS IN Bios. Embeddings from a nonlinear adversary trained to extract gender information from the erased representation. Existing nonlinear methods fail to fully protect gender, as gender-specific distributions within each profession remain distinguishable. In contrast, Obliviator effectively guards gender by overlapping representations across gender, while preserving separability by profession.
nonlinear dependencies and are vulnerable to nonlinear adversaries. Even with costly fine-tuning of PLMs, will show in this work ($5), erasure is not complete. Moreover, the dynamics of the competition between objectives, i.e. the progressive loss and gain in each objective throughout the erasure process, and how they shape the resultant utility -erasue trade-off, have not been explored.
This paper studies the cost nonlinearly guarded erasure. We propose Obliviator, which performs erasure by minimizing the statistical dependency between learned PL.M representations and unwanted attributes. Our formulated erasure problem solves a multi-ohjective optimization with competing objectives. Solving this optimization in single shot is challenging and likely leads to a poor solution. Instead, Obliviator solves this problem using a two-step iterative procedure, enabling us to gradually morph the feature space to obtain a more utility-preserving erasure. We utilize the Reproducing Kernel Hilbert Space (RKHS), which allows us to capture nonlinear statistical dependencies a closed form, facilitating the optimization. Our stable optimization approach provides reliable way to study the dynamics of the objectives during the erasure process, known as the utility- trade-off. Through these trade-off curves, across various experiments, demonstrate Obliviator outperforms baselines while guarding against nonlinear adversaries (see Figure 1). Moreover, its erasure becomes more preeeeripeee when applied to the better-disentangled representations learned by more capable PLMs.
Summary of Contributions:
1. We introduce Obliviator, a post-hoe erasure method that captures the nonlinear dependency of sensitive attributes to original representation. Obliviator guards unwanted attributes against nonlinear adversaries.($4)
2. Our stable optimization approach provides a reliable way to study the dynamics of the utility- erasure trade-off. Across all experimental setups, Obliviator consistently achieves higher task performance at every level of erasure, setting a strong benchmark for this trade-off. ($5.1)
3. We illustrate the generalizability of Obliviator through experiments across various language models. We observe that its erasure becomes more utility-prese preservveg when applied to the better-disentangled representations learned by more capable (5M5.
Related Works
Removal of sensitive attributes from learned representations was first explored in the context of fair classification [33]. The task, often termed concept erasure, gained prominence with applications such as removing binary gender labels from GloVe embeddings [22], and has since evolved into a broader research area at the intersection of fairness, interpretability, and adversarial representation learning. Existing approaches can be categorized as post-hoc and fine-tuned. Post-hoc methods directly modify learned representations without altering the parameters of the backbone model. In contrast, fine-tuned approaches require access to the model and enforce erasure by finetuning its parameters. A summary of concept-erasure methods can be found in Table 1.
Linearly Guarded Erasure: Early concept removal methods were linear, relying on identifying a direction associated with the unwanted concept and subtracting it from the representations [5, 15]. Iterative Nullspace Projection (INLP) improves on this by repeatedly projecting representations onto the nullspace of linear classifiers trained to predict the sensitive attribute, gradually removing its influence [26]. Relaxed Linear Adversarial Concept Erasure (R-LACE) formulates the task as minimax game, where adversaria classifier attempts to recover the concept being erased [27].

---

Table 1: Characteristics of Existing Concept Erasure Methods.
Method
UnSupervised
Supervised
Erasure Model
INLP [26]
Post-hoe
Post-hoc
Linear/lterative
KSAL/SAL [30]
Post-hoc
Post-hoc
Nonlincar
AdS [1]
Fine-tuning
Nonlinear
FaRM [6]
Post-hoc
Fine-tuning
Nonlincar
KRaM [2]
Post-hoc
Post-hoc
Nonlinear
Obliviator [Ours]
Post-hoc
Post-hoc
Nonlinear/Iteratie
LEAst- squures Concept Erasure (LEACE) takes a least-squares approach, yielding an oblique projec- tion for linear concept removal [3]. Spectral Attribute removal. (SAL) uses matrix decomposition to project representations into directions with lower covariance with the sensitive labels [30]. However, none of these methods is robust against nonlinear adversaries.
Nonlinearly Guarded Erasure: Kernel-based techniques have been used to extend linearly guarded erasure methods nonlinear ones. For example, Kernelized Concept Erasure (KCE) and kSAL are kernelized versions of R-LACE and SAL, respectively, However, these methods only protects the unwanted attributes against a specific class of adversaries and fail to ensure complete nonlinearly guarded erasure [28, 30]. Adversarial training has emerged as a common technique for concept erasure [12, 20, 32, 11. In this method, an adversarial network is trained predict the sensitive attribute, while the primary model learns to suppress it and preserve task-relevant information. A representative example Adversarial Scrubbing (AdS) [1]. However, adversarial removal is often incomplete, and such models can be challenging to train effectively [13]. Another line of work for nonlinear concept crasure is based on rate-distortion maximization, as in FaRM [6], and its kernelized variant, KRaM [7]. Notably, both AdS and FaRM require fine-tuning PLMs. However, we show that even these approaches are still vulnerable to nonlinear adversaries and protected attributes can still be partially recovered using proper classifier.
Obliviator addresses the gap in nonlinear guardedness by directly targeting the nonlinear statistical dependency between unwanted attributes and learned representations. We adopt a functional per- spective to formulate this dependency. Similar to prior work [10, 19], we use the Hilbert-Schmin Independence Criterion (HSIC) proxy to minimize this dependency. We propose two-step iterative optimization that minimizes this quantity while enabling more quility-preserve erasure.
3 Statistical Dependence Measures
In this section, we explain how a linear statistical test can be derived from linear functions. We then extend this approach to nonlinear test with functions from RKHS, leading to the definition of HSIC that captures nonlinear dependence of random variables and has a closed-for form solution.
We denote scalars and functions using lowercase letters, e.g., n and f. For deterministic vectors boldface lowercase letters, e.g., æ, y is used. Both scalar-valued and multidimensional random variables (RV)s are denoted by regular upper case letters, e.g., X, Y. We denote deterministic matrices by boldface upper case letters, e.g., K, L. Sets are showed by calligraphic letters, e.g.. F.
Notation:
3.1 Dependence via Witness Functions
Linear Statistical Dependence: Let X R" and Y R. be two RVs with the joint distribution px,.. To design a linear statistical test, we can look for subspaces u and v that maximize the correlation of the projected RVs. Therefore, we can write:
sup sup E = o Cynu, s.t ||u||a = ||0||2 =
(1)
where E is the cross-covariance matrix, and - E[ shows a centered RV. This optimization is bounded by the largest singular value of Cyar. with and its right and left singular vectors. Equivalently, one may consider all non-zero singular values of the cross-covariance matrix and introduce their sum of square as a Linear Independence Metric (LIM):
LIM Y) = Σ 0. = |
(2)
where ||-Ile denotes Frobenius norm and is the rank of Cy..- This form of measuring dependence similar to SAL [30] but fails to capture all modes of dependency when the dependence between X and Y is non-linean
3

---

Nonlinear Statistical Dependence: We now extend the linear test, by replacing u and with nonlinear functions. Let F + 6(x) and
- (zy) be RKHSs where ((.) and ((.) are the corresponding feature maps. Also, let (, be an inner product for the function space H. For nonlinear statistical test, we look for a pair of functions f E TEF: TR R and g E Eg R _> that maximize the following objective:
sup sup (lg(Y) (g, Covy=ol s.t Il9l|a - 11f1==
(3)
f
Here, [(X)== ((X) -L((()) and Cov EYE x()) =F G is the cross-covariance operator, where @ denotes the tensor product, which extends the outer product to function spaces. The functions and g. act as witness functions that aim to transform the dependency between the input variables X and Y into an approximately linear relationship in the transformed spaces f(X) and g(Y), The unit norm constraint on these functions encourages smoothness. Since the underlying data distribution is unknown in practice, this constraint is essential for reliably estimating the objective from finite samples. Without such a constraint, it is possible to find very complex functions that produce a nonzero objective even when X and Y are independent." To ensure the test captures meaningful dependency from finite samples, we assume that any dependency between X and Y exhibits a smooth structure. This smoothness can also be influenced by the choice of the underlying feature map in the RKHS.
3.2 Hilbert-Schmidt Independence Criterion
The objective in (3) has a closed-form solution, where f and g are the right and left singular functions of the cross-covariance operator [17]. Similar to the case of linear dependence, HSIC considers the sum of square of all singular values of the covariance operator as a metric for independence:
=
[ ye 12.
Here, -- Il is the Hilbert-Schmidt norm, which generalizes the Frobenius norm to function spaces, and re is the rank of the cross-covariance operator.
Empirical Estimation of HSIC: Let (xx) = k(.,, x) and vby) = I(., y) be the corresponding
kernels of F and G. Given a set of observations D ((x,, y: | biased estimator of HSIC is:
1
HSIC GD
trace ( HH)
(5)
n²
where K and L are Kernel matrices for and y. H I 1 1 is the centering matrix.
3.3 Independence From a Functional Perspective
Under the assumption that the corresponding kernels of J and G are characteristic, X and Y are statistically independent if no pair of witness functions can be found that yield a nonzero objective in Equation (3), or equivalently: HSIC(F, 9. P.Pxy) = [18]. This principle distinguishes our method from prior kernel- based approaches, namely kSAL and KCE. These methods operate under the assumption that mapping a representation RKHS and performing linear erasure in that space is sufficient for nonlinear guardedness. However, this only protects against linear adversaries within that specific feature space, remaining vulnerable to adversaries that exploit nonlinear patterns that same space. This approach would guarantee statistical independence only if the RKHS feature map transforms representation to an approximately Gaussian distribution [29].
In contrast, Obliviator solves a more challenging problem: it seeks a representation a(X) where unwanted attribute S remains undetectable even if transformed by subsequent, adversarially chosen feature map Φ(-). This formulation results in a cascading kernel problem that lacks a closed-for solution and more difficult to optimize. Mathematically, this is equivalent to finding a representation such that no pair of functions, f and g, from a characteristic RKHS can be found to lincarize the relationship between f((()) and g(S):
inf sup sup E (G(S) f (=(0;
(6)
i
where we assumed guarding function e is parameterized by 0. Thus, for the transformed RV Z e(X) we have HSIC(F,9, Pz = 0 and, therefore Zo .11 S.
In extreme cases, such functions can be constructed using sharply peaked forms, such as the Dirac delta.

---

OBLIVIATOR
Next X
Encoder
The mode P.D. Darch
Sileew fook Ewdeningg Him iaaty contrd vodo pust yoacs
A
Tegitwure
Closed-form Solver
Classifur
Phosicin
Abberray
Encoder gi+l
X 1L
Figure 2: Overview. Obliviator operates with two-step iterations: 1) Imposing Independence via RKHS: An encoder trained with multi-objective loss (8) to reduce statistical dependence on the unwanted attribute while preserving task-relevant information. 2) RKHS Disentanglement Representations from the previous step are refined using functions derived from a constrained optimization in RKHS (11). This refinement enhances the feature space's alignment with the target attribute, which facilitates the encoder's training the next iteration. (Image source: BERT)
4 Obliviator: Non-Linearly Guarded Erasure
Obliviator solves (6), challenging nested optimization problem. This formulation, however, lacks explicit term to enforce the preservation of task-relevant information during erasure. RKHS witness functions make the adversary search tractable and, by a similar logic, the observability of the task-relevant information by them can be utilized formalize this, let X be the original representation. with S and Y denoting the unwanted attribute and target task, respectively. Assume F, 9, and H are the RKHSs from which we draw adversary and witness functions. Recalling our guarding function as Zp =0; X), we can write:
proxy for information preservation. To
inf 0
ASIC((F,9P2.0)
HSIC(F, P2,,,,))
where transformed the nested optimization (6) into simpler bi-objective form using RKHS. This problem still lacks a closed-form solution and thus requires non-convex optimization. Due to its two competing objectives, single-shot optimization is likely converge poor solution. We adopt an iterative approach to solve it which provides intermediate representations, offering more leverage preserve utility during erasure. Moreover, employ witness functions reshape these intermediate representations such that the components of information independent of the unwanted attribute become more accessible for the encoder in the iteration (see Figure for an illustration).
Obliviator Two-step Process:
Imposing Independence via RKHS: Utilizing RKHS, an encoder is trained minimize the statistical dependence of the representation on unwanted attributes, while preserving task- relevant information by keeping it observable witness sanctions.($4.11)
RKHS Disentanglement: Solves constrained optimization in RKHS to find functions that improve alignment of representation with utility, while avoiding such alignment with the unwanted attribute to facilitate the optimization in next iteration.($4...
4.1 Encoder: Imposing Independence via RKHS
In our iterative method, we train sequence of distinct encoders, where their inputs and outputs serve as intermediate RVs. Let X' be the input to the encoder at iteration i, and 23 = s s'O ; X its output. Assume X, and S denote the RVs for the initial representation, target task, and unwanted attribute, respectively (see Figure 2). Our objective is to use witness functions as a proxy to impose the visibility of information about Y, and X within Z while minimizing this visibility for S to achieve statistical independence. Recalling the objective in (7) and empirical estimation of HSIC from (5) we can write:
inf 6'
n² trace HK TrK, - K -TyK,
(8)
where K. E R denotes the kernel matrix for the corresponding variable. Compared to (7), we introduce two modifications:: 1) are hyperparameters that weight the importance of each objective.
5

---

2) X and X are auxiliary RVs included to enforce utility preservation. These are particularly useful when target-task labels (Y) are unavailable.
A natural question, however, is why X and X are necessary when Y is available. Recall from (3) that HSIC aggregates multiple "modes of visibility" (singular values) into a single scalar. Due the competition between objectives, the optimization, which only sees this aggregate sum, is likely to discard weaker modes. The critical insight is that as information disentanglement evolves during optimization, the same mode of dependency is often weighted differently when measured relative to each RV. For instance, a specific mode of task-relevam, information might be weakly expressed relative to Y but remain more apparent relative to X or X'. Therefore, including X and X increases the likelihood that these modes are preserved during optimization.
4.2 Eigen Value Problem : RKHS Disentanglement
As discussed, the evolution of information disentanglement affects the visibility of dependency modes relative to each RV, a result of the competition between objectives in optimization. To counteract this, we introduce an intermediate step that utilizes witness functions to re-align the representation, making task-relevant information more accessible. Recalling the logic from (3) and (8), we seek a function f E F that maximizes the correlation of 7 with the corresponding witness functions yr in Gz for XX', and Y (where I = {x*, x, })). However, the search for f must be constrained to avoid modifying the visibility of S, which would effectively counteract the minimization from the previous iteration. To formalize this, we solve the following constrained optimization:
(X*) sup sup t F
To Elg (x) + Ty Blg, f(Z)
sup (lg( 0,
11g2llez = ||/||> 119.1l..
Here, To are hyperparameters that weight the importance of each objective. Let K. = L,L, be a full-rank factorization of the kernel matrix for the corresponding RV. This L. can be viewed finte-diimensional feature map. Using this map, the empirical estimation of an objective term, for example between Z² and Y, can be written as:
Elg
n Σ gu (o(()))())))
n L, = Cyer
(10)
Here, By and are the equivalence of 9y and f in the finite dimensional feature map. The optimal functions for (9) can then empirically estimated by solving the following eigenvalue problem:
(T Ourzt + Ty C Oyse T CT. Qu X
(11)
where Q is an orthonormal basis for Null(C...)) Given an eigenvalue threshold, we select m eigenvectors from (11) that correspond to this threshold. Let V ju....,, em.] denotes the selected eigen vectors, encoded representation for the next iteration is then given by:
X QV
(12)
We defer the derivation and practical implementation for Obliviator to Appendix.
Experiments
Evaluated PLMs:
We consider four models: DeepSeek-LLMM778 B-Catt [9], and
GPT-2 [23],
[16]. For BERT, we use the final hidden state of the [CLS] token as the text representation. For DeepMoji [14] we use the setup as in [26. 6). For the other models, we apply mean pooling to their token embeddings. The embedding dimensions for DeepMoji, BERT, GPT-2, DeepSeek and LLaMa 300, 768, 768, 4096, and 2048 respectively.
Evaluated Setup: Obliviator can utilize target task labels (Y), setup we term supervised erasure. In the unsupervised setup, these Y labels are unavailable. PLM representations are kept frizen with the exception of BERT, for which we also consider fine-tuned case where the model's weights are modified to improve disentanglement with respect to Y. We use three datasets in our experiments: BIAS IN BiOs [8] (Y: Profession, S: Gender), DIAL-SENTIMERT [4] (Y: Sentiment, S: Race), and

---

75
100
800
Sentment Aec.
Profession Acc.
Mention Acc.
70
50-
70-
INLP
ESAL
Ranlom
60-
60
Obliviatoo
Chance
40
50g
60
75
35
75
80
53
63
Race Acc.
Race Acc.
Gender Acc.
(a) DIAL-MENTION
(b) -ENTIMENT
(c) BIAS IN BIOS
Figure 3: Finetuned+Superivee Erasure Comparison of Obliviator with baselines for fine-tuned representations. Obliviator leverages labels during the erasure, a scheme which we refer to as supervised erasure."
DIAL-MENTION [4] (Y: Mention, Race). We compare Obliviator against INLP, AdS, kSAL, FaRM, and KRaM as baselines.
While concept erasure methods are typically evaluated on their final- stage, such evaluations overlook how competing objectives interact throughout the erasure process. Studying the utility-erasure trade-off provides deeper insight into this interaction, offering a more comprehensive assessment of method's performance and its optimization. To construct these trade-off curves, we evaluate utility and leakage at each iteration. We measure attribute leakage by probing for the unwanted concept with various nonlinear adversaries and reporting their highest accuracy. We continue the erasure process until the probe accuracy for the unwanted attribute become close to random chance accuracy, which is the majority class percentage and shown the origin of all trade-off plots. We compute trade-off curves from five independent runs.
Utility-Erasure Trade-off:
Questions Investigated in Experiments:
$5.1 How do different experimental scenarios affect utility-erssure trude-off? Here, analyze the effect of the erasure scheme (supervised vs. unsupervised), the initial representation (frozen fine-tuned), and different datasets in BERT, a common benchmark that has been evaluated by many erasure methods.
$5.2 How does Obliviator generalize across different language models? Here, goal is to investigate whether the better disentanglement of representations learned by more capable language models is reflected in the utility-crasure trade-off.
$5.3
How does biased sampling or data skewness affect the erasure? Here, we use another common benchmark in concept erasure. a controlled setup that modifies the proportion of the unwanted attribute, to investigate the resulting trade-off with the Deepmoji model.
$5.4
How do erasure schemes different PLMs affect fairness metrics? Here, explore the effect of erasure on downstream fairness under these conditions.
5.1
How does different experimental scenarios affect utility-erasure trade-off?
In this section, we compare Obliviator against baselines to examine the effects of the erasure scheme (supervised vs. unsupervised), the initial representation (frozen vs. fine-tuned), and different datasets mentioned above. We use the BERT model for this analysis, many erasure methods have been evaluated using these datasets. However, the utility-erasun trade-off these contexts has not been previously explored. We consider all four combinations of erasure scheme and representation, namely: F)Finetuned+ Supeevisse (Figure 3), Frozenn++ AerrzznnUUnuueer Unsupervised (Figure 4), 3)Frozen+ Supervised (Figure 5), 4)iintuned++ Unsupervised (Figure 5). First, we study the initial two scenarios, which are designed to assess the impact of Y label availability, either for fine-tuning or during the crasure process. The final two scenarios then investigate how the visibility of Y information affects the erasure process.
a common benchmark on which
Considering Figure 3 and Figure 4, Obliviator achieves nonlinear guardedness, capturing the full trade-off curve across all scenarios and datasets. This outcome is expected, as our method directly minimizes HSIC - O, condition equivalent to independence from functional perspective. In contrast, existing methods fail to achieve full erasure, particularly for fine-tuned representations. Notably, in Figure 3, the gap between full erasure (i.e., random chance) and the best-performing baseline is 12%. 13%. and 14% for DIAL-MENTION, DIAL-SENTIMENT and BIAS
Erasure Performance:

---

76
80
70
Profession Acc.
Mention Acc.
70
70
KRaM
65
Sentment
65
60
ESAL
INLP
Random
60-
50
Chance
60
Obliviator
-55
40
(Ours)
55
50
30
30
05
60
65
70
75
80
65
70
10
80
85
60
70
80
90
100
Race Acc.
Race Acc.
Gender Acc.
(a) DIAL- DIAL-MENTION
(b) SENTIEENT
(c) BIAS IN BIOS
Figure 4: Frozen+ Frozen+Unsupeen Unsupervied Erasure: : Comparison of Obliviator and baselines with frozen representations. In unsupervised erasure, we implicitly observe Y information from X and X' and thereby we observe a more noticeable trade-off compared to Figure 3.
IN BIOS, respectively. Similar gaps of 12%, 9%, and 8% are observed Figure 4. Obliviator consistently outperforms all baselines, achieving higher Y accuracy every level of S accuracy across all scenarios and datasets. This result stems from our algorithm's 's two core components: the utility-pressee preserving loss in (8) and the disentanglement process in (11). Notably, utilizing RKHS to formulate these two components in closed-form is key contributor to the consistency of our optimization.
Cost of Erasure: Comparing the trade-off curves for each dataset across the different setups in Figure 3 and Figure 4, we observe more noticeable trade-off in the frozen+ unsupervised sce- nario. This is consistent with our discussion Section 4.1. The key difference is that Y provides explicit proxy for the task-relevant modes. In contrast, X and X' are implicit proxies; the optimization's priority for the task-relevant modes depends on how those modes are weighted relative to all other information encoded within these RVs. Therefore, preserving Y-relevant information is more likely in the supervised erasure setup. This distinction is particularly evi- dent in DIAL-MENTION and DIAL-SENTIMENT. where Obliviator preserves mention/sentiment information with minimal utility loss while successfully erasing race-related information.
Efffet of Y Visibility Erasure: To distinguish the effect of the Y label on supervised erasure from its effect via fine-tuning, we examine two key sce- narios: Prozenn-Spperiiell and Fine-
Sup-Fd -- Sup-Fz Unsup-Fd - Unsup-P2 Random Chance
Profession Acc.
Mention Acc.
tuned+ Unsuperveed. First, we analyze Frozen+ for DIAL-MENTION (Figure 5a). The high utility preservation confirms that supervised erasure alone highly effective. strengthening our hypoth- esis from Section 4.1 that an explicit proxy for Y rrleevatt modes is more likely to pre- serve that information. Interestingly, the Finetuned+ Unsupervised setup for DIAL- MENTION also preserves utility well. This indicates that fnn-tuning strengthens the visibility of Yreeeevant information when measured implicitly using X and X giving this information higher priority in the optimization.
Race Aoc.
Gender Acc.
(a) DIAL-MENTION
(b) BIAS IN Bios
Figure 5: Supervised and unsupervised erasure fine-tuned
and frozen representations. (Sup: Supervised, Unsup: Unsuper- vised, Fd: Finetuned, and Fz: Frozen.)
However, the behavior differs for BTAS IN Bros. As shown in Figure 5b, the gap between erasure schemes on the frozen representation is less noticeable than DIAL-MENTION This could be due to the nature of the Y labels: BIAS IN B1OS has 28 classes, so it is likely that task-relevant modes have higher weights when measured by witness functions from and X In contrast, for DIAL-MENTION (binary Y). the signal is weaker, making the implicit proxies X and X less practical compared to an explicit Y proxy for preserving the information. For Finetuned+ Unsupervised erasure BIAS IN BIOS, we again observe that preserving Y information is possible up to a point, implicitly through X and X a similar reason as DIAL- STAL-MENTION. However, in both the finetuned and frozen representations, we observe that supervised erasure preserves information much better. This can be attributed to the erasure process itself: as the representation evolves and becomes more disentangled, the visibility of Y information through the implicit proxies (XX and X') may also evolve and weaken.

---

80
Profession Acc.
Profession Acc.
20
20
60
60
Profession
60
Sup.
50
50
50
Unsup- BERT
Sup-BERT BERT
40
40
Random
Chance
300
30
30
60
70
80
90
100
60
70
80
90
100
60
70
80
90
100
Gender Acc.
Gender Acc.
Gender Acc.
(a) GPT-2 Representations
(b) DeepSeek Representations
(c) LLaMA Representations
Figure 6: Erasure Across Different PLMs Compared to BERT. The figure shows supervised and unsupervised erasure using frozen representations from GPT-2, DeepSeek, and LLaMa BIAS IN BIOS.
Therefore, having explicit proxy for Y information becomes more valuable the optimization proceeds.
In supervised erasure, Obliviator utilizes explicit term to observe Y -reeevaant information via witness functions ((8) and (11)). This provides a direct optimization signal that is more likely to preserve utility, especially in the final stages where erasure targets more entangled information.
Takeaway 1.
Takeaway 2. In unsupervised erasure, the initial visibility of Y-relevant information affects its priority the competition of objectives when measured via the implicit proxies (X and X determining its preservation during erasure.
5.2 How does Obliviator generalize across different language models?
Here, we examine the generalizability of Obliviator across different PLMs. We hypothesize that more capable models should learn representations that are better disentangled. Consequently, a robust erasure approach should yield improved trade-offs when applied them. To explore this, frozen BIAS IN BIOS representations from GPT-2, LLaMa, and DeepSeek. Figure shows:
The trade-off profile of GPT-2 closely matches that of BERT in both erasure schemes, an expected result given their comparable model complexity. However, the performance of several baselines, including INLP, FaRM, and KRaM, degrades on GPT-2. Notably, FaRM and KRaM drop to near random-chance accuracy on the target task.
GPT-2 Representations:
LLaMa Representations: Compared to BERT, we observe clear improvements in both erasure schemes. As LLaMa provides representations with better initial disentanglement of Y and S, Obliviator leverages this to achieve a more generalizability of our method and the effectiveness of its optimization approach. The performance of FaRM and INLP remains largely unchanged from BERT, and while KRaM shows some improvement, still fails to achieve full erasure. Notably, Obliviator consistently outperforms all baselines.
erasure. This result demonstrates the
DeepSeek Representations: In the wnsupervised erasure, DeepSeek representations yield perfor- mance improvement BERT similar that observed with I.LaMa. Among the baselines, KRaM's erasure performance improves compared to its results on LLaMa representations, while FaRM's be- havior remains roughly unchanged. For target task accuracy, KRaM's performance degrades, whereas FaRM shows some improvement. Obliviator consistently outperforms all baselines. Interestingly, in supervised erasure, we observe significant improvement the trade-off profile, with only mild degradation in target accuracy as erasure completes. This demonstrates that achieving full erasure on the BIAS BIOS dataset while maintaining substantial target task information is possible. This strongly suggests that the observed utility-erasure trade-off is diagnostic of the specific model's learned dependencies, not necessarily an inherent real-world link.
Takeaway
With better disentangled representations learned by more capable PLMs, Obliviator's erasure becomes more pitiity-peeeriing. indicating its strong generalizability.
5.3 How does biased sampling or data skewness affect the erasure?
In this section, we explore the effect of biased sampling on erasure, using controlled setup recognized classic benchmark in concept crasure [13, 26, 6]. Following this setup, use the DIAL-

---

754
75
Sentiment Ave.
Sentiment Acc.
Sentiment Acc.
70
70
70
0.5
60-
65-
0.7
65
05
BERT
SupIBERT
55
Randoen
80
60
Unspp
DepDespSeek
5(1
50
55
00
65
70
75
80
85
55
0.05
0.1
0.15
0.2
0
0.05
0.1
0.15
0.2
Race Acc.
DP
Gapan.
(a) Controlled Setup
(b) DP
(c) Gap
Figure 7: (a). Unsupervised erasure for DeepMoji representations on DIAL-SENTIMEN,, plotted against varying levels of unwanted attribute disproportion. (b-c). Demographic Parity (DP) and Gap across different erasure scheme and PLMs in DIAL-SENTIMENT.
SENTIMENT dataset, DeepMoji representations, and an unsupervised erasure scheme. For biased sampling, we modified the proportion of the unwanted attribute (Race) within each target task class (Sentiment). For instance, in the 80% split, the "happy" sentiment class is composed of 80% African- American English (AAE) and 20% Standard American English (SAE), while the "sad" sentiment class is composed of 20% AAE and 80% SAE.We hypothesized that increasing this imbalance would worsen the trade-off, as a skewed sample provides a biased estimate of the true statistical dependence (HSIC) and makes finding atility-preserving solution more difficult. Figure 7a confirms this hypothesis, as we observe the trade-off becomes more noticeable as the sampling skew increases. This highlights inherent challenge for post-hoc erasure: the observed trade-off is highly dependent on how well the erasure dataset represents the true data distribution.
Takeaway. Skewed sampling worsens the trade-off due to increased bias in estimation of HSIC.
5.4 How do erasure schemes on different PLMs affect fairness metrics?
Here, we investigate the effect of erasure on downstream fairness, speculating that imposing statistical independence w.r.t unwanted attribute will improve faimess metrics. We use Demographic Parity,
DP = Ecllp( = C|S = 0) pÝ = C 1) l]. and Gapma = Ec((p(Y = C|Y p( = CIY = = C.S 1))3!1/2 where Y is the prediction of a model. With statistical independence w.r.t S, fairness metries should improve and theoretically DP + 0. We chose DIAL-SENTIMENT as our dataset, with DeepSeek and BERT as the two PLMs, as they encode entanglements differently. The results for DP and Gap are shown in Figure 7b and Figure 7 respectively. Obliviator improves fairness metrics acToss erasure schemes, with the supervised scheme being more peeiity-reer.inng Moreover, due to the better disentanglement of representations in DeepSeek, we observe a higher utility for any given level of unfairness compared to BERT.
= = C,S
Takeaway. Obliviator aids fairness by imposing statistical independence w.r.t unwanted attributes.
Concluding Remarks
Concept erasure aims to remove unwanted attributes from representations while guarding against all adversaries. Existing methods fail address this challenge, as they remain vulnerable to nonlinear adversaries. We propose Obliviator, which formulates erasure from functional perspective to capture nonlinear statistical dependencies. We solve the resulting, challenging nested optimization with a two-step iterative approach. We investigate the effect of erasure more comprehensively by analyzing the full utility-crasure tradc-off, a dynamic overlooked in the literature. We explored how different erasure schemes and the disentanglement of the original representation affect this trade-off. Our results show that Obliviator achieves state-of- -ftth-art performance, provides robustness against nonlinear adversaries, and sets strong benchmark for the utility-erasune trade-off. This performance remains consistent across various datasets and language models. The improved disentanglement of representations from more capable models is directly reflected in Obliviator's 's utility-erasiie trade-off profiles, indicating its strong generalizability and effectiveness of its optimization approach.
Acknowledgements: This work was supported by the National Science Foundation (award #2147116). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF
10

---

References
[1] Somnath Basu Roy Chowdhury, Sayan Ghosh, Yiyuan Li, Junier Oliva, Shashank Srivastava, and Snigdha Chaturvedi. Adversarial scrubbing of demographie information for text classification. In Conference on Empirical Methods in Natural Language Processing, 2021.
[2] Somnath Basu Roy Chowdhury, Nicholas Monath, Kumar Avinava Dubey, Amr Ahmed, and Snigdha Chaturvedi. Robust concept erasure via kernelized rate-distortion maximization. Advances in Neural Information Processing Systems, 2023.
[3] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36, 2024.
[4] Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic dialectal variation in social media: case study of African-Americes English. Conference on Empirical Methods in Natural Language Processing. 2016.
[5] Tolga Bolukbasi, Kai- Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in Neural Information Processing Systems, 29, 2016.
[6] Somnath Basu Roy Chowdhury and Snigdha Chaturvedi. Learning fair representations via rate-distortion maximization. Transactions of the Association for Computational Linguistics, 10:1 159-1174, 2022.
[7] Somnath Basu Roy Chowdhury, Nicholas Monath, Kumar Avinava Dubey, Amr Ahmed, and Snigdha Chaturvedi. Robust concept erasure via kernelized rate-distortion maximization. In Advances Neural Information Processing Systems, 2023.
[8] Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexan- dra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Conference on Fairness, Accountability, and Transparency, 2019.
[9] DeepSeck-AL. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL https://githut http://ittuuu.cm/deepeeeeeee com/deepseek-ai. DeepSeek-LLM,
[10] Sepehr Dehdashtian, Bashir Sadeghi, and Vishnu Naresh Boddeti. Uaility-faimees trade-offs and how to find them. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Pre-training of
[12] Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In Interna- lional Conference on Learning Representations, 2016.
[13] Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In Conference on Empirical Methods in Natural Language Processing, 2018.
[14] Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Conference on Empirical Methods in Natural Language Processing, 2017.
[15] Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases word embeddings but do not remove them. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.21788, 2024.

---

[17] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidd norms. In International Conference on Algorithmic Learning Theory, 2005.
[18] Arthur Gretton, Ralf Herbrich, Alexander Smola, Olivier Bousquet, Bernhard Schölkopf, and Aapo Hyvärinen. Kernel methods for measuring independence. Journal of Machine Learning Research, 6(12), 2005.
[19] Yazhe Li, Roman Pogodin, Danica Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems, 14:15543-15555
2021.
[20] Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and privacy-preserving text representations. In Annual Meeting of the Association for Computational Linguistics, 2018.
[21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[22] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing, 2014.
[23] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[24] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. 2007.
[25] Sebastian Raschka, Joshua Patterson, and Corey Nolet. Machine learning in python: Main developments and technology trends in data science, machine learning, and artificial intelligence. arXiv preprint wrXiv:2002.04800 2020.
[26] Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Annual Meeting of the Association for Computational Linguisties, 2020.
[27] Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning, 2022.
[28] Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Conference on Empirical Methods in Natural Language Processing, 2022.
[29] Bashir Sadeghi, Sepehr Dehdashtian, and Vishnu Boddeti. On characterizing the trade-off in invariant representation learning. Transactions in Machine Learning Research, 2022. ISSN 2835- 8856. URL https: ht
Featured Certification.
[30] Shun Shao, Yftah Ziser, and Shay B. Cohen. Gold doesn't always glitter: Spectral removal of linear and nonlinear guarded attribute information. In Conference of the European Chapter of the Association for Computational Linguistics, 2023.
[31] Ben Verhoeven, Walter Daelemans, and Barbara Plank. Twisty: multilingual twitter stylometry corpus for gender and personality profiling. In International Conference on Language Resources and Evaluation, 2016.
[32] Liwen Wang. Yuanmeng Yan, Keqing He, Yanan Wu, and Weiran Xu. Dynamically disen- tangling social bias from task-oriented representations with adversarial attack. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
[33] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork, Learning fair representa- tions. In International Conference on Machine Learning, 2013.
12

---

Supplementary Material for Obliviator
This supplementary material provides additional details to support the main paper. It includes mathematical proofs, implementation specifics, ablation studies, and additional results further illustrate and validate the effectiveness of Obliviator. The contents are organized as follows:
Mathematical Proofs. (SA)
Cross-Covariance Operator ( $A.1)
Hilbert-Schmidt Operators ($A.2)
Empirical Estimation of the Objective ($A.3)
Cimite-iimensional Feature Map ($A.4)
RKHS Encoders for Feature-Space Alignment ($A.5)
Implementation Details. ($B)
Obliviator's Training Procedure ($B.1)
Datasets ($B.2)
Experimental Setup ($B.3)
Computational Complexity and Scalability of Obliviator ($B.4)
Probing Networks. ($C)
-Ou Choice of Probing Networks (SC.1)
Ablattion Studies with Different Probing Networks. ($C.2)
Additional Results. ($D)
Single Step Erasure Multi-Step Erasure ($D.1)
Further Results on Fairness ($D.2)
Hyperparameter Sensitivity (SD.3)
Visualization ($D.4)
Limitation ($E)
Societal Impact ($F)
A Mathematical Proofs
A.1
Coss-Covvreencce Operator
Let F
o(x) and & y + (z) denote reproducing kernel Hilbert spaces (RKHSs) with feature maps á and vb respectively. We aim to find functions f E F and g E g that maximize the following objective:
sup sup gEg JEF
[(S(x) B[f()))) (o(v) - Elg()))) 8.l. ||||
lggll =1 (13)
Using the reproducing property of RKHSs, the centered evaluation of can be expressed as:
f(x:) (b(x),, EB[ (6(x)) f)r]
(o(z), f)r (B(o(x)) f)
(14)
(δ(x) E(d(x)),
(ō(z;) f)F
where ō(x) = o(x) - E[o(x)). A similar expression holds for g(y). Substituting into the objective, obtain:
sup sup geg FEF
[f(x) = E. [()()), }> ()(
(15)
Recall that the tensor product g 2 Ff + 9 is rank-one operator defined by:
(g 2 f)h: = glf. k)s for all h F.
(16)
13

---

Using this, the objective becomes:
E [) (b(1))
EE ((((y) (0 (x), g)
E ((b(y) @ o(x) f. 1999g]
(17)
(E(v(y) ((x))())))))))))))))))))))))(( f.9)
= (Covyz = (g. Cov
where Covyr == (v) o(x:) denotes the cross-covariance operator from g.
Remark. Strictly speaking, interchanging the expectation with the inner product requires justifying that (yy) @ ()(x) defines Hilbert-Schmidt (HS) operator and that expectation exists in the Hilbert space of HS operators. We provide minimal justification in the following section. The derivation above is retained for its close resemblance to the fiiie-dimenniooal cross-covariance formulation.
A.2 Hilbert-Schmidt Operators
Let F and G be separable Hilbert spaces and - orthonormal basis of F. For bounded operator L F the Hillbrt-SShhid(((1 HSS) norm is defined as
L its
(18)
If this series converges, L is called Filberr-Schhnee The set of from F to 9 is a Hilbert space denoted by HS(F,9) g) with the inner product
operators mapping
(5,g)e us = Σ (fq. T F
(19)
Rank one tensor product is HS. For E F and y E G:
(| |
t=1
(20)
= -11113
Σ of
t=1
11l113 |1 x
so g f HS(F, ,)). Lemma. For any I E HS(F, f F. and g E E.
(I g8 (g. Lf4
(21)
Proof. Choose an orthonormal basis of F that begins with the normalised vector f= ie. {f) BB_ with B1 = {f. p
(L, E.g0sms (z}, go Σ
(Lui got 18s4ila
BEB
(22)
Tfll= 9 (f
(g, LfG
The second term vanishes because f is orthogonal to every q e B1.
14

---

Using this lemma we have:
(v) g8f) (6(x). f)r ( (b(4).9)
(23)
and
[[f())g(y) = - ( g g ms]
(24)
Boundedness of the expectation functional. Consider the following linear functional:
F ()(FF -R. ( E (v) ( L us
(25)
Applying Jensen and then Schwarz inequalities:
(*) - (1 s Lm.l
E [1b()))0(0)) Ins | [M]
(26)
|1 [1
1 [g]
| hss E ((ar a) (y 00
so is bounded linear functional on HS(F.9).
Riesz representation and the covariance operator. Recall that on any Hilbert space, the Riesz representation theorem states that every bounded linear functional be expressed as an inner product with a unique element of that space. Applying this to the bounded functional F defined above, obtain unique operator Cov yz E HS)F, satisfying:
(Covyz: 3) ns
F (9)
& HS(F,9)
(27)
Taking S = g f recovers the cross-covariance identity used in the main text. We now turn to the empirical estimation of cross-covariance operator.
A.3 Empirical Estimation of the Objective
Given dataset D {(x, y:) the empirical estimate of the cross-covariance operator given by:
Cov
(gy. Φ(w.).
(28)
Next, by examining the action of the tensor product operator Equation (16), together with the unit-norm constraints on / and g. we conclude?:
Σ a, and g Σ 3,
(29)
Substituting this representation into the objective in Equation (17), we obtain:
E [f(=]{(<)]
a a (T) (wu)) (o(z,), ((x;))=
(30)
Letting and / denote the kernels associated with F and respectively, the expression simplifies to: 1 3T LKa
E [()) 1 ΣΣΣ Be Ta ki a
(31)
where K and L are the centered kernel matrices corresponding to k and I, respectively. Following the representation of ʃ and from Equation (29), the unit-norm constraint on f can be expressed as:
1
Σ
(
Σ a c5ي((z))))
1=1
(32)
agay (o(2)), o(2,))
a Ka
We assume k<n,, .. and I(y, 2) have finite first moments. This follows directly from the Representer Theorem.
15

---

and likewise for g:
3
LB
(33)
Therefore, the empirical estimation of the objective reduces to the following constrained optimization problem:
sup sup B
3 LK
s.t.
a Kax = -3 LB =1
(34)
Next, let K = JJ and define = J a, and similarly let L = DD with DTB. Then, the objective can be rewritten as:
sup sup D Ju
S.L.
v v v
u T u =1
(35)
This is maximized by the largest singular value of DJJ i.e. it is upper bounded by d J)T.). Next, by considering the sum of squared singular values, we obtain:
1 f (JJ (JJDDD)) DD' J) (J" n²
1
1 n²
n² tr((L)
tr (KLH)
(36)
where we used the cyclic property of the trace and the centering matrix H = 1-1111 Finally, the squared Hilbert-Schmid norm of the empirical cross-covariance operator reproduces the same scalar quantity:
111T.
Cov yz 1
(
((,))
(w))), (11))) ((0)))
(37)
(o(x;), )(
2
i=
Kig Ia
tr (KL) n
n tr (r(KLH))
In the next section we show that, in the empirical setting, one can work with a finite- dimeensaimiii feature map obtained from factorization of the kernel matrix. This representation makes subsequent covariance expressions far more transparent.
A.4 Finite-Dimensional Feature Map
Let F:: ~ (xx:) be RKHS with reproducing kernel k(.,, .). By the Representer Theorem (cf. Eq. (29)), any solution subject to norm constraints admits the form:
as o(x;)
(38)
so f lies in the finite subspace == span (d(x:;)) For f.g X we have:
(S.9)= =
(n:((::) Σ (5 = = K3,
(39)
J=1
where K the kernel matrix Kij- = k( 2z)). Because K is symmetric positive semi-definite admits factorization:
K JJ
(40)
with axr full rank. Row of (denoted by J, therefore provides an implicit finite- dimensional vector for d(x;): indeed J. Jj Ksj (0(x.), o(x,)) Moreover:
(f.g)= = JJJ 3 (J [J 33 =u
(41)
16

---

where we have set = J'a a and = J"B. Hence u, E I' are equivalence of f and in this saite-dimensional feature map. Using the reproducing property, the centred evaluation vector f = (f())) Pf. M....... ((x)) (xt,))
μ 1", with N = -==1 (a;), satisfies:
f= = Ju 1 ( H Ju)
- HJu
(42)
where H the centring matrix. Now let 9: y + vly) with Gram matrix DD Repeating
the same construction yields HD for g = د== Byr(())) Thus the empirical objective becomes:
sup sup B[f(z) = Σ (f(z;) - ms) (o(y)) - Mg)
gEg fEF
= (HJ))
(43)
e D HJu = v Cyz u
n
where D HJ the (finite-sample) cross-covariance matrix corresponding to the empirical
cross-covariance operator Covyz. In the following section derive the Equation (11).
A.5
RKHS Encoders for Feature-Space Alignment
Recall from Section 4.1 the following random (Ø' = c'(;
X encoder input at iteration
Z:: encoder output at iteration
X: initial representation
S: undesired concept labels
Y: target task labels
Let F be an RKHS in which we seek encoders f that optimize the following objective:
Elg (xx) + Elg (X) + Ty Elg, (Y)
sup (or}
sup
(44)
s.t.
sup (S) 0, 11 = = 119alee, =1
=
9x
Here, GT. (with I = {ar x, v)) and G. refer to corresponding RKHSs for gr and ga. Let the kernel matrices for Zz, X X, and be factorized as
K L LT. K J.:J gy K, K,=JJJJ
Ly K, L,
Using the finite-dimensional feature representation discussed earlier, the empirical estimate for each term the objective, for instance (Y) can be written as:
B (Y)J(Z') Σ (g))())))))
- Ly HL, = =
C Cyz:
(45)
where Uy and are the equivalent vectors gy and f in the corresponding finite dimensional feature map, and Czz- denotes the empirical cross-covariance matrix. Note that the following two equivalent:
optimization problems
sup Re,
sup
(u. yet w 2)
= sup
||Cys = w Cyer
(46)
w
This equivalence follows since the optimal u. is aligned with G w and satisfies:
Cye-
Uyy
(47)
||C 11Cyst20li
17

---

Hence, the full empirical objective becomes:
sup w (C x= Czizi To C 25 Case + Ty C Cys))
(48)
Now consider the constraint in Eq. (44), which ensures that f does not increase alignment with the undesired attribute S. Its empirical estimate becomes:
sup (()(())((Z)) sup u, C.si
(49)
ua
To satisfy this constraint, require
w Null(C.s.)
Let Q be orthonormal basis for this null space. Then we can write:
Qu
(50)
Since orthonormal transformations preserve norm, the constraint remains unchanged. Substituting into the objective yields:
sup vQQ (O ( Carst + T2
TT 20 CTe Casi + Tyy C Cyzz) Qu
(51)
| =1
This is Rayleigh quotient maximization problem. The optimal solution corresponds to the eigenvector the matrix with the largest eigenvalue. Define:
A Q (Cr Cutst + Ōzs
Cyet)
(52)
The matrix is a sum of symmetric positive semi-definite matrices, and is itself symmetric and PSD. Using its degeppcccompoiiiion:
A DAD
(53)
v Au
Aa Σ Amax
where a DT and |a]] = since the eigenvectors symmetric matrices form an orthonormal basis. Subsequent encoders, constrained to be orthogonal to the previously selected ones, can be obtained by iteratively maximizing the same Rayleigh quotient over orthogonal complements. By the Couran--Fischer min-max principle, these correspond to the eigenvectors associated with the next largest eigenvalues of A. One may then select the top d eigenvectors ranked by eigenvalue magnitude or a normalized criterion as encoder directions for the next iteration.
B
Implementation Details
B.1 Obliviator's Training Procedure
The practical implementation of Obliviator is outlined in Algorithm 1. Depending the availability of target task labels, the erasure setting is configured as either supervised (when target labels are available) or unsupervised. In the supervised case, an additional term involving Ty is included in the encoder loss (8), and correspondingly in the RKHS- based eigenvalue problem (11) as Ty ÔT G This term omitted under the unsupervised setting.
The encoder is then trained for fixed number of epochs, which must be sufficient to enable effective transfer information from the original representation. If the encoder trained for too few epochs, its capacity to preserve relevant features may be limited. In such cases, one practical strategy is to pre-train the encoder for a few epochs without applying the undesired-concep removal term, and then perform erasure for a smaller number of epochs. However, excessive pretraining can also be detrimental: the encoder may learn overly complex representations that are no longer smooth enough to be effectively constrained by the smooth witness functions. This, in turn, may necessitate stronger (and potentially less smooth) witnesses, which can hinder erasure quality. Nonetheless, since initial representations from language models are typically expressive, even random projections can retain
18

---

Algorithm Obliviator Training Procedure
1: Input: data {æi}, unwanted labels (st), optional target labels (v:)
2: for j = 1 to M do
3: if {ys } is available then
4:
RVs <- (x',,,y)
D. Supervised
5: else
6:
RVs (x))
P Unsupervised
end if
8:
Train encoder e² using loss (8) t- RVs
b Imposing Independence
9:
2² f- 5² (x²)
10:
Solve (1I) obtain encoder
RKHS Disentanglement
11:
x encode 2 via (12)
end for
most task-relevant information. *Thus, the encoder can often preserve key features with relatively few training iterations. In experiments, found that 10-15 full-batch iterations were typically sufficient.
After training the encoder, its output is passed to the cigenvalue problem defined in (11). Since the feature space induced by the kernel can be high-dimensional, even for moderate number training samples, explicit factorization of the kernel matrix is generally intractable. To address this, employ approximation methods such as the Nyström method or Random Fourier Features (RFF) [24] to obtain minee-dimensional approximation of the feature map. Next, the resulting eigenvalues normalized by the largest eigenvalue, and the top m eigenvectors are selected based on a predefined threshold. These eigenvectors correspond functions in the RKHS, which serve We apply these functions to the encoder's 's output to generate a transformed representation. This transformed representation then becomes the input to the encoder in the subsequent iteration. The implementation of Obliviator can be found at the Obliviator page.
encoder.
B.2 Datasets
We evaluate on three benchmark datasets commonly used for concept crasure: BIAS IN BIOs, DIAL- SEN-MEENT and DIAL-MENTION.
BIAS IN B1oS [8]: This dataset consists of biographical texts, each annotated with a profession (the primary task) and gender label (the sensitivc/proteeted attribute). includes 28 distinct professions and two gender categories, with 53.7% of the data associated with male subjects and 46.3% with female subjects. The most common profession the dataset is "professor," " accounting for 30% of the total samples. To ensure fair comparison with FaRM, we used the same dataset split as FaRM [6] for the fine-tuned BERT representations. For the frozen representations, we followed the dataset split used by [26].
DIAL [4]: This dataset consists of two subsets: DIAL-SENTIMENT and DIAL-MENTION.
-DIAL-SENTIMENT is labeled for sentiment analysis, with sentiment as the primary target variable (happy 54.57%, sad 45.43%). It also includes race labels (African-Americn English 36.93%. Standard American English 63.07%).
-DIAL-MENTION is a binary classification dataset for detecting whether tweet mentions another user (50% conversational, 50% non-conversational.. The race labels in this subset are equally distributed (50%-50%)
B.3 Experimental Setup
We multilayer perceptron (MLP) our encoder, consisting of a single hidden layer with 256 units and the SiLU activation function. Optimization is performed using the Adam W optimizer with default hyperparameters. We the learning rate 5 x 10-4 and apply a weight decay of 0.001. For the BIAS IN Bios dataset, the encoder is trained for 30 iterations in the first step and 25 iterations in subsequent steps. Due the infeasibility of computing the full kernel matrix on the entire dataset,
'This is consistent with the Johnson Lindenstrass lemma.
19

---

Table Hyperparameters used for the training of Obliviator across different datasets.
Encoder Parameters Eq.(8)
EVP Parameters Eq.(11)
Dataset
Erasure
Yy
Ta
Yy
Threshold
BIAS IN BIOS
Supervised
0.05
0.02
4
0.2
3
10
Unsupervised
0.1
0,05
0.5
10
DINT-SENTM
Supervised
0.05
0.02
4
0.2
3
5 x 10-6
Unsupervised
0.1
0.05
0.2
5 x 10m6
DIAL-MENTION
Supervised
0.05
0.02
4
0.2
10
Unsupervised
0.1
0.05
0.2
5 x
exact kernel-based training is only possible via stochastic gradient descent with a reasonable batch size. However, we observe that with RFF and training on the full-batch consistently yields better performance. Consequently, we approximate the kernel using RFF throughout our experiments. The RFF dimensionality set to 2500/6000 in the first iteration and reduced to 1500 later steps, as the encoder's ' input dimension is reduced after the first transformation. Sigma chosen based on the median hcuristic. To ensure that HSIC is not artificially increased through isotropic scaling, evaluated two normalization strategies: (1) feature-wise normalization unit variance and (2) sample-wise normalization. The latter proves more effective our setup. For the DIAL-MENTION and SUAL-SENTIMMNT datasets all settings are the same except the initial training is to 30 iterations. All reported trade-off curves are over three independent runs where we reported minimum performance (lowest trade-off curve). For the RKHS encoder, the Kernel matrix is approximated using RFF with dimension of 1500. The hyper-parameters used for training can be found in Table 2. Training is conducted LLaMa on BIAS IN BTOS dataset all the parameters are similar to Table except we set the threshold 10-5
single NVIDIA RTX A6000 GPU. For the training of DeepSeek and
B.4 Computational Complexity and Scalability of Obliviator
The complexity of Obliviator is determined by two steps per iteration:
Encoder
The primary cost is calculating the empirical HSIC loss. There are two general approaches for this:
Direct Kernel Matrix Calculation: This method's time and memory complexity of O(N2) (where N is the batch size) becomes intractable for large batches. This forces trade- off, restricting the method to small batch sizes which can lead to a less accurate empirical estimation of HSIC
RFF Approximation:
To enable large-batch training for better HSIC estimation, we use scalable alternative based on Random Fourier Features (RFF). This approach has time complexity of ((Ne)) and a more favorable memory complexity of (N)d)) The core computations involve large matrix multiplications, which are highly optimized for modern GPU architectures, making this method very efficient in practice.
Eigen Value Problem (EVP)
This step involves solving an EVP. We again use RFF to make this step tractable. The complexity breaks down as follows:
Constructing the matrix for the EVP involves matrix multiplications with a complexity of O(Ndf.).
The eigenvalue solver itself has complexity of (d
In typical deep learning setup data size N is larger than the RFF dimension dur and the overall cost of this step likely dominated by the matrix multiplications, making complexity effectively d(N
20

---

Table 3: Comparison of theoretical and practical time complexity each step (BIAS IN BIOs)
Time Complexity
Encoder
EVP
Theoretical
(N clemba dhddden
Nd Chidden
Nd
O
Nd
Average time per step (BERT)
177.2 ms 0.38 ms
172 ms
1.23 ms
Average time per step (DeepSeek)
219.6 ms + 1.4680 ms
165 ms 1.65 ms
Table 4: Comparison of theoretical and practical memory complexity in each step (BiAS IN BIos) Encoder
Memory Complexity
EVP
Theoretical
(demba chidden + ditt duidden + Ndw + daa)
O + Nda)
Utilized Memory
8.18 GB
6.78 GB
The following tables show the theoretical time/memory complexity of our method, alongside its practical complexity measured on NVIDIA RTX-A6000 GPU. We also report the total time taken to perform crasure across different datasets, language models, and erasure schemes.
Table 5: Erasure time for Obliviator BIAS IN BIOS-frozen
BIAS BIOs Frozen
99.4% 770% (scc)
70% 70%% (sec)
65%->606 (sec)
60% +RC (sec) 99.4% RRC
BERT
Unsupervised
21.9++0.05 27. 27.1:10.06
27.4+0006 72.26:10.1
107.76+0.26 153.5:10.3
323.7+-0.7 487.7:10.5
482.9
Supervised
749.7=117 1.7
DeepSeek
Unsupervised
74.3++0.5
67.53-+0.46
94.54=0.64
189.08
425.4-2.9
Supervised
156
100.3++0.67
390.1=2.63
1070#
1716.3=11.6
Table 6: Erasure time Obliviator on BIAS IN Bios-Fineee cros-Finetuned
BERT
(sec)
(sec)
65%- (sec) 63% +RC (sec)
99.4%-RC (sec)
Unsupervised
37.7++0.08 43.2+0.1
62.8+0.14 162.06+0.36
540.7+1.2
4225.6+99.5
4363.9+99.
Supervised
108.0440.24
5596 12.4
5900 12.2
Table 7: Erasure time for Obliviator DIAL-MENTION-Frozen
BERT
80%-+70% (sec)
70%-+6% (sec)
60%->RC (sec)
80%- (sec)
Unsupervised
55.2-10.1 126.4-10.3
193.3 144.5-00.3
1293.2+13.0 1372.9413.7
1541.67-3.6
Supervised
1643.813.88
Table 8: Erasure time for FARM and KRAM on BIAS IN BIOS
Method
Batch Size
Memory (MB)
Time (sec/epoch)
Total Time (sec)
1024
96.4
21.4+
535-:30 330.5:125.75
FaRM
4096
305.44
13.22++.003
1024
76.8
65.58+114
3279-65. 65.2
KRaM
4096
479.04
45.8111.55
22.0-
21

---

C Probing Networks
C.1 Our Choice of Probing Networks
Unlike prior work [1, 6], do not restrict the nonlinear adversary to the default MLP classifier provided by scikit-learn. Instead, we evaluate all methods using two types of adversarial classifiers: an SVM with an RBF kernel (implemented via cuML. [25]) and an MLP with two hidden layers of 128 neurons (implemented in PyTorch (21J). The MLP classifier is trained three times with different random seeds, while the SVM is evaluated under seven hyperparameter setrings (7: C) {(10, 5), (10, 10), (5, 10), (5, 5), (1, (1, 1), (0.5, 1)}. We report the maximum accuracy obtained across all ten runs (three MLP + seven SVM) for each method. For the target task classifier, we employ the same architecture as the adversarial MLP, a two-layer Multi-Layer Perceptron with 128 neurons per hidden layer. An SVM with large kernel scale (? = 10) and hard margin (C = 10) serves as coarse decision boundary, useful for testing whether erasure occurs by overlapping distributions across unwanted attributes. We must note that, the erasure process is carried by a deterministic function and only shuffling data samples across unwanted attributes is not sufficient for erasure as this shuffling can be still invertible. Therefore, true erasure happens upon distribution matching.
In our experiments, we observed that applying dimensionality reduction often improved adversarial accuracy, For instance, using a strong encoder-an MLP with four hidden layers(see &D.1) -D.hhwithout any iterative refinement led our MLP adversaries to perform random chance level while SVM with 10, C = 10 reported around 80% accuracy. However, when we applied RKHS refinement via (11) as a dimensionality reduction step and re-evaluated the MLP adversaries again, their accuracy was similar to that of SVM with 10, C' = 10. This indicates that sensitive information was still present in the representation, and the prior adversary failures were primarily due to training difficulty rather than successful erasure.
C.2 Ablation Studies with Different Probing Networks
To evaluate the robustness of our probing setup, we assess the trade-off curves using multiple types of classifiers. Specifically, we consider Multi-Layer Perceptron (MLP) with five hidden layers of size 128. as well as Random Forest classifiers with 100 estimators and maximum depths of 15, 20, and 25. For the MLP, we report the highest accuracy across three independent runs. For the Random Forest, we report the best accuracy obtained across the three depth settings. The resulting trade-off curves are shown in Figure Sa, alongside the curve obtained using our default probing classifier (as described earlier). As illustrated in the figure, our choice of classifier yields a consistently lower trade-off profile, demonstrating its robustness for evaluating erasure performance.
In contrast prior work, we do not rely on visualizations produced by t-SNE or UMAP to assess overlap with respect to the sensitive attribute (c.g., gender). While such techniques can be visually appealing. they are unreliable indicators of structure in high nigh-dimensional spaces, often distorting distances and cluster separability. Instead, we adopt an evaluation based on the training accuracy of an expressive classifier. In particular, if sufficiently flexible probing model is unable to overfit the training data, this provides stronger evidence that the representation has been overlapped across the unwanted attribute, suggesting successful erasure. However, it is important to note that we are working with finitely many samples in high dimensional space. It is unlikely that the data points become so close that no probing classifier can overfit to training samples. For this reason, the ficxibility of the probing model must be chosen reasonably so that this evaluation becomes meaningful. Figure 10 presents the train and test accuracy of the MLP with five hidden layers over the course of erasure. As the process progresses, training accuracy falling to near random chance. This indicates that even a high-capacity classifier cannot overfit to the training samples, suggesting that the representation has been successfully aligned across the sensitive groups via distributional overlap.
observe a drop in both training and test accuracy, with the
Taken together, these probing results indicate that Obliviator consistently achieves full concept erasure. This consistent with the theoretical motivation: minimizing HSIC(S, X) enforces statistical independence between the representation X and the sensitive attribute S.
22

---

80
80
Profession Ace.
Profession Avc.
70
(MLP32-SVMD
70
Train MLP55
1F-100-11.2000
20,222
60
Test MI Ps5
Random Chance
50-
50
40
30
30
60
70
80
90
100
60
70
80
90
100
Gender Acc.
Gender Acc.
(a) Comparison of ohtained trade-off with deeper MI.P (5 hidden layers) and Random Forest with our choice
(b) Train and test accuracy for MI. with 5 hidden layer at different step of the erasure.
of classifier (MLPx2-SVM),
Figure 8: Ablation studies with different probing networks. Dataset is BIAS IN BIOS and language model is BERT.
80
Profession Acc.
70
E: MLP 1 X 256 <MPP 256 Obliviator Random Chance
60
50
40
30
60
70
80
90
100
Gender Acc.
Figure 9: Multi-Step Erasure vs. Single-Step Erasure. Comparison of erasure performance using single-step encoder with either or 4 hidden layers (MLP), versus proposed multi-step erasure framework (Obliviator). Results are shown on the BIAS IN B1OS dataset using BERT representations. The multi-step approach consistently demonstrates more effective erasure.
D Additional Results
D.1 Single Step Erasure Vs Multi-Step Erasure
To highlight the importance of each component in Obliviator, we evaluate simplified baseline using four hidden layers (each containing 256 neurons). The goal is to perform erasure using only the encoder trained with the loss in (8), without employing the multi-step framework or these components and assess their effect on erasure and adversarial optimization. During training, we evaluate both the target task and undesired attribute accuracy every 30 steps. As shown in Figure neither of the single-step encoder configurations achieves full concept erasure. Although both gender and profession accuracies initially decline, they eventually plateau and begin to oscillate, failing to reach full erasure. While more expressive encoder might hypothetically improve performance, direct comparison reveals that empirical trade-off achieved by Obliviator consistently better. This demonstrates that the multi-step procedure and RKHS refinement, results in a more robust and effective erasure process.
single MLP encoder with either
refinement. This setup allows us isolate the contribution of
D.2 Further Results Fairness
Here we show the additional fairness for controlled setup experiments 5.3. Moreover, we show the fairness for BIAS IN B1OS and DIAL-MENTION before and after erasure.
23

---

Table 9: Erasure with Obliviator on DeepMoji representations in controlled setup. Split describe levels of unwanted attribute disproportion.
DeepMoji Rep.
Race Acc.
Split
50% %
60%
70%
80 %
Original
75.38
75.57
74.16
72.37
75%
75.28
74.68
73.89
70.44
Sentiment Acc(1)
65%
74.83
72.94
68.75
64.06
55%
74.53
69.62
62.82
57.82
Original
0.1078
0.2266
0.3109
0.3751
75%
0.0667
0.1028
0.2896
0,2293
DP())
65%
0.0280
0.0330
0.0848
0.0203
55%
0.0237
0.0600
0.0650
0.0671
Original
0.1360
0.2412
0.3212
0.3815
75%
0.1026
0.1324
0.2991
0.2350
GaPRMS(L)
65% %
0.0920
0.0951
0.1251
0.0917
55%
0.0910
0.1050
0.0921
0.0915
Table 10: DP and Gap for BIAS BiOs Frozen Representations
Method
BERT
DeepSeck
Original
0.0204
0.0205
DP(L)
Obliviator
0.0147
0.0120
Original
0.1573
0.1522
Gap (L)
Obliviator
0.0464
0.0662
Table 11: DP and Gap for DIAL-MENTION Frozen Representations
Method
DP(.)
Gap (4)
Original 0.219
0.2109
Obliviator
0.006
0.0419
D.3 Hyperparameter Sensitivity
We conduct two additional studies to evaluate the impact of hyperparameters on the performance of Obliviator. First, we examine the effect of the RBF kernel bandwidth parameters 722 and 7 which are defined as the inverse the kernel width (i.e., y = 1/20²). These parameters control the smoothness and expressivity of the witness functions used in dependency estimation. Intuitively, higher values (corresponding to narrower kernels) yield more complex witness functions, which are more sensitive to fine-graine correlations. This can result in more aggressive erasure by capturing spurious dependencies. This behavior consistent with the patterns observed in Figure 10a. Furthermore,
that Obliviator performs consistently across a broad, reasonable range of values for 7, demonstrating robustness to this choice. Note that, in this experiment, we considered 2 Yei
Next, we study the effect of the weighting coefficients Ty and Tr, which appear in both the adversarial loss (8) and the eigenvalue problem in (11), In this experiment, we vary only the encoder loss parameters while keeping the EVP parameters fixed, as specified in Table As before, we set Tz² Ta The results, shown in Figure 10b, indicate that very large values of lead to slight degradation performance. Moreover, increasing 7 slows convergence, requiring more iterations to reach the same level of erasure.

---

80
Profession Ace.
Profession Avc.
70
70
0.86 0.1
0.01
60
60
0.05
50
Random Chance
50
Random Chance
30
30
60
70
80
90
100
60
70
80
90
100
Gender Acc.
Gender Acc.
(a) Trade-off obtained by different values of
(b) Trade-off obtained by different values of T
7
7/
Tx Tza. Here we only change the encoder's pa- rameter and in EVP problem
in Table 2.
Figure 10: Analysis of the effect of hyperparameters on Obliviator's performance. Dataset is BIAS IN BIOS and language model is BERT.
(a) Professions
Professor
(c) Physician
Figure 11: Representations Learned by Obliviator on Finetuned BIAS IN BTOS Representations (BERT). The professions Professor and Physician are shown separately to better visualize the distribution of gender within each class. While the two professions clearly separated (green and purple), gender labels (blue and red) are indistinguishable within each profession inffessioniiniicainnn indicating that Obliviator effectively erases gender information while preserving task-relevant structure.
D.4 Visualization
Here, we visualize the representations of two professions- Professor and Physician-from the BIAS IN BIOS dataset using fine-tuned representations at an intermediate stage of erasure. This point the trade-off curve where gender classification accuracy is approximately 60%, while profession classification accuracy remains around 85% (see Figure 4c). The visualization in Figure 11 also corresponds this stage of erasure.
representation corresponds
At this point, the number of RKHS encoders obtained from (11) three, meaning we visualize the actual transformed representation used by the model. Notably, while the two professions remain well-separated, the gender labels within each profession are indistinguishable- hiibsinguuhabbll--hgghhhhhiiggg that Obliviator successfully removes gender information while preserving task-relevant structure.
E
Limitations
Obliviator requires precise definitions of unwanted attributes to effectively crase associated concepts. However, such definitions may be unavailable vary across cultural and ethical contexts. Pseudo- labeling using existing Al models can partially mitigate this, but such labels may be noisy and impact model performance, factor not investigated in this work. Similar limitations apply to supervised erasure, which assumes access to target task labels.
Our method affords more stable optimization by leveraging tractable components: HSIC to quantify statistical dependency and RKHS disentanglement to make task-relevant information more accessible during optimization. Our experiments confirmed the more
crasure of Obliviator
25

---

compared to baselines across various setups. In some scenarios, it achieved full erasure without sacrificing utility, setting strong benchmark for the crasure-utility trade-off. Nevertheless, our method solves the non-convex optimization in (4.1) numerically, and a closed-form, theoretical guarantee for optimal nonlinear erasure remains an open problem.
Societal Impact
Obliviator enables targeted concept erasure from learned representations, which can be beneficial for instance, by removing sensitive demographic attributes to reduce reliance on them in decision-mant makiing. or by erasing personal identifiers protect privacy. However, if task-critical attributes "unwanted"" such as age in healthcare or income level in finance, model performance may degrade, or the model may rely spurious correlations with remaining features like gender race. Transparent documentation, domain-informed definitions, and independent oversight are essential, especially in high-stakes applications. Without careful review, erasure may discard socially meaningful information or fail to generalize across contexts.
treated as
26

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We introduce a multi-step framework (Obliviator) for nonlinear concept erasure. Our method can guard against nonlinear adversaries, and reveals empirical upper bound on the trade-off between preserving utility and undesired attribute removal. There are extensive experiments in the paper that support claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have section for limitations of our method in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: All assumptions are clearly explained in the paper and proofs from other resourees are properly cited. Moreover, we deferred the rest of the derivations/proofs to the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The datasets that we used are publicly available and properly cited in the paper. The choice of hyperparameter and models that we used for the set of experiments are clearly explained and provided in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The datasets we have used are publicly available and properly eited in the paper. The code is not currently open-source but will be made public when the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
httpr/gguuddes/oeeubbbmiii wttpesuiiisssooooooooooooo onssiinnolycyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //npp..c..... httpcc/ppppllllcc httplliccggii//eeeeeeee Sonideee//oeSuummieeeeee cttpsuumissiioooooieioooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the experimental setups, including the hyperparameters, probing networks and dataset details are explained in detail in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: As described in the paper, we compute trade-off curves from five independent runs for each experiment. For these runs, the train/test split remains fixed, while the encoder initialization varies. To explicitly demonstrate the variance and stability of our method, we plot the results of all five runs individually rather than aggregating them. Furthermore, as detailed in the Appendix, we ensure rigorous evaluation of erasure by reporting the highest leakage found across 10 different nonlinear adversaries for each run.
Guidelines:
29

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details of computer resources is provided in supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: / htt///eerrpppp http///urrpp.ccppuliii.ee//////t EEttei csGuideliness
Answer: [Yes]
Justification:
All the experiments follow NeurlPS's Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We put the discussion in the supplementary material.
30

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited all the codes, datasets, and models in the paper we used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com condaataeess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not introduce any new assets such as datasets, models, or benchmarks beyond the code provided to reproduce results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /atpas:///uriipee cc / for what should or should not be described.
33


Reviews:
[{"review_id": "lzIazqUVMt", "reviewer": "Reviewer_4SEV", "summary": "This paper introduces Obliviator, a multi-step framework that gradually transforms the feature space to facilitate smooth and effective concept removal. It achieves this by capturing nonlinear statistical dependencies between learned representations and undesired attributes. Each iteration involves:\n- Adversarial training in Reproducing Kernel Hilbert Space (RKHS): An encoder is trained to reduce dependence on the unwanted attribute while increasing dependence on the target attribute, utilizing the Hilbert-Schmidt Independence Criterion (HSIC) to enforce independence. RKHS enables capturing nonlinear dependencies in a closed form, providing a structured framework for adversarial optimization.\n- RKHS Refinement: The representations are refined using RKHS functions derived from an eigenvalue problem, enhancing alignment and smoothness for the target attribute without increasing these properties for the unwanted concept. This refinement helps condition the feature space for the next iteration's adversarial training.\n\nThe paper claims several key contributions, supported by its empirical evaluation:\n- Robustness to Nonlinear Adversaries\n- Quantification of Erasure Cost and Empirical Upper Bound.\n- Generalizability Across Language Models.", "questions": "- The paper acknowledges that the rigorous experimental setup is \"computationally demanding\". Could the authors provide a more detailed breakdown of the computational cost (e.g., average training time per iteration or per full erasure process, GPU memory footprint) for typical configurations (e.g., Frozen+Unsupervised vs. Finetuned+Supervised) and across different models/datasets used? If possible, a comparison to the computational cost of leading baselines would be highly valuable.", "limitations": "- The rigorous experimental setup, while contributing to quality, is acknowledged as \"computationally demanding\". This could limit its scalability or adoption for researchers with fewer computational resources.\n- Obliviator requires \"precise definitions of unwanted attributes\". In real-world scenarios, such definitions may be ambiguous, unavailable, or vary culturally and ethically, potentially limiting practical applicability.\n- The paper notes that pseudo-labeling, if used, \"may be noisy and impact model performance, a factor not investigated\". Similarly, supervised erasure \"assumes access to target task labels,\" which might not always be available.\n- Despite its advancements, the method is \"still unable to quantify the minimum intervention needed for full concept erasure or determine whether erasure can occur without compromising unrelated information\". This indicates a theoretical boundary that remains unaddressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ Obliviator explicitly enforces statistical independence between the transformed representations and sensitive attributes by setting HSIC to zero (HSIC = 0 via Equation (6)). This is a sound theoretical approach to ensuring nonlinear guardedness, which is empirically validated to be effective, unlike prior nonlinear methods like kSAL that failed to provide true nonlinear guardedness.\n\n+ The application of Reproducing Kernel Hilbert Spaces (RKHS) and the Hilbert-Schmidt Independence Criterion (HSIC), which are foundational to the proposed method, is explicated in detail, including the mechanism by which nonlinear statistical dependence is quantified in a closed-form solution.", "was_revised": true, "final_justification": "The authors have provided a comprehensive and convincing response to the feedback. I would keep the final score to boardline accept.", "rebuttal": "# Summary of the Review & Response\n\nWe thank the reviewer for their valuable and positive feedback.  The reviewer acknowledged our *theoretical approach* to formulate the concept erasure problem and that it is *empirically validated to be effective* compared to previous nonlinear approaches (S1). The reviewer identifies that the application of *RKHS* and HSIC for measuring true statistical independence is *explicated in detail* (S2).\n\nThe reviewer raised three key concerns:\n\n1. Computational complexity and scalability (Q1, L1)\n2. Label availability for erasure (L2, L3)\n3. Lack of a theoretical bound on the optimal erasure trade-off (L4)\n\nThese concerns align with the challenges we already discussed in the paper’s Limitations (Section 6). Our responses:\n\n1. We clarify the central misunderstanding: the \"computational demand\" refers to our offline **experimental** **evaluation process**, not the *Obliviator* algorithm itself. To provide concrete evidence of *Obliviator's* practicality and scalability, we have added **tables TR1-TR5** with detailed time and memory analysis.\n2. While this remains a broader challenge in the field, we note that recent SoTA methods increasingly rely on **pseudo-labeling** to overcome limited access to manually annotated sensitive attributes.\n3. We clarify that while our solution is not fully closed-form, it affords stable optimization by leveraging tractable components like HSIC for dependence measure and RKHS refinement for feature space alignment.  Our experiments show *Obliviator* achieves full erasure without sacrificing utility, setting a strong benchmark for the erasure-utility trade-off.\n\n# Response to Questions:\n\n> ### **Q1) Computationally Demanding and Time/Memory Complexity BreakDown**\n> \n\nOur reference to \"computationally demanding\" (in the NeurIPS Checklist) refers to Obliviator’s evaluation across different datasets, models, and downstream non-linear classifiers. We acknowledge that this is uncommon and may have caused confusion. As we demonstrate below, Obliviator itself is computationally very efficient.\n\nAs detailed in Appendix C (“Probing Networks”), our evaluation is intentionally rigorous. To robustly measure information leakage, we train 10 distinct probing classifiers, among which the training of hard-margin SVMs was the most computationally intensive component. We report the maximum accuracy across all probes as a worst-case leakage estimate.\n\nThis exhaustive evaluation is performed offline to validate our method's effectiveness, and its cost is entirely separate from the operational cost of the erasure algorithm. We believe such thorough validation is crucial for making reliable claims about the concept erasure problem.\n\nTables [TR1] and [TR2] below detail the time and memory complexity of our method. All benchmarks were run on a single NVIDIA RTX-A6000 GPU, using the full BiasInBios dataset (250K samples) in a single batch.  \n\n### **TR1. Time Complexity**\n\n| Time  Complexity | Ads Training (Per Epoch) | RKHS Ref |\n| --- | --- | --- |\n| Theoretical | $O(Nd_{embd}d_{hidden} + Nd_{rff}d_{hidden}+ Nd^x_{rff}d^s_{rff}+Nd^x_{rff}d^y_{rff})$ | $O(d_{rff}^3 + Nd^x_{rff}d^s_{rff}+Nd^x_{rff}d^y_{rff})$ |\n| BERT | 177.2 ms ± 0.38 ms | 172 ms ± 1.23 ms |\n| DeepSeek | 219.6 ms ±1.4680 ms | 165 ms  ±  1.65 ms |\n||||\n\n### **TR2. Memory Complexity**\n\n| Memory  Complexity | Ads Training | RKSH Ref |\n| --- | --- | --- |\n| Theoretical | $O(d_{embd}d_{hidden} + d_{rff}d_{hidden} +Nd_{rff}+d_{rff}^2)$ | $O(d_{rff}^2 + Nd_{rff} )$ |\n| Utilized Memory | 8.18GB | 6.78GB |\n|  |  |  |\n\n\nThe table below reports the time required to achieve specific gender accuracy targets, with results broken down by language model, dataset, and erasure setting. (Random Chance=RC)\n\n\n### **TR3-Bias in Bios Frozen**\n\n| ***BERT*** | 99.4%→70% (sec) | 70%→65%(sec) | 65%→60% (sec) | 60%→RC (sec) | 99.4%→RC (sec) |\n| --- | --- | --- | --- | --- | --- |\n| Unsupervised | 21.9±0.05 | 27.4±0.06 | 107.76±0.26 | 323.7±0.7 | 482.9± 1.1 |\n| Supervised | 27.1±0.06 | 72.26±0.1 | 153.5±0.3 | 487.7±0.5 | 740.7± 1.7 |\n| ***DeepSeek*** |  |  |  |  |  |\n| Unsupervised | 74.3±0.5 | 67.53±0.46 | 94.54±0.64 | 189.08$\\pm$1.28 | 425.4±2.9 |\n| Supervised | 156 ±1 | 100.3±0.67 | 390.1±2.63 | 1070$\\pm$7.2 | 1716.3 ±11.6 |  \n|||||||\n\n### **TR4-Dial Mention Frozen**\n\n| ***BERT*** | 80%→70% (sec) | 70%→60% (sec) | 60%→RC (sec) | 80%→RC (sec) |\n| --- | --- | --- | --- | --- |\n| Unsupervised | 55.2±0.1 | 193.3±﻿0.5 | 1293.2 ±﻿3.0 | 1541.67±﻿3.6 |\n| Supervised | 126.4±﻿0.3 | 144.5±﻿0.3 | 1372.9±﻿3.7 | 1643.8±3.8 |\n||||||\n\n### **TR5-Bias in Bios Finetuned**\n\n| BERT | 99.4%→70%(sec) | 70%→65% (sec) | 65%→63% (sec) | 63%→RC (sec) | 99.4%→RC (sec) |\n| --- | --- | --- | --- | --- | --- |\n| Unsupervised | 37.7±﻿0.08 | 62.8±﻿0.14 | 540.7±1.2 | 4225.6±9.35 | 4866.7±9.6 |\n| Supervised | 43.2±﻿0.1 | 162.06±﻿0.36 | 108.04±0.24 | 5596±12.4 | 5909±12.2 |\n|||||||\n---\n## **Comparison with Baselines**\n\n### **TR6- FARM Bias in Bios gender(acc)=62.91**\n\n| Batch Size | Memory(MB) | Time( sec per epoch) | Total Time(sec) |\n| --- | --- | --- | --- |\n| 1024 | 96.4 | 21.4±1.2 | 535±30 |\n| 4096 | 305.44 | 13.22±1.03 | 330.5±25.75 |\n|||||\n\n### **TR7-KRAM Bias in Bios gender(acc)=68.45**\n\n| Batch Size | Memory(MB) | Time(sec per epoch) | Total Time(sec) |\n| --- | --- | --- | --- |\n| 1024 | 76.8 | 65.58±1.4 | 3279±65.2 |\n| 4096 | 479.04 | 45.81±8.5 | 2290±42.5 |\n|||||\n\n---\n# Response to Limitations:\n\n> ### **L1) Adaptation for research**\n> \n\nWe wish to respectfully clarify this point further. The \"computationally demanding\" aspect refers exclusively to the **offline evaluation protocol** we used to validate our results, not our proposed algorithm, Obliviator.\n\nTo make this distinction unambiguous, **Tables TR1-TR2** provides a detailed breakdown of the elapsed time for each step. As the table demonstrates, the runtime of Obliviator is highly practical. The intensive evaluation suite, while crucial for making reliable scientific claims, is not a required step for applying our method. The computational requirements of Obliviator itself do not pose a barrier to its adoption for research and scalability. \n\n> ### **L2) Precise Definition of Unwanted Attributes**\n> \n\nWe agree and discuss this very point in our **Limitations (Section 6, Page 9)**, acknowledging that defining unwanted attributes is challenging and context-dependent.\n\nAny concept erasure method fundamentally requires a precise definition of the concept to be removed. In the context of our paper, all discussed baselines rely on sensitive attribute labels for erasure. The challenge of defining these concepts is analogous to the often subjective task of defining fairness criteria in ML.\n\nOur work focuses on the crucial next step: *given* a defined concept, how to erase it with maximum efficacy. While we note future work could aid the definition phase (e.g., via pseudo-labeling ), our contribution is in advancing the erasure mechanism itself. \n\n> ### **L3)  Label Availability**\n> \n\nWe agree that the availability and quality of labels, for both the unwanted concept and the target task, are crucial challenges, which we acknowledge in our Limitations section (Section 6).\n\nWe believe that existing pseudo-labeling techniques, facilitated by more powerful foundation models, offer a viable path to address these challenges. Recent literature provides strong evidence for this. For instance, [R1] use an **optimal transport framework** to handle uncertainty in pseudo-labels as part of an augmented training phase, improving performance in low-resource settings. Similarly, [R2] use pseudo-labels for debiasing and show in their Table 2 that this approach is highly competitive with models trained on true labels.\n\nThese methods demonstrate that robust solutions for label generation under uncertainty are actively being developed. In contrast, our work focuses on seeking the optimal/near-optimal erasure mechanism that can operate effectively on labeled data, whether they are ground-truth or high-quality pseudo-labels.\n\n> ### **L4) Minimum Intervention**\n> \n\nWe agree and explicitly state in our **Limitations (Section 6)** that our method is \"still unable to quantify the minimum intervention needed for full concept erasure\". To our knowledge, a closed-form, theoretical guarantee for optimal nonlinear erasure remains an open problem in the field.\n\nWhile a theoretical optimum is yet to be found, our work focuses on studying the utility-erasure trade-off, and establishing a strong **empirical upper bound** on this trade-off. For instance, **Figs. 3a, 3b, and 6b** show scenarios where our method achieves nearly full erasure of the unwanted attribute while preserving a high degree of task accuracy, indicating its practical ability to preserve unrelated information during erasure.\n\nThe effectiveness of our approach stems from its design. While the main optimization (Eq. 6) does not have a closed-form solution, we make the optimization more tractable by:\n1.  Using  HSIC to capture nonlinear dependencies in a closed form.\n\n2.  An iterative process, each with two-steps that refines the feature space via an RKHS-based EVP (Eq. 11), which facilitates smoother erasure and more consistent optimization.\n\nThe significant impact of the two-step process is demonstrated in our ablation study in **Appendix D.1 (Fig. 9)**, where we observe erasure by a single step with encoder is less stable and utility-preserving, unlike *Obliviator*. This provides strong evidence that our solution, while not offering a theoretical guarantee, represents a significant step towards achieving optimal erasure in practice.\n\n---\n\n**Please consider raising the scores if we addressed your initial concerns. If you have more questions, we would be happy to answer them.**\n\n# References\n\n[R1] FFB: A Fair Fairness Benchmark for in-processing group fairness methods,  ICLR2024\n\n[R2] FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs, ICLR2024"}, {"review_id": "KRUqvEPqbb", "reviewer": "Reviewer_X9EX", "summary": "The paper focuses on non-linear concept erasure. The idea is to focus on RKHS to find concepts that are non-linear. The paper provides a battery of experiments. The paper tackles an important topic and one that is of personal interest to me. The experiments are well executed. Moreover, the", "questions": "* Can you describe and motivate Equation 1? This is what my review will mostly hinge on.\n* Can you describe the relationship with kernelized R-LACE (https://arxiv.org/abs/2201.12191) and kernelized SAL better?\n* The original guardedness paper (https://arxiv.org/abs/2210.10012) formulates guardedness in terms of V-information. Does that definition hold here? Can you give such a definition to make it easier to relate to previous work?", "limitations": "Mentioned above.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "I saved most of my text for this box. My fundamental problem with understanding this paper is Equation 1. Everything else derives from it. First and foremost, I am not sure what the random variables X and Y. What I would need to really understand this paper is to understand which of the variables X and Y correspond to neural representations and why. I simply don't understand why Eq.  (1) is the right adversarial game. This seems crucial to me and made it hard to engage with the rest of the paper. To be clear, I am willing to believe it is, but the paper needs to be substantially improved in this part. Beyond that, I think the contribution of the paper is to come up with a kernelization of Eq. (1). \n\nI also wish the paper had more background on Gretton et al. -- as I understood it, there seems to be some sort of expressivity result.", "was_revised": true, "final_justification": "This is a great paper. The authors answered my concerns.", "rebuttal": "# Summary of the Review & Response\n\nWe appreciate the reviewer's feedback and acknowledgement that our paper tackles an *important* topic with a *battery of experiments* that are *well executed (summary).*\n\nThe reviewer's primary concerns are:\n\n1. Motivation behind Eq. (1)(Q1)\n2. The relationship between previously proposed methods, Kernelized SAL and R-LACE (Q2)\n3. The connection between our framework and the V-information formalism for guardedness (Q3)\n4. The reviewer also perceived our contribution as a \"kernelization of Eq. (1)\". \n\nOur responses are summarized as:\n\n1. We clarify the motivation behind Eq. (1) through a pedagogical toy example to build intuition, showing how Eq. (1) formalizes the objective of a linear test.\n2. We provide a detailed technical comparison for Kernelized R-LACE and kSAL, highlighting a key technical flaw in these prior methods: they rely on an unverifiable Gaussian assumption in the feature space, a limitation that our adversarial model overcomes.\n3. We connect our HSIC-based approach to the V-Guardedness formalism, demonstrating that our guarantee of independence is stronger and more general.\n4. We clarify that our core contribution is not merely a \"kernelization”, but the design of an optimization algorithm that progressively erases the undesired concept information, with each iteration involving two steps. We explain how this solution, with its utility-preserving loss and RKHS refinement step, operationalizes the principles of HSIC for effective and consistent nonlinear erasure.\n\n# Response to Questions:\n\n> ### **Q1) Motivation Behind Eq. (1)**\n> \n\nFirst let us clarify the crucial point that Eq. (1) describes a linear statistical test (maximize correlation) while Eq. (6) defines an adversarial game. An adversary, using functions $f$ and $g$ from an expressive RKHS, seeks to maximize the correlation between $g(Y)$ and $f(ε(X))$. The guarding function $ε$ is simultaneously trained to minimize this correlation.\n\nTo motivate the objective behind Eq. (1), we begin by the standard assumption in statistical learning:  we model $X$(learned neural representation) and $Y$ (class label like profession/sentiment) as random variables, and our data as samples from their joint distribution $P(X,Y)$. To build intuition for why Eq. (1) is a natural starting point, consider a toy example where a process generates each data pair from three independent latent variables, $Z_1,Z_2,Z_3$, each drawn from a Gaussian $\\mathcal{N}(0,1)$. The observable random variables X and S are formed as follows:\n\n$$\nX=Z_1v_1+Z_2v_2\\qquad Y=Z_1u_1+Z_3u_2\n$$\n\nwhere $v_i$ and $u_i$ are pairs of linearly independent unit vectors. In this case, any observed correlation between samples of $X$ and $Y$ is due *only* to the latent factor $Z_1$. For a dependency test we can begin by the following question :\n\nCan we find \"viewpoints\" that isolate the shared $Z_1$? \n\nEq. (1) formalizes this  for the optimal linear projections $(u, v)$ that maximize correlation. For this example, the solution would be $v_1$ and $u_1$.\n\nFollowing this, we can also ask the inverse: Can we find viewpoints of $X$ that are independent of $Y$ ?  \nYes, if we project $X$ to orthogonal complement of $v_1$.  Therefore,  independence  requires filtering out this information (erasure) from $X$.\n\nThis example illustrates how, for Gaussian RVs, the linear test in Eq. (1) is sufficient to describe their dependence structure completely.\n\n> ### **Q2) Kernelized R-LACE/SAL**\n> \n\nTo clarify the relationship between these methods, we first examine how they address **linear dependencies**.\n\nThe key distinction lies in the approach, not the outcome:\n\n- **R-LACE** adopts an adversarial perspective, learning a projection to thwart a linear classifier attempting to predict the sensitive attribute.\n- **SAL**, akin to the Gaussian toy example, takes a statistical approach, finding projections that nullify the linear correlation between the representation and the sensitive attribute.\n\nThese are equivalent in the linear setting: zero linear correlation implies that no linear predictor can succeed. However, this only guarantees independence under approximately Gaussian distributions. For other distributions, zero correlation does not imply independence.\n\nA natural extension is to kernelize the method:\n\n1. Map inputs X to a high-dimensional space via a Mercer feature map $\\phi(X)$.\n2. Apply a **linear** erasure method in this space.\n\nBut this only protects against *linear* adversaries in that specific kernel space. As shown in Kernelized R-LACE, such protection does not generalize to other nonlinear adversaries. Like the Gaussian case, independence is only guaranteed if $\\phi(X)$ is approximately Gaussian. The same limitation applies to KSAL. Statistically, optimizing against a kernel regressor only ensures *mean independence*, not full independence [R2, R3, R4].\n\nOur method addresses these limitations by using a stronger adversary. Rather than checking for linear dependence in a fixed space $\\phi(X)$, we ask whether *any* nonlinear transformation $\\psi(\\cdot)$ exists such that a linear dependence can be found between $\\psi(\\phi(X))$ and $\\ell(Y)$ (see Eq. 6). This effectively constructs a composite feature map $\\psi(\\phi(X))$, and from a statistical standpoint, optimizes for true *independence*.\n\nThis nested optimization problem lacks a closed-form solution and is substantially more challenging than kernelization. We propose a more **structured and consistent optimization** framework to solve it effectively.\n\n> ### **Q3) Guardedness in terms of V-information**\n> \n\nThe relationship is best understood by first considering the different, but equivalent, mathematical languages used to define **statistical independence**.  There are several equivalent ways to define statistical independence between two random variables, $X$ and $Z$:\n\n- **Probabilistic:** The conditional and marginal probabilities are the same: $P(Z|X)=P(Z)$.\n- **Information-Theoretic:** Knowing $X$ does not reduce uncertainty about $Z$. Formally, the mutual information is zero: $I(X;Z)=0$.\n- **Functional Analysis:** This view states that if no functions $f$ and $g$ can be found to observe a linear correlation between $f(X)$ and $g(Z)$ then $X$ and $Z$ are independent. (HSIC$(X,Z)=0$  with characteristic kernel)\n\nV-guardedness, is a weaker form of the information-theoretic view. It asks if a classifier from a specific family V(e.g. log-linear models) can find any predictive signal.\n\nIn our approach by enforcing that HSIC $\\to$ 0, we ensure a stronger condition and guard against entire space of a characteristic RKHS, not just a specific family of classifiers. To connect this directly to the language of the V-guardedness we can say:\n\nDefinition (Nonlinear-Guardedness): Let $h$ be a guarding function, and let $\\mathcal{F}$ and $\\mathcal{G}$ be RKHSs with characteristic kernels. A representation $X$ is nonlinearly $\\epsilon$-guarded by $h$ with respect to an attribute $Z$ if HSIC$(\\mathcal{F},\\mathcal{G},P(h(X),Z))≤ϵ$. \n\n---\n\n# Response to Strengths and Weaknesses:\n\n> ### **The contribution of the Paper is Kernelizing Eq. (1)**\n> \n\nWe appreciate the opportunity to clarify our contribution. While Eq. (1) is the linear starting point, our core novelty is the formulation of the erasure problem from a functional perspective, as defined in Eq. (9,13), and the multi-step framework we propose to solve it.\n\nFirst we utilize a non-linear statistical test Eq. (3) to measure full statistical dependence. To see why, consider a simple example like $X=sin(Z)$ and $Y=cos(Z)$ where $Z \\sim \\mathcal{U}(-\\pi,\\pi)$. Here, $X$and $Y$ are dependent, yet their linear correlation (Equation 1) is zero. But are there exist functions, *f* and *g*, that can transform $X$ and $Y$ so that the dependency between $f(X)$ and $g(Y)$ becomes linear? For a clear visualization of such functions for this problem, we refer to Fig. 1 in [R1]. This question is precisely what HSIC addresses. By constraining these functions $f$ and $g$ to exist in RKHS, the search for them becomes tractable. If RKHS  has a characteristic kernel, it can approximate any function arbitrarily well, then HSIC being zero implies true statistical independence.\n\nOur contribution is to adopt a functional perspective to the adversarial game, and leverage RKHS in  Obliviator, specifically in defining the objective for our RKHS refinement step Eq. (9) and the adversarial loss Eq. (13). While we use the empirical HSIC as a tool in our approach, we architect the erasure process to be more utility-preserving by:\n\n1.  Our adversarial loss (Eq. 13) is designed to utilize HSIC for both  gradual and utility-preserving erasure.\n2. We solve a constrained optimization in RKHS, to find optimal functions that align the feature space for the *next* iteration of erasure. This is the key to make the optimization consistent and we have shown this in our ablation study (see Fig. 9 in appendix).\n\nTherefore, our contribution is not simply kernelization of Eq. (1), our contribution is formulating and solving the erasure problem from a functional perspective. \n\n> ### **More details on Gretton et. al. and expressivity**\n> \n\nFor expressivity of RKHS with a characteristic kernel see [R5]. For more mathematical details regarding the cross-covariance operator, HSIC see Appendix A.\n\n**Please consider raising the scores if we addressed your initial concerns. If you have more questions, we would be happy to answer them.**\n\n# References\n\n[R1] Statistical Consistency of Kernel Canonical Correlation Analysis, JMLR 2007\n\n[R2] Learning unbiased representations via Rényi minimization, arXiv:2009\n\n[R3] Representation learning with statistical independence to mitigate bias, WACV 2021\n\n[R4] On Characterizing the Trade-off in Invariant Representation Learning, TMLR 2022\n\n[R5]  Hilbert Space Embeddings and Metrics on Probability Measures, JMLR 2010"}, {"review_id": "7FQrvqEOtp", "reviewer": "Reviewer_BjYo", "summary": "Given the rapid progress of LLMs learning almost everything from the web data, Concept erasure is a key field focused on removing sensitive attribute information from learned representations while preserving their utility for intended tasks. To this end, the authors propose Obliviator, a post-hoc concept erasure method designed to remove attributes like social or demographic factors from learned representations of LLMs using supervised and fine-tuning strategies. It introduces a multi-step framework that progressively transforms the feature space to enable smooth concept removal through adversarial optimization, where its key aspect is its ability to capture nonlinear statistical dependencies between learned representations and undesired attributes, addressing a significant limitation of existing methods.", "questions": "Please see the Strengths and Weaknesses section for more details.", "limitations": "Please see the Strengths and Weaknesses section for more details.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The key strength of the proposed work is that, in contrast to existing methods, it effectively guards against nonlinear adversaries, which have been shown to circumvent existing nonlinear erasure techniques.\n\n2. The proposed method shows strong performance across different families of large language models, maintaining its effectiveness in nonlinear guardedness and performance.\n\n**Weaknesses and Open Questions**\n\n1. For frontier models, we don't know anything about the training data. How does one understand the existence of any confounders for the concepts that need to be erased?\n\n2. Given that Obliviator is a multi-step framework that uses multiple updates, it would be great if the authors could comment on the computational complexity of their proposed algorithm vs. the baselines.\n\n3. The authors raise the question of the \"cost of concept erasure,\" but the cost of erasure analysis is limited to utility performance and is underexplored.\n\n4. How does the proposed algorithm specifically utilize RKHS to capture nonlinear statistical dependencies, and what advantages does this offer over existing linear or less robust nonlinear methods? \n\n5. Minor: It will be beneficial for the reader if the authors could add some implementation description of Obliviator in the main document.", "was_revised": true, "final_justification": "The paper introduces a novel post-hoc concept erasure method designed to remove protected attributes from learned representations of LLMs. I vote for accepting this work.", "rebuttal": "# Summary of the Review & Response\n\nWe thank the reviewer for their positive and insightful feedback. We are encouraged that they find concept erasure to be a *key field*  (summary) and that they recognize the core strengths of our work: its ability to *capture nonlinear statistical dependencies* (summary) , *effectively guard against nonlinear adversaries (S1)*, and show *strong performance across different families of large language models (S2).*\n\nThe reviewer's primary concerns are:\n\n1. Erasure on frontier models and unobserved confounders (Q1)\n2. Computational Complexity  (Q2)\n3. Limiting the cost of erasure to utility performance (Q3)\n4. Utilization of RKHS for capturing nonlinear dependencies (Q4)\n5. Minor: Implementation detail (Q5)\n\nOur responses are summarized as follows :\n\n1. We clarify that the utility-erasure trade-off is a diagnostic of the specific model's learned dependencies, not necessarily an inherent real-world link. We discuss the causal nuances of unobserved confounders versus spurious correlations learned by the model, connecting these concepts to our empirical results comparing BERT and newer models like DeepSeek.\n2.  We provide a theoretical complexity analysis for each component of *Obliviator*, our RFF-based design ensures obliviator is both scalable and practical. We also refer to our response to Reviewer [4SEV] for detailed empirical benchmarks.\n3. We explain that our focus on the utility-erasure trade-off addresses the most fundamental axis of cost. We clarify that our work provides the best-known **empirical trade-off**, which is a foundational prerequisite for studying other costs of interest, like perplexity, etc. \n4.  We detail the twofold role of RKHS in our framework: (1) to robustly **measure** nonlinear dependence via HSIC, and (2) in our novel **refinement step**, to **guide** the optimization by constructing optimal encoders that create a better-conditioned feature space for the next iteration.\n5.  The implementation details are in Appendix B. To aid reproducibility, we will update Appendix B with a PyTorch code snippet and also make the full implementation code publicly available upon publication.\n\n# Response to Questions:\n\n> ### **Q1) Erasure on Frontier Model and Unobserved Confounders**\n> \n\nA fundamental challenge for any post-hoc erasure method operating on frontier models is the gap between statistical dependence and causal influence.\n\nOur method, *Obliviator*, is designed to enforce statistical independence between the representation $X$ and the sensitive attribute $S$, based on the data provided for erasure. The effectiveness of this process hinges on whether that data accurately reflects the dependencies we aim to remove. This leads to two scenarios, as your comment suggests:\n\n1. Missing Confounder: If a confounder $C$ influences both $X$ and $S$ (i.e., $C → X$ and $C → S$), but our erasure dataset contains no variation in $C$, our method will be blind to this causal pathway. Enforcing independence on the biased sample may not lead to robust erasure on the data where it is influenced by $C$. This is an inherent limitation of any method that learns from a finite, and potentially unrepresentative, sample of data.\n2. Spurious Correlations from Model Training: If the frontier model's original training process encoded a spurious correlation between the target label $Y$ and the sensitive attribute $S$ into the representation $X$, our method will also treat the resulting dependency between $X$ and $S$ as something to be removed. This may force the erasure to be more aggressive than is causally necessary, potentially leading to a greater loss of utility.\n\nHowever, it is important to note that since HSIC is agnostic to causal origin, it will account for all observable dependencies, whether they are direct, spurious, or confounded. Therefore, our erasure is statistically complete if the samples used for erasure are representative of the real-world data distribution.\n\nThis distinction may help explain one of our key empirical findings. We hypothesize that the sharper erasure trade-off observed on BERT compared to DeepSeek (Figure 6b) could be because older models like BERT have stronger spurious correlations between representations and attributes. Newer, more capable models like DeepSeek may have representations that are already better disentangled, making the erasure of the remaining statistical dependence less costly to the representation's utility.\n\n> ### **Q2) Computational Complexity**\n> \n\nWe have provided a detailed empirical breakdown of the computational cost (including wall-clock times and memory usage) and a comparison to baselines in our response to Reviewer[4SEV] (see tables TR1-TR5), which will be added to the appendix for your reference. For convenience, we summarize the theoretical complexity here.\n\nThe primary cost is calculating the empirical HSIC loss. There are two general approaches for this:\n\n- Direct Kernel Matrix Calculation: This method's time and memory complexity of $O(N^2)$ becomes intractable for large datasets. This restricts the method to small datasets, which can lead to a less accurate estimation of HSIC.\n- RFF Approximation: To enable full-batch training for a better HSIC estimation, we use a scalable alternative based on RFF. This approach has a time complexity of $O(N⋅d_{rff}^x.d_{rff}^y + N⋅d_{rff}^x.d_{rff}^s )$ and better memory complexity of $O(N⋅d_{rff})$.\n\nFor EVP step we use RFF to make this step tractable. The complexity breaks down is:\n\n- Constructing the matrix for the EVP involves  $O(N⋅d_{rff}^x.d_{rff}^y + N⋅d_{rff}^x.d_{rff}^s )$.\n- The eigenvalue solver itself has a complexity of $O(d_{rff}^3)$.\n\n> ### **Q3) Cost of Erasure**\n> \n\nWe agree that the \"cost of concept erasure\" is a rich, multi-faceted topic. Our work focuses on what we consider to be the most fundamental axis of this cost: the direct trade-off between erasure effectiveness and task utility.\n\nOur primary goal was to first define the optimization objective of true nonlinear guardedness (full erasure) from the perspective of the trade-off, then devise an algorithm to optimize the objective, and then to rigorously **measure the degree of erasure and information leakage**. This, in turn, allows us to analyze the resulting utility-erasure trade-off, a relationship that was previously unexplored. By establishing the best known empirical trade-off, we believe our work improves our understanding of the costs of concept erasure.\n\nWe agree there are **other  costs of interest (e.g., loss of perplexity).** A reliable method for utility-preserving erasure, like *Obliviator*, is a necessary prerequisite before other costs can be properly investigated.\n\n> ### **Q4) Capturing Nonlinear Statistical Dependencies and Its Advantage**\n> \n\nThe goal of concept erasure is to ensure that a downstream model cannot utilize information about a sensitive attribute $S$ from a representation $X$, whether for fairness, privacy, or other reasons. Linear methods are often insufficient for this goal. For example, after applying a linear erasure method like INLP to the **BiasInBios** dataset, a simple nonlinear adversary (an MLP) can still predict $S$ from the modified $X$ with over 99% accuracy. This demonstrates that significant predictive information remains. While tractable, linear approaches can provide valuable insights, but they often fall short of the ultimate goal of complete erasure.\n\nOur use of RKHS is twofold, involves both the measurement of dependence and the architecture of the erasure process itself.\n\n**1. To Robustly Measure Nonlinear Dependencies :** This is motivated by the failure of linear correlation, as seen in the example where $X=sin(Z)$ and $S=cos(Z)$ for $Z∼\\mathcal{U}(−π,π)$. The functional approach, which we adopt, asks: Are there any functions, $f$ and $g$, that can transform $X$ and $S$ so that the dependency between $f(X)$ *and* $g(S)$ *becomes linear?* For a visualization of such functions, we refer to Figure 1in [R1].\n\nThe theory behind HSIC guarantees that if we search for $f$ and $g$ within an expressive RKHS (i.e., one with a characteristic kernel), and no such functions can be found, then $X$ and $S$ are truly statistically independent. We leverage this powerful guarantee in our objective (Equation 6) to learn an erasure function $ε$ that is robust against this entire expressive class of functions.\n\n**2. To Guide the Optimization via RKHS Refinement :** Expecting that a concept can be removed from the complex representation space of an LLM in a single-step optimization without distorting other useful information would be a **very strong assumption**.\nHere, we introduce our second, novel utilization of RKHS. In our RKHS Refinement step (Equation 9), we find optimal RKHS-based encoders via a constrained optimization. The constraint in this equation confines the search for these encoders to the orthogonal complement of functions that increase alignment with the sensitive attribute $S$. Within this constrained space, the objective is to find encoders that enhance the alignment with useful information from previous steps, creating a better-conditioned feature space. This makes the optimization task for the main encoder in the next iteration significantly easier and more stable, leading to a more utility-preserving erasure.\n\n> ### **Q5) Minor: implementation description of Obliviator**\n> \n\nPractical implementation details are in Appendix B. We will update this section with a PyTorch code snippet of the core algorithm, and the full code will be made publicly available for reproducibility.\n\n---\n\n**Please consider raising the scores if we addressed your initial concerns. If you have more questions, we would be happy to answer them.**\n\n# References\n[R1] Statistical Consistency of Kernel Canonical Correlation Analysis, JMLR 2007"}, {"review_id": "evEVKUdCGI", "reviewer": "Reviewer_eWgU", "summary": "The paper presents Obliviator, a method that tackles the long-standing problem that \"concept-erasure\" methods often hide sensitive signals (like gender or race) only from linear probes while leaving them exposed to more powerful nonlinear classifiers. Building on a objective that simultaneously minimizes the Hilbert–Schmidt independence between the learned representation and the sensitive attribute while preserving task-relevant information, Obliviator iteratively manipulates the feature space to remove the information, and enables to trace a full erasure-utility curve along the iterations. Across BiasInBios and two dialect-sentiment benchmarks, it drives sensitive-attribute accuracy down to chance while matching or beating task performance of prior linear and nonlinear baselines, thereby offering a post-hoc nonlinear erasure method.", "questions": "1. BiasIbBios shows a different utility-erasure curve (in fig 4) than the two other datasets. Why is that? are profession and gender more inhernetly related to each other than dialect and mention/sentiment?\n\n2. Related to the previous question, have you considered running a controlled experiment where you modify the proportion of each sensitive group within in main-task class (or vice versa)?\n\n3. Have you considered running ablations on both the two steps in iterative methods, and also on the different loss components? are they are actually necessary? \n\n4. If I understand correctly the appendix, the paper uses RBF kernel and yet the erasure generalizes well to a strong MLP adversary. This seems to contradict [3] where it was reported that nonlinear erasure does not generalize well between kernels. Do you have thoughts on the differences?\n\n[3] Ravfogel, S., Vargas, F., Goldberg, Y., & Cotterell, R. (2022, December). Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 6034-6055).", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The problem this paper tackles is important and widely studied, and the approach is creative, where in step (i) a flexible encoder pushes the representations to regions where the RKHS separates \"good\" information from \"bad\" information, and in step (ii) a closed-form operation uses the current features to orthogonalize the sensitive information (as is done in LEACE and other methods). The paper backs this with broad experiments, showing that at every point on the utility-erasure curve Obliviator preserves more or similarly degree of task accuracy than all baselines, and that its ability to exploit task labels in a purely post-hoc setting further enhances that margin. \n\nConcept erasure is usually aiming to achieve on of two goals: (i) making the model's prediction independent or less related to the erased concept, (e.g., gender), thereby achieving a more fair model (e.g., [1]); (ii) using the erasure as an *intervention* technique, that surgically changes the forward pass with the goal of understanding how computation is carried out in the model (e.g., [2]). While the experiments are supporting the main claims of the paper -- achieving nonlinear erasure while maintaining better erasure-accuracy tradeoff than previous linear erasure methods -- they do not test the impact of inducing nonlinear erasure on end-metrics such as fairness of the resulting classifiers (e.g., true positive rate gaps), nor do they use them to interpret the model. Does nonlinear erasure translate to a more \"fair\" model (under some notion of fairness)? does it allow us to better estimate the causal effect of concepts on the model? I think that including some experiments that would emphasize the utility of the proposed method will improve the paper. \n\nI think the experimental section can be further strengthened by some additional ablation and a more controlled experimental pipeline, see the \"questions\" section.\n\n[1] Bolukbasi, T., Chang, K. W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29.\n\n[2] Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021). Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9, 160-175.", "was_revised": true, "final_justification": "The problem this paper tackles is important and widely studied, and the approach is creative. The new fairness results in the rebuttal support the empirical relevance of the new method.", "rebuttal": "# Summary of the Review & Response\n\nWe thank the reviewer for their positive and insightful feedback. We are encouraged that they recognize the *importance* of the problem and find our approach *creative,* backed by *broad experiments (S1)*. The reviewer also highlighted our key strength, *preserve a more or similar degree of task accuracy* than all baselines(S2).\n\nThe reviewer's primary concerns are:\n\n1. Difference between utility-erasure curves across different datasets. (Q1)\n2. Controlled setup experiment modifying the proportion of sensitive groups (Q2)\n3. Ablation studies on our two-step solution and its loss components (Q3)\n4. Contradiction with prior work on generalization to nonlinear adversaries (Q4)\n5. Reporting fairness metrics (W1)\n\nOur responses are summarized as:\n\n1. We clarify that the utility-erasure trade-off reflects the *model’s learned dependencies*, not intrinsic properties of the task. We show that newer models trained on BiasInBios exhibit significantly less trade-off than BERT, suggesting BERT had learned strong spurious correlations.\n2. We performed the proposed experiment on DeepMoji, varying sensitive group proportions. Results confirm that skewed sampling worsens the trade-off due to biased HSIC estimation.\n3. We conducted ablations on each loss component and the two-step procedure, demonstrating their joint necessity. We also contrast our HSIC-based approach with Kernelized R-LACE, showing that our method defends against a *cascading kernel* adversary and resolves the generalization limitations of prior work.\n4. We now report *Demographic Parity (DP)* and *GAP*, showing that *Obliviator’s* strong statistical erasure improves downstream fairness.\n\n# Response to Questions:\n\n> ### **Q1) Inherent Trade-off in Utility-Erasure**\n> \n\nThe differing trade-off curves across datasets and models reflect the causal challenges we discussed with Reviewer (BjYo-Q1). The trade-off revealed by *Obliviator* serves as a *diagnostic of the model’s learned representation*, including any spurious correlations acquired during pretraining. Since HSIC captures all statistical dependencies, *Obliviator* erases any such entanglements, regardless of their origin.\n\nWe hypothesize that the sharper trade-off observed for BiasInBios on BERT (Fig. 4c) arises not from a real-world causal link between gender and profession, but from a *spurious, model-specific correlation*. This is supported by the fact that newer models like *DeepSeek* and *LLaMA* show a milder trade-off on the same dataset (Fig. 6).\n\nAs you noted, the trade-off is also much flatter for BERT on the DIAL datasets (Fig. 3), suggesting that in BERT’s representation, *gender and profession* are more entangled than *race and sentiment/mention*.\n\nUltimately, post-hoc erasure does not measure an objective causal relationship, but how a specific model *internalizes* that relationship, shaped by its training data and inductive biases.\n\n> ### **Q2) Controlled Experiment**\n> \n\nWe ran the suggested controlled experiment, hypothesizing that an imbalanced erasure dataset would worsen the utility-erasure trade-off due to a biased HSIC estimate. To test this, we varied the proportion of the Race attribute within Sentiment classes on the DeepMoji and measured the resulting impact on sentiment accuracy at different levels of race accuracy.\n\n### **TR8- Controlled Setup**\n\n| Race Acc. | 50% | 60% | 70% | 80% |\n| --- | --- | --- | --- | --- |\n| original | 75.38 | 75.57 | 74.16 | 72.37 |\n| 75% | 75.28 | 74.68 | 73.89 | 70.44 |\n| 65% | 74.83 | 72.94 | 68.75 | 64.06 |\n| 55% | 74.53 | 69.62 | 62.82 | 57.82 |\n\nAs dataset imbalance increases, the utility-erasure trade-off becomes more severe. *Obliviator* preserves utility well on the balanced split but shows a significant performance drop at 80% imbalance for similar erasure. This finding highlights the inherent challenge of post-hoc erasure: the quality of the erasure and its ability to reflect a true causal graph between concepts is entirely dependent on how well the erasure data represents the real world, and how the model itself has encoded those relationships.\n\n> ### **Q3,Q4) Ablation on Multi-Step Erasure and Differences with Kernelized R-LACE**\n> \n\n### **Differences with Kernelized R-LACE:**\n\nAs outlined in our response to Reviewer (X9EX-Q3) and Reviewer (BjYo-Q2),  the key distinction between *Obliviator* and prior kernelized methods lies in the theoretical guarantees underpinning the approach. *Obliviator* is based on a formal statistical measure of independence (HSIC), while prior work is based on a specific adversarial game.\n\nKernelized R-LACE is a \"single-layer\" kernelization. It first maps the representation $X$ into a feature space via $\\phi(X)$ (Mercer Feature Map) and applies a linear adversarial game in that space, ensuring that the resulting representation has no linear dependency on the sensitive attribute extracted by a linear adversary in the $\\phi(.)$ space. This does not prevent another adversary from applying a different nonlinear map, say $\\psi(·)$ and reveal a form of dependency. Thus it uses a Linear test of $(φ(X), S) = 0$  and not a linear test of $(ψ(φ(X)), S) = 0$. From a statistical perspective, the former corresponds to mean independence while the later corresponds to full statistical independence.\n\nOur approach generalizes because it is based on the HSIC principle. Dependence between $X$ and $S$ exists if any functions, $f$ and $g$, can be found that make the relationship between $f(X)$ and $g(S)$ linear. The critical question is: how can we account for *all possible* functions an adversary might use? We need a space of functions that is expressive enough to approximate any potential adversary. An RKHS with a characteristic kernel has this property, as it can approximate any continuous function in an L2 norm. This is precisely how our Eq.(6) formulates the problem: it learns a guarding function $ε$ such that no adversary $f$ from this expressive function space can linearize the relationship between $f(ε(X))$ and the sensitive attribute $S$.\n\nOur method's improved generalization comes from solving a harder problem than Kernelized R-LACE. We address the \"cascading kernel\" problem, seeking a representation ε(X) that ensures ψ(ε(X)) is not linearly separable from S for any adversarial feature map ψ(·). This is a stricter condition without a closed-form solution, and our method's ability to solve it is why it generalizes.\n\n### **Ablation on Multi-Step Erasure:**\n\nWe performed ablation studies on our method's loss components and two main steps. To test the utility-preserving terms $τ_x$ and $τ_{x^i}$ we set $τ_x=0$ in an experiment on the BiasInBios dataset. The table below shows the impact on task accuracy across different erasure levels:\n\n### **TR9- Ablation on Loss Parameters**\n| Gender Acc. | 55% | 60% | 70% | 80% | 90% |\n| --- | --- | --- | --- | --- | --- |\n| $τ_x=0.05$ | 45.06 | 55.43 | 68.92 | 73.01 | 76.10 |\n| $\\tau_x=0$ | 40.58 | 48.04 | 66.27 | 72.5 | 76.01 |\n\n### **TR10- Ablation on RKHS Refinement**\n| Gender acc. | 80% | 70% | 65% |\n| --- | --- | --- | --- |\n| RKHS-Refinement | 72.92 | 65.17 | 58.46 |\n| Obliviator | 73.12 | 68.92 | 62.80 |\n\nThe utility-preserving term is most critical in the final stages of erasure, where interventions must be more precise to maintain performance. \n\nOur ablation studies show that each of the two steps in our framework is insufficient on its own. \n- Adversarial Training, as shown in Appendix D.1 (Figure 9), results in an unstable optimization that is difficult to tune for complete erasure.\n- RKHS Refinement is less efficient at preserving utility compared to the *Obliviator* framework.\n    \nThese ablations demonstrate a clear synergistic effect. While the RKHS-refinement step alone works by successively removing modes of dependence that become linear in each new space, it lacks a theoretical guarantee of converging to a full erasure in finitely many steps (required 80 steps to reach 70% accuracy). In contrast, obliviator is backed by HSIC (Equation 6), and two steps makes it highly effective in practice (requiring only 4 iterations for the same result).\n\n# Response to Weaknesses:\n\n> ### **W1 ) Ablation Result on Fairness**\n> \nWe hypothesize that enforcing statistical independence between a representation and a sensitive attribute will improve downstream model fairness. Theoretically, as our method drives HSIC → 0, fairness metrics like Demographic Parity **(DP)** should also approach 0. To test this, we evaluated how DP and GAP change at different erasure levels in our controlled DeepMoji experiment.\n\n### **TR11- DP and $Gap_g^{RMS}$ for Controlled Setup**\n\n| DeepMoji Rep. | Race Acc. | 50% | 60% | 70% | 80% |\n| --- | --- | --- | --- | --- | --- |\n| DP($\\downarrow$) | Original | 0.1078 | 0.2266 | 0.3109 | 0.3751 |\n|  | 75% | 0.0667 | 0.1028 | 0.2896 | 0.2293 |\n|  | 65% | 0.028 | 0.033 | 0.0848 | 0.0203 |\n|  | 55% | 0.0237 | 0.06 | 0.0650 | 0.0671 |\n| $Gap_g^{RMS}(\\downarrow)$ | Original | 0.1360 | 0.2412 | 0.3212 | 0.3815 |\n|  | 75% | 0.1026 | 0.1324 | 0.2991 | 0.2350 |\n|  | 65% | 0.0920 | 0.0951 | 0.1251 | 0.0917 |\n|  | 55% | 0.0910 | 0.1050 | 0.0921 | 0.0915 |\n\nThe fairness metrics for supervised erasure on frozen representation across different dataset and language models reported below.\n\n### **TR12- DP and  $Gap_g^{RMS}$ for BiasInBios Frozen**\n\n|  | Method | BERT | DeepSeek |\n| --- | --- | --- | --- |\n| DP($\\downarrow$) | Original | 0.0204 | 0.0205 |\n|  | Obliviator | 0.0147 | 0.012 |\n| $Gap_g^{RMS}$($\\downarrow$) | Original | 0.1573 | 0.1522 |\n|  | Obliviator | 0.0464 | 0.0662 |\n\n### **TR13- DP and  $Gap_g^{RMS}$ for Dial-Sentiment Frozen**\n\n| Method | DP($\\downarrow$) | $Gap_g^{RMS}(\\downarrow)$ |\n| --- | --- | --- |\n| Original | 0.219 | 0.2109 |\n| Obliviator | 0.006 | 0.0419 |\n\n---\n**If our responses addressed your initial comments, please consider raising the score. If you have more questions, we would be happy to answer them.**"}]

Author rebuttal:

Meta-review: This paper addresses the problem of concept erasure, i.e., removing sensitive or unwanted attributes from learned representations, while remaining robust to nonlinear adversaries. The proposed method, Obliviator, is a post-hoc, multi-step framework combining HSIC-based adversarial optimization in RKHS with a refinement step to recondition the feature space for stability and utility preservation. This approach captures nonlinear dependencies in closed form, enforces statistical independence against a broad class of nonlinear adversaries, and quantifies the utility-erasure trade-off, establishing an empirical upper bound on this relationship.

Reviewers noted that the problem is important, the formulation is well-motivated, and the evaluation is thorough, spanning multiple datasets, erasure scenarios (supervised/unsupervised; frozen/fine-tuned), and representations from both older and more recent language models. The method showed consistent gains over baselines, robustness to strong adversaries, and better utility preservation at equivalent levels of erasure. Reviewer questions about theoretical grounding, relation to prior work, fairness implications, ablations, and computational cost were addressed through additional explanations, experiments, and benchmarking. Overall, the work is viewed as making both a conceptual contribution by recasting erasure from a functional RKHS perspective and providing a strong empirical benchmark for nonlinear guardedness.
Decision: Accept (poster)