WARP: Weight Teleportation for Attack-Resilient Unlearning Protocols
Authors: Mohammad mahdi Maheri
Xavier Cadet
Peter Chin
Hamed Haddadi
Venue: ICLR 2026 Poster (2026)
Keywords: Machine unlearning
Approximate unlearning
Neural teleportation
Weight-space symmetries
Privacy attacks
Membership inference
Model inversion
Data reconstruction
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: A plug-and-play weight-teleportation step reduces alignment with forget-set gradients while preserving predictions, cutting MIA/DRA success across unlearning methods without harming retain-set accuracy.

Abstract: Approximate machine unlearning aims to efficiently remove the influence of specific data points from a trained model, offering a practical alternative to full retraining. However, it introduces privacy risks: an adversary with access to both the original and unlearned models can exploit their differences for membership inference or data reconstruction. We show these vulnerabilities arise from two factors: large gradient norms of forgotten samples and the close proximity of the unlearned model to the original. To demonstrate their severity, we design unlearning-specific membership inference and reconstruction attacks, showing that several state-of-the-art methods (such as NGP and SCRUB) remain vulnerable.

To mitigate this leakage, we introduce WARP, a plug-and-play teleportation defense that leverages neural network symmetries to reduce gradient energy of forgotten samples and increase parameter dispersion while preserving accuracy. This reparameterization hides the signal of forgotten data, making it harder for attackers to distinguish forgotten samples from non-members or to recover them through reconstruction. Across six unlearning algorithms, our approach achieves consistent privacy gains, reducing adversarial advantage by up to 64% in black-box settings and 92% in white-box settings, while maintaining accuracy on retained data. These results highlight teleportation as a general tool for improving privacy in approximate unlearning.

Published as a conference paper at ICLR 2026
WARP:
WEIGHT TELEPORTATION FOR ATTACK- RESILIENT UNLEARNING PROTOCOLS
Mohammad M Maheri'
Xavier Cadet
Peter Chin?
Hamed Haddadi
1 Imperiia College London
Dartmouth College
ABSTRACT
Approximate machine unlearning aims efficiently remove the influence of spe- cific data points from a trained model, offering a practical alternative to full retrain- ing. However, it introduces privacy risks: an adversary with access pre- and post-unlearning models can exploit their differences for membership inference or data reconstruction. We show these vulnerabilities arise from two factors: large gra- dient norms of forget-set samples and the close proximity of unlearned parameters the original model. To demonstrate their severity, we propose mnlearniggsspeciie membership inference and reconstruction attacks, showing that several state-of- of- the-art methods (e.g., NGP, SCRUB) remain vulnerable. To mitigate this leakage, we introduce WARP, plug-and-play teleportation defense that leverages neural network symmetries to reduce reduceforget-se gradient energy and increase parameter dispersion while preserving predictions. This reparameterization obfuscates the signal of forgotten data, making it harder for attackers to distinguish forgotten sam- ples from non-members recover them via reconstruction. Across six unlearning algorithms, our approach achieves consistent privacy gains, reducing adversarial advantage (AUC) by up 64% in black-box and 92% in white-box settings, while maintaining accuracy on retained data. These results highlight teleportation as a general tool for reducing attack success in approximate unlearning.
INTRODUCTION
Machine unlearning (MU) aims to enforce the "right to be forgotten" by updating a trained model so that a designated forget-set has no influence [Bourtoule al. (202T): Zhao al. (2024). The ideal outcome matches retraining from scratch on the remaining refain-set, with both the model's parameters and predictions unaffected by the forgotten data, and without degrading generalization. A primary motivation for machine unlearning is ensure privacy compliance for sensitive informa- tion Wang et al. (2025a). Once personal data is used for training, models may memorize specific details Ravikumar et al. (2024a). creating risks of privacy breaches Bourtoule et al. (20TT): Carlini et al. (2022b). Unlearning addresses this by eliminating such traces, preventing exposure. The most direct solution is retraining from scratch without the forget set, but this is computationally prohibitive. Exact Unlearning methods such as SISA Bourtoulc et al. (202I) reduce cost by modifying training to allow provable deletion, but they require proactive deployment and add overhead. To avoid full retraining, Approximate Unlearning methods finetune the original model to forget the target data while preserving utility Kurmanji et al. (2023): Chundawat et al. (2023a); Golatkar et (020):
Thudi et al. (2022), trading computational efficiency against formal guarantees.
At the same time, ML models are vulnerable privacy attacks Rigaki & Garcia 2023 In Mem- bership Inference Attacks (MIA), adversary determines whether given sample was part of the training set (Shokri et al. (2017). Data Reconstruction Attacks (DRA), the adversary seeks to recover raw data (or a close approximation) from model outputs parameters Yin et al. (202 1); Li ctt al. 2022); Jeon et al. (2021) Fang et al. (2023). These attacks have been demonstrated in both black- box (access outputs) and white-box (access weights) settings Nasr et al. (2019).
Ironically, MU itself can leak the very data it aims to erase. Given access to both the original and unlearned models, adversary mount differencing attacks Hu al. (2024) |ertran et al. (2024). which substantially improve reconstruction success. Even models previously resistant to MIAS can become vulnerable once deletion is performed Bertran al. 20244; Chen al. (021). The key

---

Published as a conference paper at ICLR 2026
observation is that the parameter difference between the two models approximates the gradient of the forgotten sample (up to second-order terms), effectively releasing it to the adversary. Gradient inversion techniques, as in federated learning Geiping et al. (2020), can then reconstruct the forgotten data. Thus, approximate unlearning methods, especially gradient-ascent variants Kurmanji et al. (2023), inadvertently compromise privacy instead of ensuring it.
In this work, we aim to strengthen MU against pri- vacy attacks by characterizing two key factors driving leakage. The first, illustrated in Figure D is that a forgotten sample's privacy risk correlates with its gradient norm in the original model. Intuitively, sam- ples with large gradient magnitudes during training or finetuning induce stronger parameter changes when
Larger gradient norm higher privacy risk
Aueragee Pivaacy Rick
removed, making them more detectable via MIA and more exploitable for reconstruction Ye et al. (0033)
Second, as shown in prior work Thudi et al. 20222; ;
110-
Kurmanji et al. (2023), most approximate unlearn- ing methods make minor parameter updates, typi- cally by maximizing the forget-set loss while keep- ing retain-set accuracy stable. This keeps the un- learned model close to the original, so the parameter difference encodes information about the forgotten data. In gradient-ascent-bee -ased methods Kurmanji et al. ;0 Chundawat et al. (2023a), this difference is essentially the forget-set gradient. Recent studies confirm that such updates expose information equivalent single gradient step on the forgotten sample Bertran et al. (2024), which attackers can invert to reconstruct it.
2011 Gradient norm (lag scale)
Figure 1: Privacy risk vs. gradient norms of forget-set samples, measured with U-LiRA.
To mitigate these risks, we propose WARP, alug-and-play defense that integrates into existing unlearning algorithms without training-time statistics. Our method leverages neural network telepor- tation Armenta et al. (2023), exploiting parameter-space symmetries (e.g., rescaling or permutation) that preserve predictions. By applying selective teleportation steps before or during unlearning, we reduce forget-set gradient norms while injecting ceannetry-preserving randomness. This yields unlearned models that retain accuracy yet displaced in parameter space, making it harder for an attacker to disentangle forgetting from teleportation. Consequently. membership inference and reconstruction attacks significantly weakened, shown in Sections 4.2 and 4.4
Our contributions are summarized as follows:
Tailored privacy attacks. We design MIA and DRA for the unlearning setting, where the adversary compares pre- and post-unlearning models. These attacks show that leading methods remain vulnerable, parameter updates still expose information about the forget-set.
Symmetry-based defense. We propose WARP, plug-and-play defense that, building on existing teleportation and symmetry constructions, applies loss-preserving transformations to reduce forget- sel gradient norms and increase parameter dispersion, thereby obscuring the signal exploited in reconstruction and inference, while remaining agnostic to the particular symmetry mechanism used realize these transformations. WARP integrates into gradient-base post-hoc unlearning algorithms without requiring training-time statistics.
Comprehensive evaluation. We evaluate our attacks and defense across three Tiny-ImageNet, and - ResNet-18 and ViT-B/16 models under both black-box and white-box settings. Results across multiple unlearning algorithms show that teleportation consistently reduces privacy leakage while preserving accuracy on the retain set.
Overall, our work reframes unlearning privacy risk through the lens of gradient norm reduction and links to neural network symmetry, an underexplored optimization principle that provides a conceptual foundation for designing unlearning algorithms with stronger resilience to privacy attacks. Related work discussed in more detail Appendix A
The implementation is publicly available.
Our code is available at http:////////gg.oo/////aaaaaaaar
2

---

Published as a conference paper at ICLR 2026
2
THREAT MODEL
We consider a strong adversary performing sample-wise membership inference, distinguishing whether a sample belongs the forget-set Dj or the test set Diea- The attacker has access to both the pre- and most-unlearning models.
Attacker Capabilities. The attacker has full access to both the original ( and unlearned model 0", as well complete knowledge of the unlearning algorithm Aunlearn and its hyperparameters Hunlearn (e.g., optimizer, learning rate, update steps, retain-set size).
We consider two settings: Black-box the attacker queries outputs f(r; White-box the attacker additionally accesses full internals of both models (#"T, ))., including weights.
Attack Objective Given a sample (a, y) from either the forget-set Dior or the held-out test set Dies. the attacker computes a score A' (3r, y) and predicts membership as A(x, y) - I[A' (x, y) > T]. where I[:] is the indicator function and т is a decision threshold. The attacker seeks a high true positive rate (TPR) on forgotten samples while maintaining a low false positive rate (FPR) on test samples. This directly measures privacy risk: if membership can be reliably inferred, incomplete unlearning is exposed and the forgotten samples identified. Unlike prior work Bourtoule al, (202T); Maheri et al. (2025a), our goal is to audit unlearning algorithms from privacy perspective, rather than evaluating indistinguishability between approximate and exact unlearning outcomes.
3 METHODOLOGY
3.1 PRIVACY ATTACKS
To systematically evaluate privacy leakage unlearning, we consider two complementary classes of attacks: membership inference and data reconstraction.
Black-box (U-LiRA). For the black-box setting, we adopt U-LiRA (Hayes et al., 2025 an adaptation of LiRA (Carlini al. (2022a) to unlearning. U-LiRA leverages shadow models trained and unlearned with the same algorithm the target, yielding strong adaptive baseline for auditing privacy. We defer full algorithmic details to Appendix
White-box (Gaussian Gradient--iffference gradient- differrnce framework of Leemann et al. (2023) to the unlearning case by contrasting gradients computed on both the original and unlearned models. This contrast provides a powerful signal of residual membership leakage when both model versions are available to attacker. The detailed proposed formulation and test statistic are presented Appendix C
In the white-box setting, we extend the Gaussian
Reconstruction Attack in Unlearning. We develop a white-box reconstruction attack tailored to approximate unlearning with retain-set updates. Let A6 = 8th gorg be the observed parameter change after one unlearning stage (possibly aggregating multiple optimizer steps). As gradient inversion, we seek an input whose parameter- graddeett aligns with a target vector; here the natural target A0. Our baseline (single-sample) objective is:
i, ú E
arg min min z.y
D(Vol[((x;0 v): 20),
where & is the training loss, f(; 8) the network, and D a distance (e.g.. & or negative cosine).
With approximate unlearning, the update AO mixes retain and forget gradients. For forget example (xy,3)) y5) and a retain minibatch B,,
AO ~ (g
9r
Σ (Zr.))))BB
("T
gr Vol(f(ry; PPB), us),
(2)
with effective step size and ascent weight a 0. Directly targeting A in equation Tis therefore confounded by dr- Even when equation instantiated with state-of-the-art gradient inversion
3

---

Published as a conference paper at ICLR 2026
methods, naively inverting the unfiltered update A remains ineffective, producing low accuracy of the reconstruction (see Section 4 Table
Let Gorg ) and G. (bb;;; 8th from the training distribution. We compute thin SVDs, Gorg Vorg V and G., = 2,, V
) i==1 be gradient snapshots small probe set drawn
and keep the top-k left singular vectors to obtain orthonormal bases (columns) for the dominant gradient subspaces. Define the orthogonal projectors
Morg uarg org
-
Unlearning attenuates the forget component, retain gradients are expected to persist in both models, whereas the forget component prominent fore but suppressed in d". We therefore orthogonalize the update against the unlearned subspace and keep only directions supported by the original model:
g Horg 11 (- -0)
Intuitively, I removes directions consistent with retain gradients that remain after unlearning, while aarg preserves directions active before unlearning where the forget signal resides. If the retain subspace is well captured, then IT gr 0 and org IT, (ogg)=og) Rags, yielding high-SNR estimate of the forget gradient.
We reconstruct the forgotten sample by solving the filtered inversion:
f ús
arg min D(Vo [f (00)) v). 9f)
2.
with optional priors constraints (z, y). In practice, choose k to retain fixed fraction of gradient energy (c.g., 90-95%), which stabilizes the projectors and reliably isolates the forget component via orthogonalization. We empirically validate that orthogonal subspace filtering boosts reconstruction success across models and datasets; see Section and Appendix Table 3
3.2 WARP (TELEPORTATION BASED DEFENSE)
Motivation I: Parameter closeness increases privacy leakage. We formulate post-hoc unlearning
as minimizing a composite objective that balances forgetting on Dr with utility on D,:
min 0
4(1 Di) +
X (
X20
(5)
Forget
Retain
where 8 denotes model parameters; & is any differentiable forgetting surrogate that penalizes high confidence or reduces fidelity on Dr (e.g., loss-inflation, uniform/soft labels, margin expansion); and l, is the standard training/consistencyy loss on D, to preserve performance. The trade-off coefficient À controls how strongly the unlearning step remains anchored to the retain-set: larger À keeps 6" closer to gers, preserving accuracy but reducing the parameter shift introduced by forgetting. A first-order optimizer with mini-batches B, C Dj and B, C D, yields the iterative update
0++1 02 Thc Voli(O. BT) A Vo BB)) l.)....BB....
(6)
which encompasses common post-training approximate unlearning schemes; for instance, "negative- gradient" methods are recovered by taking (x(-) = - Luain()) (-) (i.e., ascent on the standard training loss over Dr), whereas rehearsal/consisteney-bae based approaches instantiate & with supervised loss or
distillation D, Thudi et al. (2022); Kurmanji et al. (2023); Chundawat et al. (2023a).
Because equation 5 explicitly regularizes utility on Dr and is optimized with small steps and early stopping on Df, the resulting unlearned parameters 6* typically remain close to the original gong in parameter space. The displacement A0
8" (org well-approximated (to first order) by weighted combination of gradients the forget-set, mildly contaminated by retain gradients Thudi et al. (2022); Kurmanji et al." (2023); Huang et al. (2024). This proximity creates privacy attack surface: An adversary with access (6 -,KK00 8") can leverage A0 to perform membership inference or gradient-based reconstruction of D, Hu et al. (004)): Bertran et al. (2024), motivating the defenses applied over unlearning algorithms.
4

---

Published as a conference paper at ICLR 2026
Motivation II: Gradient norm and curvature amplify leakage. Recent evidence suggests that the per-sample gradient trajectory is a strong predictor of privacy vulnerability,, Tobaben et al. (2024) show that training examples that accumulate larger gradient norms during optimization more prone to MIA, reflecting the intuition from differential privacy that each update's privacy loss scales with gradient magnitude. Complementing this, Ravikumar et al. (2024b) demonstrate that curvature around training
significantly
via local sharpness of the loss serves as a reliable discriminator between members and non-members, with sharper regions implying higher membership exposure. These findings aligns with theoretical analyses such as Ye et al. (2023), who prove that large per-sample gradients at initialization inflate the KL divergence between neighboring training trajectories, directly increasing the sample's s privacy risk. Motivated by this, we hypothesize that approximate unlearning inherits the same vulnerability: samples with higher gradient norms tend to push parameters towards sharper local extrema during both training and unlearning, thereby overshooting the target update and leaving a stronger privacy footprint. Our experiments (Fig I confirm this intuition, revealing clear correlation between a sample's ' gradient norm in the original model and its susceptibility to membership inference after unlearning.
To simultaneously address (i) the parameter- srrameter-space proximity that enables differencing and (ii) the gradient- norm driver of leakage, we leverage loss-invariant symmetries of deep networks.
Symmetry framework. Let G denote set of symmetry transformations acting on parameters (and, when needed, internal representations) such that the task loss is invariant: C(X, 0) = L(x(X, 0))
for all g E Zhao al. (2022) 2023); Armenta et al. (2023) Siissk et al. (202I). A teleportation step chooses y and updates 0 + g-0, moving within the loss level set. In our defense, we select g to reduce the gradient norm of the forget-set while preserving utility on the retain-set:
g" E arg min gEg
E( [[e.weD, [ol(f( 0). v) B
0 013
}
shrink forget-set gradients
increase parameter dispersion
s.t. (g 0 D.)) s (,)
with trade-off B20 0 and tolerance e The first term reduces squared gradient norms of forget examples (Motivation II); the dispersion term adds symmetry- -rreetrr-rreeeiingg randomness, displacing parameters from Oorg (Motivation I); the constraint preserves retain performance.
WARP operates on an abstract prediction-preserviie family can instantiate the framework. practice, we use two concrete realizations- the relain-null- space projection introduced in the next paragraph, and the change-of ba-bssis teleportation detailed in Appendix [D to illustrate this generality. To complement this algorithmic view, Appendix O develops teleportation-awe aware coormation-theorete bounds gradient-based reconstruction, showing how injecting conmetry-inducee noise via To expands the symmetry orbit and provably increases the expected reconstruction error for attackers observing (QDrg (7))
symmetry map Tø. and any such symmetry
Primary instantiation:
teleportation with retain null-space projection. We first describe one convenient way to instantiate T& using staiinnnullsppaee projections Wu al. (2025). To optimize equation (atiioon/fffiii effficiently on modern architectures without explicit group actrons, we adopt teleportation with input null-space gradient projection Wu al. (2025) and instantiate using the recent projector formulation that keeps updates on the loss-invariant level set by per-layer projections onto the input null space (thus leaving the task loss unchanged up to numerical error). Concretely, define the teleportation loss
(i)(
Σ [( 3 1100 1 (x.W)EB
where B₁ is minibatch from Dj. Let Re be the per-layer representation matrix from retain minibatch (layer-E inputs), with thin SVD Rs = UVV.V We keep the top-k left singular vectors B, U2. to span the retain subspace and define the orthogonal projector onto its complement II. = I BB A teleportation step then applies the layer-wise update
WL+1
We That (0'))
(8)
5

---

Published as a conference paper at ICLR 2026
which (i) reduces the forget-set gradient norms by descending Liel, (ii) preserves the function on the retain-set by restricting motion to the retain-orthggonll subspace. The projection operator in equation 8 corresponds to the input-null-space projector. This is implemented by subtracting the component in the subspace of the core gradient, leaving only the residual for the teleport step.
To align the invariance with utility preservation, compute Be only from retain data. Let (2(D,) denote the matrix formed by stacking the layer-l inputs for a retain minibatch 5, Then:
(e( = U
B = U2,100
I = -
- B
We set & to capture a fixed fraction of retain variance (typically 95%-99%) and apply the resulting This confines each teleport step to the retain-orthogonal subspace, stabilizing predictions on D, while suppressing gradient energy on Dr. Since II removes directions spanned by retain representations, suitable choices of rank k and step size Thel ensure that
projectors in equation 8
(g D,) - (0) D.)] ( S E,
which matches the constraint below equation 7 in practice, prediction drift on D, remains within numerical tolerance (see Appendix PP for hyperparameter sensitivity). To underline that WARP is not tied to retain-nn -uul-peee projections, Appendix instantiates To using the SVD-free change-of-basis symmetries introduce in Armenta al. 20233
Plug-and- play scope. Teleportation is interlcaved with the standard unlearning update equation 6 requiring no training-time per-sample gradients or stored statisties. The update equation X is applied at intervals t E K C O..... T-1 (e.g., every S steps), keeping forget-set gradient norms low while preserving retention performance. The full algorithm appears Appendix
EXPERIMENTS
We now empirically evaluate the proposed teleportation-bart based defense across multiple unlearning algorithms, datasets, and model architectures. Our experiments designed answer the following research questions: (i) How vulnerable are statc-of-tha theth-rr unlearning algorithms to privacy attacks under both black-box and white-box threat models? (ii) To what extent does teleportation reduce membership and reconstruction leakage without sacrificing utility on the retain-set?
Experimental Setup. We conduct experiments on CIFAR-IO, Tiny-ImageNet, and ImageNet- 1K. On CIFAR-10 10 we use ResNet-18 18, while on ImageNet we evaluate ViT-B/16, covering both convolutional and transformer models. All models are trained with SGD and standard augmentation. Following prior work Kurmanji et al. (2023); Chundawat et al. (2023a), forget sets D, are sampled as roughly 1% of training data per class, with retain sets 2, comprising the rest.
Baselines. We benchmark six representative unlearning algorithms NEGGRAD+ Kurmanji et al. (2023). SCRUB Kurmanji et al. (2023), SALUN Fan et al. (2023), PGU et al. (2024). BAD TADTEAK EACHER Chundawat et al. (202a)) and SRF-ON Huang al. (2024) covering paradigms of gradient ascent, regularization, saliency, projection, and distillation. Full details are in Appendix
4.1 OVERVIEW EFFECTIVENESS OF WARP
Figure 2 summarizes privacy and utility across six unlearning methods with and without our plug-in defense. Each radar chart reports black-box membership inference risk (AUC and TPR at low FPR), accuracy on the most-memorized subset, white-box membership inference risk (AUC and TPR at low FPR), and standard test accuracy. The memoroedd subset is selected following our U-LiRA protocol in Sec. 4.2 motivated by prior findings that highly memorized samples carry elevated unlearning risk Naderloui et al. (2025). For visualization, all metrics are min-max normalized across methods. Privacy metrics in which lower is better are inverted by plotting 1 - metric, so that larger polygons correspond to stronger privacy, while higher test accuracy remains preferable.
Three key observations emerge. First, no unlearning algorithm dominates across all axes. For instance, SF performs well under blo auditing but is weaker under white-box auditing and in

---

Published as a conference paper at ICLR 2026
MSP
NGP
SCRUB
SCRUB
PGU
PGU +
AUC IMM)
TFRGL
AUC (MM)
TFA (88)
AUC (MH)
TPR
(WR))
(WAR)
(18)
Trgg1
Test Accuracy
TPg (WA))
Test Accurscy
TPAg1 (MAR)
Test Accuracy
NGP
SCRUB
PGU
Salun
(MN)
(MN).
Test Accuracy
TPR@1 Test Accuracy
Test Accuracy
Salun
SF
BT
Figure Comparison of unlearning vs. teleportation across six unlearning methods.
test accuracy, illustrating the necessity of evaluating under both threat models. Second, algorithms that appear robust under black-box evaluation such as NGP and SF still exhibit substantial leakage under our white-box test, underscoring the importance of auditing with gradient- weight-bait based evidence. Third, adding our symmetry-based teleportation module, instantiated via retain null- space projection, consistently improves privacy across both black-box and white-box metrics while maintaining utility. In some cases, such as BT and SF, teleportation even improves test accuracy. The only noticeable accuracy drop occurs for NGP (about one percentage point), for which we provide a detailed privacy-utility trade-off analysis in Appendix | The runtime overhead of teleportation analyzed separately in Appendix T and Appendix PP] presents ablations showing that WARP's performance does not hinge on fragile choices of teleportation hyperparameters. Overall, these results demonstrate that the proposed defense empirically reduces attack success consistently and effectively across a diverse set of unlearning algorithms and threat models. For completeness, we also compare WARP against the strongest voise-based alternative, namely projected DP- -angevin unlearning Chien al. (2024b), using its formally calibrated update rule; the full comparison is provided in Appendix|M
4.2 U-LIRA (BLACK-BOX)
We evaluate our teleportation defense with U-LiRA Hayes al. (2025), state-of-the-ar black- box unlcarning auditor. Following Deep Unlearn Cadet et al. (202).. we train T' = 64 shadow models with 10 random forget sets each. To model a strong adaptive adversary, shadows use the same unlearning algorithm, teleportation, and hyperparameters as the target, reducing proxy-target miscalibration Cretu et al. (2023). Details U-LiRA appear Appendix B
As emphasized in prior work Carlini et al. (2022a), the most informative regime is low false- positive rates (FPR), where practical attacks must operate. We therefore report AUC well as @PR@0.L, TPR @ and TPR @ which capture attacker success in this stringent regime. In addition, following RULI Naderloui et al. 2025), we stratify the forget-set by memorization (ranked by training confidence) and evaluate U-LiRA the most-memorized slice. These points carry elevated privacy risk, SO we report low-FPR TPR on this subset alongside aggregate metrics.
Table I shows that adding teleportation plug-in in reduces black-box membership leakage across all methods, on both the full forget-ser and the FPR. For example, NGP's TPR @ nearly halves 10.330~0.001 SCRUB's memorized-slice AUC
slice, with the largest gains at low
7

---

Published as a conference paper at ICLR 2026
Table I: Privacy (Black-box) with and without WARP. Reported are risks on all forget samples and the most-memorize 1% (AUC, TPR@0. IPR50.1/1/5%) plus test accuracy. Each row shows baseline, WARP, and relative improvement (%).
All samples (BB)
Most dlost-memorized (top 1%)
Acc.
Method
AUC
TPR
TPR@1
TPR65
AUC
TPR@0.1
TPR@1
TPR
Test
NGP (basc)
0.545
0.012
0.030
0.077
0.649
0.058
0.157
0.277
0.808
WARP
0.516
0.003
0.014
0.055
0.598
0.015
0.082
0.206
0.797
Improvement (%)
64.4
81.8
80.0
81.5
34.2
75.4
51.0
31.3
-5.7
SCRUB (buse)
0.543
0.020
0.047
0.092
0.710
0.086
0.227
0.397
0.815
WARP
0.526
0.015
0,036
0,078
0.610
0.041
0.119
0.213
01881
Improvement (%)
39.5
26.3
29.7
33.3
47.6
52.9
49.8
53.0
-1.1
PGU (busc)
0.636
0.024
0.040
0.098
0.910
0,201
0.511
0.706
0.804
WARP
0.018
0.036
0.104
0.875
0,160
0.431
0.663
0.808
Improvement (%)
3.7
26.1
13.3
-12.5
8.5
20.5
16.0
+2.0
Salun (base)
0.572
0.020
0.062
0.121
0.910
0.129
0.321
0.520
0.802
WARP
0.565
0.019
0.059
0.113
0.826
0.107
0.264
0.487
0,803
Improvement (%)
9.7
5.3
5.8
11.3
20.5
17.2
18.3
7.0
+0.5
SF (base)
0.509
0.004
0.015
0.056
0.518
0.089
0.034
0.079
0.814
WARP
0.506
0.012
0.051
0.501
0.811
Inprrvvemnn (%)
33.3
66.7
60.0
83.3
94.4
94.3
33.3
37.9
-1.6
BT (hase)
0.725
0.000
0.177
0.287
0.902
0.119
0.295
0.582
0.816
WARP
0.661
0.000
0.137
0,219
0.865
0.113
0.275
0.537
0.818
Improvement (%)
28.4
24.0
28.7
9.2
7.0
8.5
+1.1
Method
AUC
TPR@0.1
TPR@1
TPR@S
NGP (base)
0.642
0.004
0.034
0.139
+ WARP
0.614
0.002
0.021
0.097
10
Improvement (%)
17.0
50.0
40.6
34.2
SCRUB (hase)
0.700
0.011
0.102
0.287
Than Pesitive
+ WARP
0.657
0.006
0.061
0.193
Improvement (%)
14.3
54.5
42.5
33.5
PGU (base)
0.659
0.007
0.064
0.215
10
+ WARP
0.533
0.002
0.025
0.085
Improvement (%)
92.9
83.3
Salun (hase)
0.721
0.008
0.069
0.230
+ WWRP Improvement
0,705
0.006
0.062
0,214
9.5
33.3
10.1
10-5
10
Positive
SF (base)
0.670
0.005
0.043
0.161
+ WARP
0.629
0.003
0.030
0.124
Improvement (%)
29.2
50.0
23.2
SCRUB
Avg
Salur+WWRR9
SCRUB+WARP
BT+MWARP
Aug WARP
BT (base)
0.938
0.037
0.346
0.809
SF
NGP
PGU
Optimal
+ WARP
0.907
0.028
0.279
0.684
SF+WARRP
WGP+
PGU+WWARP
Improvement (%)
49.2
25.7
19.4
18.4
Figure White-box privacy with and without WARP. Gaussian gradient-diff test on 640 unlearned models. ROC curves (left) and AUC/TPRs (right); full ROC plots are in Appendix
drops by 0.10 (0.710-10.60)) and SF's AUC falls to near-random (0.501). Low-FPR TPR gains are often large even when aggregate AUC shifts are modest, showing that teleportation suppresses the high-confidence tails attacks exploit. Some methods remain leaky on memorized points, but teleportation frequently drives this slice close to random without hurting accuracy. Its impact is strongest on TPR@0, and TPR @ I, as retain-null-space projection reduces forget gradients and shrinks extreme margins, weakening the signals enabling low-FPR success.
4.3 WHITE-BOX MIA
We evaluate the Gaussian gradient-difference test of Section C under the setup of Section 4 using ResNet-18 on CIFAR-IO and ViT-B/16 on Tiny-ImageNi (tull ViT in Appendix H.. For the null background we draw m=1000 non-members from Diea. estimate (p, Σ) with ridge >=10-1 and restrict the test the top-10% 10% most-variant A (b) coordinates. Figure 3 shows ROC curves with and
8

---

Published as a conference paper at ICLR 2026
Table 2: Effect of teleportation defense on reconstruction (ImageNet-IK, ResNet-18, NGP).
Variant
PSNR (dIB)
LPIPS (VGG) 4
LPIPS (Alex)
SSIM 1
Test MSE
Feat MSE
Ours (normal unlcarning)
10.74 = 0.31
0.56 = 0.013
0.31 0.015
0.12 = 0.008
0.10 0.007
5,39 + 0.50 11.28 +
+ WARP
7.38 0.40
0.68 0.01
0.46 0.02
+: 0.006
0.21 = 0.02
Improvement of Defense (%)
+-15.5
+21.2
+26.11
+-11.6
+52.4
+52.2
Original
NGP
WGR+
Original
NGP
NGP+WARP
Figure 4: Reconstructions under NGP vs. NGP+WARP.
without teleportation (log-log for low-FPR). Across methods, teleported variants shift toward chance (TPR FPR) and flatten between 10 10-5-10-2 2 FPR, suppressing high-confidence tails. The strongest effect appears for BT and PGU, which show the largest AUC drops, while NGP, SF, and SALUN show smaller but consistent shifts. An exception is SCRUB, where teleportation lowers ROC above 10-3 FPR but slightly raises TPR at < 10-3 due to knowledge distillation interacting with symmetry moves that amplify high-leverage directions. Overall, null-space teleportation reduces white-box evidence at low FPR, with a narrow corner case for SCRUB
4.4 RECONSTRUCTION ATTACK RESULTS
We evaluate the white-box reconstruction attack of Section 3.1 on ImageNet-IK with ResNet-18. focusing NGP. We reconstruct single forgotten example and average over 100 uniformly sampled points. For each target we use a retain minibatch of size [8, = Subspace projectors are built per layer from probe gradients: we draw m==100 training samples to form Gorg. G.,. compute thin SVDs, and keep rank l preserving 90% gradient energy. We then apply II, and Iorg layerwise to obtain the filtered target 9/- The attacker knows the label yy and optimizes equation 4 with TV regularizer Geiping al. (2220). The matching loss masked per-layer gradients: for each layer, all coordinates are kept and a weighted dot-product alignment is computed Fang et al. (2023).
Effect of teleportation. Table 2 and Figure 4 compare reconstruction risk under standard NGP unlearning and its teleported variant using change-of-basie reparameterization. Despite negligible cost, this symmetry-base attacks fail recover meaningful features of forgotten data. Teleportation injects symmetry component into AO that is nearly orthogonal to per-sample gradients Armenta et al. (2023), reducing alignment with the true forget gradient 9 and driving cradient-matchine optima. It also undermines our subspace-filtens subspaces so Uorg and U,, overlap little, leaving the residuall Torg (- A0/n) small and noisy. In practice, optimization collapses to the generative prior or class cues, yielding label-consistent but semantically poor reconstructions (Figure 4p. Symmetry moves thus decouple updates from data-dependent directions, removing the geometric handle exploited by white-box reconstruction. This motivates examining how teleportation reshapes the information relationship between parameters and training data (forget-set); stronger symmetry-awee aware adaptive reconstruction attack is evaluated in Appendix(N] N and Appendix proovvdes complementary information- -theorrttt bounds showing how teleportation expands the symmetry orbit and increases expected reconstruction error.
randomization disrupts reconstruction: even strong generative-prion
toward low signal-to-noise attack (Eq. 3. since teleportation reshapes gradient

---

Published as a conference paper at ICLR 2026
5 CONCLUSION AND FUTURE WORK
Approximate unlearning provides scalability but introduces privacy risks. We showed that adversaries with access original and unlearned models can mount strong membership inference and reconstruc- tion attacks. These risks stem from two properties: parameter proximity and large forget-set gradient norms, which amplify leakage.
To counter this, we proposed WARP, a symmetry-based defense that interleaves teleportation with unlearning. By exploiting network symmetries, WARP reduces forget-set gradient energy and displaces parameters in reconstruction leakage while preserving retain performance. Across unlearning algorithms, WARP improves privacy, cutting adversarial advantage by up to 64% in black-box and 92% in white-box settings. We also stress the need for white-box bot auditing: methods seemingly robust in black-box mode (e.g.. SF HFang et al. (2024) still leak when gradients are exposed. Even simple teleportation disrupts reconstruction, reducing quality by 45%.
directions, weakening both membership and
Our findings suggest future directions. First, extending I Langevin-based privacy analyses to practical unlcarning with gradient ascent and symmetry moves is promising. Second, recent work shows approximate unlearning leaves low-rank weight signals, reversible via re-unlearning Fan et al. (025 or removed by quantization Zhang et al. (2024). Exploring teleportation directly on weights may help obscure these signals and mitigate reversals. Finally, the study of neural network symmetries continues to evolve and more efficient estimators and richer invariance families become available, WARP can directly inherit these advances by instantiating its symmetry map with stronger cheaper symmetry mechanisms, which further strengthens its resistance to unlearning attacks.
ACKNOWLEDGMENTS
The rescarch in this paper was supported by the UKRI Open Plus Fellowship (EP/W005271/1 Securing the Next Billion Consumer Devices the Edge) and EU CHIST-ERA GNNs for Network Security and Privacy (GRAPHS4SEC) projects and by the Defense Advanced Research Projects Agency (DARPA), under contract W912CG23C0001..
REFERENCES
Marco Armenta and Pierre-Marc Jodoin. The representation theory of neural networks, Mathematics, 9(24):3216, 2021.
Marco Armenta, Thierry Judge, Nathan Painchaud, Youssef Skandarani, Carl Lemaire, Gabriel Gibeau Sanchez, Philippe Spino, and Pierre-Marc Jodoin. Neural teleportation. Mathematics, (2):480, 2023.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International conference on machine learning, pp. 322-332. PMLR, 2019.
Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Commetry-invarian optimization in deep networks. arXiv preprint 111.v:151..01754. 2015.
Haolei Bai, Siyong Jian, Tuo Liang, Yu Yin, and Huan Wang. Ressvd: Residual compensated svd for large language model compression. arXiv preprint arXiv:2505.20112 12, 2025.
Martin Bertran, Shuai Tang, Michael Kearns, Jamie H Morgenstern, Aaron Roth, and Steven Z Wu. Reconstruction attacks on machine unlearning: Simple models are vulnerable. Advances Neural Information Processing Systems, 37: 17499955--00006 2024.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette- Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lic, and Nicolas Papernot. Machine unlcarning. In 2021 IEEE symposium security and privacy (SP), pp. 141-159. IEEE, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
10

---

Published as a conference paper at ICLR 2026
Xavier F Cadet, Anastasia Borovykh, Mohammad Malekzadeh, Sara Ahmadi-Abhant and Hamed Haddadi. Deep unlearn: Benchmarking machine unlearning, arXiv preprint arXiv:2410.01276. 2024.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pp. 1897-1914. IEEE, 2022a.
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papemot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 35:13263-13274 2022b.
Sungmin Cha, Sungjun Cho, Dasol Hwang, Honglak Lee, Taesup Moon, and Moontac Lee. Learning to unlearn: Instance-wist wise unlearning for pre-trained classifiers. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 11186-11194, 2024.
Min Chen, Zhikun Zhang. Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. Proceedings of the 202/ ACM SIGSAC conference on computer and communications security, pp. 196-911,2000 2021.
Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Certified machine unlearning via noisy stochastic gradient descent. Advances in Neural Information Processing Systems, 37:38852-3385 -3887,, 2024a.
Eli Chien, Haoyu Wang, Ziang Chen, and Pan Li. Langevin unlearning: A new perspective of noisy gradient descent for machine unlearning. Advances in neural information processing systems, 37: 79666-79703, 2024b.
Vikram S Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 7210-7217, 2023a.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on information Forensics and Security, 18:2345-2335, 2023b.
Ana-Maria Cretu, Daniel Jones, Yves-Alexandre de Montjoye, and Shruti Tople. Investigating the effect of misalignment on membership privacy in the white-box setting. arXiv preprint arXiv:2306..c 05093, 2023.
Simon S Du, Wei Hu, and Jason D) Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient- weight saliency in both image classification and generation. arXiv preprint 2rXiv:2310.122 125508 2023.
Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst- case forget sets in machine unlearning. In European Conference on Computer Vision, pp. 278-297. Springer, 2024.
Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. Towards 11m unlearning resilient relearning attacks: A sharpness-aware minimization perspective and beyond. arXiv preprint arXiv:2502.05374, 2025.
Hao Fang, Bin Chen, Xuan Wang, Zhi Wang, and Shu-Tao Xia. Gifd: A generative gradient inversion method with feature domain optimization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4967-4976, 2023.
Stanislav Fort, Gintare Karolina Dziugaite, Maithra Raghu Paul, Surya Ganguli, and David Schwab. Deep learning versus kernel learning: An empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Advances in Neural Information Processing Systems, volume 33, pp. 5850-5861, 2020.
11

---

Published as a conference paper at ICLR 2026
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy to break privacy federated learning? Advances in neural information processing systems, 33:16937-16947, 2020.
Kristian Georgiev, Roy Rinberg. Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. Atribbte-to-dellte: Machine unlearning via datamodel matching. arXiv preprint arXiv:2410.23232, 2024.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 2232-2241. PMLR, 2019.
Grzegorz Gluch and Rüdiger Urbanke. Noether: The more things change, the more stay the same. arXiv preprint arXiv:2104.05500, 2021.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetring outside the box: Scrubbing deep networks of information accessible from input-output observations. In European Conference on Computer Vision, pp. 383-398. Springer, 2020.
Gene H. Golub and Charles F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, MD, edition, 2013.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1 11516-11524, 2021.
Chiyuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. International Conference on Machine Learning, pp. 3832-3842. PMLR, 2020.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv: 1812::1812 2018.
Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53 (2):217-288. 2011.
Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations avoid a false sense of privacy. In 2025 IEEE Conference Secure and Trustworthy Machine Learning (SaTML), pp. 497-519. IEEE, 2025.
Tuan Hoang, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4819-4828, 2024.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.001 12, 2022.
Hongsheng Hu, Shuo Wang, Tian Dong, and Minhui Xue. Learn what you want to unlearn: Unlearning inversion attacks against machine unlearning. 2024 IEEE Symposium on Security and Privacy (SP), pp. pp. 3257-3275. IEEE, 2024.
Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient- gradient-based machine unlearning with remain geometry enhancement. Advances in Neural Information Processing Systems, 37:26377-2641 2024.
Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. Advances in neural information processing systems, 14:29898-29990 2021.
Daniel Kunin, Javier Eogaastuy-Brens Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint 2012:22012.04728. 2020.
12

---

Published as a conference paper at ICLR 2026
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances neural information processing systems, 36:1957-1987, 2023.
Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Gaussian membership inference privacy. Advances in Neural Information Processing Systems, 36:73866-73878, 2023.
Zhuohang Li, Jiaxin Zhang, Luyang Liu, and Jian Liu. Auditing privacy defenses in federated learning via generative gradient leakage. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10132-10142) 2022.
Junxu Liu, Mingsheng Xue, Jian Lou, Xiaoyu Zhang, Li Xiong, and Zhan Qin. Muter: Machine unlearning on adversarially trained models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4892-4902, 2023.
Mohammad M Maheri, Sunil Cotterill, Alex Davidson, and Hamed Haddadi. knowledge approximate personalized unlearning with executable proofs. arXiv preprint arXiv:2512.09953, 2025a.
Zk-apex: Zero-
Mohammad M Maheri, Hamed Haddadi, and Alex Davidson. Telesparse: Practical verification deep neural networks. arXiv preprint arXiv:2504. 19274, 2025b.
Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10422-10431, 2022.
Ioannis Mitliagkas, Constantine Caramanis, Prateek Jain, Shiva Prasad Kasiviswanathan, and Sanjiv Kumar. Memory limited, streaming PCA. In Advances in Neural Information Processing Systems, volume 26, 2013.
Cameron Musco and Christopher Musco. Randomized block krylov methods for stronger and faster approximate singular value decomposition. In Advances in Neural Information Processing Systems, volume 28, 2015.
Nima Naderloui, Shenao Yan, Binghui Wang, Jie Fu, Wendy Hui Wang, Weiran Liu, and Yuan Hong. Rectifying privacy and efficacy measurements in machine unlearning: A new inference attack perspective. arXiv preprint arXiv:2506. 13009, 2025.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pp. 739-753. IEEE, 2019.
Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299, 2022.
Erkki Oja. A simplified neuron model as principal component analyzer. Journal of Mathematical Biology, 153)):667-23,, 1982.
Xinbao Qiao, Meng Zhang, Ming Tang, and Ermin Wei. Hessian-free online certified unlearning. arXiv preprint arXiv:2404.01712. 2024.
Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. Unveiling privacy, memorization, and input curvature links. arXiv preprint 1877: 2024a.
Deepak Ravikumar, Efstathia Soufleri, and Kaushik Roy. Curvature clues: Decoding deep learning privacy with input loss curvature. Advances in Neural Information Processing Systems, 37: 20003-20030, 2024b.
Maria Rigaki and Sebastian Garcia. A survey of privacy attacks in machine learning. ACM Computing Surveys, 56(4):1-34, 2023.
Charbel Sakr and Brucek Khailany. Espace: Dimensionality reduction of activations for model compression. Advances in Neural Information Processing Systems, 57:17489-17517 2024.
13

---

Published as a conference paper at ICLR 2026
Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. {Updates- Leak}: Data set inference and reconstruction attacks in online learning. In 29th USENIX security symposium (USENIX Security 20), pp. 1291-1308, 2020.
Seonguk Seo, Dongwan Kim, and Bohyung Han. Revisiting machine unlearning with dimensional alignment. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3206-3215. IEEE, 2025.
Nazanin Mohammadi Sepahvand, Anvith Thudi, Berivan Isik, Ashmita Bhattacharyya, Nicolas Papernot, Eleni Triantafillou, Daniel M Roy, and Gintare Karolina Dziugaite. Leveraging per- instance privacy for machine unlearning. arXiv preprint arXiv:2505. 18786, 2025.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symme- tries and invariances. In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.
Anvith Thudi, Gabriel Deza, Varun Chandrasckaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (FuroS&P), pp. 303-319. IEEE, 2022.
Marlon Tobaben, Gauri Pradhan, Yuan He, Joonas Jälkö, and Antti Honkela. Understanding practical membership privacy of deep learning. In Privacy Regulation and Protection in Machine Learning, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Joel A. Tropp, Alp Yurtsever, Madeleine Udell, and Volkan Cevher. Practical sketching algorithms for low-rank matrix approximation. SIAM Journal Matrix Analysis and Applications, 38(4): 1454-1485, 2017.
Jana Vatter, Ruben Mayer, and Hans-Arno Jacobsen. The evolution of distributed systems for graph neural networks and their origin in graph processing and deep learning: A survey. ACM Computing Surveys, 56(1):1-37, 2023.
Qiyuan Wang, Ruiling Xu, Shibo He, Randall Berry, and Meng Zhang. Unlearning incentivizes learning under privacy risk. In Proceedings of the ACM on Web Conference 2025, pp. 1456-1467, 2025a.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377 2024.
Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Ilm v2: Optimizing singular value truncation for large language model compression. arXiv preprint wwXiv:2503.123400 2025b.
Manfred K. Warmuth and Dima Kuzmin. Randomized online PCA algorithms with regret bounds. Journal of Machine Learning Research, 910):2287-233 2008.
David Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science, 10(1-2):1-177, 2014.
Zihao Wu, Juncheng Dong, Ahmed Aloui, and Vahid Tarokh. Teleportation with null space gradient projection for optimization acceleration. arXiv preprint arXiv:2502.11362. 2025.
Song Xia, Yi Yu, Wenhan Yang. Meiwen Ding, Zhuo Chen, Ling- Yu Duan, Alex C Kot, and Xudong Jiang. Theoretical insights in model inversion robustness and conditional entropy maximization for collaborative inference systems. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8753-8763, 2025.
14

---

Published as a conference paper at ICLR 2026
Jiayuan Ye, Zhenyu Zhu, Fanghui Liu, Reza Shokri, and Volkan Cevher. Initialization matters: Privacy-utility amalysis of overparameterized neural networks. Advances in Neural Information Processing Systems, 36:5419-540 54469995446644 2023.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16337-16346, 2021.
Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure of Ilm unlearning via quantization. arXiv preprint 1xXi:2440.11655:. 2024.
Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accelerated optimization. Advances in neural information processing systems, 35:16679- 16690, 2022.
Bo Zhao, Robert M Gower, Robin Walters, and Rose Yu. Improving convergence and generalization using parameter symmetries. arXiv preprint arXiv:2305. 2023.
Kairan Zhao, Meghdad Kurmanji, George- Oetavian Bärbulescu, Eleni Triantafillou, and Peter Tri- antafillou. What makes unlearning hard and what to do about it. Advances in Neural Information Processing Systems, 37:12293-12333, 2024.
A RELATED WORK
Approximate Unlearning. The removal of training samples was introduced by Cao & Yang (2013) in the context of the "right to be forgotten." Retraining from scratch guarantees deletion but is infeasible for modern networks Vatter al. 2023). Exact unlearning methods such as SISA [Bouttoul at.l. (202T) and Amnesiac Unlearning Graves et al. (02T) lower cost through partitioning or selective retraining but still require storage and scale poorly Nguyen et al. (2022).
Approximate unlearning directly updates the trained model to erase the forget-ser Kurmanji et al. (2023); Chundawat et al. (2023a); Golatkar et al. (2020); Thudd et al. (022). These methods aim to match the predictive distribution of retraining while preserving retain accuracy, offering a practical forgetting-utin trade-off with large savings in computation and memory. Related methods target structured forget sets such as entire classes [Chundawat et I (2023b): Seo et al. (2025), ) or tackle the harder instance-wist setting, where arbitrary samples must be removed Fan al. (2024); Cha et al.) (2024): Zhao et al.) (2024). Many approaches rely on training-time time side information like per-sample gradients Qiao et (2024); Mehta al.] (2022), or assume specialized regimes with adversarial robustness Liu et al. (2023 or differential -priaacy noise Chien et al. (022b a): [Sepahvand et al. (2025). While effective, these assumptions add resource overhead, limiting post-hoe use. Our focus, therefore, is training-agnosti instance- wise unlearning that takes only pretrained classifier and a designated forget-set, without stored gradients or training modifications Kurmanji et al. (2023);
Thudi et al. (002)).
Privacy Unlearning. The effectiveness of approximate unlearning is accessed by two criteria: (I)
the model should maintain accuracy on non-forgotten data, and (II) its outputs on the forget-set should be indistinguishable from those of model with no access to it Naderloui et al. (2025). In practice, this is evaluated using MIA Shokri et al. (2017): Carlini al. (2022a), which test whether sample was part of training. Effective unlearning removes this membership advantage on the forger-set.
Most prior work evaluates unlearning by comparing outputs of the unlearned model a retrained reference on the forget-ser Cadet et al. (2024); Kurmanji et al.] (2023): (Hayess et al.] (2025): : Georgiev al. (2024); Naderloui
al. (2025). This black-box view ignores parameters, even though in practice-suu such as MU on edge devices- an adversary may access both original and unlearned models. Some studies consider this stronger setting: Chen et al. (202I) showed that output-comparison across models can detect unlearning, while others adapted reconstruction to infer forgotten data from parameter differences/alem et al. (2020); Hu et al. (2024); Bertran et al. (2024). These works, however, limited toy models and simplified updates, leaving privacy risk under realistic conditions unclear. In particular, they do not capture the robustness of recent multi-step approximate methods such as NGP or SCRUB Kurmanji et al. 2003); Chundawat et al. (202a)). where iterative
15

---

Published as a conference paper at ICLR 2026
updates with retain-set supervision weaken inversion of forget-ser gradients. We address this gap with stronger white-box MIAs (SeclC) and DRAs (Sec 3.1) tailored to realistic unlearning.
Neural Network Symmetry. Continuous symmetries in neural networks arise when transformations of the weights leave the output unchanged. Such invariances, a byproduct of overparameterizzttio.
mean that many distinct weight configurations represent the same function Gluch & Urbanke (2021).
They appear in homogeneous activations [Badrinarayanan et al. 2015): : Du et al. (2018); Maheri
et al. (2025b) and in components like softmax and batch normal lizaiio Kunin al. (2020), and
have been linked to both improved optimization and generalization. Neural teleportation leverages
these symmetries by relocating parameters within the loss-invariant level set, yielding equivalent
models that accelerate opiimizatiorAAmenna & Jodoin (20TT) Armenta et al. (023). Building on this
idea et al. (2022) introduced symmetry teleportation, which searches for beneficial relocations
while providing framework for analyzing symmetry-inducce minima. More recently, teleportation
with null-space gradient projection Wu et al. (2025) leverages the input null space: moving along
projected directions leaves the function unchanged, directly aligning with the goal of teleportation.
B U-LIRA ALGORITHM
To evaluate sample-wise privacy leakage, we employ the U-LiRA attack Cadet et al. (2024); Hayes et al. (2025), an adaptation of LiRA Carlini et al. al. (2022a) to the unlearning setting. The attack relies on shadow models to estimate two distributions for a target sample (ar, y): (i) models trained with (z, y) and subsequently unlearned using the same unlearning algorithm, and (ii) models trained from scratch without (x: y). By fitting simple parametric models (e.g., Gaussians) to the outputs of these shadow ensembles, U-LiRA computes the likelihood of the target model's output under each case and classifies membership according to a likelihood ratio test.
Crucially, all shadow models are trained with the same unlearning algorithm and hyperparameters as the audited model. This makes U-LiRA effectively an adaptive attack, since it tailors the proxies to each specific unlearning method. Such alignment minimizes miscalibration between shadow and target models and is known to increase attack success Cretu et al. (2023). Therefore, U-LiRA serves strong black-box baseline for auditing privacy in unlearning. A complete description of the algorithm can demonstrated in Algorithm
Algorithm U-LiRA (used for auditing unlearning)
Require: Target model 8", learning algorithm A, unlearning algorithm U, number of shadows T. sample (x.
Ensure: Prediction: is (x. v) the forger-set?
1: Initialize empty lists 0 + { and Ô + {
2: for / 1 to 7 do
3: Sample dataset D containing (xr, y)
4: Train 8 < A(D)
5: Unlearn 0f + U00 ; {x,vv))
6: Retrain 0r A(D \ (1)))))
7: Record O[1] + f(x; 0z), y 011 + s(x;
8: end for
9: Fit Gaussian (,,0²) to O, and (p, 02) to ò
10: Compute 0" < (2; 0r)) )
11: Compute likelihood ratio: Pmember
N(o':
N(ot:11,0 +N
12: if Pmeember > 0.5 then
13:
return "member of training"
14: else
15:
return "non-member"
16: end if
16

---

Published as a conference paper at ICLR 2026
C WHITE-BOX GAUSSIAN GRADIENT-DIFERENC ATTACK ALGORITHM
Guided by the GLiR framework of Leemann al. 2023). we formulate sample-wise MIA in the unlearning setting as a binary hypothesis test that uses both the pre-unlearning and post-unlcariig models. Let A denote the training algorithm, U the unlearning operator, S the original training set, and F C S the forget subset. For candidate example (Ir, v). we test
Ho (x,y) ~ Drests
(0or,, 8*) = (A(S), U(A(S), F)) with x 4 S, 2 F, H1 (x,, v) Drorg (0ors, (*) = (A(S), U(A(S), F)) with S and E F,
ie., under H1 the point participated in the original training and was subsequently targeted by unlearning, whereas under Ho it never used. With white-box access, form the gradient- difference statistic
A Vo((f(z;0",,)) Vo (f; Rd.
Assuming access to draws from Diest, the adversary builds a background set B = {(bs, 11 D and estimates the null (non-member) distribution of gradient differences via
lest
((b.),
- (A(b)) A
Following Leemann et al. (2023), we adopt a Gaussian model for (zx) under Ho and compute the whitened Mahalanobis statistic
((x) (1) (A(x)) -n))
with small ridge À > 0 for numerical stability. Under Hn, s(x) is approximately x3 disttrribb yielding the log-n-value score
A'(x,v) = -log Faz(s(z)))
and the final decision rule
A(x,y) I[A'(x,)) v) r].
predicting forgotten when the score exceeds threshold Algorithm provides the full details of the proposed attack.
Relation GLiR and unlearning specifies. GLiR aggregates evidence across training steps by comparing per-step sample gradients to Gaussian background of batch gradients; our adaptation replaces the (typically unavailable) per-step trajectory with the two-model contrast A(x). The geometry is unchanged: Evidence corresponds to the squared norm of the whitened difference, |I + 1/))) Unlike standard MIAs that query single model, the test exploits white-box access to gorg and 8 and targets the dllearning-speciie alternative H, (membership in both S and F), providing simple and powerful auditor for residual leakage after unlearning.
17

---

Published as a conference paper at ICLR 2026
Algorithm 2 White-box Gaussian Gradient-Differene Attack for Unlearning Audit
Require: Pre-unlearning model 6 post-unlearning model Ø", candidate sample (æ, (), loss
predictor [(;;0, background sampler (m)) that returns m i.i.d. draws from Diest Require: Hyperparameters:: background size m, repetitions T, ridge À > decision threshold > initialize cumulative evidence
1: S 4-0 0
2: for 1 to T do
3:
B, [b; {.) f (mm)(m)
if labels are unavailable, set U, = arg max f f(b;; 0org)
4: for
do
5:
A f [ol(f(b;; (0:;;"" 0) Bi) - Vol(f(b;: Rd
6:
end for
7:
Ri 4 E A;
8:
EN 4 m-1
- mm
i -A) -Re)T
9:
ΣG, < E, +.a
ridge for numerical stability
10:
A. + Vol(f(x;(") y) - (x; 0u8g), y)
U U+ A - i4
Solve Σ,,> w for w (e.g. Cholesky); 81 u.u
4 + log Fx3 (st))
log tail p-value under Ho
S+ 4e
15: end for
return FORGOTTEN >> т; else TEST
D ALTERNATIVE SYMMETRY: CHANGE-OF-BANE NEURAL TELEPORTATION.
We also support the "neural teleportation" family of symmetry moves from Armenta et al. Ta be scale attached to neuron a. For an edge ++ with weight lat the teleported weight
2023) Let
= Tb
8bb
Pab-
(10)
and if fa is the activation at neuron then the teleported activation is
ga(z) = fa
-
(1I)
which preserves the function for positively homogeneous activations and extends naturally to batch- norm scales Armenta et al. (2023). In subset of experiments, we choose т to further increase parameter dispersion under loss invariance (outputs unchanged), thereby weakening the differencing signal and making reconstruction harder; most results rely on the null space instantiation in equation In the experimental section, it is explicitly indicated when both mechanisms are enabled.
E BASELINES
We evaluate our teleportation-based defense as a plug-and-play module layered on top of several state-of-the-art approximate post-hoc unlearning methods. These baselines representative of the most widely studied approaches in recent literature, requiring no access to training-time auxiliary statistics (e.g., par-sample gradients) and operating directly on pretrained model. Specifically, consider:
1. NegGrad+ (NGP) Kurmanji et al. (2023): An improved variant of GA that incorporates a regularization term on the retain-set. The method balances ascent on the forget-set with descent on the retain-sed set, aiming to preserve model utility while unlearning.
2. SCRUB Kurmanji et al. 22023; knowledge distillation approach that aligns the unlearned model with the original model the retain-set via consistency loss, while simultancously removing the forget-ser's influence. SCRUB represents one of the most competitive baselines in recent evaluations.
18

---

Published as a conference paper at ICLR 2026
3. SalUn al. )003): ): A saliency-basee unlearning method that directs updates to subset of weights deemed saliens for forgetting, identified via gradient-base weight saliency maps. By restricting optimization to these salient weights, SalUn enhances stability and efficiency compared to updating the full parameter set, and aims to reduce the gap to exact retraining.
4. Projected Gradient Unlearning (PGU) Hoang al. 2024 A method that projects the gradient ascent update for the forget-ser onto a subspace orthogonal to retain-set, thereby mitigating catastrophic forgetting, PGU is particularly relevant addresses gradient-level entanglement between forget and retain data.
5. BadTeachhr (BT) Chundawat al. (023aa): recent distillation-baal based unlearning method where the unlearned model (student) is trained against a deliberately corrupted teacher that provides noisy or adversarial labels for the forget-set, encouraging the student to erase their influence while preserving performance the retain-set.
6. SRF-ON (SF) Huang et al. (004)) A geometry- aware unlearning method that decomposes updates into forget ascent, retain descent, and saliency modulation. By embedding updates into the manifold of retain data and approximating Hessian modulation with a fast-slow strategy, SRF-ON improves stability-plasticit trade-offs and enables efficient large-scale unlearning.
These methods span the main paradigms of approximate unlearning gradiinn ascent, retain-aware regularization, distillation, and projection-basee updates-mmaking them representative state-of-the-art baselines.
F ADDITIONAL WHITE-BOX RESULTS ON CIFAR-10
Figure reports the complete ROC curves for the Gaussian gradient diff test, covering the entire FPR range. These correspond to the same 640 unlearned models as in Figure 3 shown here without zoom to provide the full view.
10a
10-1
True Positive Rate
10
3
10
10-5
10-5
10-4
10-3
-0
10-1
10c
False Positive Rate
SCRUB
Avg Normal
Salun + WARP
SCRUB WARP
BT
Avg WARP
SF
NGP
PGU
Optimal
SF4WARP
NGP+WARP
PGU+WARP
Figure 5: Complete ROC curves for the white-box Gaussian gradient-diff test. Averaged over 640 unlearned models, identical to Figure Lower curves (closer to the random diagonal) indicate stronger privacy.
19

---

Published as a conference paper at ICLR 2026
Table 3: Reconstruction on ImageNet-1K (ResNet-18), NGP (no defense). Averages over 100 forgotten samples. Higher is better for PSNR/SSIM; lower is better for LPIPS/MSE.
Method
PSNR (dB)
LPIPS (VGG).
LPIPS (Alex) J.
SSIM
Test MSE
Feat MSE
GILD/Fang et
0.506 5.388 0.497
Ours Suusspcc--iiieee Bilered GFID)
10.74 = (1.1
0.564 = 0.013
0.345 0,015
0.117 = 0,008
0.100 + 0.007
Improvement
+10.5
+19,4
Table 4: White-box membership inference risk with and without teleportation (VTT, Tiny- ImageNet). Results are reported mean = standard deviation across five splits. Improvements are computed as advantage reduction over random guessing.
Method
AUC
TPR @0.01%
TPR @0.1%
TPR @
TPR @ 5%
NGP (base)
0.792 0.019
0.0019
0.001
0.0188 1 0.009
0.178 0.072
0.444 0.035
+ WARP
0.755 + 0.019 12.7
0.0008 + 0.000
0.0079 = 0.003
0.075 + 0.027
0.302 + 0.054
Improvement (%)
61.1
61.2
61.2
36.1
G RECONSTRUCTION ATTACK BASELINES AND COMPARISON.
Table 3 compares three strategies for unlearning: (i) simple differencing, directly inverting Hu et al. 2024]; Bertran et al. (2024); (ii) generative inversion (GIFD et al. (2023) applied to Ao; and (mi) Ours, which adds orthogonal subspace filtering (Eq. equation 3) to a generative backbone.
Results average 100 forgotten samples on ImageNet-1K K with ResNet- 18 under NGP unlearning.
H ADDITIONAL RESULTS: VIT ON TINY-IMAGENET
To extend the white-box analysis of Section 4.3 we evaluate Vision Transformer models trained on Tiny-ImageNet. We adopt ViT-B/16 as the base architecture and follow the same setup described in Section 4 with the forget-set constructed by randomly sampling 1% of the training data and the retain-set consisting of the remainder. All models are trained with SGD and standard augmentations for ViT training. Unlearning is applied with NGP (NGP) and its teleported variant (NGP+WARP))
As shown in Table 4 and Figure 6. WARP substantially reduces attack success across all thresholds, with the largest relative gains at low false-positive rates where practical attacks operate. These results confirm that the symmetry- caasee based defense proposed in WARP extends effectively to transformer models, demonstrating applicability beyond convolutional architectures.
I PRIVACY-UTILITY TRADE-OFF
Improving privacy in unlearning often comes the cost of reduced model accuracy. Since test accuracy on the retain-set is one of the primary criteria for evaluating unlearning algorithms, it is critical examine whether the proposed defense introduces unfavorable trade-offs.
We focus this analysis on NGP, as Figure indicates that teleportation applied to NGP yields the noticeable accuracy drop (roughly one percentage point), whereas for other methods accuracy remains stable or even improves. To probe this trade-off more carefully, we follow the hyperparameter tuning procedure described in Section and select the top 20 trials with the highest validation score. From this pool we examine: (i) the single best-performing trial reported in Figure 2 (ii) the two trials with the highest validation accuracy, and (iii) the two trials with the lowest validation attack AUC.
Figure 7 plots test accuracy against privacy (1 (1-AUC of black-box MIA, higher is better) for NGP and NGP+WARE across the selected hyperparameter trials. The overall trade-off is clear: higher accuracy typically coincides with lower privacy. Yet teleportation consistently shifts the Pareto frontier upward, delivering strictly better privacy at nearly every accuracy level. While NGP saturates around privacy 0.455, teleportation extends this frontier up to 0.484, breaking through the baseline ceiling. At the highest-accurant setting, teleportation still provides a ~ 18% reduction attack advantage over random, demonstrating that even stringent accuracy targets the defense yields nontrivial privacy
20

---

Published as a conference paper at ICLR 2026
1.0
10a
0.-
The Positvv Rate
Tuee Positive Rate
0.6
10
0.0
0.0
0.2
0.4
10a3
10
10
10-2
Positive Rate
MGP NGP MSP--NARP
Optimal
(a) ROC curve (linear scale)
(b) ROC curve (log-log scale, low-FPR region)
Figure White-box ROC for the Gaussian gradient-differeme test on ViT-B/16 (Tiny- ImageNet). Each curve is averaged over five different forget-set splits, with shaded regions showing the standard deviation, Both figures compare NGP and NGP+ WARP; (a) presents the full ROC on linear axis, while (b) zooms into the low-FPR regime log-log scale, which is the operational region for practical attacks.
Privacy-Utility Frontier for NGP vs. NGP+TP
0.49
NGP
NGP+TP
Privacy (1 Attack AUC, t)
0.48
0.17
0.46
0.45
0.44
0./92
0./58
0,804
0,810
Test Accuracy (1)
Figure 7: Privacy-utility trade-off for NGP with and without WARP RP. Each point is a hyperparam- eter trial, with privacy (I-AUC) averaged over 640 shadow models (64 shadows 10 forget sets) under the U-LiRA protocol. Points further to the right (higher accuracy) and upward (higher privacy) indicate better trade-offs.
gains. Across the frontier, improvements remain stable, confirming that teleportation meaningfully reshapes the privacy-utility boundary in favor of the defender.
J RUNTIME ANALYSIS
In this appendix we focus on the retain-nii ciiiin-nul-spacce instantiation of Tg. which is the only variant that requires explicit SVDs; the change-of-basis teleportation in Appendix D SVD-free and without its
21

---

Published as a conference paper at ICLR 2026
200
172
175
150
122
Runtime (s)
125
100
82
75
54
55
50
I
40
33
26
33
30
25
NGP
SCRUB
Salun
SF
BT
Figure 8: Runtime overhead of teleportation. Average runtimes (seconds) of unlearning algorithms with and without the WARP plugin, evaluated on CIFAR-10 with ResNet-18. Each bar reports the mean over five runs, with error bars showing standard deviations.
computational overhead result. Moreover, Section L introduces FastWARP, which replaces full SVD with randomized low-rank approximations and further reduces this overhead.
We benchmark the runtime of our teleportation defense across unlearning algorithms on machine equipped with an NVIDIA GeForce RTX 4090 GPU (24 GB memory) and an Intel 13th Gen Core 19-13900KF CPU (24 cores, 32 threads, base 3.0 GHz, boost up to 5.8 GHz). Each experiment was repeated five times, and Figure reports averages with standard deviations in the caption. All algorithms were run with the hyperparameters used Table D and Figure ensuring runtime reflects the same conditions as our privacy-utility evaluations.
For this particular SVD-based instantiation, teleportation increases runtime by approximately +277 relative to the baseline on average. reflecting the overhead of constructing the retain subspace. The main exception NGP, where teleportation reduces runtime by about -32%, due to more stable updates that in turn lower the required number of unlearning epochs. Since subspace computation can be pre-computed offline and does not need to be repeated after every teleportation step, this overhead can be amortized in practice. While updating the retain subspace less frequently reduce cost, the primary computational overhead from full SVD is addressed directly by an approximate low-rank implementation (Appendix D. which removes the per-step bottleneck entirely.
K TELEPORTATIIN-BASEE UNLEARNING ALGORITHM
In Algorithm 3] Ts denotes abstract symmetry operator: in experiments instantiate it either with retain retam-null-space teleportation or with change-of thange-of-basiis basis teleportation, but any other loss- preserving symmetry could be used in its place.
22

---

Published as a conference paper at ICLR 2026
Algorithm 3 WARP (retain-nulllspace instantiation); telepootaaiion-augmentee gradient-based un- learning.
Require: 8 Dr, Dr. Et, fr. X, bnch Meel. k, S or Tgaad 0², T
1: Og +- 1org
2: for 1. = 0, ....,,,, 0....,T-11 1 do
3:
sample Bt c Dj, B, C C.D
4: Oet f 0, - 1/. [Vel((, Bt) + Vgl,(0 B,))
5:
if (t mod S = 0) V Vol[ Bi) 2 > Tgrad then
6:
for layer do build RB(B,); Be Uei
7:
R = U (SVD)
8:
1 t- I -BBB
9:
end for
(0) = & [ol[](x; v)13 110 gorz
11:
for layer do
Mal I (Vw frel (0c+))) Wi+1 - We W.+
1+
12:
end for
14:
B.t1 t (W
15:
if (0 1B) > (0e B,) + then
16:
0.+1 + 0
b backtrack/safeguard
17:
end if
18: else
19:
Pct1 c 0c+
20:
end if
21:
end for
22:
return f Or
L APPROXIMATE NULL-SPACE TELEPORTATION
Low-rank structure of retain representations. For retain minibatch B, and layer é, let (D.) E [xd denote the matrix whose rows collect the layer-& inputs (dr(z)) Em- pirically, RD. exhibits strong spectral decay: its spectrum is dominated by a small number of singular values, and most of the energy lies in low-dimensional subspace. Such low-rank structure of activations, gradients and Hessians has been observed repeatedly in modern deep networks (Arora cct al.) 2019; Ghorbani et al., 2219 Fort et al., 2020: Gur-Ari et al., 2018). and is often attributed to overparameterisstion and the implicit regularisation of SGD. In WARP, the retain subspace at layer & defined by the top-k left singular vectors of (D,))):
(v( U.
U2 -1= B.B.
Since only the span of these dominant directions matters for teleportation, exact SVD required: any procedure that recovers good approximation to the top-k principal subspace suffices.
Covariance-based PCA and subspace iteration. Instead of computing full thin SVD of R3 (D,), FASTWARP estimates B, via a covariance deooppoccoooiiiionn and small number of subspace- iteration updates, following classical PCA and online PCA methods (Golub & Van Loan 2013 Oja
1982: Warmuth & Kuzmin 2008 Mitliagkas et al. 2013). We first form the covariance
C³ X [R Rd
where X₂ d is a layer-wise input matrix constructed from B, (for convolutional layers use unfolded patches; for batch-norm we aggregate per-channel features). We then compute the eigen-decompossition C₂ Q.A.Q and retain the smallest lt such that the cumulative explained
23

---

Published as a conference paper at ICLR 2026
0.442
100
0.45
0.480
350
Privacy
0.461
0.460
0.45
102
0.792
0.798
0.04
0810
Test Accuracy (tt
Figure 9: Runtime of the WARP plug-in on CIFAR-10 with ResNet-18. Each bar reports the mean over five runs. The top hatched seg- ments correspond to the additional teleportation time; the solid base is the runtime of the under- lying MU algorithm.
Privacy-utility comparison of NG+WARP and NG+FASTWARP. The ap- proximate teleportation method (FASTWARP) matches the privacy- utility frontier of the exact variant, achieving nearly identical privacy and test accuracy.
Figure 10:
variance exceeds threshold T:
k min (
E max(Ar...) 27 2
B₁ Qt,1:ks
Mn max()e,.00)
optionally capped by user-specified kimax. For subsequent teleportation steps, we update B, using few iterations of subspace iteration (Golub Van Loan 2013 Halko al. 2011 Musco & Musco 2015 Tropp et al., 2017 Woodruff] 2014):
Y + CeBr.
Be
+ qr(Y),
which amounts to an Oja-style streaming PCA update (Ojal 1982) with QR re-orthogonallsation. This reduces the cost of updating B& for new minibatch from the ((B,Idd) cost of a fresh thin SVD to (( for the covariance application plus (ddgk² for QR, with k & de. The resulting projector 11:
B, then used exactly as in the original WARP update.
Algorithm FAST WARP basis update at layer e.
Require: de, retain minibatch B., BB (or NONE), T E (0, 1]. kmaxs T Trrack
1: build X₂ E Rdd XN from B,
2: Cj t X,X,; C (C+++
+ CT)
3: if Bprev NONE then
4:
C. Qu
5:
sort Ar in descending order, permute Qr accordingly Σ1 Ar,st Σ. Ne,is
6:
+ +-min// kmax: min [k:
2 T
7:
Be 1:k
8:
else
9:
By -- Burev
10:
for t 1, 1,....
.....,Trraee Tirack do
11:
Y +- C,B.
12:
Be.
qr(Y)
13:
B. <- Bel:, 1:
14:
end for
15:
end if
11 <- la, BBB,
return Be, I
Runtime and privacy-utility impact. Figure 9 reports the runtime for NG and BT with and without teleportation on CIFAR-IO/ResNet-18 The hatched segments correspond to the teleportation
24

---

Published as a conference paper at ICLR 2026
component. Using full SVD yields moderate yet visible overhead (e.g., WARR increases runtime from 30 to 55 s). Replacing full SVD with the covariance-base based PCA and subspace iteration of Algorithm[4 (FASTWARP) (F for NG+FastWARP and BT+FastWARPP corresponding to a 2x-3x reduction in the teleportation time. The teleportation component becomes only small fraction of the overall MU cost.
shrinks this overhead substantially: total runtime drops to 29 and 35
To measure the effect of this approximation on privacy and accuracy, Figure TO compares NG+WARP and NG+FASTWARE along the privacy-utility frontier. The two curves are nearly indistinguishable: privacy (1 AUC) differs by at most 0.3-0.6% across operating points, and test accuracy changes by at most 0.2-0.3 percentage points. We also track retain-set loss during teleportation and observe that the relative drift under FASTWARE remains below 2%, indicating that the approximate projector continues to enforce practical loss invariance. some configurations, the additional numerical noise introduced by the approximation yields slightly higher privacy for the same utility. Overall, these results show that the privacy gains of WARP are robust to approximate PCA, and that FASTWARP preserves the empirical privacy-utility trade-off while significantly reducing computational overhead.
Scalability to LLMs and calibration of the retain subspace. A natural concern is whether null- space teleportation remains practical and stable LLM scale, where layer widths reach de 109-104
and a single minibatch may not span the retain subspace. Empirically, recent compression work shows that truncated SVD and related low-rank factorizations are already applied efficiently to
full LLM weight matrices with comparable or larger dimensions: SVD-LLM Wang et al. (2024) 20256) optimizes singular-value truncation for LLaMA Touvron et al. (2023) and Grown et al. (2020) -caas models while preserving perplexity and throughput, demonstrating that l SVD with < de is tractable in practice on modern hardware. Complementary methods such as ResSVD Bai et al. (2025) leverage the residual matrix left by truncation to correct the approximation, further reducing the effective loss of expressivity at fixed rank. Orthogonal lines of work, e.g., weighted low-rank factorization for LMs, explicitly introduce depandee weights in the covariance (or Gram) operator to bias the recovered subspace toward high-importance tokens or examples, and report competitive compression ratios transformer-base LMs Hsu et al. (2022); Sakr & Khailany (2024). In our setting, we can adopt the same design principles: instead of forming (B,) from an arbitrary minibatch, we maintain small buffer of retain batches with large gradient norm Sakr & Khailany (2024) or Fisher information, and construct the activation matrix E from this "high- influence" pool. This yields a weighted or importance-sampled covariance C₁- = X whose top-k eigenspace more faithfully captures the retain subspace seen over the full retain stream, while keeping the per-teleportation cost ( k). Combined with low-rank SVD implementations that are already optimized for LLM compression, these heuristics make the Fast projector construction compatible with large transformer architectures without breaking the retain loss invariance enforced by WARP We leave the adaptation to large language models for future research. Our contributions target symmetry-basee defenses for generic neural networks and established MU baselines, and do not address LLM-specific challenges in unlearning, which constitute a distinct line of investigation.
M COMPARISON WITH DP-LANGEVIN NOISE DEFENCES
While our goal is to make neural networks more resilient to privacy attacks post hoc, natural question is how WARP compares with defences based on differential privacy (DP). Ds the strongest known framework for providing indistinguishabiliy guarantees between neighbouring datasets, and a small number of recent unlearning methods have attempted translate these guarantees into certified machine unlearning. Among these, noisy-gradient (I (Langevin) approaches provide the closest analogue to our setting: we therefore include them as a comparison point.
Certified unlearning methods such as Guo et al. (2020); Chien et al. (2024b) formalise unlearning as an indistinguishabiliyy requirement between (i) a model obtained by training on the full dataset, and (ii) counterfactual model that has never seen the forget set. These works build on the principle that if the training algorithm is itself DP, then suitable proc-procsssing can yield certified removal of training points. Such guarantees make DP- the strongest known general- -urposse defence with explicit indistinguishabiiiiy guarantees, hence a meaningful baseline to evaluate privacy- trade-offs.
25

---

Published as a conference paper at ICLR 2026
Table 5: NGP+WARP vs. Langevin noise (U-LiRA, black-box). Reported are risks on all for- get samples and on the most-memorized subset (top 5%), plus test accuracy. U-LiRA AUC and TPR@0.14 1% (FPR) are shown for each setting.
All samples (BB)
Most most-memorized (top 5%)
Acc.
Method
AUC
TPR @0.1
AUC
TPR @.1
Test
Langevin (s 1)
0.523
0.004
0.671
0.029
0.682
Langevin (s 4)
0.571
0.006
0.766
0.048
0.718
Langevin (E 8)
0.627
0.020
0,912
0.166
0.771
Langevin (c 16)
0.650
0.027
0.935
0.224
0.798
NGP + WARP
0.516
0.003
0.598
0.015
0.797
What the DP guarantees actually require. The formal guarantees in Guo et al. (2020); Chien et al. 2024b) rely on assumptions that do not hold in the deep, non-convex MU regime we consider:
Convexity and strong dissipativity. Both works require (strongly) convex. fe-reuuaarsed objectives to bound the stationary distribution of the noisy dynamics. Deep convolutional networks trained with cross-entropy fundamentally violate these assumptions.
2. DP-trained initial model required. The certified-un certified-unleaming guarantee requires that the original model be obtained using the same noisy-gradient mechanism (noisy SGD or Langevin) applied throughout training the full dataset. This is explicitly stated as a necessary condition in Chien et al. 22024b). In contrast, our setting begins from a standard ERM PRM--rainee model, which is non-DP and therefore outside the scope of their certification theorem.
As result, the "z" obtained from the RDP accountant in our experiments should be interpreted purely as calibrated noise level, not as a valid DP guarantee. Our use of Langevin noise is therefore strong basse defence, not certified mechanism.
Adapting projected Langevin unlearning to MU. Following Chien et al. (2024b), we implement projected Langevin dynamics on top of the same MU objective used throughout the paper. For per-sample clipped gradient with radius C and loss
(0) = a (c,(0) + A10 - 0pll)) - (1)))
the Da-Lengevin update is
It clip VoLmu (8:). C)2
(12)
0++1 = 0, - 7:9t + 2 X E.,
& ~(0,
(13)
where À is the regularisation parameter entering the RDP privacy analysis. Given a target privacy level follow the exact Rényi-DP accounting of Chien et al. (2024b) to compute the Gaussian noise standard deviation a required by their Langevin update. our implementation, three quantities act tunable hyperparameters: the learning rate 77. the per-sample gradient-clipping radius C, and the regularisation coefficient that appears in the RDP analysis. For any chosen (n, C, A) and target the formulas of Chien et al, (2024b) uniquely determine the corresponding noise scale a.. To ensure fairness across baselines, we run the same number of hyperparameter-searce trials as for the MU baselines, jointly sweeping (7,, C, obtain the set of reported results in Table 5)
Interpretation under non-convexity. Although the privacy accountant yields a numerical s, none of the formal conditions needed for DP-certified unlearning hold for our deep ResNet models. Consequently, we reiterate that the resulting values should not be interpreted as DP guarantees but rather as a systematic way of calibrating the magnitude of injected noise. The comparison therefore isolates the empirical effect of noise injection forgetting, retention, and attack success.
Empirical privacy-utility trade-off. Table Table reveell clear tension between nominal DP guar- antees and empirical membership privacy. As the target privacy budget for Langevin is relaxed
26

---

Published as a conference paper at ICLR 2026
from E = t = 16, test accuracy gradually recovers (from 0.682 up to 0.798), but U-LiRA risk monotonically increases: the salple AUC rises from 0.523 to 0.650, and the AUC on the top-5% most memorised points grows from 0.671 to 0.935, with TPR@0.19 @0.1% FPR increasing from 0.029 simultaneously achieves competitive utility and strictly lower
to 0.224. In contrast, NGP+WAR attack success: on all forget samples it attains the best AUC and TPR @0.1 % (0.516 and 0.003), and on the most-memorised subset it reduces AUC to 0.598 and TPR @0.1 1% to 0.015, outperforming every Langevin configuration by a wide margin. Notably, relative the lowest-noise setting (e 16), NGP+WARP matches accuracy (0.797 vs. 0.798) while cutting the memorised AUC from 0.935 to 0.598 and TPR @0.1 1% from 0.224 to 0.015. For stronger nominal privacy (E or 4), Langevin noise severely degrades accuracy (down to 0.682) yet still leaves substantially higher attack AUC and TPR than WARP. Overall, these results suggest that isotropic DP noise is poorly aligned with the specific memorization patterns exploited by U-LiRA: it injects substantial randomness into all updates, harming utility without reliably protecting the most vulnerable examples, whereas WARP reshapes the parameter space in a targeted way that yields a markedly better empirical privacy-utility frontier.
Taken together, these observations clarify the roles of the two approaches. Langevin noise offers principled mechanism for certified unlearning in the restricted setting of convex, DP-trained models, but its guarantees do not extend the non-convex MU regime nor pretrained models obtained without DP noise. Consequently, applying Langevin updates post hoc to deep networks provides no formal protection and yields an unfavourable privacy-utility trade-off in practice. By contrast, WARP operates directly on arbitrary pretrained models, targets the directions most responsible for memorization, and empirically achieves substantially stronger resistance to membership inference at comparable accuracy. A compelling direction for future work is to investigate whether the geometric structure exploited by WARP can be combined with, or serve as a foundation for, certified unlearning mechanisms that simultaneously handle non-convex objectives and non-DP initialisation-2 capability not supported by current Da-Lengevin frameworks.
N ADAPTIVE RECONSTRUCTION WITH SYMMETRY-AWARE ATTACKER
Teleportation acts by composing the unlcarming update with a symmetry transform that preserves predictions but redistributes parameter mass along ioss-invariant directions (Section 3.2). This raises natural question: can stronger white-box adversary, aware of the teleportation family, invert or compensate for these symmetry moves and recover the residual forget gradient? More concretely, if the attacker parameterise and optimize over the change-of of--e---- (COB) scales used in neural teleportation (Armenta et al. 2023). does this restore reconstruction quality and defeat WARP?
It is worth noting that our privacy evaluation already includes two adaptive- attack families: U-LiRA and GLiR, both of which instantiate adaptive membership-inference attacks by optimising proxy mod- els or surrogate loss landscapes. However, the reconstruction attack considered Section 3.1 3.I-whick directly targets instance-level recovery of the forgotten data-was not adaptive: the attacker opti- mized only over the dummy image while keeping the teleportation parameters fixed. To fully test the robustness of symmetry bbsed teleportation, we now consider a strictly stronger attacker that jointly optimizes both the dummy image and the teleportation parameters themselves.
Concretely, we study whether an attacker who can parameterise and optimize over the change-of thaagg--ff-baste -bassiss (COB) symmetry scales used in neural teleportation (Armenta al.. 2023) can undo the defender's symmetry moves, thereby restoring the clean gradient geometry required for successful reconstruction. This experiment directly probes whether teleportation is merely hiding the forget gradient behind reversible reparameteriistioon, whether it fundamentally reshapes the inverse problem faced by reconstruction attacks.
Attack formulation. In the adaptive setting, we give the attacker full knowledge of the teleportation family and let them shadow the defender's operations. Specifically, starting from the original pretrained weights Oorg. the attacker first applies a change-of-basi symmetry parametrised by COB scales fra > 0), obtaining
g(T) org T- (Oorg),
(14)
27

---

Published as a conference paper at ICLR 2026
0.700
16
0.675
0.650
Test miee
VGG
0.625
Non-adaptive Attack
Non Non-adaptive Attack
Adaptive Attack
Adaptive Attack
COB std
COB std
(A) Feature MSE
(B) LPIPS (VGG)
Figure 11: Adaptive reconstruction under change-of-bast teleportation (NGP, ImageNet- (A) Feature MSE and (B) LPIPS (VGG) as a function of the COB standard deviation deob. Increasing the symmetry variance consistently worsens reconstruction quality across both metrics.
where T, is the COB teleportation map (Appendix D). They then perform single gradient step in parameter space using a dummy image-label pair (z. y):
g(r) (I, 4) = O + Matt Vol(f(<; (.)))
(15)
with attack step size Matt > O. The attacker's goal is to choose (I, 7) so that the shadowed update in equation [5 closely matches the actual unlearned parameters 0n produced by WARP. Formally, we solve
+ ē,, f E arg min [D(B(T) (2, (z, 0.) + Arv + A(7):
(16)
2,5
where D).-)) a parameter-space discrepancy (we use Eg distance over all weights), TV(x) the total-variation regulariser on the image. and (2(T) implements Gaussian prior Ta ~ (1, (..h)) on each COB scale. We optimize equation [6] by alternating gradient steps on and 7, with 7 clipped to a bounded interval around 1 to avoid degenerate scalings.
Experimental setup. For fair comparison, reuse exactly the reconstruction protocol of Section 4.4 (same model, dataset, forgotten examples, optimizer, and image priors), and only extend the attack to optimize over the COB parameters via equation 16] We vary the COB prior variance Goob that defines Q(T), treating cach Ta as scalar random variable centred at with variance aoob. We sweep Ooob 0, 0.1, 0.2, 0.4, 0.8, where acob = 0 recovers the non-adaptive attack with fixed T = 1, and larger values correspond to stronger dispersion along the symmetry orbit induced by WARP. Following the evaluation protocol of Table 2 we quantify reconstruction quality using PSNR, SSIM, LPIPS, and feature MSE, reporting averages over 30 randomly drawn forget examples.
Results and connection to theory. Figure 11 shows how reconstruction quality changes increase the COB prior std doob that controls the spread of admissible symmetry scales. When Jooh 0 the symmetry prior collapses around т. As so the attacker effectively searches over narrow neighbourhood of the defender's true teleportation and can partially recover the forgotten signal: the adaptive attack achieves substantially lower feature MSE and LPIPS than the non- adaptive WARP attack (dashed line). However, the attacker never observes the ground ground-truth truth COB scales used by the defender; as aoob grows and the symmetry orbit broadens, the optimisation over (ar, r) quickly becomes unstable. Both metrics deteriorate almost monotonically with acob: already at moderate variance the gains over the non-adaptive attack largely disappear, and for the largest tested Gcob the adaptive reconstructions are statistically indistinguishable from (or slightly worse than) the non-ade adaptive baseline. Importantly, the COB standard deviation a defender-com -onetree-contolled knob: in this symmetry family and in our main reconstruction experiments in Table we set Ccoh = 0.8, already placing the attacker in high-variance regime where adaptive reconstruction is strongly impaired.
raise Gcob up to 1.0 without changing the realised network function,
2
28

---

Published as a conference paper at ICLR 2026
This trend is consistent with our theoretical analysis in Appendix O which shows that the expected reconstruction error increases with the variance of the COB scales. Larger Coob expands the symmetry orbit of org and u.. the update A admits many symmetry-cquivllee gradients are nearly orthogonal to the true forget gradient gf- The optimisation problem in equation 16 thus becomes highly ill-posed inverse problem over the joint space (xr, 7). where many different configurations of (I, produce similar matches in parameter space. Empirically, the adaptive optimiser drifts toward such low-signal-to-nois solutions that satisfy the symmetry constraints but no longer encode the specific forgotten example, explaining the systematic degradation in reconstruction quality as symmetry variance (or std) increases.
decompositions whose
Takeaway. Even under strong white-box threat model-wheee the attacker knows the teleportation family and jointly adapts both the dummy input and the symmetry to disrupt reconstruction effectively. The injected symmetry components become entangled with the forget-induced update A0, enlarging the attacker's search space and destroying the geometric alignment between parameter differences and the underlying forgotten example. Thus, teleportation does not merely reparameterise the model in way that can be inverted; instead, by injecting symmetry variance into the update, it structurally increases reconstruction error and removes the clean gradient-base baased signal that standard reconstruction attacks depend on. This provides empirical and theoretical evidence that symmetry- -base teleportation fundamentally hardens the inverse problem faced by adaptive adversaries.
teanneterr-elleortttttiiio continues
O TELEPORTATION- AWARE IHTORMATION-THEORETI BOUNDS ON GRADIENT- MAADENNT-BASED RECONSTRUCTION
O.1 OVERVIEW OF THE THEORETICAL ANALYSIS
This appendix develops an information-thoonm lower bound on the minimal reconstruction mean- squared error (MSE) achievable by gradient-based inversion adversary within a shared probabilistic model for gradients. We first adapt standard entropy-MSE. relationships to the case where the attacker observes gradients rather than intermediate features, closely following the spirit of the analysis in Xia et al. Xia et al. (2025). We then introduce a Gaussian-mixture model (GMM) for gradient features and derive parametric lower bound on the conditional entropy H (z g). analogous to the intermediate- incorporate teleportation (change-of-basin noise as private randomness the training dynamics and analyze its impact on the same lower-boure bound pipeline, under an explicit diagonal approximation distribution. Throughout, we
analysis in Xia et al. (2025) but specialized to gradients. Finally, we
and an energy-pre wergy-ppeserving design assumption on the change-of- keep the modelling assumptions identical between the teleported and non-teleported channels, so any improvement we prove directly reflects a genuine tightening of the analytic lower bound on reconstruction error-and hence a provable gain in information- thhooretic privacy within this common generative framework. We emphasize that H is fixed by the dataset distribution, so only relative differences between the channels are meaningful.
O.2 SETUP AND THREAT MODEL
Data and model. Let & E Rd denote the d-dimensionnl input random variable, distributed according to some unknown data distribution on measurable subset X' C TR.. We assume throughout that admits a density w.r.t. Lebesgue measure and has finite second moment. (If one wishes to model discrete or manifold-supported data, the analysis can be recovered by adding an arbitrarily small Gaussian perturbation to a as is standard in differential-entropy arguments; we implicitly assume such smoothing has been applied so that conditional covariances below are positive definite.)
Consider deep network with parameters W E Rmxd and first-layer pre-activations
Wx R.
and subsequent decoder Fa. Let E(-, y) be a loss for label y. and define the gradient with respect to 2:
g= v=((Fa(=).v) "
29

---

Published as a conference paper at ICLR 2026
In the analysis below, the attacker's observation will be gradient-basee signal g (not necessarily equal to Is directly) that is deterministically related to (z, y, W) plus noise. In a white-box setting, for instance, the adversary can observe weight differences across steps, which are affine functions of the underlying gradient features; since mutual information and our entropy-based bounds are invariant under fixed invertible affine reparametrizations, is without loss of generality work with a canonical gradient feature g.
Adversarial objective. An inversion adversary aims reconstruct from the observable y. Given an estimator ö(g), we measure reconstruction quality by the mean-squared error (MSE)
1 (g(#) d [
=
(17)
The minimal MSE S, is the infimum of equation 17 over all measurable estimators 2(.). We interpret "information-theerrtii robustness"
the regime where the attacker is Bayes-optimal under the assumed generative model, ie. has access to the true posterior P( g) induced by that model and implements the Minimum Mean Square Error (MMSE) estimator. Assumption (Basic regularity). We assume:
(1) x: has density on Rd and finite second moment;
(ii) for the observation g. the conditional distribution P(x: g) admits density with finite second moment, and its covariance matrix Cov(æ g) is positive definite almost surely; (iii) all entropies, mutual informations and expectations used below are finite.
These conditions are standard in information- theoorttc MMSE analysis (see, e.g., Xia et al. (025 and ensure that all quantities are well-defined and that the maximum-entropp characterization for Gaussians be applied without degeneracy.
O.3 MINIMAL MSE FROM GRADIENTS AND AN ENTROPY-BASED LOWER BOUND
Bayes-optimal reconstruction from gradients We first recall the standard MMSE characterization. Proposition (Minimal reconstruction MSE from gradients). Let E Rd and observation satisfy Assumptionnn Consider estimators ((g) of x based on and define (g( as in equation 17 Then:
(i) The estimator that minimizes Eg (8) is the conditional mean f ('(g)= = (g)=
g).
(ii) The corresponding minimal MSE
Eg = inf (g d B. [r<(Co(( 9))}
(18)
where Cov(æ g) denotes the conditional covariance of given g and Eg is expectation w.r.t. g.
Proof. For any fixed g. the conditional risk E x
((g) is uniquely minimized by 3 (g)
E[z g] (standard MMSE theory, ef. Xia et al. 2025 )). The minimal conditional risk at is
E[I E
2 lg]
Tr Tr(Cov(r 9)),
since for any random vector X with mean μ and covariance E one has E|| 1ul13 Tr(E). Taking expectation over and dividing by d yields equation 18
Thus, when we refer the "minimal MSE achievable by an attacker" for a given observation model, we mean E, as given in equation 18 corresponding to a Bayes-optimal adversary within that model.
An entropy-based lower bound on the minimal MSE We now relate the minimal MSE & to the conditional entropy H (x g). generalizing standard entropy -MMSE inequalities (cf. Xia et al. (2025)).
Theorem 1 (Eatropy-based lower bound on gradient reconstruction). Under Assumption 1 let (( g) be the conditional differential entropy of given the observation 9. Then the minimal reconstruction MSE Eg in equation 18 satisfies
Eg 2
exp
H(x 9)).
(19)
2Te
d
30

---

Published as a conference paper at ICLR 2026
E(g) is symmetric and positive definite almost surely. For each such 9. the conditional distribution of a given has entropy bounded above by that of Gaussian with the same covariance:
Proof. Fix g and define E(g) == Cov(æ g). Under Assumption 1
H 9 g) S log((2 del(E(g))), (ee(()e))dee(((())))
with equality iff g is Gaussian. This is the usual maximum entropy property of Gaussians. Taking expectation over g gives
(x g) = Eg (( =9]] I Eg 1 log (2Te) det( det(E(g)))
(20)
Let A1 be the eigenvalues of E(g) (all positive). Then
d det (E(g)) = II (1))) Tr(E(g) = Σ A1)
j=1
By the Arithmetic Mean-Geomes Geometric Mean (AM-GM) inequality,
2y) (L) ((aa)))=((((2)) (T((2()))
II j=1
SO
Tr(I(g))
log (et(((gg)) dlog
d
Substituting into equation 20
d _ log 2 Tr(E(g))
(x g) s Bg log ((2T)) det()(g))))
Bg
Since log(-) is concave, Jensen's 's inequality yields
T((())))] < log 2n
Eg (T(EE)))
Eg log 2nee
d
d
Therefore
H(x ( S d
Eg (T((((()))
log
2me
(21)
2
d
By Proposition
B. (Tr(E())) = Eg so equation
21 becomes
d
H(x ( g) s log (2Te
2
Rearranging,
2 2 H(x (z 19),
2
exp(31(x11
1(( 19)).
log(2ne) Eg)
2ee
which yields equation 19
Note that H (ax)aand and hence the absolute scale of these lower bounds fully determined by the underlying dataset distribution and does not depend teleportation. In our comparisons between teleported and non-teleported channels, H/ (xx) cancels and only differences or ratios matter.
0.4 A PARAMETRIC LOWER BOUND H( VIA GAUSSIAN MIXTURES
We introduce specific probabilistic model for the gradient signal and derive tractable parametric lower bound on H(x g). The modelling choices mirror those used for intermediate features in Xia al. (2025), but here applied to gradients.
31

---

Published as a conference paper at ICLR 2026
0.4.1 GRADIENT FEATURE AND OBSERVATION MODEL
Clean gradient feature. Let G: Rd 1m be a deterministic mapping producing a clean gradient feature from input Specifically, let G(x) E R denote feature derived deterministically from (z, y, W) (e.g., the gradient with respect to first-layer pre-activations, or flattened stack of first-layer weight gradients). Thus is deterministic function of once the model and label are fixed.
Assumption (Gaussian Mixture Model (GMM) for u). We assume that the marginal distribution of be well approximated by Gaussian mixture
K
T NiN, E.),
m
0,
>3 >
(22)
i=1
This GMM assumption is standard in information theoretic (2025) and serves as our common surrogate model for gradient features.
theeooaiion-theortiiic analyses of representations Xia al.
Noisy gradient observation. We model the attacker's baseline observation as a noisy version of u:
Go = u u+ E E~
(O, Eg): E L (a,, ux)
(23)
where Eg > a fixed pesfiit covariance matrix. This captures gradient perturbations due to stochastic training, subsampling, other noise sources; Σg is assumed known to the attacker, as in Xia et al. (2025). We use this Gaussian channel as the standard abstraction of gradient perturbations for the subsequent informatiiontheorrtti analysis.
0.4.2 A MUTUAL-INFORMATION IDENTITY FOR DETERMINISTIC FEATURES
We will repeatedly the following simple lemma for deterministic features.
Lemma 1 (Mutual information for deterministic feature maps). Lef G(z) be deterministic function of x, and let g be random variable such that p(g x, u) = p(g u) (i.e., g depends on (z, u) only through u.). Then
[(:; g) = I(u;g).
Where I(x;g) g) denotes the mutual information between x and
Proof. Since u deterministic function of x, we have H(u x) = 0 and ((r,u) = H(x). Moreover, p(g x) p(g u) by the conditional-indeeedeeen assumption,
(g x)) = Hg x =)) Hg a G(z)) H w).
Therefore
[(:; = - H(g) H(g u) = I(u; g).
We will apply this lemma to both the baseline channel gu and the teleported channel g below.
O.4.3 PARAMETRIC GMM-BASED LOWER BOUND ON (xx 9o)
We now adapt the mixture-entropy bound used in Xia et al. 2025) to gradients.
Theorem (Parametric lower bound on H(x go)).. Under Assumption 7 and Assumption and the
channel equation 23 the conditional entropy ((x (o) satisfies
Σ ni (-log log T. log
1
De + 25g1
H( go) 2 H(x)
(24)
2
|g
Proof. Because u = G(x) is deterministic given x, and go depends on (æ, u) only through via equation 23 we have go and the conditions of Lemma Thold. Thus
T(:: go) T(u;go),
32

---

Published as a conference paper at ICLR 2026
and
Hr (00 -
- - - H() - I go)
We bound (u;g0) from above using the GMM model. We have
T [(u; 9o)
(go) (go u).
From equation 23 go 1 ~ N (1, Eg), so
(go u) 2
1 log (2me) (M (Z
Marginally, 9o is the convolution of the GMM with the Gaussian e, hence
90 Tr WW, Eg Eg))
For any mixture density p(z) >, (z) with components pi, the differential entropy satisfies the standard upper bound
H()) S H(T) + Σ (p:),
where H(x) = - Ty log n. the discrete entropy of the mixture weights (this follows by considering the joint entropy of the component index and the sample). Applying this with Gaussian components p; [N,,, E, + Eg) yields
K
T (- log n. + log log (22Te) + E.21)
((g))
2
in Xia et al. 2025
Therefore
K
(u; gu)
<
- log + log (2 + Egl)
log ((rTe) (1))))))
2
1=1
K
log क. + log log((2Te) + 2gl1)
Σ ti (-2 log log((((ee) 120)
T2g
K
Σn
log ne + log (og((27))))))) Er + Egg))
log
K
[Σ + Eigl
log प. + log
F.g.
where the (2mr)m terms cancel. Substituting into H(x gn) = H(x) [u; go) yields equation
Theorem 2 yields parametric lower bound H(: 9o) parametric in the GMM and noise covariances. Via[Therrem I] this in turn induces a lower bound on the minimal reconstruction MSE for attacker observing 9o- Our teleportation analysis will reuse exactly the same ingredients (GMM approximation and mixture-entropy bound) so comparisons are on equal footing.
O.5 TELEPORTATION / CHANGE-OF-BASI NOISE ON GRADIENTS
We now incorporate teleportation (change-of -baass:-of-basi:: CoB) symmetry as a source of private randomness in the gradient dynamics and analyze its impact on the same lower-hound pipeline used for go-
33

---

Published as a conference paper at ICLR 2026
O.5.1 TELEPORTATION AS PRIVATE MULTIPLICATIVE NOISE
Teleportation structure. For each layer f, let ا4 denote the corresponding CoB vector (with all entries nonzero). The teleported gradient layer is obtained by column-scaling with -(f-1) and row-scaling with 1/Tl9, i..
dv = -!e-1|
aw (1/r!1),
av - awld of (1/T!²).
where the left operation multiplies each column of dw( by the corresponding coordinate of 7[f-1], and the right operation multiplies each row by the corresponding coordinate of 1 1/7l(), Consequently, each gradient entry acquires multiplicative factor equal to ratio of CoB coordinates. As such, each gradient entry picks multiplicative factor equal to ratio of CoB entries. Flattening all gradient parameters into single vector, we write the clean gradient feature as u and its teleported version as
û R(T) u,
(25)
where R(T) is diagonal matrix with entries (T) = Ta()/Ta()) corresponding to the appropriate input/output channels (a(j), b(j)) of coordinate j. In practice, these ratios are constrained by the underlying channel-wise 7 structure; our analysis below treats [ry(T)} as effective per-coordinate scalings induced by that structure.
Threat model for teleportation. We adopt the following threat model. Assumption 3 (Teleportation threat model).
(i) The CoB parameters T are sampled from a distribution P. that is independent of (zz, u).
(ii) Teleportation is applied internally in the training update rule, so that the observable gradient feature (e.g., weight differences across step) function of rather than u. Algebraically, this yields observation of the form equation below.
(iii) The adversary has white-box access to the model architecture and weights but does not observe directly. They know the distribution P,.
= - This is the same additive-noise form in equation 23 applied to multiplicatively perturbed feature R(T)u.
(()++=(()))+++ R( ~~N(00 Egy, 8 (x,
(26)
O.5.2 TELEPORTATION-WWAR ENTROPY LOWER BOUND
We now derive the teleportation-awee aaaeee counterpart of Theorem 2] using the same GMM approximation for u. Here the relevant mutual- cnttullinffrmmation identity is again supplied by Lemma 1
Theorem (Teleportation-aww) aware lower bound on (xx g)). Under Assumppion
Assumption
Assumption 3 and the teleported channel equation 26 the conditional entropy H(= g) satisfies
K
(-
E, log ((()) + +Egg P Egg
gx) Z H(x)= Σ T
-1og
log n; + 2
(27)
1=1
where the expectation is taken w.r.t. P.,
Proof. As before, G(x) is deterministic given æ, and depends on (z) only through (2, T) via equation 26 particular, we have the Markov chain
x + u
(g. r) + g.
and 1 x (u, r). Integrating over the independent 7 yields p(g xx u) p(g u). and hence the conditions of Lemma hoold, giving
[(x; g) = [1; g), H(x g) = H (x) 1(x;g) = (() - I(u;
We bound (uu g) from above. By the chain rule and independence of and
T(u; g) I(u (.g..) ( T(u:g ( I(u5 (g) 7(u; r),
34

---

Published as a conference paper at ICLR 2026
since [(u; 7 ()) 0. Here g ()) is conditional mutual information and can be written
[(u;g g 7) = E, (wu g 7 7= O).
For a fixed realization 7 = t, the channel is linear with Gaussian noise:
g -== 1= R(()u + Conditionally mixture component i, (u,, Ex), so
g (i, r = (((()))) R(JE,R()) + Eg),
and g T t is GMM with components indexed by i. For this fixed L,
T(u; g = Hgg = t) Hig u, = t).
Since g (u, ~ (R(()), Eg), we obtain
H(g u, () log (22Te) (
Using the same mixture-entropy bound as before, applied to the GMM g T = L, we have
K
Hig r == t) -
Σ mx (- log n; + ((] +E,)))
Therefore
K Σ
[lug T= T= <
T;
(- logg T. + (2000((27)))1)())))) + +1g))) - log (
i=1
K
(((1) T + Egg
log + log
[g
1=1
Taking expectation yields
K
(- ne + E, log
[R(T),, + 1g0
I(ug r) = EE, t) S
Σ -i i=1
-
|gg
Combining T(u;g) S T(u;g [) with H( g) H(x) T(u;; g) gives equation 27
Theorem is the teleportation analogue of Theorem 2] obtained via the same steps, with E, replaced by R((T),R((T) and an additional expectation over T.
O.6 DIAGONAL APPROXIMATION THE ROLE THE COB DISTRIBUTION
To make the teleportation effect more interpretable at per-coordinate level, we now adopt a diagonal approximation. This is a modelling simplification, similar in spirit to Xia et al. (2025). and all comparisons between teleported and baseline channels will be made within this shared surrogate approximation.
0.6.1 DIAGONAL APPROXIMATION
Assumption (Diagonal covariance approximation). We work in the canonical channel basis in which teleportation defined and posit that, in this basis,
Σ diag(o omm))
Eg = diag(??.... ..... ?
and the teleportation matrix has the form
R(T) diag(ri (7), fm
35

---

Published as a conference paper at ICLR 2026
That is, adopt surrogate model in which gradient covariance, observation noise and CoB factors act coordinatewise in the natural channel basis, rather than attempting to diagonalize arbitrary covariances and then reinterpret teleportation in that rotated frame. This is not claimed to be exact description of real networks, but structured approximation for per-coordinate interpretation.
Under Assumption 4
R(T) + Eg diag(nf + +r())) ....... I +mm (m(())2f_m))
and hence
(((T),,((TT 1,9 Eg
m (1 aij 7
0..
(28)
j=l
Taking logs and expectation equation
27
we obtain
R(()) R( R(T) Eg
E, log
Vis
Egg
where define the per-coordinate quantities
C.j = E, [log(1 + (T)))(T)))}
(29)
Thus Theorem 3 becomes, under Anseer/tsstttionn 4
1 Σ Va.
H() 2 H(x)
- log n; +
(30)
O.6.2 BASELINE (NON-TELEPORTED) DIAGONAL BOUND
For comparison, if no teleportation is applied, we have R(T) = and rr())² = 1. Under the same diagonal surrogate,
Σ; + Eg
II
[1 + ag.)),
j=1
and the entropy lower bound equation
24
reduces to
K m
m Σ j=1
log(1 [1 + (nn)
H(x go) 2 Ho = H(x)
log m +
(31)
We explicitly introduce Hilh to denote the analytic lower bound on H 9n) obtained under the GMM and diagonal surrogate. Similarly, in the teleported diagonal setting equation 30 we define
m +
H gg) >
Hilb tele
H(x)
m log ग,
(32)
j=1
Both H/l and Hilb are computed from exactly the same modelling ingredients and diagonal surrogate.
O.6.3 ENERGY-PRESERRING COB AND IMPROVEMENT OF THE BOUND
To isolate teleportation pure source of randomization (rather than trivial global rescaling of gradient energy), we consider energy-preserving CoB distributions at the level of the per-coordinate effective ratios.
Assumption 5 (Energy-preserving CoB marginals). For each coordinate j. the marginal distribution of ( satisfies EE, (T)]
This condition enforces that, average, teleportation does not inflate shrink per-coordinate gradient energy; it only redistributes stochastically. In practice, the defender controls the sampling of and hence the induced distribution of ratios (r5(7)), subject architectural constraints (shared
36

---

Published as a conference paper at ICLR 2026
channels, etc.). We do not model those constraints explicitly here; treat (ry(T)) effective per-coordinate scalings whose marginals can be chosen to
We do not assume independence of (T( across j. only these marginals.
Define, for each (i,
Ada.j log (1 + 05..)) Vij
log(1 + 04.) E, [log(1 (r)))(r)3)
(33)
Subtracting equation 31 from equation 32 yields an exact relation between the two analytic entropy lower bounds under the diagonal surrogate.
Corollary 1 (Exact relation between diagonal entropy lower bounds). Under Assumption 4 the diagonal entropy lower bounds equation equation 32 satisfy
K Adi.j.
tele
H
(34)
2
with Duhig desined in equation 33 negative, and hence
If. in addition,
Assumption 5 holds, then each A is non-
H
H
(35)
Proof. Equation equation 34 is obtained by direct subtraction of equation from equation 32 and using equation 33 For the sign of Ati.j. fix > 0 and define @a(() = log[1 + at), which is
concave on > Under Assumptios
Vi = (1, s Se.., (E-Ir = Da.,, (1) = log(1 (1 + as.z),
A Mig 2 0 for all (i,j), implying equation 35
Remark (Scope and strength the entropy result). Within the shared modelling assumptions (GMM, diagonal surrogate, energy-preserving CoB), Corollary shows that teleportation never decreases the analytic entropy lower bound:
( 1 Z HO:
H(x 9) 2
H/b H
We stress that we do not claim Hxx 2 H(x go) for the true channels. Rather, we compare the surrogate quantities Hh and Hib arising under the same generative model; under this common lens, teleportation strictly improves the analytic lower bound on uncertainty about x.
0.7 TELEPORTATION AWARE RECONSTRUCTION LOWER BOUND
We now translate the entropy bounds into reconstruction MSE lower bounds using Theorem
O.7.1 BASELINE AND TELEPORTED MSE LOWER BOUNDS
From equation 31 31 eqqution 32 and Theorem 1 we obtain analytic lower bounds on minimal
reconstruction MSE for the baseline and teleported channels:
S.
1
exp
Hill
(36)
2Te
Stele
2ne
exp
Hlb
(37)
By construction and monotonicity the exponential,
E.r
>
S
2
Syele"
(38)
where Egg and Eg are the true minimal MSEs for the baseline and teleported channels, respectively.
Again, (xx) is common to both channels and cancels in all relative statements about / /5n-
37

---

Published as a conference paper at ICLR 2026
O.7.2 IMPROVEMENT FACTOR ON THE ANALYTIC MSE BOUND
Combining the definitions, the ratio between the teleported and baseline MSE lower bounds satisfies
Seele
exp (2 (H
Hbb)
1
K
exp
T
(39)
with Apt. as in equation 33
A bb. hence the exponential factor in equation 39 is at least 1, and the analytic teleportation-awwe aware MSE lower bound is never smaller than the baseline one. In other words, teleportation provably raises the information coommation-thhoretic theoretic floor on reconstruction accuracy captured by this shared surrogate model. We do not assert any ordering between the true minimal MSEs E.0 and Eg.
Under the energy-press energy-preserving assumption (Assumption 5
Remark (Interpretation for privacy). Equation equation 39 provides a quantitative, distribution- CoB),
aware guarantee: under the shared assumptions (GMM, diagonal surrogate, energy-presere teleportation inflates the analytic lower bound on the attacker's 's reconstruction MSE by a factor given by the RHS of equation 39 This factor depends the CoB distribution only through A Aii. which in turn are functions of the per-coordinate lignal-tonnois ratios oij and the marginals of (T) Thus teleportation is not merely a heuristic perturbation: for any attacker whose behaviour dominated by this generative model (in essentially the same sense in Xia et al. (2025)). there is a formal lower bound on how accurately they can reconstruct x.
O.8 LOG-NORMAL COB FAMILY (EFFECTIVE MODEL)
We now specialize the general diagonal amalysis effective log- normnolaal model for the CoB-induced per-coordinate scalings (T)?, to make the dependence on CoB variance explicit the analytic MSE lower bounds.
Log-normal marginal model.
We model each per-coordinate scaling as ry(r)² = exp(Y,),
where
Y, ~ W- s7))
so that
= exp( 130 =1
This ensures the energy pvergy-preeerving condition (rr Assumption 5), while the parameter >3 > 0 controls the strength of deleportation-indueed variability coordinate j. Practically, the defender can aim to implement such marginals by sampling T so that the induced ratios (T)) are approximately log-normal; we do not model the exact mapping from channel-wist 7! to ratio marginals. We emphasize that this is an effective parametric family for r3. chosen for analytical clarity; our rigorous inequalities rely only on [Assumption 5) while log-normality is used to express the dependence
small number of variance parameters.
Under this model, the per-coordinate quantities Vas and AWss admit explicit expressions.
Corollary 2 (Log-normal teleportation and analytic MSE bound improvement). Under the log-normal CoB marginal model above, for each mixture component and coordinate j.
(in()3)
LE 1y3mN((1 (23,)3)))
[log(1 + a..
(40)
and
A log(1 + ai.)) 15 B (3))) [log(1 + a).
(41)
Let En and Segte denote the analytic lower bounds on the minimal reconstruction MSE for the non-teleported and teleported channels, respectively, defined in equation 36 eequaaioion 37 Then
E (s²)
K
exp
Σ
7
log(1 aus) (.5(63))
(42)
&
i=1
where x2 (of,..., ... 22 collects the log- parameters across coordinates.
38

---

Published as a conference paper at ICLR 2026
Proof. The identities equation 40 eDeequutioo 41 are obtained by substituting r c with Y 2 89) into the definition equation 29 of thi, and the definition equation 33 of Adty The ratio equation 42 then follows immediately by plugging At),)(8)) into equation 39 which relates the analytic MSE lower bounds Stele and 2o the Ati.j.
Remark 3 (Local small-variane expansion (heuristic)). To gain intuition about the dependence on teleportation strength, it useful to consider the regime <ž < 1, where the log-normal marginals are close to the degenerate case r3 = 1. This section provides a local Taylor expansion for intuition; is not used in rigorous inequalities, which already follow from [Assumption 5
For ty = r = e with Y, ~ NV( 20 sf) we have
RR(G = 1 Var(4, = F[()] =1 exp((s)
Thus Var(4) 2f + O($)) of - 0. Writing ly + 8g. we have E[öy] and Var(ò,) Var(t;)-
Since log-normal marginals have finite moments of all orders, a second-order Taylor expansion of (1) = log(1 + at) around = yields
a
Φa + 5) log((1 + +a) + 1
8² + (8),
2(1 + a)²
with (5) < 18 for some constant C. depending on a. Taking expectations with E[ō] = 0 and E[0²] = Var(ly) gives
El + 6)] = log(1 + a.) 2(1 + a)² Var(1,) + (B
Applying this with aiy and t; r3. and recalling that ch. E[log(1 + +r,rr)), we obtain the local approximation
(sh loog11 + oa.j) (55(
2(1 + az, Var(ty),
with an error term controlled by |[55,13]. For the log-normal model, Var(ty) = exp(s?) - 1, so we arrive at the heuristic expression
a 2(1 ) (exp(s)) - 1),
A (33)
8f <
Substituting this into equation 42 yields the corresponding small-variance approximation for the logarithm of the analytic MSE bound ratio:
K
1
log
tele
(exp(s)) -1)
E9 1.
2d
(+
ai.)²
1=1
1=1
This approach highlight that, in the small-varte variance regime, the teleportation improvement in the analytic reconstruction MSE lower bound grows approximately linearly in 83 (via exp(s?) - 1), with slope governed by the per-coordinate signal-to-noise ratios Oi,j and the mixture weights
P ABLATION: SENSITIVITY OF TELEPORTATION HYPERPARAMETERS
Teleportation introduces small set of additional hyperparameters that control how strongly we move along symmetry directions. In this section we study the sensitivity of WARP to two core choices: (i) the target retain-variane variance fraction used to choose the per-layer rank & in the SVD projector (Section 3.2). and (ii) the size of the retain minibatch B, used to estimate the retain subspace. Both directly govern the geometry of the retain null space and the amount of stochasticity in the teleportation step, and were explicitly highlighted potential sources of instability.
39

---

Published as a conference paper at ICLR 2026
0.50
0.810
Privacy (1 AUC,, 1)
0.19
Test accuracy (t)
0.48
0.47
0.800
0.46
0.795
0.45
0.95
0.98
0.993
0.999
0.790
Rank fraction
Rank fraction
(A) Privacy (1 AUC) vs. retain-variance
B Test accuracy retain-variante target
target
0.50
0.810
Privacy (1- AUC, 1)
0.49
Test accuracy (t)
0.805
0.48
0.800
0.47
0.795
0.46
0.45
0.790
256
512
1024
2048
4096
256
512
1024
2048
4096
Teleportation batch size [8r]
Teleportation batch size [B,]
(C) Privacy (1 AUC) teleportation batch
(D) Test accuracy teleportation batch size
size B
B,
Figure 12: Sensitivity of teleportation hyperparameters. Plots (A.B) vary the target retain-variance
level used to set the per-layer rank ky; plots (C,D) vary the retain minibatch size [B,] used to estimate the retain subspace. Privacy is measured AUC of U-LiRA (higher is better). Markers highlight the configuration used in our main experiments (95.3% retain variance and [B|| 2048).
Setup. We perform controlled sweep on CIFAR-10 with ResNet-18 and NGP+WARP under the U-LiRA black-box auditor (Section[4.)) For the SVD projector, we vary the target variin-vriancee level from 95% 99.9%, which induces different per-layer ranks kę such that the top singular vectors of (D,,) capture the chosen fraction of retain energy. For the retain minibatch, we vary the teleportation batch size B. E (256, 512, 1024, 2048, 4096) while keeping the forget minibatch and unlearning hyperparameters fixed. For each configuration we run the full unlearning pipeline and record test accuracy as well as privacy measured by (1 AUC) of U-LiRA (higher is better).
Results and discussion. Figure 12 shows that teleportation is remarkably insensitive to both hyperparameters in the regime we consider.
Retain-variance target. Increasing the target retain-variance from 95% to 99.9% changes privacy (1 AUC) by less than 0.015 in absolute terms, while test accuracy varies in narrow band of 0.79-0.80. Privacy slightly improves as we move from 95% to around 99.3%, after which the curve flattens: very high targets effectively make the retain projector full-rank, leaving less room for teleportation to move symmetry directions and yielding diminishing returns. The configuration used in the main experiments (target our chosen rank provides a good privacy-utility compromise.
99.3%) lies near this plateau, indicating that
Retain minibatch size [Br. Varying [B,| over order of magnitude has only minor effect: privacy (1- AUC) shifts by at most ~ 0.01, and test accuracy remains within +0.2% points of 0.796. Even relatively small batches (BB,I 256) already provide sufficiently representative retain subspace for teleportation, and larger batches only yield slight, saturating gain in privacy. This suggests that the random retain minibatch need not tightly approximate the full retain set to obtain stable projector
40

---

Published as a conference paper at ICLR 2026
and effective defense; in practice, modest [B,] balances computational cost with stable subspace estimation.
Overall, these ablations show that WARP's performance does not hinge fragile hyperparameter choices: both privacy and utility are stable across wide ranges the SVD rank and retain minibatch size. Moreover, the small spread in test accuracy (< 0.6% across all settings) empirically confirms that teleportation remains approximately loss- loss-preserving on the retain set, providing an implicit bound on worst-case retain-loss drift in our experiments.
41


Reviews:
[{"review_id": "RTwYuBeNTV", "reviewer": "Reviewer_J66i", "summary": "This paper addresses the privacy risk in approximate machine unlearning (MU), where adversaries can exploit differences between pre- and post-unlearning models to reconstruct forgotten data. To mitigate this, the authors propose WARP, a plug-and-play defense that leverages neural network symmetries to reduce the forget-set's gradient signal and increase parameter distance, thus obfuscating the unlearning process.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tConnecting neural network symmetry with unlearning privacy is a novel conceptual contribution.\n2.\tThe evaluation uses strong, adaptive attacks (U-LiRA, and a custom white-box attack), making the defense's evaluation robust.", "weaknesses": "1.\tSignificant Computational Overhead is the most significant drawback. The core of this method (see Algorithm 3) requires computing the SVD for the retainer activation matrix $R_l$ of each layer in each transport step. SVD is a computationally intensive operation. The analysis in Appendix J (Figure 8) shows that this adds an average of +27% to the runtime overhead.\n2.\tThis method introduces a large number of new and seemingly highly sensitive hyperparameters: transmission step size $\\eta_{tel}$, tradeoff coefficient $\\beta$, rank $k$ preserved by SVD, transmission frequency $S$, etc. This is further evidenced by the discussion of the privacy-utility tradeoff in Appendix I (as shown in Figure 7).\n3.\tIn Table 1, for most-memorized samples, NGP+WARP still has an AUC of 0.598, and BT+WARP has an AUC as high as 0.865, which is far from reaching the \"immune\" level.\n4.\tThe entire method is heuristic. It provides no formal privacy guarantees (e.g., in relation to Differential Privacy). It is unclear if a stronger, adaptive attacker aware of WARP could defeat it.\n5.\tThe reported 92% white-box AUC reduction appears to be a best-case scenario. For other baselines (like NGP or Salun), the improvement is marginal. The paper also admits the method can worsen privacy for SCRUB under certain conditions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7qkuPtPYKB", "reviewer": "Reviewer_NyvK", "summary": "This paper proposes a privacy defense method, WARP, for machine unlearning. Despite being efficient and effective, approximate unlearning methods can leak information about the unlearned instance when an adversary analyzes the difference between models before and after unlearning. To tackle this issue, WARP leverages neural network symmetries to keep updates near a loss-invariant set for the retain data, and to suppress gradients from the forget set. It then applies layer-wise projections to teleport parameters in function-preserving directions. Experiments combining WARP with unlearning methods showing reduced attack success under both black-box and white-box settings.", "questions": "Can WARP be applied to popular NLP tasks with LLMs?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tWARP can be applied to both CNNs and ViTs, showing that it does not rely on any particular neural network architecture.\n2.\tWARP is designed to be a plug-in method and can be adopted with many unlearning methods, as demonstrated in the experiments.\n3.\tThe improvements of WARP are consistent across all the tasks and unlearning methods.", "weaknesses": "1.\tEach layer’s retain subspace is built from a randomly sampled retain minibatch, where this small batch can misrepresent the full retain set when the retain set is highly diverse. This could result in an inaccurate retain subspace and hinder defense effectiveness.\n2.\tThe paper describes teleportation as “leaving the task loss unchanged up to numerical error”, which is in fact an approximation of a loss-invariant transformation on the retain set. Yet there is no analytical worst-case bound on the retain loss drift. Also, the claim “loss-invariant” may not be accurate since it is not strictly unchanged.\n3.\tThe evaluation in this paper centers around with and without WARP but does not compare against any privacy defenses. Since the contribution of this paper is privacy defense in machine unlearning, it would be better to show how WARP outperforms other privacy defense methods, rather than only showing gains when plug WARP into many unlearning methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Ct9oCbnX8", "reviewer": "Reviewer_5bBC", "summary": "A machine unlearning algorithm that provides defense for reconstruction attacks is introduced in the paper. By applying layer-wise scaling and permutation transformations that preserve the model’s function but relocate its parameters in space, WARP ensures that the parameter difference between the pre-unlearning and post-unlearning models no longer aligns with the forgotten data’s gradients. Experiment results shows that across membership inference and gradient reconstruction attacks, the proposed algorithm reduces privacy leakage with negligible accuracy loss and minimal computational cost.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed algorithm uses inherent network symmetries to randomize parameter space without retraining or noise injection. It is easy to integrate into any unlearning pipeline, introduces negligible computational costs, and preserves model accuracy. \n\n2. The authors evaluated their algorithm across multiple datasets, unlearning algorithms, and attack types. The inclusion of detailed ablations on transformation modes and frequencies supports the robustness and generality of the proposed defense.", "weaknesses": "1. The described process involves per-layer SVDs and null-space projections, which can be expensive for larger models.\n\n2. The work does not provide a theoretical explanation of how teleportation changes the information relationship between parameters and training data. A formal analysis would make the contribution more rigorous. \n\n3. Symmetries are by design invertible. If an attacker can recreate or approximate the teleportation transform (which they probably can given the strong threat model described in the paper), is it possible that they can work their way up and subtract out it's influence to recover the residual forget gradient?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pd8BIUTgo6", "reviewer": "Reviewer_puub", "summary": "The paper introduces WARP, a defense against privacy attacks on machine unlearning. The core contribution is a method to mitigate the risk of an attacker discovering what data was forgotten by comparing a model before and after the unlearning process. WARP accomplishes this by using inherent symmetries in neural networks to teleport the model's weights to a functionally identical but distant point in the parameter space after unlearning. This engineered change effectively masks the information-rich updates from the unlearning process, empirically demonstrating a strong defense against both black-box and white-box reconstruction attacks while preserving the model's utility.", "questions": "* How does the SVD's computational overhead scale for industrial-sized models, and is the proposed amortization sufficient for practical application?\n* What is the anticipated model utility cost of using formal Differential Privacy to achieve the same low attack success rate that WARP provides?\n* How does WARP's performance and compatibility extend to non-gradient-based unlearning paradigms like those using influence functions or data partitioning?\n* How does WARP's effectiveness hold up under large-scale unlearning requests where a significant fraction of the training data must be forgotten?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Introduces a new defense mechanism for machine unlearning by using neural network symmetries.\n- Demonstrates strong empirical success in thwarting both black-box and white-box privacy attacks.\n- Effectively secures the unlearning process without causing a significant drop in model accuracy.", "weaknesses": "The authors correctly acknowledge that the SVD step introduces computational overhead but suggest this cost can be amortized. However, for truly large-scale models with very wide layers (e.g., modern LLMs), this cost can be prohibitive even if amortized. Could the authors provide a more concrete analysis of the method's scalability, perhaps commenting on the practical wall-clock time for models with billions of parameters, and elaborate on whether the proposed amortization strategies are sufficient to make WARP truly practical for industrial-scale applications?\n\nThe paper's privacy analysis is based on strong empirical results against specific attacks. An alternative approach to privacy is using a formal framework like Differential Privacy (DP), for instance, by adding calibrated noise to the unlearning gradients. Could the authors elaborate on the trade-offs between their empirical defense and a formal one? Specifically, if one were to apply DP to achieve a similarly low Attack Success Rate, what would be the anticipated cost in model utility compared to the high utility maintained by WARP?\n\nThe paper claims WARP is a \"plug-and-play\" module, yet only validates this with gradient-based unlearning. How does WARP's performance and compatibility extend to other unlearning paradigms, such as those based on influence functions or data partitioning methods (e.g., SISA)?\n\nThe experiments are conducted using relatively small forget-sets (e.g., ~1% of a class). While this is a common scenario, how would the method perform under a more large-scale unlearning request where, for instance, 20% or 50% of the training data must be forgotten? Does the clear distinction between the retain subspace and its null-space still hold when the retain set shrinks significantly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)