Towards Privacy-Guaranteed Label Unlearning in Vertical Federated Learning: Few-Shot Forgetting Without Disclosure
Authors: Hanlin Gu
Hong Xi Tae
Lixin Fan
Chee Seng Chan
Venue: ICLR 2026 Poster (2026)
Keywords: Federated Learning
Machine Unlearning
Privacy-Preserving
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose the first method for label unlearning in Vertical Federated Learning (VFL), addressing privacy risks with limited labeled data using manifold mixup and gradient-based forgetting, followed by recovery optimization.

Abstract: This paper addresses the critical challenge of unlearning in Vertical Federated Learning (VFL), a setting that has received far less attention than its horizontal counterpart. Specifically, we propose the first method tailored to *label unlearning* in VFL, where labels play a dual role as both essential inputs and sensitive information. To this end, we employ a representation-level manifold mixup mechanism to generate synthetic embeddings for both unlearned and retained samples. This is to provide richer signals for the subsequent gradient-based label forgetting and recovery steps. These augmented embeddings are then subjected to gradient-based label forgetting, effectively removing the associated label information from the model. To recover performance on the retained data, we introduce a recovery-phase optimization step that refines the remaining embeddings. This design achieves effective label unlearning while maintaining computational efficiency. We validate our method through extensive experiments on diverse datasets, including MNIST, CIFAR-10, CIFAR-100, ModelNet, Brain Tumor MRI, COVID-19 Radiography, and Yahoo Answers demonstrate strong efficacy and scalability. Overall, this work establishes a new direction for unlearning in VFL, showing that re-imagining mixup as an efficient mechanism can unlock practical and utility-preserving unlearning. The code is publicly available at https://github.com/bryanhx/Towards-Privacy-Guaranteed-Label-Unlearning-in-Vertical-Federated-Learning.

Published as a conference paper at ICLR 2026
TOWARDS PRIVACY-GUARANTERI LABEL UNLEARNING IN VERTICAL FEDERATED LEARNING: FEW-SHOT FORGETTING WITHOUT DISCLOSURE
Hanlin Gu' WeBank AI Lab
Hong Xi Tae*
Lixin Fan WcBank Al Lab
Chee Seng Chan' Universiti Malaya
Universiti Malaya
ABSTRACT
This paper addresses the critical challenge of unlearning in Vertical Federated Learning (VFL), setting that has received far less attention than its horizontal counterpart. Specifically, we propose the first method tailored to label unlearning in VFL., where labels play dual role as both essential inputs and sensitive information. To this end, we employ representation-level manifold mixup mechanism generate synthetic embeddings for both unlearned and retained samples. This is to provide richer signals for the subsequent gradient-based label forgetting and recovery steps. These augmented embeddings are then subjected to gradient-basee label forgetting, effectively removing the associated label information from the model. To recover performance on the retained data, we introduce a recovery-phase optimization step that refines the remaining embeddings. This design achieves effective label unlearming while maintaining computational efficiency. We validate our method through extensive experiments on diverse datasets, including MNIST, CIFAR- 10, CIFAR-100, ModelNet, Brain Tumor MRI, COVID-19 Radiography, and Yahoo Answers demonstrate strong efficacy and scalability. Overall, this work establishes a new direction for unlearning in VFL, showing that re- re-imngining mixup as an efficient mechanism can unlock practical and preeetiirrs unlearning. The code is publicly available at Guarantted- Labeld-Laeb-lllaie Saabel--nlenning--nn Vertical-Fedee Federated- Federated-Learning
www.//branbb//llwwdd---PP. Towards-
INTRODUCTION
Vertical Federated Learning (VFL) (Yang al. 2019) enables multiple organizations collabora- tively utilize their private datasets in privacy-preserving manner, even when they share some sample IDs but differ significantly in terms of features. In VFL, there are typically two types of parties: (i) the passive party. which holds the features, and (ii) the active party. which possesses the labels. VFL has seen the widespread application, especially in sensitive domains like banking and healthcare, where organizations benefit from joint modeling without exposing their raw data (I.i et al. 2020).
fundamental requirement in VFL is the necessity for unlearning, driven by the "right to be forgotten" " as mandated by regulations such as GDPR and CCPAR While unlearning has been explored in vertical settings. Existing studies on vertical federated unlearning (Li et al.) 2024 Wang et al. 2024 Han et al. 2025) primarily address the removal of features when an entire passive client withdraws. In contrast, this paper focuses on label unlearning, which is particularly critical in scenarios where labels encode highly sensitive information. For example, in medical diagnostics, one may wish to erase the label indicating whether patient is H/V-positive, Similarly, in credit risk assessment systems, revoking the label associated with a loan approval decision helps safeguard against faimnsss and regulatory concerns.
Horizontal Federated Learning (HFL), there has been limited attention to its application
it reveals private health status.
'These authors contributed equally to this work; authors are listed alphabetically by first name. I Cooresponding author: Chee Seng Chan chan @ um.edu.my).
http:://eag......///rvvvvv/

---

Published as a conference paper at ICLR 2026
However, the need for synchronous processing is paramount constraint in VFL architectures. In VFL, different parties possess unique feature sets over a common sample ID space, necessitating coordinated training steps in which intermediate results are exchanged and aligned precise points. This rigorous coordination implies that all participating models must wait for the slowest participant to complete its step before proceeding the next round. Consequently, this synchronization requirement significantly compounds the difficulties associated with unlearning efficiency in the context of Vertical Federated Unlearning (VFU).
To overcome this challenge, we introduce a novel few-shot unlearning framework for VFL that removes labels from both active and passive models, relying solely on a small public dataset. To achieve this, we re- re-purpose the manifold mixup (Verma et al., (011)) technique, typically used for data augmentation to generate synthetic embeddings for both unlearned and retained samples. This approach accelerates unlearning efficiency while maintaining good unlearning performance even with limited number of data samples. On these transformed embeddings, the active party performs mudient-based forgetting, while inverse gradients allow the passive party to unlearn the corresponding representations locally without accessing raw labels. final recovery phase restores accuracy on the retained data.
Our proposed unlearning solution offers three key advantages: (i) Ceetermaanne--eeerviii Unlearn- ing: It requires only small subset of labeled public data, significantly minimizing the degradation of the retain classes performance; (ii) Enhanced Unlearning Effectiveness: By leveraging the mani- fold mixup technique, it achieves effective unlearning with minimal data; and (iii) Computational Efficiency: The proposed unlearning process is highly efficient, completing within seconds.
The primary contributions of this work are as follows:
i. To the best knowledge, this is the first work to address label unlearning in VFL.
ii We propose novel few-shot label unlearning method that removes labels from both active and passive models in VFL using only a limited amount of public data. As a result, our approach enhances unlearning effecctivness through a representation-Ievee manifold mixup mechanism. ensuring a more robust and efficient unlearning process (see Section 4
ii. We conduct extensive experiments on diverse benchmark datasets, including MNIST (image), CIFAR- 10 (image), CIFAR-100 (image), ModelNet (image), Brain Tumor MRI (image), COVID- 19 Radiography (image) and Yahoo Answers (text). The results demonstrate that approach rapidly and effectively unlearns target labels, outperforming existing machine unlearning tech- niques in both efficiency and effectiveness.
RELATED WORKS
This section reviews related work in two areas, (i) machine unlcarning (MU) and (ii) federated unlearning (FU). Please see Appendix for the comparison of our method with existing FU studies.
2.1 MACHINE UNLEARNING (MU)
MU refers the process of selectively removing the influences of specific data points from machine learning (ML) model after it has been trained. This process ensures that the model "forgets" the contributions of specific data, though the data had never been part of the training process.
MU was first proposed by (Cao & Yang 2015) to remove specific data from a model without full retraining (Garg et al. 2020|[Cher
2021). It includes exact and approximate unlearning. Exact methods like SISA (Bourtoule et al. (0211) and ARCANE (Yan et al., 2022) partition data, train sub-models per partition, and retrain only the affected sections during unlearning. Approximate unlearning includes methods like fine-tuning (Golatkar et al.] [2200a] Tia et al., 2023). random label
updates (Graves et al. 2021 Chen et al.) 2023), noise injection (Tarun et al. 2024 Huang et al. 2021), gradient ascent (Goel et al. 2022 Choi & Na 2023 Abbasi et al. 2023 Hoang et al. (2024) knowledge distillation (Chundawat et al. 2023b Zhang et al. 2023b; Kurmanji et al. (022). and weight scrubbing (Golatkar et al.) 2020a b 202 Guo al. 2020 Foster al. 2024a). While these rely on data during unlearning, Chundawat et al. (Chundawat et al.) (022a)))) propose zero-shot unlearning, and Yoon et al. (Yoon al. 2022) introduce a few-shot approach using model inversion.
2

---

Published as a conference paper at ICLR 2026
2.2 FEDERATED UNLEARNING (FU)
In FU, most of the existing works are focusing in the horizontal environment (Wu et al. 2022 Gu
al. 2024a Zhao al.
2023 Romandini al. 20244] Liu et al. 2024b Zhang al. 2023a Su &
Li 2023; | Ye et al., 2024 . Gao et al. 2024; Cao et al. [023b] | Yuan et al., [023] Alam ct al. 2024]
al.
2023 Halimi al.
[0222] Xia et
Liu et al. 2023 Wang al. [202]] Dhasade al. 2023
2022 Zhao et al., 2024, Wang et al. 2022, Gu et al.
2024bb] Liu et al. 2021 Cao et al. 2023a). The
research on horizontal federated unlearning (HFU) mainly focuses on label (aag et al. 2022 Zhao
al. 2024). client (Liu et al., 202 Yuan et al., 2023 Zhang et al. 2023a Gao et al. 2024; Ye et al.
2024; Su Li 2023 Cao et al. 2023a; Wu et al, (022) and sample unlearning (Liu et al. 2022
Dhasade et al. 2023). While HFU has been widely studied, vertical federated unlearning (VFU)
remains underexplored. Existing works mainly focus on passive party unlearning, such as logistic regression (Deng et al.
2023), gradient boosting (Li et al. 2024). and deep learning models with fast
retraining (Wang et al. 2024) or backdoor defence (Han et al. 2025). However, little attention has
been given setive-party-mnitiinn unlearning with all parties actively engaged in VFI..
BACKGROUND
This section introduces the general setup of the label unlearning process.
3.1
GENERAL SETUP
VFL training. Vertical Federated Learning (VFL) allows parties with different user attributes to collaboratively train model without sharing raw data (Yang et al.) 2019; Liu et al. 2024a). The active party holds the labels and the active model, while passive parties provide features and passive models, enabling performance gains while preserving privacy. We assume that a VFL setting consists of one active party Po and passive parties {P1.... .K] who collaboratively train a VFL model θ = (01,- (k,ww to optimize:
E e(F= o (Go, (2),)), Go,(r2)), , (xK,)))
min w.01 OK
We denote the empirical training objective in Equation I Tas
n
(w..8..... 0k) Σ ((F o (Go, Gox (xx,))
where passive party Ps owns features Fe = (IR.1, Ik.n) and the passive model Ga. while the active party owns the labels y (v1.- Un } and active model F.. Before training, both parties use Private Set Intersection (PSI) to align data records by the same ID. During training, each passive party i computes a forward embedding H3 from its features and sends it to the active party. The active party concatenates embeddings { { into H, generates outputs, computes L, and updates its model using ow their models. Please refer to Appendix A.2 for the table of notations.
ac 8L It then sends DH, to passive parties, which compute 20
ac Bc all, to update aH.
59
Label unlearning in VFL. When the active party requests unlearning for subset of samples with indices I" S [1,..., n. we define the unlearned dataset as D= = [(II....... IK,,, viliez For each passive party Pk. the corresponding feature subset x (II.i)
The objective of label unlearning is to efficiently remove the influence of D" from the trained VFL system without requiring full retraining. Formally, given an unlcarning mechanism acting on the trained model θ = (#I...., .....,,se (k, the updated parameters are
(O",...,, O0 wn) D")
Following standard machine unlearning principles (Bourtoule et al. 2021) ). an effective unlearning method should satisfy: i) Cestormanne-Preserviie Unlearning, removing the impact of D" without degrading performance on retained data; ii) Enhanced Unlearning Effectiveness, requiring minimal auxiliary data; and iii) Computational Efficiency, avoiding full model retraining.
3

---

Published as a conference paper at ICLR 2026
Paseve Party
Fixx
Stzp Marifoli Misup
Votiall Chuckan-lead Amsed Lalsd Ualcansns:
Step Label Unkasning
Passive Party
Go.
Cruckir
Auoont
Skop
Ranazed Accansy Beaoway
Step Remained Accuracy Boorvrr
Skep
fly,
Active
Party
Forward
4... Rackward
Figure 1: Overview of our proposed few-shot unlearning framework in VFL setting.
4 FEW-SHOT LABEL UNLEARNING
This section outlines our proposed few-shot label unlearning method, as shown in Figure ] and Algorithm I The approach involves three steps: (1) augmenting the forward embedding with manifold mixup to address limited labeled data (Section 4.1h: (2) using gradient ascent on the augmented embedding guide label removal in both passive and active models (Section 4.2 and (3) improving model accuracy on the retained labels (Section 4.3
4.1 VERTICAL MANIFOLD MIXUP
With fewer samples, the steps involved in the unlearning algorithm, such as forward embeddings execution and gradient updates, can be executed much faster. However, this small labeled unlearning set, denoted as DP. [(= P. Jk= " Pp... is often insufficient for effective unlearning (see Figure 3. Consequently, we cast this setting as a few-shot unlearning problem, where only a small available to remove the corresponding label information from the model. Please refer to Appendix A for an explanation why a small number of samples (up to 40 per label, as shown in Tables and 10 is sufficient to achieve performance comparable to using the full dataset.
number of labeled unlearning samples
Drawing inspiration from the principles of few-shot learning, repurpose manifold mixup Verma et al. 2019) by interpolating hidden embeddings rather than directly mixing features. We propose manifold mixup framework for VFL. by optimizing the following loss function:
mp.. Σ 4.j=1
w. Mg 12? x(FL (Miva (xf:)) Go, (xP.)) ........... Mixa Mixa (r Goge (okk))) "ixx(o?"."
where
Mix (a, A (1-
(2)
The mixed coefficient ranges from 1. The advantage of the manifold mixup approach lies in its ability to flatten the state distributions (Verma al.) 2019). Once each passive party A generates its local embeddings HP.u = Go, (x these embeddings are transmitted to the active party. The
active party then constructs set of synthesized embeddings H by performing manifold mixup among embeddings originating from the same passive party, ensuring consistency in the À value while avoiding any coordination among passive parties. Similarly, for the small data with the remaining label D we also implement the manifold mixup to obtain the HJ and corresponding label y for the remaining accuracy recovery step.
4

---

Published as a conference paper at ICLR 2026
4.2 VERTICAL GRADIENT-BASED LABEL UNLEARNING
Once the augmented embeddings (Hi,..., Hg) are generated, a straightforward yet effective strategy perform gradient ascent on both the active and passive models using these augmented embeddings. Specifically, the active party concatenates all embeddings (HT) R= into a single tensor
f [M....., Hg.. and optimizes it according to the following formulation:
max (FF( (FF 111)), 3")
where 17th represents the mixture of the representative unlearned labels and is the learning rate.
Unlearning for active model Fu. On one hand, the active model undergoes unlearning for the active model Fw via gradient ascent as follows:
Algorithm Our proposed unlearning
1: Input: Bottom models parameters 0g of K passive parties, top model parameters w. unlearn data DR.,, remain data Dr:r, learning rate 7, unlearn epoch N, batch size b.
w == w + v (FF
(4)
Unlearning for passive model Gg Subsequently, the active party computes the gradients at in accor- dance with Equation to the corresponding passive party k. Finally, the passive party & updates the passive model Ga, using the following expression:
aHE B and transmits these gradients
2: Output: Unlearned bottom models pa- rameters 0% unlearned top model param- eters wt
for in N do
4: for k = 1 to K do
(5) Theorem 1. Suppose that both the trained passive model 0 and the active model achieve training loss bounded by a small value e. Then, when unlearn- ing single label, the following holds:
8 = 02 + T .- ((F.)
Each passive party h generates lo- cal embeddings and send it to Active party.
6:
end for
& Manifold Mixup
Active party generate Hg and Hīg ac- cording to Equation 2
V ft" .T) Bo H² ,²""""""" O. Butt,ge Vol(o; A", 5")" Volle H v²) > (6)
9:
b Gradient-based Label Unlearning
10:
Active party updates according to Equation 4
where (H², TTt) denotes the manifold mixup embed- dings and labels of the public data Deau associ- ated with the unlearned label, (H", y²)) denotes the embeddings and labels of the complete unlearned dataset D",, and is the main task loss of VFL
11:
Active party transfers
88 to all pas- 0Hl
sive parties.
12:
for l: to K do
13:
Each Passive party i update O₂ as Equation 5
Theorem D lindicates that the gradient update direction for label unlearning, when applied to the augmented embeddings of the public unlearned data, is positively aligned with the update direction derived from the em- beddings of the entire unlearned dataset. This result suggests that gradient-based label unlearning using only public data is effective and approximates the behavior of unlearning with access to all unlearned data. See proof in Appendix A.I
14:
end for
15:
> Remained Accuracy Recovery
16: Active party updates according to
Equation
17: Active party transfers
ar to all pas-
sive parties.
18:
for k= = to K do
19: Each Passive party i update 0g as
Equation 7
20: end for
4.3 REMAINED ACCURACY RECOVERY
21: end for
22: Return unlearned passive model On A
The preceding step focuses solely on unlearning the target labels and does not explicitly account for the model's 'ss accuracy on the retained data. As result, the predictive performance on the remaining dataset may degrade. To address this issue, we introduce a Remained Accuracy Recovery step to enhance model accuracy the retained samples. Specifically, using the small dataset with retained labels, denoted as DP-", we jointly optimize the passive and active models with respect to the main task loss, formulated follows:
0% and unlearned active model "
V.F (H)))
(7)
0 - - v AF (H),T) Vo, H
5

---

Published as a conference paper at ICLR 2026
Arcrraay
Model
Dazzoens
Morrics
Buaclae
F7
Unsic
17'
59.29
99:35 a 0.03
95,99 0.05
12.10 0.46
90.16 1192
B4.9 1.13
59.72 0.08
90:50 0.19
WUIS OJI
MNIS
6"
99.39
0.00
0.00
0.00
0.00 0.00
0.00 0.00
ASR
90.61
1.00
2,22
0.07
0.00 6.00
7.95
0.47 0000
0.00
=
CIFARIO
0.00
11.00 0.10
3.25
ASR
25.98
1545 2,33
12.51
1,62
0144
34,50
25:74
1/00
12'
11.43
71.05
1.0.12
06.36 0.71
61.04 K.6
00.05 0.01
59.53. 0.14
55.30
0.81
67.25 0.91
09.96 0.12
CIFARICO
$3.00
0.00 0.00
12.25
2.25
0.00
0.00 2: 0.00
0 0.90
3.50 a: 0.50
0.00
0.00
0.00 0.00
ASR
18.40
25.53
H: 3.56
29.30 2.70
28 04 6.10
7.60 1,30
73.70 1.70
6,09 1: 0.00
6,81 0.01
5,42 1,81
94.26
93.00 0.11
05.64 1:53
26.10 1..00.00000 0,00
73.91 1. 1.83
15.51 1. 0.06
34.07 1. 0.27
81.59 0.96
67.69 0.21
ModetNec
100000
0.00 0.00
0.00 0,00
0,000 0.00
0.00 =: 0.00
0 0.00
0.00 00000
4.79 1.00
2.00 0.00
ASR
$8.40
0.65
23.48 0.77
1,11 0.16
d: 1.25
21,16
0.77
95.81 #:0334
74.25
85.93
0.17
XIRI
46.37
0.63
24.29
52.35
92.87
93.85 4:011
73.84 0.73
40196 6.M0
72.50 0.37
06.13
H1.11 0:15
92.35 0.77
COVID 19 Radingrapy
73.53
0.00
11001.000 0.00
0.00 0.00
0.90 0.00
0.60 6:00
10.83 1.48
6.11 1.92
0.00 0.00
0:00 6.00
ASR
$8.97
46.67 2:74
20.51 2.25
49.27 +: 0.59
13.06 3.93
37.52 5.32
18.12 2,63
36.17 0.82
39.21 0.11
2""
$0.50
90.27 0.119
93.69 0.08
15.93 4.52
84.67 a: 0.22
74.74 0.72
82.69 0.1
87.19
0.39
80.97 0.18
CIAARIU
91.10
1100
1100)
4.25 1.06
11:00 11:00
0.00 0.00
0.00 0.00
2:85 +: 0105
0.00
0.00
0.13- 0.04
Vasts
ASR
$1.86
33.10
1: 1.56
21.44 2,66
42:35 1. 8:23
2.36 0.36
21.75 2.41
34.53 a: 0.65
29.19 0,81
31.33 0.36
2''
65.48
65.32 0:33
59.92 0.56
35.42 +: 1.95
55:83 = 0.13
55.78 0.59
52.21 = 0.00
64.11 0.73
64.24
6*
77.00
0.0 0.00
3.50 4
0,00
0.00 8:00
5.00 0.00
1.19 0.05
ASR
67.30
4
34.50 4.30
40.70
3.10 0.15
0.70
18.30 ::
39.11
0,67
Table I: Accuracy of DD,, y² and ASR for each unlearning method across ResNet18 and Vgg16 models in single-label unlearning.
5 EXPERIMENTAL RESULTS
This section presents the empirical analysis of the proposed method in terms of utility, unlearning effectiveness, time efficiency and some ablation studies.
5.1 EXPERIMENT SETUP
Datasets & Models. We conduct experiments on seven datasets: MNIST (Lecun et al. 1998), CIFARIO, CIFARIOR 100 (Krizhevsky et al., 2009), ModelNet (Wu et al. 2015). Brain Tumor MRI (Wang et al. 2024), COVID-19 Radiography (Rahman (022) and Yahoo Answers dataset (Fu et al. 2022a). We adopt ResNet18 on the datasets MNIST, CIFARTO, CIFARIOR ModelNet, Brain Tumor MRI and COVID-19 Radiography. We adopt Mix Text (Chen et al.) 2020) on the Yahoo Answers dataset. Additionally, we extend our experiments with Vggl6 on the dataset CIFARIO and CIFAR 100. Experiments are repeated over five random trials, and results are reported as mean and standard deviation. We conduct experiments on single NVIDIA A100 GPU.
VFL Setting & Unlearning Scenarios. We simulate a VFL setting with one active model owner and 1-8 passive model owners. single-label unlearning. one label removed from all datasets; in two-label unlearning, two labels are removed from CIFAR 10/100; and in multi-label umlearning, four labels are removed from CIFAR 100. Appendix A summarizze model names, VFL configurations, datasets, unleamed labels and hyper-parameters used for unlearning.
Evaluations Metrics. We evaluate the utility of unlearning by measuring the accuracy of remaining data, Dr, before and after unlearning. The higher accuracy on Dr indicates stronger utility. A stronger utility indicates that more information about Dr is being preserved.
To assess unlearning effectiveness, we use basic Membership Inference Attack (MIA) from (Shokri ctt al., 2017) to measure the Attack Success Rate (ASR) and the prediction accuracy on the unlearned label 1 before and after unlearning. MIA determines whether a data point was part of the model's s training. Lower accuracy on 3" suggests more effective unlearning. A 0% ASR may indicate the Streisand effect (Golatkar al. 2020a). where the model consistently mispredicts all y" samples as a single incorrect label. Ideally, the ASR should be slightly lower than that of a retrained model, signalling successful unlearning without revealing extra information.

---

Published as a conference paper at ICLR 2026
Time efficiency is measured by each method's runtime, where the shorter the better. An effective unlearning method should: a) preserve D' accuracy; b) reduce y" accuracy to near 0%; c) achieve an ASR slightly below that of retrained model; and d) run quickly.
Baselines.
We compare our method with the following baselines: Retrain, Fine- Tuning (Golatkar
et al.
2020a
Jia et al. 2023), Fisher Forgetting (Golatkar et al. 2020a), Amnesiac Unlearning
Graves et al.
2021).
UNSIR (Tarun et al.) 2024), Boundary Unlearning (Chen et al. (023) and SSD
(oottr
2024b).
The implementation details of each baseline Appendix A.8
5.2
EXPERIMENTAL RESULTS
5.2.1
UTILITY GUARANTEE
To assess the utility of our proposed method, evaluate accuracy on D" before and after unlearning An effective unlearning method should retain as much information as possible from DD".
(Table D
From Table we observe: i) Fine-tuning preserves DDr well but has low unlearning effectiveness (see Section 5.2.2): ii) Fisher forgetting severely degrades D" accuracy; iii) Amnesiac's ' random mislabeling shifts decision boundaries, hurting 'D", especially label-rich datasets like CIFAR 100 and ModelNet; iv) UNSIR's repair step fails to fully retain D", causing some degradation; v) Boundary unlcarning yields inconsistent results across datasets and models; vi) While SSD preserves Dr reasonably well, it underperforms slightly compared to approach; vii) In contrast, our method consistently achieves strong unlearning and retention performance.
5.2.2 UNLEARNING EFFECTIVENESS
To assess unlearning effectiveness, we run MIA to determine whether the unlearned model leaks information about y" by measuring the accuracy of y before and after unlearning.
From Tab. 1. we observe: i) Fine-tuning performs poorly on CIFARIO/ 10/100; ii) Fisher forgetting, Am- nesiac, and UNSIR effectively reduce y" accuracy 0.00%; iii) Boundary unlearning is inconsistent across datasets and models, with mixed results; iv) SSD shows strong unlearning effectiveness across all settings; v) In contrast, our method consistently achieves effective unlearning across all settings.
From Table I we further observe: i) Fine-tuning yields consistent ASR; ii) Fisher forgetting often shows high ASR; iii) Amnesiac consistently has low ASR; iv) UNSIR shows high ASR in most cases; v) Boundary unlearning has relatively stable ASR; vi) SSD achieves strong ASR performance across most datasets, with the exception of ModelNet and Brain MRI; and vii) our method consistently achieves strong ASR across all settings.
5.2.3 TIME EFFICIENCY
FT
Fisher
Fig. 2 shows single-label unlearning runtime on CIFAR10
600
Amnesiac Unsir
with ResNet18 across varying numbers of passive parties:
Runtime
BU
i) Methods using the full dataset or Dr (e.g., FT, Amnesiac, Fisher Forgetting) have high execution time; ii) Methods using only D.. like Boundary Unlcarning, faster; iii)
SSD
Ours
200
Our solution has the lowest runtime (16x 1200x lower).
Also, the manifold mixup step is executed the embed- dings of the same passive party (see Figure and Algo- rithm [ As result, the unlearning runtime increases lin- early with the number of passive parties. The unlearning runtimes of different methods compared for varying numbers of passive parties in Figure 2 demonstrating that our method remains the most efficient compared the alternatives.
Number
Figure 2: The runtime(s) of each unlearn- ing method in seconds.
5.2.4 OTHER MODALITY
Beyond image datasets, our method also maintains strong performance on text datasets. This demonstrates its robustness across modalities, which is particularly relevant since VFL is widely adopted in e-commerce scenarios where text is the primary modality.
7

---

Published as a conference paper at ICLR 2026
Table[[shows the effectiveness of our method in preserving the accuracy of the retained data (D") while substantially reducing the accuracy of the unlearned data (y"). ). For example, the accuracy of the unlearned data drops sharply from 41.63% to 1.41% whereas the accuracy of the retained data decreases by less than 2%. Other baseline methods are excluded from direct comparison
Accuracy (%)
Metrics
Baseline
Retrain
Ours
D y
62.92
63.14 + 0,45
61.01 + 0.91
41.63
0.00 = 0.00
1.41 + 0.35
Table 2: Single-label unlearning scenario on Yahoo
Answer dataset with Mix architecture.
as they are designed primarily for image datasets and do not generalize well to text-based tasks. In conclusion, our method consistently lowers leakage and maintains stable unlearning effectiveness across diverse domains.
5.3 ABLATION STUDY
In this section, we conduct an ablation study on the effectiveness of different sizes DP.", our method across varying numbers of passive parties and different privacy-preserving VFL mechanisms.
Module-Omission Experiments: We conduct module-omission experiments to quantify the contribution of individual components to overall performance. We evaluate four experimental variants: (i) standard gradient ascent using all data samples, (ii) few-shot gradient ascent under the same data samples our method, (iii) few- shot gradient ascent with mixup, and (iv) our proposed approach.
100
89.29
88.11
89:29
Accirasy (%)
40.48
20
This set of experiments used R architec- ture on the CIFAR-10 dataset in single-label unlearning scenario. In Figure GA-A (Gradi- ent Ascent-All) refers to the of all 5000 sam- ples from y" to perform gradient ascent. GA-S (Gradient Ascent-Small) uses only 40 data sam- ples from y". matching our method's setting, but without the Manifold Mixup and Remained Accuracy Recovery modules. GA-S + Manifold Mixup, which corresponds to our method without the Remained Accuracy Recovery component. "Ours" refers our full method, which also 40 data samples but incorporates all modules.
GAAA
GA.9
GA-Semieip
Ouita
Figure 3: Comparison of the utility and unlearning effectiveness on different sizes of DP.
As shown in the Figure
GA-A effectively unlearns y but significantly degrades the performance D
i GA-S preserves Dr well but fails effectively unlearn y".
GA-S + Manifold Mixup demonstrates improved unlearning performance but still results in moderate degradation on DD".
iv Our full method achieves both successful unlearning of y" and strong performance preservation on Dr, validating the contribution of the Remained Accuracy Recovery module.
Evaluation for different numbers passive parties: The robustness our approach is unaffected by the number of passive parties. Experiments with varying participant counts (see Figure 4 show consistent performance across all settings, demonstrating that scalability preserved.
Evaluation for different prrseviy VFL methods: Our method remains effective when applied to aivacy-preserving VFL models. To validate this, we evaluate performance under Differen- tial Privacy and Gradient Compression, two widely adopted mechanisms in VFL training (Fu et al.) 2022b). The results confirm that our approach maintains robustness, underscoring its practicality for real-world deployments. Please refer to Appendix A.5 for complete result.
Multi-label Unlearning: Similarly, our method remains highly effective and maintains strong utility under challenging conditions (i.e. unlearning multiple labels), underscoring its reliability
8

---

Published as a conference paper at ICLR 2026
(a) Two Passive Parties
(b) Four Passive Parties
(c) Eight Passive Parties
Figure 4: Accuracy of D", yd and ASR for each unlearning method across ResNet18 model in single-labe label unlearning different numbers of passive parties.
Accurcy (8)
Modd
Dutracto
Mitrice
Baseinn
Retriia
FT
Fister
Amuriia
Unii
BU
SSD
Our
y"
33.40
0.00
0.00
41,15 1:55
49.55 à: 0.40
0.00
0.00 1:
19.00
4.55 = 0.31
0.41 0.11
RoNNe18
ASR
79.61
21.66 0.64
13.22 0.37
25.60 # 0.08
1.54 1113
41.79 #: 1.35
35.40
35.60 =
24.21 = 1.04
D'
71.56
71.21 à 0.13
66.04 0.58
53.56 1: 2.54
39.52 0.03
58.02 0.37
56.37 a 0.39
66.12 2.98
07.90 0.26
CIFAR100
71.00
0.00 0.00
58.00 0.01
25.20 5.75
0.80 0.00
0.00 + 0.00
15.00 0.01
0,00 0.00
1,17 0.29
21.00
48,90
0.40
54.83 0.44
13.70
6.10 0.66
47.53
71.50
88.82
CIRARIO
0.00
Veg16
ASR
$2.64
28.31 8: 1.25
17.75 2.22
68.43 A: 1.14
1.67 = 0.01
46.21 0.72
11.72 :: 0.07
5:27 = 0.01
28.27 :: 1.51
65.75
66 59
0.17
60,79 0.37
35.24 2.21
$7.86 0.81
56.04 0:44
50.02 i: 0.18
58.97 :: 0.05
63.40 :: 0.13
CIFARIO0
58.50
0.00 0.00
11,75 1.25
11.00 =: 4.85
0.00 0.00
0.00 :: 0.00
3.25 0.25
0,00 = tu00
0.09 0,00
30.55
22,75 1,05
32.60
0.65
52.40 0,30
27.90
8,30 0.09
30.47
Table 3: Accuracy of D", yd and ASR for each unlearning method across ResNet18 and 1gg16 models in two-labels unlearning
across diverse scenarios. Tables and present the experimental results for the two-label and multi-label unlearning scenarios, respectively. Please refer to Appendix A.6 for detailed explanation.
Ablation Study for X: For each dataset, we augment the embeddings with two coef- ficients, i.e., À = 0.25 and À 0.5. Addi- tionally, we evaluate the impact of different values in Table 5 The results indicate that variations in À have a minimal impact on the effectiveness of unlearning.
À Rate
Metrics
Accuracy (%)
D,
89.11 + 0.11
[0.2, 0.5]
y"
0.00 = 0.00
D,
89.29 0.19
[0.25, 0.5]
y"
0.00 = 0.00
D,
88.91 + 0.21
[0.33, 0.5]
0.21 = 0.02
In summary, these ablation results demon- strate that method remains stable across various module configurations, numbers VFL
Table 5: Different lambda rates on single-label unlearn- ing scenarios on CIFARIO dataset with ResNet18 archi- tecture. We unlearn label "0" this experiment.
of passive parties, mechanisms, multi-label unlearning, and a wide range of mixup coefficients. Beyond these behavioural and robustness analyses, an important complementary question is how much information the unlearning procedure itself reveals to passive parties? ? We therefore next examine transcript-level leakage through the lens of process privacy.
5.4 PROCESS PRIVACY IN VERTICAL FL UNLEARNING
Unlearning data from vertically partitioned passive models introduces a distinct privacy chal- lenge in VFL. That is, during unlearning request, the active party must exchange interme- diate information, such as embeddings or gradients, that can inadvertently reveal which sam- ples labels being deleted. For example, in retraining, the active party must indicate the

---

Published as a conference paper at ICLR 2026
Accuracy
Model
Darswets
Metrics
Biseline
Retrai
FT
Fisher
Amneiiac
Unit
BU
SSD
Ouss
71.58
71.91
0.33
$4.79
1.04
50,05
0:38
ResNet88
CIRARIO0
72.00
0.00
33,87 0.88
45.38
1.13
0.00
0.00 1:
0.25
0.33 0.144
ASR
$6.65
16.05
0.85
18.23
62.78 + 3.91
1,19
68.63 1.8
38.35 +
0.34
12,40 1.82
65,83
65.66 + 0.08
60,92 0.08
36.85 + 1.00
$7.06 0.18
56,86 + 0.26
47.04 it
61.01 + 0.55
64.64 + 0.12
Ver16
CIFAR100
60.25
0.00
0.00
7.63 d: 0.13
28.75 di 1.25
0,80
0000
0.00 :: 0.00
7.13 0.11
0.00
1000
1.17 à 0.25
75,30
27.20 0.75
24.88 3.3
a.77
4,50 0015
32,83 0.58
29. a 0,00
10.96 0.59
27.50 0.29
Table 4: Accuracy of D", y" and ASR for each unlearning method across ResNet18 and Vgg16 models in multi-labels unlearning
sample IDs to be removed, directly exposing the deletion set. In boundary unlearning, gra- dients associated with the deleted label may similarly disclose sensitive membership informa- tion. As shown in Table [6] retraining reveals the full deletion set to the passive party (100%
leakage), and the risk increases further when multiple labels are unlearned (Appendix A.3)
To make this leakage channel explicit, we intro- duce the notion of process privacy, which char- acterises how much the passive party's 's belief about the deletion set can change after observ- ing the protocol transcript. Let the passive party begin with a prior belief about which samples constitute the deletion set. After receiving the transcript T, which may contain embeddings, gradients, or auxiliary updates, the passive party updates this belief. We quantify this shift us- ing the Kullback-Leibler divergence between the posterior and prior beliefs. The smaller the divergence, the less information the transcript conveys about the deleted data.
Membership Leakage Rate (%)
Datasets
Standard Retraining
Ours
CIFARTO CIFARION 100
100
14.38
100
4.04
Table 6: The averaged membership leakage rate evaluated across all class for CIFARI0/100 with ResNet18 (lower better). We set k = 5000 for CIFARIO and = 500 for CIFAR 100, where is the total number of samples in each class.
Definition 1 (Process Privacy). A VFU protocol satisfies process privacy with parameter E if the passive party's 's posterior belief about the deletion set, after observing the transcript TT, differs from its prior belief by al most s, measured via the Kullback-Leibler divergence. Formally, the protocol satisfies process privacy when DRt. (PDD (P) P(D")) S which captures the requirement that the unlearning transcript should not substantially increase the passive party's 's ability to infer which samples were deleted.
This formulation aligns naturally with our empirical leakage evaluation, which quantifies how the passive party's inference capability changes after observing the transcript under retraining, boundary unlearning, and our method. Our approach restricts information exchange to embeddings and gradient updates of a small public subset, which, supported by the Bernstein bound in Appendix AA.4Iimmts the residual influence of unlearned data. Empirically, this yields a substantial reduction in leakage, from 100% under retraining to as low as 14.38% under our method, as shown in Table
As VFU is an emerging research area, prior work has not formally characterised this transcript-level privacy dimension. To the best of our knowledge, this paper is among the first to articulate, define, and empirically study process privacy in the VFL context. While the definition provides a conceptual foundation for describing transcript leakage, deriving tight theoretical upper bounds on remains significantly more challenging and is left as an open problem.
CONCLUSION
This paper presents a pioneering approach to label unlearning within the vertical federated learning (VFL.) domain, addressing a critical gap in the existing literature. By introducing a few-shot unlearning method that leverages manifold mixup, we effectively mitigate the risk of label privacy leakage while ensuring efficient unlearning from both active and passive models. Our systematic exploration of potential label privacy risks and extensive experimental validation on benchmark datasets underscore the proposed method's 's efficacy and rapid performance. Ultimately, this work not only advances the understanding of unlearning in VFL but also sets the stage for further innovations in privacy- preserving collaborative machine learning practices.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This research is supported by the ASEAN-China Cooperation Fund (ACCF) under the project "Deep Ensemble Under Non-Ideal Conditions and Its Typical Applications in Computer Vision".
REFERENCES
Ali Abbasi, Chayne Thrash, Elaheh Akbari, Daniel Zhang, and Soheil Kolouri. Covarnav: Machine unlearning via model inversion and covariance navigation. arXiv preprint arXi:2231.12299, 2023,
Manaar Alam, Hithem Lamri, and Michail Maniatakos. Get rid of your trail: Remotely erasing backdoors in federated learning. IEEE Transactions on Artificial Intelligence, 2024.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette- Hengrui Jia. Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pp. 141-159. IEFE, 2021.
Xiaoyu Cao, Jinyuan Jia, Zaixi Zhang, and Neil Zhenqiang Gong. Fedrecover: Recovering from poisoning attacks in federated learning using historical information. In 2023 IEEE Symposium Secuuityy and Privucy (SP), pp. 1383, 2023a. doi: 10.1109/SP46215
Xiaoyu Cao, Jinyuan Jia, Zaixi Zhang, and Neil Zhenqiang Gong. Fedrecover: Recovering from poisoning attacks in federated learning using historical information. In 2023 IEEE Symposium Security and Privacy (SP), pp. 1366-1383 1383. IEEE, 2023b.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium Security and Privacy, pp. 480, 2015. doi: 10. 10.1109//..0015.35..
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguisticalll--nfornee
interpolation of hidden space for semi-supervee superviide text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2147-2157, 2020.
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang, When machine unlearming jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS '21. ACM, November 2021. doi: 10.1145/3460100. 3484756. URL UtLhttp:://dx doi 100 14 /1111//444500 3460120 3484756
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7766-7775, 2023.
Wei-Yr Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint vrXiv:1904.042333 2019.
Dasol Choi and Dongbin Na. Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems. arXiv preprint arXiv:2311.02246 2023.
Vikram S. Chundawat, Ayush K. Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning, IEEE Transactions on Information Forensics and Security, 18:2345-2354, 2023a. doi: 10.1 202333366...0.332555006
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7210-7217, 2023b.
Zihao Deng, Zhaoyang Han, Chuan Ma, Ming Ding, Long Yuan, Chunpeng Ge, and Zhe Liu. Vertical federated unlearning on the logistic regression model. Electromics, 12(14), 2023. ISSN 2079-9292. doi: 10.3390/elcctrooiiss214433 12133182. URL https: //ww///wwww/./ mapi wtt.ccmmm/200/// 2079 001199-2222222212
3182
Akash Dhasade, Yaohong Ding, Song Guo, Anne-mare Kermarrec, Martijn De Vos, and Leijie Wu. Quickdrop: Efficient federated unlearning by integrated dataset distillation. arXiv preprint arXiv:2311.15603 2023.
11

---

Published as a conference paper at ICLR 2026
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 12043-12051, 2024a.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 12043-120511 2024b.
Chong Fu, Xuhong Zhang, Shouling Ji, Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X. Liu, and Ting Wang. Label inference attacks against vertical federated learning. In 31st USENIX Security Symposium (USENIX Security 22), pp. 1397-1414, 2022a. ISBN 978-1-939133-31-1
Chong Fu, Xuhong Zhang, Shouling Jinyin Chen, Jingzheng Wu, Shanqing Guo, Jun Zhou, Alex X Liu, and Ting Wang. Label inference attacks against vertical federated learning. In 31st USENIX security symposium (USENIX Security 22), pp. 1397-1414, 2022b.
Xiangshan Gao, Xingjun Ma, Jingyi Wang, Youcheng Sun, Bo Li, Shouling Ji, Peng Cheng, and Jiming Chen. Verifi: Towards verifiable federated unlearning. IEEE Transactions on Dependable and Secure Computing, 2024.
Sanjam Garg, Shafi Goldwasser, and Prashant Nalini Vasudevan. Formalizing data deletion in the context of the right to be forgotten. Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 373-402. Springer, 2020.
Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlcarning. arXiv preprint arXiv:22010066-00 2022.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9304 9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks information accessible from input-output observations. In 16th European Conference Computer Vision (ECCV), pp. 383-398 2020b.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 792-801, 2021.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Hanlin Gu, WinKent Ong, Chee Seng Chan, and Lixin Fan. Ferrari: Federated feature unlearning via optimizing feature sensitivity. Advances in Neural Information Processing Systems, 37:24150- 24180, 2024a.
Hanlin Gu, Gongxi Zhu, Jie Zhang, Xinyuan Zhao, Yuxing Han, Lixin Fan, and Qiang Yang. Unlearning during learning: an efficient federated machine unlearning method. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 4035-4043, 2024b.
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, pp. 3832-3842, 2020.
Anisa Halimi, Swanand Ravindra Kadhe, Ambrish Rawat, and Nathalie Baracaldo Angel. Federated unlearning: How to efficiently erase a client fi? International Conference on Machine Learning, 2022.
Mengde Han, Tianqing Zhu, Lefeng Zhang, Huan Huo, and Wanlei Zhou. Vertical federated unlearning via backdoor certification. IEEF Transactions on Services Computing, 2025.
12

---

Published as a conference paper at ICLR 2026
Tuan Hoang, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Learn to unleam for deep neural networks: Minimizing unlearning interference with gradient projection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2819-4828, 2024.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. arXiv preprint arXiv:2101 00000....88..... 2021.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36:51584-5166 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in neural information processing systems, 36: 1957-1997, 2023.
Y. Lecun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-22) 1998. doi: 10. 10.1109/5.72679.
Bowen Li, Jian Zhang, Jie Li, and Chentao Wu. Securecut: Federated gradient boosting decision trees with efficient machine unlearning. International Conference on Pattern Recognition, pp- 350-365. Springer, 2024.
Guanghao Li, Li Shen, Yan Sun, Yue Hu, Han Hu, and Dacheng Tao. Subspace based federated unlearning. arXiv preprint arXiv:2302. 12448, 2023.
Li Li, Yuxi Fan, Mike Tse, and Kuo-Y Yi Lin. A review of applications in federated learning. Computers & Industrial Engineering. 149:106854, 2020.
Gaoyang Liu, Xiaoqiang Ma. Yang Yang, Chen Wang, and Jiangchuan Liu. Federaser: Enabling efficient client-level data removal from federated learning models. 2021 IEEE/ACM 29th Inter- national Symposium Quality of Service (IWQOS), pp. 1-10, 2021. doi: 10.1109/WQQS520092
2021.9521274.
Yang Liu, Yan Kang. Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, and Qiang Yang. Vertical federated learning: Concepts, advances, and challenges. IEEE Transactions on Knowledge and Data Engineering, 36(7):31153364, 2024a. doi: 10.TT0.T
2024.3352628.
Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. The right to be forgotten in federated learning: An efficient realization with rapid retraining. In IEEE INFOCOM 2022 IEEE Conference Computer Communications. IEEE, May 2022. doi: 10. http:////f.coom88000000000000000
URL
http:/ dx. doi wrt. 10. 1109/ BATP//INOOCO WNWOCOM4888800... 2022. 9796721
Ziyao Liu, Yu Jiang, Jiyuan Shen, Minyi Peng, Kwok-Ya Yan Lam, Xingliang Yuan, and Xiaoning Liu. A survey on federated unlearning: Challenges, methods, and future directions. ACM Computing Surveys, 57(1):1-38, 2024b.
Puneet Mangla, Nupur Kumari, Abhishck Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning, In Pro- ceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2218-2227, 2020.
Tawsifur Rahman. Covid-19 radiography
radid-Iga database, Mar 2022. URL https: www. kaggle http://awwwaaaaaoaooaooaaaaaaaaa
com/
Nicolò Romandini, Alessio Mora, Carlo Mazzocca, Rebecca Montanari, and Paolo Bellavista. Federated unlearning: A survey on methods, design guidelines, and evaluation metrics. IEEE Transactions Neural Networks and Learning Systems, pp. 1-21, 2024. ISSN 2162-2388. doi: 10.1 10111099mmms..22444733333 URL hctp: / dx doi owg. 1109/ C0L09/TNNLL TNNLS 2024
3478333
13

---

Published as a conference paper at ICLR 2026
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Ningxin Su and Baochun Li. Asynchronous federated unlearning. In IEEE INFOCOM 2023 - IEEE Conference on Computer Communications, pp. 1-10, 2023. doi: 10.1109/FFOOMM539999 2023. 10229075.
Ayush K. Tarun, Vikram S. Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions Neural Networks and Learning Systems, pp. 1-10. 2024. ISSN 2162-2388. doi: 10.1 1019//nmss.22333222222..2. URL http/xxx.ooo.... o
T109/ CNNLSNNLS.... 200...2222333333
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International conference on machine learning, pp. 6438 1888. PMLR, 2019.
Roman Vershynin. High- digeesiimnii probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. Federated unlearning via class-discriminative pruning. In Proceedings of the ACM web conference 2022, pp. 622-632, 632, 2022.
Pengfei Wang, Zhaohong Yan, Mohammad S. Obaidat, Zhiwei Yuan, Leyou Yang, Junxiang Zhang, Zongzheng Wei, and Qiang Zhang. Edge caching with federated unlearning for low-latency v2x communications. IEEE Communications Magazine, pp. 1-7, 2023. doi: 10.1109/MOM.001 2300272.
Zichen Wang, Xiangshan Gao, Cong Wang, Peng Cheng, and Jiming Chen. Efficient vertical federated unlearning via fast retraining, ACM Transactions on Internet Technology, 24(2):1-22, 2024.
Chen Wu, Sencun Zhu, and Prasenjit Mitra. Federated unlearning with knowledge distillation. arXiv preprint arXiv:2201.0944/) 094441 2022.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Hui Xia, Shuo Xu, Jiaming Pei, Rui Zhang, Zhi Yu, Weitao Zou, Lukun Wang, and Chao Liu. Fedme2: Memory evaluation & erase promoting federated unlearning in dtmn. IEEE Journal on Selected Areas in Communications, (1(1)):55733358 2023. doi: 10. 109///AC.202..331004))
Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient
architecture for exact machine unlearning. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joins Conference on Artificial Intelligence, IJCAI-22, pp. 4006 International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: 10.249633ijcci.220225566 URL http:/ttss//oo.oogg//////// .2....249933 2493222233333 2022 Main Track.
Minmin Yang, Weiheng Chai, Jiyang Wang, and Senem Velipasalar. Simplimix: A simplified manifold mixup for few-shot point cloud classification. In Proceedings of the IEEE/CVF Winter Conference Applications Computer Vision, pp. 3668-3677 2024.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions Intelligent Systems and Technology (TIST), 10(2):1-9, 2019.
Guanhua Ye, Tong Chen, Quoc Viet Hung Nguyen, and Hongzhi Yin. Heterogeneous decentralised machine unlearning with seed model distillation. CAAI Transactions on Intelligence Technology. (3):608-619. 2024.
Youngsik Yoon, Jinhwan Nam, Hyojeong Yun, Jacho Lee, Dongwoo Kim, and Jungseul Ok. Few-shot unlearning by model inversion. arXiv preprint arXiv: /5567, 2022.
14

---

Published as a conference paper at ICLR 2026
Wei Yuan, Hongzhi Yin, Fangzhao Wu, Shijie Zhang, Tieke He, and Hao Wang. Federated unlearning for on-device recommendation. Proceedings of the sixteenth ACM international conference on web search and data mining, pp. 393 401, 2023.
Lefeng Zhang, Tianqing Zhu, Haibin Zhang, Ping Xiong, and Wanlei Zhou. Fedrecovery: Differ- entially private machine unlearning for federated learning frameworks. IEEE Transactions on Information Forensics and Security, 18:4732-477 2023a. doi: 1010000FF.202..322222
Xulong Zhang, Jianzong Wang, Ning Cheng. Yifu Sun, Chuanyao Zhang, and Jing Xiao. Machine unlearning methodology based on stochastic teacher network. In International Conference Advanced Data Mining and Applications, pp. 250-261, 2023b.
Yang Zhao, Jiaxi Yang, Yiling Tao, Lixu Wang, Xiaoxiao Li, and Dusit Niyato. A survey of federated unlearning: A taxonomy, challenges and future directions. arXiv preprint arXiv:2310.19211 2023.
Yian Zhao, Pengfei Wang, Heng Qi, Jianguo Huang. Zongzheng Wei, and Qiang Zhang. Federated unlcarning with momentum degradation. IEEE Internet of Things Journal, 1165)8860-8870, 2024. doi: 10. 10.1100/00022203333315444
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
This appendix provides additional theoretical insights, extended experimental results and further implementation details to support the main submission. We begin with theoretical analysis of unlearning effectiveness in Section A.1 followed by table of notations in Section A.2 for reference. Section A.3 provides an explanation of label leakage during the unlearning process in VFL, while Section A.4 discusses why our method can achieve strong results with only small number of data samples. In addition, Section A.3 presents ablation experiments on applying our method to Krivey--reseerve VFL models. Section A.6 presents the experimental results for two-label and multi- label unlearning scenarios, showcasing the effectiveness of our solution handling more complex unlearning tasks. This section also includes detailed performance analysis of each baseline across three key metries: (i) accuracy on 2 (ii) accuracy on y" and (iii) ASR score. Section A.7 presents comparative table of method against existing Federated Unlearning approaches. highlighting the capability of our method to preserve label privacy during the unlearning process. In Section A.8 we provide detailed information about the experiment setup, including model architectures, unlearning scenarios, baseline methods, datasets, and the data distribution among parties. Finally, in Section A.9 we discuss the limitations of our work, including aspects beyond its scope, methodological constraints, and other relevant considerations.
A.1 THEORETICAL ANALYSIS FOR UNLEARNING EFFECTIVENESS
Theorem 2. Suppose that the trained passive model 0 and active model w achieve near-zero training loss. Then, when unlearning a single label, the following holds:
B 3n) Wellw; H",'")) ""' B(H an) Wal H :y") 0,
(8)
E F .o) Ve H T²)
Vol(O; H" y²) 0,
where (IH","")) denotes the manifold mixup embeddings and labels of the public data DR. associated with the unlearned label, (H" )²") denotes the embeddings and labels of the complete unlearned dataset D".,, and E is the main task loss of VFL.
Proof. In simplified analysis, we set the loss based on the small-size public data Du., the manifold mixup of DD.P, and the original unlearned data D are &,, Emir and & respectively. We consider two-layer linear neural network: f(x) = w(Ox). The loss function is the Mean Squared Error (MSE): L D
EiED (z()) 11.
For public unlearned sample (xt" (x).y"" let H. Ox?" be the hidden representation, 2, = wH be the output, 2, y² be the error. The gradients are:
Vul; = e Voli = T exx uT
(9)
Averaging the gradients over the any dataset Du.P are:
Val
1
Σ T. Vol
1
E w Te,
(10)
D
D
Given two samples ( y)), (x"),0 ) and a mixup coefficient À [0, 1], we have:
Hmix AHK + (1 A)H Ymix Ayy + (1 A)8)
(11)
Xmix Wiinx
>24 (1 2)2 emix = 2mix Mmix de; + (1 A)e,
Then we have the gradients on w for manifold mixup:
Emix Cmix H
()e; + -A))))))HHH +(1 (11 T)
= =.P +(1 - He He.H.T)
16

---

Published as a conference paper at ICLR 2026
The gradients for manifold mixup:
V mii
Vhbiix Hoim
w emix (Axl + (1 1)zz )
= -A'w Texx
+ A(1 - A)(w ee + +w Twezzz" (1 - N) T
Since
Eleh J Vali,
Elw ElwTex ex Voli
We have:
E mix] = 28| V. + 2( - (e)
E
V
0
l
mix
22
1 [
B[}3]) [[c]E[a]]
Due to the near-zero training loss, E(e] tends to be zero, we obtain the averaged gradients of w and on the public dataset Du.
E E Lmix 2 J.,, V 2
(12)
Since the DP- is the subset of the total unlearned dataset D", Inequality to obtain Veli Vol2 > 0 and Vuh V >> with the probability 0 where
leverage the Chebyshev's 's
Np.. is the data size of D Combining Eq.equation 12
we can obtain
E Laix
V
E Volnx]]
E[>2]V 0.
(13)
which completes the proof.
Theorem 2 indicates that the gradient update direction for label unlearning, when applied to the augmented embeddings of the public unlearned data, positively aligned with the update direction derived from the embeddings of the entire unlearned dataset. This result suggests that gradient-based label unlearning using only public data effective and approximates the behavior of unlearning with access to all unlearned data.
A.2
TABLE OF NOTATION
Table 7 summarises all notations used throughout the paper for clarity.
Notation
Mcaning
Fss Go
Active model and ken passive model
wn
0%
Unlearned active model and unlearned km passive model
K
The number of passive party Mixed coefficient
A
y
Learning rale
N
Unlearning epochs
ck
Private features own by ken passive party
y
Private label owned by active party
yd
unlearn labels
{ag)
The unlearned feature for client & corresponding to the y
I
The known features for client corresponding to the y"
He
Forward embedding of passive party k
fg. ITg
Augmented forward embedding of passive party k of unlearn data and remain data.
Table 7: Table of Notations
A.3 LABEL LEAKAGE DURING UNLEARNING
Figure Sguure5Jshone showss the label leakage (in %) of Boundary Unlearning in VFL settings for varying numbers of unlearning labels. Single label unlearning shows 100% label leakage since passive parties can
17

---

Published as a conference paper at ICLR 2026
100
100
100
100
97.8
Label
Labels
Labels
80
72.41
Label Leakage (%)
63.55
62.45
60
55.79
52.09
40
20
0
MNIST
CIFAR10 Datasets
CIFAR100
Figure 5: Illustration of label leakage (%) with Boundary unlearning in VFI. using ResNet18 model on different numbers of labels and datasets.
immediately infer that all samples involved belong to the same label. Moreover, when unlearning multiple labels (m,,), label leakage worsens as the passive party can infer on the transferred gradient by the active party. Increasing the number of labels, therefore, results in larger portion of data being exposed during the unlearning process. For instance, with four labels from CIFAR-100, total of 62.45% label leakage exposed. information from the gradients g" sent by the active party. Specifically, the passive party employs clustering on g" to derive my clusters by optimizing the following objective function:
boundary unlearning, the passive party can infer label
ma
min
10
9
(14)
S.EC,
where Cj denotes the of points assigned to cluster and Ø3 represents the centroid of cluster j. Consequently, the passive party can deduce the labels of the features in X.
A.4 WHY TINY PUBLIC SET SUFFICES
Our goal is unlearning, not full re-training. Thus, we only need a small parameter shift to cancel the target label's influence. Below, we quantify three reasons why a much smaller public set Da. can achieve this.
1. Variance reduction via manifold mixup. Manifold mixup augments each public example into an infinite family of virtual points, reducing the gradient-estimator variance proportion- ally the number of synthetic mixtures. Recent few-shot studies show that manifold-mixup greatly reduces sample requirements. For example, (Mangla et al., (2020) achieves higher accuracy than prior baselines across four vision benchmarks with only 5 to 20 samples per class. Similarly. SimpliMix (Yang et al. 2024) achieves strong gains in few-shot 3D point cloud elassification.
2. Tiny update magnitude. The pre-unlearning model already fits the data (loss 0). As shown by (Chen et al., 2019), when the source and target domains are similar, even simple few-shot methods can perform well with minimal fine-tuning. Hence, once the gradient direction is reliable, only small update is needed. In other words, because unlearning starts from a model that already fits the data, only the label-specific contribution needs to be removed, resulting in small parameter displacement. In the quadratic loss setting analysed
18

---

Published as a conference paper at ICLR 2026
in theoretical analysis (Appendix A.I). the required step length scales with the gradient norm. Once the direction is correct, even a single mini-batch update can eliminate the label.
3. Exponential concentration of the gradient direction. Extending Eq. equation 12 in our Appendix A.1 with the vector Bernstein inequality (see Section 2.8 of (Vershynin) 2018)) gives
Prfcos Lix, V.e) S 11 exp exp(=c|D=P|e3)
where с depends on B[ Thus, the chance that the mixed-sample gradient deviates substantially from the full-data gradient vanishes exponentially with the number of public samples; a few dozen already give high-probability alignment.
Specifically, the underlying tail bound for the sum of augmented embeddings follows precisely the Bernstein inequality (Vershynin) 2018):
PIEE X t) eexpl-cminff* /E, | t/max, || Hl.
where X; = cos L(V Eixxis V () and f represents in mixture gradients.
Intuition. Each augmented sample acts like an arrow pointing toward the true unlearning direction. Applying manifold mixup to only small number of data samples generates thousands synthetic gradient directions. Averaging these directions cancels out random variance, as guaranteed by the Bernstein inequality. Because our model already starts near solution, short step along this accurately aligned direction suffices to complete unlearning.
A.5 EVALUATION FOR DIFFERENT PRIVACY PRESERVING VFL METHODS
We evaluate our unlearning method under two privacy preserving VFL methods: (i) Differential Privacy (Fu et al. 2022b) and (ii) Gradient Compression (Fu et al. 2022b). Fig. presents the effectiveness of our solution on both methods across different levels of variance Gaussian noise and compression ratio. Higher Gaussian noise levels and greater gradient compression ratios enhance privacy protection in VFL but lead to increased performance degradation. It shows that even for large compression ratio and noise level, our proposed method can still unlearn effectively, while the utility of the vertical training decreases significantly.
A.6 TWO-LABEL AND MULTI-LABEL UNLEARNING
Tables present the experimental results obtained from the two-label and multi-label unlearning scenarios, respectively.
Table 3 provides a detailed breakdown of the performance metrics and outcomes observed during the two-label unlearning process across various datasets and architectures. It highlights the accuracy scores of D", y" and ASR when unlearning was applied to labels "0" and "2", demonstrating the utility guarantees and the effectiveness of the baseline unlearning methods.
Similarly, Tab. summarizes the results of the multi-label unlearning experiments, where labels "0", "2", "5", and "7" were unlearned for all datasets and architectures. This table captures critical metrics such as the accuracy scores of 'D"," y" and ASR reflecting the outcomes of the multi-label unlearning process.
A.6.1 UTILITY GUARANTEE
From Tab. B the FT method shows performance consistent with the single-label unlearning scenario. It preserves D well on datasets with a small number of labels (e.g.. CIFAR-I0) but performs poorly on datasets with many labels (e.g., CIFAR-I00, The Fisher Forgetting method consistently demonstrates poor preservation of 'D" across all datasets and model architectures. The Amnesiac approach performs similarly to FT, achieving strong preservation labels (e.g., CIFAR-10) but showing reduced performance on datasets with larger label space (e.g.. CIFAR- 100). The Unsir and BU methods also follow this trend, maintaining good preservation on simpler datasets like CIFAR-10, 10, but experiencing greater degradation in D' accuracy across all datasets compared to FT and Amnesiac. SSD demonstrates strong preservation of D' across all experiments, except for the CIFAR 100 dataset under the VGG16 architecture.
D on datasets with fewer
19

---

Published as a conference paper at ICLR 2026
100
100
Accuracy (%)
Accuracy
Baselines
Ours
40
40
20
Baselines
20
Ours
0
0
1e-04
7.5e-05
3.5e-05
1e-05
1e-04
7.5e-05
3.5e-05
1e-05
Gaussian Level (Variance)
Gaussian Level (Variance)
(a) D
(b) y
100
100
Acouracy (%)
Accuracy
Baselines
Ours
Baselines
20
Ours
0
0
99.90
99.80
99.70
99.90
99.80
99.70
Compression Ratio (%)
Compression Ratio (%)
(c) D
(d) y
Figure 6: Comparison of remained accuracy D" and unlearned accuracy y" under Gaussian noise variance (a, b) and gradient compression ratio (c, d). Subplots (a, c) represent the remained accuracy D", while (b, d) show the unlearned accuracy y".
In contrast, our method consistently achieves strong unlearning utility while perfectly preserving Dr across all experimental settings.
From Tab. 4 the FT method shows similar behavior to the single-label and two-label unlearning scenarios, with poor preservation of D on datasets that have a large number of labels (e.g., CIFAR- 100). The Fisher Forgetting method consistently performs poorly in preserving D across all datasets and model architectures. The Amnesiac approach also mirrors the performance reduced preservation datasets with many labels. Similarly, the Unsir and BU methods exhibit even greater degradation in D" performance across all datasets compared to FT and Amnesiac. SSD yields comparable results in both single-label and two-label unlearning scenarios, showing strong preservation of D", though its performance is slightly inferior to ours.
FT, showing
In contrast, our method demonstrates strong unlearning utility and achieves near- wear-perfect preservation of DD" across all experimental settings, including multi-label unlearning scenarios and for datasets with large number of labels.
In conclusion, our experimental results across single-label, two-label, and multi-label unlearning tasks demonstrate that our method consistently delivers the best balance between utility and privacy preservation. For example, unlike FT. which maintains D" accuracy only on datasets with few labels (e.g., CIFAR-10) but suffers on CIFAR-100, our method achieves near-perfect preservation of "D" regardless of label complexity. Fisher Forgetting performs poorly across all datasets and architectures, leading to significant drops in retained accuracy. Amnesiac Unlearning introduces instability through random relabeling of D,,, leading to accuracy degradation high-label datasets, whereas Unsir and BU show even greater declines, especially in multi-label settings. SSD shows good preservation of D' across different settings, but its performance is slightly inferior to ours. In contrast, our method maintains high Dr accuracy (often above 98%) while reducing unlearned data accuracy close to random chance, consistently across all scenarios. These results provide strong
20

---

Published as a conference paper at ICLR 2026
evidence that our approach offers superior robustness, effectiivnnsss, and generalizability for practical, unlearning.
A.6.2 UNLEARNING EFFECTIVENESS
We evaluate unlearning effectiveness using two key metrics: the accuracy on y" and the Attack Success Rate (ASR). A lower accuracy on y" indicates more effective unlearning, However, an ASR of 0% may signal the presence of the Streisand effect (Golatkar et al., 2020a). where the model consistently misclassifies all y" samples into the same incorrect label, w which indicates potentially leaking information about the unlearned label. Ideally, the ASR should be slightly lower than that of retrained model, reflecting successful unlearning without revealing leakage. If the model consistently assigns all y" to a specific wrong label, it might expose that the model has treated y" uniquely. suggests the model has not truly "forgotten" " the data but developed an unusual bias against it.
Analysis on yd This paragraph provides detailed analysis of each baseline method's performance with respect to the y" accuracy metric.
From Tab. both FT and Fisher Forgetting methods exhibit poor unlearning effectiveness in the two-label unlearning scenario. These methods perform effectively only in the single-label unlearning context, In contrast, both the Amnesiac and Unsir methods consistently achieve perfect unlearning effectiveness in the two-label unlearning scenario. The BU method shows a decline in unlcarning effectiveness compared to its performance in single-label unlearning, indicating challenges in handling scenarios with two labels. SSD demonstrates strong unlearning effectiveness the CIFAR 100 dataset but fails to reduce the accuracy of y" to 0.00% on CIFAR dataset. Meanwhile, our method continues demonstrate exceptional unlearning effectiveness, maintaining its outstanding performance in the two-label unlearning scenarios, further solidifying its robustness and efficiency.
From Tab. the FT and Fisher Forgetting methods exhibit similarly poor unlearning effectiveness in the multi-label unlearming scenario, comparable to their performance in the two-label unlcarning context. These methods are effective only the single-label unlearning scenario. In contrast, the Amnesiac and Unsir methods consistently achieve perfect unlearning effectiveness in the multi- label scenario, mirroring their performance in single-label and two-label contexts. The BU method performs similarly to the two-label scenario in the multi-label context, showing limited unlearning multi-label scenario. Meanwhile, our
effectiveness. SSD shows good unlearning effectiveness method continues to demonstrate exceptional unlearning effectiveness, maintaining outstanding performance in multi-label scenarios, further reinforcing its robustness and efficiency.
In conclusion, our method demonstrates superior performance across all unlearning scenarios. Specif- ically, it consistently achieves near-perier wear-perfect preservation of retained data D", often maintaining accuracy drops below 2% while reducing the accuracy of unlearned labels to near-random levels (e.g., from 41.6 41.63% to 1.41% %). In contrast, FT and Fisher Forgetting fail to scale beyond simple settings: they perform adequately only in single-label scenarios but suffer substantial degradation in DD" accuracy when applied to two-label or multi-label tasks, especially on datasets like CIFAR-100. Amnesiac and Unsir do achieve strong unlearning effectiveness across all scenarios, but at the cost of destabilizing the decision boundaries, which leads to marked drops in D accuracy for more complex datasets. BU also shows deteriorating performance as task complexity increases. SSD performs well in unlearning, but shows minor weakness on CIFAR10 under the two-label setting. Only our method balances both objectives, that is effectively unlearning target labels while preserving the rest, across all datasets, architectures, and label complexities. These concrete results affirm our approach as the most robust, generalizable, and practically deployable solution for modern unlearning needs.
Analysis on ASR This paragraph discusses how each baseline method performs in terms of ASR, highlighting their strengths and weaknesses.
From Tab. 3 and FT demonstrates excellent performance, achieving ASR scores that are consistently lower but close to those the retrain model across all datasets, with the exception of the multi-labels scenario involving the ResNet18 architecture and the CIFAR 100 dataset. In that case, the ASR score is only slightly higher than that of the retrain model. In contrast, Fisher Forgetting performs poorly in both the two-labels and multi-labed scenarios, exhibiting high ASR scores across all experiments, which indicates its ineffcctiveness in mitigating adversarial risks. The Amnesiac method shows a consistent pattern of very low ASR scores across all experiments, potentially suggesting vulnerabilities
21

---

Published as a conference paper at ICLR 2026
to the Streisand effect, observed in previous analyses. The Unsir method performs similarly to Fisher Forgetting, also achieving high ASR scores across all experiments, further highlighting its ineffectiveness. Boundary Unlearning continues exhibit inconsistent performance throughout the experiments, failing to deliver reliable results. SSD continues to perform well across the experiments, consistently achieving ASR scores lower than and close to those of the retrain model. On the other hand, our method demonstrates outstanding performance across many datasets, achieving ASR scores that are consistently lower than, and close to, those of the retrain model. This highlights our method's effectiveness and reliability in addressing adversarial risks while maintaining robust unlearning outcomes.
In conclusion, the results clearly demonstrate the effectiveness and robustness of our method in mitigating adversarial risks. Across datasets, our approach consistently achieves ASR scores that are not only low but also often lower than those of the retrained models, indicating strong unlearning without compromising security. While FT' and SSD perform reasonably well in most scenarios, other methods such Fisher Forgetting, Unsir, and Boundary Unlearning exhibit limited effectiveness, with high ASR scores and inconsistent performance. Although the Amnesiac method achieves very low ASR scores, it raises concerns about potential vulnerabilities, such as the Streisand effect. In contrast, our method strikes the optimal balance between unlearning effectiveness and adversarial robustness, making it the most reliable and secure solution among the evaluated approaches.
A.6.3 FURTHER ANALYSIS OF BASELINES
Fine-Tunine Fine-Tuning (FT): FT demonstrates adequate performance in utility preservation for datasets with smaller number of labels (e.g.. CIFARIO), its effectiveness diminishes sharply the dataset complexity increases, failing to preserve D" effectively for datasets with a large number of labels (e.g.. CIFAR100). Additionally, its unlearming effectiveness is limited to single-label scenarios, highlighting its inability to handle more complex unlearning contexts.
This method struggles across all dimensions, exhibiting poor preservation of D irrespective of the dataset or architecture. It also fails to achieve unlearning effectiveness in two-label and mulli-labels, unlearning scenarios, with high adversarial success rates (ASR) that compromise its ability to address adversarial risks. This consistent underperformance underscores the method's s lack of robustness diverse scenarios.
Fisher Forgetting:
Amnesiac Unlearning:
While the Amnesiac method demonstrates strong unlearning effectiveness across various scenarios, its utility preservation declines significantly when applied to datasets with large number of labels, such as CIFAR-100. This is primarily due to its approach of randomly assigning incorrect labels to D.. which introduces unpredictable shifts in decision boundaries and undermines reliability in more complex settings. Additionally, the method consistently yields very low ASR scores across all experiments. This implies that although low ASR scores indicate strong unlearning, they may also indicate susceptibility the Streisand effect, as observed in prior analyses.
Unsir and BU Methods: Both methods show similar limitations, with significant degradation in utility preservation dataset complexity increases. Although these methods achieve moderate unlearning effectiveness, they exhibit high ASR scores, indicating their vulnerability to adversarial risks. The inconsistency of their performance across scenarios highlights their lack of generalizability and robustness.
SSD: SSD demonstrates strong performance across all three evaluation metries, namely accuracy on D", accuracy on y" and ASR score, although it remains marginally worse than our method in each case. Moreover, SSD incurs a higher computational cost, resulting longer runtime than our approach (see Fig. 2 in Sect. 5.2.3). further limiting its practical efficiency.
Overall, the experimental results highlight the key limitations of existing methods, such as FT, Fisher Forgetting, Amnesiac, Unsir, BU and SSD in striking a balance between utility preservation, unlearning effectiveness, and low ASR scores. While some methods may perform well one aspect, they often fall short in others, especially in complex multi-label scenarios. In contrast, our method consistently delivers strong unlearning performance, near pear-eerfect utility preservation, and robust ASR scores across all datasets and settings. This comprehensive and consistent performance firmly establishes our approach as the most reliable, effective, and scalable solution for tackling diverse and challenging unlearning tasks.
22

---

Published as a conference paper at ICLR 2026
Method
Type of Fl.
Unlearning Types
Unlearning Targets
Protection for Unlearned Data
FodEraser (Liu et
2021
Exact Approximate
Client
Yuan et 2023
Client
FedRecovery Zhang et
2023a
Approximate
Client
VoriFl
G80 ot
Approximate
Client
Approximate
Client
Approximate
Client
Horizonial
FedRecover Cao et 20233
Approximate
Client
Knowledge
2023
Approximate
Client
Discriminative Pruminy Wany
Approximate
Clays
7han
2024
Approximate
Class
Rapid Retraining
Exact
Sample
QuickDrop Dhasude
Approximate
Sample
FedAU
2024h
Approximate
Sumple
Ferari
2024g
Approximate
Feature
Fast Retraining WWnne ot
2024
Exact
Passive Party
SecureCat (La al.
2124
Approximate
Instance & Passive Party
Constraint Imposing Deng ct
2023
Vertical
Approximate
Passive Party
Backdoor Centification
Han al
2025
Approximate
Passive Party
Approximate
Label
Table 8: Comparison our method with existing studies on Federated Unlearning. This table demonstrates that method is the only one that ensures privacy protection for D. during the unlearning process. To the best of our knowledge, it is also the first approach to tackle label unlearning in VFL while safeguarding the unlearned label throughout the process.
A.7 RELATED WORKS
Table compares our method with existing studies on Federated Unlearning. Most methods target Horizontal Federated Learning (HFL) and focus Client unlearning. A smaller set of methods addresses Vertical Federated Learning (VFL), which typically involves specific unlearning for passive parties. Privacy protection for unlearned data is generally lacking, shown by the prevalence of X across methods. Notably, uniquely offers protection for unlearned data, setting it apart from previous approaches.
A.8 EXPERIMENT SETUP
and This section provides detailed information on experimental settings. Tables and 10 summariie the hyper-parameters for our unlearning method. Table [T] summarizes the model name, VFL framework settings, datasets and unlearn labels involved in each unlearning scenario.
Baselines: We implement the baselines with the following details.
Retrain: The dataset DD being divided among K parties and the VFL model retrained from scratch using the same hyperparameters as the baseline.
Fine-Tuning (Golatkar et al. 2020a Jia et al., 2023): The dataset D being divided among K parties and the baseline VFL model is fine-tuned for 5 epochs with learning rate of 0.01.
Fisher Forgetting (Golatkar et al. 2020a): The dataset D" is being divided among K parties, and Fisher information matrix (FIM) being used to inject Gaussian noise to perturb the VFL baseline models towards exact unlearning. For every backward propagation, once the gradient is calculated, each party inject Gaussian noise into their respective VFL baseline model to perturb the baseline model toward exact unlcarning.
Amnesiac (ravess al.) 2021) The dataset is divided among K parties and the baseline VFL model retrained for 3 epochs, with the active party relabeling y" to an incorrect random label.
Unsir (Tarun et al. y The dataset is being divided among K parties. Each passive party introduces a noise matrix to the D,, image features they own. The noise added image features are used impair each VFL baseline model, which is then repaired using the clean Dr.
23

---

Published as a conference paper at ICLR 2026
Toym
Tounti DONT
Enerty FIXTI Tranter PRnnECTTOOTT Tran
Tramte VOIET Virte WETTH
V-1
0-2
3-1
1x3
5-7
po Pumgle
5-4
064
5.4
Table 9: Hyperparameters used for single-label unlearning in our proposed solution. In this single- label unlearning scenario, all datasets processed using the same optimization method, stochastic gradient descent (SGD), with consistent parameters: a batch size of 32, a weight decay of Sc-4, and a momentum of 0.9. The recovery rate, which is the learning rate during the recovery phase, matches the unlearning rate across all datasets. Additionally, minimal number of unlearning epochs (a maximum of 30) and unlearning data samples (DP.", capped at 40) are utilized. Across all experiments, the number of recovery data samples (DP.r) used in the recovery phase is uniformly set to 3.
Hyper-paramders
Remel (CCIAAROO
Twc-lihe
Vec16-CITARIO
Veg16-Cifr100
Reseel SenwlTF-CTPARE CIFARIOO
Moli-aba
Optemizztion Maliod
Scils
S010
SCI
Stil
SC3D
SCD
Unleurning Rate
Ie-6
9e-7
le-6
9--7
3
Ie-6
Recovery
3e-6
lc-6
Unleaming Fgechs.
17
Number Dav Samples
Nasterr De Samples
Batch Size
64
Weight Decay
5e-4
3e-4
3e-4
56-4
Se-4
Se-4
Momentum
0.9
0.9
0.9
0.9
0.9
0.9
Table 10: Hyperparameters used for two-label and multi-label unlearning in our proposed solution. In these unlearning scenarios, all datasets are processed using the same optimization method, stochastic gradient descent (SGD), with consistent parameters: weight decay of 5e-4, and a momentum of 0.9, For the two-label unlearning scenario, a batch size of 32 is used, while a batch size of 64 is employed for the multi-label unlearning scenario. The recovery rate, which is the learning rate during the recovery phase, matches the unlearning rate across all datasets. Additionally, a minimal number of unlearning epochs (a maximum of 25) and unlearning data samples (DP.", capped at 40) are utilized. Across all experiments, the number of recovery data samples (DP-") used in the recovery phase is uniformly set to 3.
The dataset D., is being divided among K parties. Passive parties create adversarial examples from the D, image features they own, and the active party assigns a new nearest incorrect adversarial label to shrink the D, to the nearest incorrect decision boundary.
Boundary Unlearning (Chen et al. 2023):
SSD (Foster et al. 2024b): The dataset is divided among K parties, and each party modifies their local model weights with the gradients full data and D..
Datasets: We conduct experiments seven widely used datasets: MNIST, CIFARIO, CIFAR100, ModelNet. Brain Tumor MRI, COVID-19 Radiography and Yahoo Answers. For the MNIST, CIFAR10/100, Brain Tumor MRI and COVID-19 Radiography datasets. For all datasets, except ModelNet and Yahoo Answer, each image split into two equal feature segments, with each segment assigned to different passive party. For the ModelNet dataset, two 2D multi-view images are rendered for each 3D mesh model by placing virtual cameras at evenly spaced positions around the model's centroid. Each passive party then assigned one of these rendered view. For Yahoo Answers dataset, each sample (i.e. a single paragraph of text) is splitted into two segments, with each passive party receiving one segment, ensuring that no single party has access to the complete text.
Figures 7 and [ illusttraee how image features of the dataset are being split among the passive party. In this setup, we consider a scenario involving two passive parties. For all image datasets, except ModelNet, each image's 's features are evenly split into two halves, as illustrated in Fig. 7 As an example in CIFARIO (i.e., class bird), each passive party is assigned one half of the image (bird) features. This means that neither party has access to the full set of image features for any given data point, ensuring that the data remains fragmented and partially hidden from each individual party. This approach is specifically designed to enhance privacy and security by ensuring that no single party has access to the full information of the original image. This partitioning enables collaborative computation machine learning tasks while maintaining data confidentiality.
24

---

Published as a conference paper at ICLR 2026
Scenanios
Midels ||
Model of Plasive Party
Micdel of Active Pany
Dicasets
Unieam Labels
ReeNetKK
30 Coew
TR:
CIIST, CHEAI (D) MerkeiSnn, CTV IN Radingophy
Single-lahl Untearinng
ResNet18
Vapis
29 Cony
13 Cory
IFC
RCC
CIFARIO, CIPARIOD
Brain Tumoc MIKI
Minfesi
13 Bei
4
Yalson Anewer
Twp lobel Uniearning
ScaNe118
Vepla
Cony
1R:
CIFARIO, CIPARIOO
CIEARIO
0.2
Mulii Uatearning
Vagto
Core
FC
CIFARIO
CHARWN
0.2.3.
0.2.5.7
Table 11: Models and datasets involved in each unlearning scenarios. FC: Fully-connected layer. Conv: Convolutional layer. Brt: Bert layer. In the single-label unlearning scenario, we perform un- learning for label "0" on the MNIST. CIFARIO, C00,R100, ModelNet, and COVID-19 Radiography datasets using the ResNet18 architecture. Additionally, label "0" is unlearned on the CIFARI 10 and CIFAR 100 datasets using the Vgg 16 architecture. For the Brain Tumor MRI dataset, we unlearn label "2" using the ResNet18 architecture. Similarly, on the Yahoo Answer dataset, label "6" is unlearned using the MixText architecture. In the two-label unlearning scenario, labels "0" and "2" unlearned across all datasets and architectures. In the multi-label unlearning scenario, extend the unlearning process to include labels "0", "2", "5", and "7" across all datasets and architectures.
For the ModelNet dataset, a different approach is used to distribute data between the two passive parties illustrated in Fig. [ Specifically, each 3D mesh model is rendered into two distinct 21) images, each capturing a unique view of the object from different perspectives. Then, these two generated 2D views are assigned separate passive parties, with each party receiving one view exclusively. As a result, no single party has access to both perspectives of the 3D mesh model, preserving information separation. This strategy not only safeguards data privacy but also enables collaborative analysis or training, ensuring that each party's 's contribution remains independent and incomplete.
Figure 9 iillustrates how text features from the Yahoo Answers dataset are split between the passive parties. this process, each paragraph is divided into two separate segments, with each segment representing portion of the original text. Each passive party receives access to only one of these segments, ensuring that no single party can view the entire paragraph. This approach helps preserve the confidentiality of the dataset.
MNIST is a dataset of handwritten digits. MNIST dataset comprises 60,000 training examples and 10,000 test examples. Each example is represented single-channel image with dimensions of 28x28 pixels, categorized into one of 10 labels.
CIFAR10/100 dataset contain 60,000 images x32x32 pixels, three color channels). CIFARIO dataset includes 10 labels, with 50,000 for training and 10,000 for testing. The CIFAR100 dataset is similar but has 100 labels, each with 600 images, divided into 500 for training and 100 for testing.
ModelNet dataset is a widely-used 3D shape classification and retrieval benchmark, which contains 127,915 3D CAD models from 662 object categories.
Brain Tumor MRT is commonly used in healthcare scenarios. The Brain Tumor MRT dataset consists of 7,023 human brain MRI images categorized into four labels: Glioma, Meningioma, No Tumor, and Pituitary.
publicly available dataset on Kaggle, designed for research and development in medical imaging, specifically for detecting COVID-19 through chest X-rays. This dataset consists of chest X-ray images categorized into four classes, which are COVID, Normal, Viral Pneumonia and Lung Opacity.
COVID-19 Radiography
Yahoo Answers is a dataset designed for text classification tasks, comprising 10 labels (topics) such as "Soociet & Culture", "Science & Mathematics", "Health", "Education & Reference", among others. Each label contains 140,000 training samples and 6,000 testing samples. For simplicity, we utilized 5,000 training samples and 2,000 testing samples from each label.
A.9 LIMITATIONS
As the first method for label unlearning in VFL, our approach is developed under several standard and clearly defined assumptions. First, we assume access to a small public dataset that shares the same label space the private data. This is assumption commonly adopted in VFL and
25

---

Published as a conference paper at ICLR 2026
Passive Party A
Passive Party
Figure 7: Illustration of how features from the CIFAR-IO dataset are split between passive parties. Passive Party A has access to segment of the image, such the left half of the original image's 's pixel data, while Passive Party B has access to the other segment, such as the right half. This division ensures that no single party has access to the complete image.
Proisc Parly
Prniiv Paryy
Figure 8: Illustration of how features from the ModelNet dataset are split between passive parties. Passive Party A is assigned one view of the 3D model, rendered from specific angle (e.g. an upper facinn perspective of the airplane model), while Passive Party receives a different view the same model, rendered from another angle (e.g.. an upper right-side perspective). This setup ensures that no single party has full access to the complete 3D information.
can loan corpretion gavish jour wages?
yes they can read your cntracts very carefilly before sigring also k Passive Party A
nou your rights as consuner in your state go to your states welsite and search for the foonuuet rights section.
Passive Party B
Figure 9: Illustration of how text from the Yahoo Answers dataset is split between passive parties. Each passive party is assigned different portion of the text, ensuring that no single party has access to the complete content. For example, the full paragraph is: can loan corporation garnish your wages? yes they can read your contracts very carefully before signing also know your rights consumer in your state go to your states website and search for the consumer rights section." " Passive Party A receives the first segment (e.g., a loan corporation garnish your wages? yes they read your contracts very carefully before signing also k"), while Passive Party B receives the remaining portion (e.g., "now your rights as consumer your state go your states website and search for the consumer rights section."). This division maintains the confidentiality of the full paragraph.
vertical unlearning settings. When this label-space alignment holds, both our theoretical analysis and empirical results show that even small number of public samples sufficient to produce well-aligned gradient directions for effective unlearning. In extreme scenarios where the public set
26

---

Published as a conference paper at ICLR 2026
mislabeled semantically unrelated, the public gradients may longer reflect the true class boundary. Exploring more robust substitutes, such as pseudo-labels or synthetic anchor samples, represents interesting direction for future work, particularly in sensitive domains like healthcare or finance, where suitable public analogues may be scarce.
Second, experiments follow the synchronous VFL setting, which is the standard operational mode in existing vertical FL. systems and prior vertical unlearning studies. While our method is designed for this synchronous setting, extending asynchronous environments would require additional mechanisms for handling stale embeddings and misaligned updates.
Third, our method focuses one-shot label unlearning rather than long sequences of deletion requests. In scenarios with many successive unlearning operations, accumulated updates may gradually shift the model.
Finally, our threat model assumes honest-but-curne participants and does not consider adversarial behaviors such intentionally corrupted embeddings or poisoned gradients.
27


Reviews:
[{"review_id": "DhNB0t5XUz", "reviewer": "Reviewer_LVms", "summary": "The paper looks into label unlearning in vertical federated learning (VFL). Standard retraining or boundary-unlearning methods leak which labels were deleted to passive parties. The proposed method uses a few-shot public dataset (≤40 samples per label), applies manifold mixup at the embedding level across passives, runs gradient ascent on mixed embeddings to push models away from forgotten labels, then performs small descent on retained labels for recovery.", "questions": "Weaknesses and Questions merged", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) VFL unlearning seems a rather underexplored area in machine learning so targets a relative problem. \n2)  Operates on concatenated embeddings with tiny public sets (≤40 samples), the extension of manifold mixing is very interesting.", "weaknesses": "- The claims about no deletion guarantees are fine and measured so while some questions can be raised about the applicability of the algorithm (especially how diverse the features are  with every client and how does that impact unlearning) I am hesitant to raise those questions given my unfamiliarity of the recent challenges in VFL.\n\n- However,  my main concern remained here as to what extent the features of the few shot labels  are correlated with the features of the unlearned labels ( is there a sufficient bound of alignment of features of the few shot for effective unlearning). If there is weak correlation the algorithm would fail right? since the method relies that the \"public\" labels provide a valid direction. If the method would still succeed then that would strengthen then that would be very good. Would appreciate if the authors can provide some details", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZkMCrxtcog", "reviewer": "Reviewer_Dw3V", "summary": "This paper studies the problem of machine unlearning in the setting of vertical federated learning, where features and labels are owned by 2 different parties, and only a set of labels need to be unlearned. The proposed unlearning method consists of 3 steps: 1) use manifold mixup to generate sufficiently many samples for the unlearning algorithm; 2) run a vertical gradient ascent algorithm; 3)recover the accuracy on the remained data by re-training on these samples. Experiments are conducted to show the effectiveness of the proposed method.", "questions": "The paper's narrative structure is disjointed. Section 3.2, which analyzes label leakage, is placed before the proposed method is introduced in Section 4. This is confusing, as it discusses the privacy properties of an algorithm the reader has not yet seen. This premature justification breaks the logical flow of the paper and makes the draft difficult to follow. The content requires polishing to present the problem, the proposed solution, and then the analysis in a more coherent order.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "- the studied problem is important\n- experiment results look promising\n- compared to a previous draft in Neurips 2025, a new SSD baseline is included in the experiments, which strengthens the experimental evaluation .", "weaknesses": "My primary and most significant concern with this work, which I also raised as a reviewer for a previous submission of this paper to NeurIPS 2025, is a fundamental ambiguity in the threat model. This core issue, which remains unaddressed, makes the paper's central motivation and claims inconsistent. \n\nIn Section 3.2, the authors tried to argue that retraining leaks 100% membership information. But for standard machine unlearning, retraining always serves as a baseline, as the goal of unlearning is just to produce a model as if the unlearned data is never used. It appears the authors are implicitly redefining the unlearning goal: not only must the model be unlearned, but the passive party must also be kept ignorant of which data is being unlearned. This is a \"process privacy\" goal, which is a valid but very different objective from standard unlearning. This new goal is never rigorously defined.\n\nEven if we accept the authors' new \"process privacy\" goal, they provide no evidence that their own method achieves it. In the propose method gradients are shared with the passive party. Gradients are well-known to leak information about the data they were computed on. The authors provide no proof or analysis that these gradients are any safer than an explicit unlearning request. They simply assume their method is private.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MFV7cQwRNm", "reviewer": "Reviewer_T6TS", "summary": "This paper tackles the underexplored problem of label unlearning in Vertical Federated Learning (VFL), where the goal is to erase the influence of specific labels while maintaining privacy between active (label-holding) and passive (feature-holding) parties. The authors first identify a privacy leakage risk in existing retraining-based and boundary-based unlearning methods, which require explicit identification of samples tied to deleted labels. To address this, the paper introduces a few-shot label unlearning framework that re-purposes manifold mixup as a privacy-preserving embedding transformation, enabling both label removal and privacy protection without direct label disclosure. A recovery phase is added to restore performance on the retained data. Experiments on multiple datasets demonstrate that the proposed method achieves competitive unlearning and runtime efficiency while reducing label leakage compared to baselines.", "questions": "Please see the weakness section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Novel problem formulation.** The work clearly identifies a meaningful and underexplored setting, label unlearning in VFL. This is a distinct and practically relevant direction compared to the more common client or sample unlearning tasks.\n\n- **Clear motivation and privacy analysis.** The authors systematically highlight the privacy risks of conventional unlearning in VFL, showing that retraining-based approaches lead to full label leakage. This analysis is well-motivated and establishes a strong need for privacy-aware unlearning mechanisms.", "weaknesses": "- **Limited algorithmic novelty and depth.** The method primarily combines two known ideas, manifold mixup and gradient ascent unlearning, without introducing fundamentally new mechanisms for privacy preservation or optimization. The theoretical component (Theorem 1) formalizes intuitive gradient alignment but does not establish strong guarantees of privacy or unlearning completeness.\n\n- **Lack of formal privacy guarantees.** Although the paper claims “privacy-guaranteed” unlearning, there is no rigorous privacy analysis (e.g., differential privacy bounds or information-theoretic leakage quantification). The argument relies on heuristic intuition and empirical leakage reduction, which does not substantiate the “guarantee” claim in the title.\n\n- **Reliance on public data undermines practicality.** The approach assumes access to a small public labeled dataset that contains the same label categories as the private data. This assumption is unrealistic in many sensitive domains (e.g., healthcare or finance), where such public analogs may not exist.\n\n- **Overstated experimental superiority.** Although results show good accuracy retention and unlearning effectiveness, the differences from SSD or Amnesiac baselines are not statistically or conceptually large. For example, in several datasets (Brain MRI, COVID Radiography), the proposed method’s ASR improvements are marginal relative to simpler baselines.\n\n- **Insufficient clarity on communication protocol and security.** The paper claims privacy preservation by only exchanging gradients on mixed embeddings, but lacks detailed analysis of what information could still be inferred from these gradients. Without a formal adversary model or leakage bound, the privacy benefit remains speculative.\n\n- **Over-extended empirical scope with limited insight.** While the paper reports numerous datasets and metrics, it lacks deeper diagnostic analyses, such as ablation on mixup parameters (λ), effect of public data size, or visualization of embedding disentanglement that could substantiate the mechanism’s behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnWaT1pdqg", "reviewer": "Reviewer_XuNo", "summary": "This paper presents a pioneering investigation into the critical yet under-explored problem of label unlearning in Vertical Federated Learning (VFL). To address the severe privacy leakage risks of directly applying existing unlearning methods in VFL, the authors propose a novel few-shot, privacy-guaranteed unlearning framework. The core innovation lies in repurposing Manifold Mixup as a privacy mechanism to generate augmented embeddings that disguise the original label information. This is followed by a gradient-ascent-based forgetting step on these embeddings and a recovery phase to maintain model utility on the retained data. Extensive experiments across diverse image and text datasets demonstrate that the method achieves effective unlearning, strong utility preservation, and significant computational efficiency, establishing a new, practical direction for machine unlearning in VFL.", "questions": "1. How would the method perform if the initial model is not well-converged (i.e., the training loss is not near zero)? What adaptations are needed to ensure robustness under such non-ideal conditions?\n\n2. Can the proposed framework be effectively adapted to function in an asynchronous VFL environment? What modifications to the protocol would be required to maintain its efficacy and privacy guarantees under such conditions?\n\n3. How sensitive is the unlearning performance to key hyperparameters, particularly the Mixup coefficient (λ)? Is there an optimal strategy for selecting or adapting λ during the unlearning process?\n\n4. How can the framework be fortified against a malicious adversary who intentionally submits corrupted embeddings or gradients to prevent unlearning or degrade the global model's performance?\n\n5. How does the method scale with an increasing number of labels to be unlearned sequentially? Does it suffer from catastrophic forgetting of retained knowledge or cumulative performance degradation over multiple unlearning requests?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This is the first work to systematically define and address the challenge of label unlearning within the VFL paradigm.\n\n2. The proposed method successfully removes the influence of target labels from both active and passive models without disclosing the identity of the labels to be forgotten, thereby mitigating privacy leakage risks inherent in unlearning approaches.\n\n3. The method's efficacy is validated on a wide range of datasets, including images and text. This proves it is a modality-agnostic framework that operates effectively on embedding spaces, enhancing its practical applicability.\n\n4. The proposed unlearning process is efficient, which demonstrates a 16x to 1200x speedup compared to the closest baseline and traditional retraining methods, respectively.", "weaknesses": "1. The theoretical guarantee of unlearning effectiveness relies on the strong assumption that the training loss is near zero, which may not hold perfectly in practice. Furthermore, the method's requirement for a small set of labeled public data might be difficult to fulfill in strict privacy scenarios where no such data can be disclosed.\n\n2. The experimental validation is conducted in a synchronous VFL environment. Its effectiveness and stability in asynchronous VFL settings remain unexplored and pose a potential limitation for practical deployment.\n\n3. The paper does not provide a comprehensive sensitivity analysis for key hyperparameters, such as the distribution of the Mixup coefficient (λ). The performance robustness under different hyperparameter choices is therefore not fully understood.\n\n4. The security analysis is confined to the semi-honest adversary model. The proposed method lacks defensive mechanisms against malicious adversaries who could potentially sabotage the unlearning process by poisoning local embeddings or gradients.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)