Subspace based Federated Unlearning
Authors: Guanghao Li
Li Shen
Yan Sun
Yue Hu
Han Hu
Dacheng Tao
Venue: Accepted by TMLR ()

Abstract: Federated learning (FL) enables collaborative machine learning among multiple clients while preserving user data privacy by preventing the exchange of local data. However, when users request to leave the FL system, the trained FL model may still retain information about their contributions. To comply with the right to be forgotten, federated unlearning has been proposed, which aims to remove a designated client's influence from the FL model. Existing federated unlearning methods typically rely on storing historical parameter updates, which may be impractical in resource-constrained FL settings. In this paper, we propose a Subspace-based Federated Unlearning method (SFU) that addresses this challenge without requiring additional storage. SFU updates the model via gradient ascent constrained within a subspace, specifically the orthogonal complement of the gradient descent directions derived from the remaining clients. By projecting the ascending gradient of the target client onto this subspace, SFU can mitigate the contribution of the target client while maintaining model performance on the remaining clients. SFU is communication-efficient, requiring only one round of local training per client to transmit gradient information to the server for model updates. Extensive empirical evaluations on multiple datasets demonstrate that SFU achieves competitive unlearning performance while preserving model utility. Compared to representative baseline methods, SFU consistently shows promising results under various experimental settings.

Han Hu Published in Transactions on Machine Learning Research (12/2025)
Subspace based Federated Unlearning
Guanghao Li
ligh24 @maalss wswngghs...iigaaa..... edu.ccm
Tsinghua Shenzhen International Graduate School, Tsinghua University College of Engineering, Southern University of Science and Technology
Li Shen
mathshendU@@maillccom com
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University
Yan Sun
yamn8899@unni..yydeee.......
School of Computer Science, Faculty of Engineering, The University of Sydney
Yue Hu
huyue11 @rrutt edu.
College of Systems Engineering, National University of Defense Technology State Key Laboratory of Digital Intelligent Modeling and Simulation
hhu@bit, edu.cn
School of Information and Electronics, Beifing Institute of Technology
Dacheng Tao
dacheng. too @ggail com
College of Computing and Data Science, Naryang Technological University
Reviewed on OpenReview: https: // openreview.w... net/ forum? KE2ZN
Abstract
Federated learning (FL) enables collaborative machine learning among multiple clients while preserving user data privacy by preventing the exchange of local data. However, when users request to leave the FL system, the trained FL model may still retain information about their contributions. To comply with the right to be forgotten, federated unlearning has been proposed, which aims to remove a designated client's influence from the FL. model. Exist- ing federated unlearning methods typically rely on storing historical parameter updates, which may be impractical in resource-constraanee, FL settings. In this paper, we propose a Subspace-basee based Federated Unlearning method (SFU) that addresses this challenge without requiring additional storage. SFU updates the model via gradient ascent constrainod within subspace, specifically the orthogonal complement of the gradient descent directions de- rived from the remaining clients. By projecting the ascending gradient of the target client onto this subspace, SFU can mitigate the contribution of the target client while maintain- requiring
ing model performance on the remaining clients. SFU is communiction--fficion only one round of local training per client to transmit gradient information to the server for model updates. Extensive empirical ovaluations on multiple datasets demonstrate that SFU achieves competitive unlearning performance while preserving model utility. Compared to representative baseline methods, SFU consistently shows promising results under various experimental settings.
1
Introduction
The traditional training approach of deep learning typically aggregates data from various participants. How- ever, certain data, such as medical records 31,, cannot be relocated from the hospital due to concerns regarding data privacy and individual preferences. response, Federated Learning (FL) P5
"Corresponding author

---

Published in Transactions on Machine Learning Research (12/2025)
emerges a prominent decentralized machine learning solution for addressing these challenges. FL facili- tates the training of a global model by exchanging model parameters between clients and a central server, effectively bypassing the need to transfer the raw data 23:
28
Recent privacy legislations 5 35: 40 provide data owners the right to be forgotten. In the context of machine learning, this right necessitates two key actions: (i) deletion of user data from the storing entity and (ii) removal of the data's influence on the model [5 Within the realm of Federated Learning (FL), federated unlearning crystallizes the embodiment of the right to be forgotten. However, achieving machine unlearning within the framework of federated learning presents heightened challenges: (1) Limited Data Access: In FL, the server lacks direct access to all data and associated operations, rendering forgetfulness (2) Multi-client Participation: The initial model of each client in every training round depends on aggregating models from clients engaged in prior-round training, resulting in the gradual propagation of effects from individual data samples across Thus, crasing data samples from one client requires substantial number of clients to engage in retraining process.
techniques reliant on dataset segmentation inapplicable to FL scenarios.
models used for local training at multiple clients 34
33
As mentioned above, retraining in FL demands a significant number of clients to participate in local training, inevitably leading to extended training durations. Some recent endeavors have been focused on addressing this challenge, such as storing the client's historical updated gradient data on the server and utilizing it to revert the trained global model 46 30, However, these methods necessitate either the client or the server to retain additional data or gradient information, which may not be practical in FL scenarios with limited storage resources. There are alternative approaches execute the unlearning process by directly modifying the final model. For instance, directly employing gradient ascent on the target client can achieve the immediate reduction of client data influence in the final model. Nevertheless, this approach may considerably compromise the model's performance.
In this paper, we focus on developing a practical approach for implementing federated unlearning within the final model. We consider unlearning as the inverse process of learning via gradient ascent. However, the loss function unbounded, requiring constraints on the gradient to preserve model quality [8] G. Consequently, we treat the entire process as a constraint-solving problem, aiming to maximize the empirical loss of the target client while maintaining acceptable model performance for other clients. Introducing updates orthogonal to the gradient directions of neural network predictions can induce minimal changes in network output [T]] Building on this idea, we propose a Subspace- based Federated Unlearning method, termed SFU.
In the SFU framework, the server only requires the ascending gradient information from the target client and the descending gradient information from the remaining clients. The server then projects the ascending gradient onto the orthogonal subspace of the descending gradient space. This constrained gradient used to update the final trained FL global model, aiming to reduce the contribution of the target client while preserving the model's utility. Fig. iilustrates the core idea of our approach.
Specifically, participants in the SFU process can be categorized into three roles: the target client to be forgotten, the remaining clients, and the server. The target client performs local gradient ascent based on the global model and sends the gradient to the server. Each remaining client performs gradient descent and transmits their respective gradients to the server. The server aggregates the descending gradients and derives the gradient space through Singular Value Decomposition (SVD) 16 37. Finally, the server projects the ascending gradient onto the orthogonal complement of the others' gradient subspace and updates the global model. The SFU framework only requires one round of local training per client and a subsequent server update, without the need to access each client's raw data or store historical gradicnts. Empírical results indicate that SFU achieves competitive unlearning performance across various datasets while maintaining reasonable model accuracy.
In conclusion, our main contributions are as follows:
We incorporate subspace learning into federated unlearning and propose a novel algorithm named SFU, which performs gradient ascent within a subspace that is orthogonal to the gradient space of the remaining clients, thereby explicitly constraining the unlearning update to reduce collateral damage.
2

---

Published in Transactions on Machine Learning Research (12/2025)
Figure 1: An illustration of SFU with three clients. wi and w2 represent the optimal model parameters for client and client 2, respectively. The ascending gradient of the target client is denoted as Additionally, and correspond to the descending gradients originating from client and client The projection of g onto the orthogonal space spanned by I1 and is denoted as ÿ. Operating within this orthogonal subspace ensures minimal perturbation to the model's performance on client and client 2.
SFU achieves effective unlearning while maintaining acceptable utility on the remaining clients, and it does so without requiring additional storage of historical client updates or intermediate checkpoints, leading to no extra storage overhead.
We conduct comprehensive experiments to evaluate SFU under diffcrent training stages and system conditions (e.g., varying client participation rates and target-client data volumes), demonstrating that SFU achieves competitive performance across multiple datasets, including MNIST, CIFAR1O, and CIFAR100.
2 Related Work
Machine unlcarning. The concept of "machine unlearning" entails the complete removal of a specific training data instance, necessitating the nullification ifs impact on both extracted features and models. The introduction of machine unlearning is attributed to Cao & Yang [7], who reformulate statistical query learning into a summation form and achieve unlearning by selectively updating subset of the summation. However, this algorithm is confined to transformable traditional machine-learning methods, prompting exploration into machine unlearning for various ML models. Ginart et al. 12 formalize the notion of effective data deletion in machine learning and propose two efficient deletion strategies for the K-means clustering algorithm. In the realm of supervised linear regression, Izzo al. [17] develop the projective residual update (PRU) for linear and logistic models. While the computational cost of PRU scales linearly with feature dimensions, its suitability is limited for more intricate models such neural networks. To address the overhead of forgetting, Bourtoule et al. 4 introduce the versatile SISA framework, which trains disjoint sub-models on data shards and only retrains the affected shard after deletion request. A recent survey [1]] provides comprehensive overview of progress, but most studies still assume centrally accessible data, which is ill-suited for federated settings.
Federated unlearning. Current work can be grouped into two main directions. Retraining-efficien methods. Liu et al. [30] cache historical gradients to accelerate retraining; Su & Li 42 cluster clients and retrain only within the group containing the target client, which is useful in highly heterogeneous scenarios. One-shot model modification. Wu et al. 4o subtract stored gradients of the target client from the final model, still requiring extra storage. Wang et al. 45 prune task-specific weights, but only support class-level forgetting. Very recently, NoT 20 introduces a storago-frce weight-negation stratogy that supports client-, class-, and instance-wise unlearning without retraining. The definition of exact. federated unlearning was formalized in FATS E0l which provides a TV-stable FedAvg variant and provable closed-form unlearning step equivalent to retraining-but it requires server-side batch data and complex aggregation schemes.
Our approach is closely related to UPGA 15 and EWC-SAG 47. All three perform gradient ascent on the final model to remove a client's influence, but differ in their constraints: UPGA constrains the magnitude of the update with an l2 ball; EWC-SAG employs an clastic weight ponalty; in contrast, SFU constrains the direction of the update to the orthogonal complement of the remaining clients' descent subspace, climinating the need for historical checkpoints while better preserving accuracy.
3

---

Published in Transactions on Machine Learning Research (12/2025)
3 Method
We propose Subspace-bused Federated Unlearning (SFU), summarized in Algorithm | SFU leverages con- strained gradient information from the target client to adapt the final trained model, effectively removing client contributions while upholding model performance across other clients. Notably, this method eliminates the need for the server to retain a historical record of parameter updates from individual clients and obviates the necessity for exlensive retraining.
3.1 Problem Setup
Suppose there are N clients, denoted as CI,..., CN. Let [N] fI,...,Ny. Client C2 possesses a local dataset D' The objective of conventional Federated Learning (FL) is to collaboratively train a machine learning model w over the combined dataset - Uie[N] D
N
D D
w² A arg min L(w) 21
Σ
L.Ww,,
(1)
where [,(w)= = ((ew)) (w: (2. v))] represents the empirical loss of client C.. Throughout the federated training process, each client minimizes their respective empirical risk L,(w). The model w the final outcome of the FL procedure.
Now let's delve into the strategy for eliminating the contribution of the target client C₁. An intuitive approach is to escalate the empirical risk (w) associated with the target client Ct, essentially reversing the learning process. However, straightforward maximization ion of the loss will impact the model's performance on other clients. Federated unlearning needs to forget the contribution of the target client C₁ while ensuring the overall model performance. Consequently, federated unlcarning can be formulated as follows:
arg max Lr(W) = (e (w; v))
(2)
s.l. E(w²) - E(w) 5
Here, represents tolerable difference in model performance, while E(w) significs the accuracy of model w evaluated on the remaining clients within the FL system. Prior work by Halimi al. [ utilized the parameter distance between and w" as constraint, although this parameter distance might not fully capture the performance disparity among different models. Conversely, the constraint presented in Eq. 2 effectively addresses this concern.
3.2 subssace-baseed Federated Unlearning (SFU)
When applying the ascending gradient update of the target client to the global model without considering other clients, there is high probability that the neural network will turn into stochastic model. SFU restricts model updates to orthogonal subspaces aligned with the gradients of other clients to mitigate this issue. This approach achieves the goal of forgetting the target client's contribution while minimizing potential disruptions to model performance for other clients. The training process SFU is depicted in Fig. 2 Participants in SFU encompass the target client, remaining clients, and the server. The target client employs gradient ascent and transmits the resulting gradient the server. Other clients engage in gradient descent. and forward their descending gradients to the server. The server calculates the gradient space of the other clients and performs unlearning updates the global model. Subsequently, will provide detailed explanation of the SFU process.
3.2.1 Local training on clients
Satisfying the constraints outlined Eq. necessitates imposing restrictions on the ascending gradient. Drawing inspiration from Fig. [ it becomes evident that moving orthogonally the gradient space yields the least impact (or even negligible change locally) on the FI. model's performance for clients. This valuable
4

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm Subspace-based Federated Unlearning (SFU)
Input: FL global model w", local dataset Di client i, learning rate
1: Target client Cy:
91 è- VLy (w")
Send gi the server
Other clients:
for each client i I do
6: - VL,(w")
7: Send 9: to the server
Server: Let Inthers S [N]] (7) denote participating others, S [others
for each network layer 1 = 1, .,,,..., L do
10: R' + [voc(g)) liETath... E *
11: Perform SVD on R' = ('
12: Choose the smallest. l s.t. | R
1
2 I|
and let U = [uq....,
13: Projection matrix p (U))
14: P+ - diag(P1, ,..,,,,, Pb)
15: 9P (I-P)
16: Wal f- w² 19P
insight prompts us to project the updated gradient of the target client Cj onto the orthogonal space of the gradient subspace associated with other clients [0]. This process requires both the ascending gradient from the target client and the descending gradients from the other clients. Hence, upon receiving a forget request from the target client, the training procedures for the target client and the other clients are follows:
Target Client: The larget client performs gradient ascent on its local loss. To implement ascent with standard descent machinery, we minimize the surrogate Ly'(w) (w) == 1/Lr(w). Concretely, during unlearning we evaluate at the final FL model w" and compute a mini-batch stochastic gradicnt 9r == V (w*), so 9r is a negative multiple (L(("*) and descending along 91 increases L1. Hence the local objective for the target client is to minimize T.p(()) (w). The target client then transmits the gradient information gr the server.
The remaining available clients follow the same gradient descend procedure as in standard FL; that is, each minimizes its local objective L,ww). During the unlearning round, for each i o I we evaluate at the final FL model w" a mini-batch stochastic gradient gi = (L:(w"), computed client i's data. Each such client then transmits to the server.
Other Clients:
3.2.2 Computation projection matrix
When the server aggregates the gradient information sent by cach client, it proceeds to compute the projection matrix of descending gradient space for each layer of the model. This process involves two key steps:
Let L denote the total number network layers. For each layer 1, the per-client gradient can be viewed matrix of E mm Let vec(g]) E R denote its vectorization (with ri = mym). To form the gradient subspace at layer the server column-stacks the vectorized gradients from the available non-target clients Iothers S [N]] \ {1}, yielding R² = [vec(g)) I. E x where S == [others
Next, compute the SVD of Re,, i..,, R (V)). Based on a user-specified coverage coefficient ef E (0, 1]. choose the smallest io and form the rank- approximation R == (V)), where U = [u]...., u/]l Σ = .iag, ,of), and V/d = [of,...,.d) v/), such that
| 2 é [
(3)
We then take the first Io left singular vectors to represent the significant subspace layer 1, i.c., spanfuf... uf with Uf as its basis matrix, which captures the directions associated with the largest singular values 88
5

---

Published in Transactions on Machine Learning Research (12/2025)
Gradient
Fr
Gradient
Server
Grudient Suhspace
Gradient
The pipeline of SFU. The entire process occurs subsequent to the training of the FL model. The orange client denotes the target client from which the contribution to be removed; the blue clients represent others. The boxes on the right of the image symbolize global model updates performed on the server.
Figure 2:
Following these steps, the server forms the orthogonal projector onto the others' gradient subspace at layer as p = T with (U%) U% = 4t. Repeating this for all layers yields a. block-diagonal projector over the whole network, p= diag( P ,.,, P*).
3.2.3 Update of the global model on the server
After recciving the client gradients and constructing P, the server updates the global model w". First project the target client's gradient onto the others' gradient subspace, Ør := Pgr, and let the orthogonal component gp = (1 P) The server then updates in the orthogonal complement:
Val w² 1gr,
(4)
where > O is the learning rate. This update removes the target client C1's contribution while limiting interference with directions favored by the remaining clients, thereby preserving overall performance.
Remarks. Subspace methods have been broadly applied continual learning BS meta learning 1 one-
shot learning [8], adversarial training [27, graph forgetting g. and speeding up deep model training 20] To the best of knowledge, SFU is the first to exploit updates in the orthogonal complement. of the remaining clients' gradient subspace for federated unlearning.
3.3 On the effectiveness of SFU
The effectiveness of SFU is twofold: it preserves overall performance the remaining clients while enforcing forgetting for the target client. In this section, analyze its efficacy. Let w, denote the model obtained by retraining from scratch, wilhout the target. client, starting from the same initialization w(0). Let T be the total number of gradient updates (equivalently, the aggregated local epochs) used in FL training. Let wat denote the model returned by a single application SFU to the final FL model w".
Performance preservation. We aim show that ||w,-w,l la is bounded, thereby ruling out pathological deviations. Before presenting the result, we state the following standard assumptions on stochastic gradients, collected in Assumption [ these are commonly used in the theoretical analysis of federated learning 50 29
Assumption 1. The expected squared norm of stochastic gradients uniformly bounded for all w: EIVVL(w)) G² (hence, EEI[L[(w)]
G). The same bound holds for retraining gradients (T))w, where L" = EiE(NNAT To (.w). Moreover, the target-cliend gradient is bounded: El|gr ll G. [D'
Theorem 1. Under Assumption let wha = n² ngr be the model returned by single SFU update, and let wo be the model obtained by retraining from scratch (without the target client) from the same initialization after T updates. Then
I.. ||,, woell2 < (2h/T + 1) G.

---

Published in Transactions on Machine Learning Research (12/2025)
This shows that the deviation between the SFU-updated model and the retrained model is bounded. Proof. Let define w(t), wy(t) as the weight parameters obtained by training from scratch with target client and without target client for t epochs with the same initialization w(0)- = (,) and Wid denote the weight parameters obtained by applying SFU on w*. To help the reader better understand the proof strategy used in this section, we derive the upper bound of (T) - War ll by first expanding the formula into two terms
Wr (T) wwil2 - Bl|l,, (T) - w(T) (T) - warll2-
(6)
First term. By the update rules w,((+1)) (1) V 1." (w) and w((+1) = w(() - V) where [21 L' ("))) = EiENNM ToT Le(w), we have
[1 w((+1)ll2 S - (00
(7)
S wt, - w(t) 2gG..
Then after T iterations, we can bound the difference between two parameters as
R (+) u: nx(T)l2 < 2nTG.
(8)
Second term. Since wal w(T 19r with gp (I P)gı, it follows that l| 119u Thercfore, (T)(T) worll 2| 17 Ellgpll2 s n Elgrrl S nn.
Adding the bound the first term and the bound of the second term we can obtain the final bound (2nlI +n)..
Forgetting of the target client. We now show that SFU enforces forgetting at the target: the update direction -9p is a descent direction for the surrogate loss Lid (equivalently, 9p is an ascent direction for L1).
Theorem 2. Let g1 VLT((w)) (w)) and let P be the orthogonal projector onto the gradient subspace spanned by the other clients. Define ğr P gr and gp == P) Then -gp is descent direction for LTf at w"; in particular, (-gp, 91) -1g 11 < 0.
Proof. For a differentiable function f, vector u is a descent direction at w (u, Vf(w)) S Taking f Lif and using that P is an orthogonal projector, we have gP ğr; hence
(-gP, gi) = gp, gP + ğr)
(10)
-11gp113 (gp, ğr) -1ggp < 0.
Therefore, -gr a descent direction for L w".
Together, the two results show that SFU updates the model in direction that removes (forgets) the target client's contribution while respecting the subspace favored by the remaining clients. They also imply a bounded deviation between the SFU-updated model and the model obtained by full retraining without the target client, thereby guaranteeing that overall performance is preserved within a controlled margin.
4 Experiments
In this section, we empirically assess the effectiveness of SFU across various model architectures on three datasets. Our experimental findings demonstrale that our unlearning strategies can adeptly eliminate the target client's impact on the global model while incurring minimal performance degradation. Furthermore, these stratogics facilitate rapid accuracy recovery within a few training rounds. We commence by outlining the experimental setup and subsequently unveil the evaluation outcomes. For comprehensive experimental settings, please consult. the Appendix.

---

Published in Transactions on Machine Learning Research (12/2025)
For Foy
Retraining
UPGA
UTSS
EWC-SGA
SFC
Dataset
distributionn
nesworic
test nce
atk ncc
test acc
atx acc
test noc
atk noc
test
acc
atk acc
Scstt ncC
atic ACC
test aec
ack ncc
MLP
0.61
88.18
0.10
53.
0.00
901.9
0:21
92.8
0:15
IID
CNN
99.15
99.63
99.21
0.20
51.59
0.0
34.95
0.0
88.9
1.56
98.62
0.06
ResNet
99.56
98.32
98.51
0.51
51.80
0..0
45.00
0.0
80.27
1.24
99.03
0.06
MLP
96.8
99.04
96.95
0.60
81.91
0.14
08.41
0.0
86.60
1.41
89.12
0.25
ANIST
Dir(0.6)
CNN
98.96
99.55
99.01
0.25
14.11
0.0
16.
0.0
85.25
0.0
98.76
0.01
ResNet
99.50
98.23
98.55
0.79
22.20
0.0
51.42
03.0
85.79
0.0
99.39
0.03
MLP
83.54
08.95
95.48
0..66
83.21
0.24
72.96
0.0
86.34
1.88
888 63
0.26
Dir(0.3)
CNN
98. 98.92
98.
0.28
32.41
38.70 76
0.0
86.83
0.0
98.39
0:03
ResNet
99.11
98.39
99.38
0.69
55.67
0.10
12.95
0.0
80.26
1.21
98. 88.88
0.03
MLP
50.80
62.35
60.6
3.77
11.00
0.001
13.75
0.0
201.45
0.088
44.76
0.91
IID
CNN
66, 10
47.14
67.56
4.42
60.91
9:40
10.0
0.0
61.48
1.02
61.68
0.46
ResNet
63.73
38.27
83.98
7.23
76.81
8.56
12:61
0.00
77.53
0.81
78.06
0.46
MLP
47.68
4.81
0.02
11.97
0.0
33.4
0.12
3.67
CIFAR1O
Diz(0.6)
CNN
63.71
39.88
65.83
3.21
64.2
8.31
53.27
0.85
56.27
0.94
61.52
6.43
ResNet
60.79
46.86
81. 61.62
2.49
75.40
7.52
67.44
0.0
71.64
0.94
77.95
2.82
MLP
46.41
54.4
45.51
6.01
13.00
0.01
10:36
0.0
34.73
0.36
45.21
1.53
Dir(0.3)
CNN
61.52
61.84
63.51
4.54
38.
0.08
10:06
0.0
30.007
1.43
54.67
3.72
RexNet
79.91
56.87
79.55
1.59
30.35
0,08
13.07
0.0
65.0
2.31
71.01
1.56
MLP
21.93
67.79
23.42
1.73
9.05
0.01
2.03
0.0
200.11
0.26
22. U1
1.46
IID
CNN
0.48
31.35
1.75
0.0
31.50
1.27
ResNet
58.02
50.38
57.75
0.51
60.03
0.11
3.06
0.0
54.05
0.18
55.71
1.36
MLP
50.89
21.8
0.78
9.72
0.D
1.50
0300
19.12
0.50
CTFAR100
Dir(0.6)
CNN
31.26
47.77
32.06
0.58
23.38
0.02
28.32
0.03
30.88
0.00
31.55
0.12
ReaNet
56.50
19.04
56.38
0.51
53,
0.05
11.83
0.0
30.42
0.12
55.24
0:12
MLP
20.17
$7.93
20.67
1.30
6.94
0.01
1.06
13.0
10.52
0.61
19.2
1.31
Dir(0.3)
CNN
31.61
45.81
92.02
0.65
31.96
0.18
1.0
0.0
30.86
0.20
31.12
2.10
ResNet
50.87
55.70
45.72
1.71
0..0
51.08
54.56
1.92
Table 1: Results of different unlearning methods, We record the attack accuracy as "atk acc,' and "test acc" represents the accuracy metric on the clean test data.
4.1 Experimental Setup
Datasets. We evaluate the performance of SFU using three widely recognized datasets: MNIST 22 CI- FAR1O, and CIFAR100 2T. The datasets exhibit increasing levels of training difficulty from MNIST CIFAR100. Our evaluation encompasses two distinct data distribution scenarios: Independent and Identi- cally Distributed (IID), as well as Non-IID (Non Independent and Identically Distributed). For the Non-IID setting, we adopt the Dirichlet distribution (3): The label distribution each device follows the Dirichlet distribution, where 3 serves as the concentration parameter (3 > 0).
Baselines. Our goal is to achieve the unlearning process by adapting the FL model. To this end, we select three prominent federated unlearning algorithms that directly modify the final FI. model as reference benchmarks: Unlearning via Projected Gradient Ascent (UPGA) [5], UL-Subtraction (ULS) A6] and EWC-SGA 47 Furthermore, we include full retraining from scratch bascline for comparative evaluation.
These methods are chosen because they (i) require multi-epoch retraining of all clients, (ii) do not assume access to auxiliary server-side data, and (iii) represent three different constraint Atrategies-lz-ball, gradient subtraction, and elastic weight regularisation thus providing a diverse yet fair yardstick for SFU.
Unlearning Assessment Approaches. We employ backdoor attacks during the target client's updates to the global model, enabling us to intuitively examine the unlcarning impact through the success rate of model backdoor attack on the unlearned global model. An effective unlearning method should diminish the success rate of such an attack post unlearning. It is worth noting that due model prediction errors, yield attack success rate greater than 0. In our experiments,
even retraining of Halimi et al. [5 and execute the backdoor attack using a "pixel pattern' trigger of size 2 x 2, altering the label to "9" on data with labels other than "9'. Furthermore, we evaluate the model's performance after unlearning using accuracy metrics on untainted test data.
follow the approach
Implementation details. We evaluated three network architectures- MLP, CNN, and ResNet-on the MNIST, CIFAR-10, and CIFAR-100 datasets. These models were implemented using PyToh 30 Our experimental sctup consists of 10 clients, with one designated as the target client. All clients fully participate in each training round. We backdoor 80% of the data on the target client. The hyperparameterr settings for
We follow the notation of Wu et
47 and denote elastic weight consolidation with single ascent gradkent as EWO-SGA.
8

---

Published in Transactions on Machine Learning Research (12/2025)
50
45
40
Accuracy
Accuracy
Method (calor)
35
UPGA
MLS EWC--SAG
30
25
Retraining
Distribution (marker))
1D
20
(ir(0.6)
EWC SAG
DKn(0.3)
15
0.00
0.25
0.50
0.75
1.00
1:25
1.50
1.75
2.00
6
7
8
9
10
11
14 15 16 18 19
Accuracy
Training
Figure 3: MNIST MLP: Accuracy Attack Accu- racy.
Figure 4: Convergence plots for SFU and other baselines on CIFAR-10 (CNN, IID).
each approach were as follows: In the case of SFU, the learning rate was set 0.01, the number of epochs set 1, and mini-batch size of 128 was utilized for gradient ascent the target client. We explored the SVD parameter within the range [0.90 0.99] and selected the optimal value. SFU constructs the gradient space by aggregating gradient information from all other clients. For UPGA and EWC-SGA, the learning rate, the number of epochs, and the mini-batch size were maintained consistent with those of SFU. Additionally, we performed a paramcter search to determine the specific values of their unique parameters. All experiments were run on a single NVIDIA RTX 4090 (24 GB) GPU with 120 GB RAM.
4.2 Main Results
Efficient unlcarning with minimal performance loss. Table summarizes the unlearning performance of SFU and representative baselines across IID datasets and different model architectures. Comparable to full retraining, SFU consistently suppresses the backdoor attack success rate (ASR) to a low level on evaluated benchmarks, indicating effective removal the target client's malicious influence. Although several baselines can also reduce ASR, they frequently achieve this by substantially degrading the clean classification accuracy, reflecting unfavorable utility-unlearning trade-off.For instance, on MNTST with the CNN backbone, UPGA reduces the attack performance but causes accuracy drop of roughly 47 percentage points, whereas SFU mainlains nearly the original utility with only Rl percentage- point decrease. Overall, these results demonstrate that SFU can achieve strong unlearning efficacy while preserving clean performance, delivering a more efficient and practical unlcarning solution under minimal performance loss.
Test Attack accuracy trade-off. Figure plots test accuracy (t) against attack accuracy (L) for all methods MNIST with MLP. Owing to stochasticity, even full retraining yield non-zero attack accuracy. We therefore take the retraining point reference and deem method to achieve successful forgetting if its attack accuracy is at or below the retraining attack accuracy (within small tolerance 8). Equivalently, points lying to the left of the retraining reference satisfy the forgetting requirement, and among those, higher test accuracy is preferable (i.c., the upper-left region). Under this criterion, SFU lies on or near the empirical Pareto frontier: delivers the highest test accuracy while keeping attack accuracy within an acceptable range, indicating a superior balance between accuracy retention and attack mitigation compared with competing baselines.
Robustness Across Different Data Settings. The heterogeneity of data across different clients and the complexity of training tasks can both impact the foderated lcarning process. Table | illustrates the influence of these factors on the unlearning process. It is evident that as the data heterogeneity increases, various unlearning methods lead to more significant performance degradation. For instance, during the transition from IID data to Dir(0.3) data, SFU's performance degradation on the CNN model trained on the CIFAR10 dataset increases from 4. 4.72% to 6.85%. However, this performance change remains smaller than the performance losses observed in other algorithms when facing changes in data heterogeneity. Furthermore,

---

Published in Transactions on Machine Learning Research (12/2025)
Method
Target data
Extra comm.
Srv. storage
Public KD
Server comp.
Time
Retraining
Cround
FL. training O(wwl)
UPGA
[wl
ULS
0
(
w=)
O(hel) + KD(F)
0+F
EWC-SGA
[w]
O(w)) ) (Fisher diag)
SFU (ours)?
(5+1)|ml
per-layer SVD: Ó mx)
Table 2: Resource comparison. Cround: total uplink-downlingg per FL round; T: # retraining rounds; R: # pre-unlcarning rounds kept by ULS (history length); F: KD fino-tuning rounds; |w]: model size (one full model upload or download); S: # participating non-targel clients. Here ry denoles the dimension of the vectorized gradient at layer ky the retained rank, and Ó(-) hides polylogarithmie factors. Time counts only the core unlearning round((); optional KD for ULS is shown as +F. All non-target clients participate each round (retraining). A single collaborative round with S non-target clients (SFU).
as the task complexity shifts from MNIST to CIFAR100, some unlearning methods may become ineffective. For instance, ULS generates a random CNN model on the CIFAR10 dataset, while SFU manages to maintain the model's performance after unlearning. This highlights SFU's superior robustness across different data settings.
Recovery of Model Accuracy After Unlearning. After undergoing different Foderated Unlcarning methods, it is common practice to further fine-tune the unlearned model on the remaining clients to restore its predictive utility. Accordingly, we finc-tuned the CNN models produced by each unlearning method on the remaining clients under the IID CIFAR-10 setting, using the same fine-tuning recipe to ensure a fair comparison. In contrast to these client-only recovery protocols, ULS additionally leverages a small amount of clean server-side data and performs knowledge-disfllatton--as extra supervision beyond the remaining clients. Fig. summarizes the recovery curves on CIFAR-10 (IID). Notably, SFU exhibits a markedly faster and more ellective recovery behavior: it reaches a higher accuracy after only one few retraining rounds, whereas other methods typically require substantially more rounds to approach similar performance. This indicates that SFU yiclds an unlearned model that remains closer to a high-utility solution on the retained data distribution, thereby serving as a stronger initialization for post-unlearning retraining and reducing the overall cost of accuracy recovery.
fine-tuning, which provides
4.3 Resource implications and deployment trade-offs
Table 2 summarizes resource profiles using unified notation: T (retraining rounds), Cround (per-round up- downllik,,, |w] (model size), S (number of participating non-target clients), R (length of stored history), and (KD fine-tuning rounds, if applicable). The Time column counts only the core unlearning round(s); optional post-processing (e.g., KD for UTSS) indicated as +F. Column abbreviations are: Targel data (whether access to the target client's data is required), Extra comm. (additional communication beyond baseline FL, measured in multiples of |w| or Croud); Sru. storage (extra server-side storage), Public KD (whether public-data knowledge distillation is required), and Server comp. (dominant server-side computa- tion).
Comparison summary. Retraining serves as the commmimimmii baseline with T rounds, no extra server storage, and per-round participation of all non-target clients; ULS incurs near-zero extra commu- nication at unlcarning time but requires storing @ @(R/wr) historical updates and typically relics on public dala for knowledge distillation, adding F rounds; UPGA/EWC-SGA require only the target client once (extra communication s (w(), need no other clients online, and impose O[ww])-eeel server compute (EWC uses a Fisher-diagonal surrogate); SFU neither stores history nor relies on public data-its core cost. is a single collaborative round in which the target uploads an ascent gradient and S non-target clients upload descent. gradients, followed by a per-layer (randomized) SVD at the server to project onto the orthogonal complement, with (S+1) extra communication and Ô rik7) compute.
Practical takeaways. Requiring target data at unlearning time is not inherently a disadvantage; it enables client-controlled, fine-grained forgetting (e.g., selecting specific subsets to forget) while keeping server-side
10

---

Published in Transactions on Machine Learning Research (12/2025)
storage zero. SFU is preferable when long-term storage of client histories is disallowed undesirable, and when the target client, can briefly be online with access to its data. In practice, the single-round cost of SFU can be further reduced by sampling a subset of non-target clients (reducing S) and by using rrando//anndmize SVD with an e-coverage critcrion to choose per-layer ranks hy.
4.4 Exploratory Study
Impact of different SVD coefficient c. As dis- cussed in Section 3.2, our approach aims to mitigate the decline in clean accuracy by taking gradient steps orthogonal to the gradient subspace spanned by the remaining clients. This subspace is constructed from bases that approximate the dominant task representa- tions of the remaining clients, and the extent of this approximation is controlled by the threshold hyperpa- rameter e' at layer I, as defined in Eq. Intuitively, determines how much gradient energy is retained when forming the subspace: & smaller e keeps fewer principal directions (a lower-rank subspace), whereas larger preserves more directions and better matches the remaining clients' gradient geometry. In our ex- periments, we maintain the same e for all layers; Fig. 5 further demonstrates a consistent test attack trade-off on CIFAR-10 (IID) across three architectures. Specif- ically, lower value (closer to 0) yields lower- rank retained subspace and leaves larger orthogo- nal complement for the ascent update, which can re- move the target client's influence more aggressively but may induce larger performance shifts on the re- maining clients, resulting in reduced clean accuracy. Conversely, a higher e value (closer to 1) preserves more gradient correlations and thus better maintains clean accuracy by restricting updates to directions less harmful to other clients, but it may not fully eliminate the target client's contribution, leading to increased attack success. Overall, Fig. 5 suggests that selecting in the range 0.9-0.95 strikes a good balance between effectively eliminating the targel client's 's contribution and preserving the utility of the learned model.
MLP
45.0
Attack
0.8
Text Testaccumcy
46.0
435
-0.5
43.0
42.5
42.0
41.5
0.7
0.8
0.92
0.95
0.97
CNN
Test socuracy Attack sccuracy
0.1
-0.3
-0.3
ResNet
Test accuracy Attack accuraty
0.45
0.35
0.300
60
0.20
00.15
0.7
0.8
0.92
0.95
0.97
Threshold
Figure 5: Effect of the SVD coverage coefficient e on the bust-attack trade-off (CIFAR-10, IID,, evaluated with MLP/CNN Rest..
Impact of client's participation rate. therefore vary the client participation rate, defined as the per-round availability (sampling) probability of clients, and evaluate how well SFU removes the target client while preserving the remaining clients' utility, As shown in Fig, aa), SFU consistently yields a favorable unlearning outcome across three architectures on CIFAR-10 10 (IID), and higher participation rates generally lead to better performance on the remaining clients, with the improvement gradually saturating as more clients become available. This trend is expected since a higher participation rate provides more non-target clients' descending gradients over rounds, resulting in a richer and more representative gradient subspace, which better constrains the unlearning update to preserve non-target directions while suppressing the target client's 's influence. In the extreme case of 0% participation, the system degenerates to purely local training without federated aggregation, and SFU cannot construct the orthogonal gradient space from non-target clients; consequently, applying the unlearning update without this subspace constraint leads to severe parameter drift and yields near lhaece-level accuracy (about 10% for 10-way classification). These results underscore the necessity of performing SFU within an orthogonal gradient space: even partial participation supplies sufficient geometric structure for meaningful unlearning, while higher participation further improves the utility of the resulting model.
In practical FL, not all clients are available in every round. We
11

---

Published in Transactions on Machine Learning Research (12/2025)
80
MLP CNN
70
Reshet
Redket
50
Test Scouracy
40
20
25%
75%
100%
30%
20% percentage target
30%
40%
50%
Participaation rate
(a) Test accuracy (CIFAR-10, IID) vs. client participa- tion rate.
(b) Outcome under the IID setting for different amounts of data on the larget client.
Figure Ablation results of SFU.
Impact of the amount of data the target client. The data volume of the target client reflects, to some extent, how strongly it influences the learned global model during FL training. Fig, G(b) reports the outcome on CIFAR-10 (IID) across three architectures when varying the target client's data proportion. As the target client holds more data, its contribution is more deeply encoded in the global parameters, and the unlearning update must remove a larger portion of this contribution; this inevitably increases the tension between forgellng the larget. and preserving the utility for the remaining clients. Consequently, the post-unlearning test accuracy on the remaining clients decreases for all backbones as the target client becomes larger. This trend is consistent across architectures, suggesting that the degradation is primarily driven by the target client's increasing influence rather than model-specific artifact. Nevertheless, SFU remains effective in removing the target client's contribution under almost all target-data settings, indicating that the method is robust to substantial variations in the target client's 's importance in the system and can operate reliably even when the target client dominates non-trivis con-trivial fraction the training data.
5 Conclusion
In this paper, propose novel federated unlearning approach, SFU, that effectively climinate the contribution of specified client to the global model while minimizing the utility loss the remaining clients. The core idea is to perform the unlcarning update in constrained manner by carrying out gradient ascent within an orthogonal subspace derived from the non-target clients' gradients, which does not require restarting the entire learning process. Notably, SFU only requires access to the target client that needs to forgotten, and does not rely on the server or other clients to store historical parameter updates or client- specific training trajectorics, making it practical for real-world deployments with limited logging, realistic communication constraints, and heterogeneous client availability. We evaluate SFU under backdoor atlacks to explicitly quantify whether the target client's influence has been removed, and extensive experiments across different models and settings demonstrate that SFU is both effective in forgetting and efficient in computation, achieving favorable trade-off between unlearning strength and model utility in practice.
Acknowledgements
Li Shen is supported by the NSFC Grant (No. 62576364), Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO.
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
[L] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938 2948. PMLR, 2020.
[2] Thomas Baumhauer, Pascal Schöutle and Malthias Zeppelzauer. Machine unlearning: Linear filtration for logit-based classifiers. Machine Learning, 111(9):3203 0226, 2022.
[3] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloc Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learn- ing at scale: System design. Proceedings of machine learning and systems, 1:374 388, 2019.
[4] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-150 TEEE, 2021.
[5] Axel Bussche. The EU General Dala Prolection Regulation (GDPR): A Pructical Guide. Springer, 2017.
[6] Xiaoyu Cao, Jinyuan Jia, Zaixi Zhang, and Neil Zhenqiang Gong. Fedrecover: Recovering from poisoning attacks in federated learning using historical information. In 2023 IEEE Symposium Security and Privucy (SP), pp. 1366 1383. IEEE, 2023.
[7] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine uulearning. In 2015 IEEE Symposium on Security and Privacy, pp. 463 480. IEEE, 2015.
[8] Yudong Chen and Martin J Wainwright. Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees. arXie preprint arXiv:1509.0m 03005, 2015.
(9] Weilin Cong and Mehrdad Mahdavi. Efficiently forgetting what you have learned in graph representation learning via projection. In International Conference on Artificial Intelligence and Statistics, pp. 6674 6703. PMLR, 2023.
[10] Mchrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Arlificial Intelligence and Statislics, pp. 3762 3773. PMLR, 2020.
[11] Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. The limitations federated learning in sybil scttings. In RAID, pp. 301-316, 2020.
[1] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
[13] Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11516-11524 2021.
[14] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnots: Identifying vulnerabilitios in the machine learning model supply chain. arXiv preprinl xrviv:1700 11111770897999 2017.
[15] Anisa Halimi, Swanand Kadhe, Ambrish Rawat, and Nathalie Baracaldo. Federated unlearning: How to efficiently erase a client in fi? arXiv preprint arXiv:2207.00 05521,, 2022.
166 Andreas Hoecker and Vakhtang Kartvelishvili. Svd approach to data unfolding. Nuclear Instraments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 372(3):469 481, 1996.
[17] Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificiaa Intelligence and Statistics, pp. 2008 2016. PMT.R, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2025)
[18] Hyejum Jeong, Shiqing Ma, and Amir Houmansadr. Sok: Challenges and opportunities in federated unlearning. arXiv preprint arXin:2033.0245 2024.
[19] Weisen Jiang, James Kwok, and Yu Zhang. Subspace learning for effective meta-learning. In Interna- tional Conference on Machine Learning, pp. 10177-10194. PMI.R, 2022.
[20] Yasser H Khalil, Leo Brunswic, Soufiane Lamghari, Xu Li, Mahdi Beitollahi, and Xi Chen. Not: Fed- erated unlearning via weight negation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25759-25769, 2025.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 2324, 2002.
[23] Guanghao Li, Yue Hu, Miao Zhang, Ji Liu, Quanjun Yin, Yong Peng, and Dejing Dou. Fedhisyn: A hi- erarchical synchronous federated learning framework for resource and data helerogeneily. In Proceedingss of the 51st International Conference on Parallel Processing, pp. 1 1-11, 2022.
[24] Li Li, Yuxi Fan, Mike Tse, and Kuo- Yi Lin. A review of applications in federated learning. Computers & Industrial Engineering, 149:106854, 2020.
[25] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 2021.
[26] Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, and Xiaolin Huang. Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2022.
[27] Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, and Xiaolin Huang. Subspace adversarial training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13409- 13418, 2022.
[28] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3))50 60, 2020.
[29] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint 1rXi::1907.0211 2019.
[30] Gaoyang Liu, Xiaoqiang Ma, Yang Yang. Chen Wang, and Jiangehuan Liu. Federaser: Enabling cffi- cient client-level data removal from federated learning models. In 2021 IEEE/ACM 29th International Symposium Quality of Service (IWQOS), pp. 1-10. IEEE, 2021.
[31] Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Alcksci Timofcev, Philip Q Nelson, Greg S Corrado, et al. Detecting cancer metastases on gigapixel pathology images. XrXiv preprint ar 008.0224,, 2017.
[32] Brendan McMahan, Eider Moore, Danicl Ramage, Seth Hampson, and Blaise Aguera y Arcas. coammnicatioon-fficcee learning of deep networks from decentralized data. In Artificial intelligence PMLR, 2017.
and statistics, pp.
[33] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP), pp. 691-706. IEEE, 2019.
[34] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), pp. 1-15, 2018.
[35] Stuart Pardau. The california consumer privacy act: Towards a european-style privacy regime in the united states J. Tech. L. & Pol'y, 23:68, 2018.
14

---

Published in Transactions on Machine Learning Research (12/2025)
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in newral information processing systems, 32, 2019.
[37]
Rowayda A Sadek. Svd based image processing applications: state of the art, contributions and research challenges. arXiv preprint
2012.
[38] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning, arXiv preprint WXXi:2103.09706 2021.
[39] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075- 2021.
[40] Supreeth Shastri, Melissa Wasserman, and Vijay Chidambaram. The seven sins of (Personal-Data) processing systems under {GDPR). In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19), 2019.
[41] Mengkai Song, Zhibo Wang, Zhifei Zhang, Yang Song, Qian Wang, Ju Ren, and Hairong Qi. Analyzing user-level privacy atlack against federated learning. IEEE Journal on Selected Areas in Commnunicationss 38(10):2430-24 2444, 2020.
42] Ningxin Su and Baochun Li. Asynchronous federated unlearning. In IEEE INFOCOM 2023-IEEE conference on computer communications, pp. 1-10. IEFE, 2023.
[43] Shangchao Su, Bin Li, and Xiangyang Xue. One-shot federated learning without server-side training. Neural Networks, 2023.
[44] Youming Tao, Cheng-Long Wang, Miao Pan, Dongxiao Yu, Xiuzhen Chong, and Di Wang. Communi- cation efficient and provable federated unlearning. arXiv preprint arXiw:2/01.11018 2024.
[45] Junxiao Wang. Song Guo, Xin Xie, and Heng Qi. Federated unlearning via clas-ddssciminative pruning. In Proceedings of the ACM Web Conference 2022, pp. 622-632, 2022.
[46] Chen Wu, Sencun Zhu, and Prasenjit Mitra. Federated unlearning with knowledge distillation. arXiv preprint orXiv:2201.0944 2022.
[47] Leijie Wu, Song Guo, Junxiao Wang, Zicong Hong, Jie Zhang, and Yaohong Ding. Federated unlearning: Guarantee the right of clients to forget. IEFE Network, 36(5):129-133 2022.
[48] Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning, pp. 10355-10366. PMLR, 2020.
[49] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against. federated learning. In International conference on learning representations, 2020.
[50] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, Xiv preprint ar Xiv: 0008080000 2018
15

---

Published in Transactions on Machine Learning Research (12/2025)
A
Appendix
We conducted experiments on real-world datasets including MNIST, CIFARIO, and CIFAR100. The exper- imental settings are described in detail below.
Datasct. We utilized real-world datasets, specifically MNIST, CIFAR1O, and CIFAR100. The MNIST dataset 22 consists of 60,000 training samples and 10,000 test samples distributed across 10 classes. Each data sample is grayscale image of dimensions 28 x 28. The CIFAR10 dataset comprises 50,000 training samples and 10,000 test samples spread across 10 classes. Each data sample is color image with dimensions 3 x 32 32. Similarly, the CIFAR100 dataset [1 includes 50,000 training samples and 10,000 test samples distributed among 100 classes, with 500 Iraining samples per class. The Normalization of pixel values for MNIST and CIFAR10/ 100 is performed using mean and standard deviation values of [0.5, 0.5, 0.5] for both.
Datascts MNIST 60,000
Training Data
Test Data
Classes
Size
10,000
10
x 28
CIFAR-10
50,000
10,000
10
3 x 32 32
CIFAR-100
50,000
10,000
100
3 x 32 x 32
Table 3: Summary of dataset characteristics
IID Distribution in CIFAR-10
Dirichlet 0.3 Distribution in CIFAR-10
130
2000
1500
Clesses
1250
1000
115
500
130
Clients
(a) IID
(b) Dir(0.3)
Figure 7: Heat maps for each client on CIFAR-10 under different data partitions. The color bar denotes the number of samples, and each cell shows the per-elass sample count for a client.
Dataset Partitions. To ensure fair comparison with other baselines, we introduce heterogeneity by partitioning the total dataset based on label ratios sampled from the Dirichlet distribution. A parameter controls the level of heterogeneity in the data partition. Fig. 7 displays heat maps illustrating the label distribution across different datascts. Notably, for heterogencity weight of in the Dirichlet distribution, approximately 10% to 20% of calegories dominate each client, depicted by the blue blocks in Fig. ( In contrast, the IID dataset is uniformly distributed across clients, as indicated by the blue block in Fig. a).
Baseline Algorithms. We evaluate our proposed federated unlearning algorithm (SFU) against four baseline methods:
Retraining: This method involves retraining the entire FL system without excluding the target client, resulting in computational and communication overhead.
UI-Subtraction 46] This approach forgets the target client by subtracting historical parameter updates unique to the larget client from the global model. Knowledge distillation is used to alleviate the distortion caused by subtraction.
Unlearning via Projected Gradient Ascent (UPGA) 16 UPGA leverages gradient ascent infor- mation from the target client revert the learning process and achieve unlearning, while constraining updates norm ball.
16

---

Published in Transactions on Machine Learning Research (12/2025)
EWC-SGA 4ri EWC-SGA employs the Fisher Information matrix to regularize the cross-entropy loss and control parameter updates, with higher importance factors imposing stricter constraints.
Network Architectures.
We employ three neural network architectures in our experiments:
A fully-connected neural network with 2 hidden layers, containing 200 and 100 neurons, respec- tively.
MLP:
CNN: A network architecture consisting of 2 convolutional layers with 64 5x5 filters, followed by 2 fully connected layers with 800 and 500 neurons and ReLU activation function.
ResNet: We adopt standard 4-stage ResNet with global average pooling and a linear classifier. For MNIST, we modify the first convolution to accept 1-channel inputs. We also evaluate a lightweight variant by reducing the stage widths while kceping the same topology.
All neural network architectures were implemented in Py'Torch 36
Implementation Details. The hyperparameters for each method were set as follows: SFU used a learning rate of 0.01, epoch as 1, and a mini-batch size of 128 for gradient ascent on the target client. SVD parameters followed the setting of Saha et al. [S)] exploring within the range [0.90 0.99] and selecting the optimal value. UPGA and EWC-SGA maintained the same learning rate and mini-batoh size as SFU. ULS on the server used public dataset formed by randomly sampling one-tenth of the total data. For the model performance recovery experiment, FL training commenced with the stochastic model for full retraining. For SFU, UPGA, and EWC-SGA, FL training started on the unlearned local model without the target client's 's involvement, Knowledge distillation with the server's public data aided model accuracy recovery for UL. Additionally, paramcter searches wore performed to determinc the best hyperparameter values.
Unlearning Metrics Comparing the distinction between the unlearned model and the retrained model serves as fundamental criterion for measuring the effectiveness of unlcarning. Common dissimilarity metries encompass model test accuracy difference A Ez-distance AN and Kullback-Leibler (KL) divergence By] Nevertheless, in the context of Federated Learning (FL), assessing the removal of specific client's 's contribu- tion through these model difference- evaluation methods can be challenging. Alternative metries involve privacy leakage within the differential privacy framework 39 and membership inference atlacks 13 2 In
this study, we adopt backdoor triggers [4] as an effective means to evaluate unlearning methods, akin to Wu et al. EG. Specifically, the target client employs a dataset containing a fraction of images with inserted backdoor triggers. Consequently, the global FL model becomes susceptible these triggers. A successful unlearning process should result. in model that reduces accuracy for images with backdoor triggers while maintaining strong performance regular (clean) images. It is important to note that we employ back- door triggers solely for the purpose of evaluating unlearning methods; we do not consider any malicious clients 49
Our Evaluation Metric. We use backdoor attacks in the target client's updates to the global model, as described earlier, to intuitively assess the unlearning effect based on the attack success rate of the unlearned global model. In Table we record the attack success rate as "atk acc'. A lower "atk acc" indicates cleaner removal of the target client's contribution. In our experiments, we implement the backdoor attack using a 'pixel pattern' trigger of size 2x2 and change the label to "9'. Due to the predictive errors of the model, even the attack success rate after retraining is greater than 0. We can consider the attack success rate after retraining as the benchmark for successful removal of the target client's contribution. Additionally, we use the accuracy metrie on clean test data lo gauge the model's performance after unlearning, denoted as 'test acc" in Table [ A high accuracy suggests that unlearning has minimal impact on the model's performance.
17


Reviews:
[{"review_id": "OZpCTrix6G", "reviewer": "Reviewer_B2Ph", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Subspace-based Federated Unlearning (SFU), which is a new method to remove a client’s influence from a federated learning (FL) model without retraining or storing historical updates. SFU performs gradient ascent on the target client but constrains it to an orthogonal subspace to the gradient space of other clients. This helps erase the target client’s contribution while preserving overall model performance. The method requires only one local training round per client and no storage of historical gradients, making it practical for resource-constrained FL systems. The paper provides theoretical guarantees and empirical experiments on MNIST, CIFAR-10, and CIFAR-100. SFU achieves strong unlearning performance (reducing attack success rates) while retaining model accuracy better than prior baselines (UPGA, EWC-SAG, ULS).\n\nKey Strengths of the paper\n1. The paper is well written to explain the idea clearly and thoroughly. \n2. The proposed SFU is very efficient given that it requires only one local training round per client and no storage of historical gradients.\n3. The paper provides a good theoretical foundation with convergence and effectiveness proofs. \n\nWeaknesses of the papaer\n1. The empirical experiments need improvement. The accuracy of CIFAR10 models are typically below 50% which is unacceptable. Even under the Federated Learning scenario, with 10 clients, I think it's reasonable to expect the accuracy to be ~80%. The extremely low accuracy may suggest either that the model is too simple or that the training process is not correct. This will weaken the readers' belief of the effectiveness of the experiments. \n2. The paper should state more clearly regarding the resource usage of SFU vs. the other method. In my understanding, the SFU will require the unlearning client to possess the training data (or at least the data that the client wants to forget). While other unlearning methods may not require this data but needs to store the historical gradient data on server end. I don't think this is a disadvantage itself; instead, this could be an advantage that clients can determine what data they want to unlearn from the model. However, I think the author should state more clearly regarding the resource requirements of their methods.", "claims_and_evidence_yn": "No", "claims_explanation": "The major issue is with the results of empirical experiments. I think the extremely low test accuracy of the trained CIFAR10 (<50%) and CIFAR100 (~20%) models will make the audience doubt the reliability of the results. Because this may suggest either that the model is too simple or that the training process is not correct.", "audience_yn": "Yes", "audience_explanation": "The idea presented in this paper is novel and well written. Subspace-based Federated Unlearning (SFU) offers a new and efficient approach to address an important problem in federated unlearning. Given the growing interest in privacy, unlearning, and resource-efficient federated learning, this method would be of broad interest to the TMLR audience, particularly those working on model unlearning, privacy-preserving machine learning, and distributed optimization. The clear presentation and strong theoretical grounding make the findings valuable for researchers pursuing similar directions.", "requested_changes": "1. Improve experimental results (Critical). The current CIFAR-10 models achieve unusually low accuracy (often below 50%), which raises concerns about the validity of the training setup or model capacity. The authors should revisit the experimental design to ensure models are properly selected and trained, and results are representative of realistic FL performance (expected 80% accuracy).\n2. Clarify resource requirements (Important but not critical). The paper should more clearly discuss the resource implications of SFU compared to other methods. Specifically, SFU requires the target client to have access to its data during unlearning, while other methods store historical gradients on the server. This clarification would help readers fairly assess the trade-offs in practicality and deployment.", "broader_impact_concerns": "No, I don't find any concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hXrqmKE56S", "reviewer": "Reviewer_8CB1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies unlearning in a federated learning (FL) system, referred to as federated unlearning for brevity. While federated unlearning has been studied previously, existing popular methods typically require additional storage for historical data in order to remove the influence of a client that wishes to leave the FL system. This paper proposes a storage-efficient strategy for federated unlearning, called subspace-based federated unlearning.\n\nStrengths: (i) The problem setup and method advantages are clearly described. (ii) Sufficient numerical results are provided for demonstration.\n\nWeaknesses: The theoretical results are basic, and the technical writing could be improved.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The method developed in this paper is based on projecting the gradient of the targeted client onto the orthogonal space of the gradients of the remaining clients. This approach is fundamentally different from retraining-based methods and is therefore storage-efficient.", "audience_yn": "Yes", "audience_explanation": "This paper provides an interesting approach to tackling federated unlearning. The algorithm's update direction is obtained by projecting the gradient of the target client onto the orthogonal space of the gradients of the remaining clients, so that it still maximizes the targeted loss while mitigating the effects on the remaining clients.", "requested_changes": "Major Comments: I have some concerns regarding the theoretical results in this paper. \n\n(i) Theorem 1 provides an upper bound for $\\|w_r(T) - w_{ul}(T)\\|$, where $w_r(T)$ denotes the weights obtained from training without the targeted client, and $w_{ul}(T)$ denotes the weights obtained by SFU. It uses $w(T)$, the weights obtained from training with all clients, as a reference, and establishes upper bounds for $\\|w_r(T) - w(T)\\|$ and $\\|w(T) - w_{ul}(T)\\|$, respectively.\n\nIt is unclear to me why Assumption 1 implies (7). I believe the right-hand side should be $\\eta\\|\\nabla L_r(w_r(t)) - \\nabla L(w(t))\\|$. Why is this term bounded by $2\\eta G$? Recall that there is no assumption made on $\\nabla L_r$, and  Assumption 1 is stated in expectation.\n\nMoreover, (9) does not imply any useful property specific to SFU. It appears that this inequality holds even without performing the projection onto the orthogonal space.\n\n(ii) Theorem 2 is somewhat confusing. The statement claims that $-g_P$ is a descent direction for $L(w)$, i.e., the global objective over all clients. However, the last sentence of the proof states that $-g_P$ is a descent direction for client $i$'s objective. This causes confusion about which objective function the descent direction is actually associated with.\n\nI believe this theorem can only be properly evaluated once its meaning is clarified.\n\nMinor comments: \n\nPlease try to fix the typos in the manuscript. For example,\n\n1. Page 6, Theorem 1. Assumptions1 should be Assumption 1\n\n2. Pages 6-7, Proof of Theorem 1. $w_{ul}$ or $w_{ul}(T)$\n\n3. Page 7, Display (7). What is $L^u$?\n\nThe displays on page 7 waste too much space, in my opinion. Please try to make them more compact.", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "OMyeikys2q", "reviewer": "Reviewer_3Efq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel method for “federated unlearning,” which aims to remove the influence of a particular client’s data from a trained model in federated learning via post-processing. The proposed algorithm is based on the idea of projecting the unlearning gradient into the orthogonal subspace of the gradient directions of the remaining clients. The effectiveness of the method is also demonstrated through numerical experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "* There are numerous concerns regarding unclear points and the logical validity of the theorems, proofs, and notation.\n    * In **Theorem 1**, it is unclear how $T$ is defined, and what exactly is meant by the “solution obtained through retraining.”\n    * In **Assumption 1**, it is assumed that the expectation of $\\|\\nabla L(w)\\|^2$ is bounded by $G^2$, but in the proof of Theorem 1 the inequality $\\|\\nabla L(w)\\| \\le G^2$ is used. This is inconsistent in two ways: taking expectations and taking squares.\n    * In **Theorem 2**, it is unclear why $\\langle g_P, g_I \\rangle \\le 0$ implies that $-g_P$ is a descent direction of $L(w)$. Under the natural interpretation, one would need to show that $\\langle -g_P, \\nabla L(w) \\rangle < 0$. On the other hand, since $L(w)$ is defined using data from all clients in equation (1), its gradient does not correspond to $g_I$, so this is confusing.\n    * The definition of $g$ in **Theorem 2** is unclear.\n    * In **Algorithm 1** Step 14 and Section 3.2.2, $S^l = \\operatorname{span}\\{u_1^l,\\ldots,u_k^l\\}$ is defined, i.e., $S^l$ is a subspace. Then $P^l$ is defined as $P^l = S^l (S^l)^T$, but this expression does not make notational sense. Is this a typo for $P^l = U^l (U^l)^T$?\n    * In **Algorithm 1** Step 15, it says $P \\leftarrow [P^1, P^2, \\ldots]$, but this notation looks like the matrices $P^l$ are concatenated horizontally. If the intention is to apply projection $P^l$ for each layer $l$, then $P$ should instead be a block-diagonal matrix with $P^l$ along the diagonal.\n    * In **Algorithm 1** Steps 2 and 6, the notation $\\nabla L_i(w^*;D^i)$ is used, but I could not find its definition in the paper. It should be clarified whether this means the gradient computed using all data in $D^i$, or a stochastic gradient obtained by sampling a subset, etc.\n    * Just before equation (3), is $(R^l)_t$ a typo for $(R^l)_k$?\n    * In the “Target Client” paragraph of Section 3.2.1, is the expression inside the argmin also a typo?\n* Although the experimental section claims the effectiveness of the proposed method, its advantage over existing approaches is not made clear. In fact, as can be seen from Table 1, for the attack success rate existing methods such as UPGA and ULS often outperform the proposed method. From the perspective of the trade-off between test accuracy and attack success rate, it is therefore difficult to judge how advantageous the proposed method actually is.", "audience_yn": "Yes", "audience_explanation": "Federated learning and unlearning are both topics that have attracted considerable attention in the machine learning community in recent years, and if the claims of this paper are correct, its findings are likely to draw interest from many researchers.", "requested_changes": "I would appreciate it if you could review the concerns I have raised regarding the evidence and explain any misunderstandings on my part or possible ways to address them.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)