You are an expert in Deep Learning and Machine Learning. As an Academic Peer Review Assistant, your role is to provide a thorough and constructive review of submitted academic papers. When a user presents a paper for review, assess it across several key dimensions to ensure scholarly rigor and clarity. Your evaluation should cover:

•              Content Accuracy and Relevance: Analyze whether the information presented is accurate, up-to-date, and relevant to the stated research questions.

•              Argumentation and Analysis: Assess the strength and coherence of the arguments. Are the claims well-supported by evidence? Is the analysis thorough and critical?

•              Methodology: Evaluate the appropriateness and execution of the research methodology. Is it suitable for the research question? Are the methods clearly explained and justified?

•              Organization and Structure: Consider the logical flow of the paper. Is it well-organized, allowing readers to easily follow the progression of ideas?

•              Writing Quality and Clarity: Review the paper for clarity of language, style, and grammar. Is the writing clear, concise, and appropriately academic?

•              References and Citations: Check for the completeness and accuracy of citations and references. Are all sources credible and properly attributed?

Provide specific feedback for each category, citing examples from the paper to support your evaluations. Offer constructive criticism and suggestions for improvement to help the author enhance the paper. Summarize the overall strengths and weaknesses, and recommend whether the paper should be accepted, revised, or rejected based on your assessment.

PAPER:

Abstract.This paper reports on findings from a comparative study on

the effectiveness and efficiency of federated unlearning strategies within

Federated Online Learning to Rank (FOLTR), with specific attention

to systematically analysing the unlearning capabilities of methods in a

verifiable manner.

Federated approaches to ranking of search results have recently garnered

attention to address users privacy concerns. In FOLTR, privacy is safe-

guarded by collaboratively training ranking models across decentralized

data sources, preserving individual user data while optimizing search

results based on implicit feedback, such as clicks.

Recent legislation introduced across numerous countries is establishing

the so called “the right to be forgotten”, according to which services based

on machine learning models like those in FOLTR should provide capa-

bilities that allow users to remove their own data from those used to

train models. This has sparked the development of unlearning methods,

along with evaluation practices to measure whether unlearning of a user

data successfully occurred. Current evaluation practices are however of-

ten controversial, necessitating the use of multiple metrics for a more

comprehensive assessment – but previous proposals of unlearning meth-

ods only used single evaluation metrics.

This paper addresses this limitation: our study rigorously assesses the

effectiveness of unlearning strategies in managing both under-unlearning

and over-unlearning scenarios using adapted, and newly proposed evalu-

ation metrics. Thanks to our detailed analysis, we uncover the strengths

and limitations of five unlearning strategies, offering valuable insights

into optimizing federated unlearning to balance data privacy and system

performance within FOLTR. We publicly release our code and complete

results^1.

```

```

Keywords:Federated Unlearning·Federated Online Learning to Rank.

```

## 1 Introduction



Online Learning to Rank (OLTR) [12,13,19,20,35,39] utilises users’ queries and

interactions on search results to train the ranking model in real-time, wherein

a central server commonly conducts this training process. This centralization



(^1) https://anonymous.4open.science/r/Unlearning-for-FOLTR-531E/





2 Anonymous Author(s)



Fig. 1:Unlearning in Federated Online Learning to Rank. While each client performs OLTR training

locally,{θ^1 t,θ^2 t,...,θnt}represent the local ranker parameters for client{ 1 , 2 ,...,N}separately.CR

denotes the set of remaining clients that have not request unlearning. At timestamptof unlearning,

local parameters fromCRwill be sent to the server and contributes to the updated global ranker

θt+1, which will be sent back to each client ofCR.



brings challenges in protecting user’s privacy, as the data that captured by the

server for training, i.e., user’s queries and interactions in the form of clicks, can

reveal private user information like demographic attributes, personal habits, or

political views [1, 3, 5]. To safeguard user’s privacy, Federated Online Learn-

ing to Rank (FOLTR) leverages training strategies from Federated Learning

(FL) [18] to perform OLTR [14, 26, 28]. In FOLTR, each client uses its own

data to update the ranker locally, then sends the ranker update to a central

server instead of sending the data itself. The server aggregates the received up-

dates to produce an updated global ranker, which is subsequently broadcast to

all clients, achieving a privacy-enhanced online training paradigm through fed-

eration. This method has been experimentally shown to deliver effective and

privacy-preserving rankers [26].



Recent legislation approved across several countries establishes the “the right

to be forgotten”, as exemplified by the European Union’s General Data Protection

Regulation (GDPR) [25]. This legislation prescribe that in any machine learning

system, a person may request the removal of their own data from the trained

model. In the context of a FOLTR system, this means that a client leaving the

federation can request the removal of the contributions made to the ranker by

their data. Machine Unlearning (MU) and Federated Unlearning (FU) meth-

ods address this need, though current FOLTR systems lack effective solutions

for removing specific user contributions from the global ranker, with the only

method in this context being the one proposed by Wanget al. [27] who adapted

the FedEraser strategy [15] to FOLTR and evaluated its unlearning effectiveness

under a sophisticated simulation process. Figure 1 illustrates the mechanism of

Unlearning for FOLTR.



However, the initial work only studied a single unlearning method and used

limited evaluation metrics. This narrow focus falls short of capturing the full

complexity of unlearning within the FOLTR environment. Also, other types

of unlearning approaches have not been explored in FOLTR. In this paper,

we conduct a comprehensive reproducibility analysis on the issue of unlearn-





```

Unlearning for Federated Online Learning to Rank: A Reproducibility Study 3

```

ing in FOLTR, focusing on evaluating the effectiveness of unlearning, risk of

over-unlearning, and recovery efficiency of five general FU strategies adapted to

FOLTR. The main contributions of this paper are summarized as follows:



- We integrate five general Federated Unlearning strategies – FedEraser [15],

Gradient Ascent [11], a Fine Tuning-based method [36], FedRemove [34], and

Retraining – within FOLTR for the first time, showcasing their versatility

and effectiveness across four popular LTR datasets.

- We employ a comprehensive set of metrics, including adapted and newly

introduced evaluations, to thoroughly analyze each strategy’s effectiveness

in handlingunder-unlearning(incomplete data removal) andover-unlearning

(excessive data removal).

- We evaluate the efficiency of each unlearning strategy, specifically focusing on

their recovery speed from poisoning, which provides insights into optimizing

strategy selection for different unlearning requirements.

- We make our source code and data publicly available to facilitate research

in this direction.



## 2 Related Work



2.1 Federated Online Learning to Rank



As a specialized distributed machine learning paradigm, FL enables collabo-

rative training while preserving user privacy. To satisfy privacy-preserving re-

quirements in OLTR, FL has also been introduced to this field termed federated

online learning to rank (FOLTR). The FOLtR-ES method does this by incorpo-

rating evolution strategies to optimize the ranking model andε-local differential

privacy for enhanced data protection [14], but this methods is found to have

reduced effectiveness and adaptability on large-scale datasets [28]. A more ef-

fective approach, FPDGD [26], has been proposed by integrating the advanced

Pairwise Differentiable Gradient Descent (PDGD) [19] into the FL framework.

Thanks to the superiority of locally-deployed PDGD ranker in handling noise

and biases from user interactions, FPDGD has demonstrated strong ranking per-

formance which is comparable to that of the state-of-the-art centralised OLTR

methods. Following this, Wang and Zuccon [29] investigate the robustness of

FPDGD to non-independent and identically distributed client data. The vulner-

ability of FPDGD to poisoning attacks in the federated system is also simulated

and verified [30], which extends the landscape of study on FOLTR.



2.2 Federated Unlearning



Federated Unlearning extends the principles of machine unlearning to FL set-

tings, allowing individual clients’ data to be removed from a global model. A

naive approach in FU is to retrain the global model from scratch without involv-

ing clients who requested deletion, ensuing complete removal but at a high com-

putational cost. In order to accelerate the FU process, FedEraser adjusts histori-

cal updates from clients to reconstruct unlearned models [15]; other computation-

efficient approaches have followed [11,16,32]. While FU has been studied across





4 Anonymous Author(s)



natural language processing and recommendation systems [34, 38], its applica-

tion to FOLTR tasks remains unexplored, with Wang et al. [27] presenting the

only approach to date. This study implemented an existing FU approach and

evaluated its unlearning performance under a user simulation inspired by poi-

soning attacks [24,30]. While Wang et al. [27] pointed out that the uniqueness

of FOLTR including optimising ranking tasks with implicit user feedback in an

online manner, the challenge of adapting existing FU methods to FOLTR and

evaluating the unlearning performance are largely unexplored. In this paper, we

close this gap by evaluating within the context of FOLTR the performance of a

broad range of unlearning methods across a diverse set of evaluation metrics.



## 3 Unlearning for Federated Online Learning to Rank



3.1 Preliminary: Federated Online Learning to Rank



In OLTR, given a user queryqand a list of candidate documents, a ranking model

fis leveraged to determine the final ranked list. The ranking system updates

the model parameterθbased on user interactions with the ranking list, e.g.

clicks. FOLTR moves this ranker updating process into a hierarchical structure

with each client deploying a local ranker trained by the OLTR algorithm and

a central server aggregating all local models to produce a global ranker update



- the global ranker is then distributed back to individual clients for the next

learning iteration. Formally: Letcirepresent theithclient in FOLTR, where the

set of all clients is denoted asC={c 1 ,...,cn}. We assume that the local data^2 of

theithclient is denoted bydi, and the full training dataset is represented asD=

{d 1 ,...,dn}.θitrepresents the local ranker parameters forci. At timestamptof

global update, the local parameters fromCare sent to the server and contribute

to the updated global rankerθt+1, which is then sent back to each client.



3.2 Problem setup: Federated Unlearning



LetCtargetdenote the set oftarget clients(i.e. those leaving the federation); their

dataDtargetneeds to be unlearned.Cremaindenotes the set ofremaining clients

that have not undergone data erasure (i.e. have not left the federation), with their

corresponding datasets represented byDremain. Note thatC=Cremain∪Ctarget

andD=Dremain∪Dtarget. The global objective functionθ∗minimizes the em-

pirical loss across the remaining clients after unlearning has been requested:

θ∗= arg min

θ



### P



```

i∈CremainwiLi(θ,di), where the weightwirepresents the propor-

```

tion of theithclient’s datasetdirelative to the total datasetD, andLi(θ,di)

denotes the loss function on clienti’s datasetdiusing the parameterθ.

In thet-th global round, theithclient runsEtimes of OLTR update locally

based on its local datadito optimize the model parametersθtreceived from the

previous global iteration. This results in the updated local model parameters



(^2) In OLTR, data used for training typically involves queries and corresponding candidate docu-

ments, feature representations of query-document pairs and user clicks.





```

Unlearning for Federated Online Learning to Rank: A Reproducibility Study 5

```

θti+1. After receiving parameter updates from theremaining clients, the server

aggregates them by calculating the weighted average:



```

θt+1=

```

P

i∈Cremainwiθ

ti+1. (1)

The purpose of FU is to remove the contribution of thetarget clientsand

obtain the unlearned global modelfwith parametersθuas similar as possible

to the modelfwithθ∗, which is trained onDremainof theremaining clients.

Model similarity can be quantified by measuring the distribution similarity of

model parameters. Then, we can define the problem as:

T(fθ∗(Dremain))≃TU(fθu(D)), (2)

whereT denotes the distribution of model parameters. The unlearning strategy

Uachieves unlearning when these two distributions are as close as possible.



3.3 Federated Unlearning Strategies



Next we introduce the five FU strategies we consider in this study, and their

adaptation to FOLTR.

Retraining.Retraining from scratch restarts the entire training process from

the beginning with onlyCremainand the server involved. The global model is

updated based on Eq.(1). This method guarantees the complete removal of data

fromtarget clients. While retraining is possible in standard offline classification

tasks, retraining for OLTR is not possible in practice, because users cannot be

asked to repeat the same interactions they have done in the past. In our study,

as a baseline for comparison, we utilise retraining by repeating the same FOLTR

process but not with the same historical interactions, noting its limitations^3.

Fine-tuning [36].Unlike retraining, FU via fine-tuning continues training of

the current global model (regarded as the "pre-trained" model) once unlearning

has been requested, collaboratively using interactions from the remaining clients

and updating the global model based on Eq.(1) (regarded as the "fine-tuning"

process) [36]. The logic behind fine-tuning is that when the model is fine-tuned

onDremain, it induces catastrophic forgetting onDtarget, which is a common

phenomenon in continual learning processes [22].

FedEraser [15,27].This method removes the influence of target clientsCtarget

by calibrating the updates from the remaining clientsCremainand aggregating

these calibrated updates. Specifically, each remaining clienticalibrates its local

update∆ˆθitusing the stored historical local update∆θit:

∆θ ̃ti= ∆

θˆit

∥∆θˆit∥



(^) ∆θit

(^) , (3)

Then each clientisends the calibrated updates∆θ ̃itto the central server and

the global model is updated:θt+1=θt+



### P



i∈Cremainwi∆

θ ̃it.

FedRemove [34].This method directly removes the parameter updates of the

target clientsCtargetfrom the global aggregation. Same as FedEraser, before the

unlearning request, the update of each client∆θitare stored after each iteration

tof the federated training. When the removal happens, the stored parameters of

Cremainare re-aggregated as the global model:θt+1=θt+



### P



```

i∈Cremainwi∆θ

```

```

i

t.

```

(^3) Retraining might still be done using offline LTR approaches like counterfactual learning; however

we do not consider this possibility in this paper.





6 Anonymous Author(s)



Compared to FedEraser, this method does not include the calibration step as

Eq.(3) and only the central server is involved and aggregates the historical local

updates fromCremainin unlearning. The advantage of this method is its rapid

unlearning speed because it does not require retraining the model and it only

needs to re-aggregate the updates of all clients except those being forgotten.



Gradient Ascent [11]. This approach utilises gradient ascent to maximizes

the empirical loss ontarget clientsfor reversing the local model training process.

To prevent excessive parameter divergence, updates are projected onto anℓ 2 -

norm ball centered at the reference model, which represents the average of the

remaining clients’ models, ensuring the unlearned model stays aligned with them.

Unlike other FU strategies, this approach performs unlearning on thetarget

clientslocal side with their local interactions. During each iteration, target client

iupdates its model parameters as follows:



θ←P(θ+ηu∇Fi(θ)) (4)

wherePdenotes the projection operator onto theℓ 2 -norm ball,ηuis the learning

rate for unlearning, andFi(θ)represents the OLTR loss function of clienti.



## 4 Unlearning Evaluation



A key challenge for unlearning lies in its evaluation practice. The goal of unlearn-

ing is to remove the contribution of target clients so as to obtain an unlearned

global model. This model should be similar to a model trained from scratch on

the same interactions but those from the target clients. Two types of error can

lead to ineffective unlearning. When the process fails to forget the clients data,

we are in presence of incomplete forgetting, orunder-unlearning. On the other

hand, when the client fails to retain contributions provided by clients still in

the federation, we are in presence of erroneous forgetting, orover-unlearning.

To ensure a robust evaluation of the studied unlearning strategies, in this paper

we employ a broad spectrum of evaluation metrics designed to detected either

under-unlearningorover-unlearning[37]: this is the first time unlearning meth-

ods for FOLTR have been considered across both evaluation dimensions.



4.1 Under-unlearning



Data Poisoning Attack.To measure the unlearning performance, previous

studies utilize a backdoor attack which puts a marker on the target data thus

the backdoor attack is only triggered by backdoor patterns in the target data [2,

31,33]. However, this evaluation approach cannot be adapted to FOLTR directly

due to the difficulties of backdoor attacks in FOLTR [27]. To overcome this, we

manipulate the target clients by employing a data poisoning attack [30] based

on a reverse simulation click strategy. In particular, a client compromised to

data poisoning will intentionally interact with documents unrelated to the search

query. In our empirical study, we simulate the poisoned user’s clicks based on the

data poisoning click instantiation shown in Table 1. This action injects mislead-

ing interaction data into the training phase and disrupts the model’s accurate





```

Unlearning for Federated Online Learning to Rank: A Reproducibility Study 7

```

prediction of user preference. The effectiveness of unlearning is thus determined

by comparing the nDCG@k score before and after unlearning. Compared to

the nDCG@k score before unlearning, a higher nDCG@k score achieved after

unlearning is indicative of the effectiveness of the unlearning strategy.

Model Poisoning Attack.Unlike data poisoning, model poisoning directly

modifies the local model updates. Previous studies have demonstrated its effec-

tiveness in compromising global model performance and evaluating unlearning

in FL systems [4, 8, 24, 27, 34]. Similar to Wang et al [27], we also evaluate the

unlearning effectiveness in the model poisoning attack scenario where we assume

that the local updatesθitoftarget clientshave been compromised before unlearn-

ing:θpoit =−γθit+μ, whereθpoit represents the poisoned local updates that will

be sent for global aggregation. The coefficientγis used to control the magnitude

of these updates, whileμdirectly adds noise. At each local training step before

unlearning,γis a random value drawn from a uniform distribution over the in-

terval[1,2]andμis drawn from a normal distribution whose mean and standard

deviation are determined by the original gradient updates. As thesetarget clients

attempt to disrupt the FOLTR training process by uploading poisoned updates,

the evaluation of unlearning performance has been switched to observe if the

unlearned model can achieve high ranking performance after unlearning. In the

Model Poisoning Attack, we use the same evaluation metrics as those used in

the Data Poisoning Attack.

Relevancy Reset.In contrast to the two invasive methods mentioned above,

we also introduce a non-invasive evaluation approach, Relevancy Reset (RelR),

aimed at preventing performance deterioration and external security risks. In-

spired by Erroneous Memory (EM) [9], RelR focuses on the samples that exhibit

high loss during the model training process, which indicates the model struggles

to produce a correct ranking. By paying attention to and relabeling these sam-

ples, it is possible to evaluate the unlearning effect. Specifically, we first select

the top K% high-loss queries from the datasets of a target clientcunder the

local modelfc; we denote these asDc. Then we modify the relevance labels of

the documents in these queries according to the predicted ranking result by the

local modelfcand we denote the relabelled dataset asDmc (Details of our im-

plementation are given in Section 5.1). Then we fine-tune the local modelfcon

Dmc to obtain a modelfˆc. This model is then uploaded to the central server for

aggregation. After the unlearning process, an increase in the loss on the subset

Dmc in the global model suggests that the model has successfully achieved the

intended unlearning effect.



4.2 Over-unlearning



Another key aspect of evaluating unlearning performance is the presence of over-

unlearning. Over-unlearning occurs when data that should be retained is inad-

vertently forgotten. To determine the occurrence of over-unlearning, we compare

the alignment degree between the retrained model and the unlearned model us-

ing model performance-based and model distribution-based metrics [37].

Model Performance.The straightforward approach to comparing the align-

ment between the unlearned model and the retrained model is through their





8 Anonymous Author(s)



consistency in model performance (e.g.,test accuracy and loss). In OLTR, we em-

ploy performance-based metric like nDCG@k for evaluation. A higher nDCG@k

value indicates better unlearning performance.

Model Distribution.An alternative approach is to evaluate the distribution

of model parameters: this quantifies the alignment between the unlearned and

retrained models by calculating the divergence between their distributions, using

statistical measures such asℓ 2 -distance, KL divergence [10] or Wasserstein dis-

tance [21]. As shown in Eq.(2), a reduced divergence indicates a higher fidelity

in the unlearning process, reflecting a more effective unlearning strategy. In this

paper, we quantify the parameter distribution difference between the unlearned

model and the retrained model usingℓ 2 -distance.



## 5 Experiments



We instruct experiments to address the following three research questions:



- RQ1: How do the unlearning strategies perform in terms of under-

unlearning?This question assesses the effectiveness of different FU meth-

ods in thoroughly removing the influence of target data.

- RQ2: How do the unlearning strategies perform in terms of over-

unlearning?This explores whether the applied unlearning strategies inad-

vertently remove more information than necessary.

- RQ3: What is the efficiency of the unlearning strategies?This ques-

tion evaluates the efficiency of strategies by comparing the recovery speed

(from poisoning) of different unlearning strategies and the baselineRetrain-

ingstrategy.



5.1 Experimental Settings



Datasets.We use four datasets commonly used for OLTR evaluation: MQ2007 [23],

MSLR-WEB10K [23], Yahoo [6], and Istella-S [17]. MQ2007 is the smallest with

1,700 queries. Each query-document pair is represented by a 46-dimensional

feature vector, and relevance labels fromnot relevant(0) tovery relevant(2).

MSLR-WEB10K, Yahoo, and Istella-S are larger, and all have a 5-level relevance

label scheme, fromnot relevant(0) toperfectly relevant(4). MSLR-WEB10K

has 10,000 queries and 136 features, Yahoo has 29,900 queries and 700 features,

and Istella-S has 33,018 queries and 220 features. MQ2007 and MSLR-WEB10K

have five data splits, while Yahoo and Istella-S have a single data split. Results

are averaged across all data splits.

User simulations.As no real-world user interactions are available, we sim-

ulate user queries and click behaviours through randomly sampling N queries for

each participating client in each training round and leveraging the click model

for generating clicks. (This is common in FOLTR research). We employ the Sim-

plified Dynamic Bayesian Network (SDBN) click model [7] to simulate users’

click behavior. Under this click model, users examine the search engine results

page from top to bottom. The SDBN click model provides conditional probabili-

ties of user clicks based on the relevance of each document. The click probability

P(click= 1|rel(d))signifies the likelihood of a user clicking on a document related





```

Unlearning for Federated Online Learning to Rank: A Reproducibility Study 9

```

Table 1:Instantiations of SDBN click model. rel(d)denotes the relevance label for documentd. The

values for MQ2007 are demonstrated in brackets (only three levels of relevance are used).

P(click= 1|rel(d)) P(stop= 1|click= 1,rel(d))

rel(d) 0 1 2 3 4 0 1 2 3 4

per. (0.0) (0.5) (1.0) (-) (-) (0.0) (0.0) (0.0) (-) (-)0.0 0.2 0.4 0.8 1.0 0.0 0.0 0.0 0.0 0.

nav. (0.05) (0.5) (0.95) (-) (-) (0.2) (0.5) (0.9) (-) (-)0.05 0.3 0.5 0.7 0.95 0.2 0.3 0.5 0.7 0.

inf. (0.4) (0.7) (0.9) (-) (-) (0.1) (0.3) (0.5) (-) (-)0.4 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.

Poison (1.0) (0.5) (0.0) (-) (-) (0.0) (0.0) (0.0) (-) (-)1.0 0.8 0.6 0.2 0.0 0.0 0.0 0.0 0.0 0.



to its relevance level. The stopping probabilityP(stop= 1|click= 1,rel(d))rep-

resents the likelihood that a user decides to end their search session after clicking

on a document they deem relevant. SDBN has three common instances. ThePer-

fectuser examines and clicks on all relevant documents, providing very reliable

feedback. TheNavigationaluser may stop searching after finding the target in-

formation. They primarily focus on finding reasonably relevant documents. The

Informationaluser does not selectively click on documents, thus providing noisy

click feedback. Table 1 illustrates the three instances of SDBN and an instance

simulating malicious click behaviour for Data Poisoning (Section 4.1).

Federated Settings.Our experiments involve 10 clients, each performing

5 local updates before the model update is sent to the central server. The global

model is updated 1,000 times for both before unlearning and during the un-

learning procedure. As ranker we employ a linear model, with a learning rate

of 0.1; we train the ranker with FPDGD [26]. When evaluating the unlearning

performance under data poisoning attack and model poisoning attack, we set

the poisoning rate to 30%, with three clients subjected to a poisoning.

For the RelR metrics and datasets like MQ2007, which have three levels of

relevance, we re-label the top 10% of samples inDcmas highly relevant (rel-

evance=2). The subsequent 20% are re-labeled as moderately relevant (rele-

vance=1), and the remaining samples are re-labeled as irrelevant (relevance=0).

In contrast, for datasets with five relevance levels (MSLR-10K, Yahoo, and

Istella-s), we re-label the top 20% of samples inDmc as highly relevant (rele-

vance=4). Each subsequent 20% interval receives a decreasingly lower relevance

score, ensuring the labels accurately reflect the predicted ranking distribution of

the local modelfc.



5.2 Results and Analysis



We present results only for the Yahoo dataset. Similar trends are observed on

the other datasets but are omitted due to space constraints. Complete results are

available athttps://anonymous.4open.science/r/Unlearning-for-FOLTR-531E/.

RQ1: How do the unlearning strategies perform in terms of under-

unlearning?

Experimental Design.Under-unlearning evaluation aims to determine if the

data contributed by the target clients has been thoroughly removed from the

ranker. In this experiment, the assessment is conducted using three metrics: Rele-

vancy Reset Difference (RelR Diff), Online performance and Offline performance.





10 Anonymous Author(s)



```

(a) Data Poisoning.

```

```

(b) Model Poisoning

```

```

(c) Without Poisoning.

```

Fig. 2:Offline nDCG@10 obtained under different poisoning strategies on Yahoo dataset, under three

click models(Perfect,Navigational,Informational).



Online performance is the cumulative discounted nDCG@10 of the rankings dis-

played to users during training, measuring the user experience throughout the

training process. ForTsequential queries withRtas the ranking displayed to

the user at timestept, this is:



```

Online Performance=

```

```

PT

t=1nDCG(R

t)·γ(t−1) (5)

```

We follow previous work [19] by choosing a discount factor ofγ= 0. 9995. Offline

performance is the averaged nDCG@10 of the global ranker on the test set,

showcasing the overall performance of the model. Online performance and offline

performance are discussed under two types of attacks: Data Poisoning and Model

Poisoning. And RelR Diff is used in the absence of attacks.

Results Analysis.Figure 2a reports the offline nDCG@10 under data poison-

ing, while Figure 2b illustrates it under model poisoning. The "Original" line

represents the globally poisoned model before unlearning, with its nDCG@

score remaining consistently low in the poisoned state. During epochs 0 to 1000,

each client performs federated training, followed by unlearning from epochs 1000

to 2000. We find that,Fine-tuningdemonstrates the best unlearning effect, out-

performing theRetrainingacross all three click models by effectively removing

the impact of target data (with a significant improvement in nDCG@10 com-

pared to the original model).FedEraser, andGradient Ascent methods per-

form comparably toRetraining, occasionally even surpassing it, demonstrating

strong unlearning capabilities. In contrast,FedRemovedisplays the weakest per-





```

Unlearning for Federated Online Learning to Rank: A Reproducibility Study 11

```

Table 2:Evaluation ofOnline Performance(Discounted Cumulative nDCG@10) on the Yahoo

dataset, under three different click models. The best results are highlighted in boldface with super-

scripts denoting statistically significant results (paired Student’s t-test withp≤ 0. 05 ).

Dataset Click Poisoning Original(a) Retrain(b) FedEraser(c) Fine-tuning(d)



```

Yahoo

```

```

Per.

```

```

Data 1041.8 1092.9 1096.6 1111.9a

Model 854.2 894.8 922.2 938.4ab

None 1124.6bc 1105.3 1110.5 1124.

```

```

Nav.

```

```

Data 874.1 942.6 945.9 967.8abc

Model 867.5 898.6 945.5 967.8abc

None 1112.8bc 1086.9 1092.4 1112.

```

```

Inf.

```

```

Data 829.7 903.2 907.2 936.6abc

Model 863.3 882.5 938.9 969.5abc

None 1103.9bc 1069.0 1057.7 1103.

```

formance, sometimes falling below the original model.

For online performance, as shown in Table 2,Fine-tuningagain leads, offering the

best user experience during unlearning, followed byFedEraserandRetraining.

SinceGradient Ascent performs unlearning on target clients andFedRemove

operates on the server, a comparison of online performance for remaining the

clients during the unlearning phase is not feasible.

Table 3 highlights RelR Diff across unlearning strategies, whereGradient Ascent

stands out with the highest value, underscoring its unlearning effectiveness.



RQ2: How do the unlearning strategies perform in terms of over-

unlearning?

Experimental Design.Another contrasting evaluation perspective to under-

unlearning in FU is over-unlearning, which assesses whether essential data that

does not require unlearning, apart from the target clients’ data, has also been

inadvertently removed. In this experiment, we investigate the prevalence of over-

unlearning by analyzing how closely various unlearning strategies align with

Retrainingoutcomes. We utilize three metrics: Distance Difference (Dist Diff),

Online performance(without poisoning) and Offline performance(without poi-

soning). Dist Diff quantifies the parameter distribution difference between the

unlearned model and the retrained model (calculated usingℓ 2 -distance). A close

alignment withRetrainingon Dist Diff suggests better unlearning performance

in terms of over-unlearning, while a higher nDCG@10 value generally indicates

better ranking performance of the unlearned model.

Results Analysis.In the evaluation of over-unlearning, different unlearning

strategies exhibit subtle differences, particularly for the Dist Diff metric (shown

in Table 3).Gradient Ascentshows a significantly higher distance compared

to other strategies, indicating its poorer ability to maintain model integrity

after unlearning, leading to over-unlearning.FedEraser,Gradient Ascent, and

Fine-tuningall demonstrate stable model accuracy (with Offline nDCG@10, as

shown in Figure 2c, comparable to Retraining). However,FedRemoveperforms

the worst in terms of both model integrity and accuracy.

In terms of online performance (shown in Table 2), without a poisoning attack,

Originalachieves the best online performance. Other strategies, such asFed-

Eraser,Fine-tuning, andRetrain, perform slightly worse thanOriginalbut still





12 Anonymous Author(s)



Table 3:RelR DifferenceandDistance Differenceon Yahoo dataset, under three different click

models. The best results are highlighted in boldface with superscripts denoting results for statistical

significance study (paired Student’s t-test withp≤ 0. 05 ).

Dataset Model Metrics Retrain(a) FedEraser(b) Gradient Ascent(c) Fine-tuning(d) FedRemove(e)



```

Yahoo

```

```

Per. RelR Diff 0.97 0.88 1.

```

```

abde 0.91 -0.

Dist Diff - 5.87 149.75 4.00bce 5.

Nav.RelR Diff 1.35 2.

```

```

ade 2.33 0.98 0.

Dist Diff - 7.86 99.43 4.80bce 9.

Inf. RelR Diff 1.67 1.56 3.

```

```

abde 1.96 -0.

Dist Diff - 10.25 102.08 6.13bce 10.

```

manage to maintain user experience during the unlearning process. For the same

reasons as in RQ1, we do not compareGradient AscentandFedRemove.



RQ3: What is the efficiency of the unlearning strategies?

Experimental Design.To evaluate the efficiency of different unlearning strate-

gies, we compare the recovery speed (from data poisoning and model poisoning)

for each method. In this experiment, Retraining serves as the baseline. If an

unlearning strategy achieves better recovery than Retraining within the same

number of global training steps, it indicates that this strategy has a faster re-

covery speed.

Results Analysis.From Figure 2a and 2b , we find thatFine-tuningreaches

convergence first, indicating that theFine-tuningstrategy has higher recovery

efficiency, as it approaches its maximum recovery level with fewer training steps.

FedEraserandGradient Ascenthave a convergence speed comparable toRe-

training. TheFedRemovecurve shows significant fluctuations across all datasets

and never stabilizes, indicating that it has not effectively converged.



## 6 Conclusion



Research on unlearning methods for FOLTR remains in its early stages. To date,

limited knowledge is available due to lacking empirical studies across unlearning

approaches as well as a not well-established evaluation practice that does not

consider a variety of important metrics. To gain more insight into unlearning

approaches for FOLTR, we reproduce five FU strategies and assess them across

four popular LTR datasets, applying a range of evaluation metrics to capture

the nuanced performance of each strategy. We find that, for under-unlearning

(RQ1),Fine-tuningperforms best under data and model poisoning conditions,

withFedEraserandGradient Ascentgenerally maintaining performance compa-

rable toRetraining, whileFedRemoveshows the weakest unlearning effectiveness.

In terms of over-unlearning (RQ2),Fine-tuningeffectively retains non-target

data,FedEraserperforms well, whereasGradient Ascenttends to over-unlearn.

Regarding efficiency (RQ3),Fine-tuningdemonstrates a significantly faster re-

covery speed (from poisoning) than other strategies, withGradient Ascentand

FedErasermaintaining efficiency comparable toRetraining. Overall, this paper

provides a comparative analysis of five FU strategies within FOLTR, revealing

key insights into differences in data retention accuracy and recovery efficiency

across strategies. These findings provide foundational support for optimising FU

strategies to balance privacy and performance in FOLTR.





```

Unlearning for Federated Online Learning to Rank: A Reproducibility Study 13

```