A Reinforcement Learning-based Bidding Strategy for Data Consumers in Auction-based Federated Learning
Authors: Xiaoli Tang
Han Yu
Xiaoxiao Li
Venue: NeurIPS 2025 poster ()
Keywords: federated learning
auction-based federated learning
Gaussian Mixture Model
Primary area: machine_learning_for_sciences

Abstract: Auction-based Federated Learning (AFL) fosters collaboration among self-interested data consumers (DCs) and data owners (DOs). A major challenge in AFL pertains to how DCs select and bid for DOs. Existing methods are generally static, making them ill-suited for dynamic AFL markets. To address this issue, we propose the R}einforcement Learning-based Bidding Strategy for DCs in Auction-based Federated Learning (RLB-AFL). We incorporate historical states into a Deep Q-Network to capture sequential information critical for bidding decisions. To mitigate state space sparsity, where specific states rarely reoccur for each DC during auctions, we incorporate the Gaussian Mixture Model into RLB-AFL. This facilitates soft clustering on sequential states, reducing the state space dimensionality and easing exploration and action-value function approximation. In addition, we enhance the $\epsilon$-greedy policy to help the RLB-AFL agent balance exploitation and exploration, enabling it to be more adaptable in the AFL decision-making process. Extensive experiments under 6 widely used benchmark datasets demonstrate that RLB-AFL achieves superior performance compared to 8 state-of-the-art approaches. It outperforms the best baseline by 10.56% and 3.15% in terms of average total utility

A Reinforcement Learning-based Bidding Strategy for Data Consumers in Auction-basee Federated Learning
Xiaoli Tang', Han Yu', Xiaoxiao Li2.3
*College of Computing and Data Science, Nanyang Technological University, Singapore Department of Electrical and Computer Engineering, University of British Columbia, Canada
3 Vector Institute, Canada
Abstract
Auction-based Federated Learning (AFL) fosters collaboration among self- interested data consumers (DCs) and data owners (DOs). A major challenge AFL pertains to how DCs select and bid for DOs. Existing methods are gen- erally static, making them ill-suited for dynamic AFL markets. To address this issuc, we propose the Reinforcement Learning-bast based Bidding Strategy for DCs Auction-based Federated Learning (RLB-AFL). We incorporate historical states into a Deep Q-Network to capture sequential information critical for bidding decisions. To mitigate state space sparsity, where specific states rarely reoccur for each DC during auctions, we incorporate the Gaussian Mixture Model into RLB-AFL. This facilitates soft clustering on sequential states, reducing the state space dimensional- ity and easing exploration and action-value function approximation In addition, enhance the e-greedy policy help the RLB-AFF agent balance exploitation and exploration, enabling it to be more adaptable in the AFL decision decision-making process. Extensive experiments under 6 widely used benchmark datasets demonstrate that RLB-AFL achieves superior performance compared to 8 srute-of-the-as approaches. It outperforms the best bascline by 10.56% and 3.15% in terms of average total utility.
1 Introduction
Driven by stringent user privacy and data confidentiality requirements, federated learning (FL) has recently attracted substantial attention from both academic and industrial domains [177. With data owners (DOs), also known as FL clients, being self-interested entities that weigh a myriad of factors (ranging from costs potential utility gains) when deciding which FL data consumer (DC) to collaborate with, the design of FL incentive mechanisms mechanisms aim to incentivize DOs to participate in FI. through various reward strategies.
g 9 has taken center stage. These
Auction-bass based federated learning (AFL) is an important sub-field FL incentive mechanism design, due to its potential to achieve both efficiency and fairness [Л) In AFL, the incentive mechanism between DOs and DCs is organized in the form of an auction. This process is overseen by an auctioneer, who acts methods can be roughly categorized into three main groups [2]] 1) DC-side methods, 2) auctioneer- side methods, and 3) DO-side methods. The DC-side methods focus how DCs select and place bids on DOs, aiming to optimize key performance indicators (KPIs) while staying within budget constraints. The auctioneer-side methods optimize DC-DO matching and pricing strategies, along with the design effective auction mechanisms. The goal is to achieve specific operational objectives, such as maximizing social welfare or minimizing social costs, for the AFL ecosystem. DOs care more about determining the allocation of local resources and setting their reserve prices for profit maximization [3
intermediary to facilitate the exchange of data for FL training. AFL
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In recent times, there has been growing research interest [14] [5] in investigating DC-side issues, developing optimal bidding strategies assist them in effectively bidding for DOs. However, existing methods are generally static approaches. They are essentially represented by either non-linear or linear functions with parameters derived from historical auction data using heuristic techniques. Then, these parameters are applied to new auctions, even if the dynamics of these new auctions might vary significantly from those in the historical data. In practice, the inherent dynamism of AFL markets poses a considerable challenge for static DC bidding methods to consistently achieve desirable outcomes.
To bridge this important gap, we propose a Reinforcement Learning- based Bidding Strategy for DCs in duction-based FL (RLB-AFL), It incorporates historical states into a Deep Q-Network to capture sequential information critical for AFL DC bidding decisions. To mitigate the state space sparsity issue in AFI., where specific states rarely re-appear for a DC, we propose to integrate the Gaussian Mixture Model into RLB-AFL to enable soft clustering on sequential states, thereby reducing the state space dimensionality, which in turn eases exploration and action-vall function approximation. Moreover, we improve the e-greedy policy to help an RLB-AFL agent strike balance between exploitation and exploration, enhancing its applicability in the decision-making process for each DC within an AFL ecosystem.
To our best knowledge, RLB-AFL is the first cluster-based reinforcement learning approach that facilitates a large number of DCs to compete for a common pool of DOs. Extensive experiments conducted on widely used benchmark datasets demonstrate the superiority of RLB-AFL compared to 8 tate-of-the--e existing approaches. It outperforms the best baseline by 10.56% and 3. 3.1%% in terms of average total utility and model accuracy, respectively.
2 Related Works
Existing AFL DC-side methods, the primary focus of this paper, can be broadly categorized into two main groups based on the auction mechanism adopted: 1) those for reverse auction scenarios, and 2) those for forward auction scenarios.
Methods like 17- designed for reverse auction scenarios aim to help the DC select DOs after receiving their asking profiles (which may include available data resources and the corresponding asking prices). [24 combined a quality-aware model aggregation algorithm with reverse auction, and proposed the FAIR method. It employs greedy algorithm based on Myerson's 's theorem (28 to determine the winning DOs and maximize the valuation for the DC. However, crucial limitation of these methods arises from their assumption that there is only one DC and multiple DOs in the AFL marketplace. This monopoly market assumption is unrealistic in practice, where multiple DCs are typically present.
15] focus on assisting DCs bidding for DOs under a competitive AFL market setting, employing forward auction mechanisms. These methods design bidding strategies to guide DCs in determining bid prices for DOs. [5] introduced the Fed- red-Bidder bidding strategy which considers DC budget constraints, DO relevance and prior auction-related knowledge to design a bidding function. It also emphasized the critical roles played by accurate estimation of DO utility and the selection of an appropriate winning function in shaping optimal bidding strategies.
Methods like (29
RLB-AFL falls into the category of methods designed for forward auction scenarios. However, is noteworthy that most existing bidding strategies designed for DCs are static, and thus may not be suitable for dynamic AFL markets.
3 Preliminaries
AFL Market:
Generally, an AFL market consists of three types of participants [12 1) Data Owners (DOs): entities possessing potentially sensitive yet valuable data, who are willing to share or sell access to their data resources for FL task training in exchange for appropriate compensation. 2) Data Consumers (DCs): organizations or individuals requiring data to train their machine learning models via FL. 3) Auctioneer: trusted third-party entity orchestrating the auction process between DOs and DCs. It facilitates the exchange of data resources for FL. training tasks through an auction mechanism, such the Second-Pric Sealed-Bi Bid (SPSB) auction.

---

When DO is ready to offer its services for FL task training, it notifies the auctioneer, specifying its bid request and the reserve price price[TThe The auctioneer then announces the auction to all DCs currently participating in the AFL. market. Any DC whose required the corresponding data resources aligns with the DO's s offering submits a bid for the auction.
Each DO can trigger the following auction process: 1) Bid Request Initiation: DO E [C,] generates a bid request about itself (e.g., identity, data quantity, etc.) and sends it along with the the reserve price (i.e., the lowest price it is willing to accept for selling the corresponding resources 32] to the
auctioneer. 2) Bid Request Dissemination: The auctioneer disseminates the received bid request to the relevant DCs whose FL tasks are relevant to the data resources of the DO being auctioned. 3) Bidding Response: Each relevant DC evaluates the potential value and cost of the received bid request, and decides on a bid price based on its bidding strategy. The DCs submit their bids to the auctioneer. When a DC has exhausted its budget, it will forfeit future auctions. 4) Outcome Determination:
Upon receiving bids from relevant DCs, the auctioneer determines the winning price based on an auction mechanism. then compares the winning price with the reserve price set by each DO. If the winning price is lower than the reserve price, the auctioneer terminates the auction and informs the DO to initiate another auction for the same resources. Otherwise, the auctioneer informs the winning DC about the cost (i.e., the winning price) it needs to pay, informs the losing DCs, and informs the DO about the winning DC shall join.
The FL training process for the target DC based on its recruited DOs commences once either their budget is depleted or all available DOs have been recruited by DCs.
Problem Formulation: The AFL DC bidding can be framed as an optimization problem within budget B limit 115 to maximize the DC's total utility with respect to a set of DOs [1, C):
max Σ
x' x v², AE[1,C]
8.1. Σ
x xxx S B
(1)
(E[1,C]
where of denotes the utility the DC can gain from DO being auctioned. The specific process to calculate of is described in the subsequent section. Here, x' E {0, denotes whether the target DC wins i, and p³ denotes the payment from the target DC to i. Notably, under the SPSB auction mechanism (3), if a DC wins the bid for a DO i, p' equals to the second-highest bid price among all the bids received by the auctioneer, otherwise, p' = 0.
In 15 it has been shown that under SPSB, the optimal bidding strategy is:
6 == v² o'/ww
(2)
is scaling factor. When the sequence of DO arrival is known in advance, the optimal value (w") can be determined using a greedy approximation algorithm R4. Unfortunately, in practice, strategies must be executed in real-time without prior knowledge of the available data resources being auctioned. Moreover, the auction environment typically exhibits high nonstationarity due to the dynamic behaviors of all participating DCs, making the derivation of wn challenging.
Data Owner Reputation Modeling:
Following 335 we assess the utility of attracting a DO i for a DC by evaluating i's reputation. To calculate r's reputation (o') for DC, we start by adopting the computationally efficient GTG-Shapley method (36), which measures i's contribution to the DC in the Shapley Value sense [37]. This value is then fed into the Beta Reputation System (BRS) (38 to obtain i reputation value.
The contribution of DO to a DC can be grouped into two categories: 1) negative (i.e., of < 0) and 2) positive (i.c., o² 2 0). We adopt the variables ne and pe' to record the number of negative contributions and the number of positive contributions made by i for the DC, respectively. Then, we employ the BRS to compute the reputation value vù for DO i on the target DC as:
pc +
e² EeBeaa(pe + 1,, nc +1]] pe + not + 2
It is essential emphasize that, as illustrated in Eq. (3) DO i's reputation undergoes dynamic updates throughout the FL model training process. Additionally, in situations where no prior information accessible, the default initial reputation value of i is established as the uniform distribution,
represented by of = UO0 1) = 1).
FFllowwig [5 assume that DOs arrive and make their bid requests sequentially, one after the other.
3

---

DATA OWNERS
Bid request
AUCTIONEER
Bid request
DATA CONSUMER
4a) Failure Notice
/ Auuction
3. Bid price
Bidding Strategy
Bidding Agent
4b) Winner Notice
4bb Winner Notice
Historical States
GMM
Global parameters
Local parameters peramterss
SSL
DQN
Figure 1: The RLB-AFL system architecture.
4
The Proposed RLB-AFL Approach
The system architecture of RLB-AFL is illustrated in Fig. T In response the limitations of current DC bidding methods within dynamic AFL markets, as well as the challenges arising from the lack of access to private information about DOs and the underlying AFL. system dynamics, we propose a reinforcement learning (RI.) framework, RLB-AFL, for modeling each DDCss bidding process as an w control problem. RL is pivotal tackling this problem due to its ability to learn optimal decision- making policies directly from interactions with the environment. Following [29], we leverage the Deep Q-Network (DQN) [39] as the underlying RL model, which uses deep neural networks to approximate the optimal action-value function, enabling effective recision-making in complex, high- dimensional state spaces. Our design incorporates Gaussian Mixture Model (GMM) into RLB-AFF to perform soft clustering the continuous state space, mitigating the curse of dimensionality. In addition, we improve the e-greedy policy to strike a balance between exploration and exploitation, enabling AFL to adapt new market conditions while capitalizing its learned knowledge.
4.1 POMDP Modeling
RLB-AFL frames the bidding agent for the target DC by sequentially regulating w under the Partially Observable Markov Decision Process (POMDP) setting [40). The objective of the DC agent is to acquire an optimal w controlling policy that maximizes the accumulated reward E " while ensuring that >
iim r' x xp < B. The fundamental components of the POMDP are:
o,1s Before the bid for DO i, the observation o consists of: 1) B²: the remaining budget, 2) (C i): the remaining DOs, and 3) o': the utility the target DC can gain from DO i, and is formulated as:
0, (B' (B*CC= - -,)))
a; We introduce several adjustment rates to w. often taking the form of an action a E A as W; w x (1 A,), where A, represents the adjustment rate related to a.
ry The reward for bidding for DO i computed as ri = x² x v², where xf E {0, 1)
indicates whether the DC wins the auction.
y: The objective of the target DC is to maximize the overall utility of winning DOs, subject to the budget constraint B, irrespective of utility over time. Therefore, the reward discount factor 7 is set as 1, i.e., = 1.
Although the state formulation in Eq. is straightforward, it resides within an infinitely continuous state space. In light of this, the occurrence of particular state might be rare in AFL processes, particularly when sequential information is incorporated. Consequently, accurately learning an approximation of the value function becomes extremely challenging. Furthermore, this issue leads to high exploration costs. Therefore, it is crucial to map the state space into a lower-dimensional and, ideally, finite space. To tackle this issue, we propose state clustering method.
'In this paper, while it might be more accurate to refer to it as an observation, we continue to use the both term state and term observation concerning the bidding process without ambiguity.

---

4.2
State Clustering
During the decision-making process, an agent can rely on historical information. This intuition has been widely adopted in multiple domains (e.g., recommendation systems [41], click-through rate estimation in computational advertising (42), and has been proven effective. This suggests that might be useful to base DC bidding strategies on the historical information from recent auctions, rather than only on the current state. Motivated by this observation, we frame the state from the perspective of modeling the sequential information of states (i.e., historical states). Specifically, define the sequential information of states as:
8, (+) (+: ;8);;
(5)
where W is a hyperparameter representing the window size. If j 0 (0 < W- 1), S1-j configured as a zero vector. Following this, we obtain the combined state &; through the application of the state mapping function fstate()) as:
8, (s))
(6)
In the context of AFL, intuitive observation is that similar historical state sequences tend to yield comparable rewards under a given bidding strategy. This key insight emphasizes the potential benefits of grouping similar state sequences together through clustering. However, within AFL market, each DC faces a continuous state space comprising numerous elements (c.g.. utility derived from auctioned data, remaining budget, available time steps). Navigating this extensive state space and learning an effective bidding strategy is a formidable challenge, as it requires capturing the intricate dynamics and stochasticity of the environment without direct access to DOs' private information.
To address this critical issue, we design a soft clustering approach over the historical state sequences based on GMM 43 It effectively reduces the dimensionality of the state space, while preserving the essential information encoded in the sequential state trajectories. Dimensionality reduction is crucial for mitigating the state space sparsity issue, which can otherwise hinder accurate value function approximation and incur excessive exploration costs during the RL process. The ability of GMM to model complex data distributions through mixture of Gaussian components makes it well-suited for clustering the continuous and tigh-dimensional state representations arising from the incorporation of historical state information.
Specifically, let K denote the number of clusters of the historical states, and {s $2, ... SNl denote the available historical states. The conditional probability of each historical state for a cluster k is modeled by a Gaussian distribution:
p(s [2
k; pk.
Ex) N Ex).
Alternatively, the prior probability of each cluster k is assumed to adhere to a Multinomial distribution M Multinomial(u)) with:
uk > 0.
Σ uk 1, p(2 = k) uk.
(8)
Then, the total log- likeliooddd of the historical states is:
N
L(u, 2)) = log W( Mx)u
Σ
(9)
k=1
Following the Expectation-Maxiie Maximizaaiinn (EM) algorithm, RLB-AFL gradually learns the parameters u. n and Σ. Specifically, in the E-step, the weight recording the affinity of historical state 8; to cluster k is calculated as:
N(ux, Ex) )uk
# = p(z; = k| Σ/ N(5, Ezjus
(10)
In the M-step, the parameters of cluster is updated as:
N Ex
w
8t
E w (s:
N
Hx) -ma)
Uk
Wk
1
N
wi
N w
(11)
5

---

The E-step and the M-step are iteratively repeated until convergence. Eventually, the weight vector ((s;) expressing the inclination of s; towards each cluster E [1, K], is:
((s;) = (wi, W2
wkk)
(12)
However, obtaining a set of historical states and subsequently estimating the parameters of the GMM in sequential manner is not practical for DCs newly joining the AFL process. Therefore, we also propose the following EM algorithm to dynamically update the GMM clusters an adaptive fashion. Given the current state 8, utilizing the prevailing GMM, we begin by computing the posterior probabilities rs,,) as outlined Eq. (1O). Then, the M-step is enhanced (the first two equations have been consolidated into single line):
ug = (1 - K. + kwk, 14 μk + kwis; Σn = Σn + kajj (s, - (s)) -m=),, (13)
where E {0, } denotes the hyperparameter balancing the weight assigned to the incoming instance.
4.3 Enhanced c-greedy Policy
DQN implements the e-greedy policy to strike balance between exploitation and exploration, where the agent selects action a" = argmax, Q(o, a) with a probability (1 - <), while taking a random action with probability of c. The parameter is typically set to larger value and slowly anneals over time to a smaller value. Yet, determining an appropriate annealing rate is crucial, as a high annealing rate limits exploration, while low one can lead slow policy convergence.
However, in the context of AFL DC bidding, the optimal bidding theory guarantees a consistent optimal w for each DO i E [1, CJ. Given the observation og, taking the optimal action is equivalent to adjusting w to approach w". Any deviation from this optimal action results in a reduction in potential value, as reflected by lower value. Hence, considering range of adjustment rates denoted as {>a]), the action-value distribution Q(o;, a,) across the action space A sorted according to the adjustment scale X₂ of action, should ideally exhibit unimodality [4). Therefore, if the distribution is not unimodal, it implies an abnormal estimation of Q. is necessary to increase to encourage exploration in this state.
action space A, which encompasses
Algorithm 1 RLB--FF
Initialize (o, a; ()) and its target network with 0 <- 0, update frequency of target network t, replay
memory D, training batch size m.
1: Initialize wp;
2: for i 1 to C do
3:
Obtain the state based on the GMM;
4:
Compute a, according to the enhanced e-greedy policy w.r.t Q:
5:
Obtain wy based on W/-1i
6:
Calculate b; according to Eq. (
7:
Submit b' to the auctioneer;
8:
Get reward r; and the payment p²:
9:
Store transition tuples in D;
10:
Sample a random minibatch m samples from D;
11:
y = r + y max,' Q(,, a': 0);
12:
Update 0 by minimizing Σm l(y Q(o, a; 0))
13:
0 <- every t steps;
14: end for
4.4 The Training Process of RLB-AFL
RLB-AFL is based on DQN. The action-value function Q(o, a) is modeled by a deep neural network (DNN) with parameter 0. To enhance training stability, we leverage target network parameterized by 0, adopting similar DNN architecture to approximate Q(o, a) (Algorithm)
Updating of the parameters 0 achieved by minimizing L(0) = 1 "(o,a,ro"'= Q(o, a; 8)) Here, D replay buffer that stores transition tuples (o, a, r, o'), where o' denotes the new ob- servation of the bidding agent following the action derived from the initial observation and

---

corresponding reward r. Randomly sampling batches of transitions during training, buffer D fa- cilitates learning from past experience. Let r denote the discount factor. The temporal difference target y is computed as = r + y max, Q(d', a'; 0). Q(o', a'; 0) denotes the predicted action-value function of the DC bidding agent for its subsequent observation of considering all feasible actions a'. i denotes the parameters of the target network, and Q(o', a'; 0) denotes the predicted action-valus function. The target network ensures training stability by maintaining a fixed target throughout the training process, periodically updated to synchronize with the current action-value network.
5 Experimental Evaluation
5.1 Experiment Settings
Datasets: We adopt six widely used datasets in FL studies: 1) MNIST 2) CIFAR- 10 3) Fashion- MNIST a. FMNIST) [45]. 4) EMNIST-digits (a.k.a. EMNISTD) [46], 5) EMNIST-letters (a.k.a.
EMNISTL) (46) and 6) Kuzushij-MNIST (a.k.a. KMNIST) 47 The FL models used are the same
as those employed in [[]].
Comparison Baselines:
We compare RLB-AFL against the following eight well-establishe bidding
approaches: 1) Constant Bid (Const) 48
2) Randomly Generated Bid (Rand) [22 23). 3) Below
Max Utility Bid (Bmub) [49], 4) Linear-Form Bid (Lin) [50). 5) Bidding Machine (BM) [51] 6) Reinforcement Learning-based Bid (RLB)
29 14 52 More detailed descriptions of these methods
can be found in (29]. In addition, we include Fed-Bidder [15] which is specifically designed for AFL DCs. guides DCs to competitively bid for DOs to maximize their utility under given budget constraint. Fed-Bidder is implemented as two variants: 7) red-Bidder-sim (FBs) with simple winning function and 8) Feed-Bidder-cee (FBc) with a complex winning function.
Experimental Scenarios: We conduct experiments under two scenarios, each involving 10,000 DOs: 1) IID data: Each DO possesses set of 1,000 images, including some noisy ones. To facilitate the effective evaluation of DOs' reputations by DCs, the 10,000 DOs are organized into five groups, each comprising 2,000 DOs. In addition, different percentages of noisy data are introduced for each DO group as follows: DOs in the first, second, third, fourth, and last groups each owns 0%. 10%, 25%, 40%, and 60% noisy data, respectively. 2) Non-IID data: By adjusting the class distribution among individual DOs, which hold 1, 1,000 images, we intentionally introduce data heterogeneity in this experimental setup. Following (35). the Non-IID setup is implemented as follows: one class (for datasets except EMNISTL) or six classes (for EMNISTL) are designated as the minority class, assigned to 100 DOs. Therefore, images for all classes are possessed by these 100 DOs, while the other nine twenty classes except the minority class, exclusively held by all other DOs. Scenarios in which the minority classes are with 10% or 25% noisy data are also included.
Implementation Details: To deal with the challenge of lacking publicly available dataset related to AFL, we conducted simulations where we tracked the behaviors of DCs under the setting of forward auction and generalized second-price sealed-bid (SPSB) over time in four distinct scenarios, each involving 160 DCs. 1) One-eighth of the DCs adopts each of the eight comparison approaches. 2) Three-sixteenths of the total population adopt each of RLB. FBs, FBc and BM, while one-sixteenth of all DCs adopt the other four approaches. 3) and RLB-AFL. we fine-tuned the ratio of DCs choosing FBc and FBs to surpass those opting for the remaining six baseline methods. Specifically, 50 DCs chose FBc and FBs each, while the other six baselines were adopted by 10 DCss each. 4) Following the settings in Scenario 3, 65 DCs chose FBc and FBs each, while the other six baselines were adopted by DCs each..
for AFL with both Fed-Bidder variants
To assess the efficacy of RFB- 9 AFL DCs are implemented, each employing one of the previously mentioned bidding methods to bid for DOs. For the action-value function utilized by the bidding agent, RLB-AFL employs fully connected neural networks. These networks consist of three hidden layers, each comprising 64 nodes. The RMSprop with a 0.0001 learning rate is adopted to train all the neural networks. The discount factor for the reward is set to 1, as the primary aim of a DC is to maximize the overall utility gained from winning DOs within the budget constraints. A replay buffer 'D of size 6,000 is used for training the action-value function Q (i.e., [D] 6, 000). During training, 32 samples from D are utilized for updating Q at each training step (i.e., m = 32). Furthermore, the
http:/yyaan.leuuu.comm///mmmmmm cwtp//////////cc//s////ooom/
"htps://www.cs wwwwwwwwwwwwwwww.......ww....... www.oott....uukriiiiiaaaaaaamma
7

---

Table I: Comparison results of the total utilities and FL model accuracy (%) across different datasets and budget settings under the IID scenario. "Bud" means budget and "Acc" means accuracy. The best results are highlighted in Bold.
MNIST
CIFAR
FMNIST
EMNISTD
EMNISTL
KMNIST
Bud
Method
Utilyy
Acc
Unilit
Acc
Unlity
Acc
Unlity
Aec
Unlit
Acc
Utiity
Acc
Const
7.28
78.03
6.68
35.28
7.53
69.95
7.32
78.52
7.33
68.82
7.05
62.67
Rand
6.73
73.23
7.40
70.55
7.84
79.83
8.08
67.40
8.25
61.52
Bmub
8.48
80.72
9.67
35.74
9.56
71.31
9.45
80.36
9.97
70.39
9.12
63.54
Lin
11.42
82.02
10.96
37.70
11.14
71.84
11.15
80.76
11.22
71.23
11.18
64.19
BM
13.21
73.81
12.96
14.10
72.08
100
FBs
15.22
83.12
14.66
39.78
15.05
73.82
14.83
81.65
14.89
73.19
14.99
68.91
FBc
15.16
83.38
15.72
40.33
15.23
74.63
14,80
81 31.66
14,90
73.23
14.89
68.63
15.91
15.30
15.36
74.18
15.41
81.90
15.33
73.36
15.71
RLB-AFL
18.62
85,86
16.84
41.83
17,68
76.82
17.95
83.69
17.56
74.79
17.37
70,66
w/o c
16.97
84.42
15.98
41.11
16.71
75.68
16.64
82.37
16.48
73.82
16.04
69.34
e
16.25
84.55
16.25
41.09
16.29
75.79
16.58
83.46
15.37
73.59
16.33
70.26
Const
9.04
81.00
9.18
37.81
9.34
69.06
8.03
79.32
9.22
70.94
8.65
63.45
Rand
8,50
81.10
8.67
38.60
10.87
71.20
8.45
79.80
9.23
70,76
9.58
61.87
Bmub
12.08
81.90
10.72
39.39
11,48
72.23
10.15
81.37
11.63
71,99
10,48
64.74
Lin
13.80
82.13
13:43
40.13
13.55
72.85
13.29
81.40
13.56
73.05
13.62
68.99
BM
15.64
84.55
16.02
41.33
16.55
75.29
15.45
82.46
16.96
73.56
17.22
71.76
200
FBs
17.73
42.24
17.84
82.26
73.93
FBc
18.15
84.53
17.53
42.12
18.48
75.25
17.55
82.10
17.80
73.79
17.76
72.16
RLB
18.46
85.14
18.03
42.47
18.40
75.03
18.17
82,
18.23
74.47
18.65
74.60
AFL.
19.98
21.31
20.37
21.77
w/o
19.18
85.75
19.46
43.55
20.93
76.26
19.25
84.19
20.41
76.15
19.81
74.80
w/o
19.24
86.
19.58
43.86
19.96
76,86
19.75
83.83
19.35
75.88
19.92
75.32
Const
7.43
81.25
70.92
7.50
80.14
9.10
71.69
Rand
10.76
80.22
7.08
39.61
10.47
71.03
7.48
79.75
8.11
72.15
8.79
71.58
Bmub
11.56
82.30
10.33
40.14
11.35
73.20
10,51
82.05
11.88
73.32
10,70
72.66
Lin
14.77
83.31
14.35
41.65
14.38
75.33
14.13
82.04
14.39
73.94
14.52
72.78
BM
17.07
84,85
17.04
42.68
17.20
75.40
16.25
82.78
17.82
74.57
18.54
73.87
400
FBs
19.58
85.14
18.66
43.86
19.28
76.74
18.73
83.51
18.73
75.12
19.05
74.17
FBc
19.31
85:20
18.45
43.83
19.34
76.31
18.52
83.42
18.63
75.20
18.71
73.95
RLB
19.83
85.77
18.97
43.70
19.42
77.10
19.15
83.70
19.06
75.06
19.68
75.94
RLB-AFL
22.06
87.63
20.14
45.94
21.67
79,47
20.71
85.24
21.65
77.91
20.91
77.89
21.94
19,10
20,38
78.11
20.24
84,99
20.22
76.69
77.38
w/o
20.43
86,28
19.56
44.46
20.79
78.75
20.53
84.17
20.86
77.03
20.44
76.52
target network for Q is updated once every 30 training steps (i.e., C 30). The window size for
historical states is fixed at 40 (i.e., W = 40), and the number of clusters is set to 10 (i.e., K == 10). The weight assigned to the incoming instance during the M-step is set to 0.5 (i.c., 0.5). Each recruited DO undergoes 30 local training epochs, with a batch size of 256.
We employ the following two metrics to assess the compared approaches: 1) Utility: It quantifies the total reputation of DOs enlisted by the corresponding target DC upon reaching either the bid request limits or the budget limit. 2) Test Accuracy (Acc): Acc denotes the accuracy of the FL models achieved until reaching either the budget limit the limits on bid requests.
Evaluation Metrics:
5.2 Results and Discussion
To perform a comprehensive comparison of all nine bidding methods, experiments are carried out on six datasets with budgets varying from low to high among (100, 200, 400).
Table I illustrates the outcomes of various comparison methods under the IID scenario. It can be observed that the proposed RLB-AFL method consistently achieves the best performance among all the comparison methods in terms of both test accuracy and utility across all three budget settings and all six datasets. In particular, compared to the best- pessfrrrroring baseline, RLB-AFI improves the total utility and the test accuracy of the resulting FL model by 12. 18% and 2.93%, respectively. Table 2 iillustrates the outcomes of various comparison methods under the Non-lID scenario. The results align with the performance shown in Table I with the proposed RLB-AFL improving the test accuracy by 3.19% average under the Non-IID scenario.
Const and Rand perform poorly compared to other methods due to their disregard for DOs' utility in their formulation. Among all the other comparison methods, Bmub and Lin exhibit inferior performance, with Lin being more effective than Bmub. This can be attributed primarily to the introduction of randomness in the bidding strategy of Bmub. The remaining five comparison methods
8

---

Table 2: FL model accuracy (%) comparison across different datasets and budget settings under
the Non-IID scenario. "Bud" means budget. 10% and 25% represent 10% and 25% noisy data,
respectively.
MNIST
CIFAR
FMNIST
EMNISTD
EMNISTL
KMNIST
Bud
Method
10%
25%
10%
25%
10%
25%
10%
25%
10%
25%
10%
25%
Coast
66.09
70.06
13.04
13.66
59.98
59.31
76.94
76.67
64.25
63.76
60.12
59.22
Rand
68.77
67.10
10.61
10.76
61.36
60.77
75.58
78.24
63.50
63.15
59.19
58.52
Bmuh
70.10
70.85
15.02
13.63
62.12
61.60
77.27
77.76
66.18
65.68
63.09
61.78
Lin
71.95
71.14
18.47
17.76
64.08
64.09
78.45
77.88
65.47
64.75
63.23
62.90
100
BM
72.18
71.91
19.52
19.59
79.35
78.83
66.
FBs
73.05
72.68
23.37
22.48
70.67
70.54
79.34
78.78
67.34
66.63
65.86
64.75
FBc
73.45
74.12
23.25
22.59
71.04
70.92
79.77
79.30
66.48
65.61
65.21
64.33
73.78
73.94
23.57
22.97
71.41
71.70
79.86
79.10
67.10
65.98
RLB AFL
74.84
74.88
25.79
25,42
72.94
73.55
80.46
81,80
69,22
67.47
68.38
65.39
w/o c
74.13
74.24
24.66
23.95
72.33
72.61
80.05
80,58
68.29
66.94
66.88
64.86
w
74.36
74.52
24.83
24.28
72.47
72.84
80.19
80.73
6%
67.26
65.12
Const
69.86
68.12
10.74
10.97
62.25
61.59
77.91
77.69
67.14
66.64
61.33
58,33
Rand
69.38
69.17
10L.31
10.28
62.10
61,32
78.56
78.35
67.66
67.27
62,11
58.42
Bmub
71.55
71.04
13.34
13.13
63.14
62.84
79.22
78.74
68.39
67.89
64.68
63.25
Lin
72.49
71.52
18.91
18.28
64.32
64.27
79.28
78.80
69.33
68.88
67.58
66.37
200
BM
73.24
72.86
20..3
20.20
66.81
67.82
80.36
79 82
69.
66.16
68.39
68.02
FBs
74.19
73.18
23.67
23.08
71.70
71.79
80.13
79.65
68.79
68.17
68.95
69.00
FBc
74.03
73.51
23.46
22.87
71.76
71.71
80.25
79.84
69.76
69.09
68.63
67.53
RLB
75.16
73.69
23.68
23.32
71.20
71.05
80.34
79 888
69, 29
68, 69
69.61
70.49
RLB-AFL
77.62
75.67
24.88
25.64
73.93
73.97
82.86
81.55
71.41
70.72
71.75
71.86
w/o
75.93
74,44
23.92
24.58
72.02
72.68
81.49
80.38
70.52
69.26
70.46
70.98
w/o e
77.34
74.68
24.41
24.76
72.55
72,94
82.23
80.77
70.93
69.84
71.11
71.26
Const
70.52
69.43
17.07
16.99
61.89
60.96
78.42
78.17
67.56
67.12
67.92
68.42
Rand
68.66
20.84
20.58
62.72
62.05
78.59
78.50
68.36
Bmub
71.87
71.06
21.96
20.98
63.86
63.64
79.82
79.30
69.04
68.57
69.25
68.95
Lin
72.60
71.81
24.00
23.29
65.49
65.47
79.86
79.37
69.94
69.52
69.90
69.46
400
BM
74.57
73.79
25.33
24.27
66.92
80.76
80.28
71.12
70.77
FBs
75.39
74.46
26.19
25,06
71.03
70.72
81.19
80.65
71.09
70.62
71.27
71.16
FBc
75.28
74.53
25.93
24.82
72.00
71.98
81.13
80.60
71.26
70.81
70.77
69.97
75.08
26.50
25.39
72.28
72.27
81.40
80.88
70.99
71.75
71.21
RLB-AFL
76.54
76,43
27.86
26.79
73.76
74.78
82.69
82,13
74.25
73.39
72.99
72.15
w/o c
75.89
75.94
27. 12
26.05
72.77
73.41
81.99
81.50
72.68
72.31
72.06
71.66
w/eer
76.07
76.15
27.39
26.22
73.16
73.95
82.33
81.74
73.07
72.37
71.94
consistently exhibit superior performance compared to the aforementioned four simpler approaches. This improved performance can be attributed to the incorporation of auction records, which encompass both bidding records and auction history, as well as the adoption of machine learningrreinforcemenn learning frameworks. BM, FBs, and FBc underperform RLB and RLB-AFL as they belong to the category of static bidding methods, lacking adaptability to the highly dynamic auction environment of AFL. Compared BM. FBs and FBc perform better. This can be attributed to the fact that these two methods use a specially designed bidding function to model the market price distribution, enhancing the accuracy of bid cost expectations. In BM, the market price distribution is obtained by marginalizing the prediction of the market price density of bid requests, which may result in overfitting. Nevertheless, these three bidding methods are formulated as either non-linear or linear functions, trained on historical auction data utilising heuristic approaches. When these functions are exposed to new auctions, which might differ from the historical ones due to the dynamism of the AFL market, achieving consistent desired outcomes becomes challenging. Although RLB adopts dynamic programming to enhance its bidding process, it not specifically designed for AFL DCs, and might face challenges related to state sparsity, potentially leading to poor performance in AFL settings. This limitation has been effectively addressed by RLB-AFF AFL. Furthermore, RLB AFL integrates an enhanced e-greedy policy into its framework to achieve advantageous trade-off between exploration and exploitation.
We created two ablated versions of RLB-AFL: 1) w/o e: excluding the states clustering part from RLB-AFL. 2) w/o e c: the proposed c-greedy policy in RLB-AFL is replaced by the general c-greedy policy. These modifications is to examine the impact of incorporating the states clustering operation and the enhanced e-greedy policy into RLB-AFL. Tables and present the results. It can be observed that RLB-AFL outperforms its ablated variants in terms of the total utility and accuracy FL models. Therefore, the two proposed designs are effective and improve the performance of AFL.
Ablation Study:

---

Sensitivity Analysis on Number of Clusters: To see the impact of the GMM cluster number on
RLB-AFL, we vary the number of GMM clusters from {3, 5, 10, 15, 20). The averaged aceuracy of the FL. models under the 400 budget settings is shown in Table 3 Initially, as the number of clusters increases, there a noticeable ascent in accuracy, followed by subsequent decline. This trend suggests that higher cluster count initially yield a more precise state mapping function. However, excessive growth leads to increased GMM representation size and state sparsity. Within experimental framework, it becomes apparent that selecting cluster size ranging between 10 to 15 leads to optimal outcomes for the model user. This range strikes balance, steering clear of the limitations associated with smaller cluster count, while also avoiding the over-expansion of GMM representation that triggers state sparsity.
Table 3: Accuracy (%) under various number of GMM clusters (K)
K
MNIST
CIFAR
FMNIST
EMNISTD
EMNISTL
KMNIST
3
85,54
41.62
75.93
83.11
13.49
70.33
5
86.09
43.88
77.42
83.85
74.84
72.41
10
87.63
45.94
79.47
85.24
77.91
77.89
15
86.21
43.46
78.38
82.93
75.72
75.84
76.59
74.65
Conclusions
To address the limitations of static bidding strategies in dynamic AFL markets, propose RLB-AFL, a novel RL-based bidding method for DCs. frames bidding as a w-control problem using a DQN architecture. Given the nigh-dimensional, continuous state space, including utility, budget, and time, training a generalizable RL model is challenging. RLB-AFL tackles this with Gaussian mixture model-based soft clustering and refined e-greedy policy to balance exploration and exploitation. However, while RLB-AFL and other methods focus solely on competition among DCs, they overlook potential collaboration, which can indirectly influence behavior. Future work will incorporate these complex inter-DC relationships.
Acknowledgments
The research is supported, in part, by the Ministry of Education. Singapore, under its Academic Research Fund Tier (RG101/24); the RIE2025 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) (Award 12301 administered by A*STAR, well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL).
References
[1] Fan, T., H. Gu, al. Ten challenging problems in federated foundation models. IEEE Transactions on Knowledge and Data Engineering, 2025.
[2] Sun, H., X. Tang, C. Yang, et al. Hifi-gas: Hierarchical federated learning incentive mechanism enhanced gas usage estimation. In he 36th Annual Conference Innovative Applications of Artificial Intelligence (IAAI-24), pages 22824-22832. 2024.
[3] Tang, X., H. Yu, R. Tang, et al. Dual calibration- personalised federated learning. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI'24) pages 4982-4990. 2024.
[4] Qi, Z., L. Meng, Z. Chen, et al. Cross-silo prototypical calibration for federated learning with non-iid data. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3099-3107. 2023.
[5] Qi, Z., L. Meng, Z. Li, et al. Cross-silo feature space alignment for federated learning on clients with imbalanced data. In The 39th Annual AAAI Conference on Artificial Intelligence (AAAI-25), pages 19986-19994, 2025.
[6] Tang, Y.P.., C. Ren, Tang, et al. Efficient heterogeneity aware federated active data selection. In Proceedings of the 42nd International Conference Machine Learning (ICML'25) 2025.
10

---

[7] Meng, L., Z. Qi, L. Wu, et al. Improving global generalization and local personalization for federated learning, IEEE Transactions on Neural Nerworks and Learning Systems, 36, 2024.
[8] Zhan, Y., J. Zhang, Z. Hong, et al. A survey of incentive mechanism design for federated learning. IEEE T EMERG TOP COM, 10(2):033-1044, 2021.
[9] Khan, L. U., S. R. Pandey, N. H. Tran, et al. Federated learning for edge networks: Resource optimization and incentive mechanism. IEEE Commun Mag, 58(10)888-93, 2020.
[10] Tang, X. Stakeholder-oriented decision support for auction-based federated learning. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI224) pages 8514 8515. 2024.
[11] Tang, X., H. Yu. Reputation-aware revenue allocation for auction-based federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, vol. 39, pages 20832-20840. 2025.
[12] Tang, X., H. Yu, X. Li, et al. Intelligent agents for auction-based federated learning: A survey. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI 22)) pages 8253-8266 2024.
[13] Tang, X., H. Yu. Agent-oriented joint decision support for data owners in auction- based federated learning. In Proceedings of the 2024 IEEE International Conference on Multimedia and Expo (ICME'24) (4M) 2023.
[14] Tang, X., H. Yu, X. Li. Multi-session budget optimization for forward auction-basee federated learning. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25) 2025.
[15] Tang, X., H. Yu. muxiit-maximizzin bidding strategy for data consumers in paction-based federated learning. In Proceedings of the 2023 IEEE International Conference on Multimedia and Expo (ICME'2) (33) pages 330-335 2023.
[16] Jiao, Y., P. Wang, D. Niyato, et al. Toward an automated auction framework for wireless federated learning services market. IEEE Transactions on Mobile Computing, 20(10):3034 3048, 2020.
[17] Batool, Z., K. Zhang, M. Toews. Block-racs: Towards reputation-aware client selection and monetization mechanism for federated learning. ACM SIGAPP Applied Computing Review, 23(3):49-66, 2023.
[18] Le, T. H. N. H. Tran, Y. K. Tun, et al. Auction based incentive design for efficient federated learning cellular wireless networks. In Proceedings of the 2020 IEEE wireless communications and networking conference (WCNC'20), pages 1-6. 2020.
[19] Zhou, R., J. Pang. Z. Wang, al. A truthful procurement auction for incentivizing heterogeneous clients in federated learning. In Proceedings of the 4/ IEEE International Conference on Distributed Computing Systems (ICDCS'21), pages 183-193. 2021.
[20] Yuan, Y., L. Jiao, K. Zhu, et al. Incentivizing federated learning under long-term energy constraint via online randomized auctions. IEEE Transactions Wireless Communications, 21(7):512955144 2021.
[21] Wu, L., S. Guo, Z. Hong, et al. Long-term adaptive veg auction mechanism for sustainable federated learning with periodical client shifting IEEE Transactions on Mobile Computing. 2023.
[22] Zhang, J., Y. Wu, R. Pan. Incentive mechanism for horizontal federated learning based on reputation and reverse auction. In Proceedings of the Web Conference 2021 (WWW'2/), page 947-956. 2021.
[23] Zhang, J., Y. Wu, R. Pan". Online auction-based incentive mechanism design for horizontal federated learning with budget constraint. arXiv preprint, page 2201.09047, 2022.
11

---

[24] Deng, Y., F. Lyu, J. Ren, et al. Fair: Quality- aware federated learning with precise user incentive and model aggregation. In Proceedings of the 40th IEEE International Conference on Computer Communications (INFOCOM 21). 2021.
awwar
[25] Yang. Y., al. Bara: Efficient incentive mechanism with online reward budget allocation in cross-silo federated learning. arXiv preprint vrxiv:2305.05221, 2023.
[26] Batool, Z., K. Zhang, M. Toews. Fl-mab: client selection and monetization for blockchain- based federated learning. In ACM SIGAPP Applied Computing Review, pages 299-307. 2022.
[27] Tang, X., H. Yu. aairness-aware reverse auction-based federated learning. IEEE Internet of Things Journal, 12(7):8862 8872, 2024.
[28] Myerson, R. B. Population uncertainty and poisson games. International Journal of Game Theory, 27(3):375-392 1998.
[29] Tang, X., H. Yu. Competitive-ccoperatte multi-agent reinforcement learning for auction-based federated learning. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI'2), pages 1262-4270 2023.
[30] Tang, X., H. Yu, Li, et al. A bias-free revenue-maximinn bidding strategy for data consumers in auction-basse based federated learning. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCA/'44, pages 4991 19999.99... 2024.
[31] Tang. X., H. Yu. cost-aware utility-maximine maximizing bidding strategy for auction-basee federated learning. IEEE Transactions on Neural Networks and Learning Systems, 36(7):12866-2287) 2024.
[32] Vincent, D. R. Bidding off the wall: Why reserve prices may be kept secret. Journal of Economic Theory, 65(2):575-584, 1995.
[33] Tang, X., H. Yu. Towards trustworthy am-oereee real-time time bidding for online advertisement auctioning. ACM Computing Surveys, 57(6):1-36, 2025.
[34] Dantzig, G. B. Discrete-variable extremum problems. Operations research, 5(2):266-288, 1957.
[35] Shi, Y. H. Yu. Faimess--wware client selection for federated learning. In Proceedings of the 2023 IEEE International Conference on Multimedia and Expo (CCM'233), pages 144-399.90 2023.
[36] Liu, Z., et al. GTG-Shapley: Efficient and accurate participant contribution evaluation in federated learning, ACM Transactions on Intelligent Systems and Technology, 13(4):1-21, 2022.
[37] Shapley, L. S., et al. value for n-person games. 1953.
[38] Josang, A., R. Ismail. The beta reputation system. In Bled eConference, vol. 5, pages 2502-2511. Citeseer, 2002.
[39] Mnih, V., et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
[40] Littman, M. L. Markov games framework for multi-agent reinforcement learning. In
Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[41] Tang, X., T. Wang, H. Yang, et al. Akupm: Attention-enhanced knowledge-aware user pref- erence model for recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (KDD') 19), pages 1891-1899. 2019.
[42] Zhou, G., N. Mou, Y. Fan, al. Deep interest evolution network for click-through rate prediction. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), pages 5941-5948. 2019.
[43] Rasmussen, C. The infinite gaussian mixture model. Proceedings of the 13th International Conference Neural Information Processing Systems (NIPS'99), 99). 1999.
12

---

[44] Wu, D., X. Chen, X. Yang, et al. Budget constrained bidding by model-free reinforcement learning in display advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM' 18), pages 1443-1451. 2018.
[45] Xiao, H., K. Rasul, R. Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint, page 1708.07747, 2017.
[46] Cohen, G., S. Afshar, J. Tapson, et al. EMNIST: Extending MNIST to handwritten letters. In Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN' 17), pages 2921 2921-2926. 2017.
[47] Clanuwat, T., M. Bober-Irizar, A. Kitamoto, et al. Deep learning for classical japanese literature. arXiv preprint, page 181 18.001118, 2018.
[48] Zhang. W., S. Yuan, J. Wang. Optimal real-time bidding for display advertising. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD' (1D, pages 1077-1086. 2014.
[49] Lee, K.-c,, Orten, A. Dasdan, al. Estimating conversion rate display advertising from past erformance data. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD'12), pages 768-776. 2012.
[50] Perlich, C., B. Dalessandro, R. Hook, et al. Bid optimizing and inventory scoring in targeted online advertising. In Proceedings of the 18th ACM SIGKDD international conference Knowledge discovery and data mining (KDD 12), pages 804-812. 2012.
[51] Ren, K., W. Zhang, K. Chang, et al. Bidding machine: Learning to bid for directly optimiz- ing profits in display advertising. IEEE Transactions on Knowledge and Data Engineering, 30(4):645 659, 2017.
[52] Tang, X., H. Yu. Efficient large-scale personalizable bidding for multi-agent auction-bast based federated learning. IEEE Internet of Things Journal, ((15):26518-253)) 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions and the scope claimed in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should elearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the proposed method has been discussed in the Conclusions section.
Guidelines:
13

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions have been explained in the preliminary section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The experimental details have been explained in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
14

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the node once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ gabbic/guidess (oodeSumiissiionooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc cc/ http:///llgggg..eee/eeeeeeuueeee http:///oooetubmsssooooooooo.ooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
15

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All these have been explained in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All these have been explained in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: These have been explained in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
16

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://p//ppoooii....coaaaaaaaaa
Answer: [Yes]
Justification: The research conducted in the paper conform and NeurIPS code of ethics. We will relese the code once the paper has been accepted.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This paper presents work whose goal is to advance the field Machine Learning. There many potential societal consequences of work, none which we feel must be specifically highlighted here.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The model has been well safeguarded.
Guidelines:
The answer NA means that the paper poses such risks.
17

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original datasets and references have been properly cited in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new asset of this paper the proposed model. has been well documented and detailed in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
18

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy //tp:::////erra cc/ //oooffeeencee/225//// for what should or should be described.
19


Reviews:
[{"review_id": "KICfODZRTW", "reviewer": "Reviewer_3okG", "summary": "This paper focuses on incentive mechanisms in federated learning, particularly auction-based approaches. The authors propose a reinforcement learning-based bidding strategy, and the experimental results demonstrate the effectiveness of the proposed method.", "questions": "Please see weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper is well-organized and easy to follow. \n\n2. The authors conduct experiments on six widely used datasets to evaluate the effectiveness of the proposed method.\n\n\nWeaknesses\n1. The technical contribution of the paper appears to be somewhat limited. The proposed method primarily integrates several well-established techniques, namely, DQN, GMM, and the ϵ-greedy policy.\n\n2. The authors point out a limitation of existing works, stating that they are generally static approaches (line 39), and claim that their proposed RLB-AFL approach addresses this issue. However, reinforcement learning-based bidding strategies have already been explored in prior work (e.g., [9,46]). It would be helpful for the authors to clarify the relationship between their approach and these earlier methods, highlighting what is novel about RLB-AFL. \n\n3. Some important related works are missing from the literature review. The authors are encouraged to consider incorporating the following references into the related work section:\n(1) Juan Li, Zishang Chen, Tianzi Zang, Tong Liu, Jie Wu, Yanmin Zhu: Reinforcement Learning-Based Dual-Identity Double Auction in Personalized Federated Learning. IEEE Trans. Mob. Comput. 24(5): 4086-4103 (2025)\n(2) Xiaoli Tang, Han Yu: Reputation-aware Revenue Allocation for Auction-based Federated Learning. AAAI 2025: 20832-20840\n(3) Xiaoli Tang, Han Yu: Fairness-Aware Reverse Auction-Based Federated Learning. IEEE Internet Things J. 12(7): 8862-8872 (2025)\n\n4. The authors should check the formatting of references [8]–[10], as these entries currently lack author information.", "was_revised": true, "final_justification": "Most of my concerns have been addressed, and after reading the other reviewers' comments, I would like to raise my score to \"borderline accept\".", "rebuttal": "We sincerely appreciate your encouraging and insightful feedback. Please find our detailed point-by-point responses to your questions below.\n\n**Weakness 1:** We appreciate your insightful comment. While our method incorporated established tools like DQN and GMM, the novelty lies in how they are strategically incorporated and adapted to address key, previously unresolved challenges in auction-based federated learning (AFL).\n\n1）*A novel problem formulation: $\\omega$-Controlled Bidding Strategy*\n\nInstead of directly learning bid values as in prior RL-based approaches [9][46], we adopt a theoretically grounded formulation that learns a scaling factor $\\omega$, which modulates the bid according to the formula $b_i = v_i / \\omega$. This strategy decouples Q-values from raw utility values, leading to significantly more stable learning, especially in dynamic markets with varying utility scales. The approach is anchored in auction theory, ensuring convergence to optimal bidding strategies and robust generalization across tasks. Even without enhanced exploration (as shown in the “w/o enhanced $\\epsilon$” ablation), the $\\omega$-control mechanism alone outperforms all baseline methods (including the bidding method based on reinforcement learning, i,e,. RLB baseline), demonstrating its standalone effectiveness.\n\n2）*First Solution to State Sparsity in AFL: State Clustering*\n\nWe introduce GMM-based state clustering to tackle the problem of state sparsity in AFL environments, where exact states rarely repeat due to dynamic budgets, utilities, and DO arrival sequences. Our solution applies an online, adaptive GMM over historical state trajectories, enabling the agent to generalize across similar but non-identical states. This soft clustering framework improves sample efficiency, reduces redundant learning, and facilitates faster policy convergence. Empirical results confirm its value: the “w/o cluster” ablation shows consistent performance degradation, and sensitivity analysis (Table 3) supports the effectiveness and efficiency of our clustering strategy.\n\n3）*Domain-Specific Exploration Strategy: Enhanced $\\epsilon$-Greedy Policy*\n\nWe propose an enhanced $\\epsilon$-greedy exploration strategy tailored specifically to the unimodal nature of Q-values under $\\omega$-control, a structure derived from auction theory. This domain-aware design promotes targeted exploration, especially when Q-value distributions deviate from unimodality, helping the agent escape suboptimal bidding regions. As demonstrated in the ablation study, disabling this mechanism results in clear drops in performance, confirming its contribution to both convergence speed and utility maximization.\n\nThese innovations work synergistically: $\\omega$-control stabilizes learning by removing value scale dependence, GMM clustering enhances generalization under sparse states, and enhanced $\\epsilon$-greedy improves exploration efficiency in dynamic environments. Together, they lead to significant performance gains (on average, +10.56% in utility and +3.15% in accuracy across six benchmark datasets) demonstrating the robustness and scalability of our approach.\n\n-----------\n\n**Weakness 2:** Thank you for the insightful comment. While [9] and [46] also adopt reinforcement learning for dynamic bidding, our approach differs in the following key aspects:\n\n1) We learn a scaling factor $\\omega$ rather than directly predicting bids, which improves learning stability across varying utility scales.\n\n2) We apply GMM-based state clustering to address state sparsity and enable generalization across similar auction contexts.\n\n3) We design a domain-specific $\\epsilon$-greedy policy that leverages the unimodal Q-value structure unique to AFL bidding.\n\nWe will revise the paper to clearly articulate these differences in the Related Work section.\n\n-----------\n\n**Weakness 3:** Thank you very much for your insightful suggestions. In the Related Work section, we focus on methods designed to support data consumers (DCs) in AFL, that is, methods that help DCs make bidding or participation decisions.\n\nThe first reference you provided focuses more on assisting the coordinator in decision-making, similar to auctioneer-side mechanisms from an ecosystem perspective, as classified in [Ref1]. The second reference falls clearly under auctioneer-side strategies, rather than DC-oriented methods, again based on [Ref1]. The third reference you mentioned is already included in our manuscript as [22].\n\nWe sincerely appreciate your helpful input and will revise the Related Work section to clarify this classification and reasonably incorporate the first reference. We will also ensure our review covers the most recent and relevant state-of-the-art methods.\n\n-----------\n\n**Weakness 4:** Thank you for pointing this out. We apologize for the oversight. We will correct the formatting  and consistent citation styles in the revised manuscript.\n\nWe appreciate your question and hope our response provides sufficient clarity.\n\n-----------\n\n[Ref1] Xiaoli Tang et al. Intelligent Agents for Auction-based Federated Learning: A Survey. IJCAI'24"}, {"review_id": "RMnKG00Da1", "reviewer": "Reviewer_HmSa", "summary": "The authors propose a reinforcement learning (RL) based bidding method for Data Consumers (DCs), where the bidding process is modeled as an $\\omega$  control problem, with $\\omega$ being a scaling factor for the utility of the Data Owners (DOs). The authors employ Deep Q-learning as the underlying RL method and incorporate historical information into the state space. To enhance exploration, the authors propose a modified $\\epsilon$-greedy strategy. In addition, they employ the Gaussian Mixture Model to manage state space dimensionality and facilitate the learning process. Extensive experiments demonstrate that RLB-AFL outperforms several state-of-the-art methods in terms of utility and accuracy.", "questions": "Q1. Do each of the DCs have access to all DOs?\n\nQ2. What is $\\phi^i$ before Eq. (3)?\n\nQ3. In $\\sum_{i=1}^C \\gamma^{t-1}r^i$, is there a summation over time missing?\n\nQ4. Can this method also be applied to other auction scenarios, such as repeated auctions?", "limitations": "See weaknesses.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nS1. The proposed dynamic AFL learning market problem is significant and addresses a meaningful challenge.The paper presents an innovative application of reinforcement learning in the context of AFL, a relatively unexplored area.\n\nS2. The proposed RLB-AFL model demonstrates superior performance on several benchmarks, suggesting high-quality research and implementation.\n\nS3. The paper is well-written with clear explanations of the methodologies and results.\n\nS4. The approach could significantly impact bidding strategies in dynamic markets, especially for data-driven environments like federated learning.\n\nWeaknesses:\n\nW1. The application scenarios of the proposed method is not so clear. \n\nW2. The experiments could be enhanced by including some auction-based datasets or data from AFL market.\n\nW3. What bidding strategies did the other bidders adopt during the experiments? Did they use similar strategies or different ones?", "was_revised": true, "final_justification": "he authors has addressed most of my previous concerns. Combing with other reviewers' comments, I decide to raise my score.", "rebuttal": "Thanks a lot for your encouraging and insightful feedback. Below, we provide point-by-point explanations to key questions raised in the review comments.\n\n**W1:** Indeed, the proposed method can be adapted for use in most FL settings that include an incentive mechanism with data consumers competing to bid for data owners [Ref1, Ref2, Ref3], particularly in auction-based FL scenarios. A practical real-world application is in gas usage estimation within the power generation and delivery industry [Ref4].\n\n\n**W2:** Since publicly available data from the AFL market is not accessible, and existing auction-based datasets, such as those from online advertising, are not directly applicable to a federated learning setting, we have followed established methods [Ref1, Ref2] to collect and generate the datasets used in our experiments. We will also explore additional auction-based datasets that could potentially be adapted for a federated learning setting.\n\n\n**W3:** As detailed in the experiment section, we compared diverse advanced bidding strategies such as FBs, FBc, among others, by various bidders. This setting is designed to address concerns regarding the potential impact of competitors with advanced bidding strategies. However, if all bidders adopt the same bidding strategy, the differences in their individual historical data could lead to variations in their utility estimation and winning price models. This variance might result in differences in their bidding behaviors. Moreover, in practice, different bidders target diverse data owners due to their distinct training tasks. These factors influence their bids through specific parameters and model approaches in their bidding strategies. Nevertheless, it remains an intriguing area for future research to analyze the dynamics and potential equilibria that arise when all data consumers adopt the same strategy.\n\n\n**Q1:** No, each DC only observes and interacts with the DOs that participate in the auction rounds in which the DC is active. DCs do not have global access to all DOs or their data. This reflects realistic AFL settings, where DOs arrive sequentially and bid responses are only visible to participating parties through the auction process.\n\n**Q2:** $\\phi^i$ denotes the reputation score of DO $i$ as perceived by the target DC.\n\n\n**Q3:** We apologize for the confusion. In this context, $C$ represents time. As explained in the preliminaries, each DO arrival triggers one auction round. Therefore, in our POMDP formulation, each time step corresponds to a DO, and the time horizon is implicitly represented by the number of DOs, i.e., $C$.\n\n\n**Q4:** Yes, our framework is well-suited to repeated auction settings, particularly those involving sequential decision-making under uncertainty. The proposed reinforcement learning-based bidding strategy and adaptive state clustering mechanisms are designed to generalize across auction rounds, making them applicable to other repeated or dynamic auction environments beyond AFL. We will include a brief discussion of this in the revised version to highlight the method’s broader applicability.\n\n-----------\n\n[Ref1] Yutao Jiao et al. Toward an automated auction framework for wireless federated learning services market. TMC, 2020.\n\n[Ref2] Rongfei Zeng et al. Fmore: An incentive scheme of multi-dimensional auction for federated learning in MEC. In ICDCS, pages 278288, 2020.\n\n[Ref3] Palash Roy et al. Distributed task allocation in mobile device cloud exploiting federated learning and subjective logic. Journal of Systems Architecture, 113(2):doi:10.1016/j.sysarc.2020.101972, 2021.\n\n[Ref4] Hao Sun et al, \"HiFi-Gas: Hierarchical Federated Learning Incentive Mechanism Enhanced Gas Usage Estimation,\" in Proceedings of the 36th Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-24), pp. 22824-22832, 2024."}, {"review_id": "APoQvVLALP", "reviewer": "Reviewer_ntVa", "summary": "This paper proposes RLB-AFL, a Reinforcement Learning-based Bidding Strategy tailored for data consumers (DCs) in auction-based Federated Learning (AFL). The method employs a Deep Q-Network (DQN) augmented with a Gaussian Mixture Model (GMM) to manage high-dimensional and sparse state spaces and introduces an enhanced ϵ-greedy policy for better exploration-exploitation tradeoff. Experiments on six datasets show that RLB-AFL significantly outperforms eight baselines under both IID and non-IID settings.", "questions": "1. This work describes the process as a POMDP. But in my understanding, the state in POMDP is NOT equal to the observation, as the agent cannot observe all the state information, different from Line 160. Could authors explain this point clearly?\n\n2. How much improvement does GMM gain over other clustering methods?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The proposed method effectively mitigates the state space sparsity issue in AFL.\n\n2. The experiment results demonstrate the superiority of RLB-AFL compared with baselines. \n\nWeakness:\n\n1. The authors choose GMM as the clustering method. While effective, it relies on a good initialization point and requires high computational complexity. The GMM may not scale well as the number of clusters or the dimensionality increases.\n\n2. The policy is learned solely based on historical data, but ignores the features and potential influence of data at the data owners in the FL process.\n\n3. While this work considers multiple DCs, the method is designed for a single DC in nature, if I understand correctly.\nIn the auction, each DC's decision is made based on its own \"individual perspective\", without modeling the strategies of other DCs or the joint dependencies of the system state.\n\n4. DQN may struggle to converge, especially in environments with sparse states. But the authors do not show the convergence process of the proposed method.", "was_revised": true, "final_justification": "I believe this paper makes proper contribution. But two simplifications make me maintain reservations about its robustness and practicality in complex, real-world applications.", "rebuttal": "We are grateful for your valuable and supportive feedback. Below, we provide detailed clarifications and responses to the issues you have raised.\n\n**Weakness 1 & Question 2:** We appreciate your concern and acknowledge that GMM can be sensitive to initialization and less scalable in high-dimensional settings. However, our implementation is carefully designed to remain efficient in the AFL context:\n\n1) The state representations are low-dimensional (≤ 3), and clustering is applied to short historical sequences (e.g., window size = 40), keeping the input space compact.\n\n2) We use a small, fixed number of clusters ($K$ = 10–20), as supported by sensitivity analysis (Table 3), which avoids overfitting and reduces computational overhead.\n\n3) To improve robustness and efficiency, we adopt an online EM algorithm with warm-start initialization, which incrementally updates cluster parameters without full retraining.\n\n4) Empirically, our method scales well in scenarios with up to 160 data consumers and across six benchmark datasets.\n\n5) The per-step GMM update cost is $O(K)$, and the clustering component contributes minimally to overall system complexity.\n\n6) We have added clarification in the revised implementation details.\n\nTo further justify the effectiveness of our GMM-based clustering module in addressing state sparsity in dynamic AFL environments, we compare it with two widely used and conceptually different clustering methods: K-means and DBSCAN. These methods are selected because they are well-established, computationally comparable, and represent two distinct clustering paradigms (partition-based and density-based, respectively).\n\nWe evaluate all methods on six benchmark datasets under the IID setting with a fixed budget of 400. The table below reports the averaged accuracy (%) of the resulting FL models across datasets:\n\n\n| Method      | MNIST | CIFAR  |FMNIST  |EMNISTD  |EMNISTL  |KMNIST  |\n| :---        |    :----:   |    :----:     |:----:     |:----:     |:----:     |:----:     |\n| Ours (GMM)  | 87.63       | 45.94   |79.47   |85.24   |77.91   |77.89   |\n| K-means\t| 86.44\t| 45.15\t| 77.51\t| 83.03\t| 77.40\t| 77.04| \n| DBSCAN\t| 86.77\t| 45.70\t| 78.64\t| 82.75\t| 76.88\t| 76.75| \n\nIt can be observed that the original proposed method with the GMM clustering methods outperforms methods with K-means and DBSCAN clustering methods. It achieves superior accuracy of FL models.\n\nThis superiority may stem from several key advantages of GMM:\n\n1) Its soft clustering mechanism allows better handling of ambiguous and non-repetitive states common in AFL.\n\n2) The online EM updates adapt effectively to dynamic auction patterns without requiring retraining.\n\n3) Unlike DBSCAN, which struggles with temporal structure and parameter sensitivity, GMM offers robust generalization with a fixed, optimized cluster number.\n\n--------\n\n**Weakness 2:** We appreciate this important observation and would like to clarify that the features and potential influence of data at the data owners are indirectly but effectively reflected in the utility value $v^t_i$, which serves as a key component of our POMDP state representation.\n\nThe detailed feature integration process is as follows:\n\n1) Direct Feature Impact: According to our \"Data Owner Reputation Modeling\" section, the data quality and characteristics of data owners directly affect their reputation through the GTG-Shapley method\n\n2) Reputation-Utility Mapping: The data owner's reputation $\\phi_i$ captures their historical contribution quality and data characteristics through the Beta Reputation System (Eq. 3)\n\n3) State Integration: These data owner features influence the POMDP state through the utility component $v^t_i$ in the observation $o^t_i = (B^t_i, C-i, v^t_i)$\n\n4) Policy Learning: Consequently, our learned policy implicitly considers data owner features and their FL influence through this reputation-based utility estimation\n\nAdvantages of this indirect approach involves:\n\n1) Privacy Preservation: Avoids requiring direct access to private data owner information\n\n2) Dynamic Adaptation: Reputation updates reflect changing data owner behavior and contribution quality over time in FL process\n\n3) Comprehensive Feature Capture: The GTG-Shapley method captures complex, multi-dimensional aspects of data owner value beyond simple static features\n\n--------\n\n**Weakness 3:** Thanks a lot for your insightful comment. Our approach assumes that data consumers (DCs) act independently in a non-cooperative, competitive environment, which reflects realistic AFL market dynamics where entities are self-interesed and do not share strategic information.\n\nAlthough each DC makes decisions from its own perspective, multi-agent interactions are captured indirectly through the shared auction mechanism and dynamic reward feedback. In this way, strategic dependencies among DCs are reflected in the evolving market context.\n\nWe agree that explicitly modeling other agents’ strategies or joint dependencies could be a valuable direction and will discuss this as future work in the revised manuscript.\n\n--------\n\n**Weakness 4:** Thank you very much for your insightful comment. Following your suggestion, we have plotted the average utility per episode over 200 rounds to illustrate the convergence behavior of the proposed method. This figure has been added to the revised manuscript.\n\nDue to the rebuttal policy, we are unable to include figures or external links in this response. However, the results clearly demonstrate that our method achieves stable and rapid convergence, even in dynamic and sparse AFL environments. This stability is largely attributed to the introduced state clustering mechanism and the enhanced ε-greedy exploration, which effectively mitigate state sparsity and improve sample efficiency during training.\n\n--------\n\n**Question 1:** We appreciate the reviewer’s attention to this detail. We agree that in a strict POMDP setting, the true state is not fully observable. In Line 160, we refer to the observation (i.e., the agent’s perceived local state) as the input to the policy, not the true environment state. To avoid confusion, we will revise the wording to clarify that the agent operates under partial observability, and the observation is used as a proxy for the underlying (unobservable) global state."}]

Author rebuttal:

Meta-review: This paper proposes RLB-AFL, a reinforcement learning–based bidding strategy for auction-based federated learning, addressing the limitations of static approaches. The key contributions include a theoretically grounded control-based bidding formulation, Gaussian mixture clustering to mitigate state sparsity, and a domain-specific exploration strategy. Strengths lie in the methodological innovations, strong experimental validation across six datasets, and clear explanations. Weaknesses include limited discussion of multi-agent interactions and the absence of real AFL market data. However, the rebuttal clearly addressed reviewer concerns, clarifying novelty and adding experimental justification. The paper can be accepted, given its technical contributions, strong empirical results, and clear contributions.
Decision: Accept (poster)