Title: A Reinforcement Learning-based Bidding Strategy for Data Consumers in Auction-based Federated Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: federated learning

auction-based federated learning

Gaussian Mixture Model
Abstract: Auction-based Federated Learning (AFL) fosters collaboration among self-interested data consumers (DCs) and data owners (DOs). A major challenge in AFL pertains to how DCs select and bid for DOs. Existing methods are generally static, making them ill-suited for dynamic AFL markets. To address this issue, we propose the R}einforcement Learning-based Bidding Strategy for DCs in Auction-based Federated Learning (RLB-AFL). We incorporate historical states into a Deep Q-Network to capture sequential information critical for bidding decisions. To mitigate state space sparsity, where specific states rarely reoccur for each DC during auctions, we incorporate the Gaussian Mixture Model into RLB-AFL. This facilitates soft clustering on sequential states, reducing the state space dimensionality and easing exploration and action-value function approximation. In addition, we enhance the $\epsilon$-greedy policy to help the RLB-AFL agent balance exploitation and exploration, enabling it to be more adaptable in the AFL decision-making process. Extensive experiments under 6 widely used benchmark datasets demonstrate that RLB-AFL achieves superior performance compared to 8 state-of-the-art approaches. It outperforms the best baseline by 10.56% and 3.15% in terms of average total utility


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RLB-AFL, a Reinforcement Learning-based Bidding Strategy tailored for data consumers (DCs) in auction-based Federated Learning (AFL). The method employs a Deep Q-Network (DQN) augmented with a Gaussian Mixture Model (GMM) to manage high-dimensional and sparse state spaces and introduces an enhanced ϵ-greedy policy for better exploration-exploitation tradeoff. Experiments on six datasets show that RLB-AFL significantly outperforms eight baselines under both IID and non-IID settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The proposed method effectively mitigates the state space sparsity issue in AFL.\n\n2. The experiment results demonstrate the superiority of RLB-AFL compared with baselines. \n\nWeakness:\n\n1. The authors choose GMM as the clustering method. While effective, it relies on a good initialization point and requires high computational complexity. The GMM may not scale well as the number of clusters or the dimensionality increases.\n\n2. The policy is learned solely based on historical data, but ignores the features and potential influence of data at the data owners in the FL process.\n\n3. While this work considers multiple DCs, the method is designed for a single DC in nature, if I understand correctly.\nIn the auction, each DC's decision is made based on its own \"individual perspective\", without modeling the strategies of other DCs or the joint dependencies of the system state.\n\n4. DQN may struggle to converge, especially in environments with sparse states. But the authors do not show the convergence process of the proposed method.\n\nQUESTIONS:\n1. This work describes the process as a POMDP. But in my understanding, the state in POMDP is NOT equal to the observation, as the agent cannot observe all the state information, different from Line 160. Could authors explain this point clearly?\n\n2. How much improvement does GMM gain over other clustering methods?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a reinforcement learning (RL) based bidding method for Data Consumers (DCs), where the bidding process is modeled as an $\\omega$  control problem, with $\\omega$ being a scaling factor for the utility of the Data Owners (DOs). The authors employ Deep Q-learning as the underlying RL method and incorporate historical information into the state space. To enhance exploration, the authors propose a modified $\\epsilon$-greedy strategy. In addition, they employ the Gaussian Mixture Model to manage state space dimensionality and facilitate the learning process. Extensive experiments demonstrate that RLB-AFL outperforms several state-of-the-art methods in terms of utility and accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1. The proposed dynamic AFL learning market problem is significant and addresses a meaningful challenge.The paper presents an innovative application of reinforcement learning in the context of AFL, a relatively unexplored area.\n\nS2. The proposed RLB-AFL model demonstrates superior performance on several benchmarks, suggesting high-quality research and implementation.\n\nS3. The paper is well-written with clear explanations of the methodologies and results.\n\nS4. The approach could significantly impact bidding strategies in dynamic markets, especially for data-driven environments like federated learning.\n\nWeaknesses:\n\nW1. The application scenarios of the proposed method is not so clear. \n\nW2. The experiments could be enhanced by including some auction-based datasets or data from AFL market.\n\nW3. What bidding strategies did the other bidders adopt during the experiments? Did they use similar strategies or different ones?\n\nQUESTIONS:\nQ1. Do each of the DCs have access to all DOs?\n\nQ2. What is $\\phi^i$ before Eq. (3)?\n\nQ3. In $\\sum_{i=1}^C \\gamma^{t-1}r^i$, is there a summation over time missing?\n\nQ4. Can this method also be applied to other auction scenarios, such as repeated auctions?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on incentive mechanisms in federated learning, particularly auction-based approaches. The authors propose a reinforcement learning-based bidding strategy, and the experimental results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is well-organized and easy to follow. \n\n2. The authors conduct experiments on six widely used datasets to evaluate the effectiveness of the proposed method.\n\n\nWeaknesses\n1. The technical contribution of the paper appears to be somewhat limited. The proposed method primarily integrates several well-established techniques, namely, DQN, GMM, and the ϵ-greedy policy.\n\n2. The authors point out a limitation of existing works, stating that they are generally static approaches (line 39), and claim that their proposed RLB-AFL approach addresses this issue. However, reinforcement learning-based bidding strategies have already been explored in prior work (e.g., [9,46]). It would be helpful for the authors to clarify the relationship between their approach and these earlier methods, highlighting what is novel about RLB-AFL. \n\n3. Some important related works are missing from the literature review. The authors are encouraged to consider incorporating the following references into the related work section:\n(1) Juan Li, Zishang Chen, Tianzi Zang, Tong Liu, Jie Wu, Yanmin Zhu: Reinforcement Learning-Based Dual-Identity Double Auction in Personalized Federated Learning. IEEE Trans. Mob. Comput. 24(5): 4086-4103 (2025)\n(2) Xiaoli Tang, Han Yu: Reputation-aware Revenue Allocation for Auction-based Federated Learning. AAAI 2025: 20832-20840\n(3) Xiaoli Tang, Han Yu: Fairness-Aware Reverse Auction-Based Federated Learning. IEEE Internet Things J. 12(7): 8862-8872 (2025)\n\n4. The authors should check the formatting of references [8]–[10], as these entries currently lack author information.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}