Title: Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Federated Learning

Regret

Policy Switching Cost

Communication Cost
Abstract: Motivated by real-world settings where data collection and policy deployment&mdash;whether for a single agent or across multiple agents&mdash;are costly, we study the problem of on-policy single-agent reinforcement learning (RL) and federated RL (FRL) with a focus on minimizing burn-in costs (the sample sizes needed to reach near-optimal regret) and policy switching or communication costs. In parallel finite-horizon episodic Markov Decision Processes (MDPs) with $S$ states and $A$ actions, existing methods either require superlinear burn-in costs in $S$ and $A$ or fail to achieve logarithmic switching or communication costs. We propose two novel model-free RL algorithms&mdash;Q-EarlySettled-LowCost and FedQ-EarlySettled-LowCost&mdash;that are the first in the literature to simultaneously achieve: (i) the best near-optimal regret among all known model-free RL or FRL algorithms, (ii) low burn-in cost that scales linearly with $S$ and $A$, and (iii) logarithmic policy switching cost for single-agent RL or communication cost for FRL. Additionally, we establish gap-dependent theoretical guarantees for both regret and switching/communication costs, improving or matching the best-known gap-dependent bounds.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of designing model-free reinforcement learning (RL) algorithms that are both sample-efficient and communication-efficient. In both single-agent and federated RL settings, existing methods either require large burn-in costs or incur frequent policy switching or communication. To mitigate this issue, this paper proposes two new algorithms: Q-EarlySettled-LowCost for single-agent RL and FedQ-EarlySettled-LowCost for federated RL. These are the first model-free methods to simultaneously achieve near-optimal regret, burn-in costs that scale linearly with the number of states and actions, and only logarithmic switching or communication costs.\n\nThe key idea seems to be the combination of a round-based learning structure with the classic LCB techniques, enabling infrequent updates without sacrificing learning efficiency. The algorithms outperform existing state-of-the-art methods by reducing the switching or communication overhead while maintaining low regret. In the theoretical aspect, the paper gives strong gap-dependent performance guarantees. In terms of technical novelty, the paper introduces new analytical tools to handle technical challenges in regret analysis. Overall, the paper provides practical and theoretically reliable solutions for efficient RL in settings where data collection or communication is costly.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\nStrengths:\n\n1, Theoretically Rigorous: The paper provides tight worst-case and gap-dependent regret bounds for both single-agent and federated reinforcement learning (FRL), matching or improving over existing results. The analysis incorporates nontrivial techniques (e.g., surrogate reference functions, LCB-based early settlement). In terms of technical novelty, the analysis successfully handles double non-adaptiveness, a nontrivial technical barrier in FRL.\n\n2, Algorithmic Soundness: The proposed algorithms—Q-EarlySettled-LowCost and FedQ-EarlySettled-LowCost—are well-motivated and robustly designed. Their use of refined bonuses and structured policy updates yields practical and theoretical benefits. The numerical experiments confirm the regret and cost improvements over strong baselines, including UCB-Advantage and FedQ-Advantage.\n\n**Clarity**\n\nStrengths:\n\n1, I think this paper has good clarity in general. The problem is clearly defined. The similarity and difference from existing works in federated RL is well described in detail. Technical novelty is explicitly mentioned. \nI also really appreciate the illustration in Appendix D, which makes it easier for the reader to understand the federated RL framework which is quite complicated to be honest.\nOther good practices include: Table 1 and 2 provide a clear comparison with existing methods. \n\nThe only issue is that the notation is a bit heavy (for example, page 6). But I think this is just a minor issue and given the theoretical nature and complexity of the federated RL problem setting, I do not think there is a significantly better way to present the theoretical results.\n\n**Significance**\n\nStrengths:\n\n1, Addresses a Key Challenge in FRL: Communication and synchronization costs are one of the key problems in Federated RL. The proposed algorithms directly address these issues with provable logarithmic cost.\n\n\n**Originality**\n\n1, I think the major novelty of the paper is its technical improvement. Specifically, I think the successful adaptation of the existing surrogate reference functions to worst-case regret framework is one of the major originalities. This is an advancement in the field of statistical methodology, which can potentially have broader applicability in not only federated learning but other problem settings as well.\n\n**Minor Weaknesses**\nOn the empirical side, only very simple numerical experiments have been conducted. \nNotation is a bit heavy. \nA lack of proof sketch in the main text (there is, though, in Appendix K).\n\nQUESTIONS:\nDo you think this result can be extend to general cases with continuous state and action space (such as MDP with function approximation)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to simultaneously reduce the **burn-in costs** and **policy switching costs** (for the single-agent RL setting)/**communication costs** (for the federated RL setting). The proposed algorithm improves over [1] by a logarithmic factor in the regret bound and a polynomial factor in the burn-in costs.\n\n[1] Zheng, Z., Zhang, H., & Xue, L. (2025). Federated Q-learning with reference-advantage decomposition: almost optimal regret and logarithmic communication cost.\n\nSTRENGTHS AND WEAKNESSES:\n* **Strengths:**\n    1. The improvements in the burn-in costs are considerable (i.e., polynomial to $S,A,$ and $H$), which could be helpful in the applications;\n    2. This paper presents a comprehensive regret analysis of the proposed algorithms and derives both the high probability and gap-dependent regret bounds.\n\n* **Weaknesses:**\n    1. The presentation is a bit unclear. For example, I think it would be better to put the exact bounds in Tables 1 and 2 to make comparisons between different algorithms. On the other hand, in the Numerical Experiments Section in Appendix B, the algorithm does not compare to [1] in terms of switching/communication costs;\n    2. The techniques utilized in this paper, including the reference-advantage decomposition and the combination of UCB and LCB methods, have been discussed in several references [1, 2], and their combinations and applications to a federated setup seem to be rather direct.\n\n[1] Zheng, Z., Zhang, H., & Xue, L. (2025). Federated Q-learning with reference-advantage decomposition: almost optimal regret and logarithmic communication cost.\n\n[2] Li, G., Shi, L., Chen, Y., Gu, Y., & Chi, Y. (2021). Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning.\n\nQUESTIONS:\n1. I think the notation $\\Omega(\\cdot)$, instead of $O(\\cdot)$, should be used in the lower bounds in Lines 24 and 50. \n2. In the numerical experiments, how do the proposed algorithms compare to FedQ-Advantage regarding communication costs?\n3. Can you explain how FedQ-EarlySettled-Advantage reduces the burn-in costs in the FedQ-Advantage algorithm [1]?\n4. I think visualizing the burn-in costs in Figures 1 and 3 also helps highlight the improvements over the previous algorithms, for example, comparing the number of iterations where those curves \"settle down\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two novel model-free reinforcement learning algorithms, Q-EarlySettled-LowCost for single-agent RL and FedQ-EarlySettled-LowCost for federated RL (FRL). The algorithms achieve near-optimal regret bounds while simultaneously ensuring low burn-in costs (i.e., the number of samples required to reach near-optimality) and logarithmic policy switching or communication costs. The design combines upper confidence bound (UCB) techniques with a reference-advantage decomposition and incorporates lower confidence bound techniques with an early-settling mechanism for reference functions in estimation. Meanwhile, the so-called ``Event-Triggered Termination of Exploration'' technique is also used in the proposed algorithm. Theoretical guarantees are provided in both worst-case and gap-dependent settings. Numerical results demonstrate superior regret performance compared to existing model-free baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n+ The paper proposes the first algorithm that simultaneously achieve near-optimal regret, linear burn-in cost in $S$ and $A$, and logarithmic switching/communication cost under both single agent RL or FRL setting.\n    \n+ The authors provide improved gap-dependent regret bounds and establish the first gap-dependent switching cost guarantee for LCB-based algorithms.\n\n+ Theoretical results are supported by experiments that verify the improved gap-dependent performance in practice.\n\n+ The algorithm is well-engineered: by carefully tuning the bonus terms, it smoothly integrates existing methods such as Q-Early Settled-Advantage and Event-Triggered Termination of Exploration.\n\n\n\nWeaknesses\n\n- The technical contribution seems incremental. The algorithm essentially combines multiple components from prior work under the same setting. Key techniques such as Reference-Advantage Decomposition, LCB-based Early Settlement, and Event-Triggered Termination have all appeared in previous studies. Furthermore, the main analytical challenge, namely the non-adaptiveness of weights and reference functions, is addressed through direct application of surrogate function constructions from earlier studies, rather than by introducing novel analytical tools. While the author(s) have clearly invested substantial effort in assembling these components into a comprehensive (over eighty-page) manuscript, it is debatable whether such a narrowly scoped variation of the tabular RL problem warrants this level of complexity, particularly given the method’s modest empirical performance, as discussed below.\n\n- Empirical performance seems not very practically useful: under $(H, S, A) = (7, 10, 5)$, the regret and switching cost grow with $\\ln T$ at a scale of roughly 1000, indicating the constants hidden in the theory are large and the algorithm may not be practical for moderate horizons.\n\nQUESTIONS:\nThe author(s) may wish to more clearly explain the technical challenges and their technical contribution --- especially which analysis, besides Lemma I.3, is new.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel variant of the Q learning algorithm, and its federated counterpart, achieving both low switching cost and lower burn-in cost.\nThese algorithm, Q-EarlySettled-LowCost and FedQ-EarlySettled-LowCost, combine an upper confidence and a lower-confidence approach, together with the construction of a reference estimator, to reduce the burn-in cost.\nBoth methods are studied in terms of worst case dependence and gap-dependent bounds, achieving low switching cost in all cases.\nIn particular, the federated method achieves regret that grows in $\\sqrt{M}$, where $M$ is the number of agents.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The proposed Q-learning methods are the first methods to achieve both low switching cost and low burn-in period. Reducing the burn-in period is crucial to achieve meaningful guarantees in RL, while low switching cost is essential in federated RL, in order to keep clients synchronized on the same policy without requiring a prohibive number of communications. Providing a method that achieves both is a significant open problem.\n2. The federated algorithm indeed benefits from collaboration with other agents, with a summed regred scaling in $\\sqrt{M}$ (i.e., averaged regret scaling in $\\sqrt{1/M}$).\n3. The proposed construction is a clever combination of upper and lower confidene bounds, allowing for early settlement of the reference function, which in turn significantly reduces the burn-in period.\n\n**Weaknesses**\n1. The proposed method unfortunately requires important number of communications, and grows in $M \\log(T)$ for worst-case bounds and $M \\log(\\log(T))$ for gap-dependent bounds. This may quickly become prohibitive in large-scale federated RL scenarios.\n2. The FL scenario that is considered is the fully heterogeneous scenario, which is not the most common in federated learning. Although I think the paper already presents interesting results, it would be interesting to discuss possible extensions to the non-homogeneous case.\n\nThe following reference, which studies Fed-UCBVI is missing from the discussion of related work, and may provide insights for possible extensions reducing the dependence on M of the communication :\n\nLabbi, S., Tiapkin, D., Mancini, L., Mangold, P., & Moulines, E. (2024). Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agents. arXiv preprint arXiv:2410.22908.\n\nQUESTIONS:\n1. Is there hope to reduce the number of communications, so that it does not scale linearly with $M$ anymore? Could the ideas presented in [1] be applied to this setting?\n2. In the experiments presented in appendix, it seems that the proposed algorithm outperforms all other flavor of Q learning. How does this compare with model-based methods? Overall, I think it would make sense to include these experiments in the main text.\n3. The obtained burn-in cost remains quite large, although it depends only linearly on S, A. Is there hope for further improvements?\n\nLabbi, S., Tiapkin, D., Mancini, L., Mangold, P., & Moulines, E. (2024). Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agents. arXiv preprint arXiv:2410.22908.\n\nLIMITATIONS:\nThe discussion on the cost of communication is missing, although it can quickly become a problem in large scale federated RL cases.\nSimilarly, the relevance of the homogeneity assumption should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}