Title: Projection-based Lyapunov method for fully heterogeneous weakly-coupled MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: weakly-coupled Markov decision processes

fully heterogeneous systems

asymptotic optimality

planning

average-reward Markov decision processes

Lyapunov analysis
Abstract: Heterogeneity poses a fundamental challenge for many real-world large-scale decision-making problems but remains largely understudied.
In this paper, we study the _fully heterogeneous_ setting of a prominent class of such problems, known as weakly-coupled Markov decision processes (WCMDPs).
Each WCMDP consists of $N$ arms (or subproblems), which have distinct model parameters in the fully heterogeneous setting, leading to the curse of dimensionality when $N$ is large.
We show that, under mild assumptions, an efficiently computable policy achieves an $O(1/\sqrt{N})$ optimality gap in the long-run average reward per arm for fully heterogeneous WCMDPs as $N$ becomes large.
This is the _first asymptotic optimality result_ for fully heterogeneous average-reward WCMDPs.
Our main technical innovation is the construction of projection-based Lyapunov functions that certify the convergence of rewards and costs to an optimal region, even under full heterogeneity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes weakly-coupled Markov decision processes (WCMDPs), a classical framework for solving large decision-making problems, under the mean-payoff criteria. An algorithm is given with suboptimality guarantees of $O(1/\\sqrt{N})$, where $N$ is the number of components of the weakly-coupled model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is very well-written, much better than most submissions I've reviewed this year.\n- The results are strong and the technics are new. This is a good paper.\n\n**Weaknesses** I don't see significant weaknesses. I only have a few questions/comments that could improve the presentation of the results and the positioning of the paper with respect to the rest of the literature on MDPs.\n\nQUESTIONS:\n**Questions**\n\n- What is the relation between WCMDPs and factored MDP or constrained MDP?\n- How to go beyond unichain? Did the authors consider weakly-communicating models?\n- l310: “let us assume …”: isn’t this implied by Assumption 1 ?\n- l363, quote: “Note that dividing each term by powers of $\\phi$ is another interesting trick, which makes the Lyapunov function strictly contractive under the optimal single-armed policies, as demonstrated in the proof of Lemma 3.” What do you mean by “strictly contractive”? Do you just mean contractive (Lipschitz-continuity with constant strictly smaller than 1)? Also in the proof of Lemma 3 (app E.2) I only found the proof of the Lipschitzness, not of the contraction property. Please clarify.\n\n**Suggestions**\n\n- Maintaining feasibility in the hard-constraint sense is crucial. The fact that the authors manage to do it is pretty strong and perhaps should be emphasized earlier in the paper / in the exposition of the results; I was wondering about this until the middle of page 6 (paragraph line 225 - 232).\n- What does “ID” mean (line 49 and onward)? Perhaps introduce the full term the first time.\n- Unichain assumption should be stated in the abstract and intro. It is crucial, but it only appears on the last-but-one page (page 8).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the setting of weakly coupled MDPs (WCMDPs), which consist of $N$ arms, each representing a distinct individual MDP. The authors focus on the fully heterogeneous case, where each underlying MDP is governed by a different model. Considering all the models of the individual arms to be fully known, the objective is to maximize the overall reward while satisfying global constraints related to $K$ different cost functions. To this end, the authors propose a policy named \"ID policy with reassignment\", which operates in two phases. The first is the pre-processing phase: for each arm, compute the optimal single-arm policy that prescribes the ideal action the agent should take in each individual MDP. The second is the real-time phase: each MDP is assigned actions from its ideal policy as much as possible without violating the cost constraints, otherwise a neutral policy is applied. By leveraging a Lyapunov function specifically designed for this setting, the authors prove that the proposed policy achieves an asymptotic sub-optimality gap of order $\\mathcal{O}(1/\\sqrt{N})$, where $N$ is the number of arms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \nThe main strength of the paper lies in the quality of the manuscript. The paper is very well written; the authors take the time to discuss every introduced concept, the manuscript is self-contained, the related work is thoroughly reviewed, the notation is clear, and every symbol used is formally defined. Moreover, the authors consider a setting that reflects real-world scenarios, and the proposed ideas are intuitive and well presented. The paper appears to be theoretically sound, and the methodological analysis seems novel (at least to the best of my knowledge, although this is not precisely my area of expertise).\n\n**Weaknesses**  \nThe main weakness I found in the paper is that the authors assume full knowledge of all the tabular MDPs. This assumption makes the problem significantly easier, as it allows the proposed policy to solve each MDP exactly. Additionally, I believe the setting would benefit (and become more challenging) by introducing a finite budget for acting in the MDPs. The paper would also gain from a more detailed discussion of the motivational examples, which are just listed; elaborating on them could help the reader better appreciate the relevance of the proposed setting. That being said, I do not believe this is strictly a reinforcement learning paper, as it does not involve learning a policy through interaction with the environment(s). Furthermore, I would have appreciated at least a toy simulation, to provide a practical illustration of the theoretical results.\n\nQUESTIONS:\n1. Do you have any insights on how results extend to the case in which we consider continuous sub-MDPs?\n2. What are the main technical challenges in considering learning policies in unknown sub-MDPs? For instance, is would be possible to apply concepts by constrained reinforcement learning?\n3. In the paragraph at lines 147--158, is it really necessary to indicate the policy $\\pi$ when considering the vectors of MDPs states and actions?\n4. Concerning the definition of $R^*(N,\\boldsymbol{S}_{0})$, is it the optimal reward only or is it accounting for constraint satisfaction too?\n5. In Lemma 1, what happens to constraint violation when employing the solution of the relaxed problem?\n\nLIMITATIONS:\nThe authors discuss widely the limitations of their work throughout the whole manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper studies the problem of fully heterogeneous, average-reward, budget-constrained weakly-coupled MDP (WCMDP) under the setting of planning (transition kernels and reward functions are known). For the first time, they design the algorithm of $\textit{ID policy with reassignment}$ and prove it achieves an $O(\sqrt{N})$ optimality gap as the number of arms $N$ grows to infinity. The authors novelly design a clean projection-based Lyapunov function analysis that bypasses the need for symmetry or state aggregation. But this paper comes without any experiments.

STRENGTHS AND WEAKNESSES:
Strength:
1. The presentation is good. Writing is easy to follow. 
2. For the first time, this paper closes the gap and provides the first asymptotic optimality result for average-reward, hard-constrained WCMDP. Also this paper is technically solid.
3. Technically, they provide a novel construction of a Lyapunov function based on projected features across heterogeneous arms.

Weakness:
1. I am not convinced by the problem setting's practicality. The average reward setting is not widely used since it is not easy to learn, and not as stable as the discounted reward setting. In this case, the authors should provide at least a synthetic toy case. Second, the hard instantaneous cost constraint is fixed for each time. It might be more practical to consider the cumulated cost for a period of time? 
2. It will be better to provide more comparisons on the techniques with the discounted WCMDP literature, since the authors mentioned the optimality result is provided in that case.

QUESTIONS:
please refer to weakness

LIMITATIONS:
I didn't find the discussion about the limitations and societal impact. Since this work is theoretical in nature, it is acceptable for that.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a class of problems known as weakly coupled Markov Decision Processes. In this framework, the linkage between individual components, or arms, is through joint cost constraints, while each arm operates as its own independent MDP. The objective is to solve for the optimal policy for the entire system, keeping these shared constraints in mind. Typically, the state space for the entire system is exponential in the number of arms, making the problem difficult to solve directly. To approach this, the paper considers the Linear Programming relaxation of the original MDP. The best policy obtained from this LP relaxation provides an upper bound on performance, since the cost constraint is relaxed to only require satisfying a time-averaged version of the constraint. The authors then propose an \"ID reassignment\" based policy to determine the optimal arm-pulling strategy, such that the optimal stationary distribution obtained from the LP optimization problem is eventually tracked. They analyze the regret associated with the LP optimal solution, which serves as an upper bound on the actual regret with respect to the original MDP. The rest of the paper is dedicated to proving that the ID assignment indeed tracks the LP optimal policy in an asymptotic sense. The technical novelty lies in the proof technique, where the a unique Lyapunov function is synthesized for this fully heterogenous setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.  The paper's primary claim is that it provides the first asymptotic optimality result for fully heterogeneous average-reward WCMDPs. This is a significant step beyond existing work, which has focused on homogeneous or simpler typed heterogeneous settings.\n2. The design of a novel Lyapunov function is a theoretical contribution which can be leveraged as a tool to analyze other heterogeneous systems. \n\nWeaknesses:\n\n1. A more complete work would characterize the performance of the ID-reassignment policy in a finite time sense. The paper presents the asymptotic performance where the performance is characterized over the infinite time horizon. \n2. As the number of arms $N$ tends to infinity, the assumption on the finiteness of mixing time may fall through.\n3. It is unclear as to what the approach should be if the LP relaxation has multiple optimal solutions. \n4. $C_{ID}$ seems to scale really badly with the problem parameters, making the result vacuous in large complex settings.\n\nQUESTIONS:\n1. In equation 8, is there a reason for choosing half the budget per cost constraint as the limit to determine the active constraints? Or is it an arbitrary choice?\n2. In equation 7, is there any intuition beyond setting the policy to be uniform in transient states?\n3. Does line 259 imply that as the arms with lower indices approach their stationary values, the cost constraints are satisfied in a sense that lets more arms follow their LP-optimal policies?\n4. It will help to include the intuition behind Algorithm 1 in the main body of the paper since this is the main algorithmic contribution of the paper. \n5. Is there any intuition behind choosing the cost and reward vectors as the feature vectors?\n6. Is the regret quantified with respect to the best stationary optimal policies or best optimal policies in a wider class of policies (as initially pointed at)?\n\nMinor comment: In the beginning the authors mention that lim inf and lim sup of the average cost are the same in the finite state action setting. However, this is true subject to certain constraints on the Markov chain induced by the policy. Unichain and aperiodicity are sufficient (as assumed later) but this assumption needs to be made clearer earlier on.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-13"}