Title: Regret Lower Bounds for Decentralized Multi-Agent Stochastic Shortest Path Problems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Lower Bounds

Decentralized MARL

Stochastic Shortest Path

Reinforcement Learning

Feature Design
Abstract: Multi-agent systems (MAS) are central to applications such as swarm robotics and traffic routing, where agents must coordinate in a decentralized manner to achieve a common objective. Stochastic Shortest Path (SSP) problems provide a natural framework for modeling decentralized control in such settings. While the problem of learning in SSP has been extensively studied in single-agent settings, the decentralized multi-agent variant remains largely unexplored. In this work, we take a step towards addressing that gap.  We study decentralized multi-agent SSPs (Dec-MASSPs) under linear function approximation, where the transition dynamics and costs are represented using linear models. Applying novel symmetry-based arguments, we identify the structure of optimal policies. Our main contribution is the first regret lower bound for this setting based on the construction of hard-to-learn instances  for any number of agents, $n$. Our regret lower bound of $\Omega(\sqrt{K})$, over $K$ episodes, highlights the inherent learning difficulty in Dec-MASSPs. These insights clarify the learning complexity of decentralized control and can further guide the design of efficient learning algorithms in multi-agent systems.


Peer reviews:
comments: SUMMARY:
The authors study decentralized multi-agent SSPs (Dec-MASSPs) under linear function approximation, where the transition dynamics and costs are represented using linear models.
Their main contribution is the first regret lower bound for this setting based on the construction of hard-to-learn instances for any number of agents.
I find this work very interesting!

STRENGTHS AND WEAKNESSES:
Strengths:
The problem studied is central in decentralized RL.

Weaknesses:
See Questions.

QUESTIONS:
Page 1:
Can you please expand the discussion on SSP so that a non-expert can better relate to the problem you are studying?

Page 2:
Can you please elaborate on (iii)?
I think that this explanation is very handwavy :)

Page 3:
Can you please elaborate on the caption of Table 1?

Is there any related work from the field of computational complexity?

Page 5:
In Equation (2), why is there a negative term?
Why are you subtracting $K \cdot V^*(s_{\rm init})$?

Page 6:
Line 238:
This explanation is nice but I am a bit confused.
Why are there exponentially many policies that differ in just two components of the global action in a single state, etc.?

Page 7:
I do not understand Equation (6).
Why do you need to sum over all $r$?

Page 8:
Can you please explain how use Pinsker's inequality to bound the regret?

Why is $N_{i, j}(\theta)$ unbounded?

Page 9:
Can you please expand the future work paragraph? :)

LIMITATIONS:
None.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines a regret lower bound for a class of multi-agent stochastic shortest path problems. The general model assumes that $n$ agents are either in the starting or the goal state, and a linearly defined transition function stochastically allows this transition based on an action which is a binary vector of dimensionality $d$ over $\\{-1,1\\}$. This implies that the optimal policy is given by letting each agent select the action minimizing the likelihood of staying in the starting state. Though the cost is uniform for each action, the transition function is dependent on how many agents are still in the starting state. \nThe authors prove two main theorems for their setting: The first basically shows that the optimal policy should constantly select the action for each agent which is maximizing the likelihood of transitioning to the goal. Furthermore, due to the homogenous and independent behaviour of agents, states having the same number of agents positioned on the starting state share the same values. Furthermore, as agents may eventually transit to the goal, the cumulated expected future cost decreases.\nIn the second theorem, the authors compute a lower bound for the expected regret based on this optimal behaviour.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper examines whether, in decentralized multi-agent stochastic shortest path problems with linear cost and transition functions, the already proposed methods might already have an optimal complexity. This is interesting as it answers the question of how well-known algorithms are performing compared to optimal policies and if further research in this direction is required.\nThe idea of finding instances of this problem setting where the optimal policy can be derived and then showing that no algorithm can achieve faster convergence towards the optimal policy also seems feasible.\n\nA drawback of the paper is that it rarely tries to give any intuition behind the assumptions, the design of the problem, or even about the assumption on the actual algorithms, which are lower bounded in the end. Thus, trying to get behind the setting requires a lot of time, and it feels that the authors could have made it far easier for the reader to get an understanding of what is proposed in the paper. \n\nAnother drawback of the paper is that the contribution feels somewhat limited as the setting is rather specific. Thus, it is unclear why the proposed setting is indicative of the broader settings from the related work.\nThough the authors define a problem that allows deriving a lower bound, the authors do not advertise the practical relevance of the setting. Also, it is not clear to what extent the results are transferable to a larger set of MASSP problems. Thus, the results may not be interesting to the broader machine learning community.\n\nTo conclude, the paper examines a very complicated setting from a theoretical point of view by finding and analyzing an instance that allows tractable results. Concerns are that the paper is hard to follow due to a lack of providing intuitions and that the assumptions required to derive  the results are too restrictive to raise attention in the broader machine learning community.\n\nQUESTIONS:\nQ1. Your main claim seems to be Theorem 2, presenting the lower bound. What are the assumptions on the algorithm $\\pi$ that is lower bounded? There are some hints in the proofs, but even from the sketch in Section 5, it remains unclear why the algorithm cannot estimate the transition likelihood of actions any quicker.\n\nQ2. To what extent are LM-MASSP problems indicative of general MASSP problems as defined in the related work? \n\nQ3. Could your results easily be extended to a larger number of nodes beyond start and goal?\n\nLIMITATIONS:\nThe authors describe the main assumptions of their model and stress that the linear transition function is a limiting factor.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the fundamental learning limits in decentralized multi-agent stochastic shortest path (Dec-MASSP) problems with linear function approximation—a setting relevant to areas like swarm robotics and traffic routing, but still theoretically underexplored. While single-agent SSPs have been well explored in the existing literature, extending regret bounds to decentralized multi-agent cases poses major challenges due to exponential state-action spaces and coupled dynamics.\n\nTo this end, the paper establishes the first ever regret lower bound for Dec-MASSPs, proving a bound of Ω(√K) over K episodes. This matches known upper bounds and recovers the single-agent case when the number of agents is one, providing a tight characterization of the problem's learning complexity. Their analysis is based on a technically novel construction of hard-to-learn problem instances with linearly parameterized dynamics and costs.\n\nTo overcome key technical barriers, the paper designs feature representations that yield valid transitions, simplify value function analysis using state partitioning and monotonicity, and bound KL divergence via symmetry in agent distributions. These techniques enable valid analysis despite the exponential state space. Overall, the paper closes a critical gap in theory and offers a foundation for evaluating and designing scalable learning algorithms in decentralized multi-agent systems.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**Quality**\n\n1. Technically rigorous: The paper presents a careful and comprehensive construction of hard instances for Dec-MASSP, supported by well-structured theoretical arguments and inductive proofs.\n\n2. Tight lower bound: The regret lower bound matches the known upper bounds up to poly-log factors, demonstrating tightness and validating the authors' analytical framework.\n\n3. The authors skillfully handle several complex issues — exponential state space, coupled agent dynamics, and KL divergence — using symmetry and feature design to make the analysis doable.\n\n**Clarity**\n\n1. Well-motivated research topic: The paper clearly identifies the knowledge gap in decentralized SSPs with function approximation and motivates the need for lower bounds.\n\n2. Each section is logically laid out, with clear transitions from the problem setup to methodology, theory, and discussion.\nNotions like state-type partitions, the monotonicity of value functions, and symmetry-based analysis are introduced and explained in a way that highlights their intuitive appeal and theoretical usefulness.\n\n3. A very clear proof sketch: This is a hard paper. But the proof sketch makes it easier for the readers to follow. The proof sketches are clear, well-motivated, and the full proofs (relegated to appendices) build on and extend techniques from prior work thoughtfully.\n\n\n**Significance**\n\nI think this paper fills in an important gap in the research direction of SSP and multi-agent RL.\n\n1. This is the first known regret lower bound in the decentralized SSP setting with linear function approximation, establishing a benchmark for future algorithms in this research area. The paper recovers the classical single-agent bounds as a special case, and thus provides a generalization of prior SSP results to the decentralized setting.\n\n2. Importantly, the paper provides a critical insight for understanding the complexity of decentralized MARL. Specifically, the paper reveals that increasing the number of agents increases the density of near-optimal policies, making learning fundamentally harder.\n\n\n**Originality**\n\n1. The hard instances are new and cleverly designed to yield analytical tractability despite exponential state-action space.\n\n2. Novel technique challenges: The use of symmetry to simplify the KL divergence analysis and characterize optimal actions is a clever extension of classical information-theoretic techniques to a more complex setting.\n\n3. First result in the field: The tight lower bound for any number of agents in Dec-MASSP with linear approximation is a good contribution in MARL theory.\n\n\n**Weaknesses (Minor and Contextual)**\n\nI do not detect any major technical weakness in the paper. Following are just some minor comments.\n\n**Quality**\n\nAs a purely theoretical work, the use of very specialized assumptions (e.g., uniform cost, linear transition features, parameter restrictions on $\\delta$) may limit the practicability of the lower bounds beyond the theory.\n\n**Significance**\n\nThe impact is currently limited to linear function approximation. While the paper acknowledges this, broader significance will depend on whether similar bounds can be extended to more general function classes (e.g., neural networks).\n\n**Overall**, I deem this as a high-quality and technically sound contribution that meaningfully contributes to our understanding of decentralized stochastic shortest path. Its careful lower bound construction and insights into the hardness of multi-agent learning problems position it as a strong candidate for acceptance, particularly in theory-focused tracks or venues.\n\nQUESTIONS:\n1. The setting is fully decentralized learning without communication. I wonder how limited communication might affect the regret bounds and instance hardness? Or am I missing something here?\n\n\n2. Just curious but I wonder what might happen if we allow some model misspecification? The model assumes exact linear approximation. How sensitive are the lower bounds or constructions to small model misspecifications?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper characterizes the theoretical lower bound on the regret of the decentralized stochastic shortest path learning problem, which is a crucial problem in path planning. The authors consider decentralized learning in multi-agent state and action spaces. There are three major contributions: (1) designing a class of hard-to-learn MASSP problems and theoretical characterization of the optimal policy and value functions; (2) leveraging that class of problems, a tight lower bound is proposed; (3) Some technical intermediate results on the exponential state-action space would be valuable for future MASSP studies. Overall, the paper appears to be the first work to propose tight lower bounds for MASSP problems, and therefore is a valuable contribution to the field.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe high-level two-step strategy of proving the theoretical lower-bound result is technically sound and original. \n\nThe constructed worst-case MDP is interesting. It can motivate more research on characterizing which class of linear MDP would have an even tighter lower bound than the one proposed in this paper, and therefore can be an impactful contribution for helping the readers understand the interplay between linear MDP structure and computational complexity of learning the optimal policy. \n\nThe linear MDP approach generalizes prior works on tabular MDP, with compact and efficient feature representation. \n\nThe four technical challenges of the multi-agent shortest path problem are clearly discussed on page 2. The paper employs various strategies to address these challenges.\n\n\nWeaknesses: \nAssumption 1 can be made more informative. In general, the theoretical lower bound of computational complexity relies on the specific information provided for the algorithm. However, the current version of Assumption 1 only specifies the linear MDP setup. A more informative assumption should also include information on how decentralized learning is implemented. What information can different agents share while learning? \n\nSome technical details are unclear. For example, this paper motivates the problem by discussing the shortest path over a graph, which typically includes a set of nodes and a set of edges; however, the paper only introduces the node set. Moreover, it is unclear whether the paper considers a discrete or continuous state space. In other words, do we allow state or action to only take discrete values, or can they take continuous values? \n\nThere is a concern that some technical proofs may have a logic gap, and therefore, it questions the quality and credibility of the proofs. In the equation (45) of Lemma 5’s proof in the supplementary material, why can we aggressively take the sum of h from h = 1 to h = infinity? \nThis part seems not very clearly explained, and it may be a bold step. Technically, I’m not sure whether it’s correct. \n\nThere appears to be no discussion about how frequently the constructed MDP would be encountered in practical applications. If it never happens, then the impact of those constructed examples is low. \n\nThe technical details are hard to follow, and limited intuition is shared behind the derivation of proofs. So, it is hard to fully justify the technical correctness of the proposed idea. For example, it would greatly improve the readability if the authors could explain how the binomial coefficients help construct the lower bound. \n\nMany typos. For example, line 28, “recent works has”; line 252, “its desirable to”; line 320, “upto”, etc.\n\nQUESTIONS:\nThis work considers multi-agent cases, so would the derived lower bound recover existing lower bound results for the single-agent shortest path problem? If so, I believe the constructed hard-to-learn example will also have an impact on the single-agent case. \n\nThe constructed lower bound relies on the strong assumption that the optimal policy can be easily found. However, for the general case where the optimal policy is hard to find, would the lower bound still be tight? \n\nThe constructed lower bound in Theorem 2 depends on B^*, but the reported lower bound of this work in Table 1 has no dependency on B^*. Why is B^* missing in the last line of Table 1? \n\nWhat is the technical challenge when extending the proposed technique to random policies? \n\nThe constructed regret lower bound seems to rely on the assumption that the value function can be estimated accurately. In practice, it may not be the case. So, how does the value function approximation error affect the regret? \n\nIn many theoretical RL papers, we assume all states and actions should have some probability mass in the transition dynamics to avoid the poor exploration issue. Do we need similar assumptions in this work? I believe so, because otherwise the denominator in equation (15) can be zero when we have deterministic transition dynamics.\n\nLIMITATIONS:\nYes, the authors provide some discussion about the limitations, such as only considering linear feature spaces instead of nonlinear feature spaces. However, the reviewer believes that there are two additional limitations to the current manuscript. Having some discussion about them will significantly improve the impact of this work.\n\nNo experiment comparing the actual computation time of different algorithms for solving those hard-to-learn examples. The result would be more convincing if a plot could be provided for visualizing that the lower bound regret is indeed tight and valid, compared to the existing algorithms’ regrets. \n\nIn many real-world planning problems, we need to consider road capacity and other types of constraints. The current formulation has no practical constraints. So, looking ahead, a natural question is how constraint satisfaction affects the computational lower bound? Some discussion on this in future work would be exciting and make the work more impactful for real-world constraint-aware shortest-path problems.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}