Title: Stochastic Shortest Path with Sparse Adversarial Costs
Venue: NeurIPS 2025
Accepted: yes
Keywords: SSP

sparsity

OMD
Abstract: We study the adversarial Stochastic Shortest Path (SSP) problem with sparse costs under full-information feedback. In the known transition setting, existing bounds based on Online Mirror Descent (OMD) with negative-entropy regularization scale with $\sqrt{\log S A}$, where $SA$ is the size of the state-action space. While we show that this is optimal in the worst-case, this bound fails to capture the benefits of sparsity when only a small number $M \ll SA$ of state-action pairs incur cost. In fact, we also show that the negative-entropy is inherently non-adaptive to sparsity: it provably incurs regret scaling with $\sqrt{\log S}$ on sparse problems. Instead, we propose a novel family of $\ell_r$-norm regularizers ($r \in (1,2)$) that adapts to the sparsity and achieves regret scaling with $\sqrt{\log M}$ instead of $\sqrt{\log SA}$. We show this is optimal via a matching lower bound, highlighting that $M$ captures the effective dimension of the problem instead of $SA$. Finally, in the unknown transition setting the benefits of sparsity are limited: we prove that even on sparse problems, the minimax regret for any learner scales polynomially with $SA$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses adversarial Stochastic Shortest Pash (SSP) with sparse costs under full-information feedback. The main take is that such sparsity should lead to a regret bound scaling with some measure of sparsity of the problem, and not with the space size. Indeed, while negative entropy regularised OMD leads to bounds scaling as SA in general, the authors show that it scales as S on sparse problem. Motivated by this, they propose a new regularisation for OMD (smoothly interpolating between neg-entropy and l2 norm) which leads to a regret scaling with the sparsity of the problem. Additionally, the authors instantiate a parameter-free version of the same algorithm that does not require knowledge of the sparsity level nor the hitting time of the optimal policy. Finally, the authors show lower bounds ( for some regimes of the sparsity factor M) matching the upper bound over these instances. The authors also show that this bound does not hold under bandit feedback, in which the M factor is substituted by SA again, regardless of the sparsity, suggesting that without further assumptions, sparsity cannot be exploited effectually under non-full feedback settings.\n\n\n**Overall Motivation for Rating**:\n\nIn general, I am not strongly leaning towards accept just because (1) I have *not* carefully read the proofs and (2) I have some knowledge of the related works but I would not call me an expert. My current score will change conditioned on: a satisfactory rebuttal showing that my questions are not so well founded; whether other reviewers raise some major concerns I was not aware of but I might agree with ex-post.\n\nSTRENGTHS AND WEAKNESSES:\nIn the following, I will refer to Strengths with (S) and Weaknesses with (W), line numbering with (l#)\n\n**Quality**: \n\n(S) The work reaches a good quality overall and it is self-sufficient in covering a problem and pointing to limitations and next step.\n\n**Clarity**:\n\n(S) The paper is extremely well written, it does introduce all the concepts clearly and it does address all the relevant limitations.\n\n(Minor W) I would format equation under l170 over a single line.\n\n**Significance**: \n\n(Minor W) Full-information feedback might sound limiting, yet I believe that (S) a formal characterisation of the problem is still relevant and provides an important stepping stone towards more realistic settings, and the authors provided enough tools to go towards this direction.\n\n**Originality**:\n\n(S) The proposed regularisation is interesting and novel (as far as I am aware of), and the counterexamples on standard neg-entropy OMD limitations are a pretty nice rethorical tool.\n(Very Minor W) Most of the other proof structures of the lower bounds are based on previous works, which is yet pretty common.\n\nQUESTIONS:\n- The lower bound goes towards non-sparse problem more than sparse ones, as it holds for M > 100. Do the authors have a take on this, or some intuition on what could happen for sparser problem (M<100)?\n- Can the authors better articulate on the limitations suggested by Thm 5.1? Which sub-portions of the algorithmic solutions could be nonetheless employed for unknown transition settings?\n\nMy current score will be undergo the following changes:\n- increased: whether my questions will be addressed properly, and further discussions about them will be included in the manuscript.\n- decreased: whether other reviewers raise some *major* concerns I was not aware of but I might agree with post discussion.\n- Kept the same: whether the answers were not addressed.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the stochastic shortest path problem under sparsity. In particular, they show the online mirror descent (OMB) with entropy regularization does not take advantage of sparsity in general, and introduce a new family of regularizers which can. With the new regularizers, they prove a bound on the regret of OMB which scales with the sparsity instead of the absolute size of the state space. They also prove a lower bound which matches their upper bound up to logarithmic factors.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n- Sparsity is a very natural setting to consider. \n- They provide a new algorithm and both upper and lower bounds \n\n# Weaknesses\n- A bit hard to follow for someone who is not an expert in the area \n- Some concerns I have about Theorem 4.3; see below.\n\nQUESTIONS:\n# Questions: \n- In the definition of D (equation 148) you swap the min and max. Why is this justified? \n- Theorem 3.1 is very interesting. Do you have any idea how common these instances are in which negative-entropy regularization fails to get the right (ie sparse) rate? Is it possible to generalize your proof to a statement such as: For any S, any M and any A, there exists an SSP instance with regret $\\Omega(\\sqrt{DKT_* \\log (SAT_*)})$? That is, showing that for all parameters, there exists an SSP instance which won’t take advantage of the sparsity. \n- Should $K\\geq MT$ be $K\\geq MT_*$ in the statement of theorem 4.3? Otherwise what is T?\n- Couldn’t the bound of Theorem 4.3 be worse than the status quo if K is very large? The second term vanishes, but if K is large enough such that $\\log(\\log(K)\\sqrt{T_\\*}) > \\log(T^\\*SA)$  then theorem 4.3 is looser than $O(DKT_*\\log(SAT^*))$? So perhaps you need to state the result with an upper bound on K as well? \n\n# Minor comments\n- Line 142: “a proper policy” -> “one proper policy” \n- Maybe move line 158 upwards to where you are discussing hitting times? Feels out of place \n- Why does $\\eta$ matter in the statement of Theorem 4.1? It’s just a parameter in the proof no?\n- Missing square root on the regret in line 273 I think\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the adversarial stochastic shortest path (SSP) problem with sparse costs in the full-information feedback setting. With known transition P, this work improves the previous best known result $O(\\sqrt{DK{T_{\\*}} \\log ({T_{\\*}}SA)})$ to $O(\\sqrt{DKT\\_{\\*} \\log (T\\_{\\*}M)})$, where $D$ is the diameter, $K$ is the number of episodes, ${T_{\\*}}$ is the expected hitting time of the optimal policy, $M$ is the sparsity parameter, and $S$ and $A$ are the state and action space size, respectively. This is achieved by a new regularizer proposed by the authors, which regularizes the (1+1/p)-norm of the occupancy measures. The authors propose two algorithms, one requiring the knowledge of $M$ and the other being a parameter-free one, without requiring the knowledge of $M$. For the same problem, the authors prove a lower bound of ${\\Omega}(\\sqrt{DK{T\\_{\\*}}\\log (M)})$, showing the tightness of the dependence on $M$ in the regret upper bound. The authors also prove an algorithm-specific lower bound, showing that the canonical Shannon entropy is not able to achieve the same regret upper bound. With unknown transition P, the authors prove a regret lower bound $\\Omega(D\\sqrt{SAK})$.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The results are new.\n\n**Weaknesses**\n1. **Significance**: Though the results are new to the literature, I am a bit concerned about the significance of the results. In particular, the authors obtain the upper bound of $O(\\sqrt{DK{T_{\\*}} \\log (T_{\\*}M)})$ in the full-information setting. Though the dependence on $M$ indeed matches the proved lower bound, the improvement from $O(\\sqrt{\\log (SA)})$ to $O(\\sqrt{\\log M})$ against previous work [1] does not seem to be very valuable to me. Further, I find that the improvement does not even recover it in the special case of the expert setting (i.e., $S=1$). On Line 39, the author mention that $O(\\sqrt{\\log A})$ has been improved to $O(\\sqrt{\\log A^{\\frac{M}{A}}})$ in the expert setting. However, letting $S=1$, the work actually achieves the improvement from $O(\\sqrt{\\log A})$ to $O(\\sqrt{\\log M})$. Yet, $\\sqrt{\\log M}\\ge \\sqrt{\\log A^{\\frac{M}{A}}}$ for all $M\\in (1,A]$. I would instead suggest the authors investigate whether significant improvements are possible in, e.g., the bandit-feedback setting.\n2. **Parameter-free case**: Theorem 4.3 in this work shows that it is possible to achieve the sparsity-dependent upper bound in the parameter-free case. However, I notice that this only holds for a sufficiently large $K$ (specifically, $K\\ge MT_*$). When the losses are not so “sparse”, this essentially means that $K\\ge SAT_*$ is required. As such, I think this limits the applicability of the proposed algorithm.\n\n[1] Chen et al. Minimax regret for stochastic shortest path with adversarial costs and known transition. COLT, 2021.\n\nQUESTIONS:\n1. Could the authors please further explain the intuition on Line 223-229? I do not quite understand why “inducing an OMD update at points near the boundary of the space that can only move to new points in very close proximity” prevents the algorithms from obtaining a sparsity-dependent bound.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}