Title: Optimal Regret Bounds via Low-Rank Structured Variation in Non-Stationary Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Non-Stationary Reinforcement Learning
Abstract: We study reinforcement learning in non-stationary communicating MDPs whose
transition drift admits a low-rank plus sparse structure. We propose
\textbf{SVUCRL} (Structured Variation UCRL) and prove the dynamic-regret bound
\[\scalebox{0.75}{$\displaystyle
  \widetilde{\mathcal O}\Bigl(
       \sqrt{SA\,T}
       + D_{\max}S\sqrt{A\,T}
       + L_{\max}B_r
       + D_{\max}L_{\max}B_p
       + D_{\max}S\sqrt{A\,B_p}
       + D_{\max}\delta_B B_p
       + D_{\max}\sqrt{K\,T}
     \Bigr),
$}\]
(up to the additional planning-tolerance term $\sum_{t=1}^T \varepsilon_{\tau(m(t))}$).

where $S$ is the number of states, $A$ the number of actions, $T$ the horizon,
$D_{\max}$ the MDP diameter, $B_r$/$B_p$ the total reward/transition variation
budgets, and $K\!\ll\! SA$ the rank of the structured drift, $L_{\max}$ is the maximum episole length. The first two terms are the statistical price of learning in stationary problems.
The structure-dependent non-stationarity contribution appears through
$D_{\max}\sqrt{K\,T}$ (low-rank drift) and $D_{\max}\delta_B B_p$ (sparse shocks),
which scale with $\sqrt{K}$ rather than $\sqrt{SA}$ when drift is low-rank. This matches the $\sqrt{T}$ rate (up to
logs) and improves on prior $T^{3/4}$-type guarantees. SVUCRL combines:
(i) online low-rank tracking with explicit Frobenius guarantees,
(ii) incremental RPCA to separate structured drift from sparse shocks,
(iii) adaptive confidence widening via a bias-corrected local-variation
estimator, and (iv) factor forecasting with an optimal shrinkage center.


Peer reviews:
comments: SUMMARY:
This paper studies reinforcement learning with non-stationary environments. It considers a structured environmental variation, i.e., low-rank drift.  Then an algorithm is designed which combines matrix factorization, robust statistics and time-series analysis.

STRENGTHS AND WEAKNESSES:
Strength:
This paper studies RL in an interesting class of non-stationary environments characterized by a low-rank structure. The theory is clean and well-structured, incorporating tools from matrix factorization, robust statistics, and time-series analysis.

Weakness:
Although this paper provides interesting theory for non-stationary RL, these results totally rely on a specific structural assumption about non-stationarity. However, the reasonableness of this assumption is not verified using experimental data. Besides, the authors do not provide any experiments, so I am concerned about the applicability of this method.

QUESTIONS:
- Can you give concrete examples where the  low-rank drift assumption holds?
- What is the justification for presenting Assumption 1 as a contribution (as stated in line 31)?

LIMITATIONS:
- The whole paper relies on the low-rank assumption, which is not verified.
- No experiments are provided to illustrate the performance of the proposed algorithm.
TITLE: Official Review
CLARITY: 3
IMPACT: 1
ORIGINALITY: 3
DATE: 2025-06-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses Reinforcement Learning in non-stationary (communicating) MDPs satisfying a structural low-rank assumption on the transition kernel drifts. For such class of MDPs, the authors propose an algorithm, SVUCRL that achieves a regret scaling with the square-root of the number of interactions and the rank of the underlying drift. SVUCRL is made of three components: (i) randomised SVD algorithm for tracking of the low-rank structure, (ii) Incremental robust PCA to decompose low-rank components with sparse ones, (iii) extended value iteration (EVI) over the estimated MDP, with confidence bounds accounting for local uncertainties.\n\n**Overall Motivation for Rating**:\n\nIn general, I am not strongly leaning towards rejection. My current score will be changed depending on: (1)  a satisfactory rebuttal showing that my doubts are not so well founded; (2) whether other reviewers had different impressions on the clarity/originality issues.\n\nSTRENGTHS AND WEAKNESSES:\nIn the following, I will refer to Strengths with (S) and Weaknesses with (W)\n\n**Quality**: \n\n(S) The overall paper structure progressively builds up relevant results on the no-regret nature of the building blocks leading to SVUCRL, each of them being rather interesting per-se and the overall contributions seem interesting.\n(W) Unfortunately, the overall quality of the paper is utterly impacted by the limited clarity and overall exposition, and limited discussion of the related works, and actual limitations. See next section.\n\n**Clarity**:\n\n(W) The current pseudo-code of SVUCRL in the main paper fails to convey a clear description of the actual behaviour of the algorithm, which is left to the Appendix. I strongly believe that the pseudo-code of the algorithm in Appendix should be moved in the main paper. \n\n(W) The overall paper fails to cover existing literature on the topic, with extremely limited reference of existing works (13 references seem surprisingly few to me) or how the proposed algorithms/analysis differ from prior works (See Questions).\n\n(W) The current manuscript often lacks on clarity, some examples:\n\n\t- Dynamic Regret: the optimal average reward has not been defined.\n\t- For clarity purposes, I believe the terms composing the low-rank drift in Assumption 1 should be described in terms of the spaces they are supported over. \n\t- References to the Appendices are broken.\n\t- (line 200-201) the bounding terms of $hat V$ have not been defined, nor the window $W_v$.\n\t- The proof outlines are not very clear to me, I would extend them or remove them.\n\t- Algorithm 3 is not really sufficient in providing information on how the algorithm work.\n\t\n\n(Minor W) I am not sure that the current style of the abstract is suited for the audience understanding as it is extremely technical while failing to provide a bird-eye view of both the problem and the ideas of the proposed solution. \n\nOverall, in order to make the paper clear at an acceptable level, I currently believe the manuscript might undergo substantial changes which are not compatible with minor revisions.\n\n\n**Significance**: \n\n(S) The paper addresses an important problem, that is RL in non-stationary MDPs. The proposed algorithm achieves sub-linear regret both in terms of episodes of interactions and rank of the problem. \n\n(W) In order to achieve such performances compared to [1], the algorithm requires an additional assumption on the non-stationarity to be low-rank decomposable (Assumption 1). The authors try to provide some intuition on the extent to which such assumption is realistic, yet I believe that some more precise and concrete examples should be added, as the assumption is the building block of the overall work.\n\n(W) I strongly believe that the Limitations section in Appendix should be moved in the main paper, as it contains extremely relevant points. For instance, the existence of an implicit assumption on the  *sparsity of changes* as well, whose practical implications are not discussed in the main paper, and the existence of a *computational overhead* in RSVD of $O(TSA(SK + S) log T)$ which seems pretty remarkable (see Questions).\n\n**Originality**:\n\n(S) I believe the low-rank structural assumption over the drifts rather than the transition kernel themselves is potentially interesting and relevant and paves the way for further contributions. \n\n(W) Unfortunately, I was not able to totally discern the originality of the work from the existing manuscript, due to extremely limited references and discussions about related works and results. For instance, it is not clear what is the contribution of this work in the adaptation of robust PCA [2] or randomised SVD [3, 4] to this setting, or in the derivation of the theoretical findings, or in the use of EVI. For further discussion, see Questions. Overall, the current presentation is detached from previous contributions and related works, making it hard to understand what is the actual contribution of the paper.\n\n[1] Cheung et al. 2020\n\n[2] Candes et al. 2011\n\n[3] Mairal et al. 2010\n\n[4] Halko et al. 2011\n\nQUESTIONS:\n- I am not sure to have understood why the overall regret is not dependent somehow on the window size $W_v$ or how is the overall behavior dependent on the many hyper-parameters. Could the authors provide some elucidations?\n- Could the authors discuss on the role of the computational overhead of RSVD with respect to existing literature and related works? Does this impact the overall relevance of the proposed algorithm? If not, why?\n- Could the author clarify the original contributions (on the algorithmic side and on the theoretical analysis side) with respect to [2,3,4, see above]?\n\nLIMITATIONS:\n- The paper claims to improve over [1, see above], yet [1] is designed for a larger class of non-stationary MDPs and does not make uses of plug-in algorithms (such as RSVD, which apparently comes with a pretty relevant computational overhead), or the additional assumptions on low-rank and sparsity. While I believe limitations or additional assumptions are acceptable while chasing for improved performances, I had the impression that the current phrasing and presentation of the paper formatting was trying to *hide* these limitations and not expressively recognising or discussing them when needed (see above).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SVUCRL, a reinforcement learning algorithm designed for non-stationary environments where the transition dynamics evolve with structured, low-rank drift and occasional sparse shocks. By modeling the temporal changes in the environment as a sum of low-rank components and sparse perturbations, the authors devise a principled method for regret minimization under dynamic conditions. SVUCRL achieves a regret bound of $\\mathcal{\\tilde{O}}(D_{max}\\sqrt{SAT} + D_{max}\\sqrt{(B_r+B_p)KST}) $. The paper combines techniques from online factor tracking, incremental robust PCA, adaptive confidence widening, and shrinkage forecast to derive its results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Novel model of change in transition probabilities as a low rank drift plus sparse shocks\n- Efficient tracking of the structured drift of transition probabilities through a randomized SVD with power iterations with provable guarantees (bounded Frobenius error)\n- Incremental RPCA elegantly differentiates between low rank drift from sparse shock components\n- Adaptive Confidence Widening cleverly saves on regret by focusing on points of highest uncertainty \n- Theoretically grounded method to optimally blend empirical and model-based transition estimates using James–Stein shrinkage  \n- SVUCRL adapts to unknown rank and drift complexity without prior knowledge, enhancing its practicality\n\n**Weaknesses:**\n- I believe the contribution of strictly improving upon the $T^{3/4}$ regret of SWUCRL-CW is overclaimed and an unfair comparison to make. Cheung et al. do not make any assumptions on the structure of the changes (i.e no low rank structure of transition probability drift) - making it a different and thereby making their algorithm more general than this paper. Further, in the average reward infinite horizon MDP with time-varying rewards and transition probabilities and no assumptions on the structure, Mao et al. presents a $\\mathcal{\\Omega}((B_r+B_p)^{1/3}T^{2/3})$ regret lower bound.\n- While the algorithm is theoretically polynomial time, the incorporation of SVD, RPCA, and forecasting adds significant computational overhead, especially for large state and action spaces.\n- As acknowledged in the limitations, windows ($W, W_v , W_f$ ), oversampling $s$, power iterations $q$ and the shrinkage threshold all require tuning and may not be easy to choose practically.\n\n**References:**\n- Mao, W., Zhang, K., Zhu, R., Simchi-Levi, D., and Basar, T. Model-free nonstationary reinforcement learning: Nearoptimal regret and applications in multiagent reinforcement learning and inventory control. In Management Science, 2024.\n- Cheung, W. C., Simchi-Levi, D., and Zhu, R. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International Conference on Machine Learning, 2020.\n\nQUESTIONS:\n- Section 9.1 / Interpretation and Tightness of Regret Bound: I believe it is unfair to compare against SWUCRL-CW that makes no assumptions on the structure of the changes.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-23"}

comments: SUMMARY:
This paper addresses non-stationary RL problems in communicating MDPs with low-rank transition drifts over time. This paper derives a regret bound of order $\widetilde{O}(\sqrt{T})$ with respect to the horizon $T$, which improves over the best previously known bound $\widetilde{O}(T^{\frac{3}{4}})$.

STRENGTHS AND WEAKNESSES:
* **Strengths:**
    1. The derived regret bound considerably improves the best previously known bound.
    2. The algorithm does not require prior knowledge of the rank and can adaptively estimate the appropriate rank based on observed data.

* **Weaknesses:**
    1. The writing and presentation of this paper make it difficult to follow. I think it would be better to present the SVUCRL algorithm first and then dive into the technical details, i.e., advancing Sections 8 and 9 before Sections 4-7.
    2. The result seems to contradict the lower bound (see Question 4 below).

QUESTIONS:
1. I think there is a typo in Line 3, where the first term of regret should be $D_{\max}S\sqrt{AT}$ according to Theorem 9.
2. Is it possible to improve the first term in the regret to $\widetilde{O}(\sqrt{D_{\max}SAT})$ as in the stationary environment [1]?
3. Does the regret guarantee (Thm. 9) require the knowledge of the rank $K$?
4. Does the result contradict the regret lower bound in non-stationary multi-armed bandits, which is of order $\Omega(T^{\frac{2}{3}})$?

[1] Zhang, Zihan, and Xiangyang Ji. "Regret minimization for reinforcement learning by evaluating the optimal bias function." NIPS 2019.

[2] Besbes, Omar, Yonatan Gur, and Assaf Zeevi. "Stochastic multi-armed-bandit problem with non-stationary rewards." NIPS 2014.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03