Title: Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Low-rank representation

RL

successor measure

local mixing
Abstract: Low-rank structure is a common implicit assumption in many modern reinforcement learning (RL) algorithms. For instance, reward-free and goal-conditioned RL methods often presume that the successor measure admits a low-rank representation. In this work, we challenge this assumption by first remarking that the successor measure itself is not approximately low-rank. Instead, we demonstrate that a low-rank structure naturally emerges in the shifted successor measure, which captures the system dynamics after bypassing a few initial transitions. We provide finite-sample performance guarantees for the entry-wise estimation of a low-rank approximation of the shifted successor measure from sampled entries. Our analysis reveals that both the approximation and estimation errors are primarily governed by a newly introduced quantitity: the spectral recoverability of the corresponding matrix. To bound this parameter, we derive a new class of functional inequalities for Markov chains that we call Type II Poincaré inequalities and from which we can quantify the amount of shift needed for effective low-rank approximation and estimation. This analysis shows in particular that the required shift depends on decay of the high-order singular values of the shifted successor measure and is hence typically small in practice. Additionally, we establish a connection between the necessary shift and the local mixing properties of the underlying dynamical system, which provides a natural way of selecting the shift. Finally, we validate our theoretical findings with experiments, and demonstrate that shifting the successor measure indeed leads to improved performance in goal-conditioned RL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSuccessor representation (SR) is a widely used approach for reward-free representation learning in reinforcement learning. This paper focuses on computing low-rank approximations of SR. However, if the underlying MDP is not low-rank, its successor representation is unlikely to be low-rank either, which introduces approximation error. To address this, the authors propose a new variant called the $k$-shifted SR, that captures the dynamics of policy $\\pi$ starting from time step  $k$.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is overall well written and easy to follow.\n- The proposed shifted SR is a natural and clever response to the limitations of low-rank SR in general MDPs.\n\nWeaknesses:\n- The relevance of the proposed approach is unclear. For instance, it remains uncertain whether obtaining the shifted-SR representation is actually useful for deriving good policies. The authors provide no theoretical guarantees about the quality of policies obtained after estimating the shifted-SR. Consequently, the practical utility of these representations is unclear.\nGiven this lack of clarity, the paper essentially proposes the problem of estimating shifted-SRs and a method to do so, but fails to establish why this is meaningful in the first place.\n\nQUESTIONS:\n- If the MDP we aim to represent is not inherently low-rank, it’s unclear why we would prefer a low-rank representation in the first place. Why not simply use the standard full-rank successor representation (SR)? The justification for opting for a low-rank SR remains unclear. Is the goal to reduce memory usage, computational cost, or sample complexity? Or does it empirically lead to faster learning? These motivations need to be more explicitly articulated.\n- The paper shows results on the regret when estimating the shifted representation $M_{\\pi,k}$. However, it is unclear how this quantity is related to the regret of a policy that is computed via the shifted representation. How is $M_{\\pi,k}$ related to $M_{\\pi}$? One could accurately estimate  $M_{\\pi,k}$ but still get a bad policy. This could happen, for example, if the policy improvement method (e.g., GPI or policy iteration) amplifies the approximation error in an undesirable way. In my view, unless it is demonstrated that using  $M_{\\pi}$​ for policy computation leads to low regret in control, the importance of estimating it accurately remains unclear. This is likely my primary concern regarding the contribution. If this issue were resolved, the significance of the work would become much more apparent.\n\nLIMITATIONS:\nSome of the weakness listed below needs further comments on the text to address this potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a theoretical work focused on challenges associated with building successor measures in goal-conditioned RL. In particular, the authors analyze a common assumption that the successor measure can be effectively modeled via low-rank representation and find that, whereas vanilla approaches are not guaranteed to admit a low-rank structure, such low-rank structure naturally emerges in shifted successor measures (where \"shift\" refers to skipping the first k time-steps of the trajectory before building the successor measure). Furthermore, the authors provide sample complexity bounds and errors for learning the shifted successor measure, which are shown to be related to the spectral recoverability of the successor matrix. Finally, the authors present a small experimental section consisting of a discrete, goal-conditioned maze. These experiments show that, in line with the provided theory, low-rank approximation works better with shifted successor measures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.Despite being quite mathematically dense, the paper is clearly written (except for the experimental section).\n2.The manuscript provides an analysis of a common assumption used in reward-free RL with results that can inform future algorithm design.\n3.To the best of my knowledge, the strategy for obtaining the theoretical results is novel in the context of RL (although I did not check the correctness of the proofs).\n\nWeaknesses:\n1.The experimental section is relatively small and dense. It would be helpful to describe the experimental settings and the results in more detail. For example, a more direct discussion of the learning algorithm (e.g. hyperparameters), implications of the experiments with respect to the presented theory, and limitations of the experimental setup would be particularly helpful for the reader. Having said this, I understand that the theoretical analysis is the central contribution of this paper.\n\nQUESTIONS:\n1.How restrictive are the assumptions made in corollary 1? In what types of setups do these hold?\n2.How do these theoretical considerations apply to continuous state/action environments?\n3.What is the significance of relaxed accuracy as compared to accuracy? Why do the policies with higher shifts perform relatively better in terms of relaxed accuracy than in terms of accuracy?\n4.Could the authors expand the intuition on why shifting would lead to a low-rank structure? Does this observation hold in more stochastic problems?\n\nLIMITATIONS:\nAuthors could slightly improve the following aspects of their work:\n1.The limitations of the theoretical results - adding a paragraph summarizing the impact of the theoretical result on the practical workflows of reward-free RL, as well as the limitations associated with the assumptions \n2.The limitations of the experimental section - adding a paragraph discussing which assumptions and other elements of the theoretical analysis correspond to the experimental setup", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes ​shift successor measures​ to better separate two types of transitions: those involving short-range dynamics (which remain inherently high-rank) from those governed by long-range interactions. The experiments demonstrate that shifted successor measures admit more accurate low-rank SVD approximations than non-shifted counterparts, and that enhanced representation yields consistent performance gains in goal-conditioned reinforcement learning tasks.

STRENGTHS AND WEAKNESSES:
**Pros:**

While I'm not familiar in low-rank RL theory, the paper demonstrate its contributions to this research area. It clearly introduces the problem setting in the introduction, followed by explanations of the methodology grounded in strong intuition that build toward a rigorous theoretical analysis with detailed proofs. Critically, the empirical validation demonstrates that shifted successor measures achieve notably better low-rank approximations than standard counterparts, confirming the method's effectiveness.

**Cons:**
1. Experiments are confined to a ​single discrete environment​. Broader validation on a continuous environment would be better to support claims of general applicability.

2. Adding comparisons against baselines would better demonstrate the advantages of the proposed method.

QUESTIONS:
1. What's the computational efficiency of deriving the shifted successor measure estimator?

LIMITATIONS:
Yes, the author presents the discussion on limitations in the appendix.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the low-rank representations in RL through a shifted successor measure, supported by strong theoretical contributions. It claims that the standard successor measure is not inherently low-rank, and a shifted successor measure, which captures dynamics subsequent to omitting earlier transitions, can demonstrate the low-rank property. This paper provides a novel perspective on the analysis of long-term dynamics. However, the proposed method may lose practicality since the assumptions in Section 5 are somewhat strong, which may fail in practice (e.g., in complex or dynamic environments).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n- Novel Perspective: The authors redefine the low-rank assumptions in RL by introducing the shifted successor measure, providing a novel perspective on the analysis of long-term dynamics.\n- Theoretical Proof: The authors provide rigorous proofs with spectral recoverability and Type II Poincaré inequalities to enhance the understanding and effectiveness of low-rank approximations in RL.\n\n# Limitations:\n- Selection of the Shift Parameter: The theoretical results explain why a  shifted successor helps, but without a guidance to choose k, which is critical in practical implementation.\n- Strong Assumptions: the proposed method may lose practicality since the assumptions in Section 5 are somewhat strong, which may fail in practice (e.g., in complex or dynamic environments). In addition, the experiments validate the results in a small, discrete envs. Demonstrating the method's effectiveness on a more challenging, high-dimensional benchmark would be better to show its practical values in real-world tasks.\n\nQUESTIONS:\n- How to choose k in practical implementation?\n- The proposed method may lose practicality since the assumptions in Section 5 are somewhat strong, which may fail in practice (e.g., in complex or dynamic environments). In addition, the experiments validate the results in a small, discrete envs. Demonstrating the method's effectiveness on a more challenging, high-dimensional benchmark would be better to show its practical values in real-world tasks.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-08"}