Spectral Bellman Method: Unifying Representation and Exploration in RL
Authors: Ofir Nabati
Bo Dai
Shie Mannor
Guy Tennenholtz
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
represetation learning
Primary area: reinforcement learning
TL;DR: A theoretically-inspired spectral method of the Bellman operator for value-based RL that unify representation learning and exploration.

Abstract: Representation learning is critical to the empirical and theoretical success of reinforcement learning. However, many existing methods are induced from model-learning aspects, misaligning them with the RL task in hand. This work introduces the Spectral Bellman Method, a novel framework derived from the Inherent Bellman Error (IBE) condition. It aligns representation learning with the fundamental structure of Bellman updates across a space of possible value functions, making it directly suited for value-based RL. Our key insight is a fundamental spectral relationship: under the zero-IBE condition, the transformation of a distribution of value functions by the Bellman operator is intrinsically linked to the feature covariance structure. This connection yields a new, theoretically-grounded objective for learning state-action features that capture this Bellman-aligned covariance, requiring only a simple modification to existing algorithms. We demonstrate that our learned representations enable structured exploration by aligning feature covariance with Bellman dynamics, improving performance in hard-exploration and long-horizon tasks. Our framework naturally extends to multi-step Bellman operators, offering a principled path toward learning more powerful and structurally sound representations for value-based RL.

Published as a conference paper at ICLR 2026
SPECTRAL BELLMAN METHOD: UNIFYING REPRESENTATION AND EXPLORATION IN RL
Ofir Nabati!A: Bo Shie,
Shie Mannor'., Guy Tennenholtz
Technion, Google DeepMind, Nvidia Research," 4 Google Research
ABSTRACT
Representation learning is critical to the empirical and theoretical success of reinforcement learning. However, many existing methods are induced from model- learning aspects, misaligning them with the RL task in hand. This work introduces the Spectral Bellman Method, a novel framework derived from the Inherent Bell- man Error (IBE) condition. It aligns representation learning with the fundamental structure of Bellman updates across a space of possible value functions, making it directly suited for value-based RL. Our key insight is fundamental spectral relationship: under the zero-IBE condition, the transformation of a distribution of value functions by the Bellman operator is intrinsically linked the feature covariance structure. This connection yields new, theoretically seareiccallygoounneedd objec- tive for learning state-action features that capture this Bellman Bellman-aligned covariance, requiring only a simple modification to existing algorithms. We demonstrate that our learned representations enable structured exploration by aligning feature co- variance with Bellman dynamics, improving performance in hard-exploration and long-horizon tasks. Our framework naturally extends to multi-step Bellman oper- ators, offering principled path toward learning more powerful and structurally sound representations for value-based RL.
INTRODUCTION
Efficient reinforcement learning (RL) in complex environments hinges two critical challenges: learning effective representations and performing efficient exploration. While many approaches tackle representation learning (Laskin et al. 2020; Schwarzer al. [021 Oh et al.] [215 Zhang
et al. 2021 Nabati et al. 2023; Barreto et al. 2017 Zhang et al.) 2022) and exploration as separate problems, deeper synergy is needed, particularly for control tasks where features must support both accurate value estimation and strategic data gathering. This work introduces novel framework learn representations that inherently unify these aspects, paving the way for more powerful and sample-efficient RL agents.
Our approach is rooted in the theory Inherent Bellman Error (IBE) (Zanette al., 2020a). The IBE quantifies the suitability of a feature space for value-based RL by measuring the minimum error representing Bellman updates. zero IBE condition, generalizing Linear MDPs (Jin et al. 2020), is highly desirable as it implies the function space is closed under the Bellman operator. However, directly discovering features that satisfy this condition a priori remains significant challenge through min-max-min optimization (Modi et al., 2024), especially with general function approximator. The challenge impedes the practical applications of IBE-based representation learning.
This paper introduces the Spectral Bellman Method (SBM), framework which makes low-IBE representation learning tractable and addresses exploration upon the learned representation. Our approach uses fundamental spectral relationship of IBE: when the IBE is zero, the transformation of a distribution of value functions by the Bellman operator is intrinsically linked to the covariance structure of the features themselves. This connection reveals that features aligned with Bellman updates naturally possess a covariance structure that can be exploited for structured exploration. Leveraging this insight, we derive novel, theoretically grounded objective function. Our method learns state-action features whose covariance inherently captures this Bellman-aligee structure, while avoiding the complicated optimization. The learned representations not only enhance value
"fiimabati & @gmail.com

---

Published as a conference paper at ICLR 2026
Σ
V T
TQ=
=
A4
U
Figure 1: Spectral Representation of the optimal Bellman operator. Under zero IBE condition, the linear representation is equivalent to an SVD decomposition rank with a singular value matrix Σ.
approximation but also naturally facilitate structured exploration strategies. Specifically, we use Thompson Sampling (TS) driven by the learned representation for efficient exploration.
Our contributions are as follows. (1) We propose novel representation learning framework, the Spectral Bellman Method, which unifies representation learning with exploration, motivated by the structural implications of the Inherent Bellman Error. Our method requires only a simple modification (2) We demonstrate the effectiveness of the Spectral Bellman Method and its learned representations on the Atari benchmark, including subset of hard exploration games. (3) The extension of this representation learning approach to capture the structure of powerful multi-step Bellman operators and targets.
to existing value-based RL algorithms.
BACKGROUND: THE ZERO- IBE CONDITION AND EFFICIENT EXPLORATION
We consider Markov Decision Process (MDP) (Puterman 1990 (S,A, A,,,,,,,, with state space S. action space A. discount y E [0, 1), reward r S - R (r(s, a)l S. Rmax). and transition kernel P S x A+ + (SS) The Q-function Q" a) = E.,r Ei ('r(st, ae) Sp = 8, ag a). The goal is to find an optimal policy π² (s) arg max, Ba-- (s)
The optimal Q-function (s, a) satisfies (s a) = rss a) + +. (s maxx E (s', The optimal Bellman operator T on Q S x A + R is defined by
(s, r(s, a) + ((s max Q(5 a'EA
,a')
(Optimal Bellman Operator)
This is a special case of the policy Bellman operator T* = r(s,a) + ylE P(laa)) (9)(0 (osss('a))) a')) Indeed, for greedy policy w.r.t Q, and T are equivalent.
2.1 THE ZERO INHERENT BELLMAN ERROR CONDITION
Q' is often approximated linearly with features & 8 A + Rd as Qa(s,a) = o(s, for E R.d. We assume [[ō(s, a)ll2 < 1.
Definition 1 (Function Space and Parameter Bounds). Let Φb S A Rdd be feature map. We
define the linear function space for Q-functions as:
2s (Qo(s,a) a) = ((s,a) a)
E Bo). B3 = (0 Rd sup (s,a)ESxA
0 < D), ld(s. (T
for a bounded parameter set Bg. typically defined such that the resulting Q-values are bounded, for some maximum value (e.g.. D Rmaan 7)).
Effectivve exploration for policy learning and data collection is crucial in RL, especially with large state/action spaces, where good representations are key (Jin et al. 2020 Zanette et al. 2020a Azizz zadenesheli al. 2018). While policy learning has advanced (Schulman et al. 2017 Kapturowski ct al. 2018 Badia et al. 2020), representation for exploration remains challenging (Laskin et al. 2020 Nabati et al.) 2023 Ren et al.) 2023). Our work offers unified framework based on low Inherent Bellman Error (IBE) and spectral theory (Ren et al., (003).. We define IBE below.
Definition 2 (Inherent Bellman Error (IBE), (Zanette al. 2020a)). Given an MDP and a feature
map &, the Inherent Bellman Error (IBE) is defined as:
Is == sup inf QEO, QEQ,
TQ Q
sup inf eEB, EEB,
||TQ0 Qúll
2

---

Published as a conference paper at ICLR 2026
If = 0, 2g is closed under T (up to projection onto Bg). Is quantifies how well T maps 2a to itself. Zero IBE means any TQ2 for Qo E Qg is perfectly representable by some Q3 E 2.. This generalizes Linear MDPs (Jin et al. 2020). where T maps any Q-function into 2.p.
2.2 FACILITATING EFFICIENT EXPLORATION WITH LOw-IBE FEATURES
representation with zero or low IBE facilitates efficient exploration. With expected feature o ((.))=d (0(8, a)] under policy (occupancy d effective exploration aims to reduce uncertainty, e.g., by minimizing dells-1 across policies, Zanette et al. (2020b) define maximum uncertainty:
Definition 3 (Max Uncertainty (Zanette et al. max, vo |1
2020b)). U(o)
max I o
Informally, our goal is to reduce maximum uncertainty. This can be done by overestimating via sampling exploration noise & E IRd (N0 (0, per rollout (i.e., Thompson Sampling), where
[klll2 S Rmax (Zanette et al. 2020b).
Motivated by the IBE, in what follows we detail a method for learning low-IBE representations and integrate them with exploration methods like Thompson Sampling (TS) to improve RI. performance.
SPECTRAL BELLMAN METHOD
This section introduces the Spectral Bellman Method (SBM), a framework designed to learn repre- sentations that satisfy the zero-IBE condition (Definition 2). While the zero-IBE condition implies that the function space is closed under the Bellman operator, directly enforcing this condition leads computationally intractable min-max optimization problems. Our analysis reveals fundamental connection between the spectral properties of the optimal Bellman operator and the covariance struc- ture of the learned features under orthogonality assumptions. By leveraging this connection, inspired by the power iteration method, we derive a new learning objective that overcomes the limitations of direct Bellman error minimization.
3.1 ZERO-IBE OBJECTIVE
Consider the mapping (()) : B3 + BB, that maps parameter B& into his correspond Bellman minimizer 0 E B, (noting that for the optimal Q-function parameter, this is the identity map):
{()) arg min T Qallo
8EB,
Our goal is to learn features (ss, a) E Rd and a corresponding mapping ((0) such that the function space 2p is closed under the Bellman operator, i.e., Is 0. However, straightforward idea for minimizing the IBE leads to a complicated optimization (Modi et al. 2024). and thus, difficult to be implemented for practical applications.
Given set of observations {ss: ag 1, sampled from distributions p(s, a) over state-action pairs and v0)) over imput Q Q-function parameters, IBE condition implies the minimization of mean squared error (MSE) between the Bellman backup of Qo and its representation in the learned feature space:
LMSE(Ø 0) = ((t.a))=((e.a..00v00
TQo(s, a) 0(s, a) TO(0)
(1)
However, even the minimization of Equation II) is still challenging. First, the Bellman operator T highly non-linear in Qo (which depends on the learned and 0) and the task parameters 0, complicating the joint optimization landscape and risking suboptimal minima poor features. Second, this MSE objective neither leverages the underlying spectral properties of TQ nor inherently promotes desirable structural properties in & and Ó. To address these limitations, we next introduce a principled learning objective.
spectral analysis of the Bellman operator to derive
The core distinction of SBM is its objective derived from the zero IBE condition, which seeks closure of Qs under the Bellman operator across distribution of parameters. This contrasts with: (1) methods focusing only on the optimal O-function (Mnih et al. 2013) Kapturowski et al.) 2018)) which may be unfeasible to learn directly (Du al. 2019); (2) Learn successor representation or Laplace
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Spectral Bellman Power Method
1: Initialize 0, do
2: for t (1,2....) do
3: Compute AI,t Bpls.a) [0r(s, ")) 22, Evjo) 0,0.7
4: Find & and 0 which satisty the following constraints: A2, a) = (s,,),0(()))
E8
(0, (TQg(),
0 E BB
(o.(). (6()),,())) (-) = 0. (0.(),,(()) =0, i 14.
(3)
5: end for
Barreto et al. 2017]] Touati & Ollivier 2021 which are task-agnostic and not task-specific as our method.
methods (Mahadevan] 2005
Farebrother et al. 2023
3.2 BELLMAN SPECTRAL DECOMPOSITION UNDER ZERO IBE
To overcome the challenges of directly minimizing Equation ( analyze the spectral properties of the optimal Bellman operator under the zero-IBE condition. This analysis will lead us to an algorithm inspired by the power iteration method for singular value decomposition (Golub & Van Loan 2013 Trefethen & Bau 2022), designed to efficiently learn a zero-IBE representation.
For conceptual clarity, let temporarily adopt matrix perspective, assuming finite state-action spaces (S x A| = n)) and finite parameter space (l3g1 = m). Let k: [n] - S x A be bijection from indices to state-action pairs. We denote the feature matrix as E Raxd (where the i-th row is (((((i)) and the matrix of input parameters θ [0,,..., 0m E Rdxm__ A Q-function Qe is then Φ0, and the collection of Q-functions for all 03 E θ is Q $0 E Raxm, Under the zero-IBE condition, for every 0 E Bg, there exists ((0) such that T(40) = Φ0(0). In matrix form, this means TQ Φ where Ô ((0)),.... ((0m) ) Rddm. To incorporate the influence of the sampling distributions p(s, a) and v0),, we augment the feature and parameter mappings:
a(,,a) == V ((s, a) ()),), 0(0) == (()) Vv()
Consequently, the Bellman operator acting on a distribution of Q-functions can be thought of through an augmented operator, (Q, a) = 5(x (os,a)((0))) 0(0). essence, the augmentation defined in Equation (2) enables us to treat the cerrrbbttoo-weiigheee problem as a standard spectral decomposition on the augmented space.
We also define the feature covariance matrix A = [( [dss, [] and the post-
Bellman parameter covariance matrix A2 = ()) (())
key insight reveals a deep structural connection under the zero-IBE condition. Informally, when the Bellman operator T transforms distribution of Q-functions (weighted by P and v), the resulting (weighted) feature matrix & and (weighted) post-Bellman parameter matrix Ô correspond to the singular vectors of this transformation.
Our spectral Bellman method aim to find representation that is aligned with the singular vectors of the (augmented) Bellman operator. Namely, A₁ and A₂ are diagonal matrices, correspond to the singular values of the Bellman operator. This structural alignment is crucial and forms the basis of our method. We refer the reader to Section A for formal statement and proof of this connection.
This spectral relationship naturally motivates an approach analogous the power iteration method for finding dominant digenvectors/singllan vectors (Golub & Van Loan 2013) Trefethen & Bau (022). The following proposition captures identities central such an iterative process (see Section for proof).
Proposition
The following identities hold:
(TQa(-), ()))) = A,0(0).
TQ (s. a),aa) = A2ō(s,
The above identities closely resemble the update rules of power iteration algorithm (Sanger 1988 Xie et al.) 2015 Guo et al. 2025), but for eigenfunction learning for symmetric operator in semi-supervised learning. This suggests an alternating optimization strategy for learning o and Algorithm [ outlines this iterative procedure. In each round, covariance matrices are computed,
4

---

Published as a conference paper at ICLR 2026
followed by solving linear equations extract the updated representation & and parameters 8. This spectral method offers significant advantages over direct minimization MSE loss (Equation ( as we will discuss in Section 3.4 The alternating updates can stabilize the optimization by decomposing the problem into simpler, often convex, subproblems. Moreover, by leveraging the power iteration structure, method aims for the faster convergence rates characteristic of such techniques, leading more efficiently to the desired representation.
3.3 PRACTICAL SBM LEARNING
While Algorithm | provides conceptual framework based on power iteration, directly solving the system of linear equations in Equation (3) at each step can be computationally prohibitive, especially in environments with large or infinite state-action spaces. To make this approach practical, formulate an objective function whose minimization effectively performs these power iteration steps.
Let A = aa) (s, a) (z a) and A2, Bu(
((()),((0)
T
denote the empirical covari-
matrices at iteration The SBM objective is given by C(o.0.u p) (o) + L2) s.t & @ M'.
(SBM Loss)
where (Ø) R (1) [ (11((s,))l3,,) a)), - 2TQo. a) 0.(00] is the representation loss;
(2) (0)p(sa) |0
(sQ (s, a) is the parameter objective; and
= (s: Rdd [f = vi Vi E [d]
the orthogonal function space of functions X' w.r.t probability P: P 00
The term L1 updates & to align with the Bellman-transffommed Q-values, using the current estimate of parameter covariance 22, and parameters 0.. The term L2 updates to best represent the Bellman- transformed Q-values given the current features de and their covariance A AI, Indeed, minimizing the objective in SBM Loss is equivalent to applying the power method. This is formally shown by the following proposition.
Proposition 2. For any I, a solution 07, of to the power method objective in Equation 3 if and only if it is minimizer of the SBM Loss
proof is provided in Section C This equivalence means that by minimizing L(o, 0) using gradient- based methods, we are effectively performing the updates of the power iteration algorithm. In practice, we solve an unconstrained relaxed version of SBM Loss(sse Section D for details).
3.4 COMPARISON THE SBM LOSS TO THE BELLMAN MSE
Lossover To better understand the advantages of the SBM Loss over the naive MSE objective (Equation IT). consider the following expansion of the MSE objective:
min E (s..))))(ss))))))) 6.0
TQols - ((., T0O0
Cc 2TQo 0(0) 2T) a) o(s, 0((0)000)) (ssa))
-inn E (s..))ppssss))00m0
0.0
x min E ((s.a))p((s.)))mm 0.0
[10((, o) - 2TQ. a))(s, T()0)
where C is constant independent of and Ø. and Å (0)) is noisy, single-sample estimate of the parameter covariance. The SBM objective offers distinct advantages. First, its separation into Li (Φ) and (² enables alternating optimization (inherent to the power method), decomposing the joint problem into tractable subproblems and promoting stability over simultancous MSE updates. Second, SBM's quadratic terms (Ilo(s, "). ( 10(0))1,, ,) use moving average covariance matrices A2, A1,: from the previous iteration, offering robust, statistically grounded regularization. In contrast, MSE's quadratic term [| a) a) uses the noisy, single-sample Å, lacking this stabilizing batch-averaged influence. Finally, SBM incorporates explicit orthogonality regularizer Lorth-
These differences demonstrate SBM's more refined approach to representation learning over direct MSE minimization. These fundamental differences highlight how SBM offers a refined approach
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 Q-Learning with SBM
1: Initialize Q-function parameters dg features do, and replay buffer D.
2: for = ....... do
3: Data Collection with TS Exploration:
4: Generate OrS ~ NNO0, Gexp Ex²) using current O, and E, from th (Equation (
5: Collect data using policy T Trns (s) = arg max ( a)T 0rS, and add D.
arg
6: Policy Optimization:
7:
Update Q-function parameters: 0.41- arg ming fou (0; 9)) (Equation 6 using D.
8: Representation Learning:
9: Define (0) NV(0+1,, Pap and Pr a) probability over D (e.g. uniform).
Update features: Φ++1 = arg ming L(o, (O), SBM Loss
11: end for
for representation learning than direct MSE minimization, which is also justified empirically in (Guo
et al. 2025).
3.5 EFFICIENT EXPLORATION USING THOMPSON SAMPLING
After covering spectral representation learning part, learned representation can be leveraged to establish an efficient exploration method, In this work, we focus on Thompson Sampling (TS)
(Osband et al. 2013 2016 Azizzadenesheli et al., 2018, Zanette et al., 2020b), which is particularly
suited for settings with low-IBE representations. Indeed, given dataset D = (8s, 04 T4, &
least-square estimator for optimal parameter 0* E B3 (such that TQ2. Qs-) can be found:
0LS E arg max E (s) ((Toc((((OOs))))
Φ(s, TO)
AEB,
We conduct exploration following Zanette et al. 2020b Specifically, the weights Ôrs for the 0rs, are sampled from posterior distribution,
behavior policy T (s) arg max aE.4 Φ(s, a) often referred to as the uncertainty set. Given the representation, the weights are sampled from the posterior before each rollout according to:
0rs
(epLs::eapET1) E Σ ((s.a)o(saa), (e.0)ED
(5)
for some positive A. TS provides an easy-to-implement method that add an exploration noise the least-squre estimate 0 (i.e., 0rs = 0 + E), which yields the sampling procedure in Equation (5) and reduce the maximum uncertainty in Definition? 3 We emphasize although mainly consider the TS for exploration, the learned spectral Bellman representation is also compatible UCB with the same covariance Σ in equation 5 discussed in (Zanette al., 2020b; Modi et al.) 2024 Zhang eet al. 2022). Having established the representation learning and exploration framework, we next explore its practical integration into value-based reinforcement learning algorithms, using Q-learning (Sutton et al. 1998) as an illustrative example due to its intrinsic connection with the optimal Bellman operator.
Q-LEARNING WITH SBM
Put the representation learning and RI. together, we obtain Algorithm 2 which describes the imple- mentation of the Spectral Bellman Method (SBM) using Q-learning and Thompson Sampling (TS) for exploration. The algorithm iteratively refines the policy and representation through three key phases: i) data collection with TS exploration, ii) policy optimization, and iii) spectral representation learning.
Algorithm begins by initializing the Q-function parameters dg feature mapping do. and an empty replay buffer 'D (Line 1). For every iteration f, the algorithm samples exploratory Q-function parameters Ors (Lines 3-5) using the current Q-parameters Ó, (as the mean for the TS posterior, akin to θr. in Equation (4) and the feature covariance Σ, (derived from the current features o, as per Equation ). The agent then interacts with the environment using a policy Tars that greedy with respect Qara (8,a) dr(s, a) Örs. The collected state-action-rewanr transitions stored in the

---

Published as a conference paper at ICLR 2026
m) A m.y!"
update
t 2
82
6
0.
NO Figure 2: Visualization of the parameter sampling distribution v2( =N Pep successive rounds. As Q-learning updates the mean 0,, v()) shifts, focusing representation learning on parameters relevant the current policy. Darker regions indicate higher probability density.
for d 2 over
replay buffer DD This mechanism ensures that data collection guided by the uncertainty captured in the learned feature space. Following data collection, the Q-function parameters are updated from 0 01+1 (Lines 6-7). This update achieved by minimizing the standard loss with Q-Jearning target: (0; 6) (s,a)>> [(TQ0- (s, a) o(s, a) TO)²].
(6)
where A- are the target parameters. This loss utilize the current features Φi and mini-batches of data sampled from the replay buffer D. This step refines the agent's policy based the existing representation of state-action values.
Finally, in the final representation learning phase (Lines 8-10), the feature mapping is updated from or to 0 1 by minimizing the SBM Loss A critical aspect of this phase is the choice of the distribution over parameters, v(0) and over state-actions p(s, a) Equation ( v(0) is centered around the newly updated Q-parameters obtained from the policy optimization phase: v(0) = N (0 ") where o is a variance hyperparameter. The state-action distribution p(s, a) for the SBM loss is implicitly defined by sampling transitions from the replay buffer D. This adaptive focus of v(0), as visualized in Figure ensures that the feature learning process concentrates satisfying the low-IBE condition in regions of the parameter space that are most relevant to the current policy.
This alternating process of exploration, policy optimization, and representation learning allows the policy and the features to co-evolve. Improved features lead to more accurate value estimates and consequently better policies. turn, an improved policy guides the representation learning towards more effective exploration.
Extension to Multi-Step Operators. The SBM framework naturally extends to h-step Bellman operators Th A low one-step IBE implies low h-step IBE (Section D). Thus, we can apply the SBM Loss by simply replacing the one-step backup T TQp,; with a multi-step target, such as one from (etrace(A) Munnos et 2016).
5
EXPERIMENTS
We evaluate the effectiveness of incorporating Spectral Bellman Representation learning into standard deep RL algorithms. We focus on challenging benchmarks and compare against established baselines.
5.1 SETTING
Environments: We use the Atari game suite (ALE, Bellemare et al. (201)). We follow standard preprocessing steps, including frame stacking (4 frames), grayscale conversion, downsampling, and
7

---

Published as a conference paper at ICLR 2026
Atari ALE
Atari Explore
Method
Mean
Median
Mean
Median
DQN (Mnih et al.
2013
1.62 + 0.12
0.52
0.24 = 0.03
0.11
Online PVN (c-greedy)
1.72 0.11
0.60
0.26 0.03
0.21
Online PVN (TS)
1.84 0.15
0.65
0.31 0.04
0.23
SBM DQN (e-greedy).
1.80
0.64
0.33
0.03
0.23
SBM + DQN (TS)
2
0.85
0.45
0.05
0.24
R2D2 (Kapturowski et 2018)
3.21
1.14
0.40
0.06
0.22
SBM + R2D2 (e-greedy)
3.3 0.24
1.14
0.45
0.05
0.22
SBM+ + R2D2 (TS)
3.53 + 0.23
1.37
0.61 = 0.03
0.30
Table 1: Aggregated Atari HNS at 100M steps. Our SBM method with TS bold.
Method
Atari ALE
Atari Explore
Features from DQN Loss (Azizzadenesheli al. 2018)
1.73 1 0.14
0.30 0.03
Features from Naive MSE Loss
1.82 + 0.12
0.37 + 0.04
SBM w/o Orthogonality Reg.
2.11 + 0.11
0.43 0.05
SBM Full
2.23 + 0.19
0.45 - 0.05
Table 2: Ablation study on the DQN backbone with TS exploration.
sticky actions unless specified otherwise. Performance is measured using mean and median Human Normalized Score (HNS) across all 55 games after 100 million environment steps.
To assess the impact of our method on exploration, we report aggregated HNS on a subset of particularly challenging Atari games identified by [Badia et al. (2020). This subset includes games known for sparse rewards or requiring long-term credit assignment: Montezuma's 's Revenge, Pitfall!, Private Eye, Skiing, Freeway, Solaris, Venture, BeamRider and Pong. We denote this benchmark as Atari Explore.
Our spectral learning method is integrated into two representative deep RL agents: DQN (Miih et al. 2013) and R2D2 (Kapturowski et al. 2018).
Implementation Details:
We minimize the Equation (SBM Loss) using stochastic gradient descent. This is also true for the Q-learning phase. Therefore. SBM was actually tested in a fully online setting similar to DQN. To satisfy the theoretical assumption [, a)ll S 1 used in the background (Section 2.1 we apply norm clipping to the output of the feature encoder network. We follow Algorithm 2 performing
alternating updates between the Q-learning objective (Fquation (6)) and the spectral representation
objective (SBM Loss The distribution v(0) is set to N (00+1,, () as explained in Section For
specific hyperparameters and implementation details refer the reader to Section E We compare
standard e-greedy exploration with the TS approach described in Section [3.3] As R2D2 operates
in distributed setting with multiple asynchronous actors, the precision matrix Σ is shared across
the actors while each of them sample 0rs before each epoch according to Equation 5 For R2D2 experiments, the spectral objective is applied using multi-step operator in the form of Retrace operator target (Munos et al., 2016) in place of TQq. (see Section D). We found that choosing Tepp is crucial for successtul learning. We conducted grid search and empirically found the best value to be of 10-2
In addition, in order to compare SBM to a Laplace based representation method, we compare SBM to PVN Faarerothhrr et al. (2023). To fairly compare with SBM, we adapted PVN to our online learning setup. We alternate between policy optimization (DQN) and representation learning. The representation learning phase uses Random Network Indicators (RNI) to define auxiliary tasks, over the collected data, as in the original PVN. We used the same network architecture and latent dimension as our SBM experiments. We evaluated Online PVN with both e-greedy and TS. Lastly, we made an ablation study with DQN backbone. We examine SBM with features extracted with naive MSE loss in Equation ( and not with our suggested spectral representation. Furthermore, we investigated
the specific case where (0) = 8(0 0t)) under the naive MSE loss. This configuration effectively corresponds to standard DQN training in Equation (6) augmented with TS exploration, suggest by /Azizzadenesheli et al. (0018). We also examine the effect of the orthogonality regularization by training SBM without it. All runs are done over 10 seeds.
8

---

Published as a conference paper at ICLR 2026
Atari ALE Average
Atari Explore Average
0.6
0.5
2.5
2.0
0.3
1.5
0.2
1.0
DCN
0.5
0.0
0.0
100
Figure 3: Average HNS over 100M steps. DQN and R2D2 against their SBM counterparts with TS across Atari ALE benchmark (left) and on the hard-expe hard-explooation subset (right).
5.2 MAIN RESULTS
T
presents results comparing the baseline agents with their SBM counterparts, which incorporate spectral representation learning and exploration. Figure presents the results against environment steps over the training. Per game results are presented in Section
We find SBM significantly outperforms vanilla DQN. Furthermore, combining spectral learning with TS exploration yields substantial gains, particularly on hard exploration tasks, suggesting that the learned representation effectively facilitates structured exploration. We observed that SBM underperforms compared DQN precision-contro games such as Breakout, This is characteristic side-effect of optimistic exploration strategies (like TS) in environments with 'narrow' optimal paths or instant-death conditions, where high-variance actions often lead termination. This effect can be mitigated by annealing the exploration noise switching greedy evaluation once sufficient performance threshold is reached.
DQN Comparison:
We augment the R2D2 agent, which utilizes recurrent networks and Retrace targets, with our spectral representation learning objective applied to the Retrace target. R2D2 with SBM demonstrates improved performance
R2D2 Comparison:
the baseline R2D2, with the most significant improvement observed when combined with TS, particularly on the Atari Explore subset.
Online PVN Comparison: While Online PVN improves over DQN, SBM with TS performs better, especially on Atari Explore. This is likely because Online PVN learns reward-agnostic features using successor measure representation, capturing state-action structure on various MDPs, whereas SBM learns reward-deye dependden features by enforcing closure under the optimal Bellman operator. SBM's approach appears better for maximizing online rewards, while PVN's may be more suited for adapting varied reward tasks.
The results are presented in Table SBM-learned features yielded significantly better results compared to features learned by minimizing the naive MSE loss, highlighting the advan- tage of the spectral objective over direct Bellman error minimization. While limiting representation learning to the current policy (effectively the DQN objective) combined with TS exploration improves performance over vanilla e-greedy DQN, it causes significant degradation compared to SBM or even the full Naive MSE loss. This indicates the critical importance of learning representations over distribution of value functions rather than solely fitting the current policy. Additionally, performance slightly lower, suggesting that while beneficial, striet
Ablation Study:
without the orthogonality regularizer orthogonality is not the primary driver of SBM's gains. The computational overhead of SBM is relatively small, resulting in 20% increase compute time per IM steps. This overhead primarily stems from training the 0 network and computing the inverse covariance for Thompson Sampling. Notably, the core representation learning cost mirrors vanilla baselines, while the Q-learning phase remains highly efficient by updating only linear weights.
RELATED WORK
Our work builds upon and distinguishes itself from several key areas reinforcement learning.
Representation Learning in RL: Many auxiliary objectives, such contrastive learning (Laskin eet al.) 2020; Schwarzer et al. 202T) and autoencoders (Kingma et al. (201)). don't explicitly optimize for Bellman compatibility. Laplace-based methods, including Successor Features (Mahadevan 2005

---

Published as a conference paper at ICLR 2026
Barreto et al. 2017; Touati & Ollivier 2021 Farebrother et al. 2023 often aim for task- agnostic, generalizable representations, potentially less potent for task-specific optimization as done in our work. Directly learning linear representations for the optimal Q-function can be sample inefficient under misspecification (Du et al. 2019; Azizzadenesheli et al.
2018 In contrast, SBM, derives its objective 2020a targeting Bellman consistency across
from structural properties under low IBE (Zanette et al.
the function space, which makes the IBE-based model free representation learning (Modi et al. 2024)
eventually practical.
Linear MDPs and IBE: Linear MDPs (Jin et al. 2020 Yang & Wang 2020) assume linear transitions and rewards in features, a practically challenging condition (Agarwal et al. 2020; Zhang et al.) 2022
Ren et al. 2022). The IBE (Zanette et al. (0200)) relaxes this, measuring if the function space 2o is nearly closed under the Bellman operator (TQ, Qa). Low IBE, more direct target than full MDP linearization, ensures theoretical guarantees, e.g.. Za.ettee et al. (2020a) presented
planning-based algorithm achieving Odd /T + vdHTI,) ) regret under low-IBE settings. While practical methods approximate linear MDPs by combining representation learning for RL with some dynamics assumptions (Zhang et al. 2022 Ren et al. 2022] [Shribak et al. 2024 Fujimoto et al.) 2025), our work inspired by the Ig condition's 's structural implications. avoiding full Linear MDP decomposition which be resource- resource-intensive and require impractical latent dimensions with heuristic nonlinear correction in (Fujimoto et al. 2025).
Bellman Error/Residual Minimization: Traditional value- RL minimizes the Bellman error via methods like gradient TD (Sutton et al.) 2009). residual algorithms (Baird) 1995), and target networks (Mnih et al. 2013). Distributional RL (Bellemare et al. (017)) matches return distributions. Unlike these, our method doesn't I directly minimize the residual for single 0, but leverages structural relationships (specifically covariance aligmment) emerging from assuming zero residual across function distribution induced by v()).
Exploration Strategies: In contrast to count-based methods (Bellemare et al., 2016) and intrinsic
motivation (Pathak et al. 2017). TS offers Bayesian uncertainty approach, effective in linear settings (Agrawal Goyal 2013 Azizzadenesheli et al. (0188). Notably, /Zanette et al. (2020b) analyzes TS for RL with low-IBE representations. Our contribution is synergistic: our spectral objective learns suitable representations s, and TS naturally utilizes the resulting feature covariance Σ, for more directed exploration.
CONCLUSION
We introduced Spectral Bellman Method (SBM), theoretically method for spectral representation learning and efficient TS based exploration. Our spectral objective derived from the zero-IBE condition allows for theoretically-goondee representation. We show that the SBM is compatible with multi-step operators such as Retrace. Experiments on Atari demonstrate consistent improvements over baselines, especially on hard-exploration games and when applied to advanced agents (R2D2), validating our approach.
key limitation includes the sensitivity of SBM the parameter distribution, which needs to be tuned carefully. Also, a broader empirical validation beyond Atari would be beneficial (e.g.. continu- control tasks). Future work may focus strengthening the theoretical analysis (convergence, non-zero IBE), developing improved algorithms (better approximations of Ô, adaptive parameter sampling), and extending the framework to other settings (e.g., distributional or actor-critic methods).
In conclusion, SBM offers promising leading to more efficient, stable, and exploratory RL agents, by explicitly optimizing features for Bellman consistency across the function space.
for representation learning and exploration, potentially
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alckh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33:20095- 20107, 2020.
Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoff's. In International conference machine learning, pp. 127-135. PMLR, 2013.
Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International conference on machine learning, pp. 507-517. PMLR, 2020.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30 Elsevier, 1995.
André Barreto. Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado P Van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, volume 30, 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count- exploration and intrinsic motivation. In Advances in neural information processing systems, volume 29, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents, Journal of artificial intelligence research, 47:253-279, 2013.
Mare G Bellemare. Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In International conference mackine learning, pp. 449-458. PMLR, 2017.
Simon S Du, Sham M Kakade, Ruosong Wang, and Lin F Yang. Is a good representation sufficient for sample efficient reinforcement learning? arXiv preprint 11000:19,. 2019.
Yonathan Efroni, Gal Dalal, Bruno Scherrer, and Shie Mannor. Beyond the one-step greedy approach in
reinforcement learning. In International Conference Machine Learning, pp. 1387-1396. PMLR, 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Sealable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMIR, 2018.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G Bellemare, Proto-value networks: Scaling representation learning with auxiliary tasks. arXiv preprint arXiv:2304.12566 2023.
Scott Fujimoto, Pierluca D' D'Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. arXiv preprint wwXiv:2501.1614 2025.
Gene H. Golub and Charles F. Van Loan. Matrix Computations - 4th Edition. Johns Hopkins University Press, Philadelphia, PA, 4th edition, 2013. doi:
Zhaohan Daniel Guo, Bernardo Avila Pires, Khimya Khetarpal, Dale Schuurmans, and Bo Dai. Representation learning via non-contrastive mutual information. arXiv preprint arXiv:2504. 2606.76677. 2025.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference learning theory, pp. 2137-2143. PMLR, 2020.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018
Diederik P Kingma. Adam: method for stochastic optimization. arXiv preprint 4rX1v:1412.6980, 2014.
Diederik P Kingma, Max Welling, al. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 1242)3377332, 2019.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp- 5639-5660.. PMI., 2020.
11

---

Published as a conference paper at ICLR 2026
Sridhar Mahadevan. Proto-value functions: Developmental reinforcement learning. In Proceedings of the 22nd international conference on Machine learning, pp. 553-560, 2005.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model-fiie free representation learning and exploration in low-rank mdps. Journal of Machine Learning Research, 25(6): 2024.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off off-policy reinforce- ment learning. Advances in neural information processing systems, 29, 2016.
Ofir Nabati, Guy Tennenholtz, and Shie Mannor. Representation-drive reinforcement learning. International Conference on Machine Learning, pp. 25588-25603. PMLR, 2023.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. conniicondiiional video prediction using deep networks in atari games. In Advances in neural information processing systems. volume 28, 2015.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. International conference on machine learning, pp. 2778-2787, PMLR, 2017.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado Van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv: 1805. 1593, 2018
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2: 331 434, 1990.
Tongzheng Ren, Tianjun Zhang, Lisa Lee, Joseph E Gonzalez, Dale Schuurmans, and Bo Dai. Spectral decomposition representation for reinforcement learning. arXiv preprint urXiv:2208. 09510, 2022.
Tongzheng Ren, Tianjun Zhang, Lisa Lee. Joseph E. Gonzalez, Dale Schuurmans, and Bo Dai. Spectral decomposition representation for reinforcement learning. 2023. (RRLLLttt://arrtttt org/ abs ab2
09515
Terence D Sanger. Optimal unsupervised learning. Neural Networks, 1(Supplementt1):127, 1988.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Gracpel, et al. Mastering atari, go, chess and shogi by planning with leameed model. Nature, 588(7839):604- 6609, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:
2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bachman. Data- efficient reinforcemntt learning with self-predictive representations. In International Conference on Learning Representations, 2021.
Dmitry Shribak, Chen-Xiao Gao, Yitong Li, Chenjun Xiao, and Bo Dai. Diffusion spectral representation for reinforcement learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 10028-1000000 10056. Cur- Associates, Inc., 2024. httpll/tta////ppp 0022//Efeee/c68.8.00222233606610
ngs neuriss wwwpspppppppppeeeeeeeeeeeeeeeeee fiwesf/aperrr.// Paper
Richard Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Richard S Sutton, Hamid Reza Maei, Doina Precup. Shalabh Bhatnagar, David Silver, Csaba Szepesvári. and Eric Wiewiora. Fast gradient-descent methods for femporal-differeee learning with linear function approximation. In Proceedings of the 26th annual international conference on muchine learning, pp. 993-1000, 2009.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13-23, 2021.
12

---

Published as a conference paper at ICLR 2026
Lloyd N Trefethen and David Bau. Numerical linear algebra. SIAM, 2022.
Stephen J Wright. Numerical optimization, 2006.
Yifan Wu, George Tucker, and Ofir Nachum, The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXiv:/810.0000 04586, 2018.
Bo Xie, Yingyu I. Liang, and Le Song. Scale up nonlinear component analysis with doubly stochastic gradients. Advances in Neural Information Processing Systems, 28. 2015.
Lin Yang and Mengdi Wang. Reinforcement learning feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pp. 10746-10756. PMLR, 2020.
Andrea Zanctte, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Muchine Leurning, pp. 10978-10989. PMLR, 2020a.
Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Provably efficient reward- agnostic navigation with linear value iteration. Advances in Neural Information Processing Systems, 33: 11756-11766, 2020b.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021.
Tianjun Zhang. Tongzheng Ren, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, and Dai. Making linear mdps practical via contrastive representation learning. In International Conference on Machine Learning, pp- 26466. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2026
A THE BELLMAN OPERATOR SPECTRAL DECOMPOSITION THEOREM
We start by incorporating distributions over states/actions and parameters, let p(s, a) be distribution over S x A. In the finite case, let Ps.a be a diagonal matrix with v /P(x(i)) on the diagonal. Similarly, let Pe represent weighting by vv(0) (conceptually, an operator diagonal matrix if Bg and finite). We define the augmented feature and parameter matrices as 4p Pe.a and θp O.. The distribution aaggmeetdd Bellman matrix becomes:
TQ = ( (T - Pa.a DOP, - pp
The following theorem highlights a structural connection between the (weighted) feature matrix, (weighted) parameter matrix and the singular matrix of the Bellman matrix.
Theorem (Bellman Operator Spectral Decomposition). Let Φ : S x A Rdd be a feature map with I& 0. Let the feature covariance matrix under p(s, a) be
A == ((ssa))=lss. [o(s, a)o(s, a) '] b ppp.
where A = diag (Ai)am)) Also, let the parameter covariance also be diagonal matrix:
(0) (())(()) T = Öppp.
with M didag(())
Consider the SVD of the augmented Bellman matrix TQ = UEV then the SVD components satisfy:
1. The non-zero singular values in E (up to d) are related to the values of A and M. Specifically, E has the form diag( "'im,,,.. Vdataa 0,...) up to permulation. Let Σn diag(o1......0a) where a; = VX,m.
2. The corresponding left singular vectors (first d columns of U, denoted Ú) and right singular vectors (first d columns of V, denoted V) satisfy:
Dp USS and 3p V
where EgLi E. Σ As, and 23 = M.
Proof. Under the assumption of zero Bellman error, we have that
TQ TQ 6 where the last equality follows from the SVD of TQ. Given that p = we have:
VΣ ΣV
θAΛÂP VΣ ΣVT.
Since assume M θ is diagonal, the rows of Ô orthogonal. This implies that Ôp can be factored as Ôp = A VT where E RRmxd has orthonormal columns. Substituting this into the expression:
= V /2M
Since and 21/2 are diagonal, they commute. Thus:
VAMİ = VΣ Σ
Comparing both sides, identify that the singular values are ΣT AM (implying the singular values are VA,m) and the right singular vectors correspond to V.
Similarly, for the left singular vectors, we have:
TQTQ" = - UTE
Using θPÔЛ- and the factorization pp UA (implied by t D = and orthogonality of U):
((AA)) = Ü(AM)TT
This confirms that US, = (AM)) = A 1/2 Rearranging for the feature matrix o using the derived singular values yields the final relation:
$ Ü- 0/2
14

---

Published as a conference paper at ICLR 2026
which completes the proof.
B
PROOF FOR PROPOSITION
Proof.
(TQ((),(()) pla.a)
o(s.a) as
(ss,a)((s, a) (())pss
= (s, a) Mss a)dsdaa(())
Ai(0).
(s)((s,)))
(s,
/ ((0)ō(s.a) ((())((0)0)
/ ((0))(00((00:00066 a)
A2o(s.0)
C PROOF FOR PROPOSITION
Proof.
The equivalent objective for the t-th iteration of the power method is given by
/ ((0) (000)) s.t (st a)] = Ev) (,)0) = vi * E [d]
min / [1A2,10(s.)) (s,, 6.0
The choice of the norm is arbitrary. Therefore, we can choose the norm induced by A and Azy for the left and right terms respectively
/ Ael0(s..) / TQo(s,
dsda
Ag7
(C1 (C1+ A2, 20(s.00 a)T
(0)))))))
C1 dsda Ep(a,)) (0(s, a) 1 a)) (0) (Q, a) [())
where C is term that does not depend d. The same practice be used for the right term in Eq 7
E0 0(0)T A ,(0)] (() (s, (s, at) + C2
where C2 is constant which does not depend on 0. Therefore, the overall equivalent objective is L( 0; p) (o) + C2(0) s.l & MMx E My
where
(0) E Ep(s.a) () A2, a)] - 2Ex(())(sa)) TQ,, a)o(s, a) [.
(2 (0) (((0) (000)TA1 0()) 22 [Q2u(s,))((
15

---

Published as a conference paper at ICLR 2026
For the practical unconstrained version of SBM Loss Section
D MULTI-STEP ALGORITHMS
D.1 THE h-STEP OPTIMAL BELLMAN OPERATOR
While the analysis in Section 3 focused on the standard one-step Bellman operator T, the spectral representation learning framework can be naturally extended to multi-stpp operators. Multi-step methods are widely used in modern RL agents they often accelerate learning by propagating information more rapidly through trajectories (Sutton et al., 1998: Pohlen et al. 2018 Schrittwieser al. 2020). Consider the h-step optimal Bellman operator, defined recursively:
==) = T(T"=-TT(TTsss. a) = r(s,a) + 7BBr)(((aa)
max 7 Q(5 a(ss.ad) a'EA
(8)
The optimal Q-function Q' is also the fixed point of Th for any h 2 1 Efroni et al. 2018).
Furthermore, T is known to be contraction mapping with factor 7 in the max-norm (Efroni et al. 2018), which is potentially beneficial for stability. Evaluating T generally requires access to model of the environment, either through simulator learned model (Schrittwieser et al. 2020 often approximated using planning techniques like Monte Carlo Tree Search (MCTS).
We define the inherent Bellman error with respect to this h-step operator:
IS == sup inf || - Qáll
(9)
#EB BEB
Crucially, if the function space p is closed under the one-step operator T, it is also closed under Th
Proposition 3 (h-step IBE Bound), Consider a feature map &. If the one-step IBE is Ig. then the h-step IBE satisfies I3 < Eiz I' S I- In particular, I& = 0 if = 0.
Proof. Using the definition Is = supQ inf & IT (QQoo there exists Q1 such that I| -10 S Is. Similarly, there exists Q2 such that [ - Q211 I Ig. Then, using the triangle incquality and the y-contraction property of T: ||T 2211 = IT(TQ) - TQ, + +TQ1 Q211
$ [T(TQ) -Qqil= + +IT Q2110
< +Ig III + Ig.
Repeating this argument times yields the bound I < Σ/ I'Zz.
This proposition implies that if features yield low (or zero) one-step IBE, they also yield low (or zero) /t-teep IBE. Therefore, the spectral representation learning objective in SBM Loss can be directly applied by replacing the one-step operator TQp., with its h-step counterpart T T"Qa, (or its approximation, e.g., via MCTS). This encourages learning features that linearly represent the outcome of h-step lookahead planning.
D.2 PRACTICAL MULTI-STEP TARGETS
While T offers theoretical appeal, practical deep RL algorithms often employ sampled multi-step targets derived from actual trajectories, such as n-step Q-learning targets or Retrace (Munos et al.) 2016). Let Qa(s, a) = a(s s, a) and (s)s) arg max, Qo(s,a) (such that T TQx). A generic n-step target involves bootstrapping off Qo after n steps.
However, using such targets within spectral learning framework, which involves sampling different 0 ~ v(0) poses a challenge in off-policy settings. The target policy тy associated with each sampled A will likely differ significantly from the behavior policy y used to generate the data in the replay buffer D. This discrepancy can lead to high variance or instability, especially for longer multi-step returns.
To mitigate this, off-policy correction techniques like Retrace(A) Munos et al., 2016) are crucial. Retrace provides return target that mixes n-step returns and function approximation values, using
16

---

Published as a conference paper at ICLR 2026
importance sampling ratios cj min(1, xo(ax (86) }:
p(aa)ss)))
8
Ro az) Qe(s,, ar) + BB
II
cj
8k
(10)
R=t
j=t+1
where Oę = TooQO ax) - Qo(sk- 0x) is the TD error. In practice, the used target policy m6 an e-greedy version of To with e, and - along the training process such that - Qo =0 1 Er Qolloc as suggested by (Munos et al. 2016).
Furthermore, to handle varying reward scales across different environments (c.g., in Atari), value transformations are often applied (Pohlen et al. 2018). A common transformation f(x) =
sign(z) / 11 + eæ for small e. The target operator becomes R2 (Q)).
While the strict condition Ip O does not guarantee that RºQa lies within 2 can still hypothesize that learning features & to linearly represent these practical, information-rich targets is beneficial. We can heuristically apply SBM Loss by replacing TQa,: with RºQ,,.. This encourages to capture structure relevant to these robust, multi-step, off-policy corrected targets used in high- performing agents like R2D2 (Kapturowski al. 2018), even if the direct IBE connection is relaxed.
E IMPLEMENTATION DETAILS: SBM
E. UNCONSTRAINED Loss FOR SBM Loss
The unconstrained objective (the Lagrangian) of SBM Loss is:
C(o, 0,,00 p) Li()) L2(0) Lorth (0, 0),
forth (0, 0) (o) [Eiyeeda Ne, 0,) + ((tt))))ts.a)) Eitjeta &i (i, (s,,))s((; a) is an orthogonality regularizer for set of positive Lagrange multipliers (Au.t P4,5 (4.) [
If 66, 0+ is solution the primal problem in Equation 7 then there exist set of positive Lagrange multipliers {Ai ) ) such that o² 0 satisfies the KKT conditions: (1) VL(6", 0r and (2) T and θ are orthogonal (Theorem 2.1 in Wright 2006
Cortn(o) 8) can be further relaxed into: Corth
(En (0)00
Σ
Pr, (e,((ean))e((sc)))((ssss..
EE
*SEE(d
which can be simplified to the one proposed by Wu et al. (2018):
Lorth(o, 0) =Eg ~u())
(0000000))
+ (s,,)) (t)
0.(s, (ils.))o((ss)))((5.d))) KE
Σ
E.2 IMPLEMENTATION DETAILS
Implementation 0 in SBM Loss The update for (implicitly or explicitly parameterized) within the spectral representation learning phase is crucial. One could approximate ((0) by explicit parameters (reset each time new 0 is sampled) or by parameterized function (e.g. neural network). In this work, we implement as residual network: (()) MLP model.
A where A A(0) is trainable
Moving Average of A. Practically, SBM Loss is designed to be optimized in mini-batch iterative manner. Therefore, for better stability, we suggest using exponential moving average (EMA) updates: 1 a (...( (s, a))x(s,a) + (1 A1, (same for A2) for some 3 [0, 1). This is
17

---

Published as a conference paper at ICLR 2026
solution for the constrained objective:
A1411 E min 1
|| R (ecss) a) "]l + 2 M|l - A 1c
A 2
where 7) > 0 and a = 147"
Network Architecture. For the DQN architecture, we follow the architecture suggested by Mnih et al. (2013) for (bs, a) where the last layer's output dimension changed d [A]. The output is reshaped into a |A| x d matrix such that for any input state we get ø(s, a) for each action E A. The network in the residual model f is 3-layer MLP with dimensions of 2d + 2d + d.
For R2D2 architecture, we follow the ResNet architecture from Espeholt et al. (2018) for the feature extraction from the visual observations here) network to process the sequence of observations. The LSTT head outputs hidden state m, and a state Cz m, with output dimension [A| x d, reshaped into a |A| x matrix such that for any input history h = (O,, Ot.........00 00), we get ((h, a) for each a E A. with the same architecture in the DQN case.
g(oz). We use an LSTTM (put reference to LSTM
f(et, m4-1). The output feature o(h, a) is done by using an MLP
Evaluation Parameters. Each experiment ran for 10 different seeds. During evaluation, we ran the policy 10 times.
Retrace Operator. We used the transformed retrace operator R.j in our R2D2 experiments as target.
E.3
HYPERPARAMETERS
Table Hyperparameters for SBM with DQN
0.99
Latent dimension
256
Learning rate
3 x
Enviornment steps
100M
Batch size
256
Number of representation learning steps
512
Number of policy optimization steps Optimizer
64
Adam (Kingma
2014)
O Ogrp
Xmin >>/
71
0 Orep
10
0.1
Reward clipping value EMA parameter a
1.0
0.1
18

---

Published as a conference paper at ICLR 2026
Table 4: Hyperparameters for SBM with R2D2
?
0.997
Latent dimension
256
Learning rate
3 x 10
Enviornment steps
100M
Batch size
64
Number of representation learning steps
512
Number of policy optimization steps
64
Number of policy optimization steps
64
Burn-in steps (Kapturowski et al. 2018)
40
Trajectory training (max) length (Kapturowski al. 2018)
120
Retrace parameter 3
0.95
Optimizer
Adam Kingma
2014)
Gepp
Amiin
/
2)
Trep
10-2
0.1
EMA parameter a
0.1
19

---

Published as a conference paper at ICLR 2026
F
ATARI SCORE TABLE
Game
Human
Random
DQN
SBM+D9N
R2032
R202-5BM
Alien
7127,70
227.50
1999.79
257
1237.88 + 108.96
679285
295.58
8389.99 1: 384,19
Amidar
1719.50
5.80
433.66
69.68
1159.18 71.13
1573.45
80.54
1596.91
64.33
Assault
742,00
222.40
341.49
54,91
1788.34 61,37
2070.25
285.82
2068
362.52
Asterix
8503, 30
210.00
6167.47 = 1180.6
8681.00 + 1182.53
6224.40 40 615.35
7547. +: 438
Astencids
74.15
1137.54 60.64
1704.3
159.39
Atlantis
29028.10
12850.00
4971166.27 + 4941 23
528266640
73334,77
90377267
97353.82
959381.33
134777.47
Rankheist
753.10
15.06
1136.08 96.93
83,11 14,91
983.00
Botilesone
37187.50 50
2360.00
17066. 67 1426 73
26840.00 + 1394,43
53993.33 = 3267.35
65191.69 1: 3059,00
Beamrider
16925.59
363.90
2241.12
381.89
5684.90 1: 411.47
4116.11 #: 404.47
5531.92 299.30
Herzerk
2630,40
299.52 36.49
746.67 10.03
897.66 92.56
841.77
Bowling
160.70
23.10
21.72 1.77
11:35 + 2.58
228.65 1.78
282.34 2 1.29
Bosing
12,10
0.62
Becakoor
30,50
1,70
74.07 = 14.16
32.47 + 2.82
197.29 32.99
183.10 + 23.84
Centipede
12017.00
2090
2479.27 545.55
316483 + 708.75
15334.34 1810.33
16902.07 :
Chopprrcmmand
2367,80
811.00
610.13 91.47
915.20 62.60
1753.27 213.60
1950,43 1: 189.47
Crayclimber
25829.40
10780.30
58781.87
4431.00
91828.00 + 7361.11
107756.13 4010.5
136573.43 3712.86
Demanattack
1971,00
4431.15
712.73
2691 4088 96
Dauhledunk
-16.40
-18.60
-2.24 1.45
-2.53 = 1.87
7.04 2.21
8.77 + 2.55
Enduru
860.50
41.65
2305 40.05
Fishingddrryy
-38.70
-91.70
3.20 = 2.76
-21.67 + 4.62
35,73 = 2.75
33.77 + 2.69
Freeway
29.60
19:07
31.29 0.30
30.82 0.07
33,00 0.14
Prosthise
4334,70
65:20
2585. 60 278.19
$35.04 117,30
7726.51 220.78
8734,95 211.13
Gopher
2412.50
257.00
694.18 3546.45
21192.16 +: 2595.47
9775 83 + 2533
11532,03 2724.36
Gravitar
3351.40
242.00
3370,03 226.38
195.74
Heno
30826.40
1027 00
16431 15 1016.03
21358.04 + 1150.31
25055.32 31.69
29W76.29 99 1 3487
lochuckey
Jamesbond
362.80
29.00
445 K7 40.24
646.50 50 + 40.03
721 78.90
734,88 + 88.68
Kangaruo
3035,00
52.00
2999.17 333.
7840.50 680.31
11265.80 478.40
12907.43 561.41
Krull
2665.50
1598.00
4504.32
11408.32 248.90
21415.33 3964.99
23311.90 +: 3284.52
Kungfumaster
22736.30
258
3136.00 843.01
19676.00 + 1179.52
$4325.20 = 1823.20
43780.17 + 1773.79
4753.30
0.00
219.29
1176.24 + 181.95
Mapocmma
6951.60
307.30
2289.07 296.70
2578.40 1: 186.92
6960.11 =: 353.58
8120.56 1: 378,52
Namethisgame
5047,89
391.33
1894.12 341.02
Phoenix
7242.60
761.40
4967 68 753.66
6285.84 + 15360
4347.37 355.99
3987.71 1: 444,14
Pitfall
6463.70
-299.00
-9..66 10.22
14,03 8.35
-522 :: 2.51
-3.69 3.23
Pong
14,60
18662
20.47 + 0.18
Privarecye
69571.30
24.00
-547 84 84.07
-181.95 2: 90.23
58212.54 = 1274.77
51369.59 + 1177.57
Qbert
13455.00
8241.20 1016.85
20062.47
18471.19
Riverraid
17118.00
1338.50
6721.28
$10.13
12111.44 1: 712.36
7401.3 + 308.99
9564.34 1: 282
Roadrunner
2845.00
33083.73
1722.53
46015.20 2269.45
42994477 2139.38
41526.14 2726.37
Robetank
11:90
2.20
27.99 5,36
17.86 + 2.72
60.61 1.81
62.75 + 1.47
Seaquest
42054.70
68.40
2896.64
363.70
2261.60 +: 209.35
3343,95 195.17
3156.72 4: 234.17
Sking
4336.90
-17099.10
-15384
278.75
-15214911 1750.84
0.00
-17024.68 0.00
Salari
12326.70
1236. 30
1690.21
421.35
1806 1 44.73
$110.99 = 378.79
4061.02 1: 491.34
Spucenndeess
1668.70
226
Stargunerr
10250.00
664.00
3336100
2218.71
6512.00 1: $49.11
2723.93 :i $48.09
2621.41 1: 690.72
Tennis
-8.30
11.69
1,57
Timepilor
$229.20
3568.00
$230.93 337.32
7594.40 + 321.73
8132.7 499.41
9550L81 + 547.06
Tutankham
167.50
11.40
124.80 + 7.70
189.64 + 10.34
112.05 18.91
134,66 + 19.94
Upodown
11693.20
84
7714495
12594.93
99513.17 9181.75
Venture
1187.50
0.00
439.47 = 115.21
1182:07 1 19:43
1365.00 45.61
1949.20 :: 39.73
Videopinball
17667.90
22379.44
2432.50
Wizardofwor
4756.50
$63.50
955.73 # 453.05
2129.60 + 136.21
10283.00
1960.67
9889.49 + 2038.
Yanseevenge
54576.90
25562.24
27827.71 + 2219.92
60559.04
3913.49
73644.13 +
Zaxxon
9173.30
32.50
5619.20 466.03
3238.40 + 384.70
13134.33 = 1342.95
17187.17 : 1234.38
20


Reviews:
[{"review_id": "MmMHE1zqhj", "reviewer": "Reviewer_5nzo", "summary": "The paper proposes a new approach, the Spectral Bellman Method, for learning representations that aid exploration. The approach is built upon the Inherent Bellman Error (IBE) condition, leveraging the key insight that connects feature covariance and zero-IBE to derive an algorithm that alternates between learning features and Q-values. Furthermore, the method also leverages the underlying structural properties to enhance exploration via the Thompson sampling technique. Experiments conducted on the full Atari game suite show improvements over other baselines, observing significant gains on deep exploration games and corroborating the effectiveness of the approach.", "questions": "- The paper's claim that \"effective exploration aims to reduce uncertainty\" (lines 110-112) is presented as a general-purpose objective for exploration. The authors should justify this framing, as it seems to imply that exploration is also a problem in policy evaluation, not just control.\n- The paper's assertion (line 149) that not resulting in task-agnostic representations is a feature is counterintuitive. Given that task-agnostic representations are generally desirable for fast adaptation to changing goals, the authors should provide a clear justification for why their task-primary approach is a benefit and not a limitation.\n- The introduction of Equation 2 lacks sufficient motivation and derivation. It appears abruptly in the text, and the authors should provide more insight into how this equation was derived and why it is the correct formulation in this context.\n- Building on the weakness of the missing ablation studies, the paper should provide the intuition or justification for the specific choices of parameters used in the experiments (e.g., for the state-action distribution and Q-function). An explanation of why these particular choices were made is necessary.\n- The algorithm, as presented in the pseudocode, appears to operate on a batch of data for each update. This raises questions about the method's feasibility and efficiency in a fully online or streaming setting. The authors should clarify whether their approach can be adapted for such scenarios.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a well-executed idea. The proposed method has solid mathematical backing, and the derived algorithm stems directly from the theoretical insights. It is encouraging to see these insights translate into meaningful empirical gains.\n- The key insight regarding the spectral properties and the zero-IBE condition is particularly elegant, as it helps reduce a complex optimization problem to a more tractable one. It is a further strength that the Thompson sampling exploration fits well with the approach, tackling both feature learning and exploration simultaneously.\n- The resulting algorithm represents only a minor modification to existing algorithms, such as DQN or R2D2, yet it achieves substantial empirical gains on many Atari games, including those where exploration is challenging.", "weaknesses": "- The mathematical derivations in Section 3 could be significantly improved for clarity. As written, the section is overly dense. The authors should consider simplifying complex notations (e.g., $\\tilde{\\theta}(\\theta)$) and adding intuitive explanations to make the theoretical statements more comprehensible\n- The paper would also benefit from a small-scale, illustrative experiment. Such an experiment would be valuable for building intuition and helping isolate the source of the method's benefits (representations, exploration, or both). Additionally, a discussion or analysis of the method's sensitivity to key parameters (such as the choice of state-action distribution pairs and Q-function parameters) is a notable omission.\n- The experimental comparison to other representation learning methods is narrow, as it only includes one such baseline (Online-PVNs). A more thorough evaluation against other common baselines&mdash;particularly those using an auxiliary loss for representation learning&mdash;is needed to fully contextualize the proposed method's performance. This is especially true since several such methods are already discussed in the related work section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKPVuKHpn7", "reviewer": "Reviewer_pqsD", "summary": "This paper introduces the Spectral Bellman Method (SBM), a novel framework designed to unify representation learning and efficient exploration in reinforcement learning.The central problem addressed is that these two critical components are often treated as separate modules, failing to leverage their potential synergy. The motivation stems from the insight that while representations with low Inherent Bellman Error (IBE) are highly desirable for value-based methods, directly optimizing for them is computationally intractable. The authors identify a key spectral relationship: under the ideal zero-IBE condition, the structure of the Bellman operator is fundamentally linked to the covariance structure of the features themselves. To solve this, SBM proposes a new learning objective inspired by the power iteration method. Instead of directly minimizing Bellman error, this objective encourages the feature covariance to align with the Bellman dynamics across a distribution of value functions. This serves as a tractable proxy for learning low-IBE representations. The method is integrated into an alternating training loop where the learned feature covariance is naturally used to drive exploration via Thompson Sampling, which in turn collects data to improve the policy and refine the representation learning target.", "questions": "1. Could the authors elaborate on the behavior of the SBM objective when the Inherent Bellman Error (IBE) is small but non-zero? Specifically, how does the relationship between the feature covariance and the Bellman operator evolve in this case, and does the SBM loss remain a more faithful proxy for minimizing IBE than the standard MSE objective? \n\n2. Given that the SBM objective depends on the Bellman operator of the current policy, how do the authors prevent the learned representation from becoming overly specialized to a suboptimal early policy? Could such coupling lead to a “representational trap,” and if so, what mechanisms or design choices mitigate this risk?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1.\tThe paper introduces a novel objective for representation learning by reframing the intractable problem of minimizing Inherent Bellman Error (IBE) into a more tractable proxy based on the spectral properties of the Bellman operator.\n\n2.\tThe framework provides a tight, natural coupling between representation and exploration. The feature covariance matrix learned by SBM is directly used to guide Thompson Sampling, creating a coherent feedback loop where better representations inform more structured exploration.", "weaknesses": "Major：\n1.\tThe theoretical motivation for the SBM objective, particularly the spectral decomposition outlined in Theorem 1, is quite elegant. This derivation hinges on the ideal assumption of zero Inherent Bellman Error (IBE), where the function space is perfectly closed under the Bellman operator. I am curious about how the proposed framework is expected to behave when this assumption is inevitably relaxed in practice, as is the case when using complex neural network approximators. Could the authors provide some additional intuition or analysis on the robustness of the SBM objective when the IBE is small but non-zero? For example, how does the connection between the feature covariance and the Bellman operator change in this scenario? Does the SBM loss still serve as a more effective proxy for minimizing IBE than the standard MSE objective? Adding a brief discussion on this point would be very helpful in bridging the compelling theory with the practical algorithm.\n\n2.\tThe paper's alternating optimization between policy and representation is interesting. However, I have a conceptual question about the representation learning target. Since the SBM objective is tied to the Bellman operator defined by the current policy, it seems the representation might become overly specialized to a suboptimal policy early in training. Could this create a \"representational trap\" that makes it difficult for the agent to later discover and represent a truly optimal policy? I would appreciate the authors' perspective on this potential issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QlRin6vgXo", "reviewer": "Reviewer_mbKF", "summary": "The work attempts to combine representation learning and exploration for better performance in RL tasks. In particular, they focus on the Atari benchmark. The method attempts to learn zero IBE features and they combines it with Thompson sampling for better exploration. They combine their approach with two algorithm - DQN and R2D2 and show performance improvement, particularly on games which demand hard exploration.", "questions": "1. Can you provide results for DQN + TS and R2D2 + TS without spectral features? If not, any justification as to why there were excluded.\n2. In Table 1, R2D2 → R2D2+SBM shows almost no improvement (in terms of median score for the explore subset), while R2D2+SBM → R2D2+SBM+TS shows +0.16 improvement. Doesn't this suggest TS contributes more than SBM? How do you reconcile this with your claim that SBM representations \"naturally facilitate structured exploration\"?\n3. What is the computational overhead of alternating between Q-learning and SBM representation learning? \n4. On games like Breakout and Frostbite, SBM+TS performs worse than baseline DQN. Can you discuss this? Is this due to harmful exploration noise in exploitation-heavy games? Can you discuss/analyze this? However, as stated, this is a minor requirement.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The work is well motivated, it makes sense to learn representations which directly support exploration/learning better strategies.\n* The theory is well motivated and sound. Specifically, spectral decomposition under zero IBE conditions and the way to use power iteration for SBM loss optimisation are useful results.\n* The performance gains are good when SBM + TS is used with baseline algorithms, this is in-particular substantial in the hard exploration games.\n* The extension to multi-step operators (Retrace for R2D2) is theoretically sound and practical.", "weaknesses": "* Baseline: The paper is missing two critical baselines - DQN + TS or any DQN based method which uses TS (for instance BDQN (Azizzadenesheli et al. 2018)) but does not use spectral features. Similar is the case with R2D2, the work is missing a critical baseline R2D2 + TS without spectral features (which I understand is might be non-trivial to implement but without this, it is very hard to truly guage SBM's effectiveness). This is extremely important to have to be able to gauge the efficacy of SBM features. The paper cites Azizzadenesheli et al. (2018) in their related work section but provides no justification for excluding it as a baseline. The current experimental design confounds two variables (representation learning + exploration strategy), making it difficult to attribute performance gains to either component independently. For instance, from table 1, it seems the major chunk of improvement is coming after adding TS to SBM (its good enough for DQN but with R2D2, R2D2 + eps → R2D2 + SBM + eps the improvement is pretty small (basically none in terms of median scores), whereas R2D2 + SBM + eps → R2D2 + SBM + TS shows +0.23 improvement overall and 0.16 on explore subset). This pattern suggests TS may contribute more to the gains than the spectral representation itself, which contradicts the paper's central claim that SBM representations \"naturally facilitate structured exploration.\"\n* For exploitation-heavy, precision-control games (breakout, frostbite), the performance sees a decline from the baseline algorithm (DQN specifically) which I am guessing is due to the fact that Thompson sampling may add harmful exploration noise. Authors should discuss a bit about this and suggest if there are ways to control this (e.g., annealing the Thompson Sampling variance parameter or adaptive mechanisms to reduce exploration noise when sufficient policy convergence is achieved). Some ablations and experiments showing when SBM helps vs. hurts performance would strengthen the paper. However, this is a minor thing.\n* The Thompson Sampling implementation (Equation 5) directly follows Zanette et al. (2020b) without modification. The only claimed contribution is using SBM-learned features in the covariance, but without DQN+TS baselines, it's unclear if standard representations would work equally well.\n* The paper provides no analysis of computational overhead with baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSZqAAb8we", "reviewer": "Reviewer_BNxx", "summary": "This paper investigates representation learning in the low-inherent Bellman error (low-IBE) setting. \n\nIt introduces a new method and loss function, termed the Spectral Bellman Method (SBM). \n\nThe core insight lies in a fundamental spectral relationship. Under the zero-IBE condition, the Bellman operator’s transformation of the value function distribution is intrinsically tied to the feature covariance structure. \n\nThe framework can also be extended to the multi-step Bellman operator.", "questions": "Is there any theoretical guarantee that the minimizer of the SBM objective is close to the optimal value function? Since the primary goal of reinforcement learning is typically to learn a near-optimal or optimal Q-function, it is crucial to understand whether the proposed loss function is well-aligned with this objective. Clarifying this connection would strengthen the theoretical foundation of the work and help readers assess the practical relevance of SBM.\n\nOn the practical side, could the authors comment on whether the SBM-based Q-learning method, particularly in its connection to Thompson Sampling, is easier or more efficient to implement in practice? Insights on implementation challenges, stability, or computational considerations would be valuable for understanding the real-world applicability of the approach.\n\nIt would also be helpful to discuss the computational complexity of the proposed algorithm, including how it scales with state-action dimensions, representation size, and sample complexity. A clear analysis would allow readers to compare the approach against existing methods and better understand its feasibility for larger or more complex problems.\n\nSince the paper emphasizes representation learning, it would be useful to clarify whether the learned features provide meaningful benefits, both theoretically and empirically. Do the features improve policy learning, generalization, or sample efficiency? \n\nDo the empirical experiments explicitly target the quality of these representations, or are the observed benefits mainly a byproduct of improved  Q-learning performance? Addressing these questions would help highlight the practical and theoretical significance of the representation learning component.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, clearly structured, and easy to follow, making the technical content highly accessible. \n\nThe proposed SBM loss function is novel and represents a meaningful contribution to the literature. \n\nIts motivation is well-justified, particularly since directly optimizing the MSE of IBE can be challenging and prone to inefficiencies.\n\nThe theoretical properties of the SBM loss and its connections to well-established techniques, such as power iteration, alternating optimization methods, and Thompson sampling, are especially compelling. These connections not only provide deeper insight into the behavior of the method but also situate it within a broader algorithmic context, highlighting potential avenues for further exploration.\n\nThe objective function itself appears more symmetric with respect to the parameters  theta and phi, which may contribute to improved stability and interpretability during optimization. Overall, the combination of novelty, theoretical grounding, and intuitive motivation makes the proposed approach both interesting and promising.", "weaknesses": "The paper appears to lack a formal theoretical guarantee for the minimizer of the SBM objective. For instance, it would be helpful to know whether there is any guarantee that the minimizer is close to the optimal value function, or more generally, how the SBM solution relates to the true underlying objective. Providing such a guarantee, or at least an informal discussion of its plausibility, would strengthen the theoretical contribution of the work.\n\nOne of the stated motivations of the paper is that the SBM objective is easier and more tractable to optimize than the traditional MSE objective. However, the paper does not provide clear evidence or empirical support for this claim. It would be valuable if the authors could elaborate on this point, for example by explaining why SBM optimization is expected to be more stable, faster, or less sensitive to hyperparameters, and by providing either empirical comparisons or theoretical intuition that substantiate the claim. This would help readers better understand the practical advantages of adopting the SBM formulation over conventional approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)