Stackelberg Coupling of Online Representation  Learning and Reinforcement Learning
Authors: Fernando Martinez
Tao Li
Yingdong Lu
Juntao Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Representation Learning
Q-learning
Stackelberg Equilibrium
Two-timescale
Primary area: reinforcement learning

Abstract: Deep Q-learning jointly learns representations and values within monolithic networks, promising beneficial co-adaptation between features and value estimates. Although this architecture has attained substantial success, the coupling between representation and value learning creates instability as representations must constantly adapt to non-stationary value targets, while value estimates depend on these shifting representations. This is compounded by high variance in bootstrapped targets, which causes bias in value estimation in off-policy methods. We introduce Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework for value-based RL that views representation and Q-learning as two strategic agents in a hierarchical game. SCORER models the Q-function as the leader, which commits to its strategy by updating less frequently, while the perception network (encoder) acts as the follower, adapting more frequently to learn representations that minimize Bellman error variance given the leader's committed strategy. Through this division of labor, the Q-function minimizes MSBE while perception minimizes its variance, thereby reducing  bias accordingly, with asymmetric updates allowing stable co-adaptation, unlike simultaneous parameter updates in monolithic solutions. Our proposed SCORER framework leads to a bi-level optimization problem whose solution is approximated by a two-timescale algorithm that creates an asymmetric learning dynamic between the two players. Extensive experiments on DQN and its variants demonstrate that gains stem from algorithmic insight rather than model complexity.

Published as a conference paper at ICLR 2026
STACKELBERG COUPLING OF ONLINE REPRESENTA- TION LEARNING AND REINFORCEMENT LEARNING
Fernando Martinez
Tao Li"
Fordham University New York, NY, USA
City University of Hong Kong
Hong Kong
fmartinezloped fordham.edu
1i.taoß e tyu. edu.hk
Yingdong Lu
Juntao Chen
IBM Research
Fordham University
Yorktown Heights, NY, USA yingdongeus. ibm. com
New York, NY. USA jchen504e fordham. edu
ABSTRACT
Deep Q-learning jointly learns representations and values within monolithic net- works, promising beneficial co-adaptation between features and value estimates. Although this architecture has attained substantial success, the coupling between representation and value learning creates instability representations must con- stantly adapt to -ta-stationary value targets. while value estimates depend on these shifting representations. This is compounded by high variance in bootstrapped targets, which causes bias in value estimation in off-policy methods. We introduce Stackelberg Coupled Representation and Reinforcement Learning (SCORER), framework for value-based RL that views representation and Q-lcarning as two strategic agents in hierarchical game. SCORER models the Q-function the leader, which commits to its strategy by updating less frequently, while the per- ception network (encoder) acts as the follower, adapting more frequently to learn representations that minimize Bellman error variance given the leader's commit- ted strategy. Through this division of labor, the Q-function minimizes MSBE while perception minimizes its variance, thereby reducing bias accordingly, with asymmetric updates allowing stable co-adaptation. unlike simultancous parameter updates in monolithic solutions. Our proposed SCORER framework leads to a bi-level optimization problem whose solution is approximated by two-timescale algorithm that creates an asymmetric learning dynamic between the two players. Extensive experiments on DQN and variants demonstrate that gains stem from algorithmic insight rather than model complexity.
INTRODUCTION
Deep Reinforcement Learning (RL) has achieved outstanding success, exemplified by the super- human performance of Deep Q-Networks (DQN) in complex environments (Mnih et al.) 2013). The dominant approach for value-based methods involves training both the representation and the value function within single neural network, encouraging beneficial co-adaptation between the features learned and value estimates, leading to significant breakthroughs (Hessel et al., 2018 Kapturowski et al.] 2018); however, this monolithic design is subject to an instability known as the deadly triad (Sutton& & Barto 2018: Van Hasselt et al. 2018). This instability results from the challenging interaction between function approximation using neural networks, bootstrapping, and off-polic learning from data stored in an experience replay buffer, where errors in the current value function bootstrapped into its learning targets, leading to risk of unbounded divergence, first formally demonstrated in Baird (1995). This risk remains a critical practical concern for modern agents; recent work has empirically linked this core instability phenomena like representation collapse and catastrophic learning failures in deep Q-learning (Lyle et al. 2022).
issue
prominent line of work seeks mitigate this instability by directly enhancing the quality of the learned representations. This is often accomplished by extending the main RL objective with auxiliary losses, borrowing from advances in self-supervised and representation learning (Jaderberg al.)
2017 Schwarzer et al. 2021 Laskin et al. 2020b). While this strategy can improve performance,
Corresponning author.

---

Published as a conference paper at ICLR 2026
Perception Ug)
Stackulbung Coupling
Contral Cisy
Updatte Eanker
Astorn U2.
Idean Spaard
Bellnan briw
Perception Teatures
Follower
- (200
Leader
Pollmwrr: Eerception (8)
Twe-Timende Suckeber
State
Reward 12
SCORER Agent
Coglld Comning
5,
Action
Londor
to Fett
4
Ceprementaiion
Leatted
Tha1
Environment
Sc+1
(a) Aggnt-Fnvvrrmment Interaction Loop
SCORER Two- Two-Timescale Coupling
Figure SCORER framework. (Leftt) Overall agent-environment interaction loop. Internally, the agent comprises perception network (Follawer, fa) and control network (Leader, Qo) that interact via Stackelberg game dynamies (Uv, Ur. representing their utility functions). The perception network produces features (s) used by the control network. (Right) Details the Stackelberg interaction within the agent.
utilizing a shared network to fulfill multiple objectives can introduce the challenge of conflicting gradients (Yu et al., 2020), wherein the gradient from an auxiliary loss opposes that of the primary value-learning objective. This interference presents fundamental trade-off, as the effort to stabilize representations introduces new obstacle to learning the value function itself.
This trade-off suggests that simply augmenting the monolithic optimization is an insufficient solution, motivating fundamental restructuring of the optimization problem itself. We therefore propose Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework that recasts the interaction between perception and control as hierarchical Stackelberg game (Lambertini 2018). In this game, the Q-function acts as slow-updating leader, providing the stable learning target necessary for fast-adapting perception network (the follower) learn robust representations by minimizing Bellman error variance, This objective incentivizes the creation of representations that can handle the noisy targets encountered during exploration, preventing the representation collapse that can afflict monolithic agents (Lyle et al. 2022). This division of labor resolves the underlying trade-off, using temporal separation create the stability required for this co-optimization.
We create SCORER's s game- same-theooeticc dynamics through a practical and computationally efficient algorithm. The Stackelberg equilibrium approximated using two-timescale gradient descent, where the follower can track a best response to the leader's slowly evolving strategy. Implementing SCORER simple, as the hierarchical coupling is achieved by assigning the two-players distinct decaying learning rates that satisfy the conditions for two two-timescale convergence, with no alterations to the underlying network architectures. This two-timescale dynamic provides a principled mechanism for achieving stable, coordinated adaptation, approach supported by established theory in stochastic approximation (Borkar 1997 Fiez et al. 2020
Figure [ provides high-level overview of the SCORER framework. Our code is available at https: / //g.t////ttttt com/ fernando- /SCCRRE In summary, our contributions are three- fold: 1) We introduce the SCORER framework, novel game-theoretic formulation that recasts the interaction between perception and control to address the core instabilities of off-policy value-based learning directly. We develop a practical and efficient two-timescale algorithm requiring only update frequency modifications, stabilizing the co-adaptation of representations and value functions. 3) We provide extensive empirical validation, demonstrating that SCORER consistently improves the sample efficiency and final performance of various off-policy Q-learning agents across multiple benchmarks.
RELATED WORK
Representation learning has been a pivotal topic in RL, even before the advent of deep neural networks. Early stage efforts concentrated value function approximation (Thrun & Schwartz/ 1993 Tsitsiklis & Roy 1997) and associated basis function selections (Li & Zhu 2019 Geramifard et al. 2013). Entering the age of deep learning. RL harnesses the representation power of neural networks and becomes capable of solving high sigh-dimensional complex tasks with multi-modal inputs, such texts eL al. 2016), images (Mnih al. 2015), and multi-modal sensor data (Liu et al.
2

---

Published as a conference paper at ICLR 2026
2017: Yin et al. 2024 Li et al. 2025). The confluence of deep representation learning and RL leads to the vibrant research field of deep RL.
Depending the purpose, representation learning methods in deep RL can be classified into two major categories: those aimed at facilitating training and those designed to enhance testing perfor- mance. In the training phase, learning appropriate representations helps with 1) dimension reduction:
extracting low-dimensional features from high-dimensional inputs, making the RL tasks tractable, a
typical example of which is vision-based control tasks (Lesort et al. 2018: Li et al.) 2023); 2) sample efficiency: capturing underlying environment dynamics and value structure (Subramanian et al. 2022]
Lee al.] 2020; Dabney al. [202T] Vincent et al., 2025), which reduce the number of training
episodes and foster faster convergence; and 3) oriented exploration: directing future exploration
strategies based on past experiences and intrinsic motivation (Kulkarni et al.. 2016), particularly in reward-free and rpwrse settings (Pathak et al.]
et al.] 2019). Some other
considerations include training stability (Greydanus et al. 2018) and explainability (Dazeley et al.
2023
Li al. 2023). Regarding testing improvement, previous work focuses on the transferability of representations for improved generalization to downstream or similar tasks (Agarwal et al. 2021a
Träuble et al. 2021).
Our proposed Stackelberg framework aims to improve sample efficiency and stability in the training stage. Weighing the three mainstream representation learning supperisedd unsupervised, and self-supervised learning- work opts for the unsupervised approach based on the Mean Squared Bellman Error (MSBE). Supervised learning requires additional labeling inform the agent of the quality of learned representations (Wang et al. 2024), which is often observed in physical control with partial observability, and the representations need to encode structural information of the environment, such as depth maps from RGB images (Mirowski et al. 2017) and physics principles underlying the wireless sensing inputs (Yin al.) [
Li al. 2025 Distinct from additive value decompositions (Anand & Precup 2023) or descriptive architectural separation (Garein et al. (025). this work explores the benefit of internal game-theoreii coupling, creating a hierarchical dependency between representation and control without external learning signals (other than task rewards) to interfere with the strategic interactions between the two lcarning processes.
A similar argument also explains why we do not consider the incorporation of self-supervised learning, such as contrastive learning (Liu et al., 202T; Stooke et al., 202T; [anino et al. 2022), temporal dynamics and state prediction (Subramanian et al.] 2022; Schwarzer al. 10211 and observation reconstruction (Lange & Riedmiller] 2010: Lesort et al. 2018), and most unsupervised learning, including mutual information (Anand et al. 2019),, entropy maximization (Hazan et al.) 2019), data augmentation (Yarats et al., 2022] [ Laskin et al.] [2020a), and bisimulation (Zhang et al.) | et al. 2021aa. Our proposed unsupervised representation learning solely relies on MSBE and its sample variance without auxiliary tasks or additional learning signals.
3 PRELIMINARY
Reinforcement Learning and Q-Learning. We consider an agent interacting with an environment formulated as a Markov Decision Process (MDP), defined by the tuple AS, R, 7). Here, S is the state space, .A is the action space, P(s'ls, (s, a) is the state transition probability, R(s, a) the reward. and 7 E [0, 1) is the discount factor. At each step, the environment transitions from state to s' and provides a reward r; if 8' is terminal state, a done signal d = 1 is captured, otherwise d = 0. The agent's ss goal is to learn policy π (a|s) that maps states to actions (or distributions over actions) to maximize the expected sum of discounted future rewards from given state st: E, [E (9
where Tr+s is the reward received i steps after time
In basue- Reinforcement Learning, the optimal action-value function Q* a) represents the expected return from taking action a in state s and following the optimal policy thereafter. It satisfies the Bellman optimality equation: Q"(ss, (s),a) = E,, [R(s, a) + y max.' (s'(', a'). Deep Q-Networks (DQN) approximate Q' using neural networks trained by minimizing the Mean Squared Bellman
Error (MSBE): L())= E.......s' AD) (Y (=(s) =))²
To stabilize training, the target value Y is calculated using a separate, periodically updated target network (Q that prevents the learning target from fluctuating at every step. For a given transition, the target is defined as Y r 711 -d) max, Quaane (s', a').
3

---

Published as a conference paper at ICLR 2026
Stackelberg Game. A Stackelberg game is hierarchical game involving two types of players: leader and one or more followers (Osborne 2004]. The leader moves first, choosing an action ao from its action set Ar. The follower, equipped with the full information on the leader's 's action ag. chooses its own action ap from its action set Ar to maximize its own utility ua(ar, ap): given az. The basic equilibrium relationship for the Stackelberg game is reflected by the following bi-level optimization problem: maX.7 Un (ar)), s.t. (ar) E arg maXar UF(aL, ap), with (az, aF) being the utility of the leader.
The defining characteristic is the leader's ability to anticipate and influence the follower's decision by committing strategy first, steering the game toward favorable equilibrium. This hierarchical dynamic inspires our proposed SCORER framework.
4 STACKELBERG COUPLED REPRESENTATION AND REINFORCEMENT LEARNING
In this section, we formalize the thee-theoretic principles outlined previously. We model the value- based interaction between perception and control as a hierarchical Stackelberg game to resolve the trade-off between representation stability and value function learning. This framework provides clear division of labor, implemented through two distinct network components: perception network that acts as the game's follower, and control network that acts as the leader.
4.1 THE STACKELBERG GAME FORMULATION
The Leader: Control Network. The Control network Qa assumes the role of the Stackelberg leader as it defines the primary optimization goal (value estimation). Its objective to learn an optimal action-value function Q0(2, a), mapping state representations and actions to expected cumulative discounted future rewards. As the leader, the control network learns on a slower timescale, providing stable target for the follower. The leader's objective function, Lleaders is the MSBE. Let D be the agent's 's data source. This source can be a replay buffer, as in traditional replay buffer-based methods, or collection of online trajectories, as in PQN (Gallici et al. 2023). The leader aims to solve:
min Lieader (caberrQQe,) (0)) BLB (o( a))²
where B is a batch of transitions sampled from the data source D, and Y is the corresponding Bellman target value. Here, 6= (0) represents the parameters the follower would ideally converge given the leader's choice of f. The hierarchical structure is established by having the control network commit its parameters on a slower timescale, while the perception network responds on a faster timescale.
The Follower: Perception Network. The Perception network fo acts as the Stackelberg follower. Its function is to learn an encoder that maps raw observations s to latent representation = = (s). Given the control network's 's committed strategy (with 0 treated fixed via foop-gradient during the follower's update), the follower learns on a faster timescale, allowing it to compute an effective best response to the leader's slowly changing strategy. The follower seeks parameters & that minimize its own loss function, Ciallower-
SCORER is agnostic to the specific choice of follower objective, allowing for different formulations depending on the desired properties of the learned representations. We investigate multiple objectives for Liollowe. including directly minimizing the MSBE and minimizing the variance of Bellman errors. Through extensive empirical evaluation (detailed in Section 5 and Appendix K). we find that minimizing the variance of Bellman errors yields superior performance, which echoes the observation that MSBE is biased due to the high variance (Baird 1995 Wu et al.] [202I). For a batch
B, let 6;(6, 0) Y, Qo Qf(fo(s;) az) be the Bellman error for transition j. where Y, is the target value. The follower's s objective becomes:
/ (0) E arg min Leslower (Jo, Qo) A (6,
0,) 2 is the sample variance of the Bellman errors EEE
1 where Varzenl6,] |Å over the batch.
EjEB
4

---

Published as a conference paper at ICLR 2026
The motivation for this variance-minin minimization objective is to counteract the pathologies of the deadly triad directly. By focusing on the consistency of the Bellman errors across a batch, the follower learns representations that more consistent across the batch, making them robust to the noisy targets inherent in TD learning. This promotes stability in the learning process, a trait for mitigating the risk of divergence that first formally demonstrated in foundational work by |Baird (1995). This objective transforms the follower into an active stabilizing agent, as it is incentivized to find representations that make the leader's value predictions more uniform across the data distribution.
4.2 APPROXIMATING THE STACKELBERG EQUILIBRIUM VIA DESCENT
GRADIENT
The Stackelberg game described in Section 4.1 translates to bi-level optimization problem given by equation 3 The control network aims to optimize its objective Lleader by choosing its parameters 0, anticipating the follower's optimal response o (0))
min Lleader (o)) subject
o" o'(()) arg min Loolooerr (for Qo).
&
Solving such bi-level problems directly high-dimensional, non-convex settings typical of deep reinforcement learning is challenging, with the primary difficulty arising from computing the gradient of the leader's 's objective with respect to its parameters 6. The leader's objective (Qe, (a)( depends on both directly (through Qo) and indirectly (through the follower's response @ (0)). Applying the chain rule yields
@Licnder VoQo + eLicder V Vela-(0))
V Veeiedeer
oQ.
05
where the first term is the direct gradient through Qp. and the second term Vole captures the indirect effect through the follower's response.
Computing the indirect gradient term is the central challenge of bi-level optimization. It requires the implicit derivative dø (0) which contains second-order information about the follower's optimization
de
landscape. While this term is computationally prohibitive to calculate directly in deep learning contexts (Chen et al. (023), body of recent work has developed first-order methods approximate (Li et al. [024]] Liu et al. 2022 Li et al.) 2022 Hong et al. 2023). However, as our focus is on the game-theoretic coupling, adopt more direct solution: approximate the leader's gradient using the first-order term and converge through a principled timescale separation between the players.
This separation ensures that the faster-learning follower can effectively track the best response to the slower-learning leader. In our framework, this asymmetry is achieved using time-dependent learning rate schedules, Og.kk and Ogg.k, & that satisfy the key conditions of two-timescale stochastic approximation (TTSA) theory (Hong et al., 2023). Specifically, both learning rates must decay over training steps k. while their ratio must also approach zero, i.c., limg->oc 09.t/a0, = 0. This guarantees that the leader learns on sufficiently slower timescale than the follower, allowing the system to converge to a first-order stationary point of the game, as detailed in Algorithm T The practical update rules for the follower and leader are thus given by:
@k+1 f- Øx ag..k V Lfolllower Brollowers Y, (k.), Dx+1 02 a9 Clender (0g; Breaders Y, db+1))
(5)
(6)
where (c.gg, 02) denotes stop-gradien operation that treats the variable as constant with respect to the gradient calculation, guaranteeing that no gradients flow from one player's update to the other's parameters. The follower first updates to dx++1 based the fixed leader parameters 0g. The leader then immediately updates to 0x41 using the follower's new state Φ 1 as a fixed input.
Our approach of having the leader optimize based the follower's 's recent state, rather than its fully converged response, is a common and practical technique for approximating bi-level opti- mization (Fiez et al. 2020) Petrulionyté al. 2024). Under standard smoothness assumptions
5

---

Published as a conference paper at ICLR 2026
for two-timest wwo-immescale stochastic approximation, including Lipschitz continuous gradients and weaker convexity condition known as the Restricted Secant Inequality (RSI) (Karimi et al., 2016), our method converges first-order stationary point of the game Li et al. (2024). This convergence is general and holds for both the MSBE and Bellman Error Variance objectives, as both measure temporal difference consistency.
We provide a detailed convergence analysis in Appendix M which explicitly characterizes the convergence rates in terms of the size of neural networks, well the learning rates schedules.
4.3 TWO-TIMESCALE STACKELBERG COUPLED LEARNING (SCORER)
The implementation details of SCORER for coplay-based agents are detailed in Algorithm Both the pereeption network (follower) and the control network (leader) are updated at the same interval, Tupdate. where the timescale separation that approximates the Stackelberg dynamic is achieved by setting the two players distinct, -imedeependent learning rate schedules, Og and ag- In line with two-timescale convergence theory, the schedules decay over the course of training and are chosen such that the follower's learning rate, &g. remains significantly larger than the leader's, s, cra,,, making sure that their ratio diminishes over time. We model the two-player game by stopping the gradient flow during backpropagation within a single update step. so the follower first updates its parameters Φx+1 based on the fixed leader parameters 0x. and subsequently, the leader updates its parameters to 0k+1 using the follower's ' newly updated state @ completing one step of the hierarchical game.
Algorithm 1 SCORER (Stackelberg Coupled Representation Learning) for Deep Q-Larnnin Variants
Require: Perception fe. Control Qo: Parameters Φ 0; Target networks Pragge,, Ourger
Require: Learning rates Org, Oe: Replay buffer D; Minibatch size Noasi Exploration strategy
Require: Total timesteps Update interval and start step Tipdate Turge, Tsani Polyak
I: Initialize online params d) 0; target params Ourget <- Φ, Oungei t- 0; replay
2: Reset env. and get initial state 8; fem <- 0
3: while fem T do
Dain environment interaction loop
fa(s) 4: Select action a ((Q)(2)))
Extract representation:
6: Execute a, observe r, s Store (s, a, r, s', in
7:
+ s'; feav +- tew +1
8:
if fem Timn then
if tav (mod Topaae) 0 then
10:
Sample minibatch Bratower from D
b Perception (Follower) Update
11:
Compute targets for Bistower using J Qawas (and Qo for DDQN-like variants)
12:
Best response to fixed Leader > Control (Leader) Update Compute targets Y for Bleader using fawrge: Qranes (and Qo for DDQN-like variants)
< 6 ae V Castoweer (co Brallower, Y,O,
13:
Sample minibatch Breatr from D
14:
15:
0 - Vo VoLreser (0; Breaker Y, Y,
16:
end
fan (mod Turya) = 0 then
Target Network Updates
18:
Charer + TΦ + (1 T Pragen Ourget +- T0 + (1 T) 7)0gge
19:
end if
20:
end if
21:
end while
Note: For recurrent variants (e.g., R2D2), fo maintains hidden states h across timesteps, resetting when
episodes terminate. The replay buffer stores sequences rather than individual transitions. For agents like PQN that do higher learning for the follower without sampling from replay buffer.
Note:
replay buffer, the asymmetric dynamic achieved using
SCORER's Generality While Algorithm T details the implementation for replay-based agents, the SCORER principle of two-timescale, Section 5
dynamic is more general. As shown in our it can be seamlessly adapted to modern agents like PQN that do not use replay buffer.
5 EXPERIMENTS AND RESULTS
In this section, we empirically evaluate the performance SCORER when integrated with multiple established value-hased RI. algorithms: DQN and its variants (Double, Dueling) (Mnih al. 2015

---

Published as a conference paper at ICLR 2026
Van Hasselt al. 2016; Wang al. 2016), and R2D2 (Kapturowski al. 2018 for partially observable environments. We also compare it against the Parallelized Q-Network (PQN) Gaalici et al. 2025). recent, seggh-erffrmance off- off-policy value-based method that forgoes the of a replay buffer. Our evaluation covers diverse domains, including the MinAtar suite to measure sample efficiency, the Atari-5 benchmark for high-dimensional visual control tasks, and MinGrid environments that test partially observable scenarios. Additionally, we investigate SCORER's 's stability properties on challenging counterexamples known to cause divergence in standard TD learning, as well as its robustness to environmental stochasticity. Our primary goal is to assess whether the proposed Stackelberg coupling (Algorithm[]] improves learning compared standard end-to-en training of these baseline methods without increasing computational overhead or architectural size.
All experiments use an efficient JAX -based framework inspired by PureJaxRI. and CleanRI. algo- rithms (Lu et al., 2022 Huang et al., (022). To provide fair comparison, the underlying network architectures and hyperparameter budgets are kept identical between SCORER variants and their respective baselines. Consequently, any observed performance gains are directly attributable to the dynamics introduced by SCORER. A detailed description experimental setups and hyperparameters is in Appendix C
5.1
PERFORMANCE ON GYMNAX ENVIRONMENTS
For the first evaluation, we use SCORER across the MinAtar suite Young & Tian 2019) through Gymnax (Lange 2022 Learning curves comparing the SCORER-enhann variants against their respective baselines are displayed in Figure 2 over Asterix, Breakout, Freeway, and Spacelnvaders. Table complements these curves by summarizing final performance.
Breakout
Spacelnvaders
Asterix
Freeway
DON
80
100
25
20
DOON
96
100
40
16
DualingOON
86
100
26
25
20
100
100
00
DuelingDO@N
86
100
25
20
40
80
100
60
60
100
20
62
80
100
20
40
80
100
100
50
PON
25
20
40
50
30
100
0
20
40
60
30
100
20
40
80
80
100
20
40
80
80
100
Training Steps (Millions)
Daselino
SCORER
Figure 2: Learning curves on MinAtar comparing SCORER variants against baselines. SCORER generally demonstrate improved sample efficiency and performance across several algorithm- environment combinations.
The results demonstrate that SCORER provides a consistent and significant performance benefit across all tested base algorithms. For the classic DQN family, the SCORER variants show dramatic gains in sample efficiency and final performance. score of standard DQN agent. More importantly, SCORER allows these replay-baset methods to become competitive with, and in some cases (e.g., SpaceInvaders) superior to the state-of-the-- methods like PQN baseline.
Breakout, for example, SCORER triples the final
The final rows of Table show that SCORER's ' benefits are not limited to replay- based agents. When applied to PQN, our framework provides further statistically significant performance gains. This
7

---

Published as a conference paper at ICLR 2026
Table 1: Final IQM return (#: 95% Bootstrap CT over 30 seeds) on MinAtar, averaged over the last 10% of training. Highlighted values indicate the best-perfrrming variant within each algorithm family per environment. Highlighted performances are not statistically worse than the best (bnotstrapp difference test, p 2 0.05).
Variant
Asterix
Breakout
Freeway
Spacelavaders
Speed Comparison
Raseline
54.95 4: 0.92
19.16 +: 1.50
62.70 +: 0.17
127.78 +: 0.55
1.00x
DQN
SCORER
54.78 + 0.42
65.69 + 2.52
63.03 = 0.08
[48.71 = 2.02
0.99x
Baseline SCORER
52.55 + 2:27
19:21 + 3.81
62.23 + 0.13
118.41 + 2.08
1.00x
DQN
52.88 1: 0.41
63.93 1: 1.86
62.75 1: 0.09
147.82 1: 4.44
0.99
Baseline SCORER
50.77 4: 1.19
36.47 1: 5.04
62.22 : 0.07
116.72
:: 3.36
1.00x
DDON
52.59 - 0:27
64.44 +: 2.48
68.68 + 0.08
146.67
+ 5:14
1.00x
DuclingDON
Baxeline SCORER
39.22 1: 9:24
27.81 1: 3.99
61.89 1: 0.09
121.21 1: 1.70
1.00x
52.28
60.04
62.27
3.42
1.01x
DuelingDDQN
Bascline SCORER
46.53 + 3.05
27.68 + 5.55
61.58 + 0.14
122.26 + 1.44
1.00x
52.44 4: 0.24
59.92 4: 3,85
62.22 + 0.08
142.71
4: 2.82
1.00s
Baseline
50.41
1.34
69,97 1.
61.77 1: 0.34
137.73
1.04
1.00x
PQN
SCORER
50.80 + 0.62
71.16 + 0.84
61.05 + 0.47
141.85 + 0.76
1.00x
demonstrates that SCORER not only restricted to replay buffer-based agents but represents an advancement in the co-adaptation of representation and control in value-based RL. Additional results on classic control benchmarks in Appendix H further corroborate this trend.
5.2 PERFORMANCE ON ATART
To efficiently evaluate SCORER's scalability to high-dimensions visual enviromments, we evaluate on Atari-5 (Bellemare et al., 2013 Aitchison et al.) (023). statistically validated subset of the Arcade Learning Environment (ALE) that provides a reliable estimate of performance across the full Atari-57 suite while requiring significantly fewer computational resources. We integrate SCORER with PQN (Gallici et al. 2025), being an SOTA algorithm that serves as a strong baseline.
Following standard protocols Machado et al. 2018), we train on each game for 200M frames using 5 independent random seeds per game. We compare vanilla PQN against SCORER PQN, using identical hyperparameters and network architectures for both variants to guarantee that any performance differences are attributable exclusively to SCORER's 's game-theoretee coupling mechanism. Results in Figure 3 indicate that in BattleZone, DoubleDunk, and NameTThisGame ThisGGme, SCORER PQN statistically matches baseline performance, while in Phoenix and Qbert outperforms the backbone model without adding computational overhead. This confirms that our game-theoretie coupling scales effectively to high-dimensional visual control without requiring additional architectural complexity.
Averngg Epicoda Ratur
=30
BattleZone
DoubleDunk
K10" NammmmieGame
X10
Phoenix
K30
Obert
16
122
-10
0.3
0.6
5.0
Baseline SCORER
0.0
300
300
Frames (Millions)
Frames (Millions)
Frames (Millons)
Frames (Millions)
(Millions)
Figure 3: Atari-5 Average Episodic Return
5.3 PERFORMANCE ON MINIGRID ENVIRONMENTS
We test SCORER on Min-
Grid "chevalier-ooiseert et al.
Enviroment
Max. Perf
SCORER
R2D2
TTT
SR
TTT
SR
2023). a suite of gridworld envi- ronments testing navigation, ob- ject interaction, and dynamic obstacle avoidance, accessed through Navix (Pignatelli et al.; 2024). As these environments are partially observable. we com- pare R2D2 and its SCORER R2D2 counterpart. Table 4 shows time-to-thresd -trreshholl analy- sis measuring timesteps to reach
DoarKey 6x6 SimpleCmssing S9N2
100%
0.4 : 0.0
100%
1.00
0.3 14:0000 0.0
100%
0.4 - 0.0
100%
DistShif22
1.00)
0.1 11:00
100%
0.1 1: 0.0
100%
LavaGap $6
1.00
0.1 0.0
100%:
0.2 1. 0.0
100%:
GoToDoor 8x8
1.00
0.2 03:000 0.0
100%
0.3 - 0.0
100%
Empty Random 8x8
0.0
100%
0.3 0.1
100%
Dynamic Obstacles 6x6
1.00
0.5 + 0.0
83%
0.7 1 0.1
73%
Four
97%
Figure 4: Time-to -otheeshoorohod analysis showing mean timesteps (in mil- lions) = 95% confidence interval to reach 99% maximum performance over seeds. SR (%) is the success rate runs reaching the threshold.
8

---

Published as a conference paper at ICLR 2026
99% maximum performance within one million timesteps. SCORER consistently outperforms R2D2, achieving 25-50% faster convergence across all environments. The benefit of the Stack- elberg coupling is particularly pronounced in tasks requiring adaptation and exploration. In Dynamic Obstacles, SCORER improves the success rate by 10 points over the baseline, and in the Four Rooms environment, SCORER achieves 97% success rate while the baseline R2D2 completely fails to solve the task. Full learning curves in Appendix [
5.4 STABILITY ANALYSIS: BAIRD'S COUNTEREXAMPLE AND STOCHASTIC ENVIRONMENTS
To better understand SCORER's stability properties, we test it against Baird's counterexample (Baird 1995), a challenging domain known to induce divergence under off-policy learning with function approximation, and against Stochastic Deep Sea (Osband et al. 2020), which combines complex exploration with intrinsic environment noise.
Baird's Counterexample: In this setting, we instantiate SCORER with linear networks that match the representational capacity of the standard linear TD baseline, and retain the original off aff-policy behavior policy. Figure[S] (Left) reports the loss on a logarithmic scale. As expected, the baseline Linear TD method diverges. Conversely, SCORER's variants (using either the MSBE BE Variance follower objective) stay stable and converge, remarking the effect of the Stackelberg structure. Here, the perception learns representations that are explicitly shaped by the leader's learning dynamics, thus disrupting the destructive feedback loop that causes divergence in the monolithic agent.
Deep Sea: Figures 5 (Center & Right) evaluate SCORER on both deterministic and stochastic Deep Sea (depth 10) using Bootstrapped DQN (Osband et al.) 2016) 6) For each configuration. we run 30 seeds and report the fraction that reach solved state (Osband et al., 2020). In the deterministic case, both SCORER variants achieve near-complete solved rates before the baseline, indicating faster and more reliable learning. This advantage carries over to the stochastic version as SCORER is able to reach cea--complete solved rates, while the baseline plateaus around 20%
Baird's Counterasample
Deterministic Deep Sea
Stochastic Deep Sea
Liner TO
Sea, SCURER SeVo
Busi 520RE0 Belo
Sconcs Unear
Los thg iokao
20
5000
1O000
5000
$2000
tpoades
Figure 5: SCORER Stability and Stochasticity Robustness. (Left) Baird's Counterexample: The standard SCORER hierarchy outperforms the inverted role configuration. (Center & Right) Deter- ministic & Stochastic Deep Sea: SCORER variants solve the task while baseline solves it -20%.
5.5 DISSECTING THE SCORER FRAMEWORK
To validate the core design of SCORER, we perform a series of ablation studies investigating on the MinAtar suite. We investigate our three central design choices: the follower's 's objective, the assignment of Stackelberg roles, and the hierarchical coupling dynamic. The main results are presented here as aggregated Interquartile Mean (IQM) scores, while detailed per-environment learning curves for these studies can be found in Appendix K
We test our role assignment by inverting the hierarchy, making perception the leader and control the follower. Figure 6 (left) shows that this inversion leads to a collapse in performance, falling well below the monolithic baseline. This result strongly supports our claim that the control objective must lead the representation learning process.
Stackelberg roles:
Follower's objective: ( shows that using a Bellman Error Variance objective for the follower substantially outperforms variant where the follower minimizes MSBE, This confirms that

---

Published as a conference paper at ICLR 2026
Perception Objective
Role
Coupling Study
10
NumatzudiOn 104 Rotum
NumalzzdiON IOM Rutum
0.4
0/
Nomaleedd
Baselne
Baneline SCORER
SCORER
SCORER JBE Var anoo)
02
04-
Synch onnus Coupling
SCORER MMSBE
Loader: Percoption
Monoithic (Highe Enooder
6o Traning Shops (Mlllions)
600
100
Go Traiiing Stops (illionss
Training Stepo (Mi)
Figure 6: Ablation studies on SCORER's s core components on MinAtar. (Left) Stackelberg Roles: The standard SCORER hierarchy outperforms the inverted role configuration. (Center) Follower's Objective: Bellman Error (BE) Variance superior to MSBE. (Right) Coupling Dynamie: SCORER's hierarchical coupling is critical for performance, outperforming basclines.
an explicit stabilization objective is critical for SCORER's performance, providing a clear advantage over redundant performance objective.
Hierarchical vs. Synchronous Coupling: from its full hierarchical dynamic or merely from the architectural separation of networks. For this, we design baseline called Synchronous Coupling, where distinct perception (f&) and control (Qo) networks are updated at the same frequency, with both minimizing MSBE. Stop-gradients operations ensure the updates are independent (V LMSBE (Sa, Qo) and VoL Qo)). removing any strategic interaction. This setup cleanly isolates the effect of using two networks from the timescale- -asd hierarchy of the Stackelberg game. We also test simpler baseline, standard monolithic DQN with per-layer learning rates matching SCORER's timescale ratio, where the encoder layers update faster than Q-head layers without architectural changes or stop- noop-gradients.
Finally, we study whether SCORER's benefits stem
The results present a clear picture of SCORER's effect (Figure right). We observe that the Synchronous Coupling baseline provides marginal benefit, though not significant, and the monolithic per-layer LR baseline does indeed outperform standard DQN, validating that faster updates for the representation are beneficial. Yet, SCORER outperforms both alternatives, and the performance gap is most notable in Breakout and SpaceInvaders (see Appendix K for per-environment results). Overall, the ablations provide evidence that all three design choices (the hicrarchical structure, the specific form of the stabilization objective, and the correct assignment of the roles) are important and complementary ingredients of the SCORER framework.
6 CONCLUSION
This work revisited and addressed the foundational instability that arises from the tight coupling of representation and control in monolithic value deep Q-lcarning agents. We present the Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework that reframes this dynamic as a hierarchical Stackelberg game. We model the control network (Qo) as the leader and the perception network (fa) as the follower. This game-theoretic interaction achieved through a practical two-timescale algorithm where the leader's slower learning rate delivers stable target, enabling the follower learn good representations by minimizing Bellman error variance. SCORER consistently exhibited improvements in sample efficiency and/or stabilized returns of wide range of replay-base based and online Q-learnig agents, validating the efficacy of its design.
Despite our promising results, some limitations provide clear directions for future work. The current work focused on value-based methods in discrete action spaces, so extending the SCORER principle to actor-critic algorithms for continuous control is an immediate next step. Additionally, our theoretical analysis, which provides convergence guarantees for the two-timescale approximation, could be extended to explore the algorithm's sample complexity, specifically to mathematically characterize how the follower's 's stabilization objective reduces the number of samples required to learn a high- quality policy. Finally, investigating alternative follower objectives derived from cooomationnthhoreete principles presents another compelling research direction. We believe SCORER provides a flexible foundation for developing new class of more stable and efficient deep RL systems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Juntao Chen acknowledges support from the Fordham Al Research (FAIR) Grant, the Fordham-IBM Research Award, and the National Science Foundation under Grant ECCS-2138956.
REFERENCES
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. In International Conference on Learning Representations, 2021a. URL
net/ ffrrmmrrmm
id=gda7-sVp0
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 29304-29320, Curran Associates, Inc., 2021b. URL
ngs neurips wwt.seseeriisppppppppeeeeeeeeeee wwtpssercpppppppeeeeeeeeeeeeeeep File/ http://11cccc111115588811 742 -
wttprrffieeeppppr///aeam es/paper/2021/ pdf
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. In International Conference on Muchine Learning, pp. 421-438.
PMLR, 2023.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, 2019. URL /tt
ps www.//apper.. ceteeer_fllsappppe / pdf
(018/Elle/6fo52e11 10017155144455511777773333333353
Nishanth Anand and Doina Precup. Prediction and control in continual reinforcement learning. Advances in Neural Information Processing Systems, 36:63779-63817, 2023.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. Machine
Learning Proceedings, 1995.
Andrea Banino, Adria Puigdomenech Badia, Jacob C Walker, Tim Scholtes, Jovana Mitrovic, and
Charles Blundell. CoBERL: Contrastive BERT for reinforcement learning. International Conference on Learning Representations, 2022. URL
1d=skZ3ChmeqS
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal artificial intelligence research, 47:
253-279,2013
Denis Belomestny, Alexey Naumov, Nikita Puchkin, and Sergey Samsonov. Simultancous ap- proximation of smooth function and its derivatives by deep neural networks with piecewise- polynomial activations. Neural Networks, 161: 001::42-220 252,,, 2023. ISSN 0893-6080. doi: https://ooo.ogggo/ooooo///neeees scciannee/arricaaaa/1/8833333333
URL https: /tt::///wwwww. /ttp://ieeeeeeeetttttttttteeeee. com/
Vivek S Borkar. Stochastic approximation with two time scales. Systems & Control Letters, 29(5):
291-294, 1997.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:://ithut: com/ jammm/jjxx
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and 01540
Wojeiech Zaremba. Openai gym, 2016. URL
George Casella and Roger Berger. Statistical inference. Chapman and Hall/CRC, 2024.

---

Published as a conference paper at ICLR 2026
Ziyi Chen, Bhavya Kailkhura, and Yi Zhou. An accelerated proximal algorithm for regularized nonconvex and nonsmooth bi-level optimization. Machine Learning. 112(5):1433-1444 14633 2023. ISSN 0885-6125. doi: 10.10077:100944-02---06333
Maxime Srevalier-Boisvere Bolun Dai, Mark Towers, Rodrigo Perez- Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances Neural Information Processing Systems, 36:73383-73399 2023.
Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G Bellemare, and David Silver. The malue-improvement path: Towards better representations for reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7160- 7168, 2021.
Richard Dazeley, Peter Vamplew, and Francisco Cruz. Explainable reinforcement learning for broad-X :AI: conceptual framework and survey. Neural Computing and Applications, 03 2023. ISSN 1433-3058. doi: 10.
Utt.httss: //do:::///oii.o org/10.00111110
500521-023-0844
Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3133-3144 PMLR, 13-18 Jul 2020. URL
mlr wtw.::/p..............p.......es (earess/v119/fiezeee/ html
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. In The Thirteenth International Conference on Learning Representations, 2025. URL //tppp:///penreview
net//oorum21d==11eL...E
Samuel Garcin, Trevor McInroe, Pablo Samuel Castro, Christopher G. Lucas, David Abel, Prakash Panangaden, and Stefano V Albrecht. Studying the interplay between the actor and critic repre- sentations in reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL
Alborz Geramifard, Thomas J. Walsh, Stefanie Tellex, Girish Chowdhary, Nicholas Roy, and Jonathan P. How. A tutorial on linear function approximators for dynamic programming and reinforcement learning. Foundations and Trends@ in Machine Learning, 6(4):375 451, 2013. ISSN -935-8237. doi: 10. 10.1561/22000000442
Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming, 2018. URL https://arxiv o ttt//////////2...000 02246
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In Proceedings of the 35th International Conference on Machine Learning, volume 80, ml
pp. 1792-1801, 2018. URL WRRL/htts:////oooooo http:///cccccceenn.....eeeeeerrr
html
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 268 1-991,, 09-15 Jun 2019. URL
ptttp:://proceedingg. m...press
press/
/ttp//9///aaaaa httppsss////haaeeeee 9a httml
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 13(1):147-180, 2023. doi: 22.1137/20m1887444 URL https
7/doi.or (.doi..og/1..1137/2200 387341
12

---

Published as a conference paper at ICLR 2026
Shengyi Huang, Rousslan Fermand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoAGo GM AraAšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference Learning Representations, 2017.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak-tojasiewizz condition. In Joint European conference on machine learning and knowledge discovery in databases, pp. 795-811. Springer, 2016.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Pro- cessing Systems, volume 29, 2016. URL htt. iew. cctp:scccppppppeeeee cc/pppprrcrpppp..reeppeppp paper
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under under-parameterizatiin inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021. /ttphtttss/::seeeviiiii
net/ httpt//formmmidd0000 hXF
Luca Lambertini. Stackelberg Games, pp. 234 254. Cambridge University Press, 2018.
Robert Tjarko Lange. gymnax: A JAX-based reinforcement learning environment library, 2022. URL
http://github. com/ cot.
Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning.
In The 2010 International Joint Conference on Neural Networks (IJCNN,, pp. 1-8, 2010. doi: 10.1 10.1109/ICNN.2200.55556
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp. 5639-5650. PMLR, 2020a.
Michael Laskin, Aravind Srinivas, and Picter Abbeel. CURL: Contrastive unsupervised repre- sentations for reinforcement learning. In Hal Daumé III and Aarti Singh (eds.), Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5639-5650. PMLR, 13-18 Jul 2020b. URL https! //proceedings.no mwa.ggmmm.r..... www.sppess...m /v www.inkkkinnn... html
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijic Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11890- 1911, 2020. URL URLhttts:://proceeddee cerrrseeeee..orr.r.eerroo.o.eeer
h1t1://8ee2200011112299992262263 8a Paper pdf
Timothée Lesort, Natalia Díaz Rodriguzz Fran-Frangis Goudou, and David Filliat. State representa- tion learning for control: An overview. Neural Networks, 108:379-392, 2018, ISSN 0893-6080.
doi: 1...016/..nuunet2011 0.000000..........
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep rein- forcement learning for dialogue generation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v//D16 1127. h/t.:///1AA...t/////aaaaaaoooooa
wtwp:/loyyor///111111111 1127/
Tao Li and Quanyan Zhu. On convergence rate of adaptive multiscale value function approximation for reinforcement learning. 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6, 2019. doi: 10. 2009///pp.2219.811111.
13

---

Published as a conference paper at ICLR 2026
Tao Li, Haozhe Lei, and Quanyan Zhu. Sampling attacks meta reinforcement learning: A minimax formulation and complexity analysis. arXiv preprint arXiv:2208 00081, 2022. doi: 10.48550/ htt..2208.0008811 [Online] Available at attttps://axxi>>>
00081
Tao Li, Haozhe Lei, and Quanyan Zhu. Self-adaptive driving in nonstationary environments through conjectural online lookahead adaptation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7205-7211, 2023. doi: 10.1109/1CRA44899..2222 1016616336....
Tao Li, Henger Li, Yunian Pan, Tianyi Xu, Zizhan Zheng. and Quanyan Zhu. Meta stackelberg game: Robust federated learning against adaptive and mixed poisoning attacks, arXiv preprint arXiv:2410. 174311,, 2024. [Online] Available at http::t/////m :tattss:/aatee /ttp:v//aavvv..v org http:/pf/221.110
Tao Li, Haozhe Lei, Hao Guo, Mingsheng Yin, Yaqi Hu, Quanyan Zhu, and Sundeep Rangan. Digital twin-enhanced wireless indoor navigation: Achieving efficient environment sensing with zero-shot reinforcement learning. IEEE Open Journal of the Communications Society, 6:2356-2372 2025.
doi: 10.110900CCOMMS2223355553
Liu, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu. BOME! bilevel optimization made easy: A simple first-order approach. In Advances in Neu- ral Information Processing Systems, volume 35, pp. 17248-17262, 2022. URL http:://pproceeinggg.nnurrrr wwwrinuu...... fitpppprrfieeeepppr//222/2111
Guan-Horng Liu, Avinash Siravuru, Sai Prabhakar, Manuela Veloso, and George Kantor, Learning end-to-end multimodal sensor policies for autonomous navigation. In Sergey Levine, Vincent Vanhoucke, and Ken Goldherg (eds.), ), Proceedings of the Ist Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 249-261. PMLR, 13-15 Nov 2017. URL htl/ittpss//ppoceediing /
mlr www..pree./v7/////m wwwp//i//11111 atml
Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Li Jian, Nenghai Yu, and Tie- Yan Liu. Return-based contrastive representation learning for reinforcement learning. In International Conference on Learning Representations, 2021. URL
1d= TM6ri7tXke
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances Neural Information Processing Systems, 35:16455, 16468, 2022.
Jianfeng Lu, Zuowei Shen, Haizhao Yang. and Shijun Zhang. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 53(5):5465-551) 2021.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022. URL
httpps://ppenevveew net htte:/oooommmoo/ccoooooloo
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Had- sell. Learning navigate in complex environments. In International Conference on Learning Representations, 2017. URL
net httpr/eewww.eeeeeroeeeooo--- rcle
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 518(7540::59-5333 2015.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
14

---

Published as a conference paper at ICLR 2026
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL id=rygf-kSYwH
ew. net. /t//ffruuu Forum?
M.J. Osborne. An Introduction Game Theory. Oxford University Press, 2004. ISBN 9780195128956. http:/tss//boos..ootttoooooooo google wwt.:/cc.aoooo..e
BMC
Yunian Pan, Tao Li, and Quanyan Zhu, Model-agnostic meta-policy optimization via zeroth-order estimation: A linear quadratic regulator perspective. arXiv preprint arXiv:2503.00338 2025.
[Online] Available at / /ttp://///..a...o////////5SSS 00385
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by selff-uppervieed prediction. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMLR, 06-11 Aug 2017. URL httpps:/ppe ptttp::/ppoceeddngg mlr
p2eas//770/pattha.77...aaa
leva Petrulionyté, Julien Mairal, and Michael Arbel. Functional bilevel optimization for machine learning. Advances in Neural Information Processing Systems, 37: 37:14016-14065 2024.
Eduardo Pignatelli, Jarek Liesen, Robert Tjarko Lange, Chris Lu, Pablo Samuel Castro, and Laura Toni. Navix: Scaling minigrid environments with jax. arXiv preprint arXiv:2407.19396 2024.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- ceff-predictive representations. In International Conference on Learning Representations, 2021.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9870-98799 PMLR, 18-24 Jul 2021. URL https: https:://proceedingg. mlr
html
Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. Approximate information state for approximate planning and reinforcement learning in partially observed systems. Journal of Machine Learning Research, 23:1-83, 83, 2022. doi: 008550//xxi..2100088888 URL http:
/ /23 20 116.. 6tmllll
Richard S Sutton and Andrew G. Barto. Reinforcement learning: An introduction. A Bradford Book, 2018.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionisl Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
Frederik Träuble, Andrea Dittadi, Manuel Wuthrich, Felix Widmaier, Peter Vincent Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. Rep- resentation learning for out-of-distribution generalization in reinforcement learning. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021. URL htttps::/oopereviiiee net
John N Tsitsiklis and Benjamin Van Roy. Analysis of -eiforra--difffrrencce learning with function approximation. IEEE Transactions on Automatic Control, 42(55)67,,690, 1997. doi: 10.1 100/ 9.580874. linear fucntion approximation: convengence proof #reinforcement learning #value function approximation.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint 2018.
15

---

Published as a conference paper at ICLR 2026
Théo Vincent, Daniel Palenicek, Boris Belousov, Jan Peters, and Carlo D'Eramo. Iterated SqS- network: Beyond one-step bellman updates in deep reinforcement learning. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https:/ http:::://oenneeiee.....
Corum?id=L.t2HBBE jF.
Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, and Adam White. Investigating the properties of neural network representations in reinforcement learning. Artificial Intelligence. 330:104100, 2024. ISSN 0004-3702. doi: ao.1016/j.artint..2 2024. 104100.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference machine learning, pp. 1995-2003, PMLR, 2016.
Xian Wu, Nevena Lazic, Dong Yin, and Cosmin Paduraru. Importance of representation learning for off-policy fitted q-evaluation. Offline Reinforcement Learning Workshop, NeurIPS, 2021. URL
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control:
Improved data-augmented reinforcement learning. International Conference on Learning Representutions, 2022. URLlhttss////peeeeeee
net/ Fiwftoouudddd____
Mingsheng Yin, Tao Li, Haozhe Lei, Yaqi Hu, Sundeep Rangan, and Quanyan Zhu. Zero-shot wireless indoor navigation through -nnfores-nformee reinforcement learning. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 51 5111-5,18, 2024. doi: 10.1 10/110////4/77447 1124.10611222
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv: 076.6, 2019.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33: 5836, 2020.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. International Conference on Learning Representations, 2021. htttps:///penrreieew.. id 2FCwDKRRBu
Yihua Zhang, Prashant Khanduri, Joannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning. IEEE Signal Process. Mag., 2024.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A CODE AVAILABILITY
The complete source code for SCORER and the experiments presented in this paper is available at https: /gi sigithe thub. cwm fomm/ffrnando-oee /SCCRER/ to ensure reproducibility. Our implemen- tation is built in JAX (Bradbury et al. 2018) and leverages significant components, particularly for environment vectorization and training loops, from the PureJaxRI. framework (Lu et al, 2022 While core architectural choices (Section[C) and hyperparameters for baseline algorithms are consis- tent with PureJaxRL defaults for MinAtar and DQN, our SCORER-specifie mechanisms (perception network objective, perception learning rate) additions.
B
LLM USAGE
Writing and editing assistants, including a large language model (LLM) and automated grammar checking tools, were used improve the clarity, conciseness, and grammatical correctness of this work. The usage of these tools was strictly limited to polishing the written text. All scientific contri- butions, including the framework ideation, conceptualization, theoretical analysis, and experimental results, are the original work of the authors.
C EXPERIMENT SETUP AND HYPERPARAMETERS
Software We used the following software versions:
Python 3.10 Python Software httppp/p///oosspppooooro///////.
html
CUDA - Software License Agreement
hatt:/ellaiinen html
Jax 0.4.28 Apache License 2.0 /ttttsss//ggtnubb.mm///////////
Flashbax 0.1.3
Apache License 2.0
http:://gtthbb.cmm//tseeeeep
flashbax
Chex 0.1.90 chex
Apache License 2.0
Bttpss//gatuuu.cccoooooooeeeeeep
Optax 0.2.5
Apache License 2.0
httpp://gthuubb.ommoooo//eeee//
optax
flax 0.10.4 Apache License 2.0
20tttpss/ggthhbb.c///oog///////1
Gymnax 0.0.9
Apache License 2.0
httess//gshhbbb.cmm//bbtttttLgg
gymnax
Navix 0.7.4 Apache License 2.0
Ottpps:/pppggnatellll.oooo/aaaa
Rlax 0.1.7 Apache License 2ttpt///s/ggttao.ooooooo/oo/eeee
Envpool 0.8.4 Apache License 2.0
OpenAl Gym 0.26.2 MIT License httpss//gtthubb.cm/pppaaaa/////
PureJaxRL
Apache
License
2.0
https://github.commlubbbb
pure aax1
All experiments conducted NVIDIA Tesla V100-PCIE-32GBB GPUs. typical experimental run, consisting of training one algorithm variant (c.g.. SCORER DQN) over 30 random seeds for 10 total environment time steps on MinAtar environment, completed in approximately 27
minutes.
17

---

Published as a conference paper at ICLR 2026
Hyperparameter Tuning Methodology For fair and rigorous comparison, we performed two-stage hyperparameter search where initially, for each base algorithm (e.g., DQN, PQN), we performed systematic grid search to identify its strongest possible configuration Then, we inherited these optimal hyperparameters for each SCORER variant and conducted a targeted search for the introduced SCORER hyperparameter (the follower's learning rate, (tz), while satisfying the timescale separation condition (a+ > (t)).
our benchmarks.
SCORER-Speccine Hyperparameters The key parameters introduced by SCORER are the learning rates for the leader (oo) and the follower (ora). As described in Section 4 the leader's learning rate (ag) is set to the optimal learning rate found for the baselines. The follower's learning rate (ax) then tuned from set of values greater than ag establish the necessary timescale separation.
Architectural Parity To confirm that performance gains are a result of SCORER and not increased model capacity, keep strict architectural parity between each baseline and its corresponding SCORER version. For a given monolithic baseline (e.g., a DQN with a three-layer MLP), the SCORER version is constructed by splitting this same architecture. The initial layers form the perception network (fo), and the final layer forms the control network (Qe). This way, we ensure that the total number of layers, hidden units, and learnable parameters is nearly identical between the baseline and SCORER agent, isolating the algorithmic contribution.
Learning Rate Schedule For all experiments, learning rates follow a linear decay schedule from their initial values to zero over the full training duration. i.c., a(t) an t/TT) where I is the total number of updates. This schedule is applied identically to both baseline agents and SCORER variants. The Leader network uses the exact same learning rate trajectory as the corresponding baseline, and the Follower uses the same decay schedule starting from a higher value to satisfy the timescale condition.
Gradient Clipping Gradient clipping via global norm is applied identically across all experiments. For baseline agents, gradients are clipped to a maximum norm of 0.5 (MinAtar DQN variants, MinAtar PQN, R2D2) 0.3 (classic control), and 5.0 for Full Atari. For SCORER variants, the same clipping threshold is applied independently to both the Leader and Follower networks.
Table 2: General Training Hyperparameters for Q-Learning Methods MinAtar
Parameter
Value
Training Configuration
Number of parallel environments
128 1 x 108
Total timesteps
Learning starts (time steps)
1 x 104
Training interval (env steps)
Replay Buffer
Buffer size
1 x 10 64
Batch size
Exploration Aopsilon-Greedy
start
1.0 0.01
finish
anneal time (env steps)
2.5 x 105
Learning Parameters
Optimizer
Adam
Discount factor (?)
0.99
Linear learning rate decay (Baseline SCORER)
True
Target network update interval (env steps) Soft update parameter (7) for target nets Q-network learning rate (ag)
1 x 103 1.0
1 x 10-4
Q-networks max gradient norm
0.5
18

---

Published as a conference paper at ICLR 2026
Table SCORER-Specifie Hyperparameters MinAtar
Parameter
Value
Leader-Follower Architecture
Optimizer
Adam
Leader (control) learning rate (oo))
x 10
Follower (perception) learning rate (as)
x 10
Max gradient norm for Leader & Follower
0.5
Table 4: General Training Hyperparameters for Q-Learning Methods for classic control environments
Parameter
Value
Training Configuration
Number of parallel environments
10 1 x 10
Total timesteps
Learning starts (time steps)
1 x 103 10
Training interval (env steps)
Replay Buffer
Buffer size
5 x 104 64
Batch size
Exploration (Epsilon-Greeddy)
start
1.0 0.01
finish
anneal time (env steps)
2.5 x 10
Learning Parameters
Optimizer
Adam 0.99 True
Discount factor (7)
Linear learning rate decay (Baseline SCORER)
Target network update interval (env steps) Soft update parameter (r) for target nets Q-network learning rate (ag)
1 x 103 1.0
1 x 10-4 0.3
Q-networks max gradient norm
Table SCORER-Specife Hyperparameters (Classic Control).
Parameter
Value
Leader-Follower Architecture
Optimizer
Adam
Leader (control) learning rate (oe)
x 10
Follower (perception) learning rate (axs)
x 10-4
Max gradient norm for Leader & Follower
0.3
19

---

Published as a conference paper at ICLR 2026
Table 6: General Training Hyperparameters for SCORER PQN (MinAtar environment defaults).
Parameter
Value
Training Configuration
Number of parallel environments
128
Total timesteps
1 x 10
Rollout length (experience collection)
32
Training epochs per rollout
Minibatch count per epoch
32
Exploration (Epsilon-Greedy)
start
1.0 0.01
finish
anneal period
40% of training updates (4 x 10? steps)
Learning Parameters
Optimizer
RAdam
Discount factor (7)
0.99
Lambda for TD(X) returns (A)
0.65
Linear learning rate decay (Baseline SCORER)
True (over first 50% of training) N/A (uses X-retums)
Target network
Q-network learning rate (ag)
2.5 x 10 5.0
Q-neworrk max gradient norm
Table 7: SCORER- -Seectfc Hyperparameters for PQN (MinAtar environment defaults).
Parameter
Value
Leader-Follower Architecture
Optimizer
RAdam
Leader (Q-Network) learning rate (ag)
2.5 10-4
Follower (Perception) learning rate (a.)
5 x 10
Max gradient norm for Leader & Follower
5.0
Table General Training Hyperparameters for SCORER PQN (Atari environment defaults).
Parameter
Value
Training Configuration
Number of parallel environments
128
Total timesteps
5 x 107
Rollout length (experience collection)
Training epochs per rollout
2
Minibatch count per epoch
32
Exploration Crpsilon-Greeddy
start
1.0 0.001
finish
anneal period
10% of training updates (5 x 10 steps)
Learning Parameters
Optimizer
RAdam
Discount factor (7)
0.99
Lambda for TD(A) returns (A)
0.65
Linear learning rate decay (Baseline & SCORER)
True (over 100% of training) N/A (uses X-returns)
Target network
Q-network learning rate (aş)
2.5 x 10 10.0
Q-network max gradient norm
20

---

Published as a conference paper at ICLR 2026
Table 9: SCORER-Speeee -Specffcc Hyperparameters for PQN (Atari environment defaults).
Parameter
Value
Leader-Follower Architecture
Optimizer
RAdam
Leader (Q-Network) learning rate (aw))
2.5 x 10-4
Follower (Perception) learning rate (axp)
5 x 10-1
Max gradient norm
10.0
Latent dimension
256
Table 10: General Training Hyperparameters for SCORER R2D2.
Parameter
Value
Training Configuration
Number of parallel environments
10
Total timesteps
1 x 106 25,000
Learning starts (env steps)
Prioritized Trajectory Replay Buffer
Buffer size (transitions)
1 x 10a
Min buffer size for sampling
5,000
Batch size (sequences)
32
Sampled sequence length
100
Burn-in length
50
N-step returns
10
PER alpha (a)
0.6
PER beta start (Bo)
0.4
PER beta end (3T)
1.0
Exploration Aessionn-Greedy
start
1.0
c finish
0.01
anneal time (env steps)
5 x 10a
Learning Parameters
Optimizer
Adam
Discount factor (n)
0.99
Linear learning rate decay (Baseline & SCORER)
True
Target network update interval (env steps)
5,000
Update interval (env steps)
16
Soft update parameter (7)
1.0
Q-network learning rate (axa)
1 x 10-4
Max gradient norm
0.5
Table 11: SCORER-Specific Hyperparameters for R2D2.
Parameter
Value
Leader-Follower Architecture
Optimizer
Adam
Leader (Control) learning rate (ax)
1 x 10
Follower (Perception) learning rate (a.))
3 x 10-4
Max gradient norm (Leader & Follower)
0.5
21

---

Published as a conference paper at ICLR 2026
C.1 NETWORK ARCHITECTURES
All agents within the DQN family (DQN, DDQN, Dueling DQN, and Dueling DDQN) are built from a shared architectural template. The monolithic baseline for each variant is constructed first, and its corresponding SCORER version is created by splitting this same architecture to maintain parity in the number of layers and parameters. Table 12 details this base architecture.
Table 12:
Base network architecture for the SCORER DQN agent family. Linear (in, out) denotes fully connected layer and L1Norm denotes layer that normalizes its input to have a unit LI norm. Key dimensions are environmentt-deeenden. The acti activatie ion function applied after hidden layers, but not on the final Q-Network output.
SCORER DQN Base Architecture
Hyperparaaetter::
state_dim Dimension of observation vector action_dim= Number of discrete actions latent_dime 64 (for Control tasks). 128 (for MinAtar)
q_hidden_dee dim 64 (for Control tasks), 128 (for MinAtta)
activation Tanh (for Control tasks), ReLU (for MinAtar)
Perception Network (Follower, fe)
Encodes raw state into latent representation z. MinAtar environment:
input_state NoNoomsstaaa
Else:
input_state state
P_10 Linear (state_dim, latent_dim)
Perception Network Forward Pass:
activation (p_10 (input_state)
ho (For Control tasks)
If MinAtar environment:
p 11 Linear (latent dim, latent_dim) h1 activation [p_11 (h0)) LiNorm(h1) (Overrides 2)
Q Network (Leader, Qa)
Predi cts Q-values from the latent representation z.
10 Linear (latent dim, q_hidden_diml dim)
11 Linear (q hq__iidee_diiae action_dim)
Q Network Forward Pass: input
h0 act iiviviiioon (q_10 (input)) vaalues q1 (hO)
Architectural Variants The architecture described in Table 12 serves as the foundation for all agents in the DQN family.
DQN and DDQN: The monolithic baselines for DQN and DDQN consist a single network formed by composing the layers described above (e.g.. for MinAtar, four-layer MLP). The SCORER variants are created by splitting this architecture exactly as shown above.
22

---

Published as a conference paper at ICLR 2026
Dueling DQN and Dueling DDQN:
For these agents, the Perception Network (fe) remains
identical. The Q-Network (Qo) is modified to implement the dueling architecture (Wang et al., 2016). Specifically, the 'h0" feature vector from the Network's forward pass is fed into two separate heads; a state-value head, (2) = Li near r and advantage head, A(2, a) Li near (q_hi dden aiden_dim, act action_ddi dim). The final Q-values are then combined using the standard aggregation method: Q = V(2)+ (A(z,a) T
dden_dimm 1),
1 Ee A(z,a'))
Table 13: Base network architecture for the SCORER PQN. Linear (in, out) denotes fully connected layer, Conv (k, C, 5) a convolutional layer, and Norm denotes LayerNorm. Key dimensions are environment-dependent. The activation function is applied in hidden layers.
SCORER PQN Base Architecture
Hyperraaaameter::
state dim. Dimension of observation vector (or image shape for Atari) action_dim = Number of discrete actions
latent_dimi 64 (Control), 128 (MinAtar), 256 (Atari) chidden_dimm 32 (Control), 64 (MinAtar), 256 (Atari) activation ReLUU (for all tasks)
norm_type = LayerNorm (for all tasks)
Perception Network (Follower, fa) > Encodes raw state into a latent representation z.
Option A: Vector Observation (Control, MinAtar)
P_10 Linear (state_dim, latent_dim)
Forward Pass:
p_10 (state)
h1 Norm (h0)
h2 activation (h1)
Norm (h2)
Option B: Pixel Observation (Atari)
enn_block = [Conv (8x8, 32, 4), Norm, Relu, Conv (4x4, 64, 2)
Norm, Relu, Conv (3x3, 64, 1), Norm, Relu]
P_10 Linear (cnn_out_dim, latent_dim)
Forward Pass:
features = enn block (state)
P_10 (features, flatten
h1 Norm (h0)
activation (h1) (Atarí ends with activation)
Q Network (Leader, Qs)
Predicts Q-values from the latent representation z.
Contains q_num_layers hidden blocks.
hidden. = Linear [latent_ding h_didden )
out Linear (q hidden dim, action dim)
Q Network Forward Pass (for q __n _am_laaers==); input
h0 h_l_eideen (input)
h1 Norm (h0)
h2 activation (h1)
q values q_ out (h2)
23

---

Published as a conference paper at ICLR 2026
Table 14: Architecture for the SCORER R2D2. Linear (in, out) denotes fully connected layer, GRUCell (in, out) denotes a Gated Recurrent Unit cell, and L2Norm denotes a layer that normalizes its input to have a unit L2 norm. The activation function is applied after hidden layers, but not on the final Control Network output.
SCORER R2D2 Base Architecture
Hyperparameters:
state_din: Dimension of observation vector action_dim= Number of discrete actions
embed_dim 128
recurrent_ ______iii_ = 128
g_hidden dim 128
activation Tanh
Perception Network (Follower, fe)
Encodes state and updares the recurrent hidden state.
>> Takes current state slate, and previous hidden state h411 as input.
p_l_embed = Linear (state_dine embed_dim)
P_gru GRUCe11 (embed_dimg recurrent_dim)
Perception Network Forward Pass:
input_staae = L2Norm (state_t)
emb p_l__mbed ( dinput_sttatee
emb_act = activation (emb)
he, r_w = p_gru (h--1, emb_act)
(h, is new hidden, r_raw is
GRU output)
2 12Norm (z_raw) (Representation passed to Control Network)
The new hidden state for the next step is 12Norm (h4)
Control Network (Leader, Qa)
> Predicts Q-values from the latent representation z.
1
hihhiddrr = Linear (recurrent_dim, s_hiidde_dim) q out Linear (q hi__hidde__diiie action_dim)
Control Network Forward Pass:
input
h0 a_l_hidden (input)
h1 activation (h0) vaalle = q out (h1)
24

---

Published as a conference paper at ICLR 2026
D OBJECTIVE FUNCTIONS
This section provides the explicit formulations used by the SCORER framework. In our game, the Control Network (Q&) acts the Leader and the Perception Network (fo) acts as the Follower.
D.1 LEADER (CONTROL NETWORK) OBJECTIVE
As detailed in Section 4.1 equation D. the control network Qo acts as the Stackelberg leader. Its objective is to learn an optimal action-value function by minimizing the Mean Squared Bellman Error (MSBE). Given a batch transitions Bieader and the follower's best-reponse representations før, the leader's objective is:
Cleader (0, Φ²) Eis,,,,s'' dl Blaks
[ - (s)
The Bellman target Y used in these objectives is defined in Section 3 In our two-timescale algorithm, the leader's update step uses the follower's 's most recent parameters 0k+1 as a proxy for the ideal Ø² and the gradient is taken with respect to & only.
D.2 FOLLOWER (PERCEPTION NETWORK) OBJECTIVE
The follower's goal is to learn representations that support the Leader's learning process. our ablation studies, we explored two objectives for the follower, both of which treat the leader's parameters 0 as fixed via a stop-gradient operation. See cquation and A Line 12.
D.2.1 MINIMIZING FOLLOWER'S 'S MEAN SQUARED BELLMAN ERROR (SCORER MSBE)
In this ablation variant, the follower's objective is identical the leader's loss. Here, the follower learns representations that directly minimize the MSBE given the leader's committed weights:
Follower (6,
1 N
Σ (yy (o(lo(t)).aa)))
The optimization is performed with respect to
D.2.2 MINIMIZING BELLMAN ERROR VARIANCE
This is the main objective used in SCORER, designed to explicitly stabilize the leader's learning signal. The intuition behind is that by focusing on the consistency of the Bellman errors across batch, the follower can learn representations that are robust to the noisy target inherent in TD learning. Let the Bellman error for a single transition be (á (ó, 0) = Y, (f(oss))) The follower's goal minimize the sample variance of these errors as:
2
followerr (0.0) 0)
(o.
1 Σ (o,(0))))² N
N Σ ( 0)
=
#=1
This formulation is the standard sample variance (Casella & Berger 2024). which uses all N samples
from the batch for both terms, and is algebraically equivalent to プー (ōj 5)2 where å denotes the sample mean.
This objective turns the follower into an active stabilizing agent. Because the leader's performance is evaluated an independent sample batch of data (Bjenter, the follower is asked to learn repre- sentations that create a generally stable learning signal, instead of overfitting the specifics of its own sampled batch (Biollowr). The follower must find representations that make the leader's 's value predictions more uniform and predictable across the entire data distribution, truly dividing the task of stabilization from the immediate task of performance maximization. Our ablation studies (Sec- tion (3.5) empirically validate that this variance- minimization objective outperforms using MSBE for the SCORER's follower objective.
25

---

Published as a conference paper at ICLR 2026
Stochasticity and the Variance Objective. A well-known theoretical challenge in minimizing Bellman residual errors is the need for unbiased gradients, which typically requires two independent samples of the next state (Baird, 1995). SCORER addresses this by minimizing the sample variance calculated over a batch as treats the batch statistics as a deterministic objective, ensuring that the gradients are exact with respect to the sampled data.
In stochastic environments, the total variance of the Bellman error naturally includes irreducible environmental noise. Yet, this total variance decomposes into aleatoric variance (environment stochasticity) and epistemic variance (prediction inconsistency):
Varisal [6]
~ Varalestoric
+ Varepistmii
(10)
Because the perception network cannot alter the environment's inherent noise (Varatemtoric). reducing the sample variance targets the reduction of eprepisemic- This way, we push the follower to learn representations that make the value function errors as consistent as possible across the batch, sim- plifying the learning signal for the leader. Our results the Stochastic Deep Sea environment (Section 5.4 confirm that this objective successfully stabilizes learning even in the presence of significant environmental noise.
E MOTIVATION FOR STACKELBERG ROLES
Assigning the role of the leader the Control network (Qo) and setting Perception (fe) as the follower is grounded in the structural properties of the corresponding bi-level optimization. The ultimate goal of Q-learning is identify the Q-function that minimizes MSBE in Sec. 3 Comparing the optimization problems faced by the control and perception networks in equation and equation is natural setting to have equation aas the outer problem (sometimes also known as the upper-level problem in the literature, see, c.g. Zhang et al.) (2024)) and equation as the inner (lower-level) problem in the bi-level optimization connecting them. The leader-follower roles for the two networks in the Stackelberg game thus in turn reflect this relationship.
discussed
F STATISTICAL SIGNIFICANCE CALCULATIONS
Throughout this work, reported performance metrics aim to provide robust understanding of algorithm behavior across multiple independent trials. This appendix details the methods used for calculating and presenting these statistics.
F.1 IQM AND CONFIDENCE INTERVALS
The learning curves represent the IQM (Interquartile Mean) of the total reward, averaged over independent runs (seeds). Following Agarwal et al. (2021b), the IQM statistic is recommended evaluation metric deep RL as it has the same properties as a 25% trimmed mean. The bottom 25% and top 25% of run scores removed each timestep before the mean is computed, which mitigates the effect of outliers without losing as much statistical power as the median.
The shaded areas represent 95% confidence intervals (CIs) estimated using the percentile bootstrap method (Agarwal et al.) 20216). We estimate the sampling distribution of the IQM by generating B 2000 bootstrap samples that resample the seeds with replacement, from which the 95% CI is given by the 2.5th and 97.5th percentiles of the bootstrap distribution, without assuming normality. The curves are smoothed by rolling mean over 50 timesteps, which is applied to both the IQM point estimates and the CI boundaries.
IQM for final performance tables For reporting final performance tables, we first compute the mean return over the last 10% of training for each seed, which results in more stable final score per run. We then compute the IQM over these final scores from all seeds using the 25% trimmed mean, and CIs estimated via percentile bootstrap with B 2000 resamples. To test for statistical significance when highlighting values in the tables, we perform a bootstrap difference test: compute the distribution of IQM differences between the hest-performing variant and each
26

---

Published as a conference paper at ICLR 2026
competitor, and count the number of bootstrap samples in which the competitor matches exceeds the best. this proportion less than 5%, we consider the difference to be statistically significant (p 0.05).
For comparative analysis across environments with different reward scales, we present normalized IQM returns in some figures. The normalization performed per environment by dividing all IQM values by the maximum IQM value reached across all methods in that specific environment. This helps to develop meaningful comparison of relative performance improvements across tasks while preserving the temporal dynamics of learning.
Normalized IQM Return
F.2
SPEED COMPARISON
The Speed Comparison column Table reports the relative wall-clock time required to complete a full training run. For each base algorithm (e.g., DQN), the runtime of its standard monolithic version normalized to 1.00x. The runtime of the corresponding SCORER variant is then reported as a multiple of this baseline. A value of 0.99x indicates that the SCORER variant was I 1% faster than its baseline, whereas value of 1.01% % would indicate it was 1% slower.
F.3 TIME-TO-THREE THRESHOLD (TTT) CALCULATION
To provide a measure of sample efficiency over the MiniGrid experiments, we use a Timme-t-TThresoldd (TTT) analysis. TTT quantifies the number of environment steps required for agent to reliably fulfill the task, or at least reach a high level of performance. Our calculation follows a two-pass process to ensure fair comparisons across environments with different reward scales and performance ceilings.
F.3.1 PERFORMANCE THRESHOLDS
Initially, for each evaluation environment, we determine the maximum asymptotic performance achieved across all runs of all methods. This sets empirical "best-case" performance for the task. The performance threshold for each environment is then set to 99% of this maximum value.
F.3.2 CALCULATING TTT AND SUCCESS RATE
In the second pass, for each individual seed of each method, we identify the first timestep which the agent's episodic return meets or exceeds the calculated 99% performance threshold. This timestep recorded as the TTT for that run. a run fails to reach the threshold within the maximum allowed timesteps for that environment, its TTT is considered undefined. The final TTT reported in Table 4 the mean over all successful runs. To capture the reliability of each method, we also present the Success Rate (SR), which is the percentage of the 30 independent runs that successfully reached the performance threshold.
F.4 STATISTICAL SIGNIFICANCE TESTING
In Table we highlight statistically sigmificant performance differences within each algorithm family (e.g. DQN vs. SCORER-DQNI DDQN per environment. We first identify the variant with the highest IQM within each group. We then use a bootstrap difference test to compare each the other variants to this top performer: for each of the B 2000 bootstrap resamples, variants and record the difference. less than 5% of bootstrap samples have the competitor matching or exceeding the top performer, we deem this difference to be statistically significant (p 0.05). All variants that are not significantly worse than the best are highlighted in green.
compute the IQM of both
G SAMPLE EFFICIENCY RESULTS
To complement the learning curves presented Section this section provides a detailed break- down of the sample efficiency for SCORER compared to the baseline agents across four MinAtar environments. Tables [5 through (arughh[[rrppe report the mean number of training steps (in millions, 95% CI) required for agent to reach pre-defined reward thresholds, averaged over 30 seeds. For each
27

---

Published as a conference paper at ICLR 2026
experimental condition, is considered to have reached threshold the first time step where its individual performance curve meets or exceeds the target value. The tables report the statistics for all seeds that successfully reached the threshold. An "N/A" indicates that the Interquartile Mean (IQM) of the agent's 's runs failed reach the threshold within 10 steps. The highlighted values indicates the variant that achieved the given threshold in the fewest timesteps.
Across the board, the data shows the substantial impact of the SCORER framework on learning speed. The effect is most pronounced in Breakout (Table [5). Here, the SCORER agents are an order of magnitude more sample-efficient than their monolithic baselines.
Table 15: Sample efficiency (time steps in millions to reach reward thresholds) for Breakout-MinAttar
Model
Variant
Threshold 17
Threshold 20
Threshold 25
Baseline
39.1 + 7.6
N/A
N/A
DQN
SCORER
3.4 + 0.2
4.8 + 0.5
8.2 + 0,9
Baseline
42.4 +: 6.6
43.8 + 5.0
62.3 +: 6.4
DDQN
SCORER
4.2 - 0.4
5.9 0.8
12.7 = 3.6
Baseline SCORER
43.6 + 8.4
50.8 + 9.7
72.0 + 9.1
DuelingDQN
4.0 0.4
5.3 # 0.6
13.5 2.7
Baseline
27.6 - 6.5
33.7 + 5.8
64.2 + 5.2
DuelingDDQN
SCORER
4.2 + 0.4
5.6 + 0.6
17.8 E 7.9
In Asterix and SpaceInvaders T6 and 17ble(T). SCORER consistently reduces the number of samples required to reach performance thresholds, demonstrating robust improvements in sample efficiency across different base algorithms. The gains are particularly notable for higher thresholds.
In Freeway (Table 18
all agents converge rapidly. Even so, the SCORER variants consistently reach all performance thresholds in approximately half the time of their baseline counterparts.
Table 16: Sample efficiency (time steps in millions to reach reward thresholds) for Asteri--MinAta
Model
Variant
Threshold 35
Threshold 40
Baseline SCORER
22.8 + 0.6
27.3 + 1.1
DQN
14.6 + 2.2
17.3 + 2.4
Baseline SCORER
23.3 + 0.9
27.7 + 1.2
DDQN
14.8 + 1.0
18.2 + 1.2
Baseline
19.8 + 1.0
22.6 1.0
DuelingDON
SCORER
15.4
18.3 + 3.4
Baseline SCORER
20.5 1.0
25.1 + 1.8
DuclingDDON
17.2 1:2::0 6.4
16.6 + 3.2
28

---

Published as a conference paper at ICLR 2026
Table 17: Sample efficiency (time steps in millions to reach reward thresholds) for SpaceInvaders- MinAtar.
Model
Variant
Threshold 75
Threshold 100
Threshold 110
Baseline
9.7 + 1.0
20.6 + 2.7
31.2 + 2.7
DQN
SCORER
9.1 + 0.5
14.3 0.8
17.8 + 1.7
Baseline
11.8 + 1.1
24.7 + 2.7
37.1 + 2.6
DDQN
SCORER
11.1 + 0.6
18.6 2.5
22.9 3.4
Baseline SCORER
10.0
23.4 + 4.1
33.2 5.7
DuelingDQN
17.1 = 0.7
26.0 + 1.8
32.0 2.5
Baseline
9.9 0.7
16.6 1.4
26.9 + 3.1
DuelingDDON
SCORER
15.4 0.7
23.8 + 1.8
30.2 + 3.4
Table 18: Sample efficiency (time steps in millions to reach reward thresholds) for Frecway-MinAtarr MinAtar.
Model
Variant
Threshold 30
Threshold 40
Threshold 50
Baseline
3.8 0.2
6.0 + 0.3
10.1 + 0.5
DQN
SCORER
1.3 - 0.1
2.3 + 0.1
4.5 = 0.1
Baseline
3.8 + 0.3
5.9 + 0.3
9.6 + 0.4
DDQN
SCORER
1.4 = 0.1
2.3 0.1
4.5 0.2
Baseline
4.7 : 0.4
6.1 + 0.3
10.0 + 0.4
DuelingDON
SCORER
1.8 0.2
2.7 + 0.2
5.0 + 0.4
Baseline
4.6 0.5
6.8 + 0.7
11.5 + 1.1
DuelingDDON
SCORER
1.6 0.2
2.6 0.2
4.9 = 0.4
H CLASSIC CONTROL RESULTS
To further assess the general applicability of the SCORER framework beyond the MinAtar suite and MiniGrid, we conducted experiments on two classic control environments from OpenAI Gym (Brockman al.) 2016) using Gymnax (Lange (022) for JAX compatibility: For these tasks, simpler Multi-Layer Perceptron architectures were used for both the perception and control networks, with details provided in Appendix C
29

---

Published as a conference paper at ICLR 2026
Acrobot-v1
CartPole-v1
DON
200
-300
230
4400
100
-500
250
1000
-100
DDON
-200
Interquartile Mean Return
-300
200
400
100
-500
500
750
1000
250
500
1000
-100
400
DuelingDON
-200
300
300
200
-400
250
500
750
1000
250
500
750
1000
DuelingDDON
-100
400
-200
300
-300
200
-400
250
500
750
1000
250
500
750
1000
Training Steps (Thousands)
Training Steps (Thousands)
Baseline
SCORER
Figure Learning curves classic control environments (CartPole-vl, Acrobot-vl), Each row corresponds a base algorithm (DQN, DDQN, DuclingDQN, DuelingDDQN), and each column to an environment. Curves show IQM return over 30 seeds; shaded regions represent 95% confidence intervals.
30

---

Published as a conference paper at ICLR 2026
I LEARNING CURVES ON MINIGRID ENVIRONMENTS
This section provides the full learning curves that support the time-t--trreshole analysis presented in Table We evaluate the performance of SCORER when applied to R2D2, a recurrent baseline for partially observable environments, across a suite of MiniGrid tasks.
DoorKey 6x6
SimpleCrossing S9N2
1.0
1.0
088
0.6-
0.0
0.0
9/0
0.0
Number
Number (xled)
DistShiftz
LavaGop 56
1.0
3.0
0.8
0.2
0.0
0.0
Number
Namber
GoToDoor
Empty Random
1.0
2.0
0.4
0.4
0.2
0.2
0.0
0.0
steps
steps
Dynamic Obstacles
Rooms
0.6
0.5
0.7
0.6
0.5
0.3
0.4
0.2
0.1
0.
0.0
0.0
2.0
0.9
Number steps (K126)
Namber stops (xLo6)
SCORER
Figure 8:
Learning curves for SCORER R2D2 SCORER versus the Vanilla R2D2 on eight Minigrid envi- ronments, averaged over 30 seeds. Shaded regions represent 95% confidence interval. The plots highlight SCORER's s consistent improvement in sample efficiency and its ability to solve challenging exploration tasks like Four Rooms where the baseline fails.
The learning curves in Figure g visually confirm Table 4 results. The SCORER-enhance agent demonstrates steeper learning curve most environments, indicating superior sample efficiency. The most significant result is observed in the Four Rooms environment, classic hard hard-exploration task. Here, the baseline R2D2 agent performs poorly, compared SCORER R2D2, which can solve the task, reaching high success rate.
31

---

Published as a conference paper at ICLR 2026
J DEEP SEA RESULTS
Even in stochastic MDPs, minimizing Bellman error variance remains beneficial. The total variance decomposes as Var[o] aaraleuoric + Varepistemic where aleatoric variance comes from environment stochasticity (non-reducible) and epistemic variance comes from value function error (reducible). The follower learns representations that minimize the total variance, which indirectly reduces epistemic variance by making the value function more consistent across the batch. We validate this using stochastic Deep Sea environment (depth 10) the suite's 's baseline (Bootstrapped DQN) (Osband et al., 2020 2016). Deep sea is a challenging exploration task available in both deterministic and stochastic variants.
Deterministic Deep Sea
Stochastic Deep Sea
Boot. DQN
Boot, DQN
SCORER MSDC
SCORER
SCORER BEVar
SCORER BEV8!
Solved
Solvad
5000
Episodes
Episodes
Figure 9: SCORER Performance Deterministic and Stochastic Deep Sea (Depth 10). (Left) Deterministic Deep Sea: Both SCORER variants converge substantially faster than baseline Boot- strapped DQN, with near-complete solved rates by episode 5000. (Right) Stochastic Deep Sea: SCORER succeeds despite increased environment noise, achieving near-complete solved rates while the baseline reaches around 20%. The suriance-minimizze objective remains effective even in the presence of irreducible stochasticity.
K DETAILED ABLATION STUDY RESULTS
This section provides detailed, per-environment learning curves for the ablation studies summarized in Section 5.5
Breakout
Spacelnandera
Astarix
Freenay
133
10
28
60
SCORT
Jader Parception
Figure 10: Per-environment results for the Stackelberg role assignment ablation. The standard SCORER configuration (Control as Leader, purple) is compared against the monolithic DQN baseline (green) and inverted hierarchy where Perception as the Leader (teal).
Stackelberg Role Assignment. Figure 10 details the results of our role-swapping experiment. The plots confirm that the standard SCORER configuration, where the control network acts as the leader, consistently and substantially outperforms both the baseline and the inverted hierarchy across all tested environments. The performance collapse observed when perception assigned the leader role is particularly pronounced in complex environments like Breakout and Spacelnvaders, providing strong evidence that for stable learning to occur, the value function must lead the representation learning process.
32

---

Published as a conference paper at ICLR 2026
Breakout
Spacelnvoders
Trabning Exps Oliikons
Trunig Stopa (18 or3)
Traiing Etpp (0/lkmm)))
Trating Stass (44) (rr))
Vantsoo)
Scorenn (4SDE)
Figure 11: Perception Objective
Follower's Objective. Figure presents a per-environment breakdown the follower's objective ablation. The results reinforce the conclusion from our main analysis: the Bellman Error Variance objective critical for SCORER's performance. every environment, the BE Variance follower achieves the highest final performance and demonstrates the best sample efficiency. While the MSBE follower offers a clear improvement over the baseline, particularly in Breakout, the explicit stabilization provided by the variance objective unlocks significantly higher level of performance, highlighting its role in enabling robust co-adaptation.
Learning Rate Sensitivity. We study SCORER's 's sensitivity to the timescale separation by testing on different learning rates for the follower (a.) while fixing the leader's rate (ag 1 x 10-A). Figure 12 compares follower rates from 2 x 10 to 10 (ratios 2:1 to 10:1). The results show that SCORER not hypersensitive to the exact ratio; nonetheless, performance drops noticeably at the lowest rate (2 10-1), confirming the necessity for the follower to adapt sufficiently fast relative to the leader to compute an effective best response.
Anderit
Braakout
Fresway
Spacelovadera
150
50
130
20
60
Training Stope (MSlOOn)
(Milloos))
Traintng (Milllors)
Training Staps (Milloos)
5.0e
1.00..0
Figure 12:
Performance across MinAtar environments with varying follower learning rates (ao)), given a fixed leader rate
Hierarchical vs. Synchronous Ablation Figure 13 shows per-environment learning curves com- paring all ablation variants. The monolithic per-layer LR baseline (encoders: 10-1, Q-heads: I x 10 10-4) outperforms the baseline in Breakout and SpaceInvaders, confirming that utilizing faster representation updates is of benefit. However, SCORER still surpasses this much simpler alternative.
Braakout
Spacelivaders
Asterix
Freeway
150
80
100
25
20
100
30
100
Training Steps (Mlloon)
Traising Steps (Milloos))
Training Steps (Millann))
Training Steps (Millton]]
Coupling
Figure 13: Team Coupling Study across MinAtar environments
33

---

Published as a conference paper at ICLR 2026
L LEARNING DYNAMICS AND REPRESENTATION ANALYSIS
To gain fürther understanding into SCORER's ' behavior, we analyze dynamics of representation rank and parameter norm on Breakout-MinAtar motivated by recent diagnostic efforts on the causes of deep RL pathologies Kumar et al. 2021
Performance
Parmeier Norm
Representation tion Rank
60-
60-
110
Parmm Morm
30
114
00
110
10
108
SCORER
Baseline
60
100
20
40 60
100
Training Stops (Millions)
Training Stops (Milliions)
Training Stops (Millions)
Learning dynamics on Breakout-MinAtar. (Left) Returns over training, showing SCORER's s sample efficiency advantage and higher performance. (Center) L2 parameter norm of all network weights; both methods eventually stabilize, but SCORER achieves larger value. (Right) Srank; SCORER maintains near-maxima, rank throughout, whereas the baseline only recov- ers a comparable value after 440M steps.
Figure 14:
Representation Rank. We track the effective rank of penultimate activations using Srank (Kumar et al. (02T), which is an estimate of the minimal number of singular value components required to account for 99% of their cumulative sum. Figure [4 Right) shows that the difference in the when of high-rank representations, rather than the whether. Bascline DQN starts at lower effective rank of around 107 (out of 128 dimensions), and takes about 40 million steps to approach meximalxi values around 119. SCORER reaches high effective rank very early in training and continues to hover around that level, though with a slight dip during initial exploration. The period of maximal rank divergence (0-40M steps) coincides precisely with the phase where SCORER demonstrates its largest performance advantage (Figure [4] Left). While our experiments cannot establish causality, this temporal correspondence is consistent with recent findings that representation collapse (the failure to utilize available representational capacity) is a primary cause of sample inefficiency in deep Q-learning (Kumar al. 2021 Lyle al., 2022). SCORER's game-theoretic structure appears to provide a natural regularization that maintains high-rank representations throughout training.
Parameter Norm. Figure [4] Center) shows the L2 norm of all network parameters throughout training. SCORER exhibits faster parameter norm growth than the baseline, with both methods stabilizing in the latter half of training.
M THEORETICAL FOUNDATIONS OF SCORER
In this section, we discuss optimization (especially bilevel optimization) related issues in the SCORER framework. Since the results are of independent interest for large family of bi-level optimization problems, we adapted a general formulation.
M.1 BILEVEL OPTIMIZATION & TWO-T'MESCALE ALGORITHMS
The SCORER framework formulates the interaction between perception and control as a Stackelberg game, which we cast as a standard bilevel optimization problem:
min ((x) = XER4
s.t. y (x) E arg min y(x,
(1))
yeR-y
Mapping to SCORER. SCORER framework as:
In the context of SCORER, this general formulation maps directly to the
The outer (leader) variable x corresponds the control network parameters 0.
34

---

Published as a conference paper at ICLR 2026
The inner (follower) variable y corresponds the perception network parameters Φ.
The outer objective f(x,y y) is the Mean Squared Bellman Error (MSBE) loss for the leader.
The inner objective g(x, y) is the Bellman Error Variance loss for the follower.
The remainder of this section will proceed with the general (x,y) notation to align with the standard optimization literature.
Stochastic Approximation (TTSA) algorithm proposed in Borkar 1997 updates the variable through the following iterative step,
y k+1 yh BaD g¹ x k+1 =x" ax Dr
(12)
(13)
where two sequences of learning rates } and (Bx) satisfying Ozk 1Bx - O, as k + oo; DJ and D represent stochastic estimates of the gradients Vx y and Vyg(x*, y²), respectively, with V fzz y) == fx, y) 2 V y) -1 y y). The following conditions on the objective functions and their approximations discussed in |Hong et al. (2023), here, while
we follow mostly their notations and basic arguments, detailed calculations are modified to fit our purposes.
Assumption 1. The outer function f(x,y) satisfies gradient Lipschitz conditions: for any Y1 * Y2 E R
IV y1) Vxf(xxy)2)1
I Ls,,
uniformly in x,
1y -22l
/ y1) (xf(x,
< Ligs
uniformly in x,
lyy y2ll
and for any x1 + X2 E Rdd,
[V,,(xx,, y & T.y
uniformly in
X1 X2
Gradient bound condition: y/ xx S Cra uniformly in and y. Assumption 2. The inner function g(r, v) C2 R*),
I V y1) (x)(
Lg uniformly in x.
||y
Y2ll
2 22,0 (x, - V2 Vx y2)ll
uniformly x,
lly1 -22l
22 9x (x. - V V y2)ll
uniformly in x,
Ily1 -y2l
V2 (xyg(x11))
V2 y)ll
< L Lggn
uniformly in x,
||x2
Xx
V2 12,9((11,)) - V2 Viyg (x2,
L gou
uniformly in x,
||x2 X2
Convexity: For any x E RRd,, g(x,) -) is strongly convex in y with modulus Mg Hessian boundedness: (xx,) S Cory. uniformly in x and y.
It is known that under these assumptions, have
35

---

Published as a conference paper at ICLR 2026
Lemma M.1 (Lemma 2.2 from Ghadimi & Wang (018)).
V V((=)] V Myy - vll. (x1) S Tyllar || - =2ll,
(14)
[x), (x1)) VZ (x2))) s ||| x2lb
(15)
with
Lig Cos + Csv
(Len Hg
L Luu Coas
I = 1.g +
+
1g
(1r + Laxx
(Ligen
Lig Caas
by= = Liga
+ Ctu
Tuy
Uxx
Pg
H4g
12
Fg
We also make assumptions on the random approximation of the gradient. The widely used assumptions
in optimization literature, see e.g. [Hong al. (2023): Li al. (2024): Pan et al. (2025), are usually in the following form Assumption 3. There are two positive constants of and Tg. and nonincreasing sequence (bx 3220
such that,
E D ş Fi) = Vg(at."²) E F = Vg(a*, +2)) + BE, ||ll| S bb:
(16)
E D Vug(E*, M| S o2 + Vzg(t )19.
(17)
B[|D (zl("", , - By 11 S o.
(18)
In this paper, the function and derivative approximation are realized through deep neural networks. Therefore, it is desired to connect the constants y; gg and [ to network parameters, such as their widths and depths. which can be adjusted to ensure the assumptions hold. This type of
quantitative relation between the effectiveness of the approximation and network parameters has been carried out recently, see, e.g. [u al. (202)) 1) and Belomestny al. (2023). Summarizing their results, for neural networks of width W and depth D, the following can be reasonably assumed.
Assumption 4. There are two positive constants o and gg. and nonincreasing sequence (bx (bx) such that.
E[D Fi] Vg(a" .8", E[D Vg(a", y + BR. |Bll S S
(19)
B - V Vot(t, |1 S 71 + yg " )
(20)
B , B& [R S 0
(21)
with S : 2222((d+dd.) D 222)
Assumption 5. For any fixed E RRd,, the inner objective function g(x, .) satisfies the Restricted Secant Inequality with respect to its minimizer y" (x). That is, for all y E TRd,, we have:
Vvg(x, -
y" (x)) 2 Holly yx))l²
where My > 0 is a constant.
M.2 CONVERGENCE ANALYSIS OF THE ALGORITIIMS FOR INNER OPTIMIZATION
The update for the inner optimization (SGD) takes the following form,
y ++1 y* D' g
(22)
where D is a random variable approximating V V.g((** ",g(x*,,()) )
The goal is to estimate Bllly*+ y²(x""))) || [R), for that, we have,
Elly (z ) =Blly* - (x** 11
=B]lb y' (a+)) 2.2 D
+
From restricted secant inequality (RSI), we know that
E(D v (x* and Vyg(a* " v² (1*^)]
RSI
2
Mg ll y² (1+
36

---

Published as a conference paper at ICLR 2026
Plug this back into the equation above, we have,
Billy+1 v² (1FF 1 23w))1e - --²(<")
|| DO -
Now, let us examine the last term,
E B 12 [Fx = B||| D 2| | 20(=", /² 1 + V V0 w'''
B - Vug("* // )1 + ااا )1
2 00 +035 V | || + 11 va )²
0 + (1+ V |00 y²
0 +1+
,²) V vglat, .v v² (1
0 + (1+ و - 1f (rt))
where (1) is due to the fact that E[DS|||
vgg(=" y²); (2) is due to the assumption on vari- ance of D in Assumption 4 (3) is due to the fact that y" (x*) is the stationary point, hence
Vyg(a*, y (x*)) = 0; and (4) due to the gradient Lipschitz assumption.
Now, with the assumption that B& goes to zero, as k + 0c, Bx + ogg) S ps always holds for sufficiently large k. Hence, we have,
Elly y" (1 5(1 Be) * - y" (2 + 2200
- (1. Next, to form a recursion, split 11o* - y" (z*))² into 1 - yn (x 111- + 1115 1115"(4 (x^-1)- Thus,
11"(a~t1)~ y²=""")""""" S 1, - -0 12 = - 12
where the first inequality follows from the Lipschitz continuity of y" (.) (Lemma M. and the equality follows directly from the definition of the unconstrained update rule in Equation equation T3 Therefore,
Bllly 3 (a [|2 SB(1 - Ba) - 1² (aat11)12 + Ro + a 12-1
Hence, we have,
E| - (++>)))
<E(1 - Bxx + L + (3)1 - - "
1)12 + 3802 of
As we know 0x/Bx= + 0, we can see that (1 Bx + 12 will be the uniform contraction factor, and 8o + ai 1. A is a correction term also tends to zero, therefore, E[lly* yy (a )|² diminished to zero, and the rate can also be quantified, especially in terms of the size of the neural networks.
M.3 CONVERGENCE ANALYSIS OF THE ALGORITIIMS FOR OUTER OPTIMIZATION
From the unconstrained update rule in equation 13 we have:
|| x 12
وز 21
||2 - 12
2ox (D) x² -²") + a
where x² denotes the global optimum of problem defined in equation I1 From the Assumption Aon the random variable Dj. we can see that,
BBD x²)) [Fn] = <V +++1)+++ + Bu, x² - -x")
[VV((=*) 1² - xx)) + (Val(x* (1 - Ve(x*) + Bu, x²).
Hence, we have,
- [11(FF $11x** - x= x²112 20 (l((z)), ), -x") + D
2ax V. ((* )h Vl(x²t) BC, - x²))
37

---

Published as a conference paper at ICLR 2026
Restricted secant inequality implies that,
(V((="), x² - xx) -VV(((=)- xx²) 2 uellx* - -0²112
We then have,
B B|||2 -+1 x
x111
FR FR 5(1 - 20m x²111 20 (V f ,2 y(+h1) - ve(I*) + BR, x - -x"
+ b D [2
x² 1|
an | of (2 x1 V((x*t) + BBxl12
pe
+ F D2 [
20tk
(o/em)))l= 1²
+ Me
[L'
6+1
(x*)1 + by.l
+ D DS [R
where (1) is the result of completing a square, and (2) again follows from Lemma M.I Hence, we can have uniformly bounded constants π., S > 0 such that
Bl|l x [R) <(1 cxpp)) || - + ou 1 - yy Ça x=112
Incorporating the above estimation into the inner and outer optimization, following similar argument to Theorem in Hong et al. (023), we can reach the following conclusion.
Theorem M.I. Under Assumptions 1 2 and when o4 <o, and BR c1ak 2/3 with constants cD C1 > O, the difference hetween the k-th step of the algorithm and the global optimum of problem defined in equation II x", can be estimated as,
E|I1** -
x01 1191
((1 a:ue)) [B[lIe" - [1]] + TBllly - " (x"))1111 + (ax-1]²
2/3
(23)
Similarly.
Tー II ==
Billy
Blll** (a* 10 +3
Elly" 3n 119 + 3x-
(24)
The consequence of the theorem is that the case of au, B 0. as oo, we know that the two quantities on the left-hand side will diminish to zero, thus the convergence of the algorithm in the sense of mean square error.
38


Reviews:
[{"review_id": "2WZX5o6LtM", "reviewer": "Reviewer_7cFX", "summary": "The work introduces a new reinforcement learning method for online representation learning. The core idea is to divide the function approximator into two parts: one for learning a useful representation and another for learning to predict future returns. Each part is treated as an agent playing a Stackelberg game, maximizing a different objective while sharing the same Q-function predictor. This leads to a bi-level optimization problem, which they address by enforcing a faster learning rate for the encoder than for the predictor. Experiments on various tasks are presented to report the method's performance.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "A. The submission focuses on an important problem of reinforcement learning, which is representation learning.\n\nB. The reading flow is good, and the method is clearly presented.\n\nC. The method is simple to implement as it simply uses two different learning rates for the encoder and the predictor, and updates the encoder and the predictor sequentially.", "weaknesses": "I. Some choices are suboptimal or unjustified.\n\n   a. The choice of minimizing the variance of the temporal difference error is problematic. Indeed, the proposed method faces the double-sampling issue, yet it is not mentioned. This means the follower's objective function is biased. More specifically, $\\mathbb{E}[\\delta_j^2] \\neq \\mathbb{E}[\\delta_j]^2$. Therefore, the presented algorithm will suffer from this issue in stochastic MDPs, where the variance of the temporal difference error exists, even for the optimal Q-function. Importantly, all experiments are made in deterministic environments. I suggest setting the follower's objective function to the temporal-difference error, which is an unbiased estimate. \n\n   b. It is claimed that \"This promotes stability in the learning process, a trait for mitigating the risk of divergence that was first formally demonstrated in foundational work by Baird (1995)\" in Line 218. However, this claim is never justified by an empirical study on the Baird counter-example. I suggest evaluating the method on this counter-example.\n\n   c. The choice of the follower and the leader is only justified by an empirical comparison. Adding a theoretical or an intuitive argument would strengthen the motivation behind this work.\n\nII. The empirical analysis remains weak and can be improved.\n\n   a. The method relies on multiple parallel environments. While this allows for faster training, it is an unrealistic assumption for real-world applications, and when a simulator is available, on-policy methods will be privileged. To fully assess the potential of the presented method, it would be useful to also have experiments in which only one environment is used at a time.\n\n   b. As mentioned in I. a., it is unlikely that the presented method performs well in stochastic environments. Therefore, I suggest evaluating SCORER in stochastic environments, for example, with sticky actions.\n\n   c. Two cofounding factors can be responsible for the performance increase. (i) The learning rate schedules are not fully disclosed. It would be more convincing to compare the proposed approach with the baselines using the same learning rate schedules (the one of the encoder and the one of the predictor). (ii) The gradients are clipped for the proposed approach but not for the baseline. Ablating this choice would help understand the benefit of SCORER.\n\n   d. An analysis of the learning dynamics would provide more insights. For example, reporting the parameter norm and the srank would help better understand the method.\n\n   e. While the interquantile-mean is reported as suggested by [1], the recommended way of reporting the confidence intervals is not followed, and no justification is provided.\n\n   f. In Table 1, the maximum IQM performance is reported. This choice is uncommon and not motivated. \n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" NeurIPS 2021.\n\n### Remarks\n\n1. The font size of every figure can be larger. Replacing the tables with visual plots would facilitate the analysis of the results.\n \n2. Two citations can be added when speaking about the fact that learning appropriate representations helps with sample efficiency: \n\n   [1] Dabney, Will, et al. \"The value-improvement path: Towards better representations for reinforcement learning.\" AAAI 2021.\n\n   [2] Vincent, Théo, et al. \"Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning.\" TMLR 2025.\n\n3. The batch seems to be missing in equation 5.\n\n4. In Figure 4, the left and middle Figures are swapped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UpsVt0GsyJ", "reviewer": "Reviewer_pMYo", "summary": "The paper proposes SCORER - a method that splits deep Q-learning into two separate networks that learn at different speeds to address training instability. The representation network updates faster and minimizes Bellman error variance, while the Q-function updates slower for stability. They test on MinAtar and MiniGrid environments where some tasks show good improvements while others show more modest gains.", "questions": "1. Why were full-scale Atari experiments not included, and do you have any preliminary results on high-dimensional visual domains?\n2. Can you provide an ablation study on different follower-to-leader learning rate ratios (e.g., 2:1, 10:1, 20:1) beyond the 5:1 used in all experiments?\n3. Have you tested a simpler baseline where a standard monolithic network uses higher learning rates for encoder layers without network separation or stop-gradients?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper tackles a real issue in deep Q-learning where representation learning and value learning are tightly coupled, causing training instability.\n- The learning curves show SCORER produces smoother, more stable training compared to baselines. The curves don't have as many sudden drops or erratic behavior.\n- The appendix provides mathematical analysis based on two-timescale stochastic approximation. The convergence guarantees and game-theoretic formulation seem reasonable.\n- Results are strong in some cases. MinAtar Breakout shows roughly 3x improvement over baseline DQN. MiniGrid Four Rooms is particularly impressive - baseline R2D2 completely fails while SCORER achieves 97% success", "weaknesses": "- Experiments only cover 4 simple MinAtar environments and basic gridworld tasks. No full-scale Atari with high-resolution images. This is a problem because representation learning challenges are fundamentally different at scale, and we can't tell if benefits hold with deep convolutional networks.\n- Outside Breakout and Four Rooms, gains are small. Asterix is basically the same as baseline, Freeway has identical final performance (just faster), SpaceInvaders shows about 10-15% improvement. The claim of \"consistently improves final performance\" oversells what the data shows.\n- The method uses 5:1 ratio (follower:leader learning rates) but there's no ablation testing 2:1, 10:1, or other values. This is important - we need to know how sensitive the method is and whether 5:1 is actually optimal.\n- A monolithic network with higher learning rate for encoder layers (without network separation or stop-gradients) would test whether the complex game structure is necessary or if just updating representations faster works. This might not be essential since current ablations show hierarchy matters, but it would be useful to rule out simpler explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TkjJn0MAzH", "reviewer": "Reviewer_HaiV", "summary": "This paper presents a new framing of value-based deep RL methods in terms of a two-player game between a representation player and a controller player. The motivation stems from the apparent instability that emerges in RL when a learner tries to simultaneously learn its representation, and its control policy. The core technical idea is to treat the learner's estimate of the Q-function as the leader, while a perception network is treated as the follower of a Stackelberg game. This framing yields \"Stackelberg Coupled Representation and Reinforcement Learning\" (SCORER), a simple value-based method for deep RL based on the principles of Stackelberg games. The game is solved using a two-level form of gradient descent, which yields a stable relationship between the representation and the value. Critically, this game only involves the exchange of the standard signals of the RL problem (based around MSBE) and does not allow incorporation of extraneous signals (such as intrinsic motivation, entropy bonuses, and so on). This methodological choice allows the work to focus purely on the game dynamics in a simple setting. The paper's primary evidence to support its central claims comes from a broad experimental study contrasting the performance of SCORER with standard value-based methods (DQN, DDQN, and so on) across a variety of standard tasks.", "questions": "My main question relating to my evaluation of the work is as follows:\n\nMQ.1: For the purposes of understanding, how do you see SCORER in relation to other approaches to deep RL that decompose the monolith into a variety of subcomponents? Can you comment on the merits of framing this as perceiver/actor in a game, rather than something else (such as the Anand and Precup approach)?\n\nSeparately, I have a one other question (OQ.X), though it is less critical to my overall appraisal:\n\nOQ.1: How do you foresee SCORER extending to another similar algorithmic templates, such as Actor-Critic, or even cases when value is not learned at all but perhaps just a policy?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is a very clean paper. The scope is well defined, the question is well-posed, all of the needed ingredients are self-contained within the paper, and the solution (two-level optimization of a Stackelberg game) is tidy. In other words, the paper does exactly what it sets out to do in the intro.\n- The experimental study is well designed, with appropriately chosen baselines, a variety of environments, and suitable ablations to explore various design choices (such as which player should be the leader vs. follower in Fig 4).\n- The writing is clear and does a good job of motivating the various design choices. I especially appreciated the discussion around whether extraneous other signals should be incorporated into the study such as standard self-supervised objectives. I found the question, the main method, and the findings to be well-motivated and communicated effectively.\n- The convergence result in the appendix adds a firm theoretical support to the approach.\n- Lastly, the idea at the heart of the paper is relatively simple: if we decouple algorithmic components, we can treat them as players in a game, and solve the game.", "weaknesses": "- The general premise of mitigating challenges of deep RL through decomposing the monolithic policy network or value network into different components has been studied extensively. The proposal to treat a two-decomposition sub-system as engaged in a Stackelberg game is, to my knowledge, new, though I do believe a deeper understanding of why this specific decomposition is valuable would strengthen the work. As two concrete examples:\n\n1. In Anand and Precup (2023), they propose to decompose the estimate of the value function into a \"permanent\" and \"transient\" component. These two components yields two separate networks, one for learning the permanent and one for the transient. They key difference between the two is that they are updated at different schedules. Then, the overall value function is determined by taking a combination of the two. I believe there is a heavy conceptual overlap in this approach and SCORER, and I wonder: if someone were to want to deploy a value-based method for RL, when should they use SCORER as approach to the approach of Anand and Precup?\n\n2. Second, the Actor-Critic architecture is naturally decomposed into separate pieces to enhance stability. Indeed, a recent paper by Garcin et al (2025) has explicitly looked at the learned representations in each of the Actor and Critic, and found that the two networks tend to specialize their representation differently. First, I believe applying this kind of analysis to SCORER could be beneficial to shed further light on what specifically is being learned by the representation and policy in each of the ablations (and in the general setup). Second, it is so far again unclear what the precise benefit of this proposed decomposition is compared to something like Actor-Critic. I see this is noted in the discussion (applying SCORER to AC algorithms), though I believe this is an important point to understand about the space.\n\n- Otherwise, no obvious weaknesses stand out. This is a clean paper with a tight focus that delivers on what it sets out to do.\n\n\nSome typos and writing suggestions:\n\n- Some repeated text at line 166 or so: \"The defining characteristic is the leader’s ability to anticipate and influence the follower’s decision by committing to a strategy first. The defining characteristic is the leader’s ability to anticipate the follower’s best response and commit to a strategy that steers the game toward a favorable equilibrium.\".\n\n- Convergence analysis in Appendix J is really nice. If you can find space, I would love to see this in the main text, even it is just a colloquial theorem statement with the assumptions and proof in the appendix. On that note, I would suggest stating the convergence result as a mathematical theorem or proposition, even if it is in the appendix, to give you a chance to succinctly assert the result with all needed quantifiers and variables in one place.\n\nReferences:\n- Anand, N., & Precup, D. Prediction and control in continual reinforcement learning. NeurIPS 2023.\n- Garcin, S., McInroe, T., Castro, P. S., Panangaden, P., Lucas, C. G., Abel, D., & Albrecht, S. V. Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning. ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qao58YFbtm", "reviewer": "Reviewer_b2sk", "summary": "This paper introduces Stackelberg Coupled Representation and Reinforcement Learning, a novel value-based RL framework inspired by game theory. The key insight is to decouple representation and value learning into a hierarchical leader-follower game: the Q-function (control network) is the leader updated on a slow timescale, while the perception network (state encoder) is the follower learning rapidly to minimize Bellman error variance under the leader’s strategy. This two-timescale design aims to solve the instability issue (deadly triad) seen in monolithic off-policy deep Q-learning, improving sample efficiency and performance without increased model complexity. Experiments on MinAtar, Atari, and MiniGrid show consistent, substantial improvements over standard DQN and its variants.", "questions": "1. Do you consider DQN and Dueling DQN as good baselines and the results naturally generalize to strong baselines such as Rainbow? \n\n2. Can you add studies or discussions on actor-critic methods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated by the Stackelberg game theory to reformulate the RL optimization problem, separating learning dynamics between representation and value estimation.\n\n2. The role assignments (control as leader, perception as follower) and the choice of Bellman error variance for the follower are both theoretically justified and empirically validated.", "weaknesses": "1. While computational cost is minimal, the framework introduces new algorithmic complexity (two-level optimization, choice of timescale schedules) that requires parameter tuning.\n\n2. The method is centered entirely on discrete value-based methods. Extension to actor-critic or continuous-control algorithms is absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)