Stackelberg Coupling of Online Representation Learning and Reinforcement Learning

## Abstract

Abstract Deep Q-learning jointly learns representations and values within monolithic networks, promising beneficial co-adaptation between features and value estimates. Although this architecture has attained substantial success, the coupling between representation and value learning creates instability as representations must constantly adapt to non-stationary value targets, while value estimates depend on these shifting representations. This is compounded by high variance in bootstrapped targets, which causes bias in value estimation in off-policy methods. We introduce Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework for value-based RL that views representation and Q-learning as two strategic agents in a hierarchical game. SCORER models the Q-function as the leader, which commits to its strategy by updating less frequently, while the perception network (encoder) acts as the follower, adapting more frequently to learn representations that minimize Bellman error variance given the leader’s committed strategy. Through this division of labor, the Q-function minimizes MSBE while perception minimizes its variance, thereby reducing bias accordingly, with asymmetric updates allowing stable co-adaptation, unlike simultaneous parameter updates in monolithic solutions. Our proposed SCORER framework leads to a bi-level optimization problem whose solution is approximated by a two-timescale algorithm that creates an asymmetric learning dynamic between the two players. Extensive experiments on DQN and its variants demonstrate that gains stem from algorithmic insight rather than model complexity.

## 1 Introduction

Deep Reinforcement Learning (RL) has achieved outstanding success, exemplified by the super-human performance of Deep Q-Networks (DQN) in complex environments (Mnih et al., 2015). The dominant approach for value-based methods involves training both the representation and the value function within a single neural network, encouraging a beneficial co-adaptation between the features learned and value estimates, leading to significant breakthroughs (Hessel et al., 2018; Kapturowski et al., 2018); however, this monolithic design is subject to an instability known as the deadly triad (Sutton & Barto, 2018; Van Hasselt et al., 2018). This instability results from the challenging interaction between function approximation using neural networks, bootstrapping, and off-policy learning from data stored in an experience replay buffer, where errors in the current value function are bootstrapped into its own learning targets, leading to a risk of unbounded divergence, an issue first formally demonstrated in Baird (1995). This risk remains a critical practical concern for modern agents; recent work has empirically linked this core instability to phenomena like representation collapse and catastrophic learning failures in deep Q-learning (Lyle et al., 2022).

A prominent line of work seeks to mitigate this instability by directly enhancing the quality of the learned representations. This is often accomplished by extending the main RL objective with auxiliary losses, borrowing from advances in self-supervised and representation learning (Jaderberg et al., 2017; Schwarzer et al., 2021; Laskin et al., 2020b). While this strategy can improve performance, utilizing a shared network to fulfill multiple objectives can introduce the challenge of conflicting gradients (Yu et al., 2020), wherein the gradient from an auxiliary loss opposes that of the primary value-learning objective. This interference presents a fundamental trade-off, as the effort to stabilize representations introduces a new obstacle to learning the value function itself.

This trade-off suggests that simply augmenting the monolithic optimization is an insufficient solution, motivating a fundamental restructuring of the optimization problem itself. We therefore propose Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework that recasts the interaction between perception and control as a hierarchical Stackelberg game (Lambertini, 2018). In this game, the Q-function acts as a slow-updating leader, providing the stable learning target necessary for a fast-adapting perception network (the follower) to learn a robust representations by minimizing Bellman error variance. This objective incentivizes the creation of representations that can handle the noisy targets encountered during exploration, preventing the representation collapse that can afflict monolithic agents (Lyle et al., 2022). This division of labor resolves the underlying trade-off, using temporal separation to create the stability required for this co-optimization.

We create SCORER’s game-theoretic dynamics through a practical and computationally efficient algorithm. The Stackelberg equilibrium is approximated using two-timescale gradient descent, where the follower can track a best response to the leader’s slowly evolving strategy. Implementing SCORER is simple, as the hierarchical coupling is achieved by assigning the two-players distinct decaying learning rates that satisfy the conditions for two-timescale convergence, with no alterations to the underlying network architectures. This two-timescale dynamic provides a principled mechanism for achieving stable, coordinated adaptation, an approach supported by established theory in stochastic approximation (Borkar, 1997; Fiez et al., 2020).

Figure: Figure 1: SCORER framework. (Left) Overall agent-environment interaction loop. Internally, the agent comprises a perception network (Follower, $f_{\phi}$) and a control network (Leader, $Q_{\theta}$) that interact via Stackelberg game dynamics ($U_{F},U_{L}$ representing their utility functions). The perception network produces features $z=f_{\phi}(s)$ used by the control network. (Right) Details the Stackelberg interaction within the agent.
Refer to caption: x1.png

Figure [1](https://arxiv.org/html/2508.07452v2#S1.F1) provides a high-level overview of the SCORER framework. Our code is available at [https://anonymous.4open.science/r/SCORER-530](https://anonymous.4open.science/r/SCORER-530). In summary, our contributions are three-fold: 1) We introduce the SCORER framework, a novel game-theoretic formulation that recasts the interaction between perception and control to address the core instabilities of off-policy value-based learning directly. 2) We develop a practical and efficient two-timescale algorithm requiring only update frequency modifications, stabilizing the co-adaptation of representations and value functions. 3) We provide extensive empirical validation, demonstrating that SCORER consistently improves the sample efficiency and final performance of various off-policy Q-learning agents across multiple benchmarks.

## 2 Related Work

Representation learning has been a pivotal topic in RL, even before the advent of deep neural networks. Early stage efforts concentrated on value function approximation (Thrun & Schwartz, 1993; Tsitsiklis & Roy, 1997) and associated basis function selections (Li & Zhu, 2019; Geramifard et al., 2013). Entering the age of deep learning, RL harnesses the representation power of neural networks and becomes capable of solving high-dimensional complex tasks with multi-modal inputs, such as texts (Li et al., 2016), images (Mnih et al., 2015), and multi-modal sensor data (Liu et al., 2017; Yin et al., 2024; Li et al., 2025). The confluence of deep representation learning and RL leads to the vibrant research field of deep RL.

Depending on the purpose, representation learning methods in deep RL can be classified into two major categories: those aimed at facilitating training and those designed to enhance testing performance. In the training phase, learning appropriate representations helps with 1) dimension reduction: extracting low-dimensional features from high-dimensional inputs, making the RL tasks tractable, a typical example of which is vision-based control tasks (Lesort et al., 2018; Li et al., 2023); 2) sample efficiency: capturing underlying environment dynamics (Subramanian et al., 2022; Lee et al., 2020), which reduce the number of training episodes and foster faster convergence; and 3) oriented exploration: directing future exploration strategies based on past experiences and intrinsic motivation (Kulkarni et al., 2016), particularly in reward-free and sparse-reward settings (Pathak et al., 2017; Hazan et al., 2019). Some other considerations include training stability (Greydanus et al., 2018) and explainability (Dazeley et al., 2023; Li et al., 2023). Regarding testing improvement, previous work focuses on the transferability of representations for improved generalization to downstream or similar tasks (Agarwal et al., 2021; Träuble et al., 2021).

Our proposed Stackelberg framework aims to improve sample efficiency and stability in the training stage. Weighing the three mainstream representation learning methods—supervised, unsupervised, and self-supervised learning—this work opts for the unsupervised approach based on the Mean Squared Bellman Error (MSBE). Supervised learning requires additional labeling to inform the agent of the quality of learned representations Wang et al. (2024), which is often observed in physical control with partial observability, and the representations need to encode structural information of the environment, such as depth maps from RGB images Mirowski et al. (2017) and physics principles underlying the wireless sensing inputs Yin et al. (2024); Li et al. (2025). Since our goal is to explore the benefit of internal game-theoretic coupling between representation and reinforcement learning, we do not allow any external learning signals (other than task rewards) to interfere with the strategic interactions between the two learning processes.

A similar argument also explains why we do not consider the incorporation of self-supervised learning, such as contrastive learning (Liu et al., 2021; Stooke et al., 2021; Banino et al., 2022), temporal dynamics and state prediction (Subramanian et al., 2022; Schwarzer et al., 2021), and observation reconstruction (Lange & Riedmiller, 2010; Lesort et al., 2018), and most unsupervised learning, including mutual information (Anand et al., 2019), entropy maximization (Hazan et al., 2019), data augmentation (Yarats et al., 2022; Laskin et al., 2020a), and bisimulation (Zhang et al., 2021; Agarwal et al., 2021). Our proposed unsupervised representation learning solely relies on MSBE and its sample variance without auxiliary tasks or additional learning signals.

## 3 Preliminary

##### Reinforcement Learning and Q-Learning.

We consider an agent interacting with an environment formulated as a Markov Decision Process (MDP), defined by the tuple $(\mathcal{S},\mathcal{A},P,R,\gamma)$. Here, $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P(s^{\prime}|s,a)$ is the state transition probability, $R(s,a)$ is the reward, and $\gamma\in[0,1)$ is the discount factor. At each step, the environment transitions from state $s$ to $s^{\prime}$ and provides a reward $r$; if $s^{\prime}$ is a terminal state, a done signal $d=1$ is captured, otherwise $d=0$.
The agent’s goal is to learn a policy $\pi(a|s)$ that maps states to actions (or distributions over actions) to maximize the expected sum of discounted future rewards from a given state $s_{t}$: $\mathbb{E}_{\pi}\left[\sum_{i=0}^{\infty}\gamma^{i}r_{t+i}|s_{t}\right]$, where $r_{t+i}$ is the reward received $i$ steps after time $t$.

In value-based Reinforcement Learning, the optimal action-value function $Q^{*}(s,a)$ represents the expected return from taking action $a$ in state $s$ and following the optimal policy thereafter. It satisfies the Bellman optimality equation: $Q^{*}(s,a)=\mathbb{E}_{s^{\prime}}[R(s,a)+\gamma\max_{a^{\prime}}Q^{*}(s^{\prime},a^{\prime})]$. Deep Q-Networks (DQN) approximate $Q^{*}$ using neural networks trained by minimizing the Mean Squared Bellman Error (MSBE): $\mathcal{L}(\theta)=\mathbb{E}_{(s,a,r,s^{\prime},d)\sim\mathcal{D}}\left[\left(Y-Q_{\theta}(s,a)\right)^{2}\right]$.

To stabilize training, the target value $Y$ is calculated using a separate, periodically updated target network ($Q_{\theta_{\text{target}}}$) that prevents the learning target from fluctuating at every step. For a given transition, the target is defined as $Y=r+\gamma(1-d)\max_{a^{\prime}}Q_{\theta_{\text{target}}}(s^{\prime},a^{\prime})$.

##### Stackelberg Game.

A Stackelberg game is a hierarchical game involving two types of players: a *leader* and one or more *followers* (Osborne, 2004). The leader moves first, choosing an action $a_{L}$ from its action set $\mathcal{A}_{L}$. The follower, equipped with the full information on the leader’s action $a_{L}$, chooses its own action $a_{F}$ from its action set $\mathcal{A}_{F}$ to maximize its own utility $u_{F}(a_{L},a_{F})$, given $a_{L}$. The basic equilibrium relationship for the Stackelberg game is reflected by the following bi-level optimization problem: $\max_{a_{L}}U_{L}(a_{L},a_{F}^{*}(a_{L})),\text{ s.t. }a_{F}^{*}(a_{L})\in\arg\max_{a_{F}}U_{F}(a_{L},a_{F})$, with $U_{L}(a_{L},a_{F})$ being the utility of the leader.

The defining characteristic is the leader’s ability to anticipate and influence the follower’s decision by committing to a strategy first. The defining characteristic is the leader’s ability to anticipate the follower’s best response and commit to a strategy that steers the game toward a favorable equilibrium. This hierarchical dynamic inspires our proposed SCORER framework.

## 4 Stackelberg Coupled Representation and Reinforcement Learning

In this section, we formalize the game-theoretic principles outlined previously. We model the value-based interaction between perception and control as a hierarchical Stackelberg game to resolve the trade-off between representation stability and value function learning. This framework provides a clear division of labor, implemented through two distinct network components: a perception network that acts as the game’s follower, and a control network that acts as the leader.

### 4.1 The Stackelberg Game Formulation

The Leader: Control Network. The Control network $Q_{\theta}$ assumes the role of the Stackelberg leader. Its objective is to learn an optimal action-value function $Q_{\theta}(z,a)$, mapping state representations $z$ and actions $a$ to expected cumulative discounted future rewards. As the leader, the control network learns on a slower timescale, providing a stable target for the follower. The leader’s objective function, $\mathcal{L}_{\text{leader}}$, is the MSBE. Let $\mathcal{D}$ be the agent’s data source. This source of can be a replay buffer, as in traditional replay buffer-based methods, or a collection of online trajectories, as in PQN (Gallici et al., 2025). The leader aims to solve:

$$ $\displaystyle\min_{\theta}\ \mathcal{L}_{\text{leader}}(Q_{\theta},f_{\phi^{*}(\theta)})\triangleq\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{B}\subset\mathcal{D}}\Big[(Y-Q_{\theta}(f_{\phi^{*}(\theta)}(s),a))^{2}\Big],$ (1) $$

where $\mathcal{B}$ is a batch of transitions sampled from the data source $\mathcal{D}$, and $Y$ is the corresponding Bellman target value. Here, $\phi^{*}(\theta)$ represents the parameters the follower would ideally converge to given the leader’s choice of $\theta$. The hierarchical structure is established by having the control network commit to its parameters on a slower timescale, while the perception network responds on a faster timescale.

The Follower: Perception Network. The Perception network $f_{\phi}$ acts as the Stackelberg follower. Its function is to learn an encoder that maps raw observations $s$ to a latent representation $z=f_{\phi}(s)$. Given the control network’s committed strategy (with $\theta$ treated as fixed via stop-gradient during the follower’s update), the follower learns on a faster timescale, allowing it to compute an effective best response to the leader’s slowly changing strategy. The follower seeks parameters $\phi$ that minimize its own loss function, $\mathcal{L}_{\text{follower}}$.

SCORER is agnostic to the specific choice of follower objective, allowing for different formulations depending on the desired properties of the learned representations. We investigate multiple objectives for $\mathcal{L}_{\text{follower}}$, including directly minimizing the MSBE and minimizing the variance of Bellman errors. Through extensive empirical evaluation (detailed in Section [5](https://arxiv.org/html/2508.07452v2#S5) and Appendix [I](https://arxiv.org/html/2508.07452v2#A9)), we find that minimizing the variance of Bellman errors yields superior performance, which echoes the observation that MSBE is biased due to the high variance (Baird, 1995; Wu et al., 2021). For a batch $B$, let $\delta_{j}(\phi,\theta)=Y_{j}-Q_{\theta}(f_{\phi}(s_{j}),a_{j})$ be the Bellman error for transition $j$, where $Y_{j}$ is the target value. The follower’s objective becomes:

$$ $\phi^{*}(\theta)\in\arg\min_{\phi}\ \mathcal{L}_{\text{follower}}(f_{\phi},Q_{\theta})\triangleq\text{Var}_{j\in B}[\delta_{j}(\phi,\theta)],$ (2) $$

where $\text{Var}_{j\in B}[\delta_{j}]=\frac{1}{|B|}\sum_{j\in B}\delta_{j}^{2}-\left(\frac{1}{|B|}\sum_{j\in B}\delta_{j}\right)^{2}$ is the sample variance of the Bellman errors over the batch.

The motivation for this variance-minimization objective is to counteract the pathologies of the deadly triad directly. By focusing on the consistency of the Bellman errors across a batch, the follower learns representations that are more consistent across the batch, making them robust to the noisy targets inherent in TD learning. This promotes stability in the learning process, a trait for mitigating the risk of divergence that was first formally demonstrated in foundational work by Baird (1995). This objective transforms the follower into an active stabilizing agent, as it is incentivized to find representations that make the leader’s value predictions more uniform across the data distribution.

### 4.2 Approximating the Stackelberg Equilibrium via Two-Timescale Gradient Descent

The Stackelberg game described in Section [4.1](https://arxiv.org/html/2508.07452v2#S4.SS1) translates to a bi-level optimization problem given by equation [3](https://arxiv.org/html/2508.07452v2#S4.E3). The control network aims to optimize its objective $\mathcal{L}_{\text{leader}}$ by choosing its parameters $\theta$, anticipating the follower’s optimal response $\phi^{*}(\theta)$:

$$ $\displaystyle\min_{\theta}\quad$ $\displaystyle\mathcal{L}_{\text{leader}}(Q_{\theta},f_{\phi^{*}(\theta)})\quad\text{subject to}\quad\phi^{*}(\theta)\in\arg\min_{\phi}\mathcal{L}_{\text{follower}}(f_{\phi},Q_{\theta}).$ (3) $$

Solving such bi-level problems directly in high-dimensional, non-convex settings typical of deep reinforcement learning is challenging, with the primary difficulty arising from computing the gradient of the leader’s objective with respect to its parameters $\theta$. The leader’s objective $\mathcal{L}_{\text{leader}}(Q_{\theta},f_{\phi^{*}(\theta)})$ depends on $\theta$ both directly (through $Q_{\theta}$) and indirectly (through the follower’s response $\phi^{*}(\theta)$). Applying the chain rule yields

$$ $\displaystyle\nabla_{\theta}\mathcal{L}_{\text{leader}}=\frac{\partial\mathcal{L}_{\text{leader}}}{\partial Q_{\theta}}\nabla_{\theta}Q_{\theta}+\frac{\partial\mathcal{L}_{\text{leader}}}{\partial f_{\phi^{*}}}\nabla_{\theta}f_{\phi^{*}(\theta)},$ (4) $$

where the first term is the direct gradient through $Q_{\theta}$, and the second term $\nabla_{\theta}f_{\phi^{*}(\theta)}$ captures the indirect effect through the follower’s response.

Computing the indirect gradient term is the central challenge of bi-level optimization. It requires the implicit derivative $\frac{d\phi^{*}(\theta)}{d\theta}$, which contains second-order information about the follower’s optimization landscape. While this term is computationally prohibitive to calculate directly in deep learning contexts (Chen et al., 2023), a body of recent work has developed first-order methods to approximate it (Li et al., 2024; Liu et al., 2022; Li et al., 2022; Hong et al., 2023). However, as our focus is on the game-theoretic coupling, we adopt a more direct solution: we approximate the leader’s gradient using the first-order term and converge through a principled timescale separation between the players.

This separation ensures that the faster-learning follower can effectively track the best response to the slower-learning leader. In our framework, this asymmetry is achieved using time-dependent learning rate schedules, $\alpha_{\phi,k}$ and $\alpha_{\theta,k}$, that satisfy the key conditions of two-timescale stochastic approximation (TTSA) theory (Hong et al., 2023). Specifically, both learning rates must decay over training steps $k$, while their ratio must also approach zero, i.e., $\lim_{k\to\infty}\alpha_{\theta,k}/\alpha_{\phi,k}=0$. This guarantees that the leader learns on a sufficiently slower timescale than the follower, allowing the system to converge to a first-order stationary point of the game, as detailed in Algorithm [1](https://arxiv.org/html/2508.07452v2#alg1).
The practical update rules for the follower and leader are thus given by:

$$ $\displaystyle\phi_{k+1}$ $\displaystyle\leftarrow\phi_{k}-\alpha_{\phi,k}\nabla_{\phi}\mathcal{L}_{\text{follower}}(f_{\phi_{k}},\overline{\theta_{k}}),$ (5) $\displaystyle\theta_{k+1}$ $\displaystyle\leftarrow\theta_{k}-\alpha_{\theta,k}\nabla_{\theta}\mathcal{L}_{\text{leader}}(\theta_{k};B_{\text{leader}},Y,\overline{\phi_{k+1}}),$ (6) $$

where $\overline{\cdot}$ (e.g., $\overline{\theta_{k}}$) denotes a stop-gradient operation that treats the variable as a constant with respect to the gradient calculation, guaranteeing that no gradients flow from one player’s update to the other’s parameters. The follower first updates to $\phi_{k+1}$ based on the fixed leader parameters $\theta_{k}$. The leader then immediately updates to $\theta_{k+1}$ using the follower’s new state $\phi_{k+1}$ as a fixed input.

Our approach of having the leader optimize based on the follower’s recent state, rather than its fully converged response, is a common and practical technique for approximating bi-level optimization (Fiez et al., 2020; Petrulionytė et al., 2024).
Under standard smoothness assumptions for two-timescale stochastic approximation, including Lipschitz continuous gradients and a weaker convexity condition known as the Restricted Secant Inequality (RSI) (Karimi et al., 2016), our method converges to a first-order stationary point of the game Li et al. (2024). This convergence is general and holds for both the MSBE and Bellman Error Variance objectives, as both measure temporal difference consistency.

We provide a detailed convergence analysis in Appendix [J](https://arxiv.org/html/2508.07452v2#A10), which explicitly characterizes the convergence rates in terms of the size of neural networks, as well as the learning rates schedules.

### 4.3 Two-Timescale Stackelberg Coupled Learning (SCORER)

The implementation details of SCORER for replay-based agents are detailed in Algorithm [1](https://arxiv.org/html/2508.07452v2#alg1). Both the perception network (follower) and the control network (leader) are updated at the same interval, $T_{\text{Update}}$, where the timescale separation that approximates the Stackelberg dynamic is achieved by setting the two players distinct, time-dependent learning rate schedules, $\alpha_{\phi}$ and $\alpha_{\theta}$. In line with two-timescale convergence theory, the schedules decay over the course of training and are chosen such that the follower’s learning rate, $\alpha_{\phi,k}$, remains significantly larger than the leader’s, $\alpha_{\theta,k}$, making sure that their ratio diminishes over time. We model the two-player game by stopping the gradient flow during backpropagation within a single update step, so the follower first updates its parameters to $\phi_{k+1}$ based on the fixed leader parameters $\theta_{k}$, and subsequently, the leader updates its parameters to $\theta_{k+1}$ using the follower’s newly updated state $\phi_{k+1}$, completing one step of the hierarchical game.

Figure: Algorithm 1 SCORER (Stackelberg Coupled Representation Learning) for Deep Q-Learning Variants

##### SCORER’s Generality

While Algorithm [1](https://arxiv.org/html/2508.07452v2#alg1) details the implementation for replay-based agents, the SCORER principle of a two-timescale, leader-follower dynamic is more general. As shown in our Section [5](https://arxiv.org/html/2508.07452v2#S5), it can be seamlessly adapted to modern agents like PQN that do not use a replay buffer.

## 5 Experiments and Results

In this section, we empirically evaluate the performance of SCORER when integrated with several established value-based RL algorithms: DQN (Mnih et al., 2015), Double DQN (DDQN) (Van Hasselt et al., 2016), Dueling DQN (Wang et al., 2016), Dueling Double DQN, and R2D2 (Kapturowski et al., 2018) for partially observable environments. We also compare against the Parallelized Q-Network (PQN) (Gallici et al., 2025), a recent, high-performance off-policy value-based method that forgoes a replay buffer. Our primary goal is to assess whether the proposed Stackelberg coupling (Algoritm [1](https://arxiv.org/html/2508.07452v2#alg1)), improves learning compared to standard end-to-end training of these baseline methods without increasing computational overhead or architectural size.

All experiments are conducted using an efficient JAX-based framework inspired by PureJaxRL, and CleanRL algorithms (Lu et al., 2022; Huang et al., 2022). To provide a fair comparison, the underlying network architectures and hyperparameter budgets are kept identical between SCORER variants and their respective baselines. The speed comparison in Table [1](https://arxiv.org/html/2508.07452v2#S5.T1) shows that the SCORER framework achieves these gains with imperceptible computational overhead, running at nearly identical speeds to the baseline algorithms. Consequently, any observed performance gains are directly attributable to the dynamics introduced by SCORER. A detailed description of experimental setups and hyperparameters is in Appendix [C](https://arxiv.org/html/2508.07452v2#A3).

### 5.1 Performance on Gymnax Environments

For the first evaluation, we use SCORER across the MinAtar suite (Young & Tian, 2019) through Gymnax (Lange, 2022). Learning curves comparing the SCORER-enhanced variants against their respective baselines are displayed in Figure [2](https://arxiv.org/html/2508.07452v2#S5.F2) over Asterix, Breakout, Freeway, and SpaceInvaders. Table [1](https://arxiv.org/html/2508.07452v2#S5.T1) complements these curves by summarizing the maximum average returns achieved.

Figure: Figure 2: Learning curves on MinAtar environments comparing SCORER variants against baselines. SCORER variants generally demonstrate improved sample efficiency and performance across several algorithm-environment combinations.
Refer to caption: figures/full_results_with_pqn.png

The results demonstrate that SCORER provides a consistent and significant performance benefit across all tested base algorithms. For the classic DQN family, the SCORER variants show dramatic gains in sample efficiency and final performance. In Breakout, for example, SCORER triples the final score of a standard DQN agent. More importantly, SCORER allows these replay-based methods to become competitive with, and in some cases (e.g., SpaceInvaders), superior to the state-of-the-arts methods like PQN baseline.

**Table 1: Maximum Interquartile Mean (IQM) return ($\pm$ 95% CI over 30 seeds) on MinAtar. Highlighted values indicate the best-performing variant within each base algorithm family per environment. Highlighted performances are those not statistically worse than the highest, based on Welch’s t-test (p < 0.05).**
|  | Variant | Asterix | Breakout | Freeway | SpaceInvaders | Speed Comparison |
| --- | --- | --- | --- | --- | --- | --- |
| DQN | Baseline | 54.95 $\pm$ 0.92 | 19.16 $\pm$ 1.50 | 62.70 $\pm$ 0.17 | 127.78 $\pm$ 0.55 | 1.00x |
| SCORER | 54.78 $\pm$ 0.42 | 65.69 $\pm$ 2.52 | 63.03 $\pm$ 0.08 | 148.71 $\pm$ 2.02 | 0.99x |  |
| DDQN | Baseline | 54.62 $\pm$ 1.05 | 44.78 $\pm$ 4.07 | 62.54 $\pm$ 0.13 | 129.06 $\pm$ 2.25 | 1.00x |
| SCORER | 54.25 $\pm$ 0.39 | 67.06 $\pm$ 1.56 | 62.89 $\pm$ 0.06 | 147.47 $\pm$ 2.93 | 1.00x |  |
| DuelingDQN | Baseline | 45.85 $\pm$ 6.21 | 29.92 $\pm$ 3.21 | 62.22 $\pm$ 0.14 | 127.72 $\pm$ 0.45 | 1.00x |
| SCORER | 53.69 $\pm$ 0.54 | 61.89 $\pm$ 1.26 | 62.42 $\pm$ 0.09 | 140.35 $\pm$ 1.75 | 1.01x |  |
| DuelingDDQN | Baseline | 50.75 $\pm$ 1.83 | 31.70 $\pm$ 4.72 | 61.98 $\pm$ 0.21 | 130.54 $\pm$ 1.40 | 1.00x |
| SCORER | 53.80 $\pm$ 0.37 | 63.18 $\pm$ 1.66 | 62.36 $\pm$ 0.07 | 143.80 $\pm$ 1.17 | 1.00x |  |
| PQN | Baseline | 51.05 $\pm$ 0.47 | 72.49 $\pm$ 0.64 | 63.52 $\pm$ 0.26 | 142.15 $\pm$ 1.56 | 1.00x |
| SCORER | 51.57 $\pm$ 0.76 | 73.50 $\pm$ 0.31 | 63.55 $\pm$ 0.27 | 145.14 $\pm$ 0.72 | 1.00x |  |

The final rows of Table [1](https://arxiv.org/html/2508.07452v2#S5.T1) show that SCORER’s benefits are not limited to replay-based agents. When applied to PQN, our framework provides further statistically significant performance gains. This demonstrates that SCORER is not only restricted to replay buffer-based agents but represents an advancement in the co-adaptation of representation and control in value-based RL. Additional results on classic control benchmarks in Appendix [G](https://arxiv.org/html/2508.07452v2#A7) further corroborate this trend.

### 5.2 Performance on Minigrid Environments

Figure: Figure 3: Time-to-threshold analysis showing mean timesteps (in millions) $\pm$ 95% confidence interval to reach 99% maximum performance over 30 seeds. SR (%) is the success rate of runs reaching the threshold.

We evaluate SCORER on MinGrid (Chevalier-Boisvert et al., 2023), a suite of gridworld environments testing navigation, object interaction, and dynamic obstacle avoidance, accessed through Navix (Pignatelli et al., 2024). As these environments are partially observable, we compare R2D2 and its SCORER R2D2 counterpart. Table [3](https://arxiv.org/html/2508.07452v2#S5.F3) shows time-to-threshold analysis measuring timesteps to reach 99% maximum performance within one million timesteps. SCORER consistently outperforms R2D2, achieving 25-50% faster convergence across all environments. The benefit of the Stackelberg coupling is particularly pronounced in tasks requiring adaptation and exploration. In Dynamic Obstacles, SCORER improves the success rate by 10 points over the baseline, and in the Four Rooms environment, SCORER achieves a 97% success rate while the baseline R2D2 completely fails to solve the task. Full learning curves in Appendix [H](https://arxiv.org/html/2508.07452v2#A8).

### 5.3 Dissecting the SCORER Framework

To validate the core design of SCORER, we perform a series of ablation studies investigating on the MinAtar suite. We investigate our three central design choices: the follower’s objective, the assignment of Stackelberg roles, and the hierarchical coupling dynamic. The main results are presented here as aggregated Interquartile Mean (IQM) scores, while detailed per-environment learning curves for these studies can be found in Appendix [I](https://arxiv.org/html/2508.07452v2#A9).

Figure: Figure 4: Ablation studies on SCORER’s core components on MinAtar. (Left) Follower’s Objective: Bellman Error (BE) Variance is superior to MSBE. (Center) Stackelberg Roles: The standard SCORER hierarchy outperforms the inverted role configuration. (Right) Coupling Dynamic: SCORER’s hierarchical coupling is critical for performance, substantially outperforming a synchronous baseline on Breakout.
Refer to caption: figures/ablation/ablation_full.png

##### Follower’s objective:

Figure [4](https://arxiv.org/html/2508.07452v2#S5.F4) (left) shows that using a Bellman Error Variance objective for the follower substantially outperforms a variant where the follower minimizes MSBE. This confirms that an explicit stabilization objective is critical for SCORER’s performance, providing a clear advantage over a redundant performance objective.

##### Stackelberg roles:

We test our role assignment by inverting the hierarchy, making perception the leader and control the follower. Figure [4](https://arxiv.org/html/2508.07452v2#S5.F4) (center) shows that this inversion leads to a collapse in performance, falling well below the monolithic baseline. This result strongly supports our claim that the control objective must lead the representation learning process.

##### Hierarchical vs. Synchronous Coupling:

Finally, we study whether SCORER’s benefits stem from its full hierarchical dynamic or merely from the architectural separation of networks. For this, we design a baseline called Synchronous Coupling, where distinct perception ($f_{\phi}$) and control ($Q_{\theta}$) networks are updated at the same frequency, with both minimizing MSBE. Stop-gradients operations ensure the updates are independent ($\nabla_{\phi}\mathcal{L}_{\text{MSBE}}(f_{\phi},\overline{Q_{\theta}})$ and $\nabla_{\theta}\mathcal{L}_{\text{MSBE}}(\overline{f_{\phi}},Q_{\theta})$), removing any strategic interaction. This setup cleanly isolates the effect of using two networks from the timescale-based hierarchy of the Stackelberg game.

The results present a clear picture of SCORER’s effect. While simply separating the networks into a synchronous coupling provides a marginal benefit (not statistically significant), introducing the Stackelberg hierarchy offers a substantial leap in performance. This gain is further boosted by the choice of the follower’s objective, with the Bellman error variance providing a consistent and statistically significant advantage over the MSBE objective. Ultimately, the ablations prove that all three design choices (the hierarchical structure, the specific stabilization objective, and the correct assignment of roles) are critical and synergistic components of the SCORER framework.

## 6 Conclusion

This work revisited and addressed the foundational instability that arises from the tight coupling of representation and control in monolithic value deep Q-learning agents. We present the Stackelberg Coupled Representation and Reinforcement Learning (SCORER), a framework that reframes this dynamic as a hierarchical Stackelberg game. We model the control network ($Q_{\theta}$) as the leader and the perception network ($f_{\phi}$) as the follower. This game-theoretic interaction is achieved through a practical two-timescale algorithm where the leader’s slower learning rate delivers a stable target, enabling the follower to learn good representations by minimizing Bellman error variance. SCORER consistently exhibited improvements in sample efficiency and/or stabilized returns of a wide range of replay-based and online Q-learning agents, validating the efficacy of its design.

Despite our promising results, some limitations provide clear directions for future work. The current work focused on value-based methods in discrete action spaces, so extending the SCORER principle to actor-critic algorithms for continuous control is an immediate next step. Additionally, our theoretical analysis, which provides convergence guarantees for the two-timescale approximation, could be extended to explore the algorithm’s sample complexity, specifically to mathematically characterize how the follower’s stabilization objective reduces the number of samples required to learn a high-quality policy. Finally, investigating alternative follower objectives derived from information-theoretic principles presents another compelling research direction. We believe SCORER provides a flexible foundation for developing a new class of more stable and efficient deep RL systems.

## References

- Agarwal et al. (2021)
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G Bellemare.
Contrastive behavioral similarity embeddings for generalization in reinforcement learning.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=qda7-sVg84](https://openreview.net/forum?id=qda7-sVg84).
- Anand et al. (2019)
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm.
Unsupervised state representation learning in atari.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 32, 2019.
URL [https://proceedings.neurips.cc/paper_files/paper/2019/file/6fb52e71b837628ac16539c1ff911667-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2019/file/6fb52e71b837628ac16539c1ff911667-Paper.pdf).
- Baird (1995)
Leemon Baird.
Residual algorithms: Reinforcement learning with function approximation.
*Machine Learning Proceedings*, 1995.
- Banino et al. (2022)
Andrea Banino, Adria Puigdomenech Badia, Jacob C Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell.
CoBERL: Contrastive BERT for reinforcement learning.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=sRZ3GhmegS](https://openreview.net/forum?id=sRZ3GhmegS).
- Belomestny et al. (2023)
Denis Belomestny, Alexey Naumov, Nikita Puchkin, and Sergey Samsonov.
Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations.
*Neural Networks*, 161:242–253, 2023.
ISSN 0893-6080.
doi: https://doi.org/10.1016/j.neunet.2023.01.035.
URL [https://www.sciencedirect.com/science/article/pii/S0893608023000473](https://www.sciencedirect.com/science/article/pii/S0893608023000473).
- Borkar (1997)
Vivek S Borkar.
Stochastic approximation with two time scales.
*Systems & Control Letters*, 29(5):291–294, 1997.
- Bradbury et al. (2018)
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.
JAX: composable transformations of Python+NumPy programs, 2018.
URL [http://github.com/jax-ml/jax](http://github.com/jax-ml/jax).
- Brockman et al. (2016)
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.
Openai gym, 2016.
URL [https://arxiv.org/abs/1606.01540](https://arxiv.org/abs/1606.01540).
- Chen et al. (2023)
Ziyi Chen, Bhavya Kailkhura, and Yi Zhou.
An accelerated proximal algorithm for regularized nonconvex and nonsmooth bi-level optimization.
*Machine Learning*, 112(5):1433–1463, 2023.
ISSN 0885-6125.
doi: 10.1007/s10994-023-06329-6.
- Chevalier-Boisvert et al. (2023)
Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry.
Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.
*Advances in Neural Information Processing Systems*, 36:73383–73394, 2023.
- Dazeley et al. (2023)
Richard Dazeley, Peter Vamplew, and Francisco Cruz.
Explainable reinforcement learning for broad-XAI: a conceptual framework and survey.
*Neural Computing and Applications*, 03 2023.
ISSN 1433-3058.
doi: 10.1007/s00521-023-08423-1.
URL [https://doi.org/10.1007/s00521-023-08423-1](https://doi.org/10.1007/s00521-023-08423-1).
- Fiez et al. (2020)
Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff.
Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study.
In Hal Daumé III and Aarti Singh (eds.), *Proceedings of the 37th International Conference on Machine Learning*, volume 119 of *Proceedings of Machine Learning Research*, pp. 3133–3144. PMLR, 13–18 Jul 2020.
URL [https://proceedings.mlr.press/v119/fiez20a.html](https://proceedings.mlr.press/v119/fiez20a.html).
- Gallici et al. (2025)
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin.
Simplifying deep temporal difference learning.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=7IzeL0kflu](https://openreview.net/forum?id=7IzeL0kflu).
- Geramifard et al. (2013)
Alborz Geramifard, Thomas J. Walsh, Stefanie Tellex, Girish Chowdhary, Nicholas Roy, and Jonathan P. How.
A tutorial on linear function approximators for dynamic programming and reinforcement learning.
*Foundations and Trends® in Machine Learning*, 6(4):375–451, 2013.
ISSN 1935-8237.
doi: 10.1561/2200000042.
- Ghadimi & Wang (2018)
Saeed Ghadimi and Mengdi Wang.
Approximation methods for bilevel programming, 2018.
URL [https://arxiv.org/abs/1802.02246](https://arxiv.org/abs/1802.02246).
- Greydanus et al. (2018)
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern.
Visualizing and understanding atari agents.
In *Proceedings of the 35th International Conference on Machine Learning*, volume 80, pp. 1792–1801, 2018.
URL [https://proceedings.mlr.press/v80/greydanus18a.html](https://proceedings.mlr.press/v80/greydanus18a.html).
- Hazan et al. (2019)
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest.
Provably efficient maximum entropy exploration.
In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), *Proceedings of the 36th International Conference on Machine Learning*, volume 97, pp. 2681–2691, 09–15 Jun 2019.
URL [https://proceedings.mlr.press/v97/hazan19a.html](https://proceedings.mlr.press/v97/hazan19a.html).
- Hessel et al. (2018)
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.
Rainbow: Combining improvements in deep reinforcement learning.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018.
- Hong et al. (2023)
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang.
A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic.
*SIAM Journal on Optimization*, 33(1):147–180, 2023.
doi: 10.1137/20m1387341.
URL [https://doi.org/10.1137/20M1387341](https://doi.org/10.1137/20M1387341).
- Huang et al. (2022)
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and JoÃĢo GM AraÃšjo.
Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.
*Journal of Machine Learning Research*, 23(274):1–18, 2022.
- Jaderberg et al. (2017)
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu.
Reinforcement learning with unsupervised auxiliary tasks.
In *International Conference on Learning Representations*, 2017.
- Kapturowski et al. (2018)
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney.
Recurrent experience replay in distributed reinforcement learning.
In *International conference on learning representations*, 2018.
- Karimi et al. (2016)
Hamed Karimi, Julie Nutini, and Mark Schmidt.
Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition.
In *Joint European conference on machine learning and knowledge discovery in databases*, pp. 795–811. Springer, 2016.
- Kulkarni et al. (2016)
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum.
Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.
In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 29, 2016.
URL [https://proceedings.neurips.cc/paper_files/paper/2016/file/f442d33fa06832082290ad8544a8da27-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2016/file/f442d33fa06832082290ad8544a8da27-Paper.pdf).
- Lambertini (2018)
Luca Lambertini.
*Stackelberg Games*, pp. 234–254.
Cambridge University Press, 2018.
- Lange (2022)
Robert Tjarko Lange.
gymnax: A JAX-based reinforcement learning environment library, 2022.
URL [http://github.com/RobertTLange/gymnax](http://github.com/RobertTLange/gymnax).
- Lange & Riedmiller (2010)
Sascha Lange and Martin Riedmiller.
Deep auto-encoder neural networks in reinforcement learning.
In *The 2010 International Joint Conference on Neural Networks (IJCNN)*, pp. 1–8, 2010.
doi: 10.1109/IJCNN.2010.5596468.
- Laskin et al. (2020a)
Michael Laskin, Aravind Srinivas, and Pieter Abbeel.
Curl: Contrastive unsupervised representations for reinforcement learning.
In *International conference on machine learning*, pp. 5639–5650. PMLR, 2020a.
- Laskin et al. (2020b)
Michael Laskin, Aravind Srinivas, and Pieter Abbeel.
CURL: Contrastive unsupervised representations for reinforcement learning.
In Hal Daumé III and Aarti Singh (eds.), *Proceedings of the 37th International Conference on Machine Learning*, volume 119 of *Proceedings of Machine Learning Research*, pp. 5639–5650. PMLR, 13–18 Jul 2020b.
URL [https://proceedings.mlr.press/v119/laskin20a.html](https://proceedings.mlr.press/v119/laskin20a.html).
- Lee et al. (2020)
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama.
Predictive information accelerates learning in rl.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 11890–11901, 2020.
URL [https://proceedings.neurips.cc/paper_files/paper/2020/file/89b9e0a6f6d1505fe13dea0f18a2dcfa-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/89b9e0a6f6d1505fe13dea0f18a2dcfa-Paper.pdf).
- Lesort et al. (2018)
Timothée Lesort, Natalia Díaz-Rodríguez, Jean-Frano̧is Goudou, and David Filliat.
State representation learning for control: An overview.
*Neural Networks*, 108:379–392, 2018.
ISSN 0893-6080.
doi: 10.1016/j.neunet.2018.07.006.
- Li et al. (2016)
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao.
Deep reinforcement learning for dialogue generation.
In Jian Su, Kevin Duh, and Xavier Carreras (eds.), *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pp. 1192–1202, Austin, Texas, November 2016. Association for Computational Linguistics.
doi: 10.18653/v1/D16-1127.
URL [https://aclanthology.org/D16-1127/](https://aclanthology.org/D16-1127/).
- Li & Zhu (2019)
Tao Li and Quanyan Zhu.
On convergence rate of adaptive multiscale value function approximation for reinforcement learning.
*2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP)*, pp. 1–6, 2019.
doi: 10.1109/mlsp.2019.8918816.
- Li et al. (2022)
Tao Li, Haozhe Lei, and Quanyan Zhu.
Sampling attacks on meta reinforcement learning: A minimax formulation and complexity analysis.
*arXiv preprint arXiv:2208.00081*, 2022.
doi: 10.48550/arXiv.2208.00081.
[Online] Available at [https://arxiv.org/pdf/2208.00081](https://arxiv.org/pdf/2208.00081).
- Li et al. (2023)
Tao Li, Haozhe Lei, and Quanyan Zhu.
Self-adaptive driving in nonstationary environments through conjectural online lookahead adaptation.
In *2023 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 7205–7211, 2023.
doi: 10.1109/ICRA48891.2023.10161368.
- Li et al. (2024)
Tao Li, Henger Li, Yunian Pan, Tianyi Xu, Zizhan Zheng, and Quanyan Zhu.
Meta stackelberg game: Robust federated learning against adaptive and mixed poisoning attacks.
*arXiv preprint arXiv:2410.17431*, 2024.
[Online] Available at [https://arxiv.org/pdf/2410.17431](https://arxiv.org/pdf/2410.17431).
- Li et al. (2025)
Tao Li, Haozhe Lei, Hao Guo, Mingsheng Yin, Yaqi Hu, Quanyan Zhu, and Sundeep Rangan.
Digital twin-enhanced wireless indoor navigation: Achieving efficient environment sensing with zero-shot reinforcement learning.
*IEEE Open Journal of the Communications Society*, 6:2356–2372, 2025.
doi: 10.1109/OJCOMS.2025.3552277.
- Liu et al. (2022)
Bo Liu, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu.
BOME! bilevel optimization made easy: A simple first-order approach.
In *Advances in Neural Information Processing Systems*, volume 35, pp. 17248–17262, 2022.
URL [https://proceedings.neurips.cc/paper_files/paper/2022/file/6dddcff5b115b40c998a08fbd1cea4d7-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/6dddcff5b115b40c998a08fbd1cea4d7-Paper-Conference.pdf).
- Liu et al. (2017)
Guan-Horng Liu, Avinash Siravuru, Sai Prabhakar, Manuela Veloso, and George Kantor.
Learning end-to-end multimodal sensor policies for autonomous navigation.
In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), *Proceedings of the 1st Annual Conference on Robot Learning*, volume 78 of *Proceedings of Machine Learning Research*, pp. 249–261. PMLR, 13–15 Nov 2017.
URL [https://proceedings.mlr.press/v78/liu17a.html](https://proceedings.mlr.press/v78/liu17a.html).
- Liu et al. (2021)
Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Li Jian, Nenghai Yu, and Tie-Yan Liu.
Return-based contrastive representation learning for reinforcement learning.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=_TM6rT7tXke](https://openreview.net/forum?id=_TM6rT7tXke).
- Lu et al. (2022)
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster.
Discovered policy optimisation.
*Advances in Neural Information Processing Systems*, 35:16455–16468, 2022.
- Lu et al. (2021)
Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang.
Deep network approximation for smooth functions.
*SIAM Journal on Mathematical Analysis*, 53(5):5465–5506, 2021.
- Lyle et al. (2022)
Clare Lyle, Mark Rowland, and Will Dabney.
Understanding and preventing capacity loss in reinforcement learning.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=ZkC8wKoLbQ7](https://openreview.net/forum?id=ZkC8wKoLbQ7).
- Mirowski et al. (2017)
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell.
Learning to navigate in complex environments.
In *International Conference on Learning Representations*, 2017.
URL [https://openreview.net/forum?id=SJMGPrcle](https://openreview.net/forum?id=SJMGPrcle).
- Mnih et al. (2015)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.
Human-level control through deep reinforcement learning.
*nature*, 518(7540):529–533, 2015.
- Osborne (2004)
M.J. Osborne.
*An Introduction to Game Theory*.
Oxford University Press, 2004.
ISBN 9780195128956.
URL [https://books.google.com/books?id=Ep7bPXVTI8MC](https://books.google.com/books?id=Ep7bPXVTI8MC).
- Pan et al. (2025)
Yunian Pan, Tao Li, and Quanyan Zhu.
Model-agnostic meta-policy optimization via zeroth-order estimation: A linear quadratic regulator perspective.
*arXiv preprint arXiv:2503.00385*, 2025.
[Online] Available at [https://arxiv.org/pdf/2503.00385](https://arxiv.org/pdf/2503.00385).
- Pathak et al. (2017)
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell.
Curiosity-driven exploration by self-supervised prediction.
In Doina Precup and Yee Whye Teh (eds.), *Proceedings of the 34th International Conference on Machine Learning*, volume 70 of *Proceedings of Machine Learning Research*, pp. 2778–2787. PMLR, 06–11 Aug 2017.
URL [https://proceedings.mlr.press/v70/pathak17a.html](https://proceedings.mlr.press/v70/pathak17a.html).
- Petrulionytė et al. (2024)
Ieva Petrulionytė, Julien Mairal, and Michael Arbel.
Functional bilevel optimization for machine learning.
*Advances in Neural Information Processing Systems*, 37:14016–14065, 2024.
- Pignatelli et al. (2024)
Eduardo Pignatelli, Jarek Liesen, Robert Tjarko Lange, Chris Lu, Pablo Samuel Castro, and Laura Toni.
Navix: Scaling minigrid environments with jax.
*arXiv preprint arXiv:2407.19396*, 2024.
- Schwarzer et al. (2021)
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman.
Data-efficient reinforcement learning with self-predictive representations.
In *International Conference on Learning Representations*, 2021.
- Stooke et al. (2021)
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin.
Decoupling representation learning from reinforcement learning.
In Marina Meila and Tong Zhang (eds.), *Proceedings of the 38th International Conference on Machine Learning*, volume 139 of *Proceedings of Machine Learning Research*, pp. 9870–9879. PMLR, 18–24 Jul 2021.
URL [https://proceedings.mlr.press/v139/stooke21a.html](https://proceedings.mlr.press/v139/stooke21a.html).
- Subramanian et al. (2022)
Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan.
Approximate information state for approximate planning and reinforcement learning in partially observed systems.
*Journal of Machine Learning Research*, 23:1–83, 2022.
doi: 10.48550/arxiv.2010.08843.
URL [http://jmlr.org/papers/v23/20-1165.html](http://jmlr.org/papers/v23/20-1165.html).
- Sutton & Barto (2018)
Richard S Sutton and Andrew G. Barto.
Reinforcement learning: An introduction.
*A Bradford Book*, 2018.
- Thrun & Schwartz (1993)
Sebastian Thrun and Anton Schwartz.
Issues in using function approximation for reinforcement learning.
In *Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum*, 1993.
- Träuble et al. (2021)
Frederik Träuble, Andrea Dittadi, Manuel Wuthrich, Felix Widmaier, Peter Vincent Gehler, Ole Winther, Francesco Locatello, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer.
Representation learning for out-of-distribution generalization in reinforcement learning.
In *ICML 2021 Workshop on Unsupervised Reinforcement Learning*, 2021.
URL [https://openreview.net/forum?id=I8rHTlfITWC](https://openreview.net/forum?id=I8rHTlfITWC).
- Tsitsiklis & Roy (1997)
John N Tsitsiklis and Benjamin Van Roy.
Analysis of temporal-diffference learning with function approximation.
*IEEE Transactions on Automatic Control*, 42(5):674–690, 1997.
doi: 10.1109/9.580874.
linear fucntion approximation: convengence proof #reinforcement learning #value function approximation.
- Van Hasselt et al. (2016)
Hado Van Hasselt, Arthur Guez, and David Silver.
Deep reinforcement learning with double q-learning.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 30, 2016.
- Van Hasselt et al. (2018)
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil.
Deep reinforcement learning and the deadly triad.
*arXiv preprint arXiv:1812.02648*, 2018.
- Wang et al. (2024)
Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, and Adam White.
Investigating the properties of neural network representations in reinforcement learning.
*Artificial Intelligence*, 330:104100, 2024.
ISSN 0004-3702.
doi: 10.1016/j.artint.2024.104100.
- Wang et al. (2016)
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas.
Dueling network architectures for deep reinforcement learning.
In *International conference on machine learning*, pp. 1995–2003. PMLR, 2016.
- Wu et al. (2021)
Xian Wu, Nevena Lazic, Dong Yin, and Cosmin Paduraru.
Importance of representation learning for off-policy fitted q-evaluation.
*Offline Reinforcement Learning Workshop, NeurIPS*, 2021.
URL [https://offline-rl-neurips.github.io/2021/pdf/17.pdf](https://offline-rl-neurips.github.io/2021/pdf/17.pdf).
- Yarats et al. (2022)
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.
Mastering visual continuous control: Improved data-augmented reinforcement learning.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=_SJ-_yyes8](https://openreview.net/forum?id=_SJ-_yyes8).
- Yin et al. (2024)
Mingsheng Yin, Tao Li, Haozhe Lei, Yaqi Hu, Sundeep Rangan, and Quanyan Zhu.
Zero-shot wireless indoor navigation through physics-informed reinforcement learning.
In *2024 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 5111–5118, 2024.
doi: 10.1109/ICRA57147.2024.10611229.
- Young & Tian (2019)
Kenny Young and Tian Tian.
Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments.
*arXiv preprint arXiv:1903.03176*, 2019.
- Yu et al. (2020)
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.
Gradient surgery for multi-task learning.
*Advances in neural information processing systems*, 33:5824–5836, 2020.
- Zhang et al. (2021)
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine.
Learning invariant representations for reinforcement learning without reconstruction.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=-2FCwDKRREu](https://openreview.net/forum?id=-2FCwDKRREu).

## Appendix

## Appendix A Code availability

The complete source code for SCORER and the experiments presented in this paper is available at [https://anonymous.4open.science/r/SCORER-530/](https://anonymous.4open.science/r/SCORER-530/) to ensure reproducibility. Our implementation is built in JAX (Bradbury et al., 2018) and leverages significant components, particularly for environment vectorization and training loops, from the PureJaxRL framework (Lu et al., 2022)(^1^11[https://github.com/luchris429/purejaxrl/tree/main](https://github.com/luchris429/purejaxrl/tree/main)). While core architectural choices (Section [C](https://arxiv.org/html/2508.07452v2#A3)) and most hyperparameters for baseline algorithms are consistent with PureJaxRL defaults for MinAtar and DQN, our SCORER-specific mechanisms (perception network objective, two-timescale updates) are additions.

## Appendix B LLM Usage

Writing and editing assistants, including a large language model (LLM) and automated grammar-checking tools, were used to improve the clarity, conciseness, and grammatical correctness of this work. The usage of these tools was strictly limited to polishing the written text. All scientific contributions, including the framework ideation, conceptualization, theoretical analysis, and experimental results, are the original work of the authors.

## Appendix C Experiment Setup and Hyperparameters

Software We used the following software versions:

- •
Python 3.10 - Python Software License [https://docs.python.org/3/license.html](https://docs.python.org/3/license.html)
- •
CUDA 12.4 - NVIDIA Software License Agreement [https://docs.nvidia.com/cuda/eula/index.html](https://docs.nvidia.com/cuda/eula/index.html)
- •
Jax 0.4.28 - Apache License 2.0 [https://github.com/jax-ml/jax](https://github.com/jax-ml/jax)
- •
Flashbax 0.1.3 - Apache License 2.0 [https://github.com/instadeepai/flashbax](https://github.com/instadeepai/flashbax)
- •
Chex 0.1.90 - Apache License 2.0 [https://github.com/google-deepmind/chex](https://github.com/google-deepmind/chex)
- •
Optax 0.2.5 - Apache License 2.0 [https://github.com/google-deepmind/optax](https://github.com/google-deepmind/optax)
- •
flax 0.10.4 - Apache License 2.0 [https://github.com/google/flax](https://github.com/google/flax)
- •
Gymnax 0.0.9 - Apache License 2.0 [https://github.com/RobertTLange/gymnax](https://github.com/RobertTLange/gymnax)
- •
Navix 0.7.4 - Apache License 2.0 [https://epignatelli.com/navix/](https://epignatelli.com/navix/)
- •
Rlax 0.1.7 - Apache License 2.0 [https://epignatelli.com/navix](https://epignatelli.com/navix)
[https://github.com/RobertTLange/gymnax](https://github.com/RobertTLange/gymnax)
- •
PureJaxRL - Apache License 2.0 [https://github.com/luchris429/purejaxrl](https://github.com/luchris429/purejaxrl)

All experiments were conducted on NVIDIA Tesla V100-PCIE-32GB GPUs. A typical experimental run, consisting of training one algorithm variant (e.g., SCORER DQN) over 30 random seeds for $10^{8}$ total environment time steps on a MinAtar environment, completed in approximately 20 to 27 minutes.

##### Hyperparameter Tuning Methodology

For a fair and rigorous comparison, we performed a two-stage hyperparameter search where initially, for each base algorithm (e.g., DQN, PQN), we performed a systematic grid search to identify its strongest possible configuration on our benchmarks. Then, we inherited these optimal hyperparameters for each SCORER variant and conducted a targeted search for the introduced SCORER hyperparameter (the follower’s learning rate ($\alpha_{\phi}$).

##### SCORER-Specific Hyperparameters

The key parameters introduced by SCORER are the learning rates for the leader ($\alpha_{\theta}$) and the follower ($\alpha_{\phi}$). As described in Section [4](https://arxiv.org/html/2508.07452v2#S4), the leader’s learning rate ($\alpha_{\theta}$) is set to the optimal learning rate found for the baselines. The follower’s learning rate ($\alpha_{\phi}$) is then tuned from a set of values greater than $\alpha_{\theta}$ to establish the necessary timescale separation.

##### Architectural Parity

To confirm that performance gains are a result of SCORER and not increased model capacity, we keep strict architectural parity between each baseline and its corresponding SCORER version. For a given monolithic baseline (e.g., a DQN with a three-layer MLP), the SCORER version is constructed by splitting this same architecture. The initial layers form the perception network ($f_{\phi}$), and the final layer forms the control network ($Q_{\theta}$). This way, we ensure that the total number of layers, hidden units, and learnable parameters is nearly identical between the baseline and SCORER agent, isolating the algorithmic contribution.

**Table 2: General Training Hyperparameters for Q-Learning Methods (primarily MinAtar environment defaults). Values for control environments may differ for some parameters (e.g., total timesteps, exploration schedule, learning rates).**
| Parameter | Value |
| --- | --- |
| Training Configuration |  |
| Number of parallel environments | 128 |
| Total timesteps | $1\times 10^{8}$ |
| Learning starts (time steps) | $1\times 10^{4}$ |
| Training interval (env steps) | 4 |
| Replay Buffer |  |
| Buffer size | $1\times 10^{5}$ |
| Batch size | 64 |
| Exploration (Epsilon-Greedy) |  |
| $\epsilon$ start | 1.0 |
| $\epsilon$ finish | 0.01 |
| $\epsilon$ anneal time (env steps) | $2.5\times 10^{5}$ |
| Learning Parameters |  |
| Optimizer | Adam |
| Discount factor ($\gamma$) | 0.99 |
| Linear learning rate decay | True |
| Target network update interval (env steps) | $1\times 10^{3}$ |
| Soft update parameter ($\tau$) for target nets | 1.0 |

**Table 3: SCORER-Specific Hyperparameters (MinAtar environment defaults).**
| Parameter | Value |
| --- | --- |
| Leader-Follower Architecture |  |
| Optimizer | Adam |
| Leader (control) learning rate ($\alpha_{\theta}$) | $1\times 10^{-4}$ |
| Follower (perception) learning rate ($\alpha_{\phi}$) | $5\times 10^{-4}$ |
| Max gradient norm for Leader & Follower | 0.5 |

**Table 4: General Training Hyperparameters for Q-Learning Methods for classic control environments**
| Parameter | Value |
| --- | --- |
| Training Configuration |  |
| Number of parallel environments | 10 |
| Total timesteps | $1\times 10^{6}$ |
| Learning starts (time steps) | $1\times 10^{3}$ |
| Training interval (env steps) | 10 |
| Replay Buffer |  |
| Buffer size | $5\times 10^{4}$ |
| Batch size | 64 |
| Exploration (Epsilon-Greedy) |  |
| $\epsilon$ start | 1.0 |
| $\epsilon$ finish | 0.01 |
| $\epsilon$ anneal time (env steps) | $2.5\times 10^{5}$ |
| Learning Parameters |  |
| Optimizer | Adam |
| Discount factor ($\gamma$) | 0.99 |
| Linear learning rate decay | True |
| Target network update interval (env steps) | $1\times 10^{3}$ |
| Soft update parameter ($\tau$) for target nets | 1.0 |

**Table 5: SCORER-Specific Hyperparameters (Classic Control).**
| Parameter | Value |
| --- | --- |
| Leader-Follower Architecture |  |
| Optimizer | Adam |
| Leader (control) learning rate ($\alpha_{\theta}$) | $1\times 10^{-4}$ |
| Follower (perception) learning rate ($\alpha_{\phi}$) | $3\times 10^{-4}$ |
| Max gradient norm for Leader & Follower | 0.3 |

**Table 6: General Training Hyperparameters for SCORER PQN (MinAtar environment defaults). Values for control environments may differ.**
| Parameter | Value |
| --- | --- |
| Training Configuration |  |
| Number of parallel environments | 128 |
| Total timesteps | $1\times 10^{8}$ |
| Rollout length (experience collection) | 64 |
| Training epochs per rollout | 2 |
| Minibatch count per epoch | 8 |
| Exploration (Epsilon-Greedy) |  |
| $\epsilon$ start | 1.0 |
| $\epsilon$ finish | 0.01 |
| $\epsilon$ anneal period | 80% of training updates |
| Learning Parameters |  |
| Optimizer | RAdam |
| Discount factor ($\gamma$) | 0.99 |
| Lambda for TD($\lambda$) returns ($\lambda$) | 0.65 |
| Linear learning rate decay | True |
| Target network | N/A (uses $\lambda$-returns) |

**Table 7: SCORER-Specific Hyperparameters for PQN (MinAtar environment defaults).**
| Parameter | Value |
| --- | --- |
| Leader-Follower Architecture |  |
| Optimizer | RAdam |
| Leader (Q-Network) learning rate ($\alpha_{\theta}$) | $2.5\times 10^{-4}$ |
| Follower (Perception) learning rate ($\alpha_{\phi}$) | $5\times 10^{-4}$ |
| Max gradient norm | 0.5 |

**Table 8: General Training Hyperparameters for SCORER R2D2.**
| Parameter | Value |
| --- | --- |
| Training Configuration |  |
| Number of parallel environments | 10 |
| Total timesteps | $1\times 10^{6}$ |
| Learning starts (env steps) | 25,000 |
| Prioritized Trajectory Replay Buffer |  |
| Buffer size (transitions) | $1\times 10^{5}$ |
| Min buffer size for sampling | 5,000 |
| Batch size (sequences) | 32 |
| Sampled sequence length | 100 |
| Burn-in length | 50 |
| N-step returns | 10 |
| PER alpha ($\alpha$) | 0.6 |
| PER beta start ($\beta_{0}$) | 0.4 |
| PER beta end ($\beta_{T}$) | 1.0 |
| Exploration (Epsilon-Greedy) |  |
| $\epsilon$ start | 1.0 |
| $\epsilon$ finish | 0.01 |
| $\epsilon$ anneal time (env steps) | $5\times 10^{5}$ |
| Learning Parameters |  |
| Optimizer | Adam |
| Discount factor ($\gamma$) | 0.99 |
| Linear learning rate decay | True |
| Target network update interval (env steps) | 5,000 |
| Update interval (env steps) | 16 |
| Soft update parameter ($\tau$) | 1.0 |

**Table 9: SCORER-Specific Hyperparameters for R2D2.**
| Parameter | Value |
| --- | --- |
| Leader-Follower Architecture |  |
| Optimizer | Adam |
| Leader (Control) learning rate ($\alpha_{\theta}$) | $1\times 10^{-4}$ |
| Follower (Perception) learning rate ($\alpha_{\phi}$) | $3\times 10^{-4}$ |
| Leader convergence steps per update | 1 |
| Max gradient norm (Leader & Follower) | 0.5 |

### C.1 Network Architectures

All agents within the DQN family (DQN, DDQN, Dueling DQN, and Dueling DDQN) are built from a shared architectural template. The monolithic baseline for each variant is constructed first, and its corresponding SCORER version is created by splitting this same architecture to maintain parity in the number of layers and parameters. Table [10](https://arxiv.org/html/2508.07452v2#A3.T10) details this base architecture.

**Table 10: Base network architecture for the SCORER DQN agent family. Linear(in, out) denotes a fully connected layer and L1Norm denotes a layer that normalizes its input to have a unit L1 norm. Key dimensions are environment-dependent. The activation function is applied after hidden layers, but not on the final Q-Network output.**
| SCORER DQN Base Architecture |
| --- |
| Hyperparameters: |
| state_dim = Dimension of observation vector |
| action_dim = Number of discrete actions |
| latent_dim = 64 (for Control tasks), 128 (for MinAtar) |
| q_hidden_dim = 64 (for Control tasks), 128 (for MinAtar) |
| activation = Tanh (for Control tasks), ReLU (for MinAtar) |
| Perception Network (Follower, $f_{\phi}$) |
| $\vartriangleright$ Encodes raw state into a latent representation z. |
| If MinAtar environment: |
| input_state = L1Norm(state) |
| Else: |
| input_state = state |
| p_l0 = Linear(state_dim, latent_dim) |
| Perception Network Forward Pass: |
| h0 = activation(p_l0(input_state)) |
| z = h0  (For Control tasks) |
| If MinAtar environment: |
| p_l1 = Linear(latent_dim, latent_dim) |
| h1 = activation(p_l1(h0)) |
| z = L1Norm(h1)  (Overrides z) |
| Q Network (Leader, $Q_{\theta}$) |
| $\vartriangleright$ Predicts Q-values from the latent representation z. |
| q_l0 = Linear(latent_dim, q_hidden_dim) |
| q_l1 = Linear(q_hidden_dim, action_dim) |
| Q Network Forward Pass: |
| input = z |
| h0 = activation(q_l0(input)) |
| q_values = q_l1(h0) |

##### Architectural Variants

The architecture described in Table [10](https://arxiv.org/html/2508.07452v2#A3.T10) serves as the foundation for all agents in the DQN family.

- •
DQN and DDQN: The monolithic baselines for DQN and DDQN consist of a single network formed by composing the layers described above (e.g., for MinAtar, a four-layer MLP). The SCORER variants are created by splitting this architecture exactly as shown above.
- •
Dueling DQN and Dueling DDQN: For these agents, the Perception Network ($f_{\phi}$) remains identical. The Q-Network ($Q_{\theta}$) is modified to implement the dueling architecture (Wang et al., 2016). Specifically, the ‘h0‘ feature vector from the Q Network’s forward pass is fed into two separate heads: a state-value head, $V(z)=\texttt{Linear(q_hidden_dim, 1)}$, and an advantage head, $A(z,a)=\texttt{Linear(q_hidden_dim, action_dim)}$. The final Q-values are then combined using the standard aggregation method: $Q(z,a)=V(z)+(A(z,a)-\frac{1}{|\mathcal{A}|}\sum_{a^{\prime}}A(z,a^{\prime}))$.

**Table 11: Base network architecture for the SCORER PQN. Linear(in, out) denotes a fully connected layer and LayerNorm denotes a layer normalization layer. Key dimensions are environment-dependent. The activation function is applied in hidden layers, but not on the final Q-Network output.**
| SCORER PQN Base Architecture |
| --- |
| Hyperparameters: |
| state_dim = Dimension of observation vector |
| action_dim = Number of discrete actions |
| latent_dim = 64 (for Control tasks), 128 (for MinAtar) |
| q_hidden_dim = 32 (for Control tasks), 64 (for MinAtar) |
| q_num_layers = 1 (for all tasks) |
| activation = ReLU (for all tasks) |
| norm_type = LayerNorm (for all tasks) |
| Perception Network (Follower, $f_{\phi}$) |
| $\vartriangleright$ Encodes raw state into a latent representation z. |
| p_l0 = Linear(state_dim, latent_dim) |
| Perception Network Forward Pass: |
| h0 = p_l0(state) |
| h1 = activation(h0) |
| z = LayerNorm(h1) |
| Q Network (Leader, $Q_{\theta}$) |
| $\vartriangleright$ Predicts Q-values from the latent representation z. |
| $\vartriangleright$ Contains q_num_layers hidden blocks. |
| q_l_hidden = Linear(latent_dim, q_hidden_dim) |
| q_l_out = Linear(q_hidden_dim, action_dim) |
| Q Network Forward Pass (for q_num_layers=1): |
| input = z |
| h0 = q_l_hidden(input) |
| h1 = LayerNorm(h0) |
| h2 = activation(h1)  (Uses pre-activation normalization) |
| q_values = q_l_out(h2) |

**Table 12: Architecture for the SCORER R2D2. Linear(in, out) denotes a fully connected layer, GRUCell(in, out) denotes a Gated Recurrent Unit cell, and L2Norm denotes a layer that normalizes its input to have a unit L2 norm. The activation function is applied after hidden layers, but not on the final Control Network output.**
| SCORER R2D2 Base Architecture |
| --- |
| Hyperparameters: |
| state_dim = Dimension of observation vector |
| action_dim = Number of discrete actions |
| embed_dim = 128 |
| recurrent_dim = 128 |
| q_hidden_dim = 128 |
| activation = Tanh |
| Perception Network (Follower, $f_{\phi}$) |
| $\vartriangleright$ Encodes state and updates the recurrent hidden state. |
| $\vartriangleright$ Takes current state $state_{t}$ and previous hidden state $h_{t-1}$ as input. |
| p_l_embed = Linear(state_dim, embed_dim) |
| p_gru = GRUCell(embed_dim, recurrent_dim) |
| Perception Network Forward Pass: |
| input_state = L2Norm(state_t) |
| emb = p_l_embed(input_state) |
| emb_act = activation(emb) |
| $h_{t}$, z_raw = p_gru($h_{t-1}$, emb_act)  ($h_{t}$ is new hidden, z_raw is GRU output) |
| z = L2Norm(z_raw)  (Representation passed to Control Network) |
| $\vartriangleright$ The new hidden state for the next step is L2Norm($h_{t}$). |
| Control Network (Leader, $Q_{\theta}$) |
| $\vartriangleright$ Predicts Q-values from the latent representation z. |
| q_l_hidden = Linear(recurrent_dim, q_hidden_dim) |
| q_l_out = Linear(q_hidden_dim, action_dim) |
| Control Network Forward Pass: |
| input = z |
| h0 = q_l_hidden(input) |
| h1 = activation(h0) |
| q_values = q_l_out(h1) |

## Appendix D Objective Functions

This section provides the explicit formulations used by the SCORER framework. In our game, the Control Network ($Q_{\theta}$) acts as the Leader and the Perception Network ($f_{\phi}$) acts as the Follower.

### D.1 Leader (Control Network) Objective

As detailed in Section [4.1](https://arxiv.org/html/2508.07452v2#S4.SS1) ( equation [1](https://arxiv.org/html/2508.07452v2#S4.E1)), the control network $Q_{\theta}$ acts as the Stackelberg leader. Its objective is to learn an optimal action-value function by minimizing the Mean Squared Bellman Error (MSBE). Given a batch of transitions $B_{\text{leader}}$ and the follower’s best-reponse representations $f_{\phi^{*}}$, the leader’s objective is:

$$ $\displaystyle\mathcal{L}_{\text{leader}}(\theta,\phi^{*})=\mathbb{E}_{(s,a,r,s^{\prime},d)\sim B_{\text{leader}}}\left[\left(Y-Q_{\theta}(f_{\phi^{*}}(s),a)\right)^{2}\right].$ (7) $$

The Bellman target $Y$ used in these objectives is defined in Section [3](https://arxiv.org/html/2508.07452v2#S3). In our two-timescale algorithm, the leader’s update at step $k+1$ uses the follower’s most recent parameters $\phi_{k+1}$ as a proxy for the ideal $\phi^{*}$, and the gradient is taken with respect to $\theta$ only.

### D.2 Follower (Perception Network) Objective

The follower’s goal is to learn representations that support the Leader’s learning process. In our ablation studies, we explored two objectives for the follower, both of which treat the leader’s parameters $\theta$ as fixed via a stop-gradient operation. See equation [5](https://arxiv.org/html/2508.07452v2#S4.E5), and Algorithm [1](https://arxiv.org/html/2508.07452v2#alg1) Line 12.

#### D.2.1 Minimizing Follower’s Mean Squared Bellman Error (SCORER MSBE)

In this ablation variant, the follower’s objective is identical to the leader’s loss. Here, the follower learns representations that directly minimize the MSBE given the leader’s committed weights:

$$ $\displaystyle\mathcal{L}_{\text{Follower}}^{\text{MSBE}}(\phi,\overline{\theta})=\frac{1}{N}\sum_{j=1}^{N}\left[\left(Y_{j}-Q_{\overline{\theta}}(f_{\phi}(s_{j}),a_{j})\right)^{2}\right].$ (8) $$

The optimization is performed with respect to $\phi$.

#### D.2.2 Minimizing Bellman Error Variance

This is the main objective used in SCORER, designed to explicitly stabilize the leader’s learning signal. The intuition behind is that by focusing on the consistency of the Bellman errors across a batch, the follower can learn representations that are robust to the noisy target inherent in TD learning. Let the Bellman error for a single transition be $\delta_{j}(\phi,\overline{\theta})=Y_{j}-Q_{\overline{\theta}}(f_{\phi}(s_{j}),a_{j})$. The follower’s goal is to minimize the sample variance of these errors as:

$$ $\displaystyle\mathcal{L}_{\text{follower}}^{\text{Var}}(\phi,\overline{\theta})=\frac{1}{N}\sum_{j=1}^{N}\left(\delta_{j}(\phi,\overline{\theta})\right)^{2}-\left(\frac{1}{N}\sum_{j=1}^{N}\delta_{j}(\phi,\overline{\theta})\right)^{2}.$ (9) $$

This objective turns the follower into an active stabilizing agent. Because the leader’s performance is evaluated on an independent sample batch of data ($B_{\text{leader}}$), the follower is asked to learn representations that create a generally stable learning signal, instead of overfitting the specifics of its own sampled batch ($B_{\text{follower}}$). The follower must find representations that make the leader’s value predictions more uniform and predictable across the entire data distribution, truly dividing the task of stabilization from the immediate task of performance maximization.

## Appendix E Statistical Significance Calculations

Throughout this work, reported performance metrics aim to provide a robust understanding of algorithm behavior across multiple independent trials. This appendix details the methods used for calculating and presenting these statistics.

### E.1 Interquartile Mean (IQM) and Confidence Intervals

The learning curves presented show the Interquartile Mean (IQM) total reward achieved at various training timesteps, averaged over independent runs each initiated with a different random seed. The IQM is a robust statistical measure that computes the mean of values between the 25th and 75th percentiles, making it less sensitive to outliers compared to the arithmetic mean. For each timestep, we calculate the IQM across all seeds and use it as our central tendency measure.

The shaded regions accompanying these IQM curves represent a 95% confidence interval (CI). This CI is calculated assuming a normal distribution of the IQM values at each timestep, using the standard error of the IQM across all the seeds:

$$ $\displaystyle\text{CI}=\text{IQM}\pm 1.96\cdot\frac{\text{Standard Deviation of IQM}}{\sqrt{\text{Number of Seeds}}}.$ (10) $$

where 1.96 is the critical value from the standard normal distribution corresponding to 95% CI. Smoothing is applied using a rolling mean over 50 timesteps for visual clarity, applied consistently to both IQM values and confidence interval boundaries.

##### IQM for Final Performance Tables

For final performance reporting in tables, we first find the maximum average episode return achieved during training for each experimental run (seed). The IQM is then calculated over these maximum returns from all seeds by sorting the values, removing the bottom and top 25%, and computing the mean of the remaining central 50%. This approach provides a robust measure of peak performance that is less influenced by exceptionally high or low performing seeds.

##### Normalized IQM Return

For comparative analysis across environments with different reward scales, we present normalized IQM returns in some figures. The normalization is performed per environment by dividing all IQM values by the maximum IQM value reached across all methods in that specific environment. This helps us to develop a meaningful comparison of relative performance improvements across tasks while preserving the temporal dynamics of learning.

### E.2 Speed Comparison

The Speed Comparison column in Table [1](https://arxiv.org/html/2508.07452v2#S5.T1) reports the relative wall-clock time required to complete a full training run. For each base algorithm (e.g., DQN), the runtime of its standard monolithic version is normalized to 1.00x. The runtime of the corresponding SCORER variant is then reported as a multiple of this baseline. A value of 0.99x indicates that the SCORER variant was 1% faster than its baseline, whereas a value of 1.01% would indicate it was 1% slower.

### E.3 Time-to-Threshold (TTT) Calculation

To provide a measure of sample efficiency over the MiniGrid experiments, we use a Time-to-Threshold (TTT) analysis. TTT quantifies the number of environment steps required for an agent to reliably fulfill the task, or at least reach a high level of performance. Our calculation follows a two-pass process to ensure fair comparisons across environments with different reward scales and performance ceilings.

#### E.3.1 Performance Thresholds

Initially, for each evaluation environment, we determine the maximum asymptotic performance achieved across all runs of all methods. This sets an empirical "best-case" performance for the task. The performance threshold for each environment is then set to 99% of this maximum value.

#### E.3.2 Calculating TTT and Success Rate

In the second pass, for each individual seed of each method, we identify the first timestep at which the agent’s episodic return meets or exceeds the calculated 99% performance threshold. This timestep is recorded as the TTT for that run. If a run fails to reach the threshold within the maximum allowed timesteps for that environment, its TTT is considered undefined. The final TTT reported in Table [3](https://arxiv.org/html/2508.07452v2#S5.F3) is the mean over all successful runs. To capture the reliability of each method, we also present the Success Rate (SR), which is the percentage of the 30 independent runs that successfully reached the performance threshold.

### E.4 Statistical Significance Testing (Welch’s t-test)

In Table [1](https://arxiv.org/html/2508.07452v2#S5.T1), we highlight those statistically significant performance differences within each base algorithm family (e.g., DQN and its SCORER variants) for each environment. To determine this, we first identify the variant with the highest Interquartile Mean (IQM) in each group. Then, other variants are compared to this top performer using an independent two-sample Welch’s t-test, which is robust to unequal variances as it does not assume they are equal between the groups being compared. Performances are considered not statistically worse (and highlighted accordingly in green) if the p-value from the Welch’s t-test comparing their set of of maximum returns to those of the top performer is greater than or equal to a significance level $\alpha=0.05$.

## Appendix F Sample Efficiency Results

To complement the learning curves presented in Section [5](https://arxiv.org/html/2508.07452v2#S5), this section provides a detailed breakdown of the sample efficiency for SCORER compared to the baseline agents across four MinAtar environments. Tables [13](https://arxiv.org/html/2508.07452v2#A6.T13) through [16](https://arxiv.org/html/2508.07452v2#A6.T16) report the mean number of training steps (in millions, $\pm$ 95% CI) required for an agent to reach pre-defined reward thresholds, averaged over 30 seeds. For each experimental condition, a run is considered to have reached a threshold at the first time step where its individual performance curve meets or exceeds the target value. The tables report the statistics for all seeds that successfully reached the threshold. An "N/A" indicates that the Interquartile Mean (IQM) of the agent’s runs failed to reach the threshold within $10^{8}$ steps. The highlighted values indicates the variant that achieved the given threshold in the fewest timesteps.

Across the board, the data shows the substantial impact of the SCORER framework on learning speed. The effect is most pronounced in Breakout(Table [13](https://arxiv.org/html/2508.07452v2#A6.T13)). Here, the SCORER agents are an order of magnitude more sample-efficient than their monolithic baselines.

**Table 13: Sample efficiency (time steps in millions to reach reward thresholds) for Breakout-MinAtar.**
| Model | Variant | Threshold 17 | Threshold 20 | Threshold 25 |
| --- | --- | --- | --- | --- |
| DQN | Baseline | 39.1 $\pm$ 7.6 | N/A | N/A |
| SCORER | 3.4 $\pm$ 0.2 | 4.8 $\pm$ 0.5 | 8.2 $\pm$ 0.9 |  |
| DDQN | Baseline | 42.4 $\pm$ 6.6 | 43.8 $\pm$ 5.0 | 62.3 $\pm$ 6.4 |
| SCORER | 4.2 $\pm$ 0.4 | 5.9 $\pm$ 0.8 | 12.7 $\pm$ 3.6 |  |
| DuelingDQN | Baseline | 43.6 $\pm$ 8.4 | 50.8 $\pm$ 9.7 | 72.0 $\pm$ 9.1 |
| SCORER | 4.0 $\pm$ 0.4 | 5.3 $\pm$ 0.6 | 13.5 $\pm$ 2.7 |  |
| DuelingDDQN | Baseline | 27.6 $\pm$ 6.5 | 33.7 $\pm$ 5.8 | 64.2 $\pm$ 5.2 |
| SCORER | 4.2 $\pm$ 0.4 | 5.6 $\pm$ 0.6 | 17.8 $\pm$ 7.9 |  |

In Asterix and SpaceInvaders Table [14](https://arxiv.org/html/2508.07452v2#A6.T14) and Table [15](https://arxiv.org/html/2508.07452v2#A6.T15)), SCORER consistently reduces the number of samples required to reach performance thresholds, demonstrating robust improvements in sample efficiency across different base algorithms. The gains are particularly notable for higher thresholds.

In Freeway (Table [16](https://arxiv.org/html/2508.07452v2#A6.T16)), all agents converge rapidly. Even so, the SCORER variants consistently reach all performance thresholds in approximately half the time of their baseline counterparts.

**Table 14: Sample efficiency (time steps in millions to reach reward thresholds) for Asterix-MinAtar.**
| Model | Variant | Threshold 35 | Threshold 40 |
| --- | --- | --- | --- |
| DQN | Baseline | 22.8 $\pm$ 0.6 | 27.3 $\pm$ 1.1 |
| SCORER | 14.6 $\pm$ 2.2 | 17.3 $\pm$ 2.4 |  |
| DDQN | Baseline | 23.3 $\pm$ 0.9 | 27.7 $\pm$ 1.2 |
| SCORER | 14.8 $\pm$ 1.0 | 18.2 $\pm$ 1.2 |  |
| DuelingDQN | Baseline | 19.8 $\pm$ 1.0 | 22.6 $\pm$ 1.0 |
| SCORER | 15.4 $\pm$ 3.3 | 18.3 $\pm$ 3.4 |  |
| DuelingDDQN | Baseline | 20.5 $\pm$ 1.0 | 25.1 $\pm$ 1.8 |
| SCORER | 17.2 $\pm$ 6.4 | 16.6 $\pm$ 3.2 |  |

**Table 15: Sample efficiency (time steps in millions to reach reward thresholds) for SpaceInvaders-MinAtar.**
| Model | Variant | Threshold 75 | Threshold 100 | Threshold 110 |
| --- | --- | --- | --- | --- |
| DQN | Baseline | 9.7 $\pm$ 1.0 | 20.6 $\pm$ 2.7 | 31.2 $\pm$ 2.7 |
| SCORER | 9.1 $\pm$ 0.5 | 14.3 $\pm$ 0.8 | 17.8 $\pm$ 1.7 |  |
| DDQN | Baseline | 11.8 $\pm$ 1.1 | 24.7 $\pm$ 2.7 | 37.1 $\pm$ 2.6 |
| SCORER | 11.1 $\pm$ 0.6 | 18.6 $\pm$ 2.5 | 22.9 $\pm$ 3.4 |  |
| DuelingDQN | Baseline | 10.0 $\pm$ 0.9 | 23.4 $\pm$ 4.1 | 33.2 $\pm$ 5.7 |
| SCORER | 17.1 $\pm$ 0.7 | 26.0 $\pm$ 1.8 | 32.0 $\pm$ 2.5 |  |
| DuelingDDQN | Baseline | 9.9 $\pm$ 0.7 | 16.6 $\pm$ 1.4 | 26.9 $\pm$ 3.1 |
| SCORER | 15.4 $\pm$ 0.7 | 23.8 $\pm$ 1.8 | 30.2 $\pm$ 3.4 |  |

**Table 16: Sample efficiency (time steps in millions to reach reward thresholds) for Freeway-MinAtar.**
| Model | Variant | Threshold 30 | Threshold 40 | Threshold 50 |
| --- | --- | --- | --- | --- |
| DQN | Baseline | 3.8 $\pm$ 0.2 | 6.0 $\pm$ 0.3 | 10.1 $\pm$ 0.5 |
| SCORER | 1.3 $\pm$ 0.1 | 2.3 $\pm$ 0.1 | 4.5 $\pm$ 0.1 |  |
| DDQN | Baseline | 3.8 $\pm$ 0.3 | 5.9 $\pm$ 0.3 | 9.6 $\pm$ 0.4 |
| SCORER | 1.4 $\pm$ 0.1 | 2.3 $\pm$ 0.1 | 4.5 $\pm$ 0.2 |  |
| DuelingDQN | Baseline | 4.7 $\pm$ 0.4 | 6.1 $\pm$ 0.3 | 10.0 $\pm$ 0.4 |
| SCORER | 1.8 $\pm$ 0.2 | 2.7 $\pm$ 0.2 | 5.0 $\pm$ 0.4 |  |
| DuelingDDQN | Baseline | 4.6 $\pm$ 0.5 | 6.8 $\pm$ 0.7 | 11.5 $\pm$ 1.1 |
| SCORER | 1.6 $\pm$ 0.2 | 2.6 $\pm$ 0.2 | 4.9 $\pm$ 0.4 |  |

## Appendix G Classic Control Results

To further assess the general applicability of the SCORER framework beyond the MinAtar suite and MiniGrid, we conducted experiments on two classic control environments from OpenAI Gym Brockman et al. (2016) using Gymnax Lange (2022) for JAX compatibility: For these tasks, simpler Multi-Layer Perceptron architectures were used for both the perception and control networks, with details provided in Appendix [C](https://arxiv.org/html/2508.07452v2#A3).

Figure: Figure 5: Learning curves on classic control environments (CartPole-v1, Acrobot-v1). Each row corresponds to a base algorithm (DQN, DDQN, DuelingDQN, DuelingDDQN), and each column to an environment. Curves show IQM return over 30 seeds; shaded regions represent 95% confidence intervals.
Refer to caption: figures/classic_control_performance.png

## Appendix H Learning Curves on MiniGrid Environments

This section provides the full learning curves that support the time-to-threshold analysis presented in Table [3](https://arxiv.org/html/2508.07452v2#S5.F3). We evaluate the performance of SCORER when applied to R2D2, a recurrent baseline for partially observable environments, across a suite of MiniGrid tasks.

Figure: Figure 6: Learning curves for SCORER R2D2 SCORER versus the Vanilla R2D2 on eight Minigrid environments, averaged over 30 seeds. Shaded regions represent a 95% confidence interval. The plots highlight SCORER’s consistent improvement in sample efficiency and its ability to solve challenging exploration tasks like Four Rooms where the baseline fails.
Refer to caption: figures/navix_plots.png

The learning curves in Figure [6](https://arxiv.org/html/2508.07452v2#A8.F6) visually confirm Table [3](https://arxiv.org/html/2508.07452v2#S5.F3)’s results. The SCORER-enhanced agent demonstrates a steeper learning curve in most environments, indicating superior sample efficiency. The most significant result is observed in the Four Rooms environment, a classic hard-exploration task. Here, the baseline R2D2 agent performs poorly, compared to SCORER R2D2, which can solve the task, reaching a high success rate.

## Appendix I Detailed Ablation Study Results

This section provides detailed, per-environment learning curves for the ablation studies summarized in Section [5.3](https://arxiv.org/html/2508.07452v2#S5.SS3).

Figure: Figure 7: Per-environment results for the Stackelberg role assignment ablation. The standard SCORER configuration (Control as Leader, purple) is compared against the monolithic DQN baseline (green) and an inverted hierarchy where Perception acts as the Leader (teal).
Refer to caption: figures/ablation/appendix_leader_role.png

##### Stackelberg Role Assignment.

Figure [7](https://arxiv.org/html/2508.07452v2#A9.F7) details the results of our role-swapping experiment. The plots confirm that the standard SCORER configuration, where the control network acts as the leader, consistently and substantially outperforms both the baseline and the inverted hierarchy across all tested environments. The performance collapse observed when perception is assigned the leader role is particularly pronounced in complex environments like Breakout and SpaceInvaders, providing strong evidence that for stable learning to occur, the value function must lead the representation learning process.

Figure: Figure 8: Perception Objective
Refer to caption: figures/ablation/appendix_perception_objective.png

##### Follower’s Objective.

Figure [8](https://arxiv.org/html/2508.07452v2#A9.F8) presents a per-environment breakdown of the follower’s objective ablation. The results reinforce the conclusion from our main analysis: the Bellman Error Variance objective is critical for SCORER’s performance. In every environment, the BE Variance follower achieves the highest final performance and demonstrates the best sample efficiency. While the MSBE follower offers a clear improvement over the baseline, particularly in Breakout, the explicit stabilization provided by the variance objective unlocks a significantly higher level of performance, highlighting its role in enabling robust co-adaptation.

## Appendix J Theoretical Foundations of SCORER

In this section, we discuss optimization (especially bilevel optimization) related issues in the SCORER framework. Since the results are of independent interest for a large family of bi-level optimization problems, we adapted a general formulation.

### J.1 Bilevel Optimization & Two-Timescale Algorithms

The SCORER framework formulates the interaction between perception and control as a Stackelberg game, which we cast as a standard bilevel optimization problem:

$$ $\displaystyle\min_{{\bf x}\in\mathbb{R}^{d_{x}}}\ell({\bf x}):=f({\bf x},{\bf y}^{*}({\bf x})),\quad\text{s.t. }{\bf y}^{*}({\bf x})\in\arg\min_{{\bf y}\in\mathbb{R}^{d_{y}}}g({\bf x},{\bf y}).$ (11) $$

##### Mapping to SCORER.

In the context of SCORER, this general formulation maps directly to the SCORER framework as:

- •
The outer (leader) variable ${\bf x}$ corresponds to the control network parameters $\theta$.
- •
The inner (follower) variable ${\bf y}$ corresponds to the perception network parameters $\phi$.
- •
The outer objective $f({\bf x},{\bf y})$ is the Mean Squared Bellman Error (MSBE) loss for the leader.
- •
The inner objective $g({\bf x},{\bf y})$ is the Bellman Error Variance loss for the follower.

The remainder of this section will proceed with the general $({\bf x},{\bf y})$ notation to align with the standard optimization literature.

Two-Timescale Stochastic Approximation (TTSA) algorithm proposed in Borkar (1997) updates the variable through the following iterative step,

$$ $\displaystyle{\bf y}^{k+1}$ $\displaystyle={\bf y}^{k}-\beta_{k}D_{g}^{k},$ (12) $\displaystyle{\bf x}^{k+1}$ $\displaystyle={\bf x}^{k}-\alpha_{k}D_{f}^{k},$ (13) $$

where two sequences of learning rates $\{\alpha_{k}\}$ and $\{\beta_{k}\}$ satisfying $\alpha_{k}/\beta_{k}\to 0$, as $k\to\infty$;
$D_{f}^{k}$ and $D_{g}^{k}$ represent stochastic estimates of the gradients ${\overline{\nabla}}_{x}f({\bf x}^{k},{\bf y}^{k+1})$ and $\nabla_{y}g({\bf x}^{k},{\bf y}^{k})$, respectively, with ${\overline{\nabla}_{x}}f(x,y):=\nabla_{x}f(x,y)-\nabla^{2}_{xy}g(x,y)\nabla_{yy}^{2}g(x,y)^{-1}\nabla_{y}f(x,y)$.
The following conditions on the objective functions and their approximations are discussed in Hong et al. (2023), here, while we follow mostly their notations and basic arguments, detailed calculations are modified to fit our purposes.

###### Assumption 1 .

The outer function $f({\bf x},{\bf y})$ satisfies gradient Lipschitz conditions: for any ${\bf y}_{1}\neq{\bf y}_{2}\in\mathbb{R}^{*}$,

$$ $\displaystyle\frac{\|\nabla_{x}f({\bf x},{\bf y}_{1})-\nabla_{x}f({\bf x},{\bf y}_{2})\|}{\|{\bf y}_{1}-{\bf y}_{2}\|}\leq L_{f_{x}},\quad\hbox{uniformly in ${\bf x}$},$ $$

$$ $\displaystyle\frac{\|\nabla_{y}f({\bf x},{\bf y}_{1})-\nabla_{y}f({\bf x},{\bf y}_{2})\|}{\|{\bf y}_{1}-{\bf y}_{2}\|}\leq L_{f_{y}},\quad\hbox{uniformly in ${\bf x}$},$ $$

and for any ${\bf x}_{1}\neq{\bf x}_{2}\in\mathbb{R}^{d_{x}}$,

$$ $\displaystyle\frac{\|\nabla_{y}f({\bf x}_{1},{\bf y})-\nabla_{y}f({\bf x}_{1},{\bf y})\|}{\|{\bf x}_{1}-{\bf x}_{2}\|}\leq{\bar{L}}_{f_{y}},\quad\hbox{uniformly in ${\bf y}$}.$ $$

Gradient bound condition: $\|\nabla_{y}f({\bf x},{\bf y})\|\leq C_{f_{y}}$, uniformly in ${\bf x}$ and ${\bf y}$.

###### Assumption 2 .

The inner function $g(x,y)\in C^{2}(\Omega\times\mathbb{R}^{*})$,

$$ $\displaystyle\frac{\|\nabla_{y}y({\bf x},{\bf y}_{1})-\nabla_{y}g({\bf x},{\bf y}_{2})\|}{\|{\bf y}_{1}-{\bf y}_{2}\|}\leq L_{g},\quad\hbox{uniformly in ${\bf x}$},$ $$

$$ $\displaystyle\frac{\|\nabla^{2}_{xy}g({\bf x},{\bf y}_{1})-\nabla^{2}_{xy}g({\bf x},{\bf y}_{2})\|}{\|{\bf y}_{1}-{\bf y}_{2}\|}\leq L_{g_{xy}},\quad\hbox{uniformly in ${\bf x}$},$ $$

$$ $\displaystyle\frac{\|\nabla^{2}_{yy}g({\bf x},{\bf y}_{1})-\nabla^{2}_{yy}g({\bf x},{\bf y}_{2})\|}{\|{\bf y}_{1}-{\bf y}_{2}\|}\leq L_{g_{yy}},\quad\hbox{uniformly in ${\bf x}$},$ $$

$$ $\displaystyle\frac{\|\nabla^{2}_{xy}g({\bf x}_{1},{\bf y})-\nabla^{2}_{xy}g({\bf x}_{2},{\bf y})\|}{\|{\bf x}_{2}-{\bf x}_{2}\|}\leq{\bar{L}}_{g_{xy}},\quad\hbox{uniformly in ${\bf x}$},$ $$

$$ $\displaystyle\frac{\|\nabla^{2}_{yy}g({\bf x}_{1},{\bf y})-\nabla^{2}_{yy}g({\bf x}_{2},{\bf y})\|}{\|{\bf x}_{2}-{\bf x}_{2}\|}\leq{\bar{L}}_{g_{yy}},\quad\hbox{uniformly in ${\bf x}$},$ $$

Convexity: For any $x\in\mathbb{R}^{d_{x}}$, $g(x,\cdot)$ is strongly convex in ${\bf y}$ with modulus $\mu_{g}>0$. Hessian boundedness: $\|\nabla^{2}_{xy}g({\bf x},{\bf y})\|\leq C_{g_{xy}}$, uniformly in ${\bf x}$ and ${\bf y}$.

It is known that under these assumptions, we have

###### Lemma J.1 (Lemma 2.2 from Ghadimi & Wang ( 2018 ) ) .

$$ $\displaystyle\|{\overline{\nabla}}_{x}f(x,y)-\nabla\ell(x)\|\leq L(y^{*}(x)-y\|,\quad\|y^{*}(x_{1})-y^{*}(x_{2})\|\leq L_{y}\|x_{1}-x_{2}\|,$ (14) $\displaystyle\|\nabla\ell(x_{1})-\nabla\ell(x_{2})\|=\|\nabla f(x_{1},y^{*}(x_{1}))-\nabla f(x_{2},y^{*}(x_{2}))\|\leq L_{f}\|x_{1}-x_{2}\|,$ (15) $$

with

$$ $\displaystyle L:=L_{f_{x}}+\frac{L_{f_{y}}C_{g_{xy}}}{\mu_{g}}+C_{f_{y}}\left(\frac{L_{g_{xy}}}{\mu_{g}}+\frac{L_{g_{yy}}C_{g_{xy}}}{\mu^{2}_{g}}\right),$ $\displaystyle L_{f}:=L_{f_{x}}+\frac{({\bar{L}}_{f_{y}}+L)C_{g_{xy}}}{\mu_{g}}+C_{f_{y}}\left(\frac{{\bar{L}}_{g_{xy}}}{\mu_{g}}+\frac{{\bar{L}}_{g_{yy}}C_{g_{xy}}}{\mu^{2}_{g}}\right),L_{y}:=\frac{C_{g_{xy}}}{\mu_{g}}.$ $$

We also make assumptions on the random approximation of the gradient. The widely used assumptions in optimization literature, see e.g.  Hong et al. (2023); Li et al. (2024); Pan et al. (2025), are usually in the following form

###### Assumption 3 .

There are two positive constants $\sigma_{f}$ and $\sigma_{g}$, and a nonincreasing sequence $\{b_{k}\}_{k\geq 0}$ such that,

$$ $\displaystyle\mathbb{E}[D_{g}^{k}|{\mathcal{F}}_{k}]=\nabla g(x^{k},y^{k}),\quad\mathbb{E}[D_{g}^{k}|{\mathcal{F}}^{\prime}_{k}]=\nabla g(x^{k},y^{k+1})+B_{k},\|B_{k}\|\leq b_{k},$ (16) $\displaystyle\mathbb{E}[\|D_{g}^{k}-\nabla_{y}g(x^{k},y^{k})\|^{2}|{\mathcal{F}}_{k}]\leq\sigma_{g}^{2}[1+\|\nabla_{y}g(x^{k},y^{k})\|^{2}],$ (17) $\displaystyle\mathbb{E}[\|D_{f}^{k}-{\overline{\nabla}}_{x}f(x^{k},y^{k+1})-B_{k}\|^{2}|{\mathcal{F}}^{\prime}_{k}]\leq\sigma_{f}.$ (18) $$

In this paper, the function and derivative approximation are realized through deep neural networks. Therefore, it is desired to connect the constants $\sigma_{f}$, $\sigma_{g}$, and $\{b_{k}\}_{k\geq 0}$ to network parameters, such as their widths and depths, which can be adjusted to ensure the assumptions hold. This type of quantitative relation between the effectiveness of the approximation and network parameters has been carried out recently, see, e.g. Lu et al. (2021) and Belomestny et al. (2023). Summarizing their results, for neural networks of width $W$ and depth $D$, the following can be reasonably assumed.

###### Assumption 4 .

There are two positive constants $\sigma_{f}$ and $\sigma_{g}$, and a nonincreasing sequence $\{b_{k}\}_{k\geq 0}$ such that,

$$ $\displaystyle\mathbb{E}[D_{g}^{k}|{\mathcal{F}}_{k}]=\nabla g(x^{k},y^{k}),\quad\mathbb{E}[D_{g}^{k}|{\mathcal{F}}^{\prime}_{k}]=\nabla g(x^{k},y^{k+1})+B_{k},\|B_{k}\|\leq b_{k}{\mathcal{S}},$ (19) $\displaystyle\mathbb{E}[\|D_{g}^{k}-\nabla_{y}g(x^{k},y^{k})\|^{2}|{\mathcal{F}}_{k}]\leq\sigma_{g}^{2}[1+\|\nabla_{y}g(x^{k},y^{k})\|^{2}]{\mathcal{S}}^{2},$ (20) $\displaystyle\mathbb{E}[\|D_{f}^{k}-\nabla g(x^{k},y^{k+1})-B_{k}\|^{2}|{\mathcal{F}}^{\prime}_{k}]\leq\sigma_{f}{\mathcal{S}}^{2}.$ (21) $$

with ${\mathcal{S}}:=W^{-2s/(d_{x}+d_{y})}D^{-2s/(d_{x}+d_{y})}$.

###### Assumption 5 .

For any fixed ${\bf x}\in\mathbb{R}^{d_{x}}$, the inner objective function $g({\bf x},\cdot)$ satisfies the Restricted Secant Inequality with respect to its minimizer ${\bf y}^{*}({\bf x})$. That is, for all ${\bf y}\in\mathbb{R}^{d_{y}}$, we have:

$$ $\langle\nabla_{\bf y}g({\bf x},{\bf y}),{\bf y}-{\bf y}^{*}({\bf x})\rangle\geq\mu_{g}\|{\bf y}-{\bf y}^{*}({\bf x})\|^{2}$ $$

where $\mu_{g}>0$ is a constant.

###### Assumption 1 .

###### Assumption 2 .

###### Lemma J.1 (Lemma 2.2 from Ghadimi & Wang ( 2018 ) ) .

###### Assumption 3 .

###### Assumption 4 .

###### Assumption 5 .

### J.2 Convergence analysis of the algorithms for inner optimization

The update for the inner optimization (SGD) takes the following form,

$$ $\displaystyle y^{k+1}=y^{k}-\beta_{k}D^{k}_{g},$ (22) $$

where $D^{k}_{g}$ is a random variable approximating $\nabla_{y}g(x^{k},y^{k})$.

The goal is to estimate $\mathbb{E}[\|y^{k+1}-y^{*}(x^{k})\|^{2}|{\mathcal{F}}_{k}]$, for that, we have,

$$ $\displaystyle\mathbb{E}[\|y^{k+1}-y^{*}(x^{k})\|^{2}|{\mathcal{F}}_{k}]=$ $\displaystyle\mathbb{E}[\|y^{k}-y^{*}(x^{k})-\beta_{k}D^{k}_{g}\|^{2}|{\mathcal{F}}_{k}]$ $\displaystyle=$ $\displaystyle\mathbb{E}[\|y^{k}-y^{*}(x^{k})\|^{2}-2\beta_{k}D^{k}_{g}[y^{k}-y^{*}(x^{k})]+\beta^{2}_{k}\|D^{k}_{g}\|^{2}|{\mathcal{F}}_{k}].$ $$

From restricted secant inequality (RSI), we know that

$$ $\mathbb{E}[D^{k}_{g}[y^{k}-y^{*}(x^{k})]|{\mathcal{F}}_{k}]\stackrel{{\scriptstyle ind}}{{=}}\nabla_{y}g(x^{k},y^{k})[y^{k}-y^{*}(x^{k})]\stackrel{{\scriptstyle RSI}}{{\geq}}\mu_{g}\|y^{k}-y^{*}(x^{k})\|^{2}.$ $$

Plug this back into the equation above, we have,

$$ $\displaystyle\mathbb{E}[\|y^{k+1}-y^{*}(x^{k})\|^{2}|{\mathcal{F}}_{k}]\leq$ $\displaystyle\mathbb{E}(1-2\beta_{k})\|y^{k}-y^{*}(x^{k})\|^{2}+\beta^{2}_{k}\mathbb{E}[\|D^{k}_{g}\|^{2}|{\mathcal{F}}_{k}].$ $$

Now, let us examine the last term,

$$ $\displaystyle\mathbb{E}[\|D^{k}_{g}\|^{2}|{\mathcal{F}}_{k}]$ $\displaystyle=\mathbb{E}[\|D^{k}_{g}\|^{2}-\|\nabla_{y}g(x^{k},y^{k})\|^{2}|{\mathcal{F}}_{k}]+\|\nabla_{y}g(x^{k},y^{k})\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(1)}}{{=}}\mathbb{E}[\|D^{k}_{g}-\nabla_{y}g(x^{k},y^{k})\|^{2}|{\mathcal{F}}_{k}]+\|\nabla_{y}g(x^{k},y^{k})\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(2)}}{{\leq}}\sigma_{g}^{2}{\mathcal{S}}^{2}+\sigma_{g}^{2}{\mathcal{S}}^{2}\|\nabla_{y}g(x^{k},y^{k})\|^{2}+\|\nabla_{y}g(x^{k},y^{k})\|^{2}$ $\displaystyle=\sigma_{g}^{2}{\mathcal{S}}^{2}+(1+\sigma_{g}^{2}{\mathcal{S}}^{2})\|\nabla_{y}g(x^{k},y^{k})\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(3)}}{{=}}\sigma_{g}^{2}{\mathcal{S}}^{2}+(1+\sigma_{g}^{2}{\mathcal{S}}^{2})\|\nabla_{y}g(x^{k},y^{k})-\nabla_{y}g(x^{k},y^{*}(x^{k}))\|^{2}$ $\displaystyle\stackrel{{\scriptstyle(4)}}{{=}}\sigma_{g}^{2}{\mathcal{S}}^{2}+(1+\sigma_{g}^{2}{\mathcal{S}}^{2})L_{g}^{2}\|y^{k}-y^{*}(x^{k})\|^{2},$ $$

where (1) is due to the fact that $\mathbb{E}[D^{k}_{g}]\|=\nabla_{y}g(x^{k},y^{k})$; (2) is due to the assumption on variance of $D^{k}_{g}$ in Assumption [4](https://arxiv.org/html/2508.07452v2#Thmasm4); (3) is due to the fact that $y^{*}(x^{k})$ is the stationary point, hence $\nabla_{y}g(x^{k},y^{*}(x^{k}))=0$; and (4) is due to the gradient Lipschitz assumption.

Now, with the assumption that $\beta_{k}$ goes to zero, as $k\to\infty$, $\beta_{k}(1+\sigma_{g}^{2})\leq\mu_{k}$ always holds for sufficiently large $k$. Hence, we have,

$$ $\displaystyle\mathbb{E}[\|y^{k+1}-y^{*}(x^{k})\|^{2}|{\mathcal{F}}_{k}]\leq$ $\displaystyle(1-\beta_{k})\|y^{k}-y^{*}(x^{k})\|^{2}+\beta^{2}_{k}\sigma_{g}^{2}.$ $$

Next, to form a recursion, split $\|y^{k}-y^{*}(x^{k})\|^{2}$ into $\|y^{k}-y^{*}(x^{k-1})\|^{2}+\|y^{*}(x^{k-1})-y^{*}(x^{k})\|^{2}$. Thus,

$$ $\displaystyle\|y^{*}(x^{k-1})-y^{*}(x^{k})\|^{2}\leq L_{y}^{2}\|x^{k}-x^{k-1}\|^{2}=L_{y}^{2}\|\alpha_{k-1}D_{f}^{k-1}\|^{2}=\alpha^{2}_{k-1}L_{y}^{2}\|D_{f}^{k-1}\|^{2},$ $$

where the first inequality follows from the Lipschitz continuity of ${\bf y}^{*}(\cdot)$ (Lemma [J.1](https://arxiv.org/html/2508.07452v2#A10.Thmlem1)), and the equality follows directly from the definition of the unconstrained update rule in Equation equation [13](https://arxiv.org/html/2508.07452v2#A10.E13).
Therefore,

$$ $\displaystyle\mathbb{E}[\|y^{k+1}-y^{*}(x^{k})\|^{2}|{\mathcal{F}}_{k}]\leq$ $\displaystyle\mathbb{E}(1-\beta_{k})\|y^{k}-y^{*}(x^{k-1})\|^{2}+\beta^{2}_{k}\sigma_{g}^{2}+a^{2}_{k-1}L_{y}^{2}\mathbb{E}[\|D_{f}^{k-1}\|^{2}|{\mathcal{F}}_{k}].$ $$

Hence, we have,

$$ $\displaystyle\mathbb{E}[\|y^{k+1}-y^{*}(x^{k})\|^{2}|{\mathcal{F}}_{k}]$ $\displaystyle\leq$ $\displaystyle\mathbb{E}(1-\beta_{k}+a^{2}_{k-1}L_{y}^{2}(1+\sigma_{g}^{2}{\mathcal{S}}^{2})L_{g}^{2})\|y^{k}-y^{*}(x^{k-1})\|^{2}+\beta^{2}_{k}\sigma_{g}^{2}+a^{2}_{k-1}L_{y}^{2}\sigma_{g}^{2}{\mathcal{S}}^{2}.$ $$

As we know $\alpha_{k}/\beta_{k}\to 0$, we can see that $(1-\beta_{k}+a^{2}_{k-1}L_{y}^{2}B)$ will be the uniform contraction factor, and $\beta^{2}_{k}\sigma_{g}^{2}+a^{2}_{k-1}L_{y}^{2}A$ is a correction term also tends to zero, therefore, $\mathbb{E}[\|y^{k+1}-y^{*}(x^{k})\|^{2}$ diminished to zero, and the rate can also be quantified, especially in terms of the size of the neural networks.

### J.3 Convergence analysis of the algorithms for outer optimization

From the unconstrained update rule in Equation equation [13](https://arxiv.org/html/2508.07452v2#A10.E13), we have:

$$ $\displaystyle\|x^{k+1}-x^{*}\|^{2}=\|x^{k}\alpha_{k}D_{f}^{k}-x^{*}\|^{2}=\|x^{k}-x^{*}\|^{2}-2\alpha_{k}\langle D_{f}^{k},x^{k}-x^{*}\rangle+\alpha^{2}_{k}\|Df^{k}\|^{2}.$ $$

From the Assumption [4](https://arxiv.org/html/2508.07452v2#Thmasm4) on the random variable $D_{f}^{k}$, we can see that,

$$ $\displaystyle\mathbb{E}[\langle D_{f}^{k},x^{k}-x^{*}\rangle|{\mathcal{F}}_{k}]=$ $\displaystyle\langle\nabla_{x}f(x^{k},y^{k+1})+B_{k},x^{k}-x^{*}\rangle$ $\displaystyle=$ $\displaystyle\langle\nabla\ell(x^{k}),x^{k}-x^{*}\rangle+\langle\nabla_{x}f(x^{k},y^{k+1})-\nabla\ell(x^{k})+B_{k},x^{k}-x^{*}\rangle.$ $$

Hence, we have,

$$ $\displaystyle\mathbb{E}[\|x^{k+1}-x^{*}\|^{2}|{\mathcal{F}}_{k}]\leq$ $\displaystyle\|x^{k}-x^{*}\|^{2}-2\alpha_{k}\langle\nabla\ell(x^{k}),x^{k}-x^{*}\rangle+\alpha^{2}_{k}\mathbb{E}\|Df^{k}\|^{2}|{\mathcal{F}}_{k}]$ $\displaystyle-2\alpha_{k}\langle\nabla_{x}f(x^{k},y^{k+1})-\nabla\ell(x^{k})+B_{k},x^{k}-x^{*}\rangle.$ $$

Restricted secant inequality implies that,

$$ $\displaystyle\langle\nabla\ell(x^{k}),x^{k}-x^{*}\rangle=\langle\nabla\ell(x^{k})-\nabla\ell(x^{*}),x^{k}-x^{*}\rangle\geq\mu_{\ell}\|x^{k}-x^{*}\|^{2}.$ $$

We then have,

$$ $\displaystyle\mathbb{E}[\|x^{k+1}-x^{*}\|^{2}|{\mathcal{F}}_{k}]\leq$ $\displaystyle(1-2\alpha_{k}\mu_{\ell})\|x^{k}-x^{*}\|^{2}-2\alpha_{k}\langle\nabla_{x}f(x^{k},y^{k+1})-\nabla\ell(x^{k})+B_{k},x^{k}-x^{*}\rangle$ $\displaystyle+\alpha^{2}_{k}\mathbb{E}\|Df^{k}\|^{2}|{\mathcal{F}}_{k}]$ $\displaystyle\stackrel{{\scriptstyle(1)}}{{\leq}}$ $\displaystyle(1-\alpha_{k}\mu_{\ell})\|x^{k}-x^{*}\|^{2}+\frac{\alpha_{k}}{\mu_{\ell}}\|\nabla_{x}f(x^{k},y^{k+1})-\nabla\ell(x^{k})+B_{k}\|^{2}$ $\displaystyle+\alpha^{2}_{k}\mathbb{E}\|Df^{k}\|^{2}|{\mathcal{F}}_{k}]$ $\displaystyle\stackrel{{\scriptstyle(2)}}{{\leq}}$ $\displaystyle(1-\alpha_{k}\mu_{\ell})\|x^{k}-x^{*}\|^{2}+\frac{2\alpha_{k}}{\mu_{\ell}}[L^{2}\|y^{k+1}-y^{*}(x^{k})\|^{2}+b_{k}{\mathcal{S}}]$ $\displaystyle+\alpha^{2}_{k}\mathbb{E}\|Df^{k}\|^{2}|{\mathcal{F}}_{k}],$ $$

where (1) is the result of completing a square, and (2) again follows from Lemma [J.1](https://arxiv.org/html/2508.07452v2#A10.Thmlem1). Hence, we can have uniformly bounded constants $\pi,\zeta>0$ such that

$$ $\displaystyle\mathbb{E}[\|x^{k+1}-x^{*}\|^{2}|{\mathcal{F}}_{k}]\leq$ $\displaystyle(1-\alpha_{k}\mu_{\ell})\|x^{k}-x^{*}\|^{2}+\alpha_{k}\pi L^{2}\|y^{k+1}-y^{*}(x^{k})\|^{2}+\zeta\alpha^{2}_{k}.$ $$

Incorporating the above estimation into the inner and outer optimization, following a similar argument to Theorem 1 in Hong et al. (2023), we can reach the following conclusion.

###### Theorem J.1 .

Under Assumptions [1](https://arxiv.org/html/2508.07452v2#Thmasm1),  [2](https://arxiv.org/html/2508.07452v2#Thmasm2) and [4](https://arxiv.org/html/2508.07452v2#Thmasm4), when $\alpha_{k}\leq c_{0}\beta_{k}^{3/2}$ and $\beta_{k}\leq c_{1}\alpha_{k}^{2/3}$ with constants $c_{0},c_{1}>0$, we have,

$$ $\displaystyle\mathbb{E}[\|x^{k}-x^{*}\|^{2}]\leq$ $\displaystyle\left\{\prod_{i=0}^{k-1}((1-\alpha_{i}\mu_{\ell})\left[\mathbb{E}[\|x^{0}-x^{*}\|^{2}]+\pi\mathbb{E}[\|y^{0}-y^{*}(x^{0})\|^{2}]\right]+\zeta\alpha_{k-1}^{2/3}\right\},$ (23) $\displaystyle\mathbb{E}[\|y^{k}-y^{*}(x^{k-1})\|^{2}]\leq$ $\displaystyle\sigma_{g}^{2}({\mathcal{S}}^{2}+3)\left[\prod_{i=0}^{k-1}\left(1-\frac{\beta_{i}\mu_{g}}{4}\right)\mathbb{E}[\|y^{0}-y^{*}(x^{0})\|^{2}]+\beta_{k-1}\right].$ (24) $$

The consequence of the theorem is that in the case of $\alpha_{k},\beta_{k}\to 0$, as $k\to\infty$, we know that the two quantities on the left-hand side will diminish to zero.

###### Theorem J.1 .

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: N/A
soundness: 2
strengths: A. The submission focuses on an important problem of reinforcement learning, which is representation learning.

B. The reading flow is good, and the method is clearly presented.

C. The method is simple to implement as it simply uses two different learning rates for the encoder and the predictor, and updates the encoder and the predictor sequentially.
summary: The work introduces a new reinforcement learning method for online representation learning. The core idea is to divide the function approximator into two parts: one for learning a useful representation and another for learning to predict future returns. Each part is treated as an agent playing a Stackelberg game, maximizing a different objective while sharing the same Q-function predictor. This leads to a bi-level optimization problem, which they address by enforcing a faster learning rate for the encoder than for the predictor. Experiments on various tasks are presented to report the method's performance.
weaknesses: I. Some choices are suboptimal or unjustified.

   a. The choice of minimizing the variance of the temporal difference error is problematic. Indeed, the proposed method faces the double-sampling issue, yet it is not mentioned. This means the follower's objective function is biased. More specifically, $\mathbb{E}[\delta_j^2] \neq \mathbb{E}[\delta_j]^2$. Therefore, the presented algorithm will suffer from this issue in stochastic MDPs, where the variance of the temporal difference error exists, even for the optimal Q-function. Importantly, all experiments are made in deterministic environments. I suggest setting the follower's objective function to the temporal-difference error, which is an unbiased estimate. 

   b. It is claimed that "This promotes stability in the learning process, a trait for mitigating the risk of divergence that was first formally demonstrated in foundational work by Baird (1995)" in Line 218. However, this claim is never justified by an empirical study on the Baird counter-example. I suggest evaluating the method on this counter-example.

   c. The choice of the follower and the leader is only justified by an empirical comparison. Adding a theoretical or an intuitive argument would strengthen the motivation behind this work.

II. The empirical analysis remains weak and can be improved.

   a. The method relies on multiple parallel environments. While this allows for faster training, it is an unrealistic assumption for real-world applications, and when a simulator is available, on-policy methods will be privileged. To fully assess the potential of the presented method, it would be useful to also have experiments in which only one environment is used at a time.

   b. As mentioned in I. a., it is unlikely that the presented method performs well in stochastic environments. Therefore, I suggest evaluating SCORER in stochastic environments, for example, with sticky actions.

   c. Two cofounding factors can be responsible for the performance increase. (i) The learning rate schedules are not fully disclosed. It would be more convincing to compare the proposed approach with the baselines using the same learning rate schedules (the one of the encoder and the one of the predictor). (ii) The gradients are clipped for the proposed approach but not for the baseline. Ablating this choice would help understand the benefit of SCORER.

   d. An analysis of the learning dynamics would provide more insights. For example, reporting the parameter norm and the srank would help better understand the method.

   e. While the interquantile-mean is reported as suggested by [1], the recommended way of reporting the confidence intervals is not followed, and no justification is provided.

   f. In Table 1, the maximum IQM performance is reported. This choice is uncommon and not motivated. 

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." NeurIPS 2021.

### Remarks

1. The font size of every figure can be larger. Replacing the tables with visual plots would facilitate the analysis of the results.
 
2. Two citations can be added when speaking about the fact that learning appropriate representations helps with sample efficiency: 

   [1] Dabney, Will, et al. "The value-improvement path: Towards better representations for reinforcement learning." AAAI 2021.

   [2] Vincent, Théo, et al. "Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning." TMLR 2025.

3. The batch seems to be missing in equation 5.

4. In Figure 4, the left and middle Figures are swapped.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. Why were full-scale Atari experiments not included, and do you have any preliminary results on high-dimensional visual domains?
2. Can you provide an ablation study on different follower-to-leader learning rate ratios (e.g., 2:1, 10:1, 20:1) beyond the 5:1 used in all experiments?
3. Have you tested a simpler baseline where a standard monolithic network uses higher learning rates for encoder layers without network separation or stop-gradients?
soundness: 2
strengths: - The paper tackles a real issue in deep Q-learning where representation learning and value learning are tightly coupled, causing training instability.
- The learning curves show SCORER produces smoother, more stable training compared to baselines. The curves don't have as many sudden drops or erratic behavior.
- The appendix provides mathematical analysis based on two-timescale stochastic approximation. The convergence guarantees and game-theoretic formulation seem reasonable.
- Results are strong in some cases. MinAtar Breakout shows roughly 3x improvement over baseline DQN. MiniGrid Four Rooms is particularly impressive - baseline R2D2 completely fails while SCORER achieves 97% success
summary: The paper proposes SCORER - a method that splits deep Q-learning into two separate networks that learn at different speeds to address training instability. The representation network updates faster and minimizes Bellman error variance, while the Q-function updates slower for stability. They test on MinAtar and MiniGrid environments where some tasks show good improvements while others show more modest gains.
weaknesses: - Experiments only cover 4 simple MinAtar environments and basic gridworld tasks. No full-scale Atari with high-resolution images. This is a problem because representation learning challenges are fundamentally different at scale, and we can't tell if benefits hold with deep convolutional networks.
- Outside Breakout and Four Rooms, gains are small. Asterix is basically the same as baseline, Freeway has identical final performance (just faster), SpaceInvaders shows about 10-15% improvement. The claim of "consistently improves final performance" oversells what the data shows.
- The method uses 5:1 ratio (follower:leader learning rates) but there's no ablation testing 2:1, 10:1, or other values. This is important - we need to know how sensitive the method is and whether 5:1 is actually optimal.
- A monolithic network with higher learning rate for encoder layers (without network separation or stop-gradients) would test whether the complex game structure is necessary or if just updating representations faster works. This might not be essential since current ablations show hierarchy matters, but it would be useful to rule out simpler explanations.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: My main question relating to my evaluation of the work is as follows:

MQ.1: For the purposes of understanding, how do you see SCORER in relation to other approaches to deep RL that decompose the monolith into a variety of subcomponents? Can you comment on the merits of framing this as perceiver/actor in a game, rather than something else (such as the Anand and Precup approach)?

Separately, I have a one other question (OQ.X), though it is less critical to my overall appraisal:

OQ.1: How do you foresee SCORER extending to another similar algorithmic templates, such as Actor-Critic, or even cases when value is not learned at all but perhaps just a policy?
soundness: 3
strengths: - This is a very clean paper. The scope is well defined, the question is well-posed, all of the needed ingredients are self-contained within the paper, and the solution (two-level optimization of a Stackelberg game) is tidy. In other words, the paper does exactly what it sets out to do in the intro.
- The experimental study is well designed, with appropriately chosen baselines, a variety of environments, and suitable ablations to explore various design choices (such as which player should be the leader vs. follower in Fig 4).
- The writing is clear and does a good job of motivating the various design choices. I especially appreciated the discussion around whether extraneous other signals should be incorporated into the study such as standard self-supervised objectives. I found the question, the main method, and the findings to be well-motivated and communicated effectively.
- The convergence result in the appendix adds a firm theoretical support to the approach.
- Lastly, the idea at the heart of the paper is relatively simple: if we decouple algorithmic components, we can treat them as players in a game, and solve the game.
summary: This paper presents a new framing of value-based deep RL methods in terms of a two-player game between a representation player and a controller player. The motivation stems from the apparent instability that emerges in RL when a learner tries to simultaneously learn its representation, and its control policy. The core technical idea is to treat the learner's estimate of the Q-function as the leader, while a perception network is treated as the follower of a Stackelberg game. This framing yields "Stackelberg Coupled Representation and Reinforcement Learning" (SCORER), a simple value-based method for deep RL based on the principles of Stackelberg games. The game is solved using a two-level form of gradient descent, which yields a stable relationship between the representation and the value. Critically, this game only involves the exchange of the standard signals of the RL problem (based around MSBE) and does not allow incorporation of extraneous signals (such as intrinsic motivation, entropy bonuses, and so on). This methodological choice allows the work to focus purely on the game dynamics in a simple setting. The paper's primary evidence to support its central claims comes from a broad experimental study contrasting the performance of SCORER with standard value-based methods (DQN, DDQN, and so on) across a variety of standard tasks.
weaknesses: - The general premise of mitigating challenges of deep RL through decomposing the monolithic policy network or value network into different components has been studied extensively. The proposal to treat a two-decomposition sub-system as engaged in a Stackelberg game is, to my knowledge, new, though I do believe a deeper understanding of why this specific decomposition is valuable would strengthen the work. As two concrete examples:

1. In Anand and Precup (2023), they propose to decompose the estimate of the value function into a "permanent" and "transient" component. These two components yields two separate networks, one for learning the permanent and one for the transient. They key difference between the two is that they are updated at different schedules. Then, the overall value function is determined by taking a combination of the two. I believe there is a heavy conceptual overlap in this approach and SCORER, and I wonder: if someone were to want to deploy a value-based method for RL, when should they use SCORER as approach to the approach of Anand and Precup?

2. Second, the Actor-Critic architecture is naturally decomposed into separate pieces to enhance stability. Indeed, a recent paper by Garcin et al (2025) has explicitly looked at the learned representations in each of the Actor and Critic, and found that the two networks tend to specialize their representation differently. First, I believe applying this kind of analysis to SCORER could be beneficial to shed further light on what specifically is being learned by the representation and policy in each of the ablations (and in the general setup). Second, it is so far again unclear what the precise benefit of this proposed decomposition is compared to something like Actor-Critic. I see this is noted in the discussion (applying SCORER to AC algorithms), though I believe this is an important point to understand about the space.

- Otherwise, no obvious weaknesses stand out. This is a clean paper with a tight focus that delivers on what it sets out to do.


Some typos and writing suggestions:

- Some repeated text at line 166 or so: "The defining characteristic is the leader’s ability to anticipate and influence the follower’s decision by committing to a strategy first. The defining characteristic is the leader’s ability to anticipate the follower’s best response and commit to a strategy that steers the game toward a favorable equilibrium.".

- Convergence analysis in Appendix J is really nice. If you can find space, I would love to see this in the main text, even it is just a colloquial theorem statement with the assumptions and proof in the appendix. On that note, I would suggest stating the convergence result as a mathematical theorem or proposition, even if it is in the appendix, to give you a chance to succinctly assert the result with all needed quantifiers and variables in one place.

References:
- Anand, N., & Precup, D. Prediction and control in continual reinforcement learning. NeurIPS 2023.
- Garcin, S., McInroe, T., Castro, P. S., Panangaden, P., Lucas, C. G., Abel, D., & Albrecht, S. V. Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning. ICLR 2025.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Do you consider DQN and Dueling DQN as good baselines and the results naturally generalize to strong baselines such as Rainbow? 

2. Can you add studies or discussions on actor-critic methods?
soundness: 3
strengths: 1. The paper is well-motivated by the Stackelberg game theory to reformulate the RL optimization problem, separating learning dynamics between representation and value estimation.

2. The role assignments (control as leader, perception as follower) and the choice of Bellman error variance for the follower are both theoretically justified and empirically validated.
summary: This paper introduces Stackelberg Coupled Representation and Reinforcement Learning, a novel value-based RL framework inspired by game theory. The key insight is to decouple representation and value learning into a hierarchical leader-follower game: the Q-function (control network) is the leader updated on a slow timescale, while the perception network (state encoder) is the follower learning rapidly to minimize Bellman error variance under the leader’s strategy. This two-timescale design aims to solve the instability issue (deadly triad) seen in monolithic off-policy deep Q-learning, improving sample efficiency and performance without increased model complexity. Experiments on MinAtar, Atari, and MiniGrid show consistent, substantial improvements over standard DQN and its variants.
weaknesses: 1. While computational cost is minimal, the framework introduces new algorithmic complexity (two-level optimization, choice of timescale schedules) that requires parameter tuning.

2. The method is centered entirely on discrete value-based methods. Extension to actor-critic or continuous-control algorithms is absent.


Decision: Accept (Poster)