Universal Value-Function Uncertainties

## Abstract

Abstract Estimating epistemic uncertainty in value functions is a crucial challenge for many aspects of reinforcement learning (RL), including efficient exploration, safe decision-making, and offline RL. While deep ensembles provide a robust method for quantifying value uncertainty, they come with significant computational overhead. Single-model methods, while computationally favorable, often rely on heuristics and typically require additional propagation mechanisms for myopic uncertainty estimates. In this work we introduce universal value-function uncertainties (UVU), which, similar in spirit to random network distillation (RND), quantify uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. Unlike RND, UVU errors reflect policy-conditional value uncertainty , incorporating the future uncertainties any given policy may encounter. This is due to the training procedure employed in UVU: the online network is trained using temporal difference learning with a synthetic reward derived from the fixed, randomly initialized target network. We provide an extensive theoretical analysis of our approach using neural tangent kernel (NTK) theory and show that in the limit of infinite network width, UVU errors are exactly equivalent to the variance of an ensemble of independent universal value functions. Empirically, we show that UVU achieves equal performance to large ensembles on challenging multi-task offline RL settings, while offering simplicity and substantial computational savings.

## 1 Introduction

Deep reinforcement learning (RL) has emerged as an essential paradigm for addressing difficult sequential decision-making problems (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019) but a more widespread deployment of agents to real-world applications remains challenging. Open problems such as efficient exploration, scalable offline learning and safety pose persistent obstacles to this transition. Central to these capabilities is the quantification of *epistemic uncertainty*, an agent’s uncertainty due to limited data. In the context of RL, uncertainty estimation relating to the *value function* is of particular importance as it reflects uncertainty about long-term consequences of actions.

However, computationally tractable estimation of value-function uncertainty remains a challenge. Bayesian RL approaches, both in its model-based (Ghavamzadeh et al., 2015) and model-free (Dearden et al., 1998) flavors, typically come with sound theoretical underpinnings but face significant computational hurdles due to the general intractability of posterior inference. Theoretical guarantees of the latter are moreover often complicated by the use of training procedures like temporal difference (TD) learning with bootstrapping. Conversely, deep ensembles (Lakshminarayanan et al., 2017) have emerged as a reliable standard for practical value uncertainty estimation in deep RL (Osband et al., 2016; Chen et al., 2017). Empirically, independently trained value functions from random initialization provide effective uncertainty estimates that correlate well with true estimation errors. Although in general more tractable than full posterior inference, this approach remains computationally challenging for larger models where a manyfold increase in computation and memory severely limits scalability. Various single-model approaches like random network distillation (RND) (Burda et al., 2019), pseudo counts (Bellemare et al., 2016) or intrinsic curiosity (Pathak et al., 2017) efficiently capture *myopic* epistemic uncertainty but require additional propagation mechanisms to obtain value uncertainties (O’Donoghue et al., 2018; Janz et al., 2019; Zhou et al., 2020) and often elude a thorough theoretical understanding. We conclude that there persists a lack of computationally efficient single-model approaches with the ability to directly estimate policy-dependent value uncertainties with a strong theoretical foundation.

To this end, we introduce *universal value-function uncertainties* (UVU), a novel method designed to estimate epistemic uncertainty of value functions for any given policy using a single-model architecture. Similar in spirit to the well-known RND algorithm, UVU quantifies uncertainty through a prediction error between an online learner $u$ and a fixed, randomly initialized target network $g$. Crucially, and in contrast to the regression objective of RND, UVU optimizes its online network $u$ using temporal difference (TD) learning with a synthetic reward $r_{g}$ generated entirely from the target network $g$. By construction, the reward $r_{g}$ implies a value learning problem to which the target function $g$ itself is a solution, forcing the online learner $u$ to recover $g$ through minimization of TD losses. UVU then quantifies uncertainty as the squared prediction error between online learner and fixed target function. Unlike previous methods, our design requires no training of multiple models (e.g., ensembles) nor separate value and uncertainty models (e.g., RND, ICM). Furthermore, we design UVU as a universal policy-conditioned model (comparable to universal value function approximators (Schaul et al., 2015)), that is, it takes as input a state, action, and policy encoding and predicts the epistemic uncertainty associated with the value function for the encoded policy.

A key contribution of our work is a thorough theoretical analysis of UVU using the framework of neural tangent kernels (NTK) (Jacot et al., 2018). Specifically, we characterize the learning dynamics of wide neural networks with TD losses and gradient descent to obtain closed-form solutions for the convergence and generalization behavior of neural network value functions. In the limit of infinite network width, we then show that prediction errors generated by UVU are equivalent to the variance of an ensemble of universal value functions, both in expectation and with finite sample estimators.

We validate UVU empirically on an offline multi-task benchmark from the minigrid suite where agents are required to reject tasks they cannot perform to achieve maximal scores. We show that UVU’s uncertainty estimates perform comparably to large deep ensembles, while drastically reducing the computational footprint.

## 2 Preliminaries

We frame our work within the standard Markov Decision Process (MDP) (Bellman, 1957) formalism, defined by the tuple $(\mathcal{S},\mathcal{A},\mathcal{R},\gamma,P,\mu)$. Here, $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathscr{P}(\mathbb{R})$ is the distribution of immediate rewards, $\gamma\in[0,1)$ is the discount factor, $P:\mathcal{S}\times\mathcal{A}\rightarrow\mathscr{P}(\mathcal{S})$ is the transition probability kernel, and $\mu:\mathscr{P}(\mathcal{S})$ is the initial state distribution. An RL agent interacts with this environment by selecting actions according to a policy $\pi:\mathcal{S}\rightarrow\mathscr{P}(\mathcal{A})$. At each timestep $t$, the agent is in state $S_{t}$, takes action $A_{t}\sim\pi(\cdot|S_{t})$, receives a reward $R_{t}\sim\mathcal{R}(\cdot|S_{t},A_{t})$, and transitions to a new state $S_{t+1}\sim P(\cdot|S_{t},A_{t})$. We quantify the merit of taking actions $A_{t}=a$ in state $S_{t}=s$ and subsequently following policy $\pi$ by the action-value function, or Q-function $Q^{\pi}:\mathcal{S}\times\mathcal{A}\xrightarrow{}\mathbb{R}$, which accounts for the cumulative discounted future rewards and adheres to a recursive consistency condition described by the Bellman equation

$$ $\displaystyle Q^{\pi}(s,a)=\mathbb{E}_{\mathcal{R},\pi,P}[R_{0}+\gamma Q^{\pi} (S_{1},A_{1})|S_{0}=s,A_{0}=a].$ (1) $$

The agent’s objective then is to maximize expected returns $J(\pi)=\mathbb{E}_{S_{0}\sim\mu,A_{0}\sim\pi(\cdot|S_{0})}[Q^{\pi}(S_{0},A_{0})]$.

Often, we may be interested in agents capable of operating a variety of policies to achieve different goals. Universal value function approximators (UVFAs) (Schaul et al., 2015) address this by conditioning value functions additionally on an encoding $z\in\mathcal{Z}$. This encoding specifies a current policy context, indicating for example a task or goal. We denote such *universal* $Q$-functions as $Q(s,a,z)$. In the context of this work, we consider $z$ to be a parameterization or indexing of a specific policy $\pi(\cdot|s,z)$, or in other words $Q:\mathcal{S}\times\mathcal{A}\times\mathcal{Z}\xrightarrow{}\mathbb{R},\quad Q
(s,a,z)\equiv Q^{\pi(\cdot|s,z)}(s,a)$.

Both in the single and multi task settings, obtaining effective policies may require efficient exploration and an agent’s ability to reason about *epistemic* uncertainty. This source of uncertainty, in contrast to *aleatoric* uncertainty, stems from a lack of knowledge and may in general be reduced by the acquisition of data. In the context of RL, we make an additional distinction between *myopic uncertainty* and *value uncertainty*.

### 2.1 Myopic Uncertainty and Neural Tangent Kernels

Myopic uncertainty estimation methods, such as RND or ensembles predicting immediate rewards or next states, quantify epistemic uncertainty without explicitly accounting for future uncertainties along trajectories. We first briefly recall the RND algorithm (Burda et al., 2019) , before introducing the neural tangent kernel (NTK) (Jacot et al., 2018) framework.

Random network distillation comprises two neural networks: A fixed, randomly initialized *target network* $g(x;\psi_{0})$, and a *predictor network* $u(x;\vartheta_{t})$. The online predictor $u(x;\vartheta_{t})$ is trained via gradient descent to minimize a square loss between its own predictions and the target network’s output on a set of data points $\mathcal{X}=\{x_{i}\in\mathbb{R}^{d_{\text{in}}}\}_{i=1}^{N_{D}}$. The RND prediction error at a test point $x$ then serves as an uncertainty or novelty signal. The loss and error function of RND are then given as

$$ $\displaystyle\mathcal{L}_{\text{rnd}}(\theta_{t})={\textstyle\frac{1}{2}}(u( \mathcal{X};\theta_{t})-g(\mathcal{X};\psi_{0}))^{2}\,,\quad\text{and}\quad \epsilon_{\text{rnd}}^{2}(x;\vartheta_{t},\psi_{0})={\textstyle\frac{1}{2}}(u( x;\vartheta_{t})-g(x;\psi_{0}))^{2}\,.$ (2) $$

This mechanism relies on the idea that the predictor network recovers the outputs of the target network only for datapoints contained in the dataset $x_{i}\in\mathcal{X}$, while a measurable error $\epsilon_{\text{rnd}}^{2}$ persists for out-of-distribution test samples $x_{T}\notin\mathcal{X}$, yielding a measure of epistemic uncertainty.

Next, we introduce the framework of neural tangent kernels, an analytical framework we intend to employ for the study of neural network and deep ensemble behavior. Consider a neural network $f(x,\theta_{t}):\mathbb{R}^{n_{\text{in}}}\rightarrow\mathbb{R}$ with hidden layer widths $n_{1},\dots,n_{L}=n$ and inputs $x\in\mathbb{R}^{n_{\text{in}}}$, a dataset $\mathcal{X}$, and labels $\mathcal{Y}=\{y_{i}\in\mathbb{R}\}_{i=1}^{N_{D}}$. Inputs $x_{i}$ may, for example, be state-action tuples and labels $y_{i}$ may be rewards. The network parameters $\theta_{0}\in\mathbb{R}^{n_{\text{p}}}$ are initialized randomly $\theta_{0}\sim\mathcal{N}(0,1)$ and updated with gradient descent with infinitesimal step sizes, also called gradient flow. In the limit of infinite width $n$, the function initialization $f(\cdot,\theta_{0})$, as shown by Lee et al. (2018), is equivalent to a Gaussian process prior with a specific kernel $\kappa:\mathbb{R}^{n_{\text{in}}}\times\mathbb{R}^{n_{\text{in}}}\xrightarrow{
}\mathbb{R}$ called the neural network Gaussian process (NNGP). The functional evolution of $f$ through gradient flow is then governed by a *gradient* inner product kernel $\Theta:\mathbb{R}^{n_{\text{in}}}\times\mathbb{R}^{n_{\text{in}}}\xrightarrow{
}\mathbb{R}$ yielding

$$ $\displaystyle\Theta(x,x^{\prime})=\nabla_{\theta}f(x,\theta_{0})^{\top}\nabla_ {\theta}f(x^{\prime},\theta_{0}),$ $\displaystyle\quad\text{and}\quad\kappa(x,x^{\prime})=\mathbb{E}[f(x,\theta_{0 })f(x^{\prime},\theta_{0})]\,.$ (3) $$

Remarkably, seminal work by Jacot et al. (2018) showed that in the limit of infinite width and appropriate parametrization(^1^11so-called NTK parametrization scales forward/backward passes appropriately, see Jacot et al. (2018)), the kernel $\Theta$ becomes deterministic and remains constant throughout training. This limiting kernel, referred to as the neural tangent kernel (NTK), leads to analytically tractable training dynamics for various loss functions, including the squared loss $\mathcal{L}(\theta_{t})={\textstyle\frac{1}{2}}\|f(\mathcal{X};\theta_{t})-
\mathcal{Y}\|^{2}_{2}$. Owing to this, one can show (Jacot et al., 2018; Lee et al., 2020) that for $t\xrightarrow{}\infty$ post convergence function evaluations $f(\mathcal{X}_{T},\theta_{\infty})$ on a set of test points $\mathcal{X}_{T}$, too, are Gaussian with mean $\mathbb{E}[f(\mathcal{X}_{T},\theta_{\infty})]=\Theta_{\mathcal{X}_{T}\mathcal
{X}}\Theta_{\mathcal{X}\mathcal{X}}^{-1}\mathcal{Y}$ and covariance

$$ $\displaystyle\mathrm{Cov}[f(\mathcal{X}_{T},\theta_{\infty})]=\kappa_{\mathcal {X}_{T}\mathcal{X}_{T}}-\bigl{(}\Theta_{\mathcal{X}_{T},\mathcal{X}}\Theta_{ \mathcal{X}\mathcal{X}}^{-1}\kappa_{\mathcal{X}\mathcal{X}_{T}}+h.c.\bigr{)}+ \Theta_{\mathcal{X}_{T}\mathcal{X}}\Theta_{\mathcal{X}\mathcal{X}}^{-1}\kappa_ {\mathcal{X}\mathcal{X}}\Theta_{\mathcal{X}\mathcal{X}}^{-1}\Theta_{\mathcal{X }\mathcal{X}_{T}},$ (4) $$

where $h.c.$ denotes the Hermitian conjugate of the preceding term and we used the shorthands $\Theta_{\mathcal{X}_{1}\mathcal{X}_{2}}=\Theta(\mathcal{X}_{1},\mathcal{X}_{2})$ and $\kappa_{\mathcal{X}_{1}\mathcal{X}_{2}}=\kappa(\mathcal{X}_{1},\mathcal{X}_{2})$. This expression provides a closed-form solution for the epistemic uncertainty captured by an infinite ensemble of NNs in the NTK regime trained with square losses. For example, the predictive variances of such ensembles are easily obtained as the diagonal entries of Eq. [4](https://arxiv.org/html/2505.21119v2#S2.E4). While requiring an idealized setting, NTK theory offers a solid theoretical grounding for quantifying the behavior of deep ensembles and, by extension, myopic uncertainty estimates from related approaches. However, this analysis does not extend to value functions trained with TD losses and bootstrapping as is common in practical reinforcement learning settings.

### 2.2 Value Uncertainty

In contrast to myopic uncertainties, value uncertainty quantifies a model’s lack of knowledge in the value $Q^{\pi}(s,a)$. As such it inherently depends on future trajectories induced by policies $\pi$. Due to this need to account for accumulated uncertainties over potentially long horizons, value uncertainty estimation typically renders more difficult than its myopic counterpart.

A widely used technique(Osband et al., 2016; Chen et al., 2017; An et al., 2021) to this end is the use of deep ensembles of value functions $Q(s,a,\theta_{t}):\mathcal{S}\times\mathcal{A}\xrightarrow{}\mathbb{R}$ from random initializations $\theta_{0}$. $Q$-functions are trained on transitional data $\mathcal{X}_{TD}=\{s_{i},a_{i}\}_{i=1}^{N_{D}}$, $\mathcal{X}^{\prime}_{TD}=\{s^{\prime}_{i},a^{\prime}_{i}\}_{i=1}^{N_{D}}$, and $r=\{r_{i}\}_{i=1}^{N_{D}}$, where $s^{\prime}_{i}$ are samples from the transition kernel $P$ and $a^{\prime}_{i}$ are samples from a policy $\pi$. $Q$-functions are then optimized through gradient descent on a temporal difference (TD) loss given by

$$ $\displaystyle\mathcal{L}(\theta_{t})$ $\displaystyle={\textstyle\frac{1}{2}}\|\,[\gamma Q^{\pi}(\mathcal{X}^{\prime}_ {TD},\theta_{t})]_{\text{sg}}+r-Q^{\pi}(\mathcal{X}_{TD},\theta_{t})\,\|^{2}_{ 2},$ (5) $$

where $[\cdot]_{\text{sg}}$ indicates a stop-gradient operation. Due to the stopping of gradient flow through $Q(\mathcal{X}^{\prime},\theta_{t})$, we refer to this operation as semi-gradient updates. Uncertainty estimates can then be obtained as the variance $\sigma_{q}^{2}(s,a)=\mathbb{V}_{\theta_{0}}[Q(s,a,\theta_{t})]$ between ensembles of $Q$-functions from random initializations. While empirically successful, TD-trained deep ensembles are not as well understood as the supervised learning setting outlined in the previous section  [2.1](https://arxiv.org/html/2505.21119v2#S2.SS1). Due to the use of bootstrapped TD losses, the closed-form NTK regime solutions in Eq. [4](https://arxiv.org/html/2505.21119v2#S2.E4) do not apply to deep value function ensembles.

An alternative to the above approach is the propagation of myopic uncertainty estimates. Several prior methods(O’Donoghue et al., 2018; Zhou et al., 2020; Luis et al., 2023) formalize this setting under a model-based perspective, where transition models $\tilde{P}(\cdot|s,a)$ are sampled from a Bayesian posterior conditioned on transition data up to $t$. For acyclic MDPs, this setting permits a consistency condition similar to the Bellman equation that upper bounds value uncertainties recursively. While this approach devises a method for obtaining value uncertainties from propagated myopic uncertainties, several open problems remain, such as the tightness of model-free bounds of this kind (Janz et al., 2019; Van der Vaart et al., 2025) as well as how to prevent *underestimation* of these upper bounds due to the use of function approximation (Rashid et al., 2020; Zanger et al., 2024).

## 3 Universal Value-Function Uncertainties

Our method, *universal value-function uncertainties* (UVU), measures epistemic value uncertainty as the prediction errors between an online learner and a fixed target network, similar in spirit to random network distillation (Burda et al., 2019). However, while RND quantifies myopic uncertainty through immediate prediction errors, UVU modifies the training process of the online learner such that the resulting prediction errors reflect value-function uncertainties, that is, uncertainty about long-term returns under a given policy.

Our method centers around the interplay of two distinct neural networks: an online learner $u(s,a,z,\vartheta_{t}):\mathcal{S}\times\mathcal{A}\times\mathcal{Z}
\xrightarrow{}\mathbb{R}$, parameterized by weights $\vartheta_{t}$, and a fixed, randomly initialized target network $g(s,a,z,\psi_{0}):\mathcal{S}\times\mathcal{A}\times\mathcal{Z}\xrightarrow{}
\mathbb{R}$, parameterized by weights $\psi_{0}$. Given a transition $(s,a,s^{\prime})$ and policy encoding $z$, we draw subsequent actions $a^{\prime}$ from a policy $\pi(\cdot|s^{\prime},z)$. Then, we use the fixed target network $g$ to generate synthetic rewards as

$$ $r_{g}^{z}(s,a,s^{\prime},a^{\prime})=g(s,a,z,\psi_{0})-\gamma g(s^{\prime},a^{ \prime},z,\psi_{0})\,.$ (6) $$

While the weights $\psi_{0}$ of the target network remain fixed at initialization, the online network $u$ is trained to minimize a TD loss using the synthetic reward $r_{g}^{\pi}$. Given a dataset $\mathcal{X}=\{s_{i},a_{i},z_{i}\}_{i=1}^{N_{D}}$, we have

$$ $\mathcal{L}(\vartheta_{t})={\textstyle\frac{1}{2N_{D}}}{\textstyle\sum\limits_ {\scriptscriptstyle i}^{\scriptscriptstyle N_{D}}}\bigl{(}\gamma\left[u(s_{i}^ {\prime},a_{i}^{\prime},z_{i},\vartheta_{t})\right]_{\text{sg}}+r^{z}_{g}(s_{i },a_{i},s_{i}^{\prime},a_{i}^{\prime})-u(s_{i},a_{i},z_{i},\vartheta_{t})\bigr {)}^{2},$ (7) $$

where $[\cdot]_{\text{sg}}$ indicates a stop-gradient operation. For any tuple $(s,a,z)$ ($\in\mathcal{X}$ or not), we measure predictive uncertainties as squared prediction errors between the learner and the target function

$$ $\displaystyle\epsilon(s,a,z,\vartheta_{t},\psi_{0})^{2}=\bigl{(}u(s,a,z, \vartheta_{t})-g(s,a,z,\psi_{0})\bigr{)}^{2}.$ (8) $$

The intuition behind this design is that, by construction, the value-function associated with policy $\pi(\cdot|s,z)$ and the synthetic rewards $r_{g}^{z}(s,a,s^{\prime},a^{\prime})$ exactly equals the fixed target network $g(s,a,z,\psi_{0})$. As a sanity check, note that the target function $g(s,a,z,\psi_{0})$ itself satisfies the Bellman equation for the policy $\pi(\cdot|s,z)$ and the synthetic reward definition in Eq. ([6](https://arxiv.org/html/2505.21119v2#S3.E6)), constituting a *random value function* to $r_{g}^{z}$ and hence achieves zero-loss according to Eq. ([7](https://arxiv.org/html/2505.21119v2#S3.E7)). Therefore, if the dataset $\mathcal{X}$ sufficiently covers the dynamics induced by $\pi(\cdot|s,z)$, the online network $u(s,a,z,\vartheta_{0})$ is able to recover $g(s,a,z,\psi_{0})$ exactly, nullifying prediction errors. However, when data coverage is incomplete for the evaluated policy, minimization of the TD loss [7](https://arxiv.org/html/2505.21119v2#S3.E7) is not sufficient for the online network $u(s,a,z,\vartheta_{0})$ to recover target network predictions $g(s,a,z,\psi_{0})$. This discrepancy is captured by the prediction errors, which quantify epistemic uncertainty regarding future gaps of the available data.

### 3.1 Building Intuition by an Example

Figure: Figure 1: Chain MDP of length $N$ with unexplored actions $b$.
Refer to caption: x1.png

To build intuition for how UVU operates and captures value uncertainty, we first consider a tabular setting with a simple chain MDP as illustrated in Figure [1](https://arxiv.org/html/2505.21119v2#S3.F1). Suppose we collect data from a deterministic policy $\pi_{d}$ using action $a$ exclusively. Given this dataset, suppose we would like to estimate the uncertainty associated with the value $Q^{\pi(\cdot|s,z)}(s,a)$ of a policy $\pi(\cdot|s,z)$ that differs from the data-collection policy in that it chooses action “*b*” in $s_{3}$. In our tabular setting, we then initialize random tables $u_{sa}$ and $g_{sa}$. For every transition $(s_{t},a_{t},s_{t+1})$ contained in our single-trajectory dataset, we draw $a_{t+1}\sim\pi(\cdot|s,z)$, compute the reward $r_{g,t}$ as $r_{g,t}=g_{s_{t}a_{t}}-\gamma g_{s_{t+1}a_{t+1}}$ and update table entries with the rule $u_{s_{t}a_{t}}\xleftarrow{}r_{g,t}+\gamma u_{s_{t+1}a_{t+1}}$. Fig. [2](https://arxiv.org/html/2505.21119v2#S3.F2) visualizes this process for several independently initialized tables (rows in Fig. [2](https://arxiv.org/html/2505.21119v2#S3.F2)) for the data-collecting policy $\pi_{d}$ (left), and for the altered policy $\pi(\cdot|s,z)$ (right), which chooses action “*b*” in $s_{3}$. We outline how this procedure yields uncertainty estimates: We first note, that one may regard $g$ as a randomly generated value-function, for which we derive the corresponding reward function as $r_{g}$. As $g_{sa}$, by construction, is the value-function corresponding to $r_{g}$, one may expect that the update rule applied to $u_{sa}$ causes $u_{sa}$ to recover $g_{sa}$. Crucially, however, this is only possible if sufficient data is available for the evaluated policy. When a policy diverges from available data, as occurs under $\pi(\cdot|s,z)$ in $s_{3}$, this causes an effective truncation of the collected trajectory. Consequently, $u_{s_{1}a}$ and $u_{s_{2}a}$ receive updates from $u_{s_{3}b}$, which remains at its initialization, rather than inferring the reward-generating function $g_{sa}$. In the absence of long-term data, the empirical Bellman equations reflected in our updates do not uniquely determine the underlying value function $g_{sa}$. Indeed, both $u_{sa}$ and $g_{sa}$ incur zero TD-error in the r.h.s. of Fig. [2](https://arxiv.org/html/2505.21119v2#S3.F2), yet differ significantly from each other. It is this ambiguity that UVU errors $(g_{sa}-u_{sa})^{2}$ quantify. To ensure $u$ recovers $g$, longer rollouts under the policy $\pi(\cdot|s,z)$ are required to sufficiently constrain the solution space dictated by the Bellman equations (as seen in Fig. [2](https://arxiv.org/html/2505.21119v2#S3.F2) left).

Figure: Figure 2: (left:) Illustration of uncertainty estimation in tabular UVU with $4$ independently initialized tables for $u$ and $g$. Access to full trajectory data allows $u$ to recover $g$. (right:) By executing action “b”, trajectories are effectively truncated, preventing $u$ from recovering $g$. All plots use $\gamma=0.7$.
Refer to caption: x2.png

Figure [3](https://arxiv.org/html/2505.21119v2#S4.F3) illustrates uncertainty estimates for the shown chain MDP using neural networks and for a whole family of policies $\pi(\cdot|s,z)$ which select the unexplored action $b$ with probability $1-z$. We analyze the predictive variance of an ensemble of $128$ universal $Q$-functions, each conditioned on the policy $\pi(\cdot|s,z)$. In the bottom row, we plot the squared prediction error of a single UVU model, averaged over $128$ independent heads. Both approaches show peaked uncertainty in early sections, as policies are more likely to choose the unknown action “*b*” eventually, and low uncertainty closer to the terminal state and for $z$ close to $1$. A comparison with RND is provided in the Appendix [B.3](https://arxiv.org/html/2505.21119v2#A2.SS3).

## 4 What Uncertainties Do Universal Value-Function Uncertainties Learn?

While the previous section provided intuition for UVU, we now derive an analytical characterization of the uncertainties captured by the prediction errors $\epsilon$ between a converged online learner $u$ and the fixed target $g$. We turn to NTK theory to characterize the generalization properties of the involved neural networks in the limit of infinite width, allowing us to draw an exact equality between the squared predictions errors of UVU and the variance of universal value function ensembles.

Figure: Figure 3: From left to right, (1. and 2.): Variance of an ensemble of 128 universal Q-functions trained on a chain MDP dataset. (3. and 4.): Value uncertainty as measured by UVU prediction errors with a single 128-headed model. All plots evaluate the “$a$” action of the chain MDP.
Refer to caption: x4.png

In the following analysis, we use the notational shorthand $x=(s,a,z)$ and $x^{\prime}=(s^{\prime},a^{\prime},z)$ and denote a neural network $f(x,\theta_{t})$ with hidden layer widths $n_{1},\dots,n_{L}=n$, transitions from $\mathcal{X}=\{(s_{i},a_{i},z_{i})\}_{i=1}^{N_{D}}$ to $\mathcal{X}^{\prime}=\{(s^{\prime}_{i},a^{\prime}_{i},z_{i})\}_{i=1}^{N_{D}}$, where $a^{\prime}_{i}\sim\pi(\cdot|s_{i}^{\prime},z_{i})$, and rewards $r=\{r_{i}\}_{i=1}^{N_{D}}$. The evolution of the parameters $\theta_{t}$ under gradient descent with infinitesimal step sizes, also called gradient flow, is driven by the minimization of TD losses with

$$ $\displaystyle{\textstyle\frac{\text{d}}{\text{d}t}}\theta_{t}=-\alpha\nabla_{ \theta}\mathcal{L}(\theta_{t})\,,\quad\text{and}\quad\mathcal{L}(\theta_{t})={ \textstyle\frac{1}{2}}\|\,[\gamma f(\mathcal{X}^{\prime},\theta_{t})]_{\text{ sg}}+r-f(\mathcal{X},\theta_{t})\,\|^{2}_{2}\,.$ (9) $$

We study the dynamics induced by this parameter evolution in the infinite-width limit $n\to\infty$. In this regime, the learning dynamics of $f$ become linear as the NTK becomes deterministic and stationary, permitting explicit closed-form expressions for the evolution of the function $f(x,\theta_{t})$. In particular, we show that the post convergence function $\lim_{t\xrightarrow{}\infty}f(x,\theta_{t})$ is given by

$$ $\displaystyle f(x,\theta_{\infty})$ $\displaystyle=f(x,\theta_{0})-\Theta_{x\mathcal{X}}\left(\Theta_{\mathcal{X} \mathcal{X}}-\gamma\Theta_{\mathcal{X}^{\prime}\mathcal{X}}\right)^{-1}\left(f (\mathcal{X},\theta_{0})-(\gamma f(\mathcal{X}^{\prime},\theta_{0})+r)\right),$ (10) $$

where $\Theta_{xx^{\prime}}$ is the NTK of $f$. Proof is given in Appendix [A.1](https://arxiv.org/html/2505.21119v2#A1.SS1). This identity is useful to our analysis as it delineates any converged function $f(x,\theta_{\infty})$ trained with TD losses [9](https://arxiv.org/html/2505.21119v2#S4.E9) through its initialization $f(x,\theta_{0})$. Theorem [1](https://arxiv.org/html/2505.21119v2#Thmthm1) leverages this deterministic dependency to express the distribution of post convergence functions over random initializations $\theta_{0}$.

###### Theorem 1 .

Let $f(x,\theta_{t})$ be a NN with $L$ hidden layers of width $n_{1},\dots,n_{L}=n$ trained with gradient flow to reduce the TD loss $\mathcal{L}(\theta_{t})={\textstyle\frac{1}{2}}\|\,\gamma[f(\mathcal{X}^{
\prime},\theta_{t})]_{\text{sg}}+r-f(\mathcal{X},\theta_{t})\,\|^{2}_{2}$. In the limit of infinite width $n\xrightarrow{}\infty$ and time $t\xrightarrow{}\infty$, the distribution of predictions $f(\mathcal{X}_{T},\theta_{\infty})$ on a set of test points $\mathcal{X}_{T}$ converges to a Gaussian with mean and covariance given by

$$ $\displaystyle\mathbb{E}_{\theta_{0}}\bigl{[}f(\mathcal{X}_{T},\theta_{\infty}) \bigr{]}$ $\displaystyle=\Theta_{\mathcal{X}_{T}\mathcal{X}}\Delta^{-1}_{\mathcal{X}}r,$ $\displaystyle\mathrm{Cov}_{\theta_{0}}\bigl{[}f(\mathcal{X}_{T},\theta_{\infty })\bigr{]}$ $\displaystyle=\kappa_{\mathcal{X}_{T}\mathcal{X}_{T}}\!-\!(\Theta_{\mathcal{X} _{T}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}\Lambda_{\mathcal{X}_{T}}\!+\!h.c.)\! +\!(\Theta_{\mathcal{X}_{T}\mathcal{X}}\Delta^{-1}_{\mathcal{X}}(\Lambda_{ \mathcal{X}}\!-\!\gamma\Lambda_{\mathcal{X}^{\prime}})\Delta_{\mathcal{X}}^{-1 ^{\top}}\Theta_{\mathcal{X}\mathcal{X}_{T}}),$ $$

where $\Theta_{xx^{\prime}}$ is the NTK, $\kappa_{xx^{\prime}}$ is the NNGP kernel, $h.c.$ denotes the Hermitian conjugate, and

$$ $\displaystyle\Delta_{\tilde{\mathcal{X}}}$ $\displaystyle=\Theta_{\mathcal{X}\tilde{\mathcal{X}}}-\gamma\Theta_{\mathcal{X }^{\prime}\tilde{\mathcal{X}}},\quad\text{ and }\quad\Lambda_{\tilde{\mathcal{ X}}}=\kappa_{\mathcal{X}\tilde{\mathcal{X}}}-\gamma\kappa_{\mathcal{X}^{\prime }\tilde{\mathcal{X}}}\,.$ $$

Proof is provided in Appendix [A.1](https://arxiv.org/html/2505.21119v2#A1.SS1). Theorem [1](https://arxiv.org/html/2505.21119v2#Thmthm1) is significant as it allows us to formalize explicitly the expected behavior and uncertainties of neural networks trained with semi-gradient TD losses, including universal value function ensembles and the prediction errors of UVU. In particular, the variance of an ensemble of universal $Q$-functions $Q(\mathcal{X}_{T},\theta_{\infty})$ over random initializations $\theta_{0}$ is readily given by the diagonal entries of the covariance matrix $\mathrm{Cov}[Q(\mathcal{X}_{T},\theta_{\infty})]$. Applied to the UVU setting, Theorem [1](https://arxiv.org/html/2505.21119v2#Thmthm1) gives an expression for the converged online network $u(x,\vartheta_{\infty})=\Theta_{x\mathcal{X}}\Delta_{\mathcal{X}}^{-1}r_{g}^{z}$ trained with the synthetic rewards $r_{g}^{z}=g(\mathcal{X},\psi_{0})-\gamma g(\mathcal{X}^{\prime},\psi_{0})$. From this, It is straightforward to obtain the distribution of post convergence prediction errors $\frac{1}{2}\epsilon(x,\vartheta_{\infty},\psi_{0})^{2}$. In Corollary [1](https://arxiv.org/html/2505.21119v2#Thmcrl1), we use this insight to conclude that the expected squared prediction errors of UVU precisely match the variance of value functions $Q(x,\theta_{\infty})$ from random initializations $\theta_{0}$.

###### Corollary 1 .

Under the conditions of Theorem [1](https://arxiv.org/html/2505.21119v2#Thmthm1), let $u(x,\vartheta_{\infty})$ be a converged online predictor trained with synthetic rewards generated by the fixed target network $g(x,\psi_{0})$ with $r_{g}^{z}=g(\mathcal{X},\psi_{0})-\gamma g(\mathcal{X}^{\prime},\psi_{0})$. Furthermore denote the variance of converged universal $Q$-functions $\mathbb{V}_{\theta_{0}}[Q(x,\theta_{\infty})]$. Assume $u$, $g$, and $Q$ are architecturally equal and parameters are drawn i.i.d. $\theta_{0},\vartheta_{0},\psi_{0}\sim\mathcal{N}(0,1)$. The expected squared prediction error coincides with $Q$-function variance

$$ $\displaystyle\mathbb{E}_{\vartheta_{0},\psi_{0}}\bigl{[}{\textstyle\frac{1}{2} }\epsilon(x,\vartheta_{\infty},\psi_{0})^{2}\bigr{]}$ $\displaystyle=\mathbb{V}_{\theta_{0}}\bigl{[}Q(x,\theta_{\infty})\bigr{]},$ (11) $$

where the l.h.s. expectation and r.h.s. variance are taken over random initializations $\vartheta_{0},\psi_{0},\theta_{0}$.

Proof is given in Appendix [A.1.3](https://arxiv.org/html/2505.21119v2#A1.SS1.SSS3). This result provides the central theoretical justification for UVU: in the limit of infinite width, our measure of uncertainty, the expected squared prediction error between the online and target network, is mathematically equivalent to the variance one would obtain by training an ensemble of universal $Q$-functions.

In practice, we are moreover interested in the behavior of finite estimators, that is, ensemble variances are estimated with a finite number of models. We furthermore implement UVU with a number of multiple independent heads $u_{i}$ and $g_{i}$ with shared hidden layers. Corollary [2](https://arxiv.org/html/2505.21119v2#Thmcrl2) shows that the distribution of the sample mean squared prediction error from $M$ heads is identical to the distribution of the sample variance of an ensemble of $M+1$ independently trained universal $Q$-functions.

###### Corollary 2 .

Under the conditions of Theorem [1](https://arxiv.org/html/2505.21119v2#Thmthm1), consider online and target networks with $M$ independent heads $u_{i},g_{i}$, $i=1,\dots,M$, each trained to convergence with errors $\epsilon_{i}(x,\vartheta_{\infty},\psi_{0})$. Let ${\textstyle\frac{1}{2}}\bar{\epsilon}(x,\vartheta_{\infty},\psi_{0})^{2}={
\textstyle\frac{1}{2M}}\sum_{i=1}^{M}\epsilon_{i}(x,\vartheta_{\infty},\psi_{0
})^{2}$ be the sample mean squared prediction error over $M$ heads. Moreover, consider $M+1$ independent converged Q-functions $Q_{i}(x;\theta_{\infty})$ and denote their sample variance $\bar{\sigma}_{Q}^{2}(x,\theta_{\infty})={\textstyle\frac{1}{M}}\sum_{i=1}^{M+1
}(Q_{i}(x;\theta_{\infty})-\bar{Q}(x;\theta_{\infty}))^{2}$, where $\bar{Q}$ is the sample mean. The two estimators are identically distributed according to a scaled Chi-squared distribution

$$ $\displaystyle{\textstyle\frac{1}{2}}\bar{\epsilon}(x,\vartheta_{\infty},\psi_{ 0})^{2}\overset{D}{=}\bar{\sigma}_{Q}^{2}(x,\theta_{\infty}),\quad\bar{\sigma} _{Q}^{2}(x,\theta_{\infty})\sim{\textstyle\frac{\sigma_{Q}^{2}}{M}}\chi^{2}(M),$ (12) $$

with $M$ degrees of freedom and $\sigma_{Q}^{2}(x,\theta_{\infty})=\mathbb{V}_{\theta_{0}}[Q(x,\theta_{\infty})]$ is the analytical variance of converged Q-functions given by Theorem [1](https://arxiv.org/html/2505.21119v2#Thmthm1).

Proof is provided in Appendix [A.2.3](https://arxiv.org/html/2505.21119v2#A1.SS2.SSS3). The distributional equivalence of these finite sample estimators provides theoretical motivation for using a multi headed architecture with shared hidden layers within a single UVU model and its use as an estimator for ensemble variances of universal $Q$-functions. While the assumptions of infinite width and gradient flow are theoretical idealizations, several empirical results suggest that insights from the NTK regime can translate well to practical finite width deep learning models (Lee et al., 2020; Liu et al., 2020; Tsilivis and Kempe, 2022), motivating further empirical investigation in Section [5](https://arxiv.org/html/2505.21119v2#S5).

###### Theorem 1 .

###### Corollary 1 .

###### Corollary 2 .

## 5 Empirical Analysis

Figure: Figure 4: (a) Ablation on GoToDoor-$10$ with different network widths. Shaded region indicates standard deviations over 5 seeds. (b) Runtime of various ensemble sizes vs. UVU. Ensembles are implemented with vmap in JAX(Bradbury et al., 2018). (c) Illustration of the GoToDoor environment. The agent (red triangle) must navigate to the door indicated by the task specification $z$.
Refer to caption: x5.png

Our empirical analysis is designed to assess whether UVU can effectively quantify value function uncertainty in practical settings, comparing its performance against established baselines, particularly deep ensembles. Specifically, we aim to address the following questions:

- 1.
Does the theoretical motivation for UVU hold in practice and do its uncertainty estimates enable effective decision-making comparable to deep ensembles?
- 2.
How are uncertainty estimates generated by UVU affected by deviations from our theoretical analysis, namely finite network width?

To address these questions, we focus on an offline multitask RL setting with incomplete data where reliable uncertainty estimation is crucial to attain high performance.

### 5.1 Experimental Setup

In our experimental analysis, we use an offline variant of the GoToDoor environment from the Minigrid benchmark suite (Chevalier-Boisvert et al., 2023). An example view is shown in Figure [4](https://arxiv.org/html/2505.21119v2#S5.F4) (c). In this task, the agent navigates a grid world containing four doors of different colors, placed at random locations and receives a task specification $z$ indicating a target door color. Upon opening the correct door, the agent receives a reward and is placed in a random different location. Episodes are of fixed length and feature a randomly generated grid layout and random door positions / colors. In our experiments, we use variations of different difficulties by increasing maximum grid sizes.

##### Dataset Collection.

A dataset $\mathcal{D}=\{(s_{i},a_{i},r_{i},z_{i},s^{\prime}_{i},)\}_{i=1}^{N_{D}}$ is collected using a policy that performs expertly but systematically fails for certain task/grid combinations (e.g., it can not successfully open doors on the “north” wall, irrespective of color or grid layout). Policies seeking to improve upon the behavior policy thus ought to deviate from the dataset, inducing value uncertainty.

##### Task Rejection Protocol.

All baselines implement a DQN-based agent trained in an offline fashion on $\mathcal{D}$. As the agents aim to learn an optimal policy for all grids and tasks contained in $\mathcal{D}$, the resulting greedy policy tends to deviate from the available data when the collecting policy is suboptimal. We employ a task-rejection protocol to quantify an agent’s ability to recognize this divergence and the associated value uncertainty. As most task/grid combinations are contained in $\mathcal{D}$, though with varying levels of policy expertise, myopic uncertainty is not sufficient for fulfilling this task. Specifically, upon encountering the initial state $s_{0}$, the agent is given opportunity to reject a fixed selection of tasks (here door colors). It is subsequently given one of the remaining, non-rejected tasks and performance is measured by the average return achieved on the attempted task. Successful agents must thus either possess uncertainty estimates reliable enough to consistently reject tasks associated with a data/policy mismatch or rely on out-of-distribution generalization. Similar protocols, known as accuracy rejection curves, have been used widely in the supervised learning literature(Nadeem et al., 2009).

### 5.2 Results

We conduct experiments according to the above protocol and perform a quantitative evaluation of UVU and several baseline algorithms. All agents are trained offline and use the basic DQN architecture (Mnih et al., 2015) adapted for universal value functions, taking the task encoding $z$ as an additional input to the state (details are provided in Appendix [B](https://arxiv.org/html/2505.21119v2#A2)). Specifically, we compare UVU against several baselines: A DQN baseline with random task rejection (DQN); Bootstrapped DQN with randomized priors (BDQNP) (Osband et al., 2019);
A DQN adaptation of random network distillation (DQN-RND) (Burda et al., 2019) and a version adapted with the uncertainty prior mechanism proposed by Zanger et al. (2024) (DQN-RND-P). Except for the DQN baseline, all algorithms reject tasks based on the highest uncertainty estimate, given the initial state $s_{0}$ and action $a_{0}$, which is chosen greedily by the agent.

**Table 1: Results of offline multitask RL with task rejection on different variations of the GoToDoor environment. Results are average evaluation returns of the best-performing policy over $10^{5}$ gradient steps and intervals are $90\%$ student’s $t$ confidence intervals.**
| Size | DQN | BDQNP(3) | BDQNP(15) | BDQNP(35) | DQN-RND | DQN-RND-P | UVU (Ours) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $5$ | $5.50\pm.15$ | $8.69\pm.24$ | $\bm{10.50\pm.04}$ | $\bm{10.58\pm.03}$ | $3.94\pm.50$ | $\bm{10.41\pm.12}$ | $\bm{10.54\pm.03}$ |
| $6$ | $4.93\pm.12$ | $7.66\pm.09$ | $9.39\pm.04$ | $\bm{9.57\pm.04}$ | $1.99\pm.40$ | $9.28\pm.12$ | $\bm{9.54\pm.03}$ |
| $7$ | $4.58\pm.09$ | $6.61\pm.16$ | $8.49\pm.05$ | $\bm{8.75\pm.06}$ | $2.66\pm.43$ | $8.12\pm.23$ | $\bm{8.73\pm.04}$ |
| $8$ | $4.06\pm.12$ | $5.91\pm.10$ | $7.68\pm.05$ | $7.92\pm.05$ | $2.53\pm.54$ | $7.40\pm.14$ | $\bm{8.03\pm.04}$ |
| $9$ | $3.66\pm.09$ | $5.04\pm.08$ | $6.69\pm.07$ | $7.03\pm.13$ | $2.39\pm.38$ | $6.39\pm.19$ | $\bm{7.29\pm.10}$ |
| $10$ | $3.39\pm.11$ | $4.64\pm.14$ | $6.09\pm.13$ | $\bm{6.53\pm.16}$ | $2.25\pm.48$ | $5.64\pm.17$ | $\bm{6.72\pm.12}$ |

Table [1](https://arxiv.org/html/2505.21119v2#S5.T1) shows the average return achieved by each method on the GoToDoor experiment across different maximum grid sizes, with average runtimes displayed in Fig. [4](https://arxiv.org/html/2505.21119v2#S5.F4) (b). This result addresses our first research question regarding the practical effectiveness of UVU compared to ensembles and other baseline methods. As shown, the standard DQN baseline performs significantly worse than uncertainty-based algorithms, indicating that learned $Q$-functions do not generalize sufficiently to counterbalance inadequate uncertainty estimation. Both small and large ensembles significantly improve performance by leveraging uncertainty to reject tasks and policies associated with missing data. RND-based agents perform well when intrinsic reward priors are used. Our approach scores highly and outperforms many of the tested baselines with statistical significance, indicating that it is indeed able to effectively quantify value uncertainty using a single-model multi-headed architecture.

We furthermore ablate UVU’s dependency on network width, given that our theoretical analysis is situated in the infinite width limit. Fig. [4](https://arxiv.org/html/2505.21119v2#S5.F4) (a) shows that UVU’s performance scales similarly with network width to DQN and BDQNP baselines, indicating that finite-sized networks, provided appropriate representational capacity, are sufficient for effective uncertainty estimates.

## 6 Related Work

A body of literature considers the quantification of value function uncertainty in the context of exploration.
Early works (Dearden et al., 1998; Engel et al., 2005) consider Bayesian adoptions of model-free RL algorithms with many modern approaches relying on efficient samplers (Russo, 2019; Ishfaq et al., 2021, 2024b). Several recent works moreover provide analyses of the Bayesian model-free setting and correct applications thereof (Fellows et al., 2021; Schmitt et al., 2023; Van der Vaart et al., 2025), which is a subject of debate due to the use TD losses. Several works furthermore derive provably efficient model-free algorithms using frequentist upper bounds on values in tabular (Strehl et al., 2006; Jin et al., 2018) and linear settings (Jin et al., 2020). Similarly, Yang et al. (2020) derive provably optimisic bounds of value functions in the NTK regime, but in contrast to our work uses local bonuses to obtain these. The exact relationship between bounds derived from local bonuses and the functional variance in ensemble or Bayesian settings remains open.

The widespread use and empirical success of ensembles for uncertainty quantification in deep learning (Dietterich, 2000; Lakshminarayanan et al., 2017) has motivated several directions of research towards a better theoretical understanding of their behavior. Following seminal works by Jacot et al. (2018) and Lee et al. (2020) who characterize NN learning dynamics in the NTK regime, a number of works have connected deep ensembles to Bayesian interpretations (He et al., 2020; D’Angelo and Fortuin, 2021). Moreover, a number of papers have studied the learning dynamics of model-free RL: in the overparametrized linear settings (Xiao et al., 2021); in neural settings for single (Cai et al., 2019) and multiple layers (Wai et al., 2020); to analyze generalization behavior (Lyle et al., 2022) with linear and second-order approximations. It should be noted that the aforementioned do not focus on probabilistic descriptions of posterior distributions in the NTK regime. In contrast, our work provides probabilistic closed-form solutions for this setting with semi-gradient TD learning.

In practice, the use of deep ensembles is common in RL, with applications ranging from efficient exploration (Osband et al., 2016; Chen et al., 2017; Osband et al., 2019; Nikolov et al., 2019; Ishfaq et al., 2024a; Zanger et al., 2024) to off-policy or offline RL (An et al., 2021; Chen et al., 2021; Lee et al., 2021) and conservative or safe RL (Lütjens et al., 2019; Lee et al., 2022; Hoel et al., 2023). Single model methods that aim to reduce the computational burden of ensemble methods typically operate as myopic uncertainty estimators (Burda et al., 2019; Pathak et al., 2017; Lahlou et al., 2021; Zanger et al., 2025) and require additional propagation mechanisms (O’Donoghue et al., 2018; Janz et al., 2019; Zhou et al., 2020; Luis et al., 2023).

## 7 Limitations and Discussion

In this work, we introduced universal value-function uncertainties (UVU), an efficient single-model method for uncertainty quantification in value functions. Our method measures uncertainties as prediction error between a fixed, random target network and an online learner trained with a temporal difference (TD) loss. This induces prediction errors that reflect long-term, policy-dependent uncertainty rather than myopic novelty. One of our core contributions is a thorough theoretical analysis of this approach via neural tangent kernel theory, which, in the limit of infinite network width, establishes an equivalence between UVU errors and the variance of ensembles of universal value functions. Empirically, UVU achieves performance comparable and sometimes superior to sizeable deep ensembles and other baselines in challenging offline task-rejection settings, while offering substantial computational savings.

We believe our work opens up several avenues for future research: Although our NTK analysis provides a strong theoretical backing, it relies on idealized assumptions, notably the limit of infinite network width (a thoroughgoing exposition of our approximations is provided in Appendix [A.3](https://arxiv.org/html/2505.21119v2#A1.SS3)). Our experiments suggest UVU’s performance is robust in practical finite-width regimes (Figure [4](https://arxiv.org/html/2505.21119v2#S5.F4)), yet bridging this gap between theory and practice remains an area for future work. On a related note, analysis in the NTK regime typically eludes feature learning. Combinations of UVU with representation learning approaches such as self-predictive auxiliary losses (Schwarzer et al., 2020; Guo et al., 2022; Fujimoto et al., 2023) are, in our view, a very promising avenue for highly challenging exploration problems. Furthermore, while our approach estimates uncertainty for given policies, it does not devise a method for obtaining diverse policies and encodings thereof. We thus believe algorithms from the unsupervised RL literature(Touati and Ollivier, 2021; Zheng et al., 2023) naturally integrate with our approach. In conclusion, we believe UVU provides a strong foundation for future developments in uncertainty-aware agents that are both capable and computationally feasible.

## References

- An et al. [2021]
G. An, S. Moon, J.-H. Kim, and H. O. Song.
Uncertainty-based offline reinforcement learning with diversified q-ensemble.
*Advances in neural information processing systems*, 34:7436–7447, 2021.
- Ba et al. [2016]
J. L. Ba, J. R. Kiros, and G. E. Hinton.
Layer normalization.
*arXiv preprint arXiv:1607.06450*, 2016.
- Bellemare et al. [2016]
M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos.
Unifying count-based exploration and intrinsic motivation.
*Advances in neural information processing systems*, 29, 2016.
- Bellman [1957]
R. Bellman.
A Markovian decision process.
*Journal of mathematics and mechanics*, 6, 1957.
- Bradbury et al. [2018]
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang.
JAX: composable transformations of Python+NumPy programs, 2018.
URL [http://github.com/jax-ml/jax](http://github.com/jax-ml/jax).
- Burda et al. [2019]
Y. Burda, H. Edwards, A. J. Storkey, and O. Klimov.
Exploration by random network distillation.
In *International conference on learning representations, ICLR*, 2019.
- Cai et al. [2019]
Q. Cai, Z. Yang, J. D. Lee, and Z. Wang.
Neural temporal-difference learning converges to global optima.
*Advances in Neural Information Processing Systems*, 32, 2019.
- Chen et al. [2017]
R. Y. Chen, S. Sidor, P. Abbeel, and J. Schulman.
UCB exploration via Q-ensembles.
*arXiv preprint arXiv:1706.01502*, 2017.
- Chen et al. [2021]
X. Chen, C. Wang, Z. Zhou, and K. Ross.
Randomized ensembled double Q-learning: Learning fast without a model.
*arXiv preprint arXiv:2101.05982*, 2021.
- Chevalier-Boisvert et al. [2023]
M. Chevalier-Boisvert, B. Dai, M. Towers, R. Perez-Vicente, L. Willems, S. Lahlou, S. Pal, P. S. Castro, and J. Terry.
Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.
In *Advances in Neural Information Processing Systems 36, New Orleans, LA, USA*, December 2023.
- D’Angelo and Fortuin [2021]
F. D’Angelo and V. Fortuin.
Repulsive deep ensembles are bayesian.
*Advances in Neural Information Processing Systems*, 34:3451–3465, 2021.
- Dearden et al. [1998]
R. Dearden, N. Friedman, S. Russell, et al.
Bayesian Q-learning.
*Aaai/iaai*, 1998:761–768, 1998.
- Dietterich [2000]
T. G. Dietterich.
Ensemble methods in machine learning.
In *Multiple classifier systems: First international workshop, MCS*. Springer, 2000.
- Engel et al. [2005]
Y. Engel, S. Mannor, and R. Meir.
Reinforcement learning with gaussian processes.
In *Proceedings of the 22nd international conference on Machine learning*, pages 201–208, 2005.
- Fellows et al. [2021]
M. Fellows, K. Hartikainen, and S. Whiteson.
Bayesian Bellman operators.
*Advances in neural information processing systems*, 34, 2021.
- Fujimoto et al. [2023]
S. Fujimoto, W.-D. Chang, E. Smith, S. S. Gu, D. Precup, and D. Meger.
For sale: State-action representation learning for deep reinforcement learning.
*Advances in neural information processing systems*, 36:61573–61624, 2023.
- Gallici et al. [2024]
M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, J. N. Foerster, and M. Martin.
Simplifying deep temporal difference learning.
*arXiv preprint arXiv:2407.04811*, 2024.
- Gerschgorin [1931]
S. Gerschgorin.
Uber die abgrenzung der eigenwerte einer matrix.
*Izvestija Akademii Nauk SSSR, Serija Matematika*, 7(3):749–754, 1931.
- Ghavamzadeh et al. [2015]
M. Ghavamzadeh, S. Mannor, J. Pineau, and A. Tamar.
Bayesian reinforcement learning: A survey.
*Foundations and trends in machine learning*, 8, 2015.
- Guo et al. [2022]
Z. Guo, S. Thakoor, M. Pîslar, B. Avila Pires, F. Altché, C. Tallec, A. Saade, D. Calandriello, J.-B. Grill, Y. Tang, et al.
BYOL-Explore: Exploration by bootstrapped prediction.
*Advances in neural information processing systems*, 35:31855–31870, 2022.
- Hasselt [2010]
H. Hasselt.
Double Q-learning.
*Advances in neural information processing systems*, 23, 2010.
- He et al. [2020]
B. He, B. Lakshminarayanan, and Y. W. Teh.
Bayesian deep ensembles via the neural tangent kernel.
*Advances in neural information processing systems*, 33, 2020.
- He et al. [2015]
K. He, X. Zhang, S. Ren, and J. Sun.
Delving deep into rectifiers: Surpassing human-level performance on Imagenet classification.
In *Proceedings of the IEEE international conference on computer vision*, 2015.
- Hoel et al. [2023]
C.-J. Hoel, K. Wolff, and L. Laine.
Ensemble quantile networks: Uncertainty-aware reinforcement learning with applications in autonomous driving.
*IEEE Transactions on intelligent transportation systems*, 2023.
- Ishfaq et al. [2021]
H. Ishfaq, Q. Cui, V. Nguyen, A. Ayoub, Z. Yang, Z. Wang, D. Precup, and L. Yang.
Randomized exploration in reinforcement learning with general value function approximation.
In *International conference on machine learning*, pages 4607–4616. PMLR, 2021.
- Ishfaq et al. [2024a]
H. Ishfaq, Q. Lan, P. Xu, A. R. Mahmood, D. Precup, A. Anandkumar, and K. Azizzadenesheli.
Provable and practical: efficient exploration in reinforcement learning via langevin monte carlo.
In *International conference on learning representations*, 2024a.
- Ishfaq et al. [2024b]
H. Ishfaq, Y. Tan, Y. Yang, Q. Lan, J. Lu, A. R. Mahmood, D. Precup, and P. Xu.
More efficient randomized exploration for reinforcement learning via approximate sampling.
In *Reinforcement learning conference*, 2024b.
- Jacot et al. [2018]
A. Jacot, F. Gabriel, and C. Hongler.
Neural tangent kernel: Convergence and generalization in neural networks.
*Advances in neural information processing systems*, 31, 2018.
- Janz et al. [2019]
D. Janz, J. Hron, P. Mazur, K. Hofmann, J. M. Hernández-Lobato, and S. Tschiatschek.
Successor uncertainties: Exploration and uncertainty in temporal difference learning.
*Advances in neural information processing systems*, 32, 2019.
- Jin et al. [2018]
C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan.
Is q-learning provably efficient?
*Advances in neural information processing systems*, 31, 2018.
- Jin et al. [2020]
C. Jin, Z. Yang, Z. Wang, and M. I. Jordan.
Provably efficient reinforcement learning with linear function approximation.
In *Conference on learning theory*, pages 2137–2143. PMLR, 2020.
- Kumar et al. [2020]
A. Kumar, A. Zhou, G. Tucker, and S. Levine.
Conservative q-learning for offline reinforcement learning.
*Advances in neural information processing systems*, 33:1179–1191, 2020.
- Lahlou et al. [2021]
S. Lahlou, M. Jain, H. Nekoei, V. I. Butoi, P. Bertin, J. Rector-Brooks, M. Korablyov, and Y. Bengio.
Deup: Direct epistemic uncertainty prediction.
*arXiv preprint arXiv:2102.08501*, 2021.
- Lakshminarayanan et al. [2017]
B. Lakshminarayanan, A. Pritzel, and C. Blundell.
Simple and scalable predictive uncertainty estimation using deep ensembles.
*Advances in neural information processing systems*, 30, 2017.
- Lee et al. [2018]
J. Lee, J. Sohl-dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri.
Deep neural networks as gaussian processes.
In *International conference on learning representations*, 2018.
- Lee et al. [2020]
J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington.
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent.
*Journal of Statistical Mechanics: Theory and Experiment*, 2020, Dec. 2020.
- Lee et al. [2021]
K. Lee, M. Laskin, A. Srinivas, and P. Abbeel.
Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning.
In *International Conference on Machine Learning*, pages 6131–6141. PMLR, 2021.
- Lee et al. [2022]
S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin.
Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.
In *Conference on Robot Learning*, pages 1702–1712. PMLR, 2022.
- Liu et al. [2020]
C. Liu, L. Zhu, and M. Belkin.
On the linearity of large non-linear models: when and why the tangent kernel is constant.
*Advances in Neural Information Processing Systems*, 33:15954–15964, 2020.
- Luis et al. [2023]
C. E. Luis, A. G. Bottero, J. Vinogradska, F. Berkenkamp, and J. Peters.
Model-based uncertainty in value functions.
In *International Conference on Artificial Intelligence and Statistics*, pages 8029–8052. PMLR, 2023.
- Lütjens et al. [2019]
B. Lütjens, M. Everett, and J. P. How.
Safe reinforcement learning with model uncertainty estimates.
In *2019 International Conference on Robotics and Automation (ICRA)*, pages 8662–8668. IEEE, 2019.
- Lyle et al. [2022]
C. Lyle, M. Rowland, W. Dabney, M. Kwiatkowska, and Y. Gal.
Learning dynamics and generalization in reinforcement learning.
*arXiv preprint arXiv:2206.02126*, 2022.
- Mnih et al. [2015]
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al.
Human-level control through deep reinforcement learning.
*Nature*, 518, 2015.
- Nadeem et al. [2009]
M. S. A. Nadeem, J.-D. Zucker, and B. Hanczar.
Accuracy-rejection curves (arcs) for comparing classification methods with a reject option.
In *Machine Learning in Systems Biology*, pages 65–81. PMLR, 2009.
- Nikolov et al. [2019]
N. Nikolov, J. Kirschner, F. Berkenkamp, and A. Krause.
Information-directed exploration for deep reinforcement learning.
In *International conference on learning representations, ICLR*, 2019.
- Osband et al. [2016]
I. Osband, C. Blundell, A. Pritzel, and B. Van Roy.
Deep exploration via bootstrapped DQN.
*Advances in neural information processing systems*, 29, 2016.
- Osband et al. [2019]
I. Osband, B. Van Roy, D. J. Russo, Z. Wen, et al.
Deep exploration via randomized value functions.
*Journal of machine learning research*, 20, 2019.
- O’Donoghue et al. [2018]
B. O’Donoghue, I. Osband, R. Munos, and V. Mnih.
The uncertainty Bellman equation and exploration.
In *International conference on machine learning*. PMLR, 2018.
- Pathak et al. [2017]
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.
Curiosity-driven exploration by self-supervised prediction.
In *International conference on machine learning*. PMLR, 2017.
- Rashid et al. [2020]
T. Rashid, B. Peng, W. Böhmer, and S. Whiteson.
Optimistic exploration even with a pessimistic initialisation.
*Proceedings of ICLR 2020*, 2020.
- Russo [2019]
D. Russo.
Worst-case regret bounds for exploration via randomized value functions.
*Advances in Neural Information Processing Systems*, 32, 2019.
- Schaul et al. [2015]
T. Schaul, D. Horgan, K. Gregor, and D. Silver.
Universal value function approximators.
In *International conference on machine learning*, pages 1312–1320. PMLR, 2015.
- Schmitt et al. [2023]
S. Schmitt, J. Shawe-Taylor, and H. van Hasselt.
Exploration via epistemic value estimation.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, 2023.
- Schwarzer et al. [2020]
M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman.
Data-efficient reinforcement learning with self-predictive representations.
*arXiv preprint arXiv:2007.05929*, 2020.
- Silver et al. [2016]
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al.
Mastering the game of go with deep neural networks and tree search.
*nature*, 529(7587):484–489, 2016.
- Strehl et al. [2006]
A. L. Strehl, L. Li, E. Wiewiora, J. Langford, and M. L. Littman.
Pac model-free reinforcement learning.
In *Proceedings of the 23rd international conference on Machine learning*, pages 881–888, 2006.
- Touati and Ollivier [2021]
A. Touati and Y. Ollivier.
Learning one representation to optimize all rewards.
*Advances in Neural Information Processing Systems*, 34:13–23, 2021.
- Tsilivis and Kempe [2022]
N. Tsilivis and J. Kempe.
What can the neural tangent kernel tell us about adversarial robustness?
*Advances in Neural Information Processing Systems*, 35:18116–18130, 2022.
- Van der Vaart et al. [2025]
P. R. Van der Vaart, M. T. Spaan, and N. Yorke-Smith.
Epistemic Bellman operators.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2025.
- Vinyals et al. [2019]
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al.
Grandmaster level in starcraft ii using multi-agent reinforcement learning.
*nature*, 575(7782):350–354, 2019.
- Wai et al. [2020]
H.-T. Wai, Z. Yang, Z. Wang, and M. Hong.
Provably efficient neural gtd for off-policy learning.
*Advances in Neural Information Processing Systems*, 33:10431–10442, 2020.
- Xiao et al. [2021]
C. Xiao, B. Dai, J. Mei, O. A. Ramirez, R. Gummadi, C. Harris, and D. Schuurmans.
Understanding and leveraging overparameterization in recursive value estimation.
In *International Conference on Learning Representations*, 2021.
- Yang [2019]
G. Yang.
Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation.
*arXiv preprint arXiv:1902.04760*, 2019.
- Yang et al. [2020]
Z. Yang, C. Jin, Z. Wang, M. Wang, and M. Jordan.
Provably efficient reinforcement learning with kernel and neural function approximations.
*Advances in Neural Information Processing Systems*, 33:13903–13916, 2020.
- Yue et al. [2023]
Y. Yue, R. Lu, B. Kang, S. Song, and G. Huang.
Understanding, predicting and better resolving q-value divergence in offline-rl.
*Advances in Neural Information Processing Systems*, 36:60247–60277, 2023.
- Zanger et al. [2024]
M. A. Zanger, W. Böhmer, and M. T. Spaan.
Diverse projection ensembles for distributional reinforcement learning.
In *International conference on learning representations*, 2024.
- Zanger et al. [2025]
M. A. Zanger, P. R. Van der Vaart, W. Böhmer, and M. T. Spaan.
Contextual similarity distillation: Ensemble uncertainties with a single model.
*arXiv preprint arXiv:2503.11339*, 2025.
- Zheng et al. [2023]
C. Zheng, R. Salakhutdinov, and B. Eysenbach.
Contrastive difference predictive coding.
*arXiv preprint arXiv:2310.20141*, 2023.
- Zhou et al. [2020]
Q. Zhou, H. Li, and J. Wang.
Deep model-based reinforcement learning via estimated uncertainty and conservative policy optimization.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2020.

## Appendix A Theoretical Results

This section provides proofs and further theoretical results for universal value-function uncertainties (UVU).

### A.1 Learning Dynamics of UVU

We begin by deriving learning dynamics for general functions with temporal difference (TD) losses and gradient descent, before analyzing the post training distribution of deep ensembles and prediction errors of UVU.

#### A.1.1 Linearized Learning Dynamics with Temporal Difference Losses

We analyze the learning dynamics of a function trained using semi-gradient temporal difference (TD) losses on a fixed dataset of transitions $\mathcal{X},\mathcal{X}^{\prime}$. Let $f(x,\theta_{t})$ denote a NN of interest with depth $L$ and widths $n_{1},\dots,n_{L-1}=n$.

###### Proposition 1 .

In the limit of infinite width $n\xrightarrow{}\infty$ and infinite time $t\xrightarrow{}\infty$, the function $f(x,\theta_{t})$ converges to

$$ $\displaystyle f(x,\theta_{\infty})$ $\displaystyle=f(x,\theta_{0})-\Theta_{x\mathcal{X}}\left(\Theta_{\mathcal{X} \mathcal{X}}-\gamma\Theta_{\mathcal{X}^{\prime}\mathcal{X}}\right)^{-1}\left(f (\mathcal{X},\theta_{0})-(\gamma f(\mathcal{X}^{\prime},\theta_{0})+r)\right),$ (13) $$

where $\Theta_{xx^{\prime}}$ is the neural tangent kernel of $f$.

###### Proof.

We begin by linearizing the function $f$ around its initialization parameters $\theta_{0}$:

$$ $f_{\text{lin}}(x,\theta_{t})=f(x,\theta_{0})+\nabla_{\theta}f(x,\theta_{0})^{ \top}(\theta_{t}-\theta_{0}).$ (14) $$

We assume gradient descent updates with infinitesimal step size and a learning rate $\alpha$ on the loss

$$ $\displaystyle\mathcal{L}(\theta_{t})$ $\displaystyle={\textstyle\frac{1}{2}}\|\,\gamma f_{\text{lin}}(\mathcal{X}^{ \prime},\theta_{t})_{\text{sg}}+r-f_{\text{lin}}(\mathcal{X},\theta_{t})\,\|^{ 2}_{2},$ (15) $$

yielding the parameter evolution

$$ $\displaystyle{\textstyle\derivative{t}}\theta_{t}$ $\displaystyle=-\alpha\nabla_{\theta}\mathcal{L}(\theta_{t}).$ (16) $$

Setting $w_{t}=\theta_{t}-\theta_{0}$ and find the learning dynamics:

$$ $\displaystyle{\textstyle\derivative{t}}w_{t}$ $\displaystyle=-\alpha\nabla_{\theta}f(\mathcal{X},\theta_{0})\left(f_{\text{ lin}}(\mathcal{X},\theta_{t})-(\gamma f_{\text{lin}}(\mathcal{X}^{\prime}, \theta_{t})+r)\right).$ (17) $$

Thus, the evolution of the linearized function is given by

$$ $\displaystyle{\textstyle\derivative{t}}f_{\text{lin}}(x,\theta_{t})$ $\displaystyle=-\alpha\nabla_{\theta}f(x,\theta_{0})^{\top}\nabla_{\theta}f( \mathcal{X},\theta_{0})\left(f_{\text{lin}}(\mathcal{X},\theta_{t})-(\gamma f_ {\text{lin}}(\mathcal{X}^{\prime},\theta_{t})+r)\right).$ (18) $$

Letting $\delta_{\text{TD}}(\theta_{t})=f_{\text{lin}}(\mathcal{X},\theta_{t})-(\gamma f
_{\text{lin}}(\mathcal{X}^{\prime},\theta_{t})+r)$, we obtain the differential equation

$$ $\displaystyle{\textstyle\derivative{t}}\delta_{\text{TD}}(\theta_{t})$ $\displaystyle=-\alpha\left(\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma \Theta^{t_{0}}_{\mathcal{X}^{\prime}\mathcal{X}}\right)\delta_{\text{TD}}( \theta_{t}),$ (19) $$

where $\Theta^{t_{0}}_{xx^{\prime}}=\nabla_{\theta}f(x,\theta_{0})^{\top}\nabla_{
\theta}f(x^{\prime},\theta_{0})$ is the (empirical) tangent kernel of $f_{\text{lin}}(x,\theta_{t})$. Since the linearization $f_{\text{lin}}(x,\theta_{t})$ has constant gradients $\nabla_{\theta}f(x,\theta_{0})$, the above differential equation is linear and solvable so long as the matrix $\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{
\prime}\mathcal{X}}$ is positive definite. With an exponential ansatz, we obtain the solution

$$ $\displaystyle\delta_{\text{TD}}(\theta_{t})=e^{-\alpha t\left(\Theta^{t_{0}}_{ \mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{\prime}\mathcal{X}} \right)}\delta_{\text{TD}}(\theta_{0}),$ (20) $$

where $e^{X}$ is a matrix exponential. Reintegrating yields the explicit evolution of predictions

$$ $\displaystyle f_{\text{lin}}(x,\theta_{t})$ $\displaystyle=f(x,\theta_{0})+\int_{0}^{t}\derivative{t^{\prime}}f_{\text{lin} }(x,\theta_{t^{\prime}})\differential t^{\prime}$ (21) $\displaystyle=f(x,\theta_{0})-\Theta^{t_{0}}_{x\mathcal{X}}\left(\Theta^{t_{0} }_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{\prime}\mathcal{ X}}\right)^{-1}\left(e^{-\alpha t(\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}- \gamma\Theta^{t_{0}}_{\mathcal{X}^{\prime}\mathcal{X}})}-I\right)\delta_{\text {TD}}(\theta_{0}).$ (22) $$

Jacot et al. [2018] show that in the limit of infinite layer widths of the neural network, the NTK $\Theta^{t_{0}}_{xx^{\prime}}$ becomes deterministic and constant $\Theta^{t_{0}}_{xx^{\prime}}\xrightarrow{}\Theta_{xx^{\prime}}$. As a consequence, the linear approximation $f_{\text{lin}}(x;\theta_{t})$ becomes exact w.r.t. the original function $\lim_{\text{width}\xrightarrow{}\infty}f_{\text{lin}}(x;\theta_{t})=f(x,\theta
_{t})$ [Lee et al., 2020].

∎

###### Proposition 1 .

###### Proof.

##### Remark on the constancy of the NTK in TD learning.

We note here, that our proof assumed the results by Jacot et al. [2018] to hold for the case of semi-gradient TD updates, namely that the NTK becomes deterministic and constant $\Theta^{t_{0}}_{xx^{\prime}}\xrightarrow{}\Theta_{xx^{\prime}}$ in the limit of infinite width under the here shown dynamics. First, the determinacy of the NTK at initialization follows from the law of large numbers and applies in our case equally as in the least squares case. The constancy of the NTK throughout training is established by Theorem 2 in Jacot et al. [2018], which we restate informally below.

###### Theorem 2 .

[Jacot et al. [2018]]
In the limit of infinite layer widths $n\to\infty$ and $n_{1},\dots,n_{L}=n$, the kernel $\Theta_{xx^{\prime}}^{t_{0}}$ converges uniformly on the interval $t\in[0,T]$ to the constant neural tangent kernel

$$ $\displaystyle\Theta_{xx^{\prime}}^{t_{0}}\to\Theta_{xx^{\prime}}\,,$ $$

provided that the integral $\int_{0}^{T}\|d_{t}\|_{2}\,dt$ stays bounded. Here, $d_{t}\in\mathbb{R}^{N_{D}}$ is the training direction of the parameter evolution such that ${\textstyle\derivative{t}}\theta_{t}=-\alpha\nabla_{\theta}f(\mathcal{X},
\theta)d_{t}$

In the here studied case of semi-gradient TD learning, the parameter evolution (as outlined above in Eq. ([17](https://arxiv.org/html/2505.21119v2#A1.E17))) is described by the gradient $\nabla_{\theta}f(\mathcal{X},\theta_{0})$ and the *training direction* $d_{t}$ according to

$$ $\displaystyle{\textstyle\derivative{t}}\theta_{t}$ $\displaystyle=-\alpha\nabla_{\theta}f(\mathcal{X},\theta_{0})\underbrace{\left (f_{\text{lin}}(\mathcal{X},\theta_{t})-(\gamma f_{\text{lin}}(\mathcal{X}^{ \prime},\theta_{t})+r)\right)}_{d_{t}}\,,$ (23) $$

where the training direction is given by $d_{t}=f_{\text{lin}}(\mathcal{X},\theta_{t})-(\gamma f_{\text{lin}}(\mathcal{X
}^{\prime},\theta_{t})+r)=\delta_{TD}(\theta_{t})$. Provided that the matrix $\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{
\prime}\mathcal{X}}$ is positive definite, the norm of the training direction $\|d_{t}\|_{2}$ decays exponentially by Eq. [20](https://arxiv.org/html/2505.21119v2#A1.E20). This implies

$$ $\displaystyle\|d_{t}\|_{2}<\|d_{0}\|_{2}e^{-t\lambda_{\text{min}}}\,,$ (24) $$

where $\lambda_{\text{min}}$ is the smallest eigenvalue of $\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{
\prime}\mathcal{X}}$. Assuming $\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{
\prime}\mathcal{X}}$ is positive definite, $\lambda_{\text{min}}$ is positive and as a consequence, we have

$$ $\displaystyle\int_{0}^{\infty}\|d_{t}\|_{2}\,dt<\int_{0}^{\infty}\|d_{0}\|_{2} e^{-t\lambda_{\text{min}}}\,dt<\infty\,,$ (25) $$

bounding the required integral of Theorem [2](https://arxiv.org/html/2505.21119v2#Thmthm2) for any $T$ and establishing $\Theta^{t_{0}}_{xx^{\prime}}\xrightarrow{}\Theta_{xx^{\prime}}$ uniformly on the interval $[0,\infty)$ (see Theorem 2 in Jacot et al. [2018] for detailed proof for the last statement).

We note, however, that the condition for $\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{
\prime}\mathcal{X}}$ to be positive definite is, for any $\gamma>0$, stronger than in the classical results for supervised learning with least squares regression. While $\Theta_{\mathcal{X}\mathcal{X}}$ can be guaranteed to be positive definite for example by restricting $\mathcal{X}$ to lie on a unit-sphere, $x_{i}\in\mathcal{X}$ to be unique, and by assuming non-polynomial nonlinearities in the neural network (so as to prevent rank decay in the network expressivity), the condition is harder to satisfy in the TD learning setting. Here, the eigenspectrum of $\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{
\prime}\mathcal{X}}$ tends to depend on the transitions $\mathcal{X}\to\mathcal{X}^{\prime}$ themselves and thus is both dependent on the discount $\gamma$ as well as the interplay between gradient structures of the NTK and the MDP dynamics.

We note here, that this is not primarily a limitation of applying NTK theory to TD learning, but is reflected in practical experience: TD learning can, especially in offline settings, indeed be instable and diverge. Instability of this form is thus inherent to the learning algorithm rather than an artifact of our theoretical treatment. Informally, one approach towards guaranteeing positive definiteness of $\Theta^{t_{0}}_{\mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{
\prime}\mathcal{X}}$ is by enforcing diagonal dominance, appealing to the Gershgorin circle theorem [Gerschgorin, 1931]. For a matrix $A=[a_{ij}]$, every real eigenvalue $\lambda$ must lie in

$$ $\displaystyle a_{ii}-R_{i}\leq\lambda\leq a_{ii}+R_{i}\,,$ (26) $$

where $R_{i}=\sum_{i\neq j}|a_{ij}|$ is the sum of off-diagonal elements of a row $i$. In other words, a lower bound on the smallest real eigenvalue can be increased by increasing diagonal entries $a_{ii}$ while decreasing off-diagonal elements $a_{ij}$. In the TD learning setting, this translates to gradient conditioning, e.g., by ensuring $\|\nabla_{\theta}f(x,\theta)\|_{2}=\|\nabla_{\theta}f(x^{\prime},\theta)\|_{2}=C$ for any pair $x,x^{\prime}$, guaranteeing cross-similarities to be smaller than self-similarities. Indeed several recent works pursue similar strategies to stabilize offline TD learning [Yue et al., 2023, Gallici et al., 2024] and rely on architectural elements like layer normalization [Ba et al., 2016] to shape gradient norms.

###### Theorem 2 .

#### A.1.2 Post Training Function Distribution with Temporal Difference Dynamics

We now aim to establish the distribution of post-training functions $f(x,t_{\infty})$ when initial parameters $\theta_{0}$ are drawn randomly i.i.d. For the remainder of this section, we will assume the infinite width limit, s.t. $f_{\text{lin}}(x,\theta_{\infty})=f(x,\theta_{\infty})$ and $\Theta^{t_{0}}_{xx^{\prime}}=\Theta_{xx^{\prime}}$. The post-training function $f(x,\theta_{\infty})$ is given by

$$ $\displaystyle f(x,\theta_{\infty})$ $\displaystyle=f(x,\theta_{0})-\Theta_{x\mathcal{X}}\left(\Theta^{t_{0}}_{ \mathcal{X}\mathcal{X}}-\gamma\Theta^{t_{0}}_{\mathcal{X}^{\prime}\mathcal{X}} \right)^{-1}\left(f(\mathcal{X},\theta_{0})-(\gamma f(\mathcal{X}^{\prime}, \theta_{0})+r)\right),$ (27) $$

and is thus a deterministic function of the initialization $\theta_{0}$.

See [1](https://arxiv.org/html/2505.21119v2#Thmthm1)

###### Proof.

We begin by introducing a column vector of post-training function evaluations on a set of test points $\mathcal{X}_{T}$, and the training data $\mathcal{X}$ and $\mathcal{X}^{\prime}$. Moreover, we introduce the shorthand

$$ $\displaystyle\Delta_{\mathcal{X}}=\Theta_{\mathcal{X}\mathcal{X}}-\gamma\Theta _{\mathcal{X}^{\prime}\mathcal{X}},$ (28) $$

and similarly $\Delta_{\mathcal{X}^{\prime}}=\Theta_{\mathcal{X}\mathcal{X}^{\prime}}-\gamma
\Theta_{\mathcal{X}^{\prime}\mathcal{X}^{\prime}}$. The vector can then be compactly described in block matrix notation by

$$ $\displaystyle\underbrace{\begin{pmatrix}f(\mathcal{X}_{T},\theta_{\infty})\\ f(\mathcal{X},\theta_{\infty})\\ f(\mathcal{X}^{\prime},\theta_{\infty})\end{pmatrix}}_{f^{\infty}}$ $\displaystyle=\underbrace{\begin{pmatrix}I&-\Theta_{\mathcal{X}_{T}\mathcal{X} }\Delta_{\mathcal{X}}^{-1}&\gamma\Theta_{\mathcal{X}_{T}\mathcal{X}}\Delta_{ \mathcal{X}}^{-1}\\ I&-\Theta_{\mathcal{X}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}&\gamma\Theta_{ \mathcal{X}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}\\ I&-\Theta_{\mathcal{X}^{\prime}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}&\gamma \Theta_{\mathcal{X}^{\prime}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}\end{pmatrix} }_{A}\underbrace{\begin{pmatrix}f(\mathcal{X}_{T},\theta_{0})\\ f(\mathcal{X},\theta_{0})\\ f(\mathcal{X}^{\prime},\theta_{0})\end{pmatrix}}_{f^{0}}+\underbrace{\begin{ pmatrix}\Theta_{\mathcal{X}_{T}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}r\\ \Theta_{\mathcal{X}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}r\\ \Theta_{\mathcal{X}^{\prime}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}r\end{pmatrix }}_{b}\,.$ (29) $$

Lee et al. [2018] show that neural networks with random Gaussian initialization $\theta_{0}$ (including NTK parametrization) are described by the neural network Gaussian process (NNGP) $f(\mathcal{X}_{T},\theta_{0})\sim\mathcal{N}(0,\kappa_{\mathcal{X}_{T}\mathcal
{X}_{T}})$ with $\kappa_{\mathcal{X}_{T}\mathcal{X}_{T}}=\mathbb{E}[f(\mathcal{X}_{T},\theta_{0
})f(\mathcal{X}_{T},\theta_{0})^{\top}]$. By extension, the initializations $f^{0}$ are jointly Gaussian with zero mean and covariance matrix

$$ $\displaystyle\mathrm{Cov}\bigl{[}f^{0}\bigr{]}$ $\displaystyle=\underbrace{\begin{pmatrix}\kappa_{\mathcal{X}_{T}\mathcal{X}_{T }}&\kappa_{\mathcal{X}_{T}\mathcal{X}}&\kappa_{\mathcal{X}_{T}\mathcal{X}^{ \prime}}\\ \kappa_{\mathcal{X}\mathcal{X}_{T}}&\kappa_{\mathcal{X}\mathcal{X}}&\kappa_{ \mathcal{X}\mathcal{X}^{\prime}}\\ \kappa_{\mathcal{X}^{\prime}\mathcal{X}_{T}}&\kappa_{\mathcal{X}^{\prime} \mathcal{X}}&\kappa_{\mathcal{X}^{\prime}\mathcal{X}^{\prime}}\end{pmatrix}}_{ K}.$ (30) $$

As the post-training function evaluations $f^{\infty}$ given in Eq. ([29](https://arxiv.org/html/2505.21119v2#A1.E29)) are affine transformations of the multivariate Gaussian random variables $f^{0}\sim\mathcal{N}(0,K)$, they themselves are multivariate Gaussian with distribution $f^{\infty}\sim\mathcal{N}(b,AKA^{\top})$.

We are content with obtaining an expression for the distribution of $f(\mathcal{X}_{T},\theta_{\infty})$ and thus in the following focus on the top-left entry of the block matrix $(AKA^{\top})_{11}$. For notational brevity, we introduce the following shorthand notations

$$ $\displaystyle\Lambda_{\tilde{\mathcal{X}}}=\kappa_{\mathcal{X}\tilde{\mathcal{ X}}}-\gamma\kappa_{\mathcal{X}^{\prime}\tilde{\mathcal{X}}}\,$ (31) $$

After some rearranging, one obtains the following expression for the covariance $\text{Cov}(f_{\mathcal{X}_{T}}^{\infty})$

$$ $\displaystyle\mathrm{Cov}_{\theta_{0}}\bigl{[}f(\mathcal{X}_{T},\theta_{\infty })\bigr{]}\!$ $\displaystyle=\!\kappa_{\mathcal{X}_{T}\mathcal{X}_{T}}\!-\!(\Theta_{\mathcal{ X}_{T}\mathcal{X}}\Delta_{\mathcal{X}}^{-1}\Lambda_{\mathcal{X}_{T}}\!+\!h.c.) \!+\!(\Theta_{\mathcal{X}_{T}\mathcal{X}}\Delta^{-1}_{\mathcal{X}}(\Lambda_{ \mathcal{X}}\!-\!\gamma\Lambda_{\mathcal{X}^{\prime}})\Delta_{\mathcal{X}}^{-1 ^{\top}}\Theta_{\mathcal{X}\mathcal{X}_{T}})\,.$ $$

∎

###### Proof.

#### A.1.3 Distribution of UVU Predictive Errors

We now aim to find an analytical description of the predictive errors as generated by our approach. For this, let $u(x,\vartheta_{t})$ denote the predictive (online) network and $g(x;\psi_{0})$ the fixed target network. We furthermore denote $\epsilon(x,\vartheta_{t},\psi_{0})=u(x,\vartheta_{t})-g(x,\psi_{0})$ the prediction error between online and target network.

See [1](https://arxiv.org/html/2505.21119v2#Thmcrl1)

###### Proof.

Since our algorithm uses semi-gradient TD losses to train $u(x,\vartheta_{t})$, the linearized dynamics of Theorem ([1](https://arxiv.org/html/2505.21119v2#Thmthm1)) apply. However, we consider a fixed target network $g(x;\psi_{0})$ to produce synthetic rewards according to

$$ $\displaystyle r_{g}=g(x,\psi_{0})-\gamma g(x^{\prime},\psi_{0}).$ (32) $$

With the post training function as described by Eq. [27](https://arxiv.org/html/2505.21119v2#A1.E27), the post-training prediction error in a query point $x$ for this reward is given by

$$ $\displaystyle u(x,\vartheta_{\infty})-g(x,\psi_{0})=$ $\displaystyle\qquad u(x,\vartheta_{0})-g(x,\psi_{0})-\Theta_{x\mathcal{X}} \Delta_{\mathcal{X}}^{-1}(u(\mathcal{X},\vartheta_{0})-(\gamma u(\mathcal{X}^{ \prime},\vartheta_{0})+g(\mathcal{X},\psi_{0})-\gamma g(\mathcal{X}^{\prime}, \psi_{0}))).$ (33) $$

We again use the shorthand $\epsilon^{t}=(\epsilon(\mathcal{X}_{T},\vartheta_{t},\psi_{0}),\epsilon(
\mathcal{X},\vartheta_{t},\psi_{0}),\epsilon(\mathcal{X}^{\prime},\vartheta_{t
},\psi_{0}))^{\top}$ and reusing the block matrix $A$ from Eq. [29](https://arxiv.org/html/2505.21119v2#A1.E29), we can write

$$ $\displaystyle\epsilon^{\infty}=A\epsilon^{0}.$ (34) $$

By assumption, $u(x,\vartheta_{0})$ and $g(x,\psi_{0})$ are architecturally equivalent and initialized i.i.d., and $\epsilon^{0}$ is simply the sum of two independent Gaussian vectors with covariance $\mathrm{Cov}[\epsilon^{0}]=2K$. We conclude that prediction errors $\epsilon^{\infty}$ are Gaussian with distribution $\epsilon^{\infty}\sim\mathcal{N}(0,2AKA^{\top})$. Taking the diagonal of the covariance matrix $AKA^{\top}_{11}$, we obtain

$$ $\displaystyle\mathbb{E}_{\vartheta_{0},\psi_{0}}\bigl{[}{\textstyle\frac{1}{2} }\epsilon(x,\vartheta_{\infty},\psi_{0})^{2}\bigr{]}$ $\displaystyle=\mathbb{V}_{\theta_{0}}\bigl{[}Q(x,\theta_{\infty})\bigr{]},$ (35) $$

where

$$ $\displaystyle\mathbb{V}_{\theta_{0}}\bigl{[}Q(x,\theta_{\infty})\bigr{]}\!$ $\displaystyle=\!\kappa_{xx}\!-\!(\Theta_{x\mathcal{X}}\Delta_{\mathcal{X}}^{-1 }\Lambda_{x}\!+\!h.c.)\!+\!(\Theta_{x\mathcal{X}}\Delta^{-1}_{\mathcal{X}}( \Lambda_{\mathcal{X}}\!-\!\gamma\Lambda_{\mathcal{X}^{\prime}})\Delta_{ \mathcal{X}}^{-1^{\top}}\Theta_{\mathcal{X}x})\,.$ (36) $$

∎

###### Proof.

### A.2 Multiheaded UVU

We now show results concerning the equivalence of multiheaded UVU prediction errors and finite ensembles of Q-functions. We first outline proofs for two results by Lee et al. [2018] and Jacot et al. [2018], which rely on in our analysis.

#### A.2.1 Neural Network Gaussian Process Propagation and Independence

Consider a deep neural network $f$ with $L$ layers. Let $z_{i}^{l}(x)$ denote the $i$-th output of layer $l=1,\dots,L$, defined recursively as:

$$ $\displaystyle z_{i}^{l}(x)$ $\displaystyle=\sigma_{b}b_{i}^{l}+{\textstyle\frac{\sigma_{w}}{\sqrt{n_{l-1}}} }\sum_{j=1}^{n_{l-1}}w_{ij}^{l}x_{j}^{l}(x),\quad x_{j}^{l}(x)=\phi(z_{j}^{l-1 }(x)),$ (37) $$

where $n_{l}$ is the width of layer $l$ with $n_{0}=n_{\text{in}}$ and $x^{0}=x$. Further, $\sigma_{w}$ and $\sigma_{b}$ are constant variance multipliers, weights $w^{l}$ and biases $b^{l}$ are initialized i.i.d. with $\mathcal{N}(0,1)$, and $\phi$ is a Lipschitz-continuous nonlinearity. The $i$-th function output $f_{i}(x)$ of the NN is then given by $f_{i}(x)=z_{i}^{L}(x)$.

###### Proposition 2 ( Lee et al. [ 2018 ] ) .

At initialization and in the limit $n_{1}\dots,n_{L-1}\xrightarrow{}\infty$, the $i$-th output at layer $l$, $z_{i}^{l}(x)$, converges to a Gaussian process with zero mean and covariance function $\kappa_{ii}^{l}$ given by

$$ $\displaystyle\kappa_{ii}^{1}(x,x^{\prime})$ $\displaystyle={\textstyle\frac{\sigma_{w}^{2}}{n_{0}}}x^{\top}x^{\prime}+ \sigma_{b}^{2},\quad\text{and}\quad k_{ij}^{1}=0,\quad i\neq j.$ (38) $\displaystyle\kappa_{ii}^{l}(x,x^{\prime})$ $\displaystyle=\sigma_{b}^{2}+\sigma_{w}^{2}\mathbb{E}_{z_{i}^{l-1}\sim\mathcal {GP}(0,\kappa_{ii}^{l-1})}[\phi(z_{i}^{l-1}(x))\phi(z_{i}^{l-1}(x^{\prime}))].$ (39) $$

and

$$ $\displaystyle\kappa_{ij}^{l}(x,x^{\prime})$ $\displaystyle=\mathbb{E}[z_{i}^{l}(x)z_{j}^{l}(x^{\prime})]=\begin{cases} \kappa^{l}(x,x^{\prime})&\quad\text{if }i=j,\\ 0&\quad\text{if }i\neq j.\end{cases}$ (41) $$

###### Proof.

The proof is done by induction. The induction assumption is that if outputs at layer $l-1$ satisfy a GP structure

$$ $\displaystyle z_{i}^{l-1}\sim\mathcal{GP}(0,\kappa_{ii}^{l-1}),$ (42) $$

with the covariance function defined as

$$ $\displaystyle\kappa_{ii}^{l-1}(x,x^{\prime})$ $\displaystyle=\mathbb{E}[z_{i}^{l-1}(x)z_{i}^{l-1}(x^{\prime})]=k_{jj}^{l-1}(x ,x^{\prime}),\quad\forall i,j,$ (43) $\displaystyle\kappa_{ij}^{l-1}(x,x^{\prime})$ $\displaystyle=\mathbb{E}[z_{i}^{l-1}(x)z_{j}^{l-1}(x^{\prime})]=0,\quad\text{ for }i\neq j,$ (44) $$

then, outputs at layer $l$ follow

$$ $\displaystyle z_{i}^{l}(x)\sim\mathcal{GP}(0,\kappa_{ii}^{l}),$ (45) $$

where the kernel at layer $l$ is given by:

$$ $\displaystyle\kappa_{ii}^{l}(x,x^{\prime})$ $\displaystyle=\mathbb{E}[z_{i}^{l}(x)z_{i}^{l}(x^{\prime})]=\kappa_{jj}^{l}(x, x^{\prime}),\quad\forall i,j,$ (46) $\displaystyle\kappa_{ij}^{l}(x,x^{\prime})$ $\displaystyle=\mathbb{E}[z_{i}^{l}(x)z_{j}^{l}(x^{\prime})]=0,\quad\text{if }i \neq j.$ (47) $$

with the recursive definition

$$ $\displaystyle\kappa_{ii}^{l}(x,x^{\prime})=\sigma_{b}^{2}+\sigma_{w}^{2} \mathbb{E}_{z_{i}^{l-1}\sim\mathcal{GP}(0,k_{ii}^{l-1})}[\phi(z_{i}^{l-1}(x)) \phi(z_{i}^{l-1}(x^{\prime}))].$ (48) $$

*Base case $(l=1)$*. At layer $l=1$ we have:

$$ $\displaystyle z_{i}^{1}(x)={\textstyle\frac{\sigma_{w}}{\sqrt{n_{0}}}}\sum_{j= 1}^{n_{0}}w_{ij}^{1}x_{j}+\sigma_{b}b_{i}^{1}.$ (49) $$

This is an affine transform of Gaussian random variables; thus, $z_{i}^{1}(x)$ is Gaussian distributed with

$$ $\displaystyle z_{i}^{1}(x)\sim\mathcal{GP}(0,\kappa_{ii}^{1}),$ (50) $$

with kernel

$$ $\displaystyle\kappa_{ii}^{1}(x,x^{\prime})={\textstyle\frac{\sigma_{w}^{2}}{n_ {0}}}x^{\top}x^{\prime}+\sigma_{b}^{2},\quad\text{and}\quad\kappa_{ij}^{1}=0, \quad i\neq j.$ (51) $$

*Induction step $l>1$.* For layers $l>1$ we have

$$ $\displaystyle z_{i}^{l}(x)=\sigma_{b}b_{i}^{l}+{\textstyle\frac{\sigma_{w}}{ \sqrt{n_{l-1}}}}\sum_{j=1}^{n_{l-1}}w_{ij}^{l}x_{j}^{l}(x),\quad x_{j}^{l}(x)= \phi(z_{j}^{l-1}(x)).$ (52) $$

By the induction assumption, $z_{j}^{l-1}(x)$ are generated by independent Gaussian processes. Hence, $x_{i}^{l}(x)$ and $x_{j}^{l}(x)$ are independent for $i\neq j$. Consequently, $z_{i}^{l}(x)$ is a sum of independent random variables. By the Central Limit Theorem (as $n_{1},\dots,n_{L-1}\rightarrow\infty$) the tuple $\{z_{i}^{l}(x),z_{i}^{l}(x^{\prime})\}$ tends to be jointly Gaussian, with covariance given by:

$$ $\displaystyle\mathbb{E}[z_{i}^{l}(x)z_{i}^{l}(x^{\prime})]$ $\displaystyle=\sigma_{b}^{2}+\sigma_{w}^{2}\mathbb{E}_{z_{i}^{l-1}\sim\mathcal {GP}(0,\kappa_{ii}^{l-1})}[\phi(z_{i}^{l-1}(x))\phi(z_{i}^{l-1}(x^{\prime}))].$ (53) $$

Moreover, as $z_{i}^{l}$ and $z_{j}^{l}$ for $i\neq j$ are defined through independent rows of the parameters $w^{l},b^{l}$ and independent pre-activations $x^{l}(x)$, we have

$$ $\displaystyle\kappa_{ij}^{l}=\mathbb{E}[z_{i}^{l}(x)z_{j}^{l}(x^{\prime})]=0, \quad i\neq j,$ (54) $$

completing the proof.
∎

###### Proposition 2 ( Lee et al. [ 2018 ] ) .

###### Proof.

#### A.2.2 Neural Tangent Kernel Propagation and Independence

We change notation slightly from the previous section to make the parametrization of $f_{i}(x,\theta^{L})$ and $z_{i}^{l}(x;\theta^{l})$ explicit with

$$ $\displaystyle z_{i}^{l}(x,\theta^{l})$ $\displaystyle=\sigma_{b}b_{i}^{l}+{\textstyle\frac{\sigma_{w}}{\sqrt{n_{l-1}}} }\sum_{j=1}^{n_{l-1}}w_{ij}^{l}x_{j}^{l}(x),\quad x_{j}^{l}(x)=\phi(z_{j}^{l-1 }(x;\theta^{l-1})),$ (55) $$

where $\theta^{l}$ denotes the parameters $\{w^{1},b^{1},\dots,w^{l},b^{l}\}$ up to layer $l$ and $f_{i}(x,\theta^{L})=z_{i}^{L}(x;\theta^{L})$. Let furthermore $\phi$ be a Lipschitz-continuous nonlinearity with derivative $\dot{\phi}(x)=\derivative{x}\phi(x)$.

###### Proposition 3 ( Jacot et al. [ 2018 ] ) .

In the limit $n_{1}\dots,n_{L-1}\xrightarrow{}\infty$, the neural tangent kernel $\Theta^{l}_{ii}(x,x^{\prime})$ of the $i$-th output $z_{i}^{l}(x,\theta^{l})$ at layer $l$, defined as the gradient inner product

$$ $\displaystyle\Theta^{l}_{ii}(x,x^{\prime})=\nabla_{\theta^{l}}z_{i}^{l}(x, \theta^{l})^{\top}\nabla_{\theta^{l}}z_{i}^{l}(x^{\prime},\theta^{l}),$ (56) $$

is given recursively by

$$ $\displaystyle\Theta_{ii}^{1}(x,x^{\prime})$ $\displaystyle=\kappa_{ii}^{1}(x,x^{\prime})={\textstyle\frac{\sigma_{w}^{2}}{n _{0}}}x^{\top}x^{\prime}+\sigma_{b}^{2},\quad\text{and}\quad\Theta_{ij}^{1}(x, x^{\prime})=0,\quad i\neq j.$ (57) $\displaystyle\Theta_{ii}^{l}(x,x^{\prime})$ $\displaystyle=\Theta_{ii}^{l-1}(x,x^{\prime})\dot{\kappa}_{ii}^{l-1}(x,x^{ \prime})+\kappa_{ii}^{l}(x,x^{\prime}),$ (58) $$

where

$$ $\displaystyle\dot{\kappa}_{ii}^{l}(x,x^{\prime})$ $\displaystyle=\sigma_{w}^{2}\mathbb{E}_{z_{i}^{l-1}\sim\mathcal{GP}(0,\kappa_{ ii}^{l-1})}[\dot{\phi}(z_{i}^{l-1}(x))\dot{\phi}(z_{i}^{l-1}(x^{\prime}))]$ (60) $$

and

$$ $\displaystyle\Theta_{ij}^{l}(x,x^{\prime})$ $\displaystyle=\nabla_{\theta^{l}}z_{i}^{l}(x,\theta^{l})^{\top}\nabla_{\theta^ {l}}z_{j}^{l}(x^{\prime},\theta^{l})=\begin{cases}\Theta^{l}(x,x^{\prime})& \quad\text{if }i=j,\\ 0&\quad\text{if }i\neq j.\end{cases}$ (61) $$

###### Proof.

We again proceed by induction. The induction assumption is that if gradients satisfy at layer $l-1$

$$ $\displaystyle\Theta_{ij}^{l-1}(x,x^{\prime})=\nabla_{\theta^{l-1}}z_{i}^{l-1}( x,\theta^{l-1})^{\top}\nabla_{\theta^{l-1}}z_{j}^{l-1}(x^{\prime},\theta^{l-1} )=\begin{cases}\Theta^{l-1}(x,x^{\prime})&\quad\text{if }i=j,\\ 0&\quad\text{if }i\neq j,\end{cases}$ (62) $$

then at layer $l$ we have

$$ $\displaystyle\Theta_{ii}^{l}(x,x^{\prime})$ $\displaystyle=\Theta_{ii}^{l-1}(x,x^{\prime})\dot{\kappa}_{ii}^{l}(x,x^{\prime })+\kappa_{ii}^{l}(x,x^{\prime})$ (63) $$

and

$$ $\displaystyle\Theta_{ij}^{l}(x,x^{\prime})$ $\displaystyle=\nabla_{\theta^{l}}z_{i}^{l}(x,\theta^{l})^{\top}\nabla_{\theta^ {l}}z_{j}^{l}(x^{\prime},\theta^{l})=0\quad\text{if }i\neq j.$ (64) $$

*Base Case ($l=1$).* At layer $l=1$, we have

$$ $\displaystyle z_{i}^{1}(x)$ $\displaystyle=\sigma_{b}b_{i}^{1}+{\textstyle\frac{\sigma_{w}}{\sqrt{n_{0}}}} \sum_{j}^{n_{0}}w_{ij}^{1}x_{j},$ (65) $$

and the gradient inner product is given by:

$$ $\displaystyle\nabla_{\theta^{1}}z_{i}^{1}(x,\theta^{1})^{\top}\nabla_{\theta^{ 1}}z_{i}^{1}(x^{\prime},\theta^{1})={\textstyle\frac{\sigma_{w}^{2}}{n_{0}}}x^ {\top}x^{\prime}+\sigma_{b}^{2}=\kappa_{ii}^{1}(x,x^{\prime}).$ (66) $$

*Inductive Step ($l>1$).* For layers $l>1$, we split parameters $\theta^{l}=\theta^{l-1}\cup\{w^{l},b^{l}\}$ and split the inner product by

$$ $\displaystyle\Theta_{ii}^{l}(x,x^{\prime})$ $\displaystyle=\underbrace{\nabla_{\theta^{l-1}}z_{i}^{l}(x,\theta^{l})^{\top} \nabla_{\theta^{l-1}}z_{i}^{l}(x^{\prime},\theta^{l})}_{l.h.s}+\underbrace{ \nabla_{\{w^{l},b^{l}\}}z_{i}^{l}(x,\theta^{l})^{\top}\nabla_{\{w^{l},b^{l}\}} z_{i}^{l}(x^{\prime},\theta^{l})}_{r.h.s}.$ (67) $$

Note that the $r.h.s$ involves gradients w.r.t. last-layer parameters, i.e. the post-activation outputs of the previous layer, and by the same arguments as in the NNGP derivation of Proposition [2](https://arxiv.org/html/2505.21119v2#Thmprp2), this is a sum of independent post activations s.t. in the limit $n_{l-1}\xrightarrow{}\infty$

$$ $\displaystyle\nabla_{\{w^{l},b^{l}\}}z_{i}^{l}(x,\theta^{l})^{\top}\nabla_{\{w ^{l},b^{l}\}}z_{j}^{l}(x^{\prime},\theta^{l})$ $\displaystyle=\begin{cases}k_{ii}^{l}(x,x^{\prime}),&\quad i=j,\\ 0,&\quad i\neq j.\end{cases}$ (68) $$

For the $l.h.s.$, we first apply chain rule to obtain

$$ $\displaystyle\nabla_{\theta^{l-1}}z_{i}^{l}(x,\theta^{l})={\textstyle\frac{ \sigma_{w}}{\sqrt{n_{l-1}}}}\sum_{j}^{n_{l-1}}w_{ij}^{l}\dot{\phi}(z_{j}^{l-1} (x,\theta^{l-1}))\nabla_{\theta^{l-1}}z_{j}^{l-1}(x,\theta^{l-1}).$ (69) $$

The gradient inner product of outputs $i$ and $j$ thus reduces to

$$ $\displaystyle\nabla_{\theta^{l-1}}z_{i}^{l}(x,\theta^{l})^{\top}\nabla_{\theta ^{l-1}}z_{j}^{l}(x^{\prime},\theta^{l})=$ $\displaystyle\qquad{\textstyle\frac{\sigma_{w}^{2}}{n_{l-1}}}\sum_{k}^{n_{l-1} }w_{ik}^{l}w_{jk}^{l}\dot{\phi}(z_{k}^{l-1}(x,\theta^{l-1}))\dot{\phi}(z_{k}^{ l-1}(x^{\prime},\theta^{l-1}))\Theta_{kk}^{l-1}(x,x^{\prime}).$ (70) $$

By the induction assumption $\Theta_{kk}^{l-1}(x,x^{\prime})=\Theta^{l-1}(x,x^{\prime})$ and again by the independence of the rows $w^{l}_{i}$ and $w_{j}^{l}$ for $i\neq j$, the above expression converges in the limit $n_{l-1}\xrightarrow{}\infty$ to an expectation with

$$ $\displaystyle\Theta_{ij}^{l}(x,x^{\prime})=\begin{cases}\Theta^{l-1}(x,x^{ \prime})\dot{\kappa}_{ii}^{l}(x,x^{\prime})+\kappa_{ii}^{l}(x,x^{\prime})& \quad i=j,\\ 0&\quad i\neq j.\end{cases}$ (71) $$

This completes the induction.
∎

###### Proposition 3 ( Jacot et al. [ 2018 ] ) .

###### Proof.

#### A.2.3 Multiheaded UVU: Finite Sample Analysis

We now define multiheaded predictor with $M$ output heads $u_{i}(x,\vartheta_{t})$ for $i=1,\dots,M$ and a fixed multiheaded target network $g_{i}(x_{t};\psi_{0})$ of equivalent architecture as $u$ with the corresponding prediction error $\epsilon_{i}(x,\vartheta_{t},\psi_{0})$ accordingly. Let $u_{i}(x,\vartheta_{t})$ be trained such that each head runs the same algorithm as outlined in Section [3](https://arxiv.org/html/2505.21119v2#S3) independently.

See [2](https://arxiv.org/html/2505.21119v2#Thmcrl2)

###### Proof.

By Collorary. [1](https://arxiv.org/html/2505.21119v2#Thmcrl1), the prediction error of a single headed online and target network $\epsilon(x,\vartheta_{t},\psi_{0})=u(x,\vartheta_{t})-g(x,\psi_{0})$ converges in the limit $n_{1}\dots,n_{L-1}\xrightarrow{}\infty$ and $t\xrightarrow{}\infty$ to a Gaussian with zero mean and variance $\epsilon(x,\vartheta_{\infty},\psi_{0})\sim\mathcal{N}(0,2\sigma_{Q}^{2})$ where

$$ $\displaystyle\sigma_{Q}^{2}=\mathbb{V}_{\theta_{0}}\bigl{[}Q(x,\theta_{\infty} )\bigr{]}\!$ $\displaystyle=\!\kappa_{xx}\!-\!(\Theta_{x\mathcal{X}}\Delta_{\mathcal{X}}^{-1 }\Lambda_{x}\!+\!h.c.)\!+\!(\Theta_{x\mathcal{X}}\Delta^{-1}_{\mathcal{X}}( \Lambda_{\mathcal{X}}\!-\!\gamma\Lambda_{\mathcal{X}^{\prime}})\Delta_{ \mathcal{X}}^{-1^{\top}}\Theta_{\mathcal{X}x})\,.$ (72) $$

By Propositions [2](https://arxiv.org/html/2505.21119v2#Thmprp2) and  [3](https://arxiv.org/html/2505.21119v2#Thmprp3), the NNGP and NTK associated with each online head $u_{i}(x,\vartheta_{\infty})$ in the infinite width and time limit are given by

$$ $\displaystyle\kappa_{ij}(x,x^{\prime})$ $\displaystyle=\mathbb{E}[u_{i}(x,\vartheta_{\infty})u_{j}(x^{\prime},\vartheta _{\infty})]=\begin{cases}\kappa(x,x^{\prime})&\quad\text{if }i=j,\\ 0&\quad\text{if }i\neq j,\end{cases}$ (73) $\displaystyle\Theta_{ij}(x,x^{\prime})$ $\displaystyle=\nabla_{\vartheta}u_{i}^{l}(x,\vartheta_{\infty})^{\top}\nabla_{ \vartheta}u_{j}^{l}(x^{\prime},\vartheta_{\infty})=\begin{cases}\Theta(x,x^{ \prime})&\quad\text{if }i=j,\\ 0&\quad\text{if }i\neq j.\end{cases}$ (74) $$

Due to the independence of the NNGP and NTK for different heads $u_{i}$, prediction errors $\epsilon_{i}(x_{t};\vartheta_{\infty},\psi_{0})$ are i.i.d. draws from a zero mean Gaussian with variance equal as given in Eq. [72](https://arxiv.org/html/2505.21119v2#A1.E72). Note that this is despite the final feature layer being shared between the output functions. The empirical mean squared prediction errors are thus Chi-squared distributed with $M$ degrees of freedom

$$ $\displaystyle{\textstyle\frac{1}{M}}\sum_{i=1}^{M}{\textstyle\frac{1}{2}} \epsilon_{i}(x_{t};\vartheta_{\infty},\psi_{0})^{2}\sim{\textstyle\frac{\sigma _{Q}^{2}}{M}}\chi^{2}(M)$ (75) $$

Now, let $\{Q_{i}(x;\theta_{t})\}_{i=1}^{M+1}$ be a deep ensemble of $M+1$ Q-functions from independent initializations. By Corollary [1](https://arxiv.org/html/2505.21119v2#Thmcrl1), these Q-functions, too, are i.i.d. draws from a Gaussian, now with mean $\Theta_{x\mathcal{X}}\Delta_{\mathcal{X}}^{-1}r$ and variance as given in Eq. [72](https://arxiv.org/html/2505.21119v2#A1.E72). The sample variance of this ensemble thus also follows a Chi-squared distribution with $M$ degrees of freedom

$$ $\displaystyle{\textstyle\frac{1}{M}}\sum_{i=1}^{M+1}{\textstyle\frac{1}{2}} \left(Q_{i}(x;\theta_{\infty})-\bar{Q}(x;\theta_{\infty})\right)^{2}\sim{ \textstyle\frac{\sigma_{Q}^{2}}{M}}\chi^{2}(M),$ (76) $$

where $\bar{Q}(x;\theta_{\infty})={\textstyle\frac{1}{M+1}}\sum_{i}^{M+1}Q_{i}(x;
\theta_{\infty})$ is the sample mean of $M+1$ universal Q-functions, completing the proof.
∎

###### Proof.

### A.3 Limitations and Assumptions

In this section, we detail central theoretical underpinnings and idealizations upon which our theoretical analysis is built.

A central element of our theoretical analysis is the representation of neural network learning dynamics via the Neural Tangent Kernel (NTK), an object in the theoretical limit of infinite network width. The established NTK framework, where the kernel is deterministic despite random initialziation and and constant throughout training, typically applies to fully connected networks with NTK parameterization, optimized using a squared error loss [Jacot et al., 2018]. Our framework instead accommodates a semi-gradient TD loss, and thereby introduces an additional prerequisite for ensuring the convergence of these dynamics: the positive definiteness of the matrix expression $\Theta_{\mathcal{X}\mathcal{X}}-\gamma\Theta_{\mathcal{X}^{\prime}\mathcal{X}}$. This particular constraint is more a characteristic inherent to the TD learning paradigm itself than a direct consequence of the infinite-width abstraction. Indeed, the design of neural network architectures that inherently satisfy such stability conditions for TD learning continues to be an active area of contemporary research [Yue et al., 2023, Gallici et al., 2024]. The modeling choice of semi-gradient TD losses moreover does not incorporate the use of target networks, where bootstrapped values do not only stop gradients but are generated by a separate network altogether that slowly moves towards the online learner. Our analysis moreover considers the setting of *offline policy evaluation*, that is, we do not assume that additional data is acquired during learning and that policies evaluated for value learning remain constant. The assumption of a fixed, static dataset diverges from the conditions of online reinforcement learning with control, where the distribution of training data $(\mathcal{X},\mathcal{X}^{\prime})$ typically evolves as the agent interacts with its environment, both due to its collection of novel transitions and due to adjustments to the policy, for example by use of a Bellman optimality operator. Lastly, our theoretical model assumes, primarily for simplicity, that learning occurs under gradient flow with infinitesimally small step sizes and with updates derived from full-batch gradients. Both finite-sized gradient step sizes and stochastic minibatching has been treated in the literature, albeit not in the TD learning setting [Jacot et al., 2018, Lee et al., 2020, Liu et al., 2020, Yang, 2019]. We believe our analysis could be extended to these settings without major modifications.

## Appendix B Experimental Details

We provide details on our experimental setup, implementations and additional results. This includes architectural design choices, algorithmic design choices, hyperparameter settings, hyperparameter search procedures, and environment details.

### B.1 Implementation Details

All algorithms are self-implemented and tuned in JAX [Bradbury et al., 2018]. A detailed exposition of our design choices and parameters follows below.

##### Environment Setup.

We use a variation of the GoToDoor environment of the minigrid suite [Chevalier-Boisvert et al., 2023]. As our focus is not on partially observable settings, we use fully observable $35$-dimensional state descriptions with $\mathcal{S}=\mathbb{R}^{35}$. Observation vectors comprise the factors:

$$ $\displaystyle o=\left(o_{\text{agent-pos}}^{\top},o_{\text{agent-dir}}^{\top}, o_{\text{door-config}}^{\top},o_{\text{door-pos}}^{\top}\right)^{\top},$ (77) $$

where $o_{\text{agent-pos}}\in\mathbb{R}^{2}$ is the agent position in $x,y$-coordinates, $o_{\text{agent-dir}}\in\mathbb{R}$ is a scalar integer indicating the agent direction (takes on values between 1 and 4), $o_{\text{door-config}}\in\mathbb{R}^{24}$ is the door configuration, comprising $4$ one-hot encoded vectors indicating each door’s color, and $o_{\text{door-pos}}\in\mathbb{R}^{8}$ is a vector containing the $x,y$-positions of the four doors. The action space is discrete and four-dimensional with the following effects

$$ $\displaystyle a_{\text{effect}}=\begin{cases}\text{turn left}&\text{ if }a=0, \\ \text{turn right}&\text{ if }a=1,\\ \text{go forward}&\text{ if }a=2,\\ \text{open door}&\text{ if }a=3.\end{cases}$ (78) $$

Tasks are one-hot encodings of the target door color, that is $z\in\mathbb{R}^{6}$ and in the online setting are generated such that they are achievable. The reward function is an indicator function of the correct door being opened, in which case a reward of $1$ is given to the agent and the agent position is reset to a random location in the grid. Episodes terminate only upon reaching the maximum number of timesteps ($50$ in our experiments).

In the task rejection setting described in our evaluation protocol, an agent in a start state $s_{0}$ is presented a list of tasks, which may or may not be attainable, and is allowed to reject a fixed number of tasks from this list. In our experiments, the agent is allowed to reject $4$ out of $6$ total tasks at the beginning of each episode.

##### Data Collection.

Figure: Figure 5: Illustration of the used architecture. $\odot$ indicates elementwise multiplication.
Refer to caption: x7.png

Our offline datasets are recorded replay buffers from a DQN-agent deployed to the GoToDoor environment with an $\epsilon$-greedy exploration strategy and a particular policy: When the door indicated by the task encoding $z$ provided by the environment lies at the south or west wall, the regular policy by the online DQN agent is executed. If the target door lies at the north or east wall, however, actions are generated by a fixed random $Q$-network. This mixture policy emulates a policy that exhibits expert performance on certain combinations of tasks and states, but suboptimal behavior for other combinations. The replay buffer does, however, contain most combinations of states and tasks, albeit some with trajectories from suboptimal policies. Hyperparameter details of the online agent are provided in section [B.2](https://arxiv.org/html/2505.21119v2#A2.SS2).

##### Algorithmic Details.

All tested algorithms and experiments are based on DQN agents[Mnih et al., 2015] which we adapted for the task-conditioned universal value function [Schaul et al., 2015] setting. While our theoretical analysis considers full-batch gradient descent, in practice we sample minibatches from offline datasets with $\mathcal{X}_{mb}=\{(s_{i},a_{i},z_{i})\}_{i=1}^{N_{mb}}$, $\mathcal{X}_{b}^{\prime}=\{(s^{\prime}_{i},a^{\prime}_{i},z_{i})\}_{i=1}^{N_{b}}$, where next-state actions are generated $a_{i}^{\prime}=\text{argmax}_{a\in\mathcal{A}}Q(s_{i}^{\prime},a,z_{i},\theta_
{t})$ and rewards are $r=\{r_{i}\}_{i=1}^{N_{mb}}$. Moreover, we deviate from our theoretical analysis and use target networks in place of the stop-gradient operation. Here, a separate set of parameters $\tilde{\theta_{t}}$ is used to generate bootstrap targets in the TD loss which is in practice given by

$$ $\displaystyle\mathcal{L}(\theta_{t})$ $\displaystyle={\textstyle\frac{1}{2}}\|\,\gamma Q(\mathcal{X}_{mb}^{\prime}, \tilde{\theta_{t}})+r-Q(\mathcal{X}_{mb},\theta_{t})\,\|^{2}_{2}.$ (79) $$

The parameters $\tilde{\theta_{t}}$ are updated towards the online parameters $\theta_{t}$ at fixed intervals through polyak updating, as is common. We use this basic algorithmic pipeline for all tested algorithms, including the online agent used for data collection.

##### Architectural Details.

We use a hypernetwork MLP architecture adapted to the DQN setting, as depicted in Fig. [5](https://arxiv.org/html/2505.21119v2#A2.F5). Specifically, this means we pass states $s$ and task encodings $z$ through single-layer encoders, which are then joint by elementwise multiplication. The resulting vector is normalized by its $l^{2}$ norm, $x^{\prime}={\textstyle\frac{x}{\|x\|_{2}}}$. This joint vector is passed thorugh a $3$-layer MLP with network width $512$, again normalized by its $l^{2}$ norm and finally passed through a fully-connected layer to obtain a vector of dimension $\mathbb{R}^{|\mathcal{A}|}$. Although our experiments are conducted in the offline RL setting, preliminary experiments showed no benefits of using ensemble-based pessimism [An et al., 2021] or conservative $Q$-updates [Kumar et al., 2020]. Instead, our normalization pipeline appears to sufficiently address overestimation issues as is suggested by several recent works [Yue et al., 2023, Gallici et al., 2024].

##### Independent Bootstrapping.

For the ensemble-based BDQNP baseline and our UVU model, we perform independent bootstrapping in the TD loss computation. By this, we mean that both the bootstrapped value and actions are generated by individual $Q$-functions. In the case of BDQNP, this means we compute Loss [79](https://arxiv.org/html/2505.21119v2#A2.E79) for each model $Q_{k}$, indexed by $k\in[1,\dots,K]$ with $\mathcal{X}_{mb,k}=\mathcal{X}_{mb}$ and bootstraps are generated as

$$ $\displaystyle\mathcal{X}_{mb,k}^{\prime}=\{(s^{\prime}_{i},a^{\prime}_{ik},z_{ i})\}_{i=1}^{N_{mb}}\,,\quad\text{and }\quad a_{ik}^{\prime}=\text{argmax}_{a \in\mathcal{A}}Q_{k}(s_{i}^{\prime},a,z_{i},\theta_{t})\,.$ (80) $$

Note, that this procedure is established [Osband et al., 2016] and serves the purpose of maintaining independence between the models in the ensemble. In order to conduct the same procedure in our UVU method, where we have access to only one $Q$-function, we generate $K$ distinct $Q$-estimates by computing

$$ $\displaystyle Q^{UVU}_{k}(s,a,z,\theta_{t}):=Q(s,a,z,\theta_{t})+\epsilon_{k}( s,a,z,\vartheta_{t},\psi_{0})\,,$ (81) $$

that is, by adding the UVU error of the $k$-th output head. Bootstraps are then generated according to Eq. [80](https://arxiv.org/html/2505.21119v2#A2.E80).

##### Intrinsic reward priors.

Intrinsic reward priors are a trick suggested by Zanger et al. [2024] to address a shortcoming of propagation methods used for intrinsic reward methods like RND[Burda et al., 2019, O’Donoghue et al., 2018]. The issue is that while learning a $Q$-function with intrinsic rewards can, with the right choice of intrinsic reward, provide optimistic estimates of the value function, but only for state-action regions covered in the data. A potential underestiation of the optimistic bound, however, counteracts its intention, a phenomenon also described by Rashid et al. [2020]. Intrinsic reward priors are a heuristic method to address this issue by adding local, myopic uncertainty estimates automatically to the forward pass of the intrinsic $Q$-function, leading to a “prior” mechanism that ensures a

$$ $\displaystyle\hat{Q}_{intr}(s,a,z,\theta_{t})={Q}_{intr}(s,a,z,\theta_{t})+{ \textstyle\frac{1}{2}}\epsilon_{rnd}(s,a,z,\theta_{rnd})^{2}\,$ $$

where $\epsilon_{rnd}(s,a,z,\theta_{rnd})$ denotes a local RND error as an example. The altered function $\hat{Q}_{intr}(s,a,z,\theta_{t})$ is trained as usual with Loss [79](https://arxiv.org/html/2505.21119v2#A2.E79) and intrinsic rewards ${\textstyle\frac{1}{2}}\epsilon_{rnd}(s,a,z,\theta_{rnd})^{2}$.

### B.2 Hyperparameter Settings

To ensure a consistent basis for comparison across our findings, all experimental work was carried out using a shared codebase. We adopted standardized modeling approaches, including uniform choices for elements like network architectures and optimization algorithms, where appropriate. Specifically, every experiment employed the same architecture as detailed in Appendix [B.1](https://arxiv.org/html/2505.21119v2#A2.SS1). Key hyperparameters, encompassing both foundational and algorithm-specific settings, were tuned through a grid search on the $10\times 10$ variation of the GoToDoor environment. The search grid and final hyperparamters are provided in Tables [2](https://arxiv.org/html/2505.21119v2#A2.T2) and  [3](https://arxiv.org/html/2505.21119v2#A2.T3) respectively. DQN in Table [3](https://arxiv.org/html/2505.21119v2#A2.T3) refers to the online data collection agent.

**Table 2: Hyperparameter search space**
| Hyperparameter | Values |
| --- | --- |
| $Q$ Learning rate (all) | $[1\cdot 10^{-6},3\cdot 10^{-6},1\cdot 10^{-5},$ |
|  | $3\cdot 10^{-5},1\cdot 10^{-4},3\cdot 10^{-4},1\cdot 10^{-3}]$ |
| Prior function scale (BDQNP) | $[0.1,0.3,1.0,3.0,10.0]$ |
| RND Learning rate (RND, RND-P) | $[1\cdot 10^{-6},3\cdot 10^{-6},1\cdot 10^{-5},$ |
|  | $3\cdot 10^{-5},1\cdot 10^{-4},3\cdot 10^{-4},1\cdot 10^{-3}]$ |
| UVU Learning rate (UVU) | $[1\cdot 10^{-6},3\cdot 10^{-6},1\cdot 10^{-5},$ |
|  | $3\cdot 10^{-5},1\cdot 10^{-4},3\cdot 10^{-4},1\cdot 10^{-3}]$ |

**Table 3: Hyperparameter settings for GoToDoor experiments.**
| Hyperparameter | DQN | BDQNP | DQN-RND | DQN-RND+P | UVU |
| --- | --- | --- | --- | --- | --- |
| Adam $Q$-Learning rate | $3\cdot 10^{-4}$ | $3\cdot 10^{-4}$ | $3\cdot 10^{-4}$ | $3\cdot 10^{-4}$ | $3\cdot 10^{-4}$ |
| Prior function scale | n/a | 1.0 | n/a | n/a | n/a |
| N-Heads $1$ | $1$ | $1$ | $1$ / $512$ | $1$ / $512$ | $1$ / $512$ |
| Ensemble size | n/a | $3$ / $15$ | n/a | n/a | n/a |
| MLP hidden layers | $3$ |  |  |  |  |
| MLP layer width | $512$ |  |  |  |  |
| Discount $\gamma$ | $0.9$ |  |  |  |  |
| Batch size | $512$ |  |  |  |  |
| Adam epsilon | $0.005/\text{batch size}$ |  |  |  |  |
| Initialization | He uniform [He et al., 2015] |  |  |  |  |
| Double DQN | Yes [Hasselt, 2010] |  |  |  |  |
| Update frequency | $1$ |  |  |  |  |
| Target lambda | $1.0$ |  |  |  |  |
| Target frequency | $256$ |  |  |  |  |

**Table 4: GoToDoor Environment Settings**
| Parameter | Value |
| --- | --- |
| State space dim | 35 |
| Action space dim | 3 |
| Task space dim | $6$ |
| N Task Rejections | $4$ |
| Max. Episode Length | 50 |

### B.3 Additional Experimental Results

Figure: Figure 6: Top Row: RND errors. 2nd Row: Value uncertainty as measured by an intrinsic $Q$-function. 3rd Row: Value uncertainty as measured by an intrinsic $Q$-function with intrinsic reward priors.
Refer to caption: x8.png

We report additional results of the illustrative experiment shown in Section [3](https://arxiv.org/html/2505.21119v2#S3). Fig. [6](https://arxiv.org/html/2505.21119v2#A2.F6) shows different uncertainty estimates in the described chain environment. The first row depicts myopic uncertainty estimates or, equivalently, RND errors. The second and third row show propagated local uncertainties with and without the intrinsic reward prior mechanism respectively. This result shows clearly the shortcoming of the standard training pipeline for intrinsic rewards: in a standard training pipeline, the novelty bonus of RND is given only for transitions $(s_{i},a_{i},z_{i},s_{i}^{\prime})$ already present in the dataset and is never evaluated for OOD-actions. To generate reliable uncertainty estimates, RND requires, in addition to the RND network and the additional intrinsic $Q$-function, an algorithmic mechanism such as the intrinsic reward priors or even more sophisticated methods as described by Rashid et al. [2020].

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Could the authors provide results demonstrating that the method improves sample efficiency in online RL algorithms or facilitates safer exploration?

2. Is the method inherently limited to offline data and multi-task settings—where the number of policies is constrained and policy conditioning is easier to learn—compared to online RL, where the policy continually evolves as the agent learns the task?
soundness: 2
strengths: 1. Improving the computational efficiency of epistemic uncertainty quantification is a key objective for RL algorithms.

2. The paper provides a formal proof, grounded in Neural Tangent Kernel (NTK) theory, that its single-network uncertainty estimate is mathematically equivalent to the variance of an infinite ensemble, moving it beyond a simple heuristic methods like Random Network Distillation (RND).

3. The policy-conditioned nature of the value function enables UVU to capture long-term, plan-dependent uncertainty. By making the policy representation an explicit input, the model is forced to learn a "universal" value function that understands how outcomes change under different plans. This design is crucial because it allows the temporal difference (TD) learning process to properly propagate uncertainty over time. If a specific policy leads to novel or data-sparse states far in the future, the TD updates naturally "back up" that high uncertainty to the agent's current state, providing a holistic, long-term risk assessment for that specific plan, rather than just a myopic "novelty" score of the current state.
summary: The paper proposes a novel and computationally efficient single-network method (called "UVU") for estimating epistemic uncertainty in reinforcement learning. This approach measures uncertainty as the prediction error between an "online" network and a fixed, randomly initialized "target" network. The online network is trained using temporal difference learning on a synthetic reward signal generated by the target network itself, allowing the error to capture long-term value uncertainty. The paper provides a strong theoretical proof showing that, under infinite-width network assumptions, this single-model error is mathematically equivalent to the variance of a large, computationally expensive ensemble.
weaknesses: **Lack of empirical results** 

A potential weakness is the paper's limited evaluation of the intrinsic quality of the uncertainty estimates. The experiments focus on a binary task-rejection benchmark, which demonstrates the utility of the uncertainty but doesn't thoroughly analyze its calibration, its correlation with true value error across different states, or how it behaves in simpler, more controlled environments.

1. The paper compares its approach to Random Network Distillation (RND) and ensemble-based methods for epistemic uncertainty estimation, but it does not provide results on standard online RL benchmarks, where these methods have traditionally been evaluated. Including such results is crucial to demonstrate the practical utility of the learned uncertainty for efficient or safe exploration.

2. Because the method estimates policy-conditioned value uncertainty, its applicability in single-task settings—where the policy continuously evolves as the agent learns about the environment—remains unclear. This raises questions about the method’s feasibility in such scenarios.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: - You have only demonstrated the performance of UVU on an offline RL task. Do you expect UVU to behave differently in an online RL setting? During online training, the state distribution shifts, leading to a shifting critic as well (the critic network might be subject to catastrophic forgetting). Can this be accurately captured by UVU?
- During reading, i was strugling a bit with the introduction of the multi-head variant of UVU (maybe just due the clarity of writing). Could you strenghten the reasoning behind this analysis? In which case would a single UVU network no be sufficient as an estimator?
soundness: 3
strengths: + The paper is crafted very well, with high-quality writing, figures, and appendices
+ The topic of uncertainty estimation for RL (including for critics/value functions) remains relevant, with no widely accepted approaches. Existing techniques like deep ensembles or MC Dropout are available, but are not widespread (e.g., due to the additional computational demands or potential training instability), and MC Dropout in particular has been shown to provide mixed results
+ The motivation and preliminary sections are comprehensive and seem to cover all important aspects
+ The theoretical analysis utilizing NTK to show equality of uncertainty estimation to ensembles is convincing, and provides a strong justification of the approach

Neutral: The paper draws on existing work (RND and NTK). Compared to RND, it provides a theoretically grounded analysis and extension; compared to work on NTK, it is a novel application domain.
summary: This paper presents an approach for general uncertainty estimation of a value function, inspired by random network distillation. The authors present a theoretical motivation for their approach and showcase the effectiveness in an empirical analysis with a gridworld RL task.

This is a well-crafted paper, presenting a convincing and rigorous theoretical analysis. While strictly speaking, the novelty is limited, and the empirical analysis is restricted to a specialized setting, I still think it is a valuable contribution.
weaknesses: - The empirical analysis is somewhat limited (although it is an illustrative use case). I would love to see some additional experiments beyond this scenario. As the method is generalizable, I feel it should be possible to apply it to common benchmark environments (similar to the original RND paper)
- The lack of any implementation limits impact, and also makes it difficult to fully assess reproducibility.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: My major concern is about the coverage rate of the true values. The authors may consider a simulated environment for which the true value functions are exactly known and demonstrate that the confidence intervals constructed by the proposed method can achieve the nomial coverage rate for the true value. Such an evaluation criterion is critically important for real decisions.
soundness: 3
strengths: The paper aims to address an important issue in RL, and the theoretical analysis is strong.
summary: This paper introduces universal value-fucntion uncertainties (UVU), quantifying uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. The paper also provides a theoretical analysis of the proposed approach using neural tangent kernel (NTK) theory, showing that  UVU errors are equivalent to the variance of an ensemble of independent universal value functions in the limit of infinite network width.
weaknesses: The central concern with the paper is the numerical study. Suppose that the true values are known for the simulated environment, the experiments should include coverage rates (and related calibration metrics). Reporting only the best policy’s returns with confidence intervals does not fully characterize estimator performance or uncertainty.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is there any observed correlation between the smallest eigenvalue of $\Theta_{XX}-\gamma\Theta_{X'X}$ and TD loss divergence?
2. How does the implementation gap such as inter-head correlations affect the uncertainty calibration?
3. Do you have plans to extend UVU to continuous control (e.g., D4RL) or stochastic gridworld environments?
soundness: 3
strengths: - **Conceptual simplicity**: UVU replaces RND’s myopic novelty signal with policy-conditioned, future-aware value uncertainty. The concept of modeling TD error between learner and fixed random teacher provides an intuitive mechanism for epistemic uncertainty estimation.
- **Theoretical grounding**: The paper rigorously analyzes TD learning dynamics under NTK, presenting explicit mean-covariance forms for post-training predictions and proving the equivalence between UVU’s prediction error and ensemble variance.
- **Empirical effectiveness**: The proposed multi-head network achieves ensemble-level uncertainty quality while reducing computational cost by roughly an order of magnitude.
summary: This paper proposes Universal Value-Function Uncertainties (UVU), a single network method for estimating policy-conditioned value uncertainty. The key idea is to train an online predictor via semi-gradient TD learning on synthetic rewards generated by a fixed random target network, and interpret the prediction error as an uncertainty measure. Using neural tangent kernel (NTK) theory, the authors derive post-training distributions of TD trained functions and show that, under the infinite-width and gradient-flow limit, the expected UVU prediction error is equivalent to the ensemble variance of universal $Q$ function. A multi-head version further yields a finite-sample equivalence to a scaled $\chi^2$ distribution.
weaknesses: 1. **Idealized stability condition**: Theoretical TD learning convergence condition (i.e., $\Theta_{XX}-\gamma\Theta_{X'X}\succ0$) depends on the transition structure and the choice of $\gamma$, which is difficult to verify or enforce in practice. Although the appendix suggests using Gershgorin’s theorem and gradient-norm conditioning to ensure diagonal dominance, it remains unclear whether this condition holds empirically or how violation manifests as instability (e.g., divergence of TD loss).
2. **Theory–implementation gap**: The theory assumes gradient flow, full-batch optimization, and stop-gradient TD, whereas the implementation employs mini-batches, target networks, and finite training steps. These deviations could introduce bias or affect convergence, yet no quantitative analysis is provided. In particular, the multi-head version uses a shared trunk, violating the independence assumption in Corollary 2 and potentially leading to systematic underestimation of uncertainty.
3. **Limited empirical generality**: Experiments are limited to a single offline MiniGrid setting. Evaluation on continuous control, stochastic, or partially observable environments would strengthen generalizability and robustness.


Decision: Accept (Poster)