Universal Value-Function Uncertainties
Authors: Moritz Akiya Zanger
Max Weltevrede
Yaniv Oren
Pascal R. Van der Vaart
Caroline Horsch
Wendelin Boehmer
Matthijs T. J. Spaan
Venue: ICLR 2026 Poster (2026)
Keywords: Uncertainty Quantificaiton
Epistemic Uncertainty
Exploration
Offline RL
Neural Tanget Kernel
Multitask RL
Primary area: reinforcement learning
TL;DR: We introduce a single-model method for directly estimating value-function uncertainty for any given policy.

Abstract: Estimating epistemic uncertainty in value functions is a crucial challenge for many aspects of reinforcement learning (RL), including efficient exploration, safe decision-making, and offline RL. While deep ensembles provide a robust method for quantifying value uncertainty, they come with significant computational overhead. Single-model methods, while computationally favorable, often rely on heuristics and typically require additional propagation mechanisms for myopic uncertainty estimates. In this work we introduce universal value-function uncertainties (UVU), which, similar in spirit to random network distillation (RND), quantify uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. Unlike RND, UVU errors reflect policy-conditional $\textit{value uncertainty}$, incorporating the future uncertainties $\textit{any policy}$ may encounter. This is due to the training procedure employed in UVU: the online network is trained using temporal difference learning with a synthetic reward derived from the fixed, randomly initialized target network. We provide an extensive theoretical analysis of our approach using neural tangent kernel (NTK) theory and show that in the limit of infinite network width, UVU errors are exactly equivalent to the variance of an ensemble of independent universal value functions. Empirically, we show that UVU achieves equal performance to large ensembles on challenging multi-task offline RL settings, while offering simplicity and substantial computational savings.

Published as a conference paper at ICLR 2026
UNIVERSAL
UNCERTAINTIES
Moritz A. Zanger, Max Weltevrede, Yaniv Oren, Pascal R. Van der Vaart, Caroline Horsch, Wendelin Böhmer & Matthijs T. J. Spaan
Department of Intelligent Systems, Delft University of Technology Delft, 2628 XE, The Netherlands
Correspondence to: m. â zanger @ teterrruuuuueee tudelft todelft..n
ABSTRACT
Estimating epistemic uncertainty in value functions is a crucial challenge for many aspects of reinforcement learning (RL), including efficient exploration, safe decision-makeen making, and offline RL. While deep ensembles provide a robust method for quantifying value uncertainty, they come with significant computational over- head. Single-model methods, while computationally favorable, often rely on heuristics and typically require additional propagation mechanisms for myopic uncertainty estimates. In this work we introduce universal tainties (UVU), which, similar in spirit to random network distillation (RND), quantify uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. Unlike RND, UVU enrors reflect policy- conditional value uncertainty, incorporating the future uncertainties any given policy may encounter. This due the training procedure employed in UVU: the online network is trained using temporal difference learning with synthetic reward derived from the fixed, randomly initialized target network. We provide an extensive theoretical analysis of our approach using neural tangent kernel (NTK) theory and show that the limit of infinite network width, UVU errors are exactly equivalent to the variance of an ensemble of independent universal value functions. Empirically, we show that UVU achieves equal performance to large ensembles on challenging multi-task offline RL settings, while offering simplicity and substantial computational savings.
uncer-
INTRODUCTION
Deep reinforcement learning (RL) has emerged as an essential paradigm for addressing difficult sequential decision-making problems (Mnih et al.) 2015 Silver et al., 2016 Vinyals et al. 2019) but a more widespread deployment of agents to real-world applications remains challenging. Open problems such efficient exploration, scalable offline learning and safety pose persistent obstacles to this transition. Central to these capabilities the quantification of epistemic uncertainty, an agent's uncertainty due to limited data. In the context of RL, uncertainty estimation relating to the value function of particular importance as it reflects uncertainty about long-term consequences of actions.
However, computationally tractable estimation of falue-function uncertainty remains challenge. Bayesian RL approaches, both in its model-based (Ghavamzadeh et al., 2015) and model-free (Dear) den al. 1998) flavors, typically come with sound theoretical underpinnings but face significant computational hurdles due to the general intractability of posterior inference. Theoretical guarantees of the latter are moreover often complicated by the use of training procedures like temporal difference (TD) learning with bootstrapping. Conversely, deep ensembles (Lakshminarayanan et al., (017) have emerged as reliable standard for practical value uncertainty estimation in deep RL (Osband et al. 2016] Chen et al.) (017). Empirically. independently trained value functions from random initialization provide effective uncertainty estimates that correlate well with true estimation errors. Al- though in general more tractable than full posterior inference, this approach remains computationally challenging for larger models where a manyfold increase in computation and memory severely limits scalability. Various single-moder approaches like random network distillation (RND) (Burda et al.) 2201),, pseudo counts (Bellemare et al. 2016) or intrinsic curiosity (Pathak et al. 2017)) efficiently capture myopic epistemic uncertainty but require additional propagation mechanisms obtain value

---

Published as a conference paper at ICLR 2026
uncertainties (O (O'onoghue et al. 20188 anz et al., 2019 Zhou al. 2020 and often elude thor-
ough theoretical understanding. We conclude that there persists a lack of computationally efficient single-model approaches with the abiliity to directly estimate policy- seieeydeeeendent value uncertainties with a strong theoretical foundation.
To this end, we introduce universal value-function uncertainties (UVU), novel method designed to estimate epistemic uncertainty of value functions for any given policy using a single-model architecture. Similar in spirit to the well-known RND algorithm, UVU quantifies uncertainty through a prediction error between an online learner u and a fixed, randomly initialized target network Crucially, and in contrast to the regression objective of RND, UVU optimizes its online network using temporal difference
(TD) learning with synthetic reward rgg generated entirely from the target network g. By construction, the reward rgg implies a value learning problem to which the target function itself is a solution, forcing the online learner u. to recover g through minimization of TD losses. UVU then quantifies uncertainty as the squared prediction error between online learner and fixed target function. Unlike previous methods, our design requires no training of multiple models (e.g.. ensembles) nor separate value and uncertainty models (e.g., RND, ICM). Furthermore, design UVU as universal colicy-connitioone approximators (Schaul et al. 2013)), that is, it takes as input a state, action, and policy encoding and predicts the epistemic uncertainty associated with the value function for the encoded policy.
model (comparable to universal value function
A key contribution of our work is a thorough theoretical analysis of UVU using the framework of neural tangent kernels (NTK) (Jacot et al. 2018). Specifically, we characterize the learning dynamics of wide neural networks with TD losses and gradient descent to obtain closed-form solutions for the convergence and generalization behavior of neural network value functions. In the limit of infinite network width, we then show that prediction errors generated by UVU are equivalent to the variance of an ensemble of universal value functions, both in expectation and with finite sample estimators.
We validate UVU empirically on an offline multi-task benchmark from the minigrid suite where agents are required to reject tasks they cannot perform to achieve maximal scores. We show that UVU's uncertainty estimates perform comparably to large deep ensembles, while drastically reducing the computational footprint.
2 PRELIMINARIES
We frame our work within the standard Markov Decision Process (MDP) (Bellman 1957
formalism, defined by the tuple AS, 7, p.). Here, is the state space, A is the action space, : S SxA A o (R) is the distribution of immediate rewards, 7 E [0, 1) is the discount factor, P: S x A - (S) is the transition probability kernel, and μ (S) is the initial state distribution. An RL agent interacts with this environment by selecting actions according to policy π : S + 9 (A). At each timestep f, the agent is in state St, takes action A, ~ TT-(Si), receives a reward R₂ ~ R(-|S, At), and transitions to new state S1+1 ~ P(. A.). We quantify the merit of taking actions A, in state S, = s and subsequently following policy π by the action-value function, or Q-function Q² S x A + IR, which accounts for the cumulative discounted future rewards and adheres to recursive consistency condition described by the Bellman equation
Q" a) = ER..,, (Ro + " At = Ao = a).
(1)
The agent's objective then is to maximize expected returns ((T) Au
(So, Ao)]
Often, we may be interested in agents capable of operating a variety of policies to achieve different goals. Universal value function approximators (UVFAs) (Schaul et al. 2015) address this by conditioning value functions additionally on an encoding E Z. This encoding specifies a current policy context, indicating for example a task or goal. We denote such universal Q-functions as Q(s, a, 2). In the context of this work, we consider z to be a parameterization or indexing of a specific policy (- z). or in other words Q : x A x Z + R, Q(s, 2) Q=(1)s.) (s, a).
Both in the single and multi task settings, obtaining effective policies may require efficient exploration and an agent's ' ability to reason about epistemic uncertainty. This source of uncertainty, in contrast to aleatoric uncertainty, stems from a lack of knowledge and may in general be reduced by the acquisition of data. In the context of RL, we make an additional distinction between myopic uncertainty and value uncertainty.
2

---

Published as a conference paper at ICLR 2026
2.1 MYOPIC UNCERTAINTY AND NEURAL TANGENT KERNELS
Myopic uncertainty estimation methods, such as RND or ensembles predicting immediate rewards or next states, quantify epistemic uncertainty without explicitly accounting for future uncertainties along trajectories. We first briefly recall the RND algorithm (Burda et al. 2019). before introducing the neural tangent kernel (NTK) (Jacot al.
2018 framework.
Random network distillation comprises two neural networks: A fixed, randomly initialized targer network g( Vu), and predictor network u(z; &t). The online predictor u(x; 0,) is trained via gradient descent to minimize a square loss between its own predictions and the target network's 's output on set of data points X' = {ax E Id No The RND prediction enror at a test point z then serves as an uncertainty or novelty signal. The loss and error function of RND are then given as
(2) This mechanism relies the idea that the predictor network recovers the outputs of the target network only for datapoints contained in the dataset z; E X, while a measurable error e persists for oft--of-disrrbuun
fidd (0)) = [ (u(X; 0,) - ((xx=))))
and (ma de t/)) = 1 (u(2; ie) - g(;; (0)))
test samples FT & X, yielding measure of epistemic uncertainty.
Next, we introduce the framework of neural tangent kemelss, an analytical framework we intend to employ for the study of neural network and deep ensemble behavior. Consider a neural network f(r, 0.) TR - RR with hidden layer widths 71. 71....,, Rt.. = n. and inputs x E R==,, dataset X, and labels y = {u E R. Inputs 2; may, for example, be state-action tuples and labels may be rewards. The network parameters Bu E Ih' are initialized randomly Bu (N, 1) and updated with gradient descent with infinitesimal step sizes, also called gradient flow. In the limit of infinite width n, the function initialization [( 0n). as shown by Lee et al. (2018), is equivalent to a Gaussian process prior with a specific kernel i. Ii x R - R called the neural network Gaussian process (NNGP). The functional evolution of f through gradient flow is then governed by a gradient inner product kernel θ
Th
IR yielding
O x Vos(z, 0o) Vol(2', 0n), and <(x, 2') = Ei,(x, (o) ,0)]
(3)
Remarkably, seminal work by Jacot et al. (2018) showed that in the limit of infinite width and appro- priate parametrization] the kernel θ becomes deterministic and remains constant throughout training. This limiting kernel, referred to as the neural tangent kernel (NTK), leads to analytically tractable training dynamics for various loss functions, including the squared loss [().) = 1 (xx: 0.) - ylB-- Owing to this, one can show (Jacot al., 2018 [ee et al.) 2020) that for - oo post conver- gence function evaluations f(XT, 0so) on a set of test points Xr, too, are Gaussian with mean ([S(Kr, 0x)) = xx,x x and covariance
Cor 0x)] = Kx - (0x,,x xx + h.c.) + xx xxxxxxxxxxxx xX
(4)
where h.c. denotes the Hermitian conjugate of the preceding term and we used the shorthands Ox,x, (X, and -x = x(X,,, This expression provides a closed-form solution for the epistemic uncertainty captured by an infinite ensemble of NNs in the NTK regime trained with square losses. For example, the predictive variances of such ensembles are easily obtained as the diagonal entries of Eq. 4 While requiring idcalized setting. NTK theory offers solid thcoretical grounding for quantifying the behavior of deep ensembles and, by extension, myopic uncertainty estimates from related approaches. However, this analysis does not extend to value functions trained with TD losses and bootstrapping as is common in practical reinforcement learning settings.
2.2 VALUE UNCERTAINTY
In contrast to myopic uncertainties. value uncertainty quantifies a model's lack of knowledge in the value (*((s, a). As such it inherently depends on future trajectories induced by policies т. Due to this need to account for accumulated uncertainties over potentially long horizons, value uncertainty estimation typically renders more difficult than its myopic counterpart.
widely used (echnique(Osband et al.) 2016 Chen al. 2017: An et al.) 202I) to this end is the use of deep ensembles of value functions ( a, 0.) S x A RR from random initializations Og- Q- functions are trained on transitional data XTD {st, a; j XtD { N.p.No and [r
No
so-called NTK parametrization scales forward/hackward passes appropriately, see Jacot et al. 2018
3

---

Published as a conference paper at ICLR 2026
where oi are samples from the transition kernel P and af are samples from policy Q-functions are then optimized through gradient descent on a temporal difference (TD) loss given by
((,,)= 11 h2Q" (KyD. 0s)l +r Q" 6t) 11..
(5)
where Hss indicates stop-graatee gradient operation. Due to the stopping of gradient flow through Q(X', Oz), we refer to this operation as semi amiiggradeent updates. Uncertainty estimates can then be obtained as the variance of a) Ve,1Q(e (s,, a, 8t)] between ensembles of Q-functions from random initializations. While empirically successful, TD-trained deep ensembles are not as well understood as the supervised learning setting outlined in the previous section 2.1 Due to the use of bootstrapped TD losses, the closed-form NTK regime solutions in Eq. do not apply to deep value function ensembles.
An alternative to the above approach is the propagation of myopic uncertainty estimates. Several prior methods O' O'Donoghue et al. 2018) Zhou et al. 2020 Luis et al. 2023) formalize this setting under a model-based perspective, where transition models P[-|s,0 a) are sampled from a Bayesian posterior conditioned on transition data up to t. For acyclic MDPs, this setting permits consistency condition similar to the Bellman equation that upper bounds value uncertainties recursively. While this approach devises a method for obtaining value uncertainties from propagated myopic uncertainties, several open problems remain, such as the tightness of model-free bounds of this kind (Janz et al., 2019) Van der Vaart et al. 2025) as well as how to prevent underestimation of these upper bounds due to the use of function approximation (Rashid et al. 2020 Zanger et al. 2024).
UNIVERSAL VALUE-FUNCTION UNCERTAINTIES
Our method, universal value-function uncertainties (UVU), measures epistemic value uncertainty as the prediction errors between an online learner and fixed larget network, similar in spirit random network distillation (Burda et al.) 2019). However, while RND quantifies myopic uncertainty through immediate prediction errors, UVU modifies the training process of the online learner such that the resulting prediction errors reflect value-function uncertainties, that is, uncertainty about long-term returns under a given policy.
Our method centers around the interplay of two distinct neural networks: online learner u(s, a, z, 0.) : S A 2 + R. parameterized by weights 0., and a fixed, randomly initial- ized target network g(s, tho) S Ax Z R. parameterized by weights v/o. Given a transition (s, a, s') and policy encoding z, we draw subsequent actions from a policy (-Iss, z). Then, we the fixed target network g generate synthetic rewards as
as, a') g(s, a, 2, vo) - 7g(3' 2,'
(6)
While the weights thu of the target network remain fixed at initialization, the online network u is trained to minimize a TD loss using the synthetic reward rz. Given a dataset X' = {sr, as 21 N we have
1 No
((U,) 2ND (n
d4, Dellag + (s a, 6,, o.) u(s;, as, 24, 02))
where Fog indicates a stop-gradient operation. For any tuple (s, a, 2) (E X' or not), we measure predictive uncertainties squared prediction errors between the learner and the target function
(s, a, 2, 0,, 22o)2 (u(s, a, 0.) gs, (1, a, 2, 2,)))
8
The intuition behind this design is that, by construction, the value-function associated with policy T ([1ss and the synthetic rewards (s,, a, 5', .a) exactly equals the fixed target network g(s, a, vo). As sanity check, note that the target function g(s, a, z, v'o) itself satisfies the Bellman equation for the policy (-1ss, 2) and the synthetic reward definition in Eq. (6), constituting a random value valuefuuccie function to rī and hence achieves zero-loss according to Eq. 7 Therefore, if the dataset X' sufficiently covers the dynamics induced by T[-|s, =). the online network u(s, a, z, 00) is able recover g(s, a, 2, v/)) exactly, nullifying prediction errors. However, when data coverage is incomplete for the evaluated policy, minimization of the TD loss 7 is not sufficient for the online network u( 19) to recover target network predictions ((s, a, 2, vo). This discrepancy is captured by the prediction errors, which quantify epistemic uncertainty regarding future gaps of the available data.
4

---

Published as a conference paper at ICLR 2026
26
9
g
14
g
g
14
4.5
4.3
(#1.a)
(#2: a)
(Aa: a)
(#s.
[86, a)
(51.0)
($2, a)
(5a,
Figure 2: (left:) Illustration of uncertainty estimation in tabular UVU with independently initialized
tables for u and g. Access to full trajectory data allows u to recover g. (right:) By executing action "b", trajectories are effectively truncated, preventing from recovering g. All plots use 7 = 0.7.
3.1 BUILDING INTUITION BY AN EXAMPLE
To build intuition for how UVU operates and captures value uncertainty, we first consider a tabular setting with simple chain MDP as illustrated in Figure [ Suppose we collect data from deter- ministic policy ga using action a exclusively. Given this dataset, suppose we would like to estimate the uncertainty associated with the value Q 8.2) a) of policy π T(1s,,) 2) that differs from the data-collection pol- icy in that it chooses action "b" in $3. In our tabular set- ting, we then initialize random tables Usa and Isa- For ev- ery transition (s,, at 81+1) contained in our tragle-trajectory dataset, we draw a++1 ~ I (-Is, z). compute the reward rg,t ggt Sara 790411044 and update table entries with the rule Us.a, f rg. + 1u Fig. visualizes this process for several independently initialized tables (rows in Fig. ( for the data-collecting policy za (left). and for the altered policy (+Is, 2) (right), which chooses action "b" in 83. We outline how this procedure yields uncertainty estimates: We first note, that one may regard g as a randomly generated fulutefunetion, for which we derive the corresponding reward function as rg. As Jea. by construction, is the value-function corresponding to Ty. one may expect that the update rule applied to Usa causes Usa to recover Jsa- Crucially, however, this is only possible if sufficient data is available for the evaluated policy. When policy diverges from available data, as occurs under т (-|18, 2) in 83, this causes an effective truncation of the collected trajectory. Consequently, Maja and Maza receive updates from Magh. which remains at its initialization, rather than inferring the reward- generatingnn function Isa- In the absence of long-term data, the empirical Bellman equations reflected in updates do not uniquely determine the underlying value function Isa- Indeed. both Usa and Isa incur zero TD-error in the r.h.s. of Fig. 2 yet differ significantly from each other. It is this ambiguity that UVU errors (g.a Uga)² quantify. To ensure recovers g, longer rollouts under the policy T (-188, 2) are required to sufficiently constrain the solution space dictated by the Bellman equations (as seen in Fig. lefft)
Figure 1: Chain MDP of length N with unexplored actions 6.
Figure Jillustrates uncertainty estimates for the shown chain MDP using neural networks and for whole family of policies =(-1s, 2) which select the unexplored action b with probability 1 1- z. We analyze the predictive variance of an ensemble of 128 universal Q-functions, each conditioned on the policy π (-1s,2) 2). In the bottom row, we plot the squared prediction error of single UVU model, averaged over 128 independent heads. Both approaches show peaked uncertainty in early sections, as policies are more likely to choose the unknown action "b" eventually, and low uncertainty closer to the terminal state and for close to 1. A comparison with RND is provided in the Appendix B.3
WHAT UNCERTAINTIES Do UNIVERSAL VALUE-FUNCTION UNCERTAINTIES LEARN?
While the previous section provided intuition for UVU, we now derive analytical characterization of the uncertainties captured by the prediction errors e between a converged online learner u and the fixed target g. We turn to NTK theory to characterize the generalization properties of the involved neural networks in the limit of infinite width, allowing us to draw an exact equality between the squared predictions errors of UVU and the variance of universal value function ensembles.
5

---

Published as a conference paper at ICLR 2026
38 10 M
Figure 3: From left right, (1. and 2.): Variance of an ensemble of 128 universal Q-functions trained on a chain MDP dataset. (3. and 4.): Value uncertainty as measured by UVU prediction errors with single 128-headed model. All plots evaluate the "a" action of the chain MDP.
In the following analysis,
the notational shorthand (8, a, =) and x' (s',a', and denote a neural network [(x, 0₂) with hidden layer widths 7i,--., P2 n, transitions from X = {r:} The evolution of the parameters 0g under gradient descent with infinitesimal step sizes, also called gradient flow, is driven by the minimization of TD losses with
{(s;, az, 2:) =)} No to X" = {(${, a4, x)) ) where df T(- [- =x), and rewards
a 0; = -V, and [(O,) = (1f 0.)lg +r (r--f(x,0)13
We study the dynamics induced by this parameter evolution in the infinite-width limit + oc. In this regime, the learning dynamics of f become linear as the NTK becomes deterministic and stationary, permitting explicit closed- frosed-form expressions for the evolution the function f(z 8t). particular, we show that the post convergence function lim, fxr 8,) is given by
(xx,00)) fxx, 0o) 2ax (Oxx xxxx) (f(X, (7f(X', 0o) +r)),
(10)
where Ozz' is the NTK of f. Proof is given in Appendix A. This identity is useful to our analysis as it delineates any converged function fxx Ozx) trained with TD losses 9 through initialization (zz,0)). Theorem [ leverages this deterministic dependency to express the distribution of post convergence functions over random initializations 0o.
Theorem Let fxx,0)) be NN with L hidden layers of width nı,.... nr n trained with gradient flow to reduce the TD loss [(0,) = }1l (X' 0t)ley + - f(X, 0₂) 11.. In the limit of infinite width - oo and time f - oC, the distribution of predictions f (Xr, 0z0) on a set of test points X7 converges to a Gaussian with mean and covariance given by
Eo, [f(XI, 0x0)] = AxxxAx
Cove, [f(XT, 0x)) = KXTXX - (0xx
Oxxx);
where 2zx is the NTK, Kays is the NNGP kernel, h.c. denotes the Hermitian conjugate, and
Dx = θ xx 7 7 &
and Ax= = X x
Proof provided in Appendix A. Theorem Tis significant as allows us to formalize explicitly the expected behavior and uncertainties of neural networks trained with semi-gradient TD losses, including universal value function ensembles and prediction errors UVU. In particular, the variance of ensemble of universal Q-functions Q( 0x0)) over random initializations 9 is readily given by the diagonal entries of the covariance matrix Cov[Q(Xr: d. Applied to the UVU setting, Theorem gives an expression for the converged online network u(x, 0x)) ) = θ AxxAxx trained with the synthetic rewards r = g(X,, vho) - (X"",, 1/0). From this, It is straightforward to obtain the distribution of post convergence prediction errors { Doo. 1n) In Corollary I we use this insight to conclude that the expected squared prediction errors of UVU precisely match the variance of value functions Q(x, 0xx) from random initializations fy.
Corollary 1. Under the conditions of Theorem 7 let u(æ, 020) be a converged online predictor trained with synthetic rewards generated by the fixed target network g(x, vo)) with re 9(X, , th) (2(" vo). Furthermore denote the variance of converged universal Q-functions Vo, (Q(r, 0z0))

---

Published as a conference paper at ICLR 2026
Width ablation
Jose Rustine (tan) D00k Graltnn Step
Rontime Analysis
(a)
(b)
(c)
Avg. Reture
BUUNNPU:
UVU DQN
125
512
1024
2045
UVD
BEgNPPI
Network width
Figure 4: (a) Ablation on GoToDoor-10 with different network widths. Shaded region indicates standard deviations over 5 seeds. (b) Runtime of various ensemble sizes vs. UVU. Ensembles are implemented with vmap in JAX (AXaBrayb et al., 2018). (c) Illustration of the GoToDoor environment. The agent (red triangle) must navigate to the door indicated by the task specification 2.
Assume u. 9, and Q are architecturally equal and parameters are drawn i..dd On 6o, to ~N 1). The expected squared prediction error coincides with Q-function variance
Ed [2,, So who) Vo [Q(z, 0x)))
(11)
where the Lh.. expectation and rh.s. variance are taken over random initializations do, 1o, On-
Proof is given in Appendix A.1.3 This result provides the central theoretical justification for UVU: in the limit of infinite width, our measure of uncertainty, the expected squared prediction error between the online and target network, mathematically equivalent to the variance one would obtain by training an ensemble of universal Q-functions.
In practice, we are moreover interested in the behavior of finite estimators, that is, ensemble variances estimated with a finite number of models. We furthermore implement UVU with number of multiple independent heads and 9, with shared hidden layers. Corollary 2 shows that the distribution of the sample mean squared prediction error from M heads identical to the distribution of the sample variance of an ensemble of M + independently trained universal Q-functions.
Corollary 2. Under the conditions of Theorem ! consider online and target networks with M independent heads Us, 9ú. = I..... M, each trained to convergence with errors (x(x, 0 vo) Let 5((x, 9oo, 120)2 = 2M Ein ej 9,,, v/022 2 be the sample mean squared prediction error over M heads. Moreover, consider M independent converged Q-functions Qi(ı; 0.0)) and denote their
sample variance o2 0x0) = 38 EM++ (z; 0z0) - Q(; o)))) where Q is the sample mean. The two estimators are identically distributed according scaled Chi-squared distribution
1E(2, 0.. v0)² 2z 0x0) (2)2,000)
2g
M (MM)
(12)
with M degrees of freedom and (3 00w) Voo [Qxx; (r, 0ze))
is the analytical variance of converged
O-functions given by Theorem 7
Proof is provided in Appendix A.2.3 The distributional equivalence of these finite sample estimators provides theoretical motivation for using a multi headed architecture with shared hidden layers within a single UVU model and its use as an estimator for ensemble variances of universal Q-functions. While the assumptions of infinite width and gradient flow are theoretical idealizations, several empirical results suggest that insights from the NTK regime can translate well to practical finite width deep learning models (Lee et al. 2020 Liu et 2020 Tsilivis and Kempe 2022). motivating further empirical investigation in Section
EMPIRICAL ANALYSIS
Our empirical analysis is designed to assess whether UVU can effectively quantify value function uncertainty in practical settings, comparing its performance against established baselines, particularly deep ensembles. Specifically, we aim to address the following questions:
7

---

Published as a conference paper at ICLR 2026
1. Does the theoretical motivation for UVU hold in practice and do its uncertainty estimates enable effective decision decision-making comparable to deep ensembles?
2. How are uncertainty estimates generated by UVU affected by deviations from our theoretical analysis, namely finite network width?
To address these questions, we focus on an offline multitask RL setting with incomplete data where reliable uncertainty estimation crucial to attain high performance.
Code for experimental reproduction is available at github. ctt. http://www............./aaaaaaa.
5.1 EXPERIMENTAL SETUP
In experimental analysis, we an offline variant of the GoToDoor environment from the Minigrid benchmark suite (Chevalier- Boisvert et al.] (023). An example view is shown in Figure D
Shlevvlier-Boisvert
(c). In this task, the agent navigates grid world containing four doors of different colors, placed at random locations and receives a task specification indicating target door color. Upon opening the correct door, the agent receives reward and placed in random different location. Episodes are of fixed length and feature a randomly generated grid layout and random door positions / colors. In our experiments, we use variations of different difficulties by increasing maximum grid sizes.
Dataset Collection. A dataset D = {(s,, ae, T4, 24, sy ) 3 is collected using a policy that per- forms expertly but systematically fails for certain task/grid combinations (e.g., it can not successfully open doors on the "north" wall, irrespective of color or grid layout). Policies seeking to improve upon the behavior policy thus ought to deviate from the dataset, inducing value uncertainty.
Task Rejection Protocol. All baselines implement a DQN-based agent trained in an offline fashion on D. the agents aim to learn an optimal policy for all grids and tasks contained in D, the resulting greedy policy tends to deviate from the available data when the collecting policy is suboptimal. We employ a task-rejection protocol to quantify an agent's 's ability to recognize this divergence and the associated value uncertainty. As most task/grid combinations are contained in D. though with varying levels of policy expertise, myopic uncertainty not sufficient for fulfilling this task. Specifically, upon encountering the initial state So, the agent is given opportunity to reject fixed selection of tasks (here door colors). It is subsequently given one of the remaining, non-rejected tasks and performance measured by the average return achieved the attempted task. Successful agents must thus either possess uncertainty estimates reliable enough to consistently reject tasks associated with a data/policy mismatch or rely on oft-o--distribution generalization. Similar protocols, known accuracy rejection curves, have been used widely in the supervised learning literature (Nademm et al. 2009).
5.2 RESULTS
We conduct experiments according to the above protocol and perform a quantitative evaluation of UVU and several baseline algorithms. All agents are trained offline and use the basic DQN architecture (Mnih et af.] 2015) adapted for universal value functions, taking the task encoding 2 as an additional input to the state (details are provided in Appendix [B). Specifically, we compare UVU against several baselines: A DQN baseline with random task rejection (DQN); Bootstrapped DQN with randomized priors (BDQNP) (Osband et al., 2019); A DQN adaptation of random network distillation IDQN-RND (Burda et al.) 2019) and a version adapted with the uncertainty prior mechanism proposed by Zanger et (2024) LDON-RND- Except for the DQN baseline, all algorithms reject tasks based on the highest uncertainty estimate, given the initial state so and action ag, which is chosen greedily by the agent.
Table shows the average return achieved by each method on the GoToDoor experiment across different maximum grid sizes, with average runtimes displayed in F (b). This result addresses our first research question regarding the practical effectiveness of UVU compared ensembles and other baseline methods. As shown, the standard DQN baseline performs significantly worse than uncertainty-be based algorithms, indicating that learned Q- Q-functions do not generalize sufficiently to counterbalance inadequate uncertainty estimation. Both small and large ensembles significantly improve performance by leveraging uncertainty to reject tasks and policies associated with missing
8

---

Published as a conference paper at ICLR 2026
Table Results of offline multitask RL with task rejection on different variations of the GoToDoor environment. Results are average evaluation returns of the best-performing policy over 10 gradient steps and intervals are 90% student's confidence intervals from 10 independent seeds.
Size
DQN
B3QNP(3)
BDQNP(15)
BDQNP(35)
DQN-RND
DQN-RND-P
UVU (Ours)
5
5.50
+ .15
8.60
.24
10.50 + .04
10.58 +: .03
3.94
+: .50
10.41 +
.12
10.54 + .03
6
4.93
L. .12
7.66
L .09
9.39
1 .04
9.57
1 .04
1.99
1 .40
9.28
1 .12
9.54
I .03
4.58
1: .09
6.61
.16
6.49 1 .05
8.75 1 .06
2.66
1 .43
8.12
.23
8.73
.04
+.
.10
7.68
7.92
2.53
7.40
9
3.66
5.04
.08
6.69
.07
7.03
.13
2.39
1 .38
6.39
.19
7.29
10
3.39
4 111
4.64
.14
6.09
+ .13
6.53
+
.16
2.25
1
.48
5.64
1
.17
6.72
- .12
data.
agents perform well when intrinsic reward priors used. Our approach scores highly and outperforms many of the tested baselines with statistical significance, indicating that is indeed able to effectively quantify value uncertainty using kingle-mode multi-heades architecture.
We furthermore ablate UVU's dependency on network width, given that our thcoretical analysis is situated in the infinite width limit. Fig. 4 (a) shows that UVU's performance scales similarly with network width to DQN and BDQNP baselines, indicating that finite-sized networks, provided appropriate representational capacity, are sufficient for effective uncertainty estimates.
RELATED WORK
body of literature considers the quantification of value function uncertainty in the context of exploration. Early works (Dearden et al. 1998 Engel al. 2005) consider Bayesian adoptions of
model-free RI. algorithms. More recent works provide theoretical analyses of the Bayesian model-
free setting and correct applications thereof (Fellows al. 2021 Schmitt al. 2023 Van der Vaart et al., 2025). which is subject of debate due to the use TD losses. Several works furthermore derive provably efficient model-free algorithms using frequentist upper bounds on values in tabular (Strehl et al. 2006 Jin et (018)) and linear settings (Jin et al.) (202)). Similarly, Yang et al. (2020) derive provably optimisic bounds of value functions in the NTK regime, but in contrast to our work uses local bonuses to obtain these. The exact relationship between bounds derived from local bonuses and the functional variance in ensemble or Bayesian settings remains open.
The widespread use and empirical success of ensembles for uncertainty quantification in deep learning (Dietterich] 22000 Lakshminarayanan et al. 2017) has motivated several directions research towards better theoretical understanding of their behavior. Following seminal works by Jacot et al. (2018) Lee et al. (2020) who characterize NN learning dynamics in the NTK regime, a number works
and Lee have connected deep ensembles Bayesian interpretations (He al. 2020 D Angelo and Fortuin 2021 Moreover, a number of papers have studied the learning dynamics of model-free RL: in the
overparametrized linear settings (iiao et al.) 2021): ; in neural settings for single (Cai et al.) 2019) and multiple layers (Wai et al. (020); to analyze generalization behavior (Lyle et al.) 2022) with
linear and second-order approximations. It should be noted that the aforementioned do not focus on probabilistic descriptions of posterior distributions in the NTK regime. In contrast, our work provides probabilistic closed-form solutions for this setting with semi- comi-gradient TD learning. In practice, the use of deep ensembles is common in RL, with applications ranging from efficient exploration (Osband jet al. 2016 [Cenn et al.)
2017 Osband et al., 22099: Nikolov et al. 2019 Zanger et al. 2024) to
off-policy offline RL
An et al. 2021 Chen et al. 202 Lee et al. 2021 and conservative or safe
RL (Lütjens et al. 2019
Lee et al. 2022 Hoel et al. 2023). Single model methods that aim to reduce
the computational burden of ensemble methods typically operate myopic uncertainty estimators
(Burda et al. 2019 Pathak et al. 2017] Lahlou et al. 2021 | Zanger et al. 2025) and require additional
mechanisms (0 D'Dooggh et al. 2018: Janz al. 20191
Zhou et al.) 2020 Luis et al. 2023).
LIMITATIONS AND DISCUSSION
In this work, we introduced universal value-function uncertainties (UVU), an efficient single-mode method for uncertainty quantification in value functions. Our method measures uncertainties as prediction error between fixed, random target network and an online learner trained with a tem- poral difference (TD) loss. This induces prediction errors that reflect long-term, policy-dependent

---

Published as a conference paper at ICLR 2026
uncertainty rather than myopic novelty. One of contributions is thorough theoretical analysis this approach via neural tangent kernel theory, which, in the limit of infinite network width, establishes an equivalence between UVU emrors and the variance of ensembles of universal value functions. Empirically, UVU achieves performance comparable and sometimes superior to sizeable deep ensembles and other baselines in challenging offline task-rejection settings, while offering substantial computational savings.
We believe our work opens up several avenues for future research: Although our NTK analysis provides a strong theoretical backing, it relies on idealized assumptions, notably the limit of infinite network width (a thoroughgoing exposition of our approximations is provided in Appendix A.3) Our experiments suggest UVU's performance is robust in practical finite-width regimes (Figure 4 yet bridging this gap between theory and practice remains an area for future work. On related note, analysis in the NTK regime typically eludes feature learning. Combinations of UVU with representation learning approaches such as self aelf-predictive auxiliary losses (Schwarzer al. 2020) Guo al. 2022 Fujimoto al. 2023) are, in view, very promising avenue for highly challenging exploration problems. Furthermore, while our approach estimates uncertainty for given policies, it does not devise method for obtaining diverse policies and encodings thereof. We thus believe algorithms from the unsupervised RL Perrature(Toulle and Ollivier 2021 Zheng et al. (2023) naturally integrate with our approach. In conclusion, believe UVU provides strong foundation for future developments in uncertainty-awee agents that are both capable and computationally feasible.
8
ACKNOWLEDGEMENTS
This project has received funding from the EU Horizon 2020 programme Epistemic Al under grant number 964505 and the Dutch Research Council (NWO) project Reliable Out-of-Distribuute Gener- alization in Deep Reinforcement Learning with project number OCENW.M.22 .234, Computational resources for experimental studies were provided by the Delft High Pertormance Computing Centre (DPC) and the Delft Artificial Intelligence Cluster (AIC)..
10

---

Published as a conference paper at ICLR 2026
REFERENCES
G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty-hased offline reinforcement learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-74477 2021.
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
M. Bellemare, Srinivasan, Ostrovski, Schaul, D. Saxton, and Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
R. Bellman. Markovian decision process. Journal of mathematics and mechanics, 6, 1957.
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPias, Wanderman-Miine Milne, and Q. Zhang. JAX: composable transformations of Python+NumPPy programs, 2018. URL http: /ttt::///iitae thub com/ hammm/ax-mm/333
Y. Burda, H. Edwards, Storkey, and O. Klimov. Exploration by random network distillation. In International conference on learning representations, ICL.R, 2019.
Q. Cai, Z. Yang, D. Lee, and Z. Wang. Neural temporal-diiffrencc learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019.
R. Chen, S. Sidor, Abbeel, and Schulman. UCB exploration via Q-ensembles. arXiv preprint orXiv:1706.0// 2017.
X. Chen, C. Wang, Zhou, and K. Ross. Randomized ensembled double Q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
M. Chevalier-Boisverr B. Dai, M. Towers, Perez-Vicente Vicente, L.. Willems, S. Lahlou, S. Pal, Castro, and J. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
Perez-
D' Angelo and Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34:3451-3466 2021.
R. Dearden, N. Friedman, S. Russell, et al. Bayesian Q-learning. Aaai/iaai, 1998:761-768, 1998.
Delft Artificial Intelligence Cluster (DAIC), 2024.
Delft High Performance Computing Centre (DHPC). DelftBlue Supercomputer (Phase 1), 2022.
T. G. Dietterich. Ensemble methods in machine learning. In Multiple classifier systems: First international workshop, MCS. Springer, 2000.
Y. Engel, S. Mannor, and R. Meir. Reinforcement leamning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning, pages 201-208, 2005.
M. Fellows. K. Hartikainen, and S. Whiteson. Bayesian Bellman operators. Advances in neural information processing systems, 34, 2021.
S. Fujimoto, W.-D. Chang. E. Smith. S. S. Gu, D. Precup, and D. Meger. For sale: State-action representation learning for deep reinforcement learning. Advances neural information processing systems, 36:61573-61624 2023.
M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, N. Foerster, and M. Martin. Simplifying deep temporal difference learning. arXiv preprint 2xXiv:240.0088/1 2024.
Gerschgorin. Uber die abgrenzung der eigenwerte einer matrix. Izvestija Akademii Nauk SSSR, Serija Matematika, 7(3):79--754, 1931.
M. Ghavamzadeh, S. Mannor, J. Pineau, and A. Tamar, Bayesian reinforcement learning: A survey. Foundations and trends in machine learning, 8, 2015.

---

Published as a conference paper at ICLR 2026
Z. Guo, S. Thakoor, M. Pislar, B. Avila Pires, F. Altché, C. Tallec, A. Saade, D. Calandriello, J.-B. Grill, Y. Tang, et al. BYOL-Explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35:31855-31885 2022.
H. Hasselt. Double Q-learning. Advances in neural information processing systems, 23, 2010.
B. He, B. Lakshminarayanan, and Y. Teh. Bayesian deep ensembles via the neural tangent kernel. Advances in neural information processing systems, 33, 2020.
K. He. X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on Imagenet classification. Proceedings of the IEEE international conference on computer vision, 2015.
C.-J. Hoel, K. Wolff, and L. Laine. Ensemble quantile networks: Uncertainty aware reinforcement learning with applications in autonomous driving. IEEE Transactions on intelligent transportation systems, 2023.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization neural networks. Advances in neural information processing systems, 31, 2018.
D. Janz, J. Hron, P. Mazur, K. Hofmann, J. M. Hernández-Lobato, and S. Tschiatschek. Successor uncertainties: Exploration and uncertainty in temporal difference learning. Advances in neural information processing systems, 32, 2019.
C. Jin, Z. Allen-Zhu, S. Bubeck, and M. Jordan. q-learning provably efficient? Advances in neural information processing systems, 31, 2018.
C. Jin, Z. Yang. Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory, pages 2137-2143. PMLR, 2020.
A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances neural information processing systems, 33:1179-1191, 2020.
Lahlou, M. Jain, H. Nekoei, Butoi, P. Bertin, Rector-Brooks, M. Korablyov, and Y. Bengio. Deup: Direct epistemic uncertainty prediction. arXiv preprint 2rXX1:21002.08501,2221
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
J. Lee, J. Sohl-dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri. Deep neural networks as gaussian processes. In International conference on learning representations, 2018.
Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. Journal of Statistical Mechanics: Theory and Experiment, 2020, Dec. 2020.
K. Lee, M. Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pages PMLR, 2021.
S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin. Offline-to-onlnne reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
C. Liu, I., Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the tangent kerel is constant. Advances in Neural Information Processing Systems, 33:15954-15964, 2020.
C. E. Luis. A. G. Bottero, J. Vinogradska, F. Berkennamp, and J. Peters. Model-based uncertainty in value functions. In International Conference on Artificial Intelligence and Statistics, pages 8029-8052 PMLR, 2023.
B. Lütjens, M. Everett, and J. P. How. Safe reinforcement learning with model uncertainty estimates. In 2019 International Conference on Robotics and Automation (ICRA), pages 8662-8661 IEEE, 2019.
12

---

Published as a conference paper at ICLR 2026
C. Lyle, M. Rowland, W. Dabney, M. Kwiatkowska, and Y. Gal. Learning dynamics and generalization in reinforcement learning. arXiv preprint arXiv:2206.02126, 2022.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518, 2015.
M. S. A. Nadeem, J.-D. Zucker, and B. Hanczar. Accuracy-rejection curves (arcs) for comparing classification methods with reject option. In Machine Learning in Systems Biology, pages 65-81. PMLR, 2009.
N. Nikolov, J. Kirschner, F. Berkenkamp, and A. Krause. Information-directed exploration for deep reinforcement learning. In International conference on learning representations, ICLR, 2019.
Oshand, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. Advances in neural information processing systems, 29, 2016.
I. Osband, B. Van Roy, D. Russo, Z. Wen, et al. Deep exploration via randomized value functions. Journal of machine learning research, 20, 2019.
B. Donoghue, I. Osband, R. Munos, and Mnih. The uncertainty Bellman equation and explo- ration. In International conference machine learning. PMLR, 2018.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning. PMLR, 2017.
T. Rashid, B. Peng, W. Böhmer, and S. Whiteson. Optimistic exploration even with a pessimistic initialisation. Proceedings of ICLR 2020, 2020.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
S. Schmitt, J. Shawe-Taylor, and H. van Hasselt. Exploration via epistemic value estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023.
M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data-efficient reinforcement learning with seef-predietive representations. arXiv preprint arXiv:2007.05929 2020.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
A. L. Strehl, L. Li, E. Wiewiora, J. Langford, and M. L. Littman. Pac model-free reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pages 88 1888888 2006.
A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13 2--3,,2222222 2021.
N. Tsilivis and J. Kempe. What can the neural tangent kernel tell us about adversarial robustness? Advances in Neural Information Processing Systems, 35:181 16-18830, 2022.
P. R. Van der Vaart, M. T. Spaan, and N. Yorke-Smith. Epistemic Bellman operators. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, nature, 575(7782):350-35 3544, 2019.
H.-T. Wai, Z. Yang, Z. Wang, and M. Hong. Provably efficient neural gtd for off-policy learning. Advances in Neural Information Processing Systems, 18:10431-14444 2020.
13

---

Published as a conference paper at ICLR 2026
C. Xiao, B. Dai. J. Mei, O. A. Ramirez, R. Gummadi, C. Harris, and D. Schuurmans. Understanding and leveraging overparameterization in recursive value estimation. In International Conference on Learning Representations, 2021.
G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv: (e5i::1902.047660 2019.
Z. Yang, C. Jin, Z. Wang, M. Wang. and M. Jordan. Provably efficient reinforcement learning with kernel and neural function approximations. Advances in Neural Information Processing Systems, 1e11390--19166 2020.
Y. Yue, R. Lu, B. Kang, S. Song, and G. Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. Advances in Neural Information Processing Systems, 36:60247- 60277, 2023.
M. A. Zanger, W. Böhmer, and M. T. Spaan, Diverse projection ensembles for distributional reinforcement learning. International conference on learning representations, 2024.
M. Zanger, R. Van der Vaart, W. Böhmer, and Spaan. Contextual similarity distillation: Ensemble uncertainties with single model. arXiv preprint arXiv:2503.11339, 2025.
C. Zheng, R. Salakhutdinov, and B. Eysenbach. Contrastive difference predictive coding. arXiv preprint arXiv:2310.2014// 2023.
Q. Zhou, H. Li, and J. Wang. Deep model-based reinforcement learning via estimated uncertainty and conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence. 2020.
14

---

Published as a conference paper at ICLR 2026
A THEORETICAL RESULTS
This section provides proofs and further theoretical results for universal value-function uncertaintics (UVU).
A.1 LEARNING DYNAMICS UVU
We begin by deriving learning dynamics for general functions with temporal difference (TD) losses and gradient descent, before analyzing the post training distribution of deep ensembles and prediction errors of UVU.
A.1.1 LINEARIZED LEARNING DYNAMICS WITH TEMPORAL DIFFERENCE LOSSES
We analyze the learning dynamics of function trained using semi- ammi-gradient temporal difference (TD) losses fixed dataset of transitions N,X' Let (æx 0,) denote NN of interest with depth L and widths ,-,-... 1.I-1 Y..
Proposition 1. In the limit of infinite width n and infinite time 0o, the function fxx 0z) converges to
f(x, 00c) = f(x, - (0xx - ~x (f(X, - (nf(2', ((2 + +r),
(13)
where Ozx is the neural tangent kernel of f.
Proof. We begin by linearizing the function f around its initialization parameters og: fim (ar, 0.) fxx 0o) Vof (x, To)T(e))) 0g)
(14)
We assume gradient descent updates with infinitesimal step size and learning rate a on the loss
[(,)) = 1ll 7fin 0r)ss + r fin (X. 0,) 113.
(15)
yielding the parameter evolution
d 0. - -Voc(())
(16)
Setting wi = 02 Og and find the learning dynamics:
at Vef 0o) (fin (X', 00) - (i''ii + +r)).
(17)
Thus, the evolution of the linearized function is given by
de fi, 02) = aVof(2 0o) Vof(X 0o) (X, 0:) fim 0,) + +r))
(18)
Letting (TD(6,) fin(X' 0,) - (r.fiin .,) + r), we obtain the differential equation
ar (0.) = aa (o xx - 7 (T)().)
(19)
where 0 = (of(x, ( oo' -x'
, 0b) is the (empirical) tangent kernel of fun 6t). Since the linearization fin( d.) has constant gradients fef(z, 0o), the above differential equation is linear and solvable so long the matrix O * x is positive definite. With exponential ansatz, we obtain the solution
(1.( or (0 - xxx ( (TD(
(20)
where ex is matrix exponential. Reintegrating yields the explicit evolution of predictions
d fan 01) = f(x, - f.'' ftm dyt
(21)
= f(x, 0n) - 1 (0 xx 70 ( -a-a010p -7 - (TD))
(22)
Jacot al. 2018) show that in the limit of infinite layer widths of the neural network, the NTK 6 .2.
becomes deterministic and constant eo ggr - Ozz". As a consequence, the linear approximation
finm a2; f,) becomes exact w.r.t. the original function lim fin (z;; 0,) f(x 8,) (Lee et al.
2020
15

---

Published as a conference paper at ICLR 2026
Remark on the constancy of the NTK in TD learning. We note here, that our proof assumed the results by) Jacot et al. (2018) to hold for the case of semi-gradient TD updates, namely that the NTK becomes deterministic and constant 0 dynamics. First, the determinacy of the NTK at initialization follows from the law of large numbers and applies in our case equally as in the least squares case. The constancy of the NTK throughout training is established by Theorem 2 in Jacot et al. (218), which we restate informally below.
aar in the limit of infinite width under the here shown
Theorem 2. .Jacot et al. [2018)] J In the limit of infinite layer widths + oo and 1i...., RL = n, the kernel O converges uniformly on the interval E [0, T) to the constant neural tangent kernel
1 Zx'
Oxx'
provided that the integral 1-T || l2 dt stays bounded. Here, de E RND is the training direction of the parameter evolution such that d 0₁ -Vooff 0)d,
ät
In the here studied case of semi mriiiigrddentt TD learning, the parameter evolution (as outlined above in Eq. (77 is described by the gradient Vof (X, Bo) and the training direction d, according
d de
f: aa Vof 0o) (fin 01) (r.fua (2', 06) + +r))
(23)
de
where the training direction is given by de (i, - (nfii B.) + r) = ( Provided that the matrix O X'x x'xx is positive definite, the norm of the training direction ||dell decays exponentially by Eq. 20 This implies
||dell2 < [1doll2e-che taain
(24)
where Amia is the smallest eigenvalue of 0% 7 Assuming 0' -702 is positive definite, Amin is positive and as a consequence, we have
| 122 di / |doll2e Ounn di < xxi,
(25)
bounding the required integral of Theorem 2 for any T and establishing ug - Ozz' uniformly on the interval [0, oo) (see Theorem 2 in Jacot al. (2018) for detailed proof for the last statement).
We note, however, that the condition for 0 - stronger than in the classical results for supervised learning with least squares regression. While Oxx be guaranteed to be positive definite for example by restricting X' to lie unit-sphere, I; E X to be unique, and by assuming non-polynomial nonlincarities in the neural network (so as to prevent rank decay in the network expressivity), the condition harder satisfy in the TD learning setting. Here, the eigenspectrum of 6x ~0' x tends to depend on the transitions X - 2" themselves and thus is both dependent on the discount 7 as well as the interplay between gradient structures of the NTK and the MDP dynamics.
x to be positive definite is, for any 7 > 0.
We note here, that this not primarily limitation of applying NTK theory TD learning, but is reflected in practical experience: TD learning can, especially in offline settings, indeed be instable and diverge. Instability of this form thus inherent to the learning algorithm rather than an artifact of our theoretical treatment. Informally, one approach towards guarantecing positive definiteness of 0 70 2 by enforcing diagonal dominance, appealing to the Gershgorin circle theorem [a.yl. every real eigenvalue À must lie in
(Gerschgorin 1931). h. For matrix A
0₂ R, S S a; + R.
(26)
where R² = Eite (aigl is the sum of off- dif-diagonal elements of a row i. In other words, a lower bound on the smallest real eigenvalue can be increased by increasing diagonal entries a, while decreasing off-diagonal elements aij. In the TD learning setting, this translates to gradient conditioning, e.g., by ensuring of, 0) = || / (2'6 = C' for any pair x, x', guaranteeing cross cross-similad similarities to be smaller than self-similarities.. Indeed several recent works pursue similar strategies stabilize offline TD learning (uue et al. 2023) Gallici et al. 2024) and rely on architectural elements like layer normalization (Ba et 2016) to shape gradient norms.
16

---

Published as a conference paper at ICLR 2026
A.1.2 POST TRAINING FUNCTION DISTRIBUTION WITH TEMPORAL DIFFERENCE DYNAMICS
now aim to establish the distribution of post-training functions f(x, (oo) when initial parameters Oo are drawn randomly i.i.d. For the remainder of this section, we will assume the infinite width limit,
s.t. (2n 0x)) fxx 20) and 0 2x Ozzz. The post-training function f(z dea) is given by
f(x, (cc = f(x Bo) Oxx (0 x (f(2) - (7f(X', (0) tr), (27)
and is thus a deterministic function of the initialization 0g-
Theorem 1. Let f(x,8) 9.) be a NN with L hidden layers of width 11....., ,,, n trained with gradient flow to reduce the TD loss [(0s) b (["(", 0z)loy + f(X, 0t) Il.- In the limit of
infinite width n - co and time t - oc, the distribution of predictions f (Xy, 30)) on a set of test points NT converges to a Gaussian with mean and covariance given by
Eeo [f(XT, 0z0)] = -x
Cova, [S(TT, 0zx)] = Kx - 10xx Ax, +h..) (xxx Oxxx))
where Ozx is the NTK, Kizr is the NNGP kernel, h.c. denotes the Hermitian conjugate, and
Ax O xx x and Ax= = x/
Proof. We begin by introducing a column vector of post-training function evaluations on a set of test points Nr, and the training data X' and X'. Moreover, we introduce the shorthand
x = ~
(28)
and similarly Ax = Oxx - x' The vector can then be compactly described in block matrix notation by
f(Xr, f(X ((x f(2', 000)
T -xxxA -1
7xxx & *xxa x xxXAA
f(Zr. 0o) f(X.0n) f f(X', (o)
@xxx Oxx x
3
=
I -xxAx
+
(29)
I -xxAAx
OxxAx
pm
70
Lee et al. (2018) show that neural networks with random Gaussian initialization 6, (including NTK parametrization) are described by the neural network Gaussian process (NNGP) f(Xr, 0g) N(0 xx, with Xx, [f fo (o) By extension, the initializations fo are jointly Gaussian with zero mean and covariance matrix
XXxX
Cov
ExxX
ExX
Exx
(30)
As the post-training function evaluations 1 given in Eq. (9) are affine transformations of the multivariate Gaussian random variables f (0 K), they themselves are multivariate Gaussian with distribution f ~ (b,, AKA)
We are content with obtaining an expression for the distribution of f(Xr, 0zo) and thus in the following focus on the top-left entry of the block matrix (AKAT))) For notational brevity, we introduce the following shorthand notations
Ag Kxž 7xxxx
(31)
After some rearranging, obtains the following expression for the covariance Cov(f
Cova, [1(T,, 0xx)] =, (0x, A Ax, th.c.) ( (0x, (Nx-7Ax) 0xxx).
17

---

Published as a conference paper at ICLR 2026
A.1.3 DISTRIBUTION OF UVU PREDICTIVE ERRORS
We now aim find an analytical description of the predictive errors as generated by our approach. For this, let u(x, 8e) denote the predictive (online) network and g(æ; th) the fixed target network. We furthermore denote c(z, Dr. 1o) u(x, 0.) g(z, tho) the prediction error between online and target network.
Corollary 1. Under the conditions of Theorem let u(æ, 02)) be a converged online predictor trained with synthetic rewards generated by the fixed target network g(æ, 1b) with r. = g(X, cb)) gg((2' v0). Furthermore denote the variance of converged universal Q-functions Ve, [I, 0x Assume u, 9. and Q are architecturally equal and parameters are drawn i..dd og 9o, 40 ~ NO, 1). The expected squared prediction error coincides with Q-function variance
Edo.vo e2 20) V [(xx, 0zo)],
(11)
where the Lhs. expectation and rh.ss. variance are taken over random initializations do, vo, On-
Proof. Since our algorithm uses semi-gradient TD losses to train u(z, 0z), the linearized dynamics of Theorem ① apply. However, we consider a fixed target network g(æ; vo) to produce synthetic rewards according to
og (22, vho) 79(2'' 1bo)
(32)
With the post training function as described by Eq. 27 the post-training prediction error in query point for this reward is given by
u(x, dx) ) g(x, v/n
u(I, 0o) g(x, vho) Onx Ax [u(X, 0o) - (ru(d"' (0) + g(xg 10) - -70("", vo))). (33)
We again use the shorthand e¹ = (c(Ar, 0c, vo), ((2' 02, vho), 0e, 1)) and reusing the block matrix A from Eq. 29 we can write
Ac
(34)
By assumption, u(x; Uo) and g(, v/) are architecturally equivalent and initialized i..d., and el is simply the of two independent Gaussian vectors with covariance Cov(e") = 2K. We conclude that prediction errors e00 are Gaussian with distribution y00 ~ NN0, ((0,2AAAT)) Taking the diagonal of the covariance matrix AKAII obtain
Es [2,, 1x0, vo) Va. (0(1z 0z));
(35)
where
Ve, [Q(x, - (/ Oxam)
(36)
A.2 MULTIHEADED UVU
We now show results concerning the equivalence of multiheaded UVU prediction errors and finite ensembles of Q-functions. We first outline proofs for two results by Lee al. (2018) and Jacot et al.) (2018), which rely on in our analysis.
A.2.1 NEURAL NETWORK GAUSSIAN PROCESS PROPAGATION AND INDEPENDENCE
Consider a deep neural network I with I layers. Let 2x(2) denote the i-th output of layer I = 1, 1,..., L. defined recursively
2(()== = =
(x), x)(x) = ((2)
(37)
where n; is the width of layer / with no n4m and -ي x. Further, ow and o constant variance multipliers, weights w/ and biases 5' are initialized i...d. ((O, 1). and & is nonlinearity. The i-th function output fa(a)) of the NN is then given by f((x)) (x).
18

---

Published as a conference paper at ICLR 2026
Proposition? 2 (Lee et al. 2018)). At initialization and in the limit nh ...... ,L.,, oc, the i-th output at layer 1, (z). converges to a Gaussian process with zero mean and covariance function K given by
21, - x' + oi: and k: =0, it
(38)
ng
(1)(xxx)) = oه + σ 1 (c)
(39)
(40)
and
M = = (o (2,))x²)
if if j.
(41)
Proof. The proof is done by induction. The induction assumption is that if outputs at layer - 1 satisfy a GP structure
~ m
(42)
with the covariance function defined as
x (2, x') = B[22-1 (z))2-1(())) k3j (z,z), Vi,j. 1- Ki (2, = (x)2 = 0, for j.
(43)
(44)
then, outputs at layer / follow
~ ((t)~~P(()) m(g),
(45)
where the kernel at layer is given by:
= n (x, Vi,j. k (x, = (z) 0, if
= Elz((z) =
(46)
(47)
with the recursive definition M1, x²) = oj + o. - oP (00))=))
(48)
Base case (I = 1). At layer I = we have:
ng eua. vno
waya bb
(49)
j=1
This is affine transform of Gaussian random variables; thus, 2 (a)) is Gaussian distributed with 27 ~ GP()), GP((, mt)),
(50)
with kernel
K/ x') = 2 x Tx' + of and mij =0, i j.
(51)
Induction step 1 > 1. For layers l > 1 we have
T2
< = obbi +
Σ (x), 2f(x)) ((2) (x)).
(52)
By the induction assumption, (x:) are generated by independent Gaussian processes. Hence, x/(2)) and (z) are independent for i j.. j. Consequently, 2x(xr) is a sum of independent random variables. By the Central Limit Theorem (as n1, n + oo) the tuple (z((x), (x')) tends to
be jointly Gaussian, with covariance given by:
[l=((r)=<(<<') of + a B -.00. (2))
(53)
Moreover, as 2/ and 2/ for i + j are defined through independent rows of the parameters w b and independent pre-activations x' (x:), we have
kky (l2(())2 = o. i
(54)
completing the proof.
19

---

Published as a conference paper at ICLR 2026
A.2.2 NEURAL TANGENT KERNEL PROPAGATION AND INDEPENDENCE
We change notation slightly from the previous section to make the parametrization of fi(a, 04) and (z; 0') explicit with
2x(2, = onby + vi-1 Σ (t) df(w)= - (x;
(55)
where 0 denotes the parameters (w b. bl...., w², 1/3 up to layer and fi(0, = of (x; 0h). Let d
furthermore & be a Lipschitz-continuon nonlinearity with derivative x(xx) = dz ((z).
Proposition 3 (Jacot et al. (2018). In the limit na....,AL-I - co, the neural tangent kernel O, x²) of the i-th output z (z, 0') at layer I, defined as the gradient inner product
(1((r==')= = 2 0 2
(56)
is given recursively by
x0)) = n² (2) =
no T + o and 0ly(2,x') 0. i
(57)
x1,) 0 (c. n') " (x,,',
(58)
(59)
where
[a(x,x')= -rot (z)) (x'))
(60)
and
(x))))
ifi=
() 10
(61)
if
Proof. We again proceed by induction. The induction assumption is that if gradients satisfy at layer 1-1
= (o) --
0 22,2²)
if j. if + j,
(62)
0
then at layer we have
01((xxx)== = 1% (z,2") (1,,, (')((,,,))))) x/()),z))
(63)
and
(x,,x'')) = V 2 0 V 2o 0¹) = 0 if
(64)
Base Case (1 1). At layer| I = 1, we have
2 (z) = obb
+ du vno
2 w Ljj
(65)
and the gradient inner product is given by:
V VoL (z. 01) T V Voizt (20,0'000)))) 04) zzzz x' + toī = -k] x').
(66)
Inductive Step (1 >>> 1). For layers >> 1, we split parameters 0f = -00-1 u (wt, b') and split the inner product by
(x, Vo (z, ) Vo = + V (1d Vo
0') Vi V ((
(67)
L.h.s
r.h.s
Note that the r.h.s involves gradients w.r.t. last-layer parameters, i.e. the post-activation outputs of the previous layer, and by the same arguments as in the NNGP derivation of Proposition 2 this is sum of independent post activations s.t. in the limit n + 00
V (wr uy (10, )')TV (nd , (a', 0²) = { [k, xz,
i i= j. i + j.
(68)
0,
20

---

Published as a conference paper at ICLR 2026
For the s.h.s., we first apply chain rule to obtain
ni-1
V (0,-12((166)) =
Σ
w (x,
(x, (x, (x,
(69)
The gradient inner product of outputs i and j thus reduces to
pi- 2/
8')
T 2 6')=
ni-1
Σ
why u
2x 6( (2, (2, 1)00))))(((+ "d',, () (x,
(70)
By the induction assumption 0 (z, x²) 0'- (z, x') and again by the independence of the rows /a and w/ for i + j. the above expression converges in the limit n4-1 + oo to an expectation with
01,(x,,0)) ( (x, + (2,))
j. i i j.
(71)
This completes the induction.
A.2.3 MULTIHEADED UVU: FINITE SAMPLE ANALYSIS
We define multiheaded predictor with M output heads u,( 02) for i 1,..., fixed multiheaded target network 9: ( thu) of equivalent architecture as u with the corresponding prediction error (2, 9,, 1h)) accordingly. Let u, (a, 0,) be trained such that each head runs the same algorithm as outlined in Section (eationnjiddependdee inddependentll.
N....,M. M and
Corollary 2. Under the conditions of Theorem 7 consider online and target networks with M independent heads ui, Yi. i I,.... M, each trained to convergence with errors ei(2,, 9 10). Let 2 ((x,, 9z0, rho)2 1 EM c2 220, v/u)2 2 be the sample mean squared prediction error over M 2M
heads. Moreover, consider M + independent converged Q-functions Qi(;; 000) and denote their (x; (x))) where Q is the sample mean. The two estimators are identically distributed according to scaled Chi-squared distribution
sample variance (2, 2x) 15 I (Qi(z; 00x)
D (x, Doc; 1ho)2 _ oz(an 0oc); oz(2,60)) (2)) MM'""X'"- (M),
og
(12)
with M degrees of freedom and oz (2. Ozo) Voo (x; 0za)) the analytical variance of converged Q-functions given by Theorem 7
Proof. By Collorary. ! the prediction error of a single headed online and target network e(x, U2, v/o) u(x; De) - g(2, v/o) converges in the limit nı .... nT-11 + do and + Gaussian with zero mean and variance e(xr, 9oo, vh) ~ (0, 20%) where
00 Vo, [Q(x, 0x)) Kaz.
(Oxx A2 th. (0 (Ax θxx) (72)
By Propositions and 3 the NNGP and NTK associated with each online head u, (x, foo) in the infinite width and time limit are given by
Ra)(x,x') Ex,,(1, a Dxcuu( 0x)) _
[x(2,22)
if
$10
if + j.
(73)
[0(z, x')
if = j. if + j.
01((xxx')= = Voullz, )20) T TV Vou 0xx)
(74)
0
Due to the independence of the NNGP and NTK for different heads ui, prediction errors Gaussian with variance equal given in Eq. 72
<:(x;;; ox vi) are i.i.d. draws from Note that this is despite the final feature layer being shared between the output functions. The empirical mean squared prediction errors are thus Chi-squared distributed with M degrees of freedom
M
1 (x dor 3 dg x2 (M)
(75)
21

---

Published as a conference paper at ICLR 2026
Now, let (Q((2; |:= M+1 be deep ensemble of M + +114000 Q-functions from independent initializations. By Corollary 1 these Q-functions, too, are i.i.d. draws from a Gaussian, now with mean xx''x''' and variance as given in Eq The sample variance of this ensemble thus also follows a Chi-squared distribution with M degrees of freedom
Σ
2
(Q,(2;0<)) Q12;0<))) aà (M), M
(76)
i=1
where Q( 0x)) = 1
Σ. MM+1
Qi(x; 020) is the sample mean of M + universal Q-functions,
M+1
completing the proof.
A.3 LIMITATIONS AND ASSUMPTIONS
In this section, we detail central theoretical underpinnings and idealizations upon which our theoretical analysis is built.
A central element of our theoretical analysis is the representation of neural network learning dynamics via the Neural Tangent Kernel (NTK), an object in the theoretical limit of infinite network width. The established NTK framework, where the kernel is deterministic despite random initialziation and and constant throughout training, typically applies to fully connected networks with NTK parameterization, optimized using squared error loss (Jacot et al., 2018). Our framework instead accommodates semi- graddearddnnn TD loss, and thereby introduces an additional prerequisite for ensuring the convergence of these dynamics: the positive definiteness of the matrix expression Oxx ~ x- This particular constraint is more a characteristic inherent to the TD learning paradigm itself than direct consequence of the infinite-width abstraction. Indeed, the design of neural network architectures that inherently satisfy such stability conditions for TD learning continues to be an active area of contemporary research (Yue et al. 2023 Gallici et al., 2024).. The modeling choice of semi- cuni-gradient TD losses morcover does not incorporate the use of target networks, where bootstrapped values do not only stop gradients but are generated by separate network altogether that slowly moves towards the online learner. Our analysis moreover considers the setting of offline policy evaluation, that is, do not assume that additional data is acquired during learning and that policies evaluated for value learning remain constant. The assumption of fixed, static dataset diverges from the conditions of online reinforcement learning with control, where the distribution of training data (X, X') typically evolves as the agent interacts with its environment, both due to its collection of novel transitions and due to adjustments to the policy, for example by use of a Bellman optimality operator. Lastly, our theoretical model assumes, primarily for simplicity, that learning occurs under gradient flow with infinitesimally small step sizes and with updates derived from full-batch gradients. Both finite-sized gradient step sizes and stochastic minibatching has been treated in the literature, albeit not in the TD learning setting (Jacot al.) 2018 Lee et al. 2020; Liu et al. 2020 Yang 2019). We believe our analysis could be extended to these settings without major modifications.
B EXPERIMENTAL DETAILS
We provide details on our experimental setup, implementations and additional results. This includes architectural design choices, algorithmic design choices, hyperparameter settings, hyperparameter search procedures, and environment details.
B. IMPLEMENTATION DETAILS
All algorithms are self-implementte and tuned in JAX (Bradbury al. 2018). A detailed exposition of our design choices and parameters follows below.
Environment Setup. We use a variation of the GoToDoor environment of the minigrid suite Chhealler-Boiisel et al. 2023). As our focus is not on partially observable settings, we use fully observable 35 di-einnnsl state descriptions with
35 Observation vectors comprise the
factors:
(0 (Ongent-poa> Oagent-dir Odoor-conigg ddoor-pos)
T
T
T
(77)
22

---

Published as a conference paper at ICLR 2026
where Ougent-pos IR2 is the agent position in æ; y-coordinates, Ougent-dir IR is scalar integer indicating the agent direction (takes on values between 1 and 4), Odvar-config E IR244 is the door configuration, comprising 4 one-hot encoded vectors indicating each door's s color, and Oducr-pos E R vector containing the z. y-positions of the four doors. The action space is discrete and four- dimensional with the following effects
turn left
if O,
turn right
if 1,
Ocffect
go forward
if a = 2,
(78)
open door
if 3.
Tasks are one-hot encodings of the target door color, that is R and in the online setting are generated such that they are achievable. The reward function is an indicator function of the correct door being opened, which case a reward of is given to the agent and the agent position reset to random location in the grid. Episodes terminate only upon reaching the maximum number of timesteps (50 in our experiments).
In the task rejection setting described in our evaluation protocol, agent in start state is presented a list of tasks, which may or may not be attainable, and allowed to reject a fixed number of tasks from this list. In our experiments, the agent is allowed to reject 4 out of 6 total tasks the beginning of each episode.
Data Collection. Our offline datasets are recorded replay buffers from DQN-agent deployed to the GoToDoor environment with an e-greedy exploration strategy and particular policy: When the door indicated by the task encoding provided by the environment lies at the south or west wall, the regular policy by the online DQN agent is executed. If the target door lies at the north or east wall, however, actions are generated by a fixed random Q-network. This mixture policy emulates a policy that exhibits expert performance on certain combinations of tasks and states, but suboptimal behavior for other combinations. The replay buffer does, however, contain most combinations of states and tasks, albeit some with trajectories from suboptimal policies. Hyperparameter details of the online agent are provided in section B.2
Algorithmic Details. All tested algorithms and experiments are based on DQN agents(Mnij Mnihh al. 2015) which we adapted for the task- conditioned universal value function (Schaul al. 2015) setting. While our theoretical anal- ysis considers full-batch gradient descent, in practice we sample minibatches from offline datasets with Xmb = {(*,, a, = Xy = ((<), d/e } generated af argmax Q(5/ a, zz, 0z) and
A
Norm
FC 512
x FC ReLU
ReLU
No where next-state actions are
A
FC 512
rewards are r = (ri Non. Jin Moreover, we devi- ate from theoretical analysis and use target networks in place of the stop- dop-gradieet operation. Here, a separate set of parameters 0g is used to generate bootstrap targets in the TD loss which is in practice given by
Norm
FC
FC512
A
ReLU
ReLU
((0,) 11l Q(X θ.) + Q(xmm, 0,) 112- (79)
512
FC 512
The parameters O, are updated towards the on- line parameters O, at fixed intervals through polyak updating, as is common. We use this basic algorithmic pipeline for all tested algo- rithms, including the online agent used for data collection.
Figure 5: Illustration of the used architecture. ? indicates elementwise multiplication.
23

---

Published as a conference paper at ICLR 2026
Architectural Details. We use a hypernet-
work MLP architecture adapted to the DQN setting, as depicted in 5 Specifically, this means we pass states and task encodings through single-layer encoders, which then joint by elementwise multiplication. The resulting vector is normalized by its 12 norm, x' = T This joint vector is passed thorugh 3-layer MLP with network width 512, again normalized by l2 norm and finally passed through fully-connected layer to obtain vector of dimension RIÁI_ Although
z
experi- ments are conducted in the offline RI setting, preliminary experiments showed no benefits of using ensemble-based pessimism (An al. [021] conservative Q-updates (Kumar et al. 2020). Instead, our normalization pipeline appears to sufficiently address overestimation issues as is suggested by
several recent works Yue al. 2023 Gallici et al. 2024).
Independent Bootstrapping. For the ensemble-based BDQNP baseline and our UVU model, we perform independent bootstrapping in the TD loss computation. By this, we mean that both the bootstrapped value and actions are generated by individual Q-functions. In the case of BDQNP, this means we compute Loss 79 for each model Qk, indexed by h E [1...., with Xmb,, = Xm and bootstraps generated
X M {(5 du 2,) ) and
de = argmax .E.A (8,, a, 0t)
(80)
Note, that this procedure is established (Osband et al. 2016 and serves the purpose of maintaining independence between the models the ensemble. In order to conduct the same procedure in our UVU method, where have access only one Q-function, we generate K distinct Qesstimates by computing
Q (s, == Q(s, 0, 2,),) + ex a, vo),
(81)
that is, by adding the UVU error of the k-th output head. Bootstraps are then generated according to Eq. 80
Intrinsic reward priors. Intrinsic reward priors are trick suggested by Zanger et al. (2024) to address a shortcoming of propagation methods used for intrinsic reward methods like RND(Burda et al. 2019 O' O'Donoghue et al. 2018). The issue is that while learning Q-function with intrinsic rewards can, with the right choice of intrinsic reward, provide optimistic estimates of the value function, but only for state-action regions covered in the data. A potential underestiation of the optimistic bound, however, counteracts its intention, a phenomenon also described by Rashid et al. (2020). Intrinsic reward priors a heuristic method address this issue by adding local, myopic uncertainty estimates automatically to the forward pass the intrinsic Q- Q-function, leading to a "prior" mechanism that ensures
Qintr (s. a, 2,), 02) = (s, a, 2, 02) + (rna a, 2, Grnd)²
where (rna(s, a 2, Ornd) denotes local RND error as an example. The altered function Qintr (8, 0, 0c) is trained as usual with Loss 79 and intrinsic rewards 2 (ndd (8, a, 2, 0rnd)²
B.2 HYPERPARAMETER SETTINGS
To ensure a consistent basis for comparison across our findings, all experimental work was carried out using a shared codebase. We adopted standardized modeling approaches, including uniform choices for elements like network architectures and optimization algorithms, where appropriate. Specifically, every experiment employed the same architecture detailed Appendix B. Key hyperparameters. encompassing both foundational and algorithm-specific settings, were tuned through a grid search on the 10 10 variation of the GoToDoor environment. The search grid and final hyperparamters are provided in Tables 2 and 3 respectiiyy DQN in Table to the online data collection agent.
B.3 ADDITIONAL EXPERIMENTAL RESULTS
We report additional results of the illustrative experiment shown Section 3 Fig. shows different uncertainty estimates in the described chain environment. The first row depicts myopic uncertainty estimates or. equivalently, RND errors. The second and third row show propagated local uncertainties with and without the intrinsic reward prior mechanism respectively. This result shows clearly the
24

---

Published as a conference paper at ICLR 2026
Table 2: Hyperparameter search space
Hyperparameter
Values
Learning rate (all)
1.1066,3-00 10-6 10-00
3 10-5, 1.
Prior function scale (BDQNP)
1011,00,111333311100,0
RND Learning rate (RND, RND-P)
[1.10 10-11-1-11111111110000
3 10-5 1-111-133-100111-00000
UVU Learning rate (UVU)
111006:3-110-01-111-
3.10-5.1.100 1-100-1-10-33110----------
Table 3: Hyperparameter settings for GoToDoor experiments.
Hyperparameter
DQN
BDQNP
DQN-RND
DQN-RND+P
UVU
Adam Q-Learning rate
3 10-4
3
10-4
10
3-10-
3 10
Prior function scale
n/a
1.0
n/a
n/a
n/a
N-Heads
1/512
1 1/512
1/512
Ensemble size
n/a
3/
n/a
n/a
n/a
MLP hidden layers
3
MLP layer width
512
Discount y
0.9
Batch size
512
Adam epsilon
0.005/batch size
Initialization
He uniform (He et al. 2015)
Double DQN
Yes (Hasselt 2010
Update frequency
Target lambda
1.0
Target frequency
256
Table GoToDoor Environment Settings
Parameter
Value
State space dim
35
Action space dim
3
Task space dim
N Task Rejections
4
Max. Episode Length
50
25

---

Published as a conference paper at ICLR 2026
NO 2a1 wo 1 mede
0 20
O-ND So mod4s
0500070800
10
20
Figure 6: Top Row: RND errors. 2nd Row: Value uncertainty as measured by intrinsic Q-function. 3rd Row: Value uncertainty as measured by an intrinsic f-function with intrinsic reward priors.
shortcoming of the standard training pipeline for intrinsic rewards: in standard training pipeline, the novelty bonus of RND is given only for transitions (s;, as, s already present in the dataset and is never evaluated for OOD-actions. To generate reliable uncertainty estimates, RND requires, in addition to the RND network and the additional intrinsic Q-function, algorithmic mechanism such as the intrinsic reward priors or even more sophisticated methods described by |aahhdd et al.) (2020).
USE OF LARGE LANGUAGE MODELS
Large language models (LLMs) were used to assist in the preparation of this paper. Their usage was limited to refining sentence structure and verifying grammar, punctuation, and general language usage. No content or substantive research contributions were generated by L.MSs.
26


Reviews:
[{"review_id": "DmBV0hloRf", "reviewer": "Reviewer_wiBU", "summary": "The paper proposes a novel and computationally efficient single-network method (called \"UVU\") for estimating epistemic uncertainty in reinforcement learning. This approach measures uncertainty as the prediction error between an \"online\" network and a fixed, randomly initialized \"target\" network. The online network is trained using temporal difference learning on a synthetic reward signal generated by the target network itself, allowing the error to capture long-term value uncertainty. The paper provides a strong theoretical proof showing that, under infinite-width network assumptions, this single-model error is mathematically equivalent to the variance of a large, computationally expensive ensemble.", "questions": "1. Could the authors provide results demonstrating that the method improves sample efficiency in online RL algorithms or facilitates safer exploration?\n\n2. Is the method inherently limited to offline data and multi-task settings—where the number of policies is constrained and policy conditioning is easier to learn—compared to online RL, where the policy continually evolves as the agent learns the task?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Improving the computational efficiency of epistemic uncertainty quantification is a key objective for RL algorithms.\n\n2. The paper provides a formal proof, grounded in Neural Tangent Kernel (NTK) theory, that its single-network uncertainty estimate is mathematically equivalent to the variance of an infinite ensemble, moving it beyond a simple heuristic methods like Random Network Distillation (RND).\n\n3. The policy-conditioned nature of the value function enables UVU to capture long-term, plan-dependent uncertainty. By making the policy representation an explicit input, the model is forced to learn a \"universal\" value function that understands how outcomes change under different plans. This design is crucial because it allows the temporal difference (TD) learning process to properly propagate uncertainty over time. If a specific policy leads to novel or data-sparse states far in the future, the TD updates naturally \"back up\" that high uncertainty to the agent's current state, providing a holistic, long-term risk assessment for that specific plan, rather than just a myopic \"novelty\" score of the current state.", "weaknesses": "**Lack of empirical results** \n\nA potential weakness is the paper's limited evaluation of the intrinsic quality of the uncertainty estimates. The experiments focus on a binary task-rejection benchmark, which demonstrates the utility of the uncertainty but doesn't thoroughly analyze its calibration, its correlation with true value error across different states, or how it behaves in simpler, more controlled environments.\n\n1. The paper compares its approach to Random Network Distillation (RND) and ensemble-based methods for epistemic uncertainty estimation, but it does not provide results on standard online RL benchmarks, where these methods have traditionally been evaluated. Including such results is crucial to demonstrate the practical utility of the learned uncertainty for efficient or safe exploration.\n\n2. Because the method estimates policy-conditioned value uncertainty, its applicability in single-task settings—where the policy continuously evolves as the agent learns about the environment—remains unclear. This raises questions about the method’s feasibility in such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wcoxtQ36GK", "reviewer": "Reviewer_DHG9", "summary": "This paper presents an approach for general uncertainty estimation of a value function, inspired by random network distillation. The authors present a theoretical motivation for their approach and showcase the effectiveness in an empirical analysis with a gridworld RL task.\n\nThis is a well-crafted paper, presenting a convincing and rigorous theoretical analysis. While strictly speaking, the novelty is limited, and the empirical analysis is restricted to a specialized setting, I still think it is a valuable contribution.", "questions": "- You have only demonstrated the performance of UVU on an offline RL task. Do you expect UVU to behave differently in an online RL setting? During online training, the state distribution shifts, leading to a shifting critic as well (the critic network might be subject to catastrophic forgetting). Can this be accurately captured by UVU?\n- During reading, i was strugling a bit with the introduction of the multi-head variant of UVU (maybe just due the clarity of writing). Could you strenghten the reasoning behind this analysis? In which case would a single UVU network no be sufficient as an estimator?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper is crafted very well, with high-quality writing, figures, and appendices\n+ The topic of uncertainty estimation for RL (including for critics/value functions) remains relevant, with no widely accepted approaches. Existing techniques like deep ensembles or MC Dropout are available, but are not widespread (e.g., due to the additional computational demands or potential training instability), and MC Dropout in particular has been shown to provide mixed results\n+ The motivation and preliminary sections are comprehensive and seem to cover all important aspects\n+ The theoretical analysis utilizing NTK to show equality of uncertainty estimation to ensembles is convincing, and provides a strong justification of the approach\n\nNeutral: The paper draws on existing work (RND and NTK). Compared to RND, it provides a theoretically grounded analysis and extension; compared to work on NTK, it is a novel application domain.", "weaknesses": "- The empirical analysis is somewhat limited (although it is an illustrative use case). I would love to see some additional experiments beyond this scenario. As the method is generalizable, I feel it should be possible to apply it to common benchmark environments (similar to the original RND paper)\n- The lack of any implementation limits impact, and also makes it difficult to fully assess reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "idSKGs6nSk", "reviewer": "Reviewer_4xcK", "summary": "This paper introduces universal value-fucntion uncertainties (UVU), quantifying uncertainty as squared prediction errors between an online learner and a fixed, randomly initialized target network. The paper also provides a theoretical analysis of the proposed approach using neural tangent kernel (NTK) theory, showing that  UVU errors are equivalent to the variance of an ensemble of independent universal value functions in the limit of infinite network width.", "questions": "My major concern is about the coverage rate of the true values. The authors may consider a simulated environment for which the true value functions are exactly known and demonstrate that the confidence intervals constructed by the proposed method can achieve the nomial coverage rate for the true value. Such an evaluation criterion is critically important for real decisions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper aims to address an important issue in RL, and the theoretical analysis is strong.", "weaknesses": "The central concern with the paper is the numerical study. Suppose that the true values are known for the simulated environment, the experiments should include coverage rates (and related calibration metrics). Reporting only the best policy’s returns with confidence intervals does not fully characterize estimator performance or uncertainty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tall26eLcE", "reviewer": "Reviewer_1jtb", "summary": "This paper proposes Universal Value-Function Uncertainties (UVU), a single network method for estimating policy-conditioned value uncertainty. The key idea is to train an online predictor via semi-gradient TD learning on synthetic rewards generated by a fixed random target network, and interpret the prediction error as an uncertainty measure. Using neural tangent kernel (NTK) theory, the authors derive post-training distributions of TD trained functions and show that, under the infinite-width and gradient-flow limit, the expected UVU prediction error is equivalent to the ensemble variance of universal $Q$ function. A multi-head version further yields a finite-sample equivalence to a scaled $\\chi^2$ distribution.", "questions": "1. Is there any observed correlation between the smallest eigenvalue of $\\Theta_{XX}-\\gamma\\Theta_{X'X}$ and TD loss divergence?\n2. How does the implementation gap such as inter-head correlations affect the uncertainty calibration?\n3. Do you have plans to extend UVU to continuous control (e.g., D4RL) or stochastic gridworld environments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Conceptual simplicity**: UVU replaces RND’s myopic novelty signal with policy-conditioned, future-aware value uncertainty. The concept of modeling TD error between learner and fixed random teacher provides an intuitive mechanism for epistemic uncertainty estimation.\n- **Theoretical grounding**: The paper rigorously analyzes TD learning dynamics under NTK, presenting explicit mean-covariance forms for post-training predictions and proving the equivalence between UVU’s prediction error and ensemble variance.\n- **Empirical effectiveness**: The proposed multi-head network achieves ensemble-level uncertainty quality while reducing computational cost by roughly an order of magnitude.", "weaknesses": "1. **Idealized stability condition**: Theoretical TD learning convergence condition (i.e., $\\Theta_{XX}-\\gamma\\Theta_{X'X}\\succ0$) depends on the transition structure and the choice of $\\gamma$, which is difficult to verify or enforce in practice. Although the appendix suggests using Gershgorin’s theorem and gradient-norm conditioning to ensure diagonal dominance, it remains unclear whether this condition holds empirically or how violation manifests as instability (e.g., divergence of TD loss).\n2. **Theory–implementation gap**: The theory assumes gradient flow, full-batch optimization, and stop-gradient TD, whereas the implementation employs mini-batches, target networks, and finite training steps. These deviations could introduce bias or affect convergence, yet no quantitative analysis is provided. In particular, the multi-head version uses a shared trunk, violating the independence assumption in Corollary 2 and potentially leading to systematic underestimation of uncertainty.\n3. **Limited empirical generality**: Experiments are limited to a single offline MiniGrid setting. Evaluation on continuous control, stochastic, or partially observable environments would strengthen generalizability and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)