Adaptive Exploration for Data-Efficient General Value Function Evaluations
Authors: Arushi Jain
Josiah P. Hanna
Doina Precup
Venue: NeurIPS 2024 poster (2024)
Keywords: general value functions
GVFs
multiple policy evaluations
exploration for GVFs
variance-minimization
reinforcement learning
Primary area: reinforcement_learning
TL;DR: Designing exploration strategy for multiple General Value Function (GVF) evaluations

Abstract: General Value Functions (GVFs) (Sutton et al., 2011) represent predictive knowledge in reinforcement learning. Each GVF computes the expected return for a given policy, based on a unique reward. Existing methods relying on fixed behavior policies or pre-collected data often face data efficiency issues when learning multiple GVFs in parallel using off-policy methods. To address this, we introduce *GVFExplorer*, which adaptively learns a single behavior policy that efficiently collects data for evaluating multiple GVFs in parallel. Our method optimizes the behavior policy by minimizing the total variance in return across GVFs, thereby reducing the required environmental interactions. We use an existing temporal-difference-style variance estimator to approximate the return variance. We prove that each behavior policy update decreases the overall mean squared error in GVF predictions. We empirically show our method's performance in tabular and nonlinear function approximation settings, including Mujoco environments, with stationary and non-stationary reward signals, optimizing data usage and reducing prediction errors across multiple GVFs.

Adaptive Exploration for Data-Efficient General Value Function Evaluations
Arushi Jain arushi.jan jain @ mai..megill.ca
Josiah P. Hanna
Doina Precup dprecup @coupp@ss.mmm eercuppps.mmcgil.caa McGill University Mila
jphanna @aannna@sss.. welecs...scc.m
McGill University Mila
The University of Wisconsin Madison
Abstract
General Value Functions (GVFs) (Sutton et al., 2011) represent predictive knowl- edge in reinforcement learning. Each GVF computes the expected return for a given policy, based on unique reward. Existing methods relying on fixed behav- ior policies or pre-collected data often face data efficiency issues when learning multiple GVFs in parallel using off-policy methods. To address this, we introduce GVFExplorer, which adaptively learns a single behavior policy that efficiently collects data for evaluating multiple GVFs in parallel. Our method optimizes the behavior policy by minimizing the total variance in return across GVFs, thereby reducing the required environmental interactions. We use existing temporal- difference-style variance estimator to approximate the return variance. We prove that each behavior policy update decreases the overall mean squared error in GVF predictions. We empirically show our method's performance in tabular and nonlinear function approximation settings, including Mujoco environments, with stationary and non-stationary reward signals, optimizing data usage and reducing prediction errors across multiple GVFs.
Introduction
The ability to make multiple predictions is a key attribute human, animal, and artificial intelligence. Sutton al. (201I) introduced General Value Functions (GVFs) which consists of several indepen- dent sub-agents, each responsible for answering specific predictive knowledge about the environment. Each GVF consists of unique policy, custom reward function called cumulant and state-dependent discount factor to calculate the cumulative discounted cumulant. For example, GVF can predict the expected number of times an agent will bump into the wall under a given policy (White al.) 2015) Schlegel et al., 2021 Sherstan 2020). In essence, GVFs generalizes the standard value function to address a wider range predictive questions, making them powerful tool for intelligent systems.
Prior works have used either a fixed random behavior policy (Sutton et af., 2011 or pre-collected datasets (Xu et al. 2022) update all GVFs in parallel using off-policy learning. However, these methods can result in large value estimation errors if the behavior policy significantly diverges from the GVF policies. Our work addresses this gap by focusing on the question of exploration for evaluating GVFs: how can we adapt an agent's 's behavior policy for data-efficient estimation of mulriple GVFs in parallel? While exploration has been extensively studied in the context of optimal control in Markov Decision Processes (MDP), the question of constructing a policy that can learn multiple value functions parallel has remained largely unexplored.
To accurately evaluate multiple GVFs in parallel, we aim to design a behavior policy that minimizes the overall mean squared error (MSE) of their predictions. A natural approach might involve following each GVF's target policy for some period of time (e.g. one episode) in a round-robin manner while concurrently updating all GVFs off-policy. However, this approach can be highly
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

data inefficient as actions are sampled according to given policies, potentially overlooking actions from states where the expected return is uncertain. To achieve better value estimation with fewer samples, it is essential to focus on state-action pairs with high variance in return, as these pairs would exhibit greater uncertainty their mean return. Therefore, a behavior policy should visit such pairs more frequently to offset higher variance in return. Consider an analogy of a two-arm bandit problem: fewer samples are needed to accurately evaluate a constant reward arm, whereas an arm with variable reward demands greater number of samples to achieve the same level of certainty, We empirically support this claim by comparing round-robin and our approach later in this paper.
With this motivation, we introduce CVFExplorer, that adaptively learns a behavior policy which minimize the total MSE across all GVF predictions. GVFExplore leverages the existing off- off-policy temporal difference (TD) based estimator of variance in return distribution (Sherstan et al. 20188) Jain et al. 202T) to guide the behavior policy. The strategy is to frequently take actions that might have more unpredictable outcomes (high variance in return). By sampling them more, agent can estimate the mean return better with fewer interactions, thus effectively lowering the overall MSE in the GVF predictions.
GVFExplorer optimizes the data usage and reduces the prediction error, offering a scalable solution for complex environments. This is particularly valuable for real-worle applications like personalized recommender systems (Parapar & Radlinski] [202T] Tang et al., 2015). where can enable efficient
evaluation of personalized policies based on diverse user preferences (reward functions)
(Li et al.
2024), leveraging shared knowledge for improved accuracy.
Contributions: (1) We design an adaptive behavior policy that enables accurate and efficient
learning of multiple GVF predictions in parallel [Algorithm D (2) We derive an iterative behavior
update rule that directly minimizes the overall prediction error [Theorem [.1. (3) We prove in the
tabular setting that each iterative update to the behavior policy causes the total MSE across GVFs
to be less than equal to one from the old policy [Theorem 4.2]. (4) We establish the existence of a variance operator that enables us to use TD-based variance estimation [Lemma [.J]. (5) We empirically demonstrate in both tabular and Mujoco environments that GVFExplorer lowers the total MSE when estimating multiple GVFs compared to bascline approaches and enables evaluating
a larger number of GVFs in parallel.
2 Related Work
Exploration in reinforcement learning (RL) has predominantly focused on improving policy per-
formance for a single objective (Oudeyer al., 2007; Schmidhuber 2010 Jaderberg et al. 2016 Machado et al., 2017 Eysenbach et al., 20188) Burda et al.] 2018) Guo et al. 2022). Refer to Ladosz ct al. (2022) for detailed survey on exploration techniques in RL. While related to exploration, these works differ from ours, as they concentrate on optimizing policies for single objective rather than evaluating multiple GVFs (policy-cumulant pair) simultancously.
Our work is most closely related to other works on learning multiple GVFs. Xu et al. (2022) address similar problem by evaluating multiple GVFs using offline dataset, but method operates online, avoiding the data coverage limitations of offline approaches. Linke et al. (2020) develops exploration strategies for GVFs in stateless bandit context, which does not deal with the off-policy learning or function approximation challenges present in the full Markov Decision Process (MDP) context. In single bandit problem, Antos et al. 22008; Carpentier et al. (015), show that the optimal data collection strategy to estimate mean rewards of arms is to sample proportional to each arm's 's variance in reward. Prior works like Hanna et al. (2017) learned behavior policy for single policy evaluation problem using REINFORCE-Style (Williams) 1992) variance-based method called BPS. This idea extends on the similar principles of using Importance Sampling in Monte Carlo simulations for finding optimal sampling policy based on variance minimization (Owen) 2013 Frank et al. 2008). Metelli et al. (2023) extends this idea the control setting. However, these methods are limited to single-task evaluation or control. Evaluating multiple policies simultaneously is more complex, requiring careful balance in action selection among interrelated learning problems. Perhaps the closest work to ours is by Mcleed et al. (2021), ). which uses the changes in the weights of Successor Representation (SR) (Dayan 1993) as an intrinsic reward to learn behavior policy that supports multiple predictive tasks. GVFExplorer approach is simpler, as it directly optimizes the behavior

---

policy to minimize the total prediction error over GVFs, resulting in an intuitive variance-proportion proportional sampling algorithm. We will compare the two approaches empirically as well.
3
Preliminaries
Consider an agent interacting with the environment to obtain estimates of N different General Value Function (GVF) (Sutton et al., 2011). We assume episodic, discounted Markov decision process (MDP) where S is the set of states, A is the action set, P: : S A + Ag is the transition probability function, Aş is the diimessoill probability simplex, and y [0, 1) is the discount factor.
Each GVF is conditioned on fixed policy я; S - A, i = (1,...,, N} and has cumulant S A + R. For simplicity, we assume that all cumulants are scalar, and that the GVFs share the environment discount factor y. This eases the exposition, but our results can be extended to general multidimensional cumulants and state dependent discount factor. Each GVF is value function Vs, (+) = Ex.p st = s] where Gj = Ci,t + G Each GVF can be viewed as answering the question, "what is the expected discounted sum of received while following TT?" We can also define action-value GVFs: Qz, (s, = c; cs a) (o0)++))) B.BarP((lsa [V,, (s')) with V*, (s)
(- 1Q,, (s, a)}-
At each time step t, the agent in state st, takes an action at and receives cumulant values c for all i E [L...., N.,, N. transitioning to a new state S1+1- This repeats until reaching a terminal state or a maximum step count. Then the agent resets to a new initial state and starts again. The agent interacts with environment using a behavior policy, μ : -->> AA. The goal is approximate values Vi corresponding to the true GVFs value V=.- We formalize the objective as minimizing the Mean Squared Error (MSE) under some state weighting d(s) for all GVFs:
V) -E d(s)) (Vz, vi(s))
aES
In our experiments, we use the uniform distribution for d(s).. This objective can generalized to prioritize certain GVFs using weighted MSE.
Importance Sampling (IS). To estimate multiple GVFs with distinct target policies m, in parallel,
off-policy learning is essential. Importance sampling (IS) is one of the primary tools for off-policg
value learning (Hesterberg 1988 Precup 2000; Rubinstein & Kroese 2016), allowing estimation of
value function under target policy π using samples from different behavior policy p. In the context
of off-policy Temporal Difference (TD) learning (Sutton & Barto 2018), the IS ratio, Pr p(a,)s.)* used adjust the updates to ensure unbiased value estimates. The update rule is given as (Se. a4) = ((s..a)) + (c + +PPpt+10 17P++1Q(E+11:) 0t+1) - Q(s;,
=(o.[s.)
where a is the learning rate.
This update rule ensures that estimated value function Q converges to correct value Qr under policy π. despite the samples being generated from a behavior policy
4 Behavior Policy Optimization
As described in the previous section, the goal of the agent is to minimize the total mean squared error (MSE) across the given GVFs (Eq, (1).. Note that MSE Variance + Bias². For the algorithm's s derivation, we will use unbiased IS estimation for off-policy correction, which shifts the task of minimizing MSE to reducing the total variance across GVFs. Thus, the core problem is to design behavior policy that collects data to minimize the variance in return across all GVFs in order to accurately estimate multiple GVF value functions.
The problem of estimating single target policy's 's value, V,, is well studied in the literature (see Sec. [ In Monte Carlo sampling literature, it is well known that there exists an optimal sampling distribution (i.e., behavior policy) that provides optimal variance reduction compared to simply running the target policy Kahn & Marshall (1953); Owen (2013). Unfortunately, the analytical solution of obtaining this optimal behavior policy, n², requires foreknowledge of V., making impractical when our overall purpose is to estimate Vz. Nonetheless, in this work, we take inspiration from this earlier work and develop a practical method that iteratively solves for a single behavior policy that minimizes the total variance when estimating multiple general value functions in parallel.
3

---

4.1 Objective Function
We propose GVFExplorer to address the above limitation and extend the problem to accurately estimate multiple GVF values. GVFExplorer takes as input the GVF target policies (1..).... collects data from a single (non-stationary) behavior policy μ4, and outputs the GVF estimates Vi= Since our objective is to find a behavior policy that minimizes the variance return across multiple GVFs, we use existing off-policy TD-style variance estimator (Sherstan et al. 2018). This estimator allows us to bootstrap the target values and iteratively update the variance function, making the solution scalable complex domains.
We define the variance function by M (s), which measures the variance in the return of target policy π in given state when actions are sampled under a different behavior policy We describe how to learn this function Sec. []] The variance function for given state and a given state-action pair are defined respectively as:
M (s) V (G₂ (t = 8,a p) and (s, Var= = s, = a, a' ~ p).
Our objective is to find an optimal behavior policy μ- that efficiintly collects single stream of experience to minimize the sum of variances (L....r under some state distribution d(s), as,
N
1e arg min
Σ Σ d(s) (8) s.t. mlals) 2 && Im(als)= =
a
We solve the above objective function iteratively. At each iteration k, GVFExplorer produces a behavior policy jgg. The behavior policy interacts with the environment and gathers data. Using this data, any off-policy TD algorithm can be used to iteratively estimate the variance function M This variance function plugged into the optimization problem given in Eq. (3) to update a policy Mk+1 that reduces variance. The iterative procedure is analogous to policy iteration, which alternates policy evaluation with policy improvement, Here, we alternate between the variance evaluation and the improvement of behavior policy to minimize the overall sum of variance across all given GVFs.
Here, aim is iteratively improve behavior policy and decrease variance functions estimate the GVF values V= (1.2.......) with reducing MSE:
E Po
M 1,-122...
L Mn E, M
E
JKK
where E, denotes variance estimation and 4 denotes behavior policy improvement. Next, we present Theorem 4.1 which principally derives the behavior policy update from P4 to μi++1 by solving the objective in Eq. ( We demonstrate that the behavior policy update in Eq. (3) minimizes the objective by showing that M+++1 is a better policy than P4. The policy fo++1 is considered as good as, or better
than μıa if it obtains lesser or equal total variance across all GVFs: MR 1(8) < Σ, M (s).
The proof of behavior policy improvement is detailed in Theorem 4.2
4.2 Theoretical Solution
Theorem 4.1. (Behavior Policy Update:) To find the behavior policy μ that minimize the variance
objective across N given target policies {T, ]=s) we iteratively update μ by solving the objective in
Eq. (2). Given state-action variance function M (s, a). the solution to Eq. (2) at iteration k is:
Σ F. (als) (8,a)
(a+1(a)s)=
(3)
E.. E T. (a S (s,a')
Proof. The proof is presented in App.
Theorem describes how to iteratively update the behavior policy μ4 that minimizes objective in Eq. 2 by using the return variance MP. The policy #4+1 selects actions proportional to their variance, meaning high-variance return (s, a) pairs are explored frequently. By visiting high-variance return pairs, policy gains informative samples and reduce the overall uncertainty. Consequently, this process improves the GVF value predictions and decrease the number of interactions needed for effective learning.

---

Next Theorem 4.2 ensures that behavior policy p&+1 either decreases or maintains the total variance across all GVFs relative to Pt, ensuring consistent progress towards minimizing the variance without oscillation. In simple terms, each policy update ensures the variance does not increase.
Theorem 4.2. (Behavior Policy Improvement:) The behavior policy update in Eq.3) ensures that the aggregated variances across all target policies { either decreases or remains unchanged at each iteration k. This non-increasing variance property demonstrates that each successive behavior policy p4+1 1 is improvement over pa in terms of reducing the total variance. Formally we have,
M (6) S Kc
M (0), Vk, V8 E.
Proof. The proof is in App. A.1
5
Variance Function
The theorems provided in the previous section rely the variance function Mus. Here, we study this variance function in detail.
What is the Variance Function M? In an off-policy context [Sherstan al. 2018). Jain al. (202T)), introduced the variance function M which estimates the variance in return under a target policy using data from a different behavior policy μ.. We will directly use this function M as our variance estimator and present it here for completeness. The function M for state-action pair x(a.le)) is defined as: under n, with importance sampling correction factor Pt ((a.)a.)>
MD (s. a)) Varasp (Gt, s,a, = Eawy [0t + r Pi+ M (s ac+1) =,aa
Here, Ge.r is the return time t. and 6, F1 + yE ~ 12 1c) (r. is the TD error. We use Expected Sarsa (Sutton & Barto 2018) to compute o, eliminating the need for IS by using the expected value of the next state-action pair under for bootstrapping, thus stabilizing the update and lowering the variance. (s,s,, a) relates the variance under и from the current state- action pair to the next, where actions sampled under μ. This allows effective bootstrapping and iterative update using a TD-style method. The state variance function is defined as M (8) Ea p(als)pp(()))))) (s, MH(,
Note, the true Qn is required to compute the TD error d, in Eq. 4 Following Sherstan et al. 2018 we substitute the value estimate Q for the true function Qr to compute & in Eq. (4). Additionally, we use variance estimates M to update the next step policy p4++1 instead of true variance in Eq. (3).
This approach is similar generalized policy iteration (Sutton & Barto 2018). which alternatively updates the value estimator and then improves the policy.
Next, we prove the existence of MG in Lemma 5.1 which not covered in Jain et al. (2021) This
proof establishes a loose upper bound on the IS ratio p, limiting the divergence of the behavior policy from the target policy for effective off-policy variance estimation. This aligns with methods like TRPO (Schulman et al. 2015) and Retrace (Munos et al. 2016), which stabilize policy updates by controlling divergence.
Lemma 5.1. (Variance Function M Existence:) Given a discount factor 0 < S 1, the variance function M exists, if the below condition satisfies, Eanp [p²(s, a)] < + for all states.
Proof. Proof in App. A.2
Note, the optimal μ for the objective in Eq. ②2. might violate the above constraint on pt we empirically clip p to mitigate this problem. Additionally, IS requires μ{als) = 0 when r(als) 0. We empirically ensure p(als) > E << for all actions. The same constraint added for all the basclines for fair comparison.
5

---

Algorithm
We present GVFExplorer algorithm, detailed in Algorithm [ for value function and M.. for variance, each with N heads (one head for each GVF). Starting with randomly initialized behavior policy, the agent observes cumulants for N GVFs each step and updates Qo using off-policc TD. We use Expected Sarsa (Sutton & Barto 2018) for both Q and M, eliminating off- off-policy corrections. The target Q updates follow:
Our approach uses two networks: Qe
Qtar ae, St-1) cz + Σ (00
(5)
We use the TD error from Q-learning, 8G = Qtar Qs. to update target M,
Mtar (8t, ats St+1) 6 Σ T(a||ss41)
(6)
Both networks are updated via an MSE loss. The behavior policy is iteratively updated using the new variance estimates for K steps, with learned Q values used for MSE metrics Eq. ①
To ensure reliable estimates, we initialize M values to small non-zero constants and apply epsilon exploration, which decays that agents visit broad range of state-action pairs early on, preventing issues of zero variance for unvisited pairs. We applied epsilon-exploration both GVFExplorer and the baselines for fair comparison.
time, ensuring coverage of the state-action space. This guarantees
We also techniques like experience replay Lin (1992) for data reuse and target networks for both Q and M to improve learning stability. Expected Sarsa is used consistently across all baselines for fair comparison. Refer to Algorithm [ for further details.
Algorithm 1: GVFExplorer: Efficient Behavior Policy Iteration for Multiple GVFs Evaluations
Input: Target policies (I..)....)) initial behavior policy /41, replay buffer D, primary networks Qo, Ma (small non-zero M), target networks Qa. Ma. learning rates aQ. aMM mini-batch size b, trajectory length T, target update frequency = 100, value/variance update frequencies p = 4, m 8, training steps K, exploration rates En, Edecay Ein
for environment step k = 1....KK K do
Set exploration rate: &s = max
edwaa)
Select action at (=
if random() Ek
Uniform Policy otherwise
Observe next state s++1 and cumulants c Vecton(size(n)) Store transition (st, ae, St-1. in D 1% = then
//Update the Value Qo network
Sample mini-batch of size of transition (8t, de, 8141, D.
Update Qo using MSE loss (Qtar ax) (o a2)),, where Qtar is Eq, 55)
end 10
if 1% = 0 then 11
12
//Update the Variance Mi network Sample mini-batch of size b of transition (st, 0t 8141, D
13
Update M,, using MSE loss (Mta(Ss, ax) - (s a2)), where Maar is Eq. ( end 15
14
if 1:% = then
17
T6 and 0 //U //Update both target networks weights
18
end
//Update the behavior policy y using the new Variance M..
V [aaass V.ES,aEA.
Behavior policy becomes: (a+1(
ENEA >E n.. M
end
Returns GVFs Values Vi(s) E. (T(als)) a) for [1....,m)

---

7 Experiments
We investigate the empirical utility of our proposed algorithm in both discrete and continuous state environments. Our experiments are designed to answer the following questions: (a) How does GVFExplorer compare with the different baselines (explained below) in terms of convergence speed and estimation quality? (b) Can CVFExplorer handle a large number of GVFs evaluations? (c) Can GVFExplorer work with non-stationary GVFs which change with time? (d) Can GVFExplorer work with non-linear function approximations and complex Mujoco environments?
Baselines. We use Off-policy Expected Sarsa updates for parallel GVF estimations for all the experiments (including baselines) for fair comparison. We benchmark against several different baselines: (1) RoundRobin: uses a round-robin strategy sampling episodically from all target policies (2) MixturePolicyy Aggregated policy sampling from all target policies; (3) SR: Successor Representation (SR) method using intrinsic reward of total change in SR and reward weights to learn behavior policy (McLeod et al. (02T). (4) BPS: behavior policy search method originally designed for single policy evaluation using a REINFORCE variance estimator (Hanna et al. 2017); we adapted it by averaging variance across multiple GVFs (similar to our objective). BPS results are limited to tabular settings due to scalability issues with it. (5) UniformPolcc Ply: a uniform sampling policy over the action space. Implementation details and hyperparameters are in App. B
Type of Cumulants. We experiment with three different types of cumulants, similar to McLeod al. (202T) constant with a fixed value; distractor, a stafionary signal with fixed mean and constant variance (normal distribution): drifter, non-stationary cumulant with walk with low variance (vary with time). Further description of cumulants is in App. B.2
random
Experimental Settings. To answer the questions presented above, we consider different settings: (Two Distinct Policies & Identical Cumulants): tabular setting, we examine two GVFs with distinct target policies but identical distractor cumulant, (T1 c). (T2, c). (Two Distinct Policies & Distinet Cumulants): In the same enviromment, we assess two GVFs with distinct target policy and distinct distractor cumulant with different fixed means, (TI, 01), (T2, (Large Scale Evaluation with 40 distinct GVFs): To verify the scalability of proposed method with high number of GVFs, we evaluate combinations of 4 different target policies T1
with 10 different constant cumulants C10. resulting in 40 GVFs. (Non-Stationary Cumulants in FourRooms): FourRooms environment, we assess with two distinct GVFs stationary distractor and non-stationary drifter cumulant - (TT. C1) (T2) (Non-Linear Function Approximation): In a continuous state environment with non-linear function approximator, we evaluate two distinct distractor GVFs, (T1, c1), (T2, 02). (Mujoco environments): In Mujoco environments - walker and cheetah evaluate different GVF tasks like walk, run and flip. Across these varied settings, measure the averaged MSE across multiple GVFs.
7.1 Tabular Experiments
We conducted experiments in 20 20 gridworld with four cardinal actions and tabular 20 x 20 FourRooms environment for added complexity. The discount factor is 7 = 0.99, and the environment stochastic with 0.1 probability of random movement. The cumulants are zero everywhere except for at the goals. Episode terminates after 500 steps or upon reaching the goal. True value function for MSE computation is calculated analytically V, (I yP*) -
cr. Detailed description of target policies and cumulants is provided in App. B.3 Table summarizes the below results for tabular experiments.
In Two Distinct Policies Identical Cumulants, we consider gridworld environment with distractor cumulant at top left corner with a reward drawn from normal distribution. Fig. Ia shows the averaged MSE across the two GVFs, with GVFExplorer showing much lower MSE compared to baselines.
Next, in Two Distinet Policies & Distinct Cumulants, we consider two distinct distractor cumulant (with different mean) GVFs placed at left and top-right corner respectively. Fig. [2] shows GVFEXplorer with reduced MSEs compared to baselines. Figs. 26 and 2c qualitatively analyze the average absolute difference between true and estimated GVF values across states, [
'The code is available on Github: http://gaahbb.ccmmoaaaaiiiaaa/vf

---

VS [l. showing smaller enrors (duller colors) for GVFExplorer. Fig. (in App. B.3.2) presents the individual variance and MSE for both GVFs GVFExplorer. Further, conduct an ablation study to experiment with how GVFExplorer performance changes with poorer feature approximations Fig.TT (I App. (33.3) shows that MSE increases as the feature quality deteriorates, but GVFExplorer remains robust with moderately coarse approximations
For Non-Stationary Cumulant in FourRooms, we evaluate the performance in FourRooms (FR) environment (Sutton et al., 1999) with two distinct GVFs; stationary distractor cumulant and non- stationary drifter cumulant which changes value over time. As shown in Fig. [b] GVFExpl orer re- duces MSE faster than other baselines, even with the non-stationary cumulant. Fig. [I (in App. [B3.4) demonstrates the effectiveness of GVFExplores tracking the non-stationary cumulant signal the later stages of learning.
In Large Scale Evaluation with 40 Distinct GVFs, we evaluate method's scalability to large number of GVFs (refer App. B.3.5). We use constant cumulants with values ranging in [50, 100]. Fig. compares the average MSE across the GVFs, showing that GVFExplores scales well with an increasing number of GVFs. In contrast, the SR baseline struggles with scalability due to the varying cumulant scales affecting the intrinsic reward (the summation of all SRs and reward weights) of behavior policy.
Avcrago MSE
Averaged MSE
10
Averaged MSF
Gara
102
Histuehollyy Uniosmfaky
Samplos
(a) Two Distinet Policies & Iden- tical Cumulants
(b) Non-Stationary Cumulant
(c) Large Scale Evaluation with
in FourRooms
40 Distinct GVFs
Figure 1: MSE Performance: Averaged MSE over 25 runs with standard error in different experi-
mental settings. GVFExplorer demonstrate notably lower MSE compared the baselines.
Raundiictee
InftrmPaaiyy
201
MiturePikyy
BPS
10
Averae MSE
1.00
10-1
0.0
0.5
1.0
Semplee
(a) Averaged MSE
(b) Value Error (RoundRobin)
(c) Value Eroo (GVFExplorer)
Figure 2: Two Distinct Policies Distinct Cumulants: Evaluate averaged MSE over 25 runs with two distinct distractor GVFs (1), (m2, 02) in gridworld Green dots top show two GVF goals. (a) Averaged MSE, (b) averaged absolute error in GVFs value predictions for baseline RoundRobin and (c) GVFExplorer. The color bar uses log scale & vibrant colors indicate higher values.
7.2 Continuous State Environment with Non-Linear Function Approximation
We use continuous grid environment that extends the tabular experiments to continuous state space (similar to McL.ed et al. (202T)) and four discrete actions. For Non-Linear Function Approximation, consider two distinct GVFs with distractor cumulants. An Experience Replay

---

Buffer with capacity of 25K and a batch size of 64 is used for all experiments. Further details on computing true value functions using Monte Carlo and network architectures are in App. B.4
Prioritized Experience Replay (PER). We investigate the integration of PER (Schaul et al. 2013) with algorithm. Unlike the standard Experience Replay Buffer, which uniformly samples experiences, PER assigns priorities based on the TD error magnitude in the Q-network. PER and GVFExplorer are complementary approaches: PER re-weights the collected data in replay buffer based on the priority, while GVFExplorer adjusts the behavior policy to influence data collection.
Combining PER with GVFExplorer drastically lowers MSE compared to other baselines (even when compared to all baselines + PER). We use the absolute sum of TD errors across multiple GVF Q-functions as a priority metric for PER in all baselines, including GVFExplorer. Placing the priority on the TD error of the variance function in GVFExplorer yields less favorable results compared priority
Q-function's 's TT error. In Fig. [3 we present the MSE for both standard experience replay (solid lines) and PER (dotted lines) for all algorithms. PER generally reduces MSE, but its integration with GVFExplores orer shows much lower MSE. This is likely as GVFEXplores could over-sample high variance return samples, causing a skewed buffer distribution. PER's non-uniform sampling maintains balanced data distribution, which helps in stringent MSE reduction. For the SR baseline, using the TD error in SR predictions as priority for PER led to performance degradation, suggesting non-stationarity in SRs' TD errors might mislead PER to prioritize less relevant states under the current policy. The original SR work by McLeod et al. (2021) does not use PER in the experiments. For PER scenario, we qualitatively compare the absolute value error for baseline (asss//ajan
RoundRobin and GVFExplorer by discretizing the state space in Figs. 3b]and [c] and observe that our algorithms results in smaller value prediction error. Further insights into the variance estimation by GVFExplores ore is shown in Figs. [1] and 16 (App. B.4). Table (App. B.4) summarizes the results highlighting the performance of various algorithms.
Shep: 905200
Step: 500000
Oci
Can Fot
1.
10
wasticy
Malaohhd 784
06-
Aug
0.4
02
0.0
0.2
0.4
0.5
0.5
2.0
Num of Samples
(a) Avg. MSE
(b) Value error (RoundRobin)
(c) Value error (GVFExplorer)
Figure 3: Non-Linear Function Approximation: (a) Averaged MSE over 50 runs with standard error using Experience Replay Buffer (solid lines) and PER (dotted lines). GVFExplorer show lower MSE with both buffers. PER generally reduces MSE across all algorithms except SR. Log-scale absolute value error for RoundRobin (b) and GVFExplorer (c); GVFExplorer achieves smaller errors (vibrant colors represent higher values).
7.3 Mujoco Environments with Continuous State-Action Tasks
We use DM-Control (Tassa et al. 2018) based continuous state-action tasks to experiment with Mujoco environments, Walker and Cheetah domain. To expand the proposed method to continuous action environments, any -raiicttrrdienttt (PG) based algorithm can be used. In experiments, we use Soft Actor-Critic (SAC) algorithm (Haarnoja et al. 2018) as a base PG method to incorporate the proposed
objective.
separate network for variance estimation is added to SAC, Further implementation details are provided in App. B.3 To experiment in Walker environment, we use two GVF tasks, namely "walk' and 'flip'. Similarly, for Cheetah environment, we use 'walk' and 'run' GVF tasks. We also added KI. regularize between the learned behavior policy and the given GVFs target policies to prevent divergence. We use MC to compute the true Q-value GVF estimates and compare the MSE between these MC values and the Q-critic network. We use the same Q-critic architecture for the baseline

---

algorithms UniformPolie oiicy and RoundRobin for fair comparison. In Fig. we observe that orrexplorer reduces MSE faster than the baselines.
Outs RoundRoamin
UniformPolicy
Avi MSE
ANgg MIE
Ours
RoundRobin UnifermPolicy
Steps
Steps
(a) Averaged MSE Walker
Averaged MSE in Cheetah
Figure 4: MSE in Mujoco: Averaged MSE over 5 runs with standard error in Mujoco envi- romment with continuous state-actions for (a) Walker and (b)Cheetah domains for GVFExplorer. UniformPolicy and RoundRobin. GVFExplorer consistently lowers averaged MSE compared to the baselines.
8 Conclusion
We addressed the problem of parallel evaluations of multiple GVFs, each conditioned on given target policy and cumulant. We developed method to adaptively learn behavior policy that uses single experience stream to estimate all GVF values parallel. The resulting behavior policy update selects the actions in proportion to the total variance of the return across GVFs. This guides the policy to frequently explore less understood areas (high variance in return), which helps to better estimate the mean return with fewer samples. Therefore, our approach lowers the overall MSE in GVF predictions while reducing the number of interactions required. We theoretically proved that each behavior policy update reduces or maintains the total prediction error. Empirically, we showed that GVFExplorer scales effectively with an increasing number of distinct GVFs, robustly handles non-stationary cumulants in tabular setting, and adapts well to non-linear function approximation. Additionally, we showcased its performance in complex continuous state-action Mujoco environments, showing that CVFExplorer can be seamlessly integrated with existing policy-gradde gradient methods.
Limitations and Future Work. One notable drawback of GVFExplorer is the increased time com- plexity, due to simultaneously learning two networks for value and variance estimation respectively. Additionally, @VFExplorer has not been evaluated in environments with significant difference in the cumulant value range. Such disparities could lead to varying variances, potentially resulting in oversampling areas with higher cumulant values. Calibration across cumulants may be necessary in these cases.
In this work, we focused on minimizing the total MSE, but other loss functions, such as weighted MSE could also be considered. However, weighted MSE requires prior knowledge about the weighting of errors in different GVFs, which is not readily available. A potential future direction could to use variance scales to automatically adjust these weights to provide uniform MSE reduction across all GVFs. Looking ahead, we are interested in testing our approach with multi-dimennional cumulants and general state-dependent discount factors, well as, extending the applicability of GVFExplorer to control settings where the target policies are unknown.
Acknowledgments and Disclosure of Funding
We are grateful to the anonymous reviewers for their valuable feedback. We also extend our thanks to Nishanth Anand, Kshitij Jain, Ayush Jain, Pierre-Lue Bacon, and Subhojyoti Mukherjee for their insightful suggestions. Josiah Hanna acknowledges support from NSF Family Insurance through a research partnership with the University of Wisconsin Sisconsin-MMadioo''' Science Institute, and the Wisconsin Alumni Research Foundation.
American Madison''s Data
10

---

References
András Antos, Varun Grover, and Csaba Szepesvári. Active learning in multi-armed bandits. In Algorithmic Learning Theory: 19th International Conference, ALT 2008, Budapest, Hungary, October 13-16, 2008. Proceedings 19. pp. 287-302. Springer, 2008
Pierre-Luc Bacon. Temporal Representation Learning. McGill University (Canada), 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation, 2018.
Alexandra Carpentier, Remi Munos, and András Antos. Adaptive strategy for stratified monte carlo sampling. Much. Learn. Rex., 16:2231-2271, 2015.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural computation, 5(4):613-624 1993.
Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. arXiv preprint
0rXiv:1205.4833 2012.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint wwXivv/802.0660 2018.
Jordan Frank, Shie Mannor, and Doina Precup. Reinforcement learning in the presence of rare events.
In Proceedings of the 25th international conference on Machine learning, pp. 336-343, 2008
Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernaro Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, al. Byol-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35: 31855-31870, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Josiah P Hanna, Philip S Thomas, Peter Stone, and Scott Niekum. Data-efficient policy evaluation through behavior policy search. In International Conference on Machine Learning, pp. 1394-1403. PMLR, 2017.
Timothy Classen Hesterberg. Advances in importance sampliig. Stanford University, 1988.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.080 10539, 2016.
Arushi Jain, Gandhary Patil, Ayush Jain, Khimya Khetarpal, and Doina Precup. Variance penalized on- policy and off-policy actor-critie. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 7899-7907, 2021.
Herman Kahn and Andy W Marshall. Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of America, 1(5):263-278 1953.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22 2022.
Xinyu Li, Zachary Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. arXiv preprint 43X5v2240.001133, 2024.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8:293-321, 1992.
Cam Linke, Nadia M Ady, Martha White, Thomas Degris, and Adam White. Adapting behavior via intrinsic reward: survey and empirical study. Journal of artificial intelligence research, 69: 1287-1332, 2020.
11

---

Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv: 1089, 2017.
Matthew McLeod, Chunlok Lo, Matthew Schlegel, Andrew Jacobsen, Raksha Kumaraswamy, Martha White, and Adam White. Continual auxiliary task learning. Advances in Neural Information Processing Systems, 14:12549-12562 2021.
Alberto Maria Metelli, Samuele Meta, and Marcello Restelli, On the relation between policy improvement and off-policy minimun-variance policy evaluation. In Uncertainty in Artificial Intelligence, pp- 1423-1433. PMIR, 2023.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Pierre Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286, 2007.
Art Owen. Monte carlo theory, methods and examples, 2013.
Javier Parapar and Filip Radlinski. Diverse user preference elicitation with multi-armed bandits. In Proceedings of the 14th ACM international conference on web search and data mining, pp. 130-138, 2021.
Doina Precup. Eligibility traces for off- oif--poicy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Reuven Y Rubinstein and Dirk P Kroese. Simulation and the Monte Carlo method. John Wiley & Sons, 2016.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint M.X.::1511..095555 2015.
Matthew Schlegel, Andrew Jacobsen, Zaheer Abbas, Andrew Patterson, Adam White, and Martha White. General value function networks. Journal of Artificial Intelligence Research, 70:497-543, 2021.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions on autonomous mental development, 2(3):230-24) 2010.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
Craig Sherstan. Representation and general value functions. 2020.
Craig Sherstan, Dylan R Ashley, Brendan Bennett, Kenny Young, Adam White. Martha White, and Richard Sutton. Comparing direct and indirect cemporal-differene methods for estimating the variance of the return. In UAI, pp. 63-72, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(1-2)181--111 1999.
Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Volume 2, 2011.
12

---

Liang Tang, Yexi Jiang, Lei Li, Chunqiu Zeng. and Tao Li. Personalized recommendation via parameter contextual bandits. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 323-332,
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint PrXiiv1801.006900 2018.
David S Watkins. Fundamentals of matrix computations. John Wiley & Sons, 2004.
Adam White et al. Developing predictive approach knowledge. 2015.
Ronald J Williams. Simple statistical pradient-followin algorithms for connectionist reinforcement learning, Machine learning, 8:229-255 1992.
Tengyu Xu, Zhuoran Yang, Zhaoran Wang, and Yingbin Liang, A unifying framework of off-policy general value function evaluation. Advances in Neural Information Processing Systems, 35: 13570-133 13583, 2022.
13

---

A Proofs
A.1 Behavior Policy Update Theorems
Theorem 4.1. (Behavior Policy Update:) To find the behavior policy μ that minimize the variance objective across N given target policies ( iteratively update is by solving the objective in
Eq. 2
Given state-action variance function M (s, a), the solution to Eq. 2 at iteration k is:
V ( M (8,0)
M (a++)a)s)=
Ea. Ea (a ? (s,
Proof. We formulate Eq. (2 as a Lagrangian equation below to solve for the optimal behavior policy μ1"
2(,,)s,as Ws) ΣΣ
d(s)
d(s)
Σ )aau(ss + wo(1 (1 Eu(s,a)),
(7)
part
11 part
part
Here, À E IR and w E RISI satisfy:
denotes the Lagrangian multipliers. The following KKT conditions
1. C(s.a)
2. (s,af((s,a) =
3. Aa,a 20
4. p(s, a) 20
5. Ea ((s,a)))
Gradient of The gradient p(s, a) w.r.t u(als),
V vlls,a)(ss,a) T(als) T(ass) p(als)2 p(als)
_ a)
Vu(als)
Solving part. We will compute the gradient of Mg, (8) in Eq. w.rt to given pls, a). Here,
p(s, a) =(als) is IS weight. We expand MH (s) relation with M (s) to derive the gradient, p(als)
M (5) Σ (5
(ulsaa-M Mu (5) v((sa)
Σ (5) (5.
= pr(s, (
Vpi(s,a)
MH (s.a) + (s, MM (s,a)
ai(s,a)
IV part
m(als)
=1(8, MM (M (s, (0, 2 (s,a)
=-p((8, M M (s,a).
The final term V MM, (s,a) is difficult to estimate in an iterative off-policy setting. Hence, we drop the (IV part) from the above gradient, which is similar to Degris et al. (2012)[eec 2.2], where the gradient of Q(s, a) was dropped while deriving the policy update.
Solving for the Lagrangian Eq. further by substituting the part), and taking derivation of II & III part and using the (1) KKT condition.
V (((a,) C(u, de.a w.) = -- d(s)))(s, (s, a) + >o.a W. =0
(8)
14

---

From (2)KKT condition, we know that either As,a = 0 or μı(als) = 0. Following the arguments of IS, support for p(a]s) can only be 0 when the support for target policy ({ass) Solving for the case when support for target policy in non-zero, then let As,a = 0. We can simplify the gradient of Lagrangian in Eq. (8)
Σ d(s) (a(a)s
We Σ d(s))
M (s.a)
u(als)²
(9)
E. me(als))MM..ss.. - /d))(s))
u(als)
We know that the numerator always positive (variance M is positive), therefore w, < 0. Let Ys - /d(). From condition (5), know that Σ. p(als) 1. Using Eq. (9 and summing over all the actions we get,
Σ, (als)) (s.
Σ Eu(als) = -
y
Hence, Vya= Σ
Therefore, the update for optimal behavior policy becomes.
(= (s,
u(als)*
E. m. (s,a)
As the optimal policy μ² appear on both the sides, this can be interpreted as iterative update, where denotes the iterate number.
(a (8,0)
(a+1(a)s) Ea (( (8,0)
Theorem 4.2. (Behavior Policy Improvement:) The behavior policy update in Eq. ensures that the aggregated variances across all target policies ) either decreases remains unchanged at each iteration k. This non-increasing variance property demonstrates that each successive behavior policy Mk+1 is improvement over Pk in terms reducing the total variance. Formally we have,
Σ M (8) S
M me (s), Vks Vs E.
suggests, for any given μi behavior policy, the next successive approximation Mk+1 minimizes the objective function Eq. (2. i.c..
Proof. Theorem 4.
pk+1 min Σ ( (*)
min Σ d(s) Σ u(als) (a(ss)
(10)
M (s. u(als)2
M (8)
will omit writing d(s) and assume that s d(s). Further, we will use the notation pi(s, a) = (a(s))) for ease writing. From Eq. (10). can establish the relation,
Pa (s|s)
Σ Me (als) sa)s))
Mu+t mut1 (als) Tz(als)² (a41(a)s)3
Mi (s, Σ
M (8,
Na (a|s)²
(11)
$,8,a
MC. (s)
15

---

Now, we will use Eq. (II) relation to further simplify the equation and establish that variance decreases with every update step k. We will use the notation Pettn =oP+t to denote the products.
Σ
(s)
Σ 1,3,a
pa+ (als)) a) M (s,
Σ 3,9,a
Mk+1 (a+1( a) ( 188 + 72 M (st+1))8: =
Bawputt
(pt) (pt2² M ($t+1) 186
1
expand
Bawwet
(0))200 + (p,) Eanmurt 1(p(+1) + (pt+ (S(22)))55 81
1,8
Σ BaraR+1 [o)))00 + 7 (1(4))) + (01) 2 (5:42) =
1
Ba-mun [Pa 2 (+he42)) + +.../s=-
M (s). mz
1
(12)
O
A.2 When does Variance Function Exists?
Let Cy. R SxAl denote the pseudo-reward a) E. and P E RisxAxSxA] represent the transition probability matrix Pa(s,a,s',d) ((ssls..c)pldd)s)) d'). The matrix form of ME is:
MH c₁ + PP
M - cc.
(13)
The existence of M hinges the invertibility of matrix (I - 2 ).. Lemma 5.1 establishes the existence the above inverse using Definition A.1 A. and Lemmas (A.2 and A.3
Definition A.1. (Spectral Radius) The spectral radius of a matrix A Rax= is denoted by sr(A) maxx(X1, A2,... "2,,.,,,,...... ..), where A; denotes the ith eigenvalue of A.
Lemma A.2. The spectral radius sr( A) of matrix A E Raxn follows the relation, sr () < A|] where, ||| max; E3 A (i. is the infinity norm over matrix.
Proof. Following the derivation from Bacon (018) Ph.D. thesis and work of Watkins (2004),
use the eigenvalue of a matrix to show that sr (A) < || We can write Xx Ax. when is the
|.
eigenvalue of A. For any sub- mo--muutpplicaaive matrix norm, |B]| S [|] Using this property,
Ax|| = ||||||| = ||x|| < |||] ||x|||
|I| < ||]]].
The above is true for any cigenvalue À of A. So this must also be true for the maximum eigenvalue of A. Therefore, can express.
sr(A) < ||||]-
Lemma A.3. When the spectral radius of sr(A) < 1, then (I (-A)- A) exits and is equal to, A)-1 - A
Proof. The proof for the Lemma is presented in Puterman 2014//ropoosition A.3].
16

---

Lemma 5.1. (Variance Function M Existence:) Given a discount factor 0 < < S 1, the variance function M exists, the below condition satisfies, Eazp [p²(s, < f for all states. 77
(s, a)
Proof. Following Lemma A.3 the existence of M hinges on the existence of inverse p
Further, (I - - P²) exists if spectral radius pr()"Pp) p)) that for any given matrix A, spectral radius satisfies, sr( (A) I |||]] Hence, using the above two lemmas, we can express,
1. Further, from Lemma A.2 we know
sr( s 1 pll S |
Further, if spectral radius satisfies the below condition, then the inverse exists,
Pr()"P,,) S > IP < 1. We expand the middle infinity norm term and get
Pu Pu
max
8/.a!
EP(s'ss Eu(d'ls'p' < 1 72
max
We further express the above condition as Eanyy [p(s,a]]] < g2, S.
B Experiments
This section provides detail about the experiments in the main paper as well as additional experiments in the Appendix. All the experiments require less than 1GB of memory and have used combined compute less than total CPU months and GPU month.
For all the experiments, we consider the two target policies with four cardinal directions left (L), right (R), up (U) and down (D) for the tabular and non-linear function approximation environments. These policies are specified follows for every state S:
क (8) (L. : 0.175, R 0. U 0.25, D 0.4)
(14)
(8))) [L: 0.25, R 0.15, U 0.25, D 0.35).
B.1 Baselines
(I) RoundRobin: We used round robin strategy to sample data from all given n target policies episodically. We used Expected Sarsa to estimate all GVF value functions in parallel when a transition given (st, ae, St41, Ci (I....)).
Qi(s o,) Q, a.) + ((())))))) + [ Q (st(
(2) MixturePolicy, UniformPolicy are also evaluated using Expected Sarsa. MixturePolicy is defined as,
T. (als)
MMixture (als) = E. N (a(ls)
(3) SR: Based on (McLeod et al. Representation (SR) and reward weights obtain the intrinsic reward for behavior policy updates. We use Expected Sarsa to learn both the SR and the Q-value function from the intrinsic reward. The behavior policy is generated using a Boltzmann policy over the learned function, as it empirically performs better than a greedy policy. We apply simple TD Expected Sarsa updates instead of Emphatic TD(A) shown in Algo 2 in McLeed et al. (202T). The learning rates for SR, reward weights, and behavior policy Q function are kept the same.
2021 SR uses the summation of weight changes in the Successor
(4)BPS: (Hanna et al.
2017) Use a Reinforce style estimator to learn IS(T, σ) G(T) (a.(a)) mentioned in the onginal paper. Since, the original work is only about single policy evaluation.
(()))))
17

---

we extended for multiple GVFs by updating behavior policy summation over Σ, IS(T, The behavior policy weight is updated as:
T Vo log u4 (
P 8 +
IS(T, T
B.2 Types of Cumulants
We consider three different types of cumulants similar Mcl.edd et (22TT) as shown in Fig. 5
Constant: Fixed value cumulant, ci =
Comulase Valee
Distractor: Stationary cumulant with reward drawn from nor- mal distribution with fixed mean and variance, = N (N,,, 04)
Drifter: Non-Stationary cumulant whose value change over time, of =c + ((s, 0,), 100.
B.3 Tabular Experiments
Figure 5: Visual represen- tation of cumulants.
We consider tabular 20 x grid environment with stochastic dy- namics. We use the above target policies for the different experimental settings. The Table [ summarizes the averaged MSE with the same 2 x 10t samples for different experimental settings.
Table Avg. MSE Summary in Tabular Env: Compares the average MSE across multiple GVFs in different experimental settings in Tabular environment. We compare baselines with WFExxlorer at same 10 steps of learning. We show the improvement in GVFExplorer w.r.t. to best baseline RoundRobin. Note; Smaller MSE indicates better performance.
Avg MSE
GVFExplorer (Ours)
Improvement of Ours (against best baseline)
BPS
SR
UniformPol
RoundRobin
MixPol
2c6 steps
Distinet policies
26.13
2.76
1.22
1.15
1.15
0.24
79%
same cumulant
Distinet policies
8.2
4.1
0.54
0.47
0.44
0.04
91%
distinct cumulants
Non-Stationary
18.57
1.78
0.9
1.28
1.08
0.46
48%
in FR env (4M steps)
Large num of GVFs
53.3
2.66
2.35
2.64
1.66
29%
Table 2: Optimized Hyperparameters: We show the optimized hyperparameters for different Experimental Settings. aq learning rate for value function. 0.M learning rate for variance function.
distinct policies distinct policies
large scale
non-linear fune approx
non-stationary cumulant in FR
Exp. Settings
identical cumulants distinct cumulants
40 GVF eval
GVFExplorer
(aq = 0.25,
(aeQ 0.1.
(aq = 0.5.
(aq = 5e 3.
(aQ = 0.5.
(Ours)
0MM 0.8)
0M 0.8)
0M 0.95)
0M 5e -3
QM 0.8)
RoundRobin
0.95
0.8
0.8
5e 4
0.8
MixturePolicy
0.95
0.8
0.8
5e
0.8
UniformPolicy
0.95
0.8
0.8
0.8
SR
0.25
0.5
0.25
Ie -
0.8
BPS
0.5
0.8
0.8
Hyperparameter Tuning:
our experiments, we use linearly de-
caying learning rates that starts with initial value of 1.0 and gradually decreased to an optimized minimum value within 500K steps of environmental interactions. We used different learning rates for value and variance function in GVFExpl orer. The minimum learning rate parameter was swept within 10.1, 0.25, 0.5, 0.8, 0.9, 0.95) for both value and variance function. The optimal minimum learning rate determined based the one achieving the lowest average Mean Squared Error
18

---

(a) RoundRobin
(b) SR
(c) BPS
(d) GVFExplorer
Figure 6: Impact of Learning Rate Averaged MSE in Two Distinct Policies & Identical Cumulants scenario: Demonstrate the effect of changing minimum value of learning rate on the averaged MSE (performance averaged over 10 runs) across GVFs. The optimal hyperparameter is selected based on the least MSE in these plots. LR_Q: value learning rate, LR_M: variance learning rate.
(MSE) after 800K sample interactions. This hyperparameter tuning approach was consistently applied for algorithms including baselines. Fig. O shows the sensitivity analysis of varying learning rates for value functions (all baselines) and variance functions (our method) with the averaged MSE performance in Two Distinet Policies & Identical Cumulants. The learning rate resulting in the lowest MSE was selected optimal. In Table 2 four experimental settings discussed in the paper earlier (refer Experimental Settings Sec. 7
we show the optimal hyperparameters for the
B.3.1 Two Distinet Policies & Identical Cumulants
In tabular 20 x 20 grid, we consider distinct target policies with identical distractor cumulant r N(u = 100, 5). Fig. 7 we depict the individual MSE over runs for both the GVFs c), (TT2, e); showing decreased MSE for GVFExplorer compared to the baselines.
Cura MCE3
MCES
101
Inchiiddd
10"
Nur Sumpess
Figure 7: Two Distinet Policies & Identical Cumulants in 20x20 Grid: Averaged MSE over 25 runs for two GVFs (Fi,c) c), (T2,c) with same cumulant. We show individual MSE,, MSE2 GVFExplorer shows lower MSE compared to other basclines.
B.3.2 Two Distinct Policies & Distinct Cumulants
In tabular 20 x 20 grid, we consider distinct target policies and distinct distractor cumulants N(u = 100, a = 5) placed at top-left corner and 02 = (u = 50, a = 5) placed in top-right corner. Fig. 8 shows the individual MSE for all algorithms and the estimated variance in GVFExplorer for both the GVFs.
Semi-greedy ı for Two Distinct Policies & Distinct Cumulants: We evaluated semi comi-greedy target policies with distinct cumulants, (m), (1) and (m2, 02) within 20x20 grid. The target policies are designed with a bias towards top-left and top-right goals respectively.
(s) (T. 0.4, R 0.1, U 0.4, D 0.1)Y' m2 (8) = (T. 0.1, 0.4,U 0.4, D 0.1)
(15)
We keep the same cumulants same as in previous experiment. Fig. 9 compares the average MSE performance, where GVFExplorer exhibits comparable MSE to RoundRobin baseline but requires
19

---

10
Recrolloon
Indiiide M915
10"4
0.0
0.5
1.0
1.5
2.0
Numn Sampless
(a) MSE for individual GVFs
(b) Var (VV
(c) Var GVF=(GVFExploree)
Figure 8: Two Distinet Policies & Distinct Cumulants in Grid env: Evaluate two distinct GVFs (TI, (1) and (T2, 62) averaged over 25 runs. Compared baselines - RoundRobin, MixturePolicy, UniformPolicy SR, BPS with GVFExplorer. Green dots show GVF goals. (a) Individual MSE, for GVF (TI, c1), and MSE₂ for GVF (T2, c2). (b,c) Estimated variance M MM in GVFExplorer. Variance plots show log-scale empirical values; most areas appear black, due to their relatively small magnitude compared to high variance regions. The color bar uses log scale & vibrant colors indicate higher values.
more samples to converge. This outcome can be attributed to the near-greedy nature of the target policies, which inherently guides RoundRobin along goal directed trajectories, enabling it to achieve nearly accurate predictions with fewer samples. The optimal hyperparameters for RoundRobin, UniformPolicy and MixturePolicy is aq = 0.95. We used aq = 0.5, a. = 0.8 for ours GvFExplorer. Another baseline SR has aq 0.8 and BPS as aq 0.9 optimal hyperparameters.
Suuddlee MCC
107
30
2012
Ap4
30
(a) Averaged MSE
(b) MSE1, MSE
Figure 9: Semi-greedy target policies in Two Distinet Policies & Distinet Cumulants: Analysis of MSE averaged over 25 runs with semi-greedy target policies. (a) Averaged MSE (b) MSE, MSE, We observe a slower convergence of GVFExplorer as compared to baselines like RoundRobin, MixturePolicy due to target policies being semi-greedy.
B.3.3 Ablation Study on Effect in Performance on using Poor Feature Approximator
In this section, we study the effect of using degraded approximations or feature quality on the performance metries. We conducted an ablation study in a 20x20 grid with two distinct distractor GVFs with cumulants, 50, 5) on the top-right corner. We reduced the state space into 10x20 and 5x20 feature grids (grouping factors of 2 and 4, respectively). and compared results against the original setup (no approximation, factor 1). As shown in Fig. [0 the MSE increases as the feature quality deteriorates. Despite this, GVFExplorer outperforms RoundRobin and MixturePolicy though with very poor approximations (factor= = 4), the UniformPol: cy performs better due to inaccurate variance estimates. These results demonstrate that GVFExpl orer is robust with moderately coarse approximations but can degrade with significantly poor feature representations, as expected. Further, these results are strengthened by the performance of GVFExplorer in Mujoco environments Sec. 7.3
N(p 100, a 5) placed on the top-left corner and C2 = N N(u =
20

---

(Groupingfacter-1]
(Grougingfacter-22)
(Gmupingfactor-d
(Groupinghcttr--)
(Groupingfactor-2)
(Groupingfactor-())
MisturePlicy (Groupingfattr-1]
MistrePalicy (Grupingfactor-2)
Mixtreeaiiyy (Groupingfectorr4))
UniformPoliy (Groupingfacter-11
UniforPoliiy (Groupingfacter-22)
UniformPolicy (Grupingfactor--)
6
5
4
Avg MSE
3
2
1
0
0.5
1.0
1.5
2.0
Steps
1e7
Figure 10: Impact Feature Approximation on MSE: Averaged MSE over 10 runs with standard error in tabular environment. Increasing Grouping indicates coarser feature mapping (more states mapped to the same feature). As expected, overall MSE increases with coarser mapping. GVFExplorer outperforms baselines given reasonable feature approximator.
B.3.4 Non-Stationary Cumulant in FourRooms
complex FourRooms environment, we consider two distinct target policies in Eq. (14) and different cumulants stationary distractor with N(u 100, = 2) in top-left room, non-stationary drifter signal of a 0.5 top-right room. Figs. lla and TTb shows the change in estimated variance M of GVFExplorer from early learning steps to later steps (vibrant color shows higher numerical value). We experimented with changing the value of drifter cumulant to see the effect on MSE. In Fig. Tce we observe that MSE increases with increasing the value of driftness (a) over time.
(a) Avg M in carly (earn-(b) Avg M in later learn- ing stage
(c) Avg MSE with differ- ent of drifter cumulant
ing state
Figure 11: Non-Stationary Cumulant in FourRooms: placed a stationary distractor cumulant in the tep-left room and non-stationary drifter cumulant in the top-right room. (a & b) show the change in estimated variance M over time, highlighting the effectiveness of GVFExplorer in tracking the non-stationary cumulant placed in top-right corner, later learning process over stationary cumulant (top-left). (c) shows the average MSE for GVFExplores with different levels of driftness (a) in the cumulant value; higher driftness leads to higher MSE.
21

---

B.3.5 Large Scale Evaluation with 40 Distinct GVFs
We evaluate GVFExplores ability to handle large number of GVFs. We examine four target policies (Tne1...4), each aligned with a cardinal direction, and ten cumulants (Cme1...au). aiming to predict 40 GVF combinations (v ... v ). Each GVF is associated with state space region ("goal"), uniformly sampled and assigned constant cumulant value ranging [50, 100), demonstrated in Fig. 12 In this setting, we considered target policies in the four cardinal directions, namely:
TN (s) = 0 R: U.1,U 0.7, D 0.1 E S
(E()) [T.: R: 0.7,U 0.1, D: 0.1) E
(s))) = 11:011,::0010:000000000
Tw (s) (4.07,R:0010:000000000333
Figure 12: 10 different cumulants for Large Scale Evaluation with 40 Distinct GVFs in 20 x 20 grid. The color depict the cumulant empirical value.
B.3.6 Ablation: IS ratios Expected Sarsa Update
In FourRooms environment with distractor and drifter cumulants (two distinct GVFs), compare the following off-poll oef-policy update styles: (1) Off- -oflcyl TD updates using IS correction,
Q(tr: ar) = Q(tt, ae) + aqle + 7Pp+ 1 (11+ ar+1) ((s,, a:))
=og
M(s, .x) (st, a.) + ass (62 7 pi M (8841,
M ((t,aa) a:)
and (2) Expected Sarsa update in Eqs. 5) and (6) In Fig. we observe that Expected Sarsa leads lower MSE, hence we use Expected Sarsa for all the TD updates in all the algorithms including baseline for further update stability.
13
107
Figure 13: IS vs Expected Sarsa update FR env: We show the averaged MSE (over runs) in Fourrooms by doing off-policy IS corrections in TD updates and off-policy Expected Sarsa in GVFExplorer algorithm. Expected Sarsa leads smaller MSE and faster convergence.
B.4 Continuous State Environment with Non-linear Function Approximation
Continuous Environment. We extend the tabular GridWorld environment to continuous state
space, similar to the approach by McLeod et al. (202T). The environment is square of dimension
1 x with four discrete actions. We evaluate two GVFs: the first GVF has a cumulant at the top-left
corner, ei AN 100, = 5), and the second at the top-right corner, 62 ~Nu = 50, a = 5).
The target policies consistent with those used in the tabular environment. The agent receives
22

---

zero cumulant signal elsewhere and moves 0.025 units in the selected direction with added uniform noise 2(.0.011 0.01]. Episodes start randomly, excluding a 0.05 radius from the goal state, and end after 500 steps upon reaching 0.05 radius from the goal.
Fig. 14 presents the individual MSE for both GVFs under standard Experience Replay and PER. Our method, GVFExplorer, consistently achieves lower MSE compared to baselines. Fig. [5] shows the absolute GVF value prediction error with PER for both the baseline RoundRobin and GVFExplorer. Fig. T6 illustrates the estimated variance from each GVF, underscoring the necessity for a sampling strategy that prioritizes high-variance return areas to reduce data interactions and ultimately reducing the variance and MSE. Fig. 17 depicts the trajectories sampled from baseline RoundRobin and CVFExplorer. Table summarizes the performance of various algorithms in this continuous environment.
Computation of True GVF Values. The true GVF values in continuous environment com- puted using a Monte Carlo (MC) method. The continuous state space discretized into a grid, with initial state sampled from each grid cell. We calculate the average discounted return over 200, 000 trajectories following policy n. with cumulant c. The mean squared error (MSE) between the estimated and true GVF values is then computed using these discretized states, expressed as E;
2
Σ.
Vee
Vin (+))
for all algorithms.
Network Architecture. We use distinct deep networks for learning value Q and variance M. Both networks share a similar architecture, with a shared feature extractor for input states and separate output heads for each GVF, producing multidimensional outputs for both value and variance. The variance network includes a Softplus layer before each head's output to ensure positive numerical values.
Table Avg. MSE Summary for Continuous Env. Averaged MSE across two GVFs for different algorithms in the continuous environment. GVFExplorer orer performance measured against others using standard and prioritized experience replay after 1 x 10t learning steps. Note: Smaller MSE indicates better performance.
Avg MSE
GVPExplorer (Ours)
% Improvement of GVFExplorer (agninst best haseline)
SR
MixturePolicy RoundRobin
@1e6 steps
Standard
21.7
18.25
16.78
5.19
69%
Replay Buffer
Prioritized
112
14.7
11.62
3.87
66%
Exp. Replay
Dera
Out 301
107
51
11.99
JuP-
0.0
1.0
0.0
1.0
Num Sumples
Num Samples
(a) MSE Left Goal
(b) MSE Right Goal
Figure 14: Individual MSE in Continuous Env.: Compare the MSE metries in baselines RoundRobin, MixturePolicy, SR and CVFExplorer (averaged over 50 runs with standard er- rors) for both standard Experience Replay Buffer (solid lines) and with Priority Experience Replay (PER) (dotted lines). CVFExplores demonstrates lower MSE with both types of replay buffers. PER generally reduces MSE across all algorithms, except for SR.
B.5 Mujoco Environment with Continuous State-Action Tasks
We conducted additional experiments using the DM-Control suite in the Mujoco environment, focusing the Walker and Cheetah domains. For the Walker domain, we defined two distinct
23

---

(a) Avg Vi emoorRoundRobin)
(b) 2 BoundRoon))
(c) V:3 enror(ounndRobin
8300
(d) Avg Vi; enor(GVFExplorer) (e) V. enor(GVFExplorer)
(1) Vag enoo((VFExplorer)
Figure 15: Value Prediction Errors in Continuous Env: Compares log-scale absolute errors between actual and predicted values for two GVFs. Top row: RoundRobin baseline errors; Bottom row: GVFExplorer results at equivalent steps. (Col 1): Mean error, (Col 2): Error in GVF (Col 3): Error in GVF 2. GVFExplorer specially achieves smaller errors in areas where RoundRobin has higher MSE, due to the focus on reducing overall MSE (indicated by lighter colors).
(a) E. (1.2) [M*(s)
(b) M (s) (s)
(c) (s)
Figure 16: Estimated Variance in Continuous Env:
The two GVF goals are depicted in Green. We show the estimated variance M (log values) over states from GVFExplorer method highlighting the motivation for behavior policy to visit high variance areas. (a) Mean variance, (b) Variance for left goal GVF, (c) variance for right goal GVF. These variance plots show log scale empirical values; most areas appear black due to their relatively small magnitude compared to high variance regions.
Taa
Teabdy
(a) RoundRobin Ti
(b) RooundRobin 72
(c) GVFExplorer Ti
(d) GVFExplorer
Figure 17: Sampled trajectories in Continuous Env: CVFExplorer generates trajectories which reduces the overall variance, thus minimizing the total MSE. Contrary, RoundRobin collects data according to given target policies. Green dots show GVF goals and red depicts the start state.
GVFs: walk and flip. In Cheetah domain, we evaluated two GVFs: walk and run. To handle the continuous action space in these environments, we leveraged policy gradient methods, which are essential when working with continuous actions, as value-based methods like Q-learning are not directly applicable.
24

---

Our method can be incorporated with any policy gradient (PG) algorithm. For these experiments, we used Soft Actor-Critic (Harnoaa et al. 2018) which provides stability in such settings. SAC uses entropy regularizer to encourage exploration, where the regularization coefficient learned adaptively. This allows the agent balance exploration and exploitation effectively. We present the SAC-CVFExplore in Algorithm 2 modified version of SAC designed to efficiently handle parallel estimation multiple GVFs, each aligned with a specific target policy and cumulant.
The first modification involves incorporating separate variance network, Mw, (s, a), which estimates the variance of returns and supports the behavior policy's objective to reduce mean squared error (MSE) in GVF estimation. This variance network, parameterized by weights and W2, updates alongside the Q-value critics capture return variability more effectively.
The Q-value target calculated like in standard SAC algorithm, which uses double-Q trick and entropy term for enhanced exploration. A slight difference from standard SAC is sampling the next state action a' from the target policy rT(--s").
min Q8 (s', d' ((((ss))) - a log (o(a1))) a ~ (a(-)')) d=1,2
au) = c
where a is the learned entropy coefficient and ps is the parametrized behavior policy. The variance target update, Mar, then incorporates the Q-value temporal difference (TD) error, óQ Qua min/=1.2 Qa (s.
(ar a) = -53 +72
(min
min Ma (s', ~ m((16)) - log (oalaaa))) d=1,2
To ensure the behavior policy does not diverge from target GVF policies Ty we introduce KL regularization term that maintains aligmment with each target policy by minimizing (L(pp(-Is) (-(s))
As computing exact Q-values is infeasible for continuous state-action domains, we approximate these values using Monte Carlo (MC) rollouts over 100 episodes for fixed set of randomly selected 50 sampled state-actions. This MC approximation enables accurate computation of MSE by comparing learned Q-values with empirical MC estimates.
Finally, the behavior policy Pe is updated to minimize the MSE by focusing on areas of high variance, effectively improving estimation efficiency. Further, the entropy term is similar standard SAC algorithm for improving the exploration. This update step, driven by the variance network, is computed as:
min Mw (s, a) log + (LL(po(1s)) m((()))) d=1,2
where action ä is sampled from (-1s). To support adaptive exploration, we update a by optimizing: -a log us (-|s) + H,
where is the target entropy.
With these adaptations, SAC-GVFExplorer effectively estimates GVFs values in continuous state-action domains, achieving lower MSE compared to baselines such as RoundRobin and UniformPolicy and performing well in complex Mujoco environments.
We used TD3 train target policies for each GVF and selected mid-level performing policies as target policies for the respective GVF tasks. This setup allows the behavior policy to efficiently gather data parallel GVF estimation.
25

---

Algorithm 2: SAC-based GVFExplorer
Input:
Target policies TTE( initialized behavior policy Ma, replay buffer D, primary networks Q with 01 02 primary M variance with W1, W2, target networks Q8 Qb2- M May. learning rates aq. aM, mini-batch size b, entropy coefficient a, KL
regularizer 3, update frequencies p. L, target entropy H, training steps K
for environment step k: = 1,..., K do
Select action (=((-s))
Observe next state s' and cumulants c
Store transition (s, a, 8', in replay buffer 'D if' step% == 0 then
Sample mini-batch D as,
//Q-critic update
Compute Qiar c (mina- 1 m((+ls)) a loguu(alb')).
a~ Mo(+)s')
Update Qo with MSE loss: (Qtar Q0, for d = 1,2
10
//Compute TD error
11
8Q = Qtr min Qe, (s,a)
12
//Variance-critic update
Compute (ta a) = 22 + 72 (min (an d' ti(-(s')) - a (lggua(a)s))
13
à ~ (-((-ss))
14
Update My with MSE loss: (Maor Mws (s,, az)) for d = 1,2 end
15
16
if step% == 0 then
17
Update target networks: 0u = 9d, We = wd end
19
step%m
== = 0 then // Update behavior policy (bb
21
Update s using
V+E. ming=1.2 Mas (s. (āls) E.n, KL(Ho(-Is) |I TT(-1s)))
22
where à sampled from (-(s).
//pdaate entropy regularizer
Update with loss (-a log u + H)
end 26 end
Returns Estimated GVF values (8, for {1, .I.
NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:[Yer
Justification: We propose GVFExplorer and support the algorithm with proofs of Theo- rem 4. and Theorem 4.2 in the App.[A A Further, we empirically show the performance in Sec. in tabular and non-linear func. approx. setting.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
26

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer Sec. g for limitations, where we discuss the limitation with extra
computation cost of the variance function, as compared to baselines which might just require value function.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Refer App. A for complete assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
27

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the exact proposed GVFExplorer in Algorithm with all the parameters. The optimized hyperparameters for all algorithms presented in Table 2 The learning rate sensitivity graph is shown in Fig. 6 We provided the code link in the paper. The environmental details and network architecture described in App. B.3 and App. B.3.5
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We present the code link in the paper: https://github. com/ arushi brruhhjaing4//VFEE..e aing4/GVFExplorer to produce all main and supplementary results. We don 't use any dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
28

---

including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. coips..cc/ublic//guuidee http://uuuddeeeeeeSbbbbbiaiaa / sinioooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The hyperparameters are presented App. B.3 and App. B.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report standard error in the empirical results and averaged performance over more than 25 runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
29

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Refer directly text immediately below Appendix Experiment section, Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://eeee.sss..cueccee///eeiie
Answer: [Yes]
Justification: We follow all the ethical standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work presented here proposes a data-efficient approach to evaluate multi- ple objectives (GVFs) and does not experiment with real-world data. In the passage before the Contributions section we outline potential extension to real-world applications like per- sonalized recommender systems. The proposed method can unlock positive societal impacts. By requiring less data for evaluation, it enhances user privacy and reduces computational footprint for more sustainable future. Though this work offers benefits for individual users, applying it across users necessitates privacy-preserving approaches. This work opens exciting future directions integrating or differential privacy with our method.
approaches like federated learning
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out
30

---

that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed work is more suited towards academic and research setting, posing no such described above risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use classical RL benchmark environments like FourRooms (cited them) and invented few environments. Our code link is added in the paper. We use existing EMDP library extend the gridworld envs which uses MIT License.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi www.swtthhhce...c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
31

---

Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: We provide the link to our codebase in the paper which has been properly anonymized.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not have any crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Paper does not involve any crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "4m3mFx8tAB", "reviewer": "Reviewer_pomH", "summary": "This paper presents a new algorithm for collecting data needed to learn multiple GVFs in parallel. By focusing data collection on high-variance (s,a) pairs, an agent is able to collect data that will reduce the variance of estimated GVFs. The authors contribute a sort of contraction-mapping proof that using their algorithm will result in non-increasing variances.", "questions": "I wonder if there are potential degeneracies in the algorithm. For example, if a behavior policy never tries a certain action, it seems like the M's for that action will be 0, and the new behavior policy will assign 0 probability to taking that action in the future, leading to a situation where you never get the data you need.  Similarly, if a certain state is never visited (in the tabular case), would the estimate of M be zero?\n\nIf so, does that imply that there is an unstated constraint on the initial behavior policy -- something about exploring states and actions sufficiently?\n\nA related question: if the cumulant function is sparse, is it possible to not get enough non-zero data to get non-zero M estimates?\n\n(by the way, it's these sorts of questions where I can kind of see that degeneracies may arise that will never get ironed by your algorithm, which is why the difference between \"<\" and \"<=\" in your proof is important)", "limitations": "It seems like a central limitation of the work is a disconnect between what was theoretically proven (which relies on a perfect knowledge of the variances M), and what will happen in practice (the M's must be estimated).\n\nIt would be nice if the paper outlined what happens to the algorithm as a function of the error in the estimates of M.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The algorithm is simple, with some reasonable theoretical properties.\nThe paper addresses a good problem.\nThe paper is very well written\nThe experiments seem reasonable and informative\nThe method performs well against other baselines", "weaknesses": "I have one minor-but-important quibble about the paper. In Thm. 4.2, the authors prove that aggregated variance is \"<=\" upon successive iterations. However, in the english description of the result, the authors state that the aggregated variances \"decrease with each update step\". \n\nThis is NOT what you proved - you proved that aggregated variance \"does not increase.\"  The same claim is made in the abstract, and again in the conclusion.  I think it's important to be clear on this point, so I would ask the authors to rephrase this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WGG8WZguwJ", "reviewer": "Reviewer_A7c7", "summary": "This paper propose a new algorithm to solve the general value function evaluations problem. In essence, GVFs can be seen as high dimensional value functions. The authors propose a temporal difference learning algorithm that minimizes the overall variance in the return distribution, in the hope to improve the behavior policy for better exploration, such that the samples the behavior policy produces better suffices for off-policy evaluations for each GVFs.", "questions": "Please see weakness above.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of minimizing the variance of the return of the behavior policy is novel. While the idea might now be groundbreaking, it is new in this field. \n2. Incorporating a temporal difference to approximate the overall variance of the return of the each general values are interesting. Indeed, for large scale problems, TD is a better solution overall. \n3. The derivation and analysis of the algorithm is sounds and rigorous. \n4. Overall the paper is clearly presented.", "weaknesses": "1. The core idea behind this paper is to propose a solution to solve the data collection problem. This paper does answer the question of how to minimize the variance of the returns, but fail to convince me entirely why we should do that at the first place, either through proofs, or empirical investigation. I think this is the biggest weakness of the paper.\n2.  The experiment section is nice and clear but the problem class is a bit simple (gridworld). For a paper without strong theoretical results, experiments are usually expected to have more materials. In this sense, the results are not very convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and positive feedback on our algorithm's rigorous derivation and analysis. Based on your suggestion, we have now **added Mujoco experiments result in the main comment**. We respond to each query below.\n\n1. **“Why did we choose to minimize variance of return as our objective”?**\n\n   We think there may be a misunderstanding regarding the question. Below, we have tried to clarify our rationale, but please let us know if further discussion would be beneficial.\n\n   The primary objective is to identify a behavior policy that minimizes MSE when evaluating multiple GVFs. As outlined in lines 120-125, $\\text{MSE} = \\text{bias}^2 + \\text{variance}$. Using the unbiased IS estimator, we reduce the problem to minimizing the variance, which is the core objective. Minimizing variance improves the accuracy of value estimates by reducing uncertainty. This ensures that the behavior policy collects the most informative samples, thereby reducing estimation error quickly.\n\n\n   Furthermore, Owen et al. (2013) demonstrated that using a minimum-variance optimal behavior policy ($\\mu^*$) for a single target policy ($\\pi$) in scenarios with known dynamics can lead to performance improvements. The value obtained under $\\mu^*$ is greater than under $\\pi$, where the improvement is directly related to the variance reduction achieved. This indicates that the higher the variance reduction under $\\mu^*$ policy, the more significant the performance improvement. We hypothesize that similar effects could be observed in multiple GVF policy “Control” scenarios, which we aim to investigate in future research. This work lays the foundations under the policy evaluation context.\n\n   We will include this rationale in the camera-ready version to further substantiate our approach.\n\n2. **“Query regarding more experiments in Mujoco”**\n\n   We have now added the empirical results in the Mujoco environment with continuous actions. Please refer to the attached PDF and the main comment for the performance graphs and explanation on Mujoco experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GVHBUUKE66", "reviewer": "Reviewer_3wne", "summary": "This paper presents a novel method named GVFExplorer for efficiently evaluating multiple Value Functions with different policy (GVFs) in parallel using off-policy methods. It adaptively learns a single behavior policy that minimizes the total variance in return across GVFs, thus reducing required environmental interactions. The method uses a temporal-difference-style variance estimator and proves that each behavior policy update decreases the overall mean squared error in GVF predictions. The performance of GVFExplorer is empirically validated in various settings, including tabular and non-linear function approximation, with stationary and non-stationary reward signals.", "questions": "- The paper starts with minimizing MSE, and argues that it uses unbiased IS estimation, leading to a policy that minimizes the variance of return. However, GVF in algorithm is estimated with $Q_\\theta$, which has been learned with expected Sarsa, which is a baised estimate given that the target contains $Q_\\theta$ instead of GT $Q^\\pi$. How do we ensure that the analysis above also works in algorithm 1? What happens if our function approximator is crude so that the bias is large?\n\n- Why does PER drastically improve the performance of GVFExplorer? If we have infinite number of data in our experience replay, GVFExplorer would have very similar effect as PER. In my opinion, it seems more natural to have reduced effect of PER as we are sampling hard states more with GVFExplorer, and it is what PER tries to do as well. (it also does not go well with the rest of the paper, as PER is about training efficiency where GVFExplorer is about the choice of behavior policy)", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- idea of choosing behavior policy for simultaeneous learning of multiple policies is quite novel, and seem to be able to be applied to some applications.\n- systematic way of deriving algorithm is novel and interesting, and makes sense as well", "weaknesses": "- More experiments would help understand the behavior of the algorithm.\n  - uniform policy being the best baseline does not seem to be a good baseline choice. Having some more competetive baselines, e.g. ablations, would have been much better.\n  - experimented environments seem too synthetic. would like to see results on typical (and challenging) RL environments, e.g., mujoco.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and valuable experimental suggestions, which we have now incorporated into our rebuttal. Based on your input, we have also **added results in Mujoco in the main comment (refer Fig 1 in PDF)**. We appreciate your recognition of the novelty of adaptively learning the behavior policy to efficiently evaluate multiple GVFs in parallel and acknowledgment of the systematic derivation of the algorithm. Detailed responses to your queries are provided below.\n\n1. **\"How can Uniform sampling be a competitive baseline?\"**\n\n   We use several baselines: uniform, mixture policy, round-robin policy, SR-based policy, and BPS. Learning a behavior policy for multiple GVFs is relatively unexplored, making it challenging to identify more baselines. Our empirical results indicate that the relative performance of these baselines can vary depending on target policy characteristics.\n\n   For example, in Fig. 7 of the main paper, round-robin and mixture policies perform better when using semi-greedy fixed target policies. Conversely, the uniform policy can be advantageous when the target policies exhibit low goal-reaching probability, because it helps hit the goal by chance.\n\n2. **\"Query: Show experiments in Mujoco\"**\n\n   We have now expanded our algorithm to continuous actions and shown results in Mujoco. Please refer to **Fig 1 in the attached PDF** and the main comment for the graphs and explanation of the Mujoco experiments. We believe these additions significantly strengthen our paper and provide a more comprehensive evaluation of our proposed method.\n\n3. **\"How does using an approximated Q in the target, rather than the true Q, affect our algorithm's analysis? What is the impact on MSE performance when the function approximator is crude?\"**\n\n   All TD-based methods, including IS-TD, are biased because of bootstrapping. Only with MC return, IS is unbiased. We use the unbiased estimator as a motivation to reduce the problem to minimizing the variance. Empirically, we used TD-based methods in all experiments to approximate both value and variance, due to the better stability and efficiency of these estimators in large-scale problems.\n\n   To better understand the impact of using a crude function approximator on MSE, we have now added experiments on this ablation study. Specifically, we examined the average MSE in a 20x20 grid with two distinct GVFs. We produced features by reducing the state space into a 10x20 feature grid (grouping factor = 2) and a 5x20 feature grid (grouping factor = 4). An approximation factor of 1 shows the result without any function approximation.\n\n   In **Fig 2 (PDF attached in main comments)**, as expected, the overall MSE increases with cruder approximations. Despite this, GVFExplorer outperforms round-robin and mixture policies. However, when the approximation is very crude (factor = 4), the uniform policy performs better due to poor variance estimates. These results suggest that GVFExplorer is robust with reasonable function approximators, but can degrade with extremely coarse ones, which is to be expected.\n\n   Further, these results are also strengthened by the performance in the attached Mujoco environment (Fig 1 in PDF in main comments) where function approximators are used.\n\n4. **\"How does PER improve GVFExplorer performance? How is PER different from our algorithm? Given infinite data in the replay buffer, would GVFExplorer have the same effect as PER?\"**\n\n   PER complements GVFExplorer by enhancing data efficiency. While GVFExplorer optimizes the behavior policy to sample informative data, PER reweights the collected samples in the buffer according to priority, ensuring that the mini-batches sampled for gradient descent are selected efficiently. Please refer to lines 307-309 for reasoning on performance boosts of GVFExplorer with PER.\n\n   With infinite data, the impact of both GVFExplorer and PER would diminish, and direct sampling from the target policies would suffice. However, in practical settings with limited data, GVFExplorer's ability to actively influence data collection provides a significant advantage.\n\n   **It is crucial to distinguish between GVFExplorer and PER**. GVFExplorer adapts the behavior policy based on variance estimates, while PER reweighs the sampling of the existing data already collected within the buffer. We used PER with all algorithms, including baselines, as shown in Fig 3a of the main paper; all baselines, except the SR method, show improved performance with PER.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for the valuable and constructive feedback. We are encouraged by your recognition of the *novelty of the problem* and acknowledgment of our *algorithm derivations as systematic and rigorous*.

Based on your feedback, we have extended our experimental results in the **Mujoco environments (Fig 1)** and included these results here. We have also added the codebase to the code repository. Further, based on Reviewer 2's suggestion, we are including an ablation study on the effect of using **feature approximator on the MSE performance metric (Fig 2)**. Both the results in Mujoco and the ablation study further support the benefits of using GVFExplorer for data-efficiently evaluating multiple GVFs in parallel.

$~~~~~~~~~~~$
## Mujoco Experiments (Reviewer 1 and 2)

We have now conducted additional experiments using the DM-control suite to experiment with the Mujoco “walker” and “cheetah” domains. For the walker, we define two distinct GVFs: walk and flip. For the cheetah, we also evaluate two GVFs: walk and run. To extend the implementation in a continuous action environment, any policy gradient algorithm can be used. Similar to the Q-value critic, we implement a separate M-variance critic using neural networks for both. The behavior policy interacts with the environment and gathers samples. It uses the samples to first update the two critics, followed by an update to the behavior policy network to minimize the MSE (our objective). We also added a KL regularizer between the behavior policy and the two target policies of the GVFs to prevent divergence. We use Monte Carlo estimates of the true GVF values and compare the MSE between these values and the output of the Q-critic network. We use the same Q-critic architecture for all algorithms, including the baselines.

**Fig 1 of the attached PDF:** GVFExplorer significantly reduces the average MSE compared to baselines such as RoundRobin and UniformPolicy, demonstrating that an adaptive behavior policy collects more informative samples.

$~~~~~~~~~~~$
## Ablation study on effects of using feature approximator on MSE (Reviewer 2)

We conducted an ablation study to understand the effects of using a very coarse function approximator on MSE. We evaluated the averaged MSE over two distinct GVFs in a 20x20 grid. To simulate approximation, we mapped each 2x1 grid region to the same feature, resulting in a grouping factor of 2 and a 10x20 feature grid. Similarly, mapping each 4x1 grid region to the same feature resulted in a grouping factor of 4 and a 5x20 feature grid. As shown in **Fig 2 in attached PDF**, the overall MSE increases with cruder approximations, as expected. GVFExplorer generally outperforms the baselines, but with a very crude approximation (factor = 4), the uniform policy performs better due to poor variance estimates. These results highlight GVFExplorer's robustness with reasonable function approximators. More details of the experiment are presented in R2’s response.

Meta-review: The reviewers and I generally liked this paper. While the core algorithm is a bit obvious, the paper is well-written and makes a positive contribution to the RL literature on GVFs.  The paper has a nice mix of theory and empirical work, and addresses a problem that has not been previously studied, and provides proofs of correctness (never easy to do in an RL setting!).  The paper also provides a clear, well-written discussion of the issues surrounding GVFs.  Based on the alignment of all of these positive factors, I recommend acceptance.
Decision: Accept (poster)