Learning State Representations from Random Deep Action-conditional Predictions
Authors: Zeyu Zheng
Vivek Veeriah
Risto Vuorio
Richard Lewis
Satinder Singh
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
state representation learning
auxiliary task

Abstract: Our main contribution in this work is an empirical finding that random General Value Functions (GVFs), i.e., deep action-conditional predictions---random both in what feature of observations they predict as well as in the sequence of actions the predictions are conditioned upon---form good auxiliary tasks for reinforcement learning (RL) problems. In particular, we show that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted auxiliary tasks like value prediction, pixel control, and CURL in both Atari and DeepMind Lab tasks. In another set of experiments we stop the gradients from the RL part of the network to the state representation learning part of the network and show, perhaps surprisingly, that the auxiliary tasks alone are sufficient to learn state representations good enough to outperform an end-to-end trained actor-critic baseline. We opensourced our code at https://github.com/Hwhitetooth/random_gvfs.

Learning State Representations from Random Deep Action-conditional Predictions
Zeyu Zheng University of Michigan zeyu@umich. edu
Vivek Veeriah University of Michigan wveeriah@umi ch. edu
Risto Vuorio University of Oxford
risto. vuorio@cs. ox. ac uk
Richard Lewis
Satinder Singh University of Michigan baveja@umi ch. edu
University of Michigan rickl@umicoo ch, edu
Abstract
Our main contribution this work is an empirical finding that random General Value Functions (GVFs), i.c., deep contiiinoniiionall predictions-ranne in what feature of observations they predict as well as in the sequence actions the predictions conditioned upon- form good auxiliary tasks for reinforcement learning (RL) problems. In particular, we show that random deep action action-conditional predictions when used auxiliary tasks yield state representations that produce control performance competitive with state-of-the-ar hand-crafted auxiliary tasks like value prediction, pixel control, and CURL in both Atari and DeepMind Lab tasks. In another set of experiments we stop the gradients from the RL part of the network the state representation learning part of the network and show, perhaps surprisingly, that the auxiliary tasks alone are sufficient to learn state representa- tions good enough outperform an end-to-end trained actor-critic baseline. We opensourced our code at wttht:tt//// battp://ggithut com
both
Introduction
Providing auxiliary tasks to Deep Reinforcement Learning (Deep RL) agents has become an important class of methods for driving the learning of state representations that accelerate learning on main task. Existing auxiliary tasks have the property that their semantics are fixed and carefully designed by the agent designer. Some notable examples include pixel control, reward prediction, termination prediction, and multi-horizon value prediction (these are reviewed in more detail below). Unlike the prior approaches that require careful design of auxiliary task semantics, explore here different approach in which a set of random action-cont cirioo--ondiiionall prediction tasks are generated through rich space of general value functions (GVFs) defined by a language predictions of random features of observations conditioned on random sequence of actions.
Our main, and perhaps surprising, contribution in this work is an empirical finding that auxiliary tasks of learning random GVFs-again, random in both predicted features and actions the predictions are conditioned upon- yield state representations that produce control performance that is competitive with sfate-of-the art auxiliary tasks with hand-crafted semantics. We demonstrate this competitiveness in Atari games and DeepMind Lab tasks, comparing multi-horizon value prediction (8). pixel control (13,, and CURL [1S] as our baseline auxiliary tasks. Note that while we present reasonable approach to generating the semantics of the random GVFs we employ in our experiments, the specifies of our approach is not by itself contribution (and thus not evaluated against other approaches to producing semanties for random GVFs). and alternative reasonable approaches for generating random GVFs could do as well.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Additionally, through empirical analyses on illustrative domains we show the benefits of exploiting the richness of GVFs temporal depth and cetion-condiiioality We also provide direct evidence that using random GVFs learns useful representations for the main task through stop-gradient experiments in which the state representations are trained solely via the random-GVF auxiliary tasks without using the usual RI. learning with rewards to influence representation learning. We show that, again, surprisingly, these stop-gradient agents outperform the end-to-end-trann -rainee actor-critic baseline.
2 Background and Related Work
Horde and PSRs. Auxiliary tasks were formalized and introduced to RL. in (77 through the Horde architecture. Horde is an off-policy learning framework for learning knowledge represented as GVFs from an agent's experience. Our work is related to Horde in the use rich subspace of GVF predictions but differs in that our interest is in the effect of learning these auxiliary predictions on the main task via shared state representations rather than to show the knowledge captured in these GVFs Our work is also related to predictive state representations (PSRs) 17 24]. PSRs use predictions as state representations whereas in the use of deep neural networks in RL as powerful function approximators, various auxiliary tasks have been proposed to improve the latent state representations of Deep RL agents. We review these auxiliary tasks below. Our work belongs to this family of work in that the auxiliary prediction tasks used to improve the state representations of Deep RI. agents.
work learns latent state representations from predictions. Recently,
Auxiliary tasks using predefined GVF targets. UNREAL [3] uses reward prediction and pixel control and achieved significant performance improvement in DeepMind Lab but only marginal improvement in Atari. Termination prediction 14] is shown to be an useful auxiliary task in episodic RL settings. SimCore DRAW [9] learns a generative model of future observations conditioned on action sequences and uses it as an auxiliary task to shape the agent's belief states in partially observable environments. Fedus et al. [8] found that simply predicting the returns with multiple different discount factors (MHVP) serves as effective auxiliary tasks. MHVP relies on the availability of rewards and thus is different from our work and other unsupervised auxiliary tasks.
Information-theerrtic auxiliary tasks. information-theoreee representations that informative about the future trajectory of these representations the agent interacts with the environment. CPC (30), CPC|action [00), ST-DIM L.. DRIML [19], and ATC (25) apply different forms of temporal contrastive losses to learn predictions in latent space. CURL [SS
approaches to auxiliary tasks learn
ignores the long-term future and applies a contrastive loss on the stack of consecutive frames to learn good visual representations. PBL [11 focuses on partially observable environments and introduces separate target encoder to set the prediction targets. The target encoder is trained to distill the learned state representations. SPR (3 replaces the target encoder in PBL with a moving average of the state representation function. In addition to being predictive, PI-SAC [66] also enforces the state representations to be compressed. SPR and PI-SAC focuses data efficiency and only conducted experiments under low data budgets. In these information-theone GVFs and despite some empirical success, none could learn long-term predictions effectively. This is in contrast to GVF-like predictions which can be effectively learned via TD as in our work as well as the work presented above.
approaches, the targets are not
Theory. A few recent works 7 18 have studied the optimal representation in RL from a geometric perspective and provided theoretical insights into why predicting GVF-like targets is helpful in learning state representations. Our work is consistent with this thcoretical motivation.
GVF discovery. Veeriah et al. (31 used metagradients to discover simple GVFs (discounted sums of features of observations) In this work, we show that random choices of features and random but rich GVFs are competitive with the state-of-the art of hand-crafted GVFs as auxiliary tasks.
GVF RNNs. Rather than using GVFs as auxiliary tasks, General Value Function Networks (GVFNs) (22 are a new family of recurrent neural networks (RNNs) where each dimension of the hidden state is a GVF prediction. GVFNs are trained by TD instead of truncated backprop through time. Our work relates to GVFNs in that both works use GVFs to shape state representations. However, unlike GVFNs, our work uses GVFs as auxiliary tasks and does not enforce any semantics to the state representations. Moreover, our empirical study mainly focuses on the control setting where the agent needs to maximize its long-term cumulative rewards, whereas (2 mainly focuses on time series modelling tasks and online prediction tasks and demonstrated the superior performance of GVFNs over conventional RNNs when the truncated input sequences are short during training.

---

T,
1, j.ans
8
1
n,, 0; ORL
yt 0ans
0rope
a
6
Ou, Au Au...., O2
(b)
Figure I: (a) An example of question network. The squares represent feature nodes and circles represent prediction nodes. (b) The agent architecture. The dashed cross denotes an optional stop-gradiens operation.
Method
In this section we first describe the specific GVFs we studied in this work. Then we describe an algorithm for the construction of random GVFs. We finish this section with description of the agent architecture used in our empirical work.
3.1 GVFs with Interdependent TD Relationships
In this work, study auxiliary prediction tasks where the semantics of the predictions are defined by set of GVFs with interdependent TD relationships (this family of GVFs are often referred as temporal-difference networks in the literature [28 (9). The TD relationships among the GVFs can be described by a graph with directional edges, which we call the question network as it defines the semantics of the predictions.
Figure la shows an example of a question network. The two squares represent two feature nodes and the six circles represent six prediction nodes. Node 1 (labeled in the circles) predicts the expected value of feature f at the next step. Implicitly, this prediction is conditioned on following the current policy. Node 2 predicts the expected value of node at the next step. Note that we can "unroll" the target of node 2 to ground it on the features. In this example, node 2 predicts the expected value of feature f after two steps when following the current policy. Node 5 has self-loop and predicts the expectation of the discounted sum of feature fa with a discount factor y. We call node 5 a discounted sum prediction node. Node 3 is labeled by action a. It predicts the expected value of node 1 at the next step given action a is taken at the current step. We say node 3 is conditioned on action Similarly, node 4 predicts the same target but conditioned on action Node 6 has two outgoing edges. It predicts the sum (in general a weighted sum, but in this paper we do not explore the role of these weights and instead fix them to be 1) of feature f' and the value of node 5, both at the next step. In this case, it is hard to describe the semantic of node 6's prediction in terms of the features, but we can see that the prediction is still grounded on feature fl and f2.
Generalising from the example above, question network with prediction nodes and ny feature nodes defines ny predictions of n, features. We use Np to denote the set of all prediction nodes and Ny to denote the set of all feature nodes. Let W be the adjacency matrix of the question network. Wij denotes the weight on the edge from node i to node j. We define Wij 0 if there is no each
edge from node to node j. Now consider an agent interacting with the environment. step f, it receives an observation O, and takes an action A, according to its policy л. Then at the next step it receives an observation O.+1- The feature f*(O,, Ar, O++1) is a scalar function of the transition. The agent makes prediction ÿ' (O, Ao...., .,) for each prediction node i based on its history; this is computed by neural network in our work. For brevity, we use ff, and o to denote (O,, A,, Os41) and j (OO. Ao....,0) O₂) respectively. The target for prediction i at step / is

---

denoted by v/. If prediction node is not conditioned on any action, its target is
v
ET JENP
Σ
Woyl+++ +
Σ
Winsital
REN,
otherwise, if it is conditioned on action a', its target is
y = ET Σ
WoUL +
Σ
W Witffil44 = a²].
JEN,
KEN,
By the construction of the targets, the agent can learn the prediction of via TD. If is not conditioned on any action, then z7 is updated by
0 + Σ
W + Σ
Wafe
JEN
AEN,
otherwise, if i is conditioned on action a', then of is updated by
W Wish+ Σ
Wast if a
0 +-
JEN,
keN,
01
otherwise
In episodic setting, if the episode terminates at step T, we define v prediction nodes
and Ø 0 for all
These GVFs represent a broad class of predictions. Many existing auxiliary prediction tasks can be expressed by a question network. Reward prediction [13] can be represented by question network with single feature node representing the reward and single prediction node predicting the reward. Multi-horizon value prediction [ can be represented by similar question network but with multiple self-loop prediction nodes with different discounts. Termination prediction [14] can be represented by a question network with a feature node of constant 1 and a self-loop node with discount 1.
3.2 A Random Question Network Generator
In this work, instead of hand-crafting a new question network instance as in previous work on the use of predictions for auxiliary tasks, we verify conjecture that large number of random deep. action-conditional predictions is enough to drive the learning of good state representations without needing to carefully hand-design the semantics of those predictions. To test this conjecture, designed generator of random question networks from which we can take samples and evaluate their performance as auxiliary tasks. Specifically, we designed heuristic algorithm that generates question networks with random features and random structures.
Random Features. We random features, each computed by scalar function g with random parameters. For any transition (Or, Ar, O++1), ), the feature is computed as f = (0*+()+)) g² (O.)). Instead of directly using the output of g² as the feature, we use the amount of change in g². similar transformation was used in pixel control [3]
Random Structure. The intuition behind our heuristic for generating random question network structures create predictions that correspond executing an open-loop action sequence and then following the policy while accumulating one random feature at each time step. As we will illustrate in Section these predictions can provide rich training signals for learning good representations. Specifically, the generator takes 5 arguments as input: number of features n the discrete action set A. a discount factor 7, depth D, and repeat R. Its output is a question network that contains ny feature nodes as defined above, and D + layers, each layer contains R x |A| prediction nodes except the first layer which contains n, prediction nodes. We construct the question network layer by layer incrementally from layer 0 to layer D. First, layer O has n, feature nodes and n, prediction nodes; each prediction node has an edge to a distinct feature node with weight on the edge and has self-loop with weight Each prediction node in layer 0 predicts the discounted sum of its corresponding feature and are not conditioned on actions. Then for each layer / (1 S l < D), we create R |A] prediction nodes. Each prediction node is conditioned on one action and there are exactly R nodes that are conditioned on the same action. Each prediction node has two edges, one to random node in layer ( - 1 and one to a random feature node in layer O. Note that prediction

---

b
(a)
(b)
(c)
Figure 2: The question networks we studied in our illustrative experiment. (a) A discounted sum prediction. (b) A depth-2 tree question network with 2 actions. The bottom right prediction node predicts the sum of the values of the feature in the next two steps if action b were taken for both the current and the next step. Other prediction nodes have similar semantics. (c) A random question network sampled from rGVFs. There are 2 features and 2 actions, with depth and repeat set to 2.
nodes in layer 1 do not necessarily connect to self-loop prediction node in layer 0 they may only connect to a feature node. A constraint for preventing duplicated predictions is included so that any two prediction nodes in layer I that are conditioned on the same action cannot connect to the same prediction node layer 1. our preliminary experiments, we tried adding self-loops to deeper layers and allowing denser connections between nodes. Sometimes those additional loops and dense edges caused instability during training. We leave the study of more sophisticated question network structures to future work. The Appendix includes pseudocode for the random generator algorithm.
3.3 Agent Architecture
We used standard wallllary-tssk-agggne agent on the actor-critic architecture and it consists of 3 modules. The state representation module, parameterized by (repe, maps the history of observations and actions (On, ....... Ot) state vector S,. The RL module, parameterized by (RL, maps the state vector S, to a policy distribution over the available actions π (-) and an approximated value function i i(S.). The answer network module, parameterized by gTn maps the state vector St to set of predictions ((S,) equal in size the number of prediction nodes in the question network. Like in previous work, the augmented agent has more parameters than the base A2C agent due to the answer network module. However, the policy space and the value function space remain the same and these auxiliary parameters are only used for providing richer training signals for the state representation module.
agent architecture, shown in Figure Tb We base
We trained the network in two separate ways. In the auxiliary task setting, the RL loss CRL is backpropagated to update the parameters of the state representation (0ePr) and the RL (ORL) modules, while the answer network loss L is backpropagated to update the parameters of the answer network (gans) and state representation (d"P)) modules. Note that the answer network loss only affects the RL module indirectly through the shared state representation module. In the stop- gradigra setting, we stopped the gradients from the RL loss from flowing from L to greprr This allows us to do a harsher and more direct evaluation of how well the auxiliary tasks can train the state representation without any help from the main task. For LRL, we used the standard actor-critic objective with an entropy regularizer. For Cans, we used the mean-squared loss for all the targets and predictions.
4 Illustrating the Benefits of Deep Action-conditional Questions
The main aim of the experiments this section is illustrate how deep cectionconniiiioall predictions can yield good state representations. We first use a simple grid world to visualize the impact of depth and action conditionality on the learned state representations. Then we demonstrate the practical benefit of exploiting both of these two factors by an ablation study on six Atari games. In addition, to test the robustness of the control performance to the hyperparameters of the random GVFs, we conducted random search experiment on the Atari game Breakout.
5

---

0.5
0.50
rand.
0.4
0.75
1EVTs. (16 rend. leab)
1EV7s rand. teak)
0.3
d scounted sum
0.70
IDV7s (tsuch)
(louct)
random g
tree. depth
0.15
0.2
free. depth
death
0.10
tree, 0.05
end to end
0.0
0.4
0.6
0.8
1.0
0.0
Millins of frames
Hillons of framess
(a)
(b)
(c)
Figure 3: (a) The illustrative grid world environment. The blue circle denotes the agent and the yellow star denotes the rewarding state. (b) MSE between the learned value function and the true value function the tree question networks experiment. (c) MSE between the learned value function and the true value function in the random question networks experiment.
(a)
(b)
S
Figure 4: Visualization of the learned value functions in the empty room environment. Bright indicates high value and dark indicates low value. (a) The true values. (b) The discounted sum predictions the touch feature. (c)- (f) The prediction are defined by a full-tree-structure question network regarding the touch feature. The depth of the tree increases from to 4 from (c) to (f).
4.1 Benefits of Depth and Action-conditionaliy:: Illustrative Grid World
Although our primary interest (and the focus of subsequent experiments) is learning good policies, in this domain we study policy evaluation because this simpler objective is sufficient to illustrate points and shows the environment, a by 7 grid surrounded by walls. The observation is a top-down view including the walls. There 4 available actions that move the agent horizontally or vertically adjacent cell. The agent gets a reward of upon arriving at the goal cell located in the top row, and 0 otherwise. This is continuing environment so achieving the goal does not terminate the interaction. The objective is to learn the state-value function of a random policy which selects each action with equal probability. We used discount factor of 0.98.
compute and visualize the true value function for comparison. Figure 3a
an
Specifying question network requires specifying both the structure and the features. Later we explore random features, but here has a clear semantics. touch is if the agent's 's move is blocked by the wall and is otherwise.
single hand-crafted touch feature that every prediction
Using the touch feature we constructed two types of question networks. The first type is the discounted sum prediction of touch (we used a discount factor 0.8) (Figure 2ab. The second type is a full action-conditional tree of depth D. There is only one feature node in the tree which corresponds to the touch feature. Each internal node has child nodes conditioned on distinct actions. Each prediction node also has skip edge directly to the feature node (except for the child nodes of the feature node). Figure 26 shows an example of a depth-2 tree (the caption describes the semantics of some of the predictions). We also compared to randomly initialized state representation module as a baseline where the state representation was fixed and only the value function weights were learned during training.
Neural Network Architecture. The empty room environment is fully observable and so the state representation module is a feed-forward neural network that maps the current observation O, to a state vector S.. parameterized by a 3-layer multi-layer perceptron (MIP) with 64 units in the first two layers and 32 units in the third layer. The RL module has hidden layer with 32

---

units and one output head representing the state value. (There is no policy head as the policy was given). The answer network module also has one hidden layer with 32 units and one output layer. We applied a stop-gradien between the state representation module and the RI. module (Figure Tbp.
More implementation details are provided in the Appendix.
Results. We measured the performance by the mean-squared error (MSE) between the learned value function and the true value function across all states. The true value function was obtained by solving a system of linear equations [26]. Figure shows the MSE during training. Both the random baseline and the discounted sum prediction target performed poorly. But even tree question network of depth 1 (i.e., four prediction targets corresponding to the four action conditional predictions of touch after one step) performed much better than these two baselines. Performance increased monotonically with increasing depth until depth 3 when the MSE matched end-to-end training after 1 million frames.
Figure 4 shows the different value functions learned by agents with the different prediction tasks. Figure 4a visualizes the true values. Figure [4b] shows the learned value function when the state representations are learned from discounted sum predictions of touch. Its symmetric pattern reflects the symmetry of the grid world and the random policy, but is inconsistent with the asymmetric true values. Figure 4c shows the learned value function when the state representations are learned from depth-1-tree predictions. It clearly distinguishes 4 corner states, 4 groups of states on the boundary, and states in the center area, as this grouping reflects the different prediction targets for these states.
For the answer network module to make accurate predictions of the targets of the question network, the state representation module must map states with the same prediction target to similar representations and states with different targets to different representations. As the question network tree becomes deeper, the agent learns finer state distinctions, until MSE of O is achieved at depth 3 (Figure 4e).
4.2 Benefits of Random Question Nets: Illustrative Grid World
The previous experiment demonstrated benefits of temporally deeper action-conditonal prediction tasks. But achieving this by creating deeper and deeper full-branching action-conditional trees is not tractable the number of prediction targets grows exponentially. The previous experiment also used a single feature formulated using domain knowledge; such feature selection is also not scalable. The random generator described in Section 3 provides a method to mitigate both concerns by growing random question networks with random features.
Specifically, used discount 0.8, depth 4, and repeat equal to the number of features for generating random GVFs, Figure Be shows the MSE of different random GVF variants. The performance of random GVFs with touch-that is, random but not necessarily full branching trees of depth 4 --performed as well as touch with a full tree of depth 4. Random GVFs with a single random feature performed suboptimally; a random feature is likely less discriminative than touch. However, as the number of random features increases, the performance improves, and with 64 random features, random GVFs match the final performance of touch with full depth 4 tree.
The results on the grid world provide preliminary evidence that random deep action-condition GVFs with many random features can yield good state representations. We next test this conjecture on a set of Atari games, exploring again the benefits of depth and action conditionality.
4.3 Ablation Study of Benefits of Depth and Action Conditionality: Atari
Here we use six Atari games 4 (these six are often used for hyperparameter selection for the Atari benchmark (20) to compare four different kinds of random GVF question networks: (a) random GVFs in which all predictions are discounted sums of distinct random features (illustrated in Figure 2a and denoted IGVFs-discoonteed-su in Figure 5); (b) random GVFs in which all predictions are shallow action-conditional predictions, a set of depth-1 trees, each for a distinct random feature (denoted sGVFs-shallo in Figure 5):: (c) random GVFs without conditionnn (denoted rGVFs- no-actions in Figure 5) and (d) random GVFs that exploit both action conditionality and depth (illustrated in Figure 2c and denoted simply by rGVFs in Figure
Random Features for Atari. The random function y for computing the random features are designed as follows. The 84 x 84 observation O, is divided into 16 disjoint 21 x 21 patches, and a shared random linear function applies to each patch to obtain 16 random features gi, ..."" Finally, 16
process these features as described in 3.2
7

---

rGVFs discounted
BeardBuder
Breakout
Pung
Qcent
Seargues
Spar wwtweeeesss
32505
800
1390
1752
2240
1580
10000
1382
2505
110
580
50 160 150
200
50
333
150
200
23
305
123
200
23
100
356
302
160
150
260
50
100
150
200
frames
Figure 5: Learning curves of different question networks in six Atari games. x-axis denotes the number of frames and y-axis denotes the episode returns. Each curve is averaged over independent runs with different random seeds. Shaded area shows the standard error.
Neural Network Architecture. We used A2C 20 with standard neural network architecture for Atari [21 as our base agent. Specifically, the state representation module consists of convolutional layers. The RL module has one hidden dense layer and two output heads for the policy and the value function respectively. The answer network has one hidden dense layer with 512 units followed by the output layer. We stopped the gradient from the RL module to the state representation module.
Hyperparameters. The discount factor, depth, and repeat were set to 0.95, 8, and 16 respectively. Thus there are 16 + 8 16 |A|| total predictions. Random GVFs without action-conditioning has the same question network except that no prediction was conditioned on actions. To match the total used 16 + 8 16 |A] random features for discounted sum and 8 16
number of predictions, features for shallow action-conditional predictions. Additional random features were generated by applying random linear functions to the image patches. The discount factor for discounted sum predictions also 0.95. More implementation details are provided in the Appendix.
Results. Figure 5 shows the learning curves in the 6 Atari games. rGVFs-shallow performed the worst in all the games, providing further evidence for the value of making deep predictions. rGVFs consistently outperformed KGVFs-no-uctions, providing evidence that action-condittoning beneficial. And finally, rGVFs performed better than LEVFss-iscounted-aee (large difference in BeamRider and small differences in Breakout and Qbert), was comparable in 2 the other 3 games, and performed worse in one me-deespiee using many fewer features than rGVFs- discounted-sumn This suggests that structured deep action-conditional predictions can be more effective than simply making discounted sum predictions about many features.
in out games
4.4 Robustness and Stability
We tested the robustness of rGVFs with respect its hyperparameters, namely discount, depth, repeat, and number of features. We explored different values for each hyperparameter independently while holding the other hyperparameters fixed the values used in the previous experiment. For each hyperparameter, we took 20 samples uniformly from a wide interval and evaluated rGVFs on Breakout using the sampled value. The results are presented in Figure The lines of best fit (the red lines) in the left two panels indicate a positive correlation between the performance and the depth of the predictions, which is consistent with the previous experiments. Each hyperparameter has a range of values that achieves high performance, indicating that rGVFs are stable and robust to different hyprraaraeeter choices. Additional results in BeamRider and SpaceInvaders are in the Appendix.
Comparison to Baseline Auxiliary Tasks on Atari and Deep Lab
In this section, we present the empirical results of comparing the performance of rGVFs against the A2C baseline (20) and three other auxiliary tasks, i.e., multi-horizon value prediction (MHVP) [
Atari games and 12 DeepMind Lab enviromments 2 It is unclear how to apply CURL to partially observable environments which require long-term memory because CURL specifically designed to use stack of recent frames the inputs. Thus we did not compare to CURL in the DeepMind Lab environments. Our implementation of rGVFs for this experiment is available at wthhtps::// ttteps://githann thub comm/Haaiteetoccchraadmmnggge
pixel control (PC) 13,, and CURL [[5]. We conducted the evaluation
8

---

750
750
750
750
700
700
650
650
650
650
600
550
550
550
550
500
450
20
40
60
80
100
4.0
450
16
24 450
16
deeth
repeet
#feastures
Figure 6: Scatter plots of scores in Breakout obtained by rGVFs with different hyperparameters. x-axis denotes the value the hyperparameter. y-axis denotes the final game score after training for 200 million frames. The red line in each panel is the line of best fit. The dotted horizontal lines denote the performance of the end-to-end A2C baseline. The solid vertical lines denotes the values used in our final experiments.
Alani
Deepliind
20.0
120
25
100
20
A2C
80
12.5
Moai capppec hummnn
1.5
AZC rSVFs
MHVP
60
10.0
PC
1.0
5.0
stop-gradlent
2.5
0.0
Milliors frames
frames
Millions frames
(a)
(b)
(C)
Figure 7: (a) Median human-normalized score across 49 Atari games. (b) Mean record-normalized score across 49 Atari games. (c) Mean capped human-normalized score across 12 DeepMind Lab environments. In all panels, the x-axis denotes the number of frames. Each dark curve is averaged over 5 independent runs with different random seeds. The shaded area shows the standard error.
Atari Implementation. We used the same architecture for rGVFs as in the prior section. For MHVP, used 10 value predictions following [8]. Each prediction has unique discount factor, chosen to be uniform terms of their effective horizons from to 100 (f0, 1 10 20 ... 90 }).
The architecture for MHVP is the same rGVFs. For PC, we followed the architecture design and 13 For CURL, we implemented it in our experiment setup by using the code hyperparameters in
accompanying the paper a reference When not stopping gradient from the RL loss, mixed the RI updates and the answer network updates by scaling the learning rate for the answer network with coefficient We searched e {0.1, 0.2, 0.5, 1, 2} on the 6 games in the previous section. c 1 worked the best for all methods. More details are in the Appendix.
DeepMind Lab Implementation. We used the same RL module and answer network module as Atari but used a different state representation module to address the partial observability. Specifically, the convolutional layers in the state representation module were followed by dense layer with 512 units and a GRU core
6 with 512 units.
Results. Figure 7a and Figure 7b shows the results for both the stop-gradient and end-to-end architectures on Atari, comparing to two standard numan-normalized score measures (median human- normalized score [1 and mean record-normalinn end-to-end through a combined main task and auxiliary task loss, the performance rGVFs matches substantially exceeds the three baselines. Although the original paper shows that CURL improves agent performance the iafa-effficiett regime (i.e., 100K interactions in Atari), our results indicate that it hurts the performance in the long We conjecture that CURL is held back by failing to capture long-term future in representation learning. Surprisingly. the stop- nop-gradient rGVFs agents outperform the end-to-end A2C baseline, unlike stop-gradien versions of the baseline auxiliary task agents. Figure 7c shows the results for both stop-gradient and end-to-end architectures on 12 DeepMind Lab environments (using mean capped normaizd scores). Again, rGVFs
score [12). When training representations

---

substantially outperforms both auxiliary task baselines, and the siopgraddien version matches the final performance of the end-to-end A2C. Taken together the results from these 61 tasks provide substantial evidence that rGVFs drive the learning of good state representations, outperforming auxiliary tasks with fixed hand-crafted semantics.
6 Conclusion and Future Work
In this work we provided evidence that learning how to make random deep action-conditional predictions can drive the learning of good state representations. We explored rich space of GVFs that can be learned efficiently with TD methods. Our empirical study on the Atari and DeepMind Lab benchmarks shows that learning state representations solely via auxiliary prediction tasks defined by random GVFs outperforms the end-to-end trained A2C baseline. Random GVFs also outperformed pixel control, multi-horizon value prediction, and CURL when being used part of combined loss function with the main RL task.
In this work, the question network was sampled before learning and held fixed during learning. An interesting goal for future research is to find methods that can adapt the question network and discover useful questions during learning. The question networks studied are limited to discrete actions. It is unclear how to condition a prediction on continuous action. Thus another future direction to explore extend action-conditional predictions continuous action spaces.
Acknowledgement
This work was supported by DARPA's L2M program as well as grant from the Open Philanthropy Project to the Center for Human Compatible AL. Any opinions, findings, conclusions, recom- mendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors.
References
[1] Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R. Devon Hjelm. Unsupervised state representation learning in atari. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages
2019.
[2] Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittiieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CORR, 0tn//612.03801, 2016.
[3] Mare G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taīga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle. A geometric perspective on optimal representations for reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garettt editors, Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 4360 4371,, 2019.
[4] Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253-22 279, 2013.
[5] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoode approaches. In Dekai Wu, Marine Carpuat, Xavier Carreras, and Eva Maria Vecchi, editors, Proceedings of SSST@ EMNLP 2014, Eighth Workshop Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, 25 October 2014, pages 103-111. Association for Computational Linguistics, 2014.
[6] Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555, 2014.
10

---

[7] Will Dabney, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Mare G. Bellemare, and David Silver. The value- inprovveeto path: Towards better representations for reinforcement learning. CoRR, hix/2006.022444 2020.
[8] William Fedus, Carles Gelada, Yoshua Bengio, Mare G. Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons. CoRR, abs/ 0666.. 2019.
[9] Karol Gregor, Danilo Jimenez Rezende. Frederic Besse, Yan Wu, Hamza Merzic, and Aäron van den Oord. Shaping belief states with generative environment models for RL. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bus Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 20/9, NeurIPS 2019, December 8-14, 14, 2019, Vancouver, BC, Canada, pages (3475-13487, 2019.
[10] Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo A. Pires, Toby Pohlen, and Rémi Munos. Neural predictive belief representations. CoRR, abs/1811 1.060777 2018.
[11] Zhaohan Daniel Guo, Bernardo Ávila Pires, Bilal Piot, Jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of PMLR, 2020.
Machine Learning Research, pages
[12] Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. CoRR, abs/2010.02193, 2020.
[13] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[14] Bilal Kartal, Pablo Hermandez-Leal, and Matthew E. Taylor. Terminal prediction an auxiliary task for deep reinforcement learning. In Gillian Smith and Levi Lelis, editors, Proceedings of the Fifteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AlIDE 2019, October 8-12, 2019, Atlanta, Georgia, USA, pages 38 18-44. AAAI Press, 2019.
[15] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised repre- sentations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5639 -6500. PMLR, 2020.
[16] Kuang-Huek Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Ser- gio Guadarrama. Predictive information accelerates learning in RL. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 12, 2020, virtual, 2020.
[17] Michael L. Littman, Richard S. Sutton, and Satinder Singh. Predictive representations of state. In Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani, editors, Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], pages 1555 1561, MIT Press, 2001.
[18] Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In Arindam Banerjee and Kenji Fukumizu, editors, The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 1-9. PMLR, 2021.
[19] Bogdan Mazoure, Remi Tachet des Combes, Thang Doan, Philip Bachman, and R. Devon Hjelm. Deep reinforcement and infomax learning. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.

---

[20] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap. Tim Harley. David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICMI. 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928-1937. JMLR.org 2016.
[21] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540))5529-55333 533, 2015.
[22] Matthew Schlegel, Andrew Jacobsen, Zaheer Abbas, Andrew Patterson, Adam White, and Martha White. General value function networks. J. Artif. Intell. Res., 70:497-543, 2021.
[23] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05922 2020.
[24] Satinder Singh, Michael R. James, and Matthew R. Rudary. Predictive state representations: A new theory for modeling dynamical systems. In David Maxwell Chickering and Joseph Y. Halpern, editors, UAI '04, Proceedings of the 20th Conference in Uncertainty in Artificial Intelligence. Banff. Canada, July 7-11, 2004, pages 512-518. AUAI Press, 2004.
[25] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9870-9879, PMLR, 2021.
[26] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[27] Richard S. Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M. Pilarski, Adam White, and Doina Precup. Horde: scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Liz Sonenberg, Peter Stone, Kagan Tumer, and Pinar Yolum, editors, 10th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2011), Taipei, Taiwan, May 2-6, 2011, Volume 1-3, pages 761-768. IFAAMAS, 2011.
[28] Richard S. Sutton, Eddie J. Rafols, and Anna Koop. Temporal abstraction in temporal cempora--diffrenne difference networks. In Advances in Neural Information Processing Systems 18 [Neural Information Processing Systems, NIPS 2005, December 5-8, 2005, Vancouver, British Columbia, Canada], pages 1313-1320, 2005.
[29] Richard S. Sutton and Brian Tanner. Temporal -iffereeee networks. In Advances in Neu- ral Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada), pages 1377-1384, 2004.
[30] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807 007.03748, 2018.
[31] Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L. Lewis, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Bue Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 9306-9317,
2019. 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See abstract and Section I
(b) Did you describe the limitations of your work? [Yes] See Section
6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen-
tary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Figure 3b 3c and
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Appendix.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
We do not release any new assets.
(d) Did you discuss whether and how consent obtained from people whose data you're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "hQ0c5RkJuak", "reviewer": "Reviewer_7T1V", "summary": "This paper introduces a new auxiliary task for learning state representations. This is done by predicting general value functions (GVFs), which are predictions of random features of observations conditioned on sequences of actions. The GVFs of interest can be represented with graphs which they call \"question networks\". The edges in this graph define nodes predicting the value of other nodes in a previous time-step, and many existing prediction tasks in the literature such as reward prediction, multi-horizon value prediction, and termination prediction can be represented using a question network.\n\nThe key contribution here is that predicting the value of nodes in a randomly generated question can provide a strong learning signal for learning representations. Several experiments are provided that show that: (1) representations learned by predicting the values of large random question networks (rGVF) outperform baselines like pixel-control and MHVP; (2) rGVF was the only method that learned good representations even when using a stop-gradient to prevent the RL learning signal from shaping the representation; and (3) several ablation tests were done to show the importance of action conditioning and depth.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: As this paper highlights, the idea of using GVFs as targets for auxiliary tasks is not new. However, for the most part the GVFs used in the past were hand defined. Work has been done on trying to discover useful GVFs, and this paper shows that interestingly, randomly generating large and deep enough GVFs can be enough for learning useful representations even in complex environments and tasks.\n2. Quality: Overall, the quality of this paper is high. I understand that the experiments are mainly done to compare rGVFs with other GVFs. However, there has been significant recent work such as [1, 2, 3] that also shows that auxiliary tasks can accelerate learning or even separate representation learning (with stop-gradients) and still learn strong policies. The one major question that I have after reading this paper is: how does this method of representation learning compare to the SOTA approaches that use contrastive learning? This would be useful to know for practitioners looking to implement one of these algorithms.\n3. Clarity: The clarity of the paper is generally high. I think one addition that would help readers understand the method would be an algorithm box which describes how the all the losses are generated from a batch of trajectories (my assumption is that the TD learning happens using the predictions from the answer network and that there are some stop-gradients in this process, but I want this to be explicitly stated).\nI appreciate the clarity of the writing in the paper, and the experiment section stands out as being especially well-written.\n4. Significance: Designing ways to improve representation learning in RL is an important problem that has received a lot of attention from researchers lately. This paper also gives an important result about the design of GVFs (randomly generating a large number of them can outperform a single hand-designed GVF).\n\n[1] Srinivas, Aravind, et al. “CURL: Contrastive Unsupervised Representations for Reinforcement Learning.” ArXiv:2004.04136 [Cs, Stat], Sept. 2020. [arXiv.org](http://arxiv.org/), [http://arxiv.org/abs/2004.04136](http://arxiv.org/abs/2004.04136).\n\n[2] Stooke, Adam, et al. “Decoupling Representation Learning from Reinforcement Learning.” ArXiv:2009.08319 [Cs, Stat], May 2021. [arXiv.org](http://arxiv.org/), [http://arxiv.org/abs/2009.08319](http://arxiv.org/abs/2009.08319).\n\n[3] Yang, Mengjiao, and Ofir Nachum. “Representation Matters: Offline Pretraining for Sequential Decision Making.” ArXiv:2102.05815 [Cs], Feb. 2021. [arXiv.org](http://arxiv.org/), [http://arxiv.org/abs/2102.05815](http://arxiv.org/abs/2102.05815).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper provides quite a generic analysis of the potential negative societal impact. I would like to see a bit more done here. For instance, while rGVFs can outperform hand designed auxiliary tasks, does this come at a cost in computational efficiency?", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "geRNB5cbE8_", "reviewer": "Reviewer_FnXv", "summary": "Summary\n-------\n\nOwing to the importance of state representation in RL, this paper\nprovides an empirical investigation of random General Value Functions\n(GVFs) for shaping state representation in RL. The paper briefly\noutlines the random generation process for their GVFs before conducting\nextensive experiments on GridWorld, as well as ablation studies in Atari\nand DeepMind Lab environments. The authors find that sufficiently deep\nrandom GVFs are able to provide state representations for policy\nevaluation in a visual gridworld environment. Surprisingly, constructing\nthe state representation as an auxiliary task alone is enough to\noutperform a baseline actor-critic on Atari and DeepMind lab.\n\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Decision\n--------\n\nWhile the paper focuses on a relatively narrow problem regarding the\nbenefits of random GVFs as auxiliary tasks, the extensive investigation\nand surprising findings deserves interest from the RL community. The\npaper is held back by some clarity issues, and as such I am not able to\neasily follow critical sections. Based on this, I am giving this paper a\n5 (\" Marginally below the acceptance threshold\"). I have outlined below\nsome comments to further improve the paper. I am willing to raise my\nscore either after revisions or discussion with the authors / other\nreviewers.\n\nStrengths\n---------\n\n-   The contribution is well motivated and contextualized with respect\n    to the current literature. State representation is an important\n    problem in reinfo\nrcement learning, and conventional methods using\n    state representations that optimize a \"deep RL objective\" are known\n    to produce representations that are not robust. With this in mind,\n    it makes sense to investigate how good random features can be\n    because intuition would leave us to think that random features are\n    not good.\n-   Extensive experiments covering simple environments (GridWorld),\n    pixel control (Atari) and continuous control (Deepmind Lab). Because\n    this is the main contribution of the paper, it is important that\n    this is good. Both the breadth and depth of the experiments are\n    commendable. Although I have some issues with specifics in the\n    experiments, I think they are good overall\n\nWeaknesses\n----------\n\n-   Explanations of general value functions and the random generation\n    process is confusing or not given sufficient reasoning. I have\n    outlined detailed comments below.\n-   Some empirical findings are unclear. I have outlined specific\n    concerns below as well.\n\nDetailed Comments\n-----------------\n\n-   Line 121/123: I thought the targets $z_i$ would be used to update\n    the predictions $y_i$, but they seem separate from $y_i$ and do not\n    appear in the update. Moreover, you say that this is a TD update.\n    The way that the update is written does not look like a TD update.\n    Is it TD update because the update to $y^i_t$ has $y^i_{t+1}$ on the\n    left hand side? This should be explained a bit more.\n-   Line 115/140: The definition of feature here is a feature of the\n    transition and not the observation. This seems like an important\n    characterization and should merit some discussion. You point to\n    Jaderberg et al. (2016) but I am unable to find any justification in\n    that paper either.\n-   Lines 144-165: While I am able to understand the generation process\n    after a couple reads of this paragraph, I think there is a better\n    way to convey such information. The pseudocode in the appendix is a\n    bit more readable and, if possible, should be condensed using the\n    plain english of the paragraph.\n-   Line 152: Is it necessary to have exactly one prediction node per\n    feature node, or is this a design choice?\n-   Line 156: Why are R repeats necessary for each action prediction\n    node?\n-   Line 157: Why must the prediction nodes at layer l need to be\n    connected to both a prediction node in layer 0 and layer l - 1? I\n    don't see why these skip connections are critical for the generation\n    process.\n-   Line 200: Why is the gridworld environment treated as a continuing\n    environment and not episodic? Is the learning curve in Figure 3b/c\n    from one single stream of experience?\n-   Figure 6: Dont the scatter plots for depth/repeat/num.features\n    contradict the findings in the gridworld experiments? The scatter\n    plots suggest that there is no trend in performance and these\n    hyperparameters? Also, why is there such a large difference in\n    performance between similar scatter points (e.g. the two left most\n    points in the depth plot)?\n\nMinor Comments\n--------------\n\n-   Figure 1a: I think the action label for nodes 3 and 4 should be in\n    the circle to denote that they are part of that node. Without the\n    text explaining the figure, it makes information hard to read off. I\n    would suggest that nodes 3 and 4 be relabelled to 3 \\| a and 4 \\| b.\n-   It is unclear whether the visual gridworld is a good environment\n    choice for evaluating GVFs. For one, the feature set is not entirely\n    meaningful as the states / observations provided to the agent is the\n    entire grid as opposed to some local region of the agent. Second,\n    while the environment seems simple, the number of iterations needed\n    to learn anything meaningful is on the order of hundreds of\n    thousands.\n\nPost Rebuttal\n--------------\nIn light of the discussion below, I have increased my score from a 5 -> 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "fySVYyjRC48", "reviewer": "Reviewer_p2uV", "summary": "Summary. Prior works have used auxiliary tasks (such as pixel control, value prediction) to help with representation learning in RL. These auxiliary tasks can be represented in the form of a question network (or TD network) of a generalized value function (GVF). Rather than handcrafting these question networks (of generalized value function), the paper proposes to randomly generate these question networks and use them to help with representation learning in RL. It shows that these randomly generated question networks can lead to better learning than handcrafted auxiliary tasks. Furthermore, the paper shows that only using these randomly generated question networks still leads to a good representation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The use of randomly generated question (or TD) networks to aid with representation learning (in RL) is novel.\n\nQuality: The claims in the paper are backed up by experiments on gridworld, Atari, and deepmind lab domain. It contains proper ablation studies on the gridworld domain which help with the understanding of the method.\n\nClarity: The paper is written clearly. I wish the authors introduced TD networks in a better way by first giving some high-level motivation and examples of special well-known TD networks (like multi horizon value prediction) \n\nSignificance: The paper presents a nice and simple way of learning representations in RL with help of TD networks which outperforms existing methods using hand-crafted auxiliary tasks.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is adequately addressed in the paper", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hr"}, "extra_text": {}}, {"review_id": "VumPFdf6c26", "reviewer": "Reviewer_QPtM", "summary": "This work proposes random General Value Functions (rGVFs) as auxiliary tasks to extract useful representations for reinforcement learning. The target feature and the connections between the prediction nodes are randomized. The proposed method outperforms end-to-end A2C and other baselines with auxiliary training in 49 Atari games and 12 DeepMind Lab environments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1.Originality\n\nTo my knowledge, it is the first paper in which both the target and the connections of the target network are randomized, so I think the originality is sufficient. The related work is adequately cited.\n\nOne missing related work that introduces compositional GVFs with action conditions and discounts.\n\n[1] Andrew Patterson and Matthew Schlegel, A Comparison of General Value Functions and Temporal-Difference Networks, AAMAS 2016.\n\n\n2. Quality\n\nAs this is a study without theoretical results, the quality in terms of the experimental results was mainly reviewed. I have a very mixed feeling about the experimental results. The empirical gain in Figure 7 is impressive, but the reasons for such improvement are not well discussed.\n\n2.1) Significant empirical gain\n\nThe performance gain of rGVFs over the end-to-end A2C in Figure 7 is impressive. Especially it is a very valuable result that the stop-gradient version of rGVFs outperforms the Vanilla A2C in Atari. This is a meaningful result because the experiment was conducted on a large number of tasks (49 in Atari and 12 in DMLab) without cherry-picking specific tasks that are advantageous to their method.\n\nA2C is a fairly weak baseline and its performance is far below the current state of the art (e.g., MuZero and R2D2). However, since rGVFs are orthogonal to the RL training, it is expected to bring similar performance gain when combined with SOTA methods. The strength of this paper will be emphasized even more by adding such empirical results.\n\n2.2) Lack of intuitive explanation and empirical analysis.\n\nThe greatest weakness of this work is that it only lists the empirical gains and lacks a detailed analysis of why such gain is achieved.\n\nThe illustrative result in Figure 3b is appreciated. It makes sense that the random $\\theta^{repr}$ suffers from high MSE since the value network has only one hidden layer and increasing the depth of the tree decreases the MSE as low as that of end-to-end. However, it does not provide intuition on why rGVFs should outperform end-to-end methods (as in Figure 7). This work will be largely strengthened with empirical analysis on what is learned to improve performance. Here are a few ideas that come to mind:\n\n- Analysis of successful random graphs: since the graphs are randomly generated, It seems that certain types of random graphs may be more effective than others (maybe graphs similar to TD are helpful? or semantics as 'touch' in Figure 2?). Maybe the improvement is from the restricted structures of the graph. Although the authors name their work as 'random' GVF, it seems that the structure is largely restricted (2 connections for each prediction nodes, self-loop in layer 0, each layer containing the same number of nodes based on R and D, weighted sum with weight 1, ..etc.)\n\n- Analysis of successful tasks: the effectiveness of rGVFs may vary depending on the properties of target tasks as the authors state in line 283. Also in Figure 2 in Appendix, it seems that the amount of gain varies from game to game.\n\nI'm willing to raise my score if such analysis is provided.\n\n\n3. Clarity\n\nThis work is generally well organized and the main claim of the work is well delivered. The source code in the supplementary material is superbly organized and provides enough information to reproduce the results. However, I think the overall clarity can be largely improved, and there seem to be some conflicts between the contents of the manuscript.\n\n- Section 3.1 and 3.2 introduce the core ideas of this study but they are somewhat complicated to understand. Maybe providing a figure with all the notations ($f^k, y^i, n_f, n_p, z^{t}_{i}, W, R, D$) will make it much easier to understand the overall structure. Or simply providing example update equations for figure 1(a) with time index would be helpful.\n- There are some sentences too long to understand (e.g. line 144). I'd appreciate it if the authors could break up such sentences.\n- It seems that the input for the representation network is just $O_t$, not $O_0, A_0, \\ldots, O_t$ if the environment is not POMDP. It would be great if there are corresponding explanations at the beginning of the paragraph in line 109 and in Figure 1b.\n- Adding the vanilla A2C result in Figure 5 will be appreciated (plotting the results of Figure 2 in appendix)\n- The star symbol in Figure 3a should be moved one cell to the right to match the results in Figure 4?\n- Line 263 describes a depth-1 tree and refers Figure 2b, but it is confusing since the tree in Figure 2b seems to be a depth-2 tree without a self-loop.\n\n**(Post rebuttal update)** I appreciate the detailed response and additional experimental results from the authors. It seems that the ablation study in the common response addresses my main concern on why rGVFs stop-gradient should outperform end-to-end A2C.  Therefore, I am willing to raise my rating from 5 to 6.\n\n4. Significance\n\n- The empirical result is significant and as the authors mention in the conclusion section, there is much room for improvement to generate more general value functions.\n- Although the authors append rGVFs to A2C only, it seems to bring improvements when combined with other SOTA RL methods.\n- The authors evaluate their work on Atari and DMLab environments, but I think rGVFs can be extended to solve generalization problems of RL since it may help to learn task-agnostic representations (e.g. pretrain rGVFs in source tasks and use the learned features to finetune different target tasks.) \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is nice that the authors addressed the limitation of rGVFs to discrete actions. This limitation may be problematic in certain continuous control tasks. But referring to Figure 5, it seems that the performance of rGVFs-no-actions is on par with that of rGVFs except for BeamRider. It would be interesting to evaluate rGVFs-no-actions in continuous control tasks such as MuJoCo.\n\n Another limitation not mentioned by the authors is the computation load required. Since rGVFs perform auxiliary training with deep multiple graphs, it seems to bring a great increment in computation, especially for tasks with large action space (4 to18 for 49 Atari games). If I'm not mistaken the amount of computation is not described in the appendix, contrary to what is indicated in checklist 3(d).\n\nThe last limitation is the expressibility of the question network in complex state space. In Figure 7(a) and 7(b), the scores of rGVFs with and without stop-gradient are nearly the same. However, in DMLab experiment, rGVFs with stop-gradient are on par with end-to-end A2C. I think this gap is from the fact that DMLab is a POMDP, therefore it is more challenging to obtain rich state representation (from the GRU with the entire past trajectory) just using the random targets. There is a concern that the representation learned by rGVFs will not be effective in situations in more complex state space.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a surprising result that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted value prediction and pixel control auxiliary tasks in both Atari and DeepMind Lab tasks. All reviewers unanimously vote to accept the paper which I agree with. 

In the camera-ready version, I suggest the authors to address the reviewer's comments. It would also be nice to see how the performance gap changes as a function of how of powerful the on-policy learner is. E.g., with A3C / PPO, do we have a similar performance gap? This will help verify whether the same advantage can be gained by improving the policy gradients or not. 
Decision: Accept (Poster)