Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary. Prior works have used auxiliary tasks (such as pixel control, value prediction) to help with representation learning in RL. These auxiliary tasks can be represented in the form of a question network (or TD network) of a generalized value function (GVF). Rather than handcrafting these question networks (of generalized value function), the paper proposes to randomly generate these question networks and use them to help with representation learning in RL. It shows that these randomly generated question networks can lead to better learning than handcrafted auxiliary tasks. Furthermore, the paper shows that only using these randomly generated question networks still leads to a good representation. 
main_review: Originality: The use of randomly generated question (or TD) networks to aid with representation learning (in RL) is novel.

Quality: The claims in the paper are backed up by experiments on gridworld, Atari, and deepmind lab domain. It contains proper ablation studies on the gridworld domain which help with the understanding of the method.

Clarity: The paper is written clearly. I wish the authors introduced TD networks in a better way by first giving some high-level motivation and examples of special well-known TD networks (like multi horizon value prediction) 

Significance: The paper presents a nice and simple way of learning representations in RL with help of TD networks which outperforms existing methods using hand-crafted auxiliary tasks.  

limitations_and_societal_impact: It is adequately addressed in the paper
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary
-------

Owing to the importance of state representation in RL, this paper
provides an empirical investigation of random General Value Functions
(GVFs) for shaping state representation in RL. The paper briefly
outlines the random generation process for their GVFs before conducting
extensive experiments on GridWorld, as well as ablation studies in Atari
and DeepMind Lab environments. The authors find that sufficiently deep
random GVFs are able to provide state representations for policy
evaluation in a visual gridworld environment. Surprisingly, constructing
the state representation as an auxiliary task alone is enough to
outperform a baseline actor-critic on Atari and DeepMind lab.



main_review: Decision
--------

While the paper focuses on a relatively narrow problem regarding the
benefits of random GVFs as auxiliary tasks, the extensive investigation
and surprising findings deserves interest from the RL community. The
paper is held back by some clarity issues, and as such I am not able to
easily follow critical sections. Based on this, I am giving this paper a
5 (" Marginally below the acceptance threshold"). I have outlined below
some comments to further improve the paper. I am willing to raise my
score either after revisions or discussion with the authors / other
reviewers.

Strengths
---------

-   The contribution is well motivated and contextualized with respect
    to the current literature. State representation is an important
    problem in reinfo
rcement learning, and conventional methods using
    state representations that optimize a "deep RL objective" are known
    to produce representations that are not robust. With this in mind,
    it makes sense to investigate how good random features can be
    because intuition would leave us to think that random features are
    not good.
-   Extensive experiments covering simple environments (GridWorld),
    pixel control (Atari) and continuous control (Deepmind Lab). Because
    this is the main contribution of the paper, it is important that
    this is good. Both the breadth and depth of the experiments are
    commendable. Although I have some issues with specifics in the
    experiments, I think they are good overall

Weaknesses
----------

-   Explanations of general value functions and the random generation
    process is confusing or not given sufficient reasoning. I have
    outlined detailed comments below.
-   Some empirical findings are unclear. I have outlined specific
    concerns below as well.

Detailed Comments
-----------------

-   Line 121/123: I thought the targets $z_i$ would be used to update
    the predictions $y_i$, but they seem separate from $y_i$ and do not
    appear in the update. Moreover, you say that this is a TD update.
    The way that the update is written does not look like a TD update.
    Is it TD update because the update to $y^i_t$ has $y^i_{t+1}$ on the
    left hand side? This should be explained a bit more.
-   Line 115/140: The definition of feature here is a feature of the
    transition and not the observation. This seems like an important
    characterization and should merit some discussion. You point to
    Jaderberg et al. (2016) but I am unable to find any justification in
    that paper either.
-   Lines 144-165: While I am able to understand the generation process
    after a couple reads of this paragraph, I think there is a better
    way to convey such information. The pseudocode in the appendix is a
    bit more readable and, if possible, should be condensed using the
    plain english of the paragraph.
-   Line 152: Is it necessary to have exactly one prediction node per
    feature node, or is this a design choice?
-   Line 156: Why are R repeats necessary for each action prediction
    node?
-   Line 157: Why must the prediction nodes at layer l need to be
    connected to both a prediction node in layer 0 and layer l - 1? I
    don't see why these skip connections are critical for the generation
    process.
-   Line 200: Why is the gridworld environment treated as a continuing
    environment and not episodic? Is the learning curve in Figure 3b/c
    from one single stream of experience?
-   Figure 6: Dont the scatter plots for depth/repeat/num.features
    contradict the findings in the gridworld experiments? The scatter
    plots suggest that there is no trend in performance and these
    hyperparameters? Also, why is there such a large difference in
    performance between similar scatter points (e.g. the two left most
    points in the depth plot)?

Minor Comments
--------------

-   Figure 1a: I think the action label for nodes 3 and 4 should be in
    the circle to denote that they are part of that node. Without the
    text explaining the figure, it makes information hard to read off. I
    would suggest that nodes 3 and 4 be relabelled to 3 \| a and 4 \| b.
-   It is unclear whether the visual gridworld is a good environment
    choice for evaluating GVFs. For one, the feature set is not entirely
    meaningful as the states / observations provided to the agent is the
    entire grid as opposed to some local region of the agent. Second,
    while the environment seems simple, the number of iterations needed
    to learn anything meaningful is on the order of hundreds of
    thousands.

Post Rebuttal
--------------
In light of the discussion below, I have increased my score from a 5 -> 7.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a new auxiliary task for learning state representations. This is done by predicting general value functions (GVFs), which are predictions of random features of observations conditioned on sequences of actions. The GVFs of interest can be represented with graphs which they call "question networks". The edges in this graph define nodes predicting the value of other nodes in a previous time-step, and many existing prediction tasks in the literature such as reward prediction, multi-horizon value prediction, and termination prediction can be represented using a question network.

The key contribution here is that predicting the value of nodes in a randomly generated question can provide a strong learning signal for learning representations. Several experiments are provided that show that: (1) representations learned by predicting the values of large random question networks (rGVF) outperform baselines like pixel-control and MHVP; (2) rGVF was the only method that learned good representations even when using a stop-gradient to prevent the RL learning signal from shaping the representation; and (3) several ablation tests were done to show the importance of action conditioning and depth.
main_review: 1. Originality: As this paper highlights, the idea of using GVFs as targets for auxiliary tasks is not new. However, for the most part the GVFs used in the past were hand defined. Work has been done on trying to discover useful GVFs, and this paper shows that interestingly, randomly generating large and deep enough GVFs can be enough for learning useful representations even in complex environments and tasks.
2. Quality: Overall, the quality of this paper is high. I understand that the experiments are mainly done to compare rGVFs with other GVFs. However, there has been significant recent work such as [1, 2, 3] that also shows that auxiliary tasks can accelerate learning or even separate representation learning (with stop-gradients) and still learn strong policies. The one major question that I have after reading this paper is: how does this method of representation learning compare to the SOTA approaches that use contrastive learning? This would be useful to know for practitioners looking to implement one of these algorithms.
3. Clarity: The clarity of the paper is generally high. I think one addition that would help readers understand the method would be an algorithm box which describes how the all the losses are generated from a batch of trajectories (my assumption is that the TD learning happens using the predictions from the answer network and that there are some stop-gradients in this process, but I want this to be explicitly stated).
I appreciate the clarity of the writing in the paper, and the experiment section stands out as being especially well-written.
4. Significance: Designing ways to improve representation learning in RL is an important problem that has received a lot of attention from researchers lately. This paper also gives an important result about the design of GVFs (randomly generating a large number of them can outperform a single hand-designed GVF).

[1] Srinivas, Aravind, et al. “CURL: Contrastive Unsupervised Representations for Reinforcement Learning.” ArXiv:2004.04136 [Cs, Stat], Sept. 2020. [arXiv.org](http://arxiv.org/), [http://arxiv.org/abs/2004.04136](http://arxiv.org/abs/2004.04136).

[2] Stooke, Adam, et al. “Decoupling Representation Learning from Reinforcement Learning.” ArXiv:2009.08319 [Cs, Stat], May 2021. [arXiv.org](http://arxiv.org/), [http://arxiv.org/abs/2009.08319](http://arxiv.org/abs/2009.08319).

[3] Yang, Mengjiao, and Ofir Nachum. “Representation Matters: Offline Pretraining for Sequential Decision Making.” ArXiv:2102.05815 [Cs], Feb. 2021. [arXiv.org](http://arxiv.org/), [http://arxiv.org/abs/2102.05815](http://arxiv.org/abs/2102.05815).
limitations_and_societal_impact: The paper provides quite a generic analysis of the potential negative societal impact. I would like to see a bit more done here. For instance, while rGVFs can outperform hand designed auxiliary tasks, does this come at a cost in computational efficiency?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes random General Value Functions (rGVFs) as auxiliary tasks to extract useful representations for reinforcement learning. The target feature and the connections between the prediction nodes are randomized. The proposed method outperforms end-to-end A2C and other baselines with auxiliary training in 49 Atari games and 12 DeepMind Lab environments.
main_review: 
1.Originality

To my knowledge, it is the first paper in which both the target and the connections of the target network are randomized, so I think the originality is sufficient. The related work is adequately cited.

One missing related work that introduces compositional GVFs with action conditions and discounts.

[1] Andrew Patterson and Matthew Schlegel, A Comparison of General Value Functions and Temporal-Difference Networks, AAMAS 2016.


2. Quality

As this is a study without theoretical results, the quality in terms of the experimental results was mainly reviewed. I have a very mixed feeling about the experimental results. The empirical gain in Figure 7 is impressive, but the reasons for such improvement are not well discussed.

2.1) Significant empirical gain

The performance gain of rGVFs over the end-to-end A2C in Figure 7 is impressive. Especially it is a very valuable result that the stop-gradient version of rGVFs outperforms the Vanilla A2C in Atari. This is a meaningful result because the experiment was conducted on a large number of tasks (49 in Atari and 12 in DMLab) without cherry-picking specific tasks that are advantageous to their method.

A2C is a fairly weak baseline and its performance is far below the current state of the art (e.g., MuZero and R2D2). However, since rGVFs are orthogonal to the RL training, it is expected to bring similar performance gain when combined with SOTA methods. The strength of this paper will be emphasized even more by adding such empirical results.

2.2) Lack of intuitive explanation and empirical analysis.

The greatest weakness of this work is that it only lists the empirical gains and lacks a detailed analysis of why such gain is achieved.

The illustrative result in Figure 3b is appreciated. It makes sense that the random $\theta^{repr}$ suffers from high MSE since the value network has only one hidden layer and increasing the depth of the tree decreases the MSE as low as that of end-to-end. However, it does not provide intuition on why rGVFs should outperform end-to-end methods (as in Figure 7). This work will be largely strengthened with empirical analysis on what is learned to improve performance. Here are a few ideas that come to mind:

- Analysis of successful random graphs: since the graphs are randomly generated, It seems that certain types of random graphs may be more effective than others (maybe graphs similar to TD are helpful? or semantics as 'touch' in Figure 2?). Maybe the improvement is from the restricted structures of the graph. Although the authors name their work as 'random' GVF, it seems that the structure is largely restricted (2 connections for each prediction nodes, self-loop in layer 0, each layer containing the same number of nodes based on R and D, weighted sum with weight 1, ..etc.)

- Analysis of successful tasks: the effectiveness of rGVFs may vary depending on the properties of target tasks as the authors state in line 283. Also in Figure 2 in Appendix, it seems that the amount of gain varies from game to game.

I'm willing to raise my score if such analysis is provided.


3. Clarity

This work is generally well organized and the main claim of the work is well delivered. The source code in the supplementary material is superbly organized and provides enough information to reproduce the results. However, I think the overall clarity can be largely improved, and there seem to be some conflicts between the contents of the manuscript.

- Section 3.1 and 3.2 introduce the core ideas of this study but they are somewhat complicated to understand. Maybe providing a figure with all the notations ($f^k, y^i, n_f, n_p, z^{t}_{i}, W, R, D$) will make it much easier to understand the overall structure. Or simply providing example update equations for figure 1(a) with time index would be helpful.
- There are some sentences too long to understand (e.g. line 144). I'd appreciate it if the authors could break up such sentences.
- It seems that the input for the representation network is just $O_t$, not $O_0, A_0, \ldots, O_t$ if the environment is not POMDP. It would be great if there are corresponding explanations at the beginning of the paragraph in line 109 and in Figure 1b.
- Adding the vanilla A2C result in Figure 5 will be appreciated (plotting the results of Figure 2 in appendix)
- The star symbol in Figure 3a should be moved one cell to the right to match the results in Figure 4?
- Line 263 describes a depth-1 tree and refers Figure 2b, but it is confusing since the tree in Figure 2b seems to be a depth-2 tree without a self-loop.

**(Post rebuttal update)** I appreciate the detailed response and additional experimental results from the authors. It seems that the ablation study in the common response addresses my main concern on why rGVFs stop-gradient should outperform end-to-end A2C.  Therefore, I am willing to raise my rating from 5 to 6.

4. Significance

- The empirical result is significant and as the authors mention in the conclusion section, there is much room for improvement to generate more general value functions.
- Although the authors append rGVFs to A2C only, it seems to bring improvements when combined with other SOTA RL methods.
- The authors evaluate their work on Atari and DMLab environments, but I think rGVFs can be extended to solve generalization problems of RL since it may help to learn task-agnostic representations (e.g. pretrain rGVFs in source tasks and use the learned features to finetune different target tasks.) 



limitations_and_societal_impact: It is nice that the authors addressed the limitation of rGVFs to discrete actions. This limitation may be problematic in certain continuous control tasks. But referring to Figure 5, it seems that the performance of rGVFs-no-actions is on par with that of rGVFs except for BeamRider. It would be interesting to evaluate rGVFs-no-actions in continuous control tasks such as MuJoCo.

 Another limitation not mentioned by the authors is the computation load required. Since rGVFs perform auxiliary training with deep multiple graphs, it seems to bring a great increment in computation, especially for tasks with large action space (4 to18 for 49 Atari games). If I'm not mistaken the amount of computation is not described in the appendix, contrary to what is indicated in checklist 3(d).

The last limitation is the expressibility of the question network in complex state space. In Figure 7(a) and 7(b), the scores of rGVFs with and without stop-gradient are nearly the same. However, in DMLab experiment, rGVFs with stop-gradient are on par with end-to-end A2C. I think this gap is from the fact that DMLab is a POMDP, therefore it is more challenging to obtain rich state representation (from the GRU with the entire past trajectory) just using the random targets. There is a concern that the representation learned by rGVFs will not be effective in situations in more complex state space.
review_title: null

Meta-review: The paper presents a surprising result that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted value prediction and pixel control auxiliary tasks in both Atari and DeepMind Lab tasks. All reviewers unanimously vote to accept the paper which I agree with. 

In the camera-ready version, I suggest the authors to address the reviewer's comments. It would also be nice to see how the performance gap changes as a function of how of powerful the on-policy learner is. E.g., with A3C / PPO, do we have a similar performance gap? This will help verify whether the same advantage can be gained by improving the policy gradients or not. 
Decision: Accept (Poster)