Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work addresses the multi-prediction problem in MDPs, with key focus on how to generate and adapt the behavior to optimize auxiliary predictions in the absence of task rewards.  They show that the rewards for the behavior have inherent non-stationarity. The proposed method approximates successor features (via expected SARSA and Tree Backup) and then uses them to learn value predictions (via regression updates). This specific separation allows for a faster convergence rate as proved by the authors.
main_review: **Strengths**:  

The work addresses an important problem of learning how to generate behavior such that multiple predictions can be learned, which remains under-explored in the literature.  The approach being fully online adds an additional challenge to tackle under non-stationarity - this is an important contribution. With theoretical and practical considerations, the proposed method seems an effective strategy to learn multiple predictions. This is a very valuable contribution as a lot of the existing literature has explored learning multiple predictions without paying much heed towards learning the behavior policy itself to be able to result in meaningful multiple predictions. While the paper can be significantly improved in writing, the contributions outweigh its flaws.

**Weaknesses**:
* One of the primary limitations of this work is that the cumulants are hand-specified and not being learned or discovered. There is no discussion about this. Any insights would be useful to add. For example how does the work on discovering useful cumulants might tie to the ideas presented in this paper for overcoming user designed cumulant equations?
* The phrasing of the paper title is misleading. The word continual learning is typically used in the context of lifelong learning or never ending learning (also referred to as continual learning). But what the authors really imply here is online learning (e.g “continually interacts with the environment”), which warrants the current title misplaced and not suitable for this work. I would recommend updating it to reflect more clearly what the paper addresses. 
 

**Empirical Analysis**: 
The empirical analysis provided support for the claims. The experiments cover a range of tasks. However, I have the following questions:  

* In the current pipeline, the value estimates of the prediction have a huge dependence on the accurate learning of the successor features. What happens if the SF is not well learned? Would the method still work?
* How does the change in weights impact the scalability of the algorithm? A potential failure case is that while the weights have changed a lot but nothing meaningful has been learned. How would one interpret this in the context of the proposed approach?
* Besides, it would be interesting and useful to show the failure cases for the proposed ideas in general. Please add additional analysis if any towards this.
* An interesting baseline would be an agent such as Horde where the behaviour policy is not being learned or adapted smartly. Would it be a good idea to add this? Any insights from the authors on this would be appreciated. 
 

**Writing and Presentation**:  
* The paper is very well motivated. 
* The experiments section is less clear as compared to the other sections. It might be useful to revisit it to ensure better understanding.
* The checklist has many links missing which show as ?
* L359- the above results are obtained by learning online.
* L375 fourth → forth
* I recommend at least one pass of proofread to check the paper for typos and minor writing errors. 

limitations_and_societal_impact: The authors do not discuss limitations. I recommended adding this with the conclusion discussing failure modes and future work. As it stands, nothing along these lines have been covered in the manuscript.   

I did not see a broader impact statement, although the work is abstract enough to have any direct concerns.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers the setting where an RL agent has to learn to make auxiliary predictions, and captures each prediction task as an RL task. In particular, the goal is to maximize the learning progress for each prediction task. Specifically, this is measured by the l1-norm of the change in weights and introduced as an intrinsic reward.

However, this multi-prediction problem is difficult because of multiple sources of non-stationarity, i.e., from the changing rewards (which shrink as the agent makes better predictions) and from the changing agent behavior. To address this non-stationarity, the successor features framework is leveraged. That is, they can use learned successor features to efficiently track the non-stationary rewards and to correct state distribution changes by importance sampling.
main_review: Originality:
- The use of successor features to track and handle non-stationarity is novel. Applying this to efficiently learn multiple auxiliary tasks is also new.

- The introduction also provides a good overview of the related work in the literature, and describes how the ideas in this paper relate to the prior work.

Quality:
- The paper provides a theoretical analysis of the sample efficiency and shows that it is better compared to standard algorithms.

- The algorithm is evaluated in a simple maze environment and in the Mountain Car task, which are pretty simple RL tasks. Learning multiple auxiliary tasks in simpler environments may be more inefficient than solving a main task, when given one. I’m curious how well this approach scales to more complex environments.

- The number of auxiliary tasks studied in the experiments is also quite small (4 in the maze setting and 2 in Mountain Car). It’d be nice to study how a larger number of auxiliary tasks impacts learning.

Clarity:
- Using the change in the weights as the RL reward seems to introduce an unnecessary source of non-stationarity. In the settings studied in the experiments, the auxiliary tasks involve reaching specific goal states. Could the intrinsic reward instead be defined in terms of reaching these goals which would then give stationary reward functions?

- The different parts of Section 4 currently read a bit disconnected and could be better organized. Including a unified algorithm box would be helpful. Also, are the weights of the GVF learners the same weights as in Alg. 2?

- There is a lot of notation and terms that get abbreviated, which gets difficult to keep track of. A table that summarizes all of the notation and abbreviations would be helpful.

Significance:
- Learning a set of auxiliary tasks in the absence of external rewards ideally allows for efficient transfer when an external reward is actually specified, and is hence an important problem setting to study. 

- The successor features framework allows for transfer between these auxiliary reward functions, which are assumed to be linear in some features. Is this a limiting assumption when we might want to consider more complex auxiliary tasks that do not share a common feature set?
limitations_and_societal_impact: Yes. See main review for other limitations.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a framework for continually learning multiple generalized value functions, where each value function is learned off-policy and the behaviour policy learns to maximize the learning progress. The paper identifies the non-stationarity of the action value function during the process of maximizing learning progress and proposes an algorithm that separately learns the immediate reward and the successor feature. The experimental results show that the proposed algorithm significantly improves sample efficiency and perfroms well with the use of replay.
main_review: ### Originality

The idea of using successor feature for MDPs with changing rewards are not new (Barreto et al.18'), but its application in learning auxiliary tasks is novel.

### Quality

The motivation of addressing the non-stationary learning problem in using auxiliary tasks for exploration using successor feature (SF) is very convincing. The paper also shows theoretically that given accurate SF, better convergence can be achieved. The assumption of accurate SF fits the problem setup where rewards are stationary while the dynamics are static. The empirical experiments are also convincing to me.

### Clarity

The paper is well written and easy to follow.

typo: line 277, equation 3 -> equation 2
limitations_and_societal_impact: The formulation of a GVF question was confusing to me in the first read as it is described as a goal-conditioned value function in Sutton et al. 11'. It seems like the formulation in this paper follows Veeriah et al. 19'. it would be better to clarify this point.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the continual RL setting in which an agent behaves according to a policy designed to improve predictions on a number of auxiliary tasks. It proposes an algorithm based on successor features that facilitates tracking under non-stationary rewards, demonstrating both empirical and theoretical benefits of this approach. 
main_review: This paper studies an important problem, namely that of learning about the environment without external rewards, in a very challenging setting, namely continual RL with inherent non-stationary for both the prediction learners and the behavior learner. The paper is well written, it introduces a new problem formulation, and proposes a new method, which is thoroughly evaluated empirically and analyzed from a theoretical perspective. I think this paper could be of value for the community and open up multiple directions for future work. 

I particularly appreciated that the authors report results over 30 different runs of the models, which increases confidence in the drawn conclusions. I also liked the fact that the experiments were designed in simple environments, where it is easier to understand the strengths and limitations of these methods. 

One limitation of this work is that the approach seems quite similar to prior work which uses successor features for transfer to new tasks (e.g. Barreto et al. 2017, Barreto et al. 2018 Borsa et al. 2018, Ma et al. 2020). Hence, the algorithmic contribution is not particularly novel, but I don't think this should prevent acceptance given that the empirical evaluation is quite thorough, the problem setting is novel and important, and the results look promising. However, I do think the authors should discuss in greater detail how the approach differs from prior work. 

Another potential limitation of this work is the reliance upon the choice of policies / tasks to learn. I think it would be valuable to analyze or discuss how the algorithm depends on this choice and how one should choose the set of auxiliary tasks or policies. What if one doesn't have access to such tasks or policies -- how could these be discovered so that they are (most) useful for learning about the environment or downstream tasks without prior knowledge? How could one combine this with skill discovery or other related research directions?

Given all the above, I recommend acceptance for this paper, but hope the authors can take into account the feedback to further improve the paper. 
limitations_and_societal_impact: None
review_title: null

Meta-review: The reviewers have come to a strong consensus for acceptance of this work. Its strong points include novelty, where the important combination of multi-prediction and non-stationarity is well motivated, as well as pleasing theory and experiments to address the same. 

I would ask the authors to carefully consider the suggested writing improvements from GBFf and suggestions for clarity from NevW. I would also like to see additionally comentary about the number of tasks and the potential to discover "useful" tasks/skills in the context of your work (these are not things I expect a solution to today, but something where guidance will help the authors bound to study the topic in the future).
Decision: Accept (Poster)