Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper focuses on reinforcement learning problems with known successor features and rewards expressible as their linear combination. Building on recent research, it presents a concept of independent features and independent policies and way to construct them. Theoretically it shows that the set of independent policies and their combination with GPE & GPI is enough to solve any induced task. Experimentally, the authors verify the theory and compare to existing approaches to create policy sets, outperforming all. They also provide a set of relevant questions and answers, supported by separate experiments. Finally, they perform experiments in problems without the linear combination assumption and lifelong RL setting, with positive results.
main_review: This is a strong paper, methodically and clearly presenting its concepts. It is well written, interesting to read, presenting theory before supporting it with experiments. It addresses the main questions with separate experiments. Based on the theory and results, the presented algorithm is better than prior-art in every case. The experiments beyond the original assumptions show that the new algorithm is valuable and interesting for general RL community -- i.e., the last two experiments (reward cannot be expressed as linear combination of features / lifelong RL) show that it outperforms DQN. It would be interesting to see the algorithm also in another domain than 2D item collection (and especially without the linear-combination assumption), but I deem the scope of the paper already well sufficient.

Typos:
p. 3 - it leaveS open the question
p. 3 - clear by the end OF this section
p. 3 - the problem weE want to tackle
p. 7 - tasks that do noT satisfy
p. 7 - test whether IF learning
summary_of_the_review: Well written and interesting paper with strong theory and results, well conducted experiments. Recommending acceptation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper extends the successor features framework to answer the following question: which policies should we learn and store so that, when presented with a new task, we achieve the best performance possible? The paper defines the notion of independent policies (forming a kind of basis over policy space), which can then be combined to solve new tasks (whose rewards are expressible as a linear combination of features) immediately. Experimental results support the theoretical results and show that it is best to learn these independent policies if we wish to maximise performance on downstream tasks.
main_review: This paper tackles an interesting and important question that is clearly critical for developing lifelong agents: which policies or value functions should an agent store in its "library" for later use?  The approach taken here is to learn a set of basis policies that are provably optimal (by optimal, we mean that there are no other policies that will result in improved performance after GPI) on downstream tasks within the successor features framework. Given the recent success and adoption of the framework, the results here will likely be of great interest to the community.

I found the paper overall to be extremely well-written and easy to follow (albeit with some details left out, which I will discuss later), and I think that the overall idea is conceptually the correct one concerning the specific question and under the assumptions made. Both theoretical and experimental results are provided, which helps bolster the paper's claims. However, I have certain reservations (and clarifying questions) about some of the results and positioning of the paper, to which I would appreciate a response. 

**On related work:**

The big downside for me is the lack of related work, which makes it difficult to determine the novelty and answer the question of why one should pick this method over existing ones. The paper builds on the successor features framework and so, unsurprisingly, there are numerous references and comparisons to this line of prior work.  However, there is insufficient coverage of other related work outside of this, and particularly on approaches that set out to do precisely the same thing studied here.

Of course, the literature regarding the transfer of policies and value functions for multitask RL is large and it is not possible to survey the entire field in the space of a conference paper. However, several approaches accomplish the same kinds of things in this paper and the paper would be made much stronger by distinguishing its contributions from past approaches. For instance, Abel et al (2018) provide theoretical results (and a simple approach that maximises over the value functions) about how best to reuse learned policies to solve new tasks, while van Niekerk et al (2019) study composition in a similar setup (see Theorem 2 in particular). Closely related to this is also Tirinzoni et al (2018).  Fernandez and Veloso (2006) build on Thrun and Schwartz (1995) to determine how to reuse past policies to accelerate learning on new tasks. 

The most important piece of related work is the recently published paper by Nemecek and Parr (2021). Their approach tackles the exact same problem, but they are able to iteratively build a policy cache without having to rely on the notion of independent features/policies. One of the most important aspects of their work is that they have motivating experiments as to why you would only want to store certain policies. In the experimental setup here, there are very few tasks, and so why not just store all the policies? In their work, they have 1000+ tasks, and so it is clearly infeasible to store every single policy learned. This motivation is in general missing from the paper. Please comment on the difference between this work and theirs.

It is worth including either direct empirical comparisons, or brief discussions of the differences to this work. Older approaches such as PolicyBlocks (Pickett and Barto 2002) may also be of relevance, since they focus on collecting a cache of policies that are used to inform future learning.

**On theoretical results:**

The theoretical results speak to the idea that, given a set of policies, is a subset we can choose to keep in our "library" to maximise the outcome of doing generalised policy improvement. While this is true, it would have been even nicer to see results that say something about how close to optimal we can get with these policies. For example, closely-related work by Abel et al (2018) provides PAC bounds when maximising a set of previously-learned action value functions, and I can't help but think that a similar kind of result here would be much more impactful. 

The definition of independent features seems quite restrictive to me, and from the intuition provided, I can only see it being useful for goal or subgoal-reaching tasks, where the features are 0 everywhere until a subgoal is reached. I'm also not 100% sure I understand the definition of an independent policy. When it says "a policy is induced by each feature", does this mean a policy that "achieves" a feature by reaching a state where its value is non-zero? If so, then I take it to mean that independent policies are policies that cause different features to be set to non-zero (and all others remain 0). Thus, to use the given domain as an example, two independent policies would be a policy that collects triangles, and one that collects diamonds. 

If so then I wonder how useful Lemma 1 and the results in Figure 3 are, since the expected features are essentially the Q-value function for solving a task with rewards of 0 everywhere and C when a goal is achieved. These results basically say: the policy for collecting diamonds has a high expected return for collecting diamonds and 0 for collecting triangles (and vice versa), which does not seem particularly surprising.

Furthermore, if this is the case, then the setup seems almost identical to Nangue Tasse et al (2020a) who also introduce the concept of a "policy basis". While that work does not deal with preferences, it seems like it is extremely related, and even if not used as a baseline, it would be helpful to describe how the notion of basis policies differs from that here.  

**On empirical results:**

The empirical experiments are well aligned with the theory and the story being told, but I wonder if they do not go far enough. I really liked Figure 2 as an initial start, but there are a few assumptions baked into that result and it would be better if it could be shown how to remove them. In order of importance, these are:
  1. The agent is essentially told which policies are independent, since these weight vectors are provided up front. However, in general the agent does not get to control the order in which it sees the task, and so a better setup would be to have tasks randomly sampled from a distribution, and require the agent to determine whether a policy is independent. This way, the library can be grown in an informed manner, instead of being provided to the agent directly. 
  2. The above experiment could also be combined with the regression procedure used in Question 7 to infer the weights. Then the agent would essentially be receiving no information, and would have to learn everything directly from data.  
  3. The lifelong experiment doesn't have any "lifelong-oriented" baselines. I think a much stronger experiment would be to test it against MAXQ (Abel et al 2018), Nangue Tasse (2020b) or Nemecek and Parr (2021).
  4. The theory only holds for deterministic cases, but it would be interesting to test empirically in the stochastic setting, which could easily be done by adding a "slip" probability to actions

I also worry a bit about the results in Figure 5. While DQN was used to show what optimal performance looks like, it is remarkably similar to the method presented here. This is despite the fact that DQN must learn everything from scratch, whereas the other approaches only need to learn a mapping from states to weights and then combine them with the pretrained independent (and non-independent) policies. I would have expected the latter approach to completely dominate DQN because of its pretraining, and the fact that this doesn't happen is slightly worrying. Incidentally, the approach taken in Question 6 reminds me of Peng et al (2019) who have a set of policies and attempt to learn weights with which to compose them.

In line with my concern in the theoretical setting, the method here was tested on a single domain that has these nice subgoal properties, but another domain would go a long way to convincing me how applicable the notion of independent features/policies is. What would happen, for instance, if we were in a classic four-rooms domain, and the policies were to get to the doorways and to get to the centre of the rooms? In this case, there's no way to get to the centre of a room without passing through a doorway, and so would things break down here? My main concern again is just how widely applicable these definitions are. 

**Clarifying questions and minor comments:**

  1. I realise space is an issue, but it would be better not to direct the reader to a different paper to try find details of the environment setup (which is actually only in the supplementary material of that paper). Please include the description of the domain in the appendix. Incidentally, I could not locate the statement that describes what the feature function is (is it 0 everywhere and 1 when a particular object is picked up?). 
  2. What is meant by normalised rewards? Is this just scaling the rewards to be in 0-1? Another thing that could be done here is instead to train optimal value functions, and then show the difference between the returns of the composed policies compared to the optimal returns. 
  3. At present, Figure 2 shows that the independent policies are the best, but there's no indication of how far from optimal they are
  4. Typos: "wee" (bottom page 3), missing "to" at top of page 8

**References**

  1. Abel, David, et al. "Policy and value transfer in lifelong reinforcement learning." International Conference on Machine Learning. PMLR, 2018.
  2. Van Niekerk, Benjamin, et al. "Composing value functions in reinforcement learning." International Conference on Machine Learning. PMLR, 2019.
  3. Tirinzoni, Andrea, Rafael Rodríguez-Sánchez, and Marcello Restelli. "Transfer of Value Functions via Variational Methods." NeurIPS. 2018.
  4. Fernández, Fernando, and Manuela Veloso. "Probabilistic policy reuse in a reinforcement learning agent." Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems. 2006.
  5. Thrun, Sebastian, and Anton Schwartz. "Finding structure in reinforcement learning." Advances in neural information processing systems (1995): 385-392.
  6. Nemecek, Mark W., and Ron Parr. "Policy Caches with Successor Features." International Conference on Machine Learning. PMLR, 2021.
  7. Pickett, Marc, and Andrew G. Barto. "Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning." ICML. Vol. 19. 2002.
  8. Nangue Tasse, Geraud , Steven James, and Benjamin Rosman. "A Boolean Task Algebra for Reinforcement Learning." Advances in neural information processing systems (2020a).
  9. Nangue Tasse, Geraud , Steven James, and Benjamin Rosman. "Logical Composition in Lifelong Reinforcement Learning." Lifelong Machine Learning Workshop at ICML (2020b).
  10. Peng, Xue Bin, et al. "MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies." Advances in Neural Information Processing Systems 32 (2019): 3686-3697.
summary_of_the_review: While the ideas presented here are sound and backed by theoretical and empirical results, the lack of context with respect to prior approaches makes it hard to judge the impact and novelty of the work. The experiments, in particular the lifelong learning one, could benefit from more appropriate baselines. Furthermore, I have concerns about whether the assumptions and definitions are too restrictive, reducing the applicability of the approach more generally, especially compared to more recent work (Nemeck and Parr 2021). I am, of course, open to having this score upgraded based on future discussions and changes.

******************************
POST-REBUTTAL
******************************

The paper has been significantly strengthened during the rebuttal period, and as such I am increasing my score. One important point to clarify is the setting - the "unsupervised RL" setting is a bit ambiguous and could lead to confusion, since it could mean the reward-free RL (see Jin et al 2020) and is separate from the lifelong setting. Clarifying this in the paper is important. 

The main improvement that can still be made is extending it to the lifelong learning setting, where the agent doesn't get to pick its task/reward function. Such an agent would be fully autonomous and would be able to discover and store a library of policies on the fly. This would be much more in line with existing work, and make comparisons to other work (Abel et al, Nemeck and Parr) apples-to-apples. It would also make Algorithm 1 far more interesting than it currently stands. Nonetheless, I feel there is enough here as it is to vote for acceptance
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem in RL of reusing policies resulting from previously learned (different) tasks in order to perform well on downstream tasks via generalized policy improvement. The authors empirically demonstrate (e.g., using a 2D item collection environment and a lifelong RL scenario) that under certain assumptions about the environment, learning a diverse set of policies bootstrap the learning process on new tasks.
main_review: The paper is, in general terms, interesting for the scientific community in RL. It is well structured and written, it reads very well, the introduction correctly states the problem to be addressed (built around previous works), the limitations that we currently find in the literature and, finally, the contributions. I have not found problems in the notation used or in the mathematical formulations.

However, there are a couple of aspects that make me doubt about the merit of this contribution:

It is a clearly incremental work built on the work of Barreto et al. The novelty of the paper and the contributions offered are limited to answering a series of questions (theoretically and experimentally) about the required conditions and the reliability/guarantee provided by the approximations using SFs & GPI.  On the other hand, the experiments shown, although illustrative, are limited and "toy-like". Once again, the settings, frameworks and approximations already presented by Barreto et. al. are used.


summary_of_the_review: See above.
review_title: null

Meta-review: This work extends the successor feature framework by focusing on the question of which policies should be learned in order to get the best generalization performance. The reviewers all agree that the question being addressed is interesting and important. One concern raised by two of the reviewers is that the work is rather incremental, providing a relatively small extension from the work of Barreto et al. Nevertheless, the authors have provided a convincing rebuttal, resulting in an increase in score of two of the reviewers. Hence, I recommend acceptance. I do want to ask the authors to carefully read the post-rebuttal point mentioned by reviewer 3QcK about clarifying the unsupervised RL setting.
Decision: Accept (Poster)