Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper notes that Universal successor feature approximators (USFA) $\psi(s,a,z)$ (here $z$ is the policy vector) exploits the smoothness of optimal value functions across different tasks but its approximation error could be large when target task vectors are far away. To improve generalization to a new task $w$ (which can be expressed as a linear combination of source tasks), the paper proposes constrained training of successor features that reduces its approximation error on the new task. It then shows that a similar effect can be achieved by using similar constraints at test time without changing the training of the successor features. The proposed method is able to improve task generalization on scavenger and reacher domains.
strengths_and_weaknesses: **Strengths:**
  1. The resulting *constrained GPI* has good theoretical motivations
  2. It exhibits better task generalization than GPI baselines

**Weaknesses:**
  1. The baselines used for comparisons are inadequate. For eg: recent works on bilinear decomposition of Q functions (bilinear value networks (Hong et al., 2022, https://arxiv.org/pdf/2204.13695.pdf)) showed improved generalization on novel tasks. Specifically, for goal condtitioned task with goal $g$, they choose the parameterization $Q(s,a,g) = \phi(s,a)^Tw(s,g)$ (as opposed to $Q(s,a,g) = \phi(s,a,g)^Tw$ or $Q(s,a,g) = \phi(s,a)^Tw(g)$. It would be nice to see comparisons to this bilinear decomposition.
  2. Furthermore, the domains (i.e. scavenger and reacher) on which the method is tested is limited. I would appreciate if authors can provide a more extensive evaluation on various goal conditioned tasks (such as the ones in fetch-gym (https://github.com/jmichaux/gym-fetch)).

questions: **Recommendation:** I think this is a borderline paper and I am happy to increase my score if the authors can evaluate their method on more tasks (such as the ones in fetch-gym) and compare it to better baselines (such as bilinear value networks).
limitations: The authors discuss the limitations of their work in Section 5
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper looks at the problem of transfer in reinforcement learning from source tasks to target tasks when the reward signal changes across tasks but the state and action spaces remain the same. Authors address a limitation of a current approach in this setting: GPI with universal successor features approximators (USFAs). The works argues that USFAs can make high approximation errors on test targets if their solutions are distant from the source targets' ones, and proposes a solution to mitigate this limitation. 

The solution involves constraining the approximation action-value error on new target task, using a lower and upper bounds introduced in the paper. The experiments show a zero-shot transfer improvement in performance compared to baselines (USFA + GPI) on synthetic scavenger tasks and robotic locomotion tasks. 
strengths_and_weaknesses: Strengths:

- The paper addresses relevant and important open problem in RL (transfer learning).
- The writing is clear, rigorous and the paper is easy to follow. A pleasant read!
- I found theoretical contribution to be main significant result, extending Nemecek and Parr result to about the value of an optimal policy for a new task beyond the conical hull is an important result.
- The proposed method, constrained GPI, is an elegant way of improving GPI.

Weaknesses:
The limitations of the work are not really discussed, and no direction for future work is given. This make it difficult to gauge if the method and its guarantees would extend to more complex and large scale environments.


questions: 

Questions
- Is there a way to extend constrained GPI if the reward cannot be linearly decomposed (but an arbitrary non linear function of the features)?
- The theoretical and empirical results presented rely on the prior knowledge of the feature vector $\phi$. How would constrained GPI perform with an unperfected (e.g. learned) feature vector instead?

Minor clarifications and comments:
- It would be helpful to spell out the different methods acronyms of figures 2 and 3 in the main text or in the caption. 
- I did not understand caption of figure 2b
limitations: Very limited discussions on the limitations. In fact, I did not really understand the remark on not getting the "full effectiveness".

No negative societal impacts were reported, and I don't see any myself either.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Successor Features / Generalized Policy improvement provide a method for generalizing to new tasks that are a linear combination of existing tasks. USFA combine this approach to generalization with neural networks with also generalize by training function approximators conditioned on the task.

This work extends USFAs by showing that one can establish lower and upper bounds on the generalization performance. When estimating action-values on a new task, these bounds are used to constrain the function approximator estimate.

They test this approach (comparing to USFAs) on two simple environments and find using these constraints outperforms USFAs.

strengths_and_weaknesses: 
Strengths:

- Paper is well communicated.

- Paper will be of interest to community working on successor features and related approaches.

- Paper introduces both theoretical grounding and empirical results.

Weaknesses:

- In common with a lot of work in SF, the test environments are "toy" and its unclear whether these approaches scale to more interesting tasks. For this reason, it may be of interest only to a smaller subset of the NeurIPS community.

- It would be interesting to compare very different approaches, such as a conditional decision transformer [1] or GATO style policy distillation [2].

- [3] introduces an alternative method for improving the performance of GPI (in the max-ent RL framework) that reduces the generalization error and should be cited, and ideally compared against.

[1] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." Advances in neural information processing systems 34 (2021): 15084-15097.

[2] Reed, Scott, et al. "A generalist agent." arXiv preprint arXiv:2205.06175 (2022).

[3] Hunt, Jonathan, et al. "Composing entropic policies using divergence correction." International Conference on Machine Learning. PMLR, 2019.

questions: Could you comment on the scalability of this approach to larger domains and tasks?

limitations: Yes although some comments (and ideally testing) on scaling to more complex domains would be useful.
review_title: null

Meta-review: metareview: The reviewers agree that the paper is a valid contribution to the line of research on successor features (SFs) and generalized policy improvement (GPI). They also agree that the paper is well written and easy to follow.

Three points that may be worth taking into account when preparing the final version of the paper, all related to the presentation:

1. The paper has two main contributions: bounds that improve upon those of Nemecek and Parr [1] and a new application of bounds of this type to detect approximation errors in universal successor features approximators (USFAs). Although these two contributions are listed at the end of Section 1, in the rest of the paper the derived bounds are always mentioned in the context of their specific use with USFAs. In particular, I believe the authors never mention that their bounds could also be applied to decide when to add new policies to a set of policies to be used with GPI, as suggested by Nemecek and Parr. In summary, the authors may want to have a presentation that clearly disentangles the two contributions of the paper.

2. Although the writing is mostly clear, I feel like the core idea of the paper is never spelled out in a concise way (this is especially important for the abstract). Given a set of vectors $\mathbf{w}_1$,$\mathbf{w}_2$, $\dots$, $\mathbf{w}_n$, and an associated space of tasks $\mathcal{W}$ composed of all possible linear combinations of the vectors $\mathbf{w}_i$, the paper derives lower and upper bounds for the value functions of all tasks in $\mathcal{W}$ in terms of the value functions of the tasks $\mathbf{w}_i$. These bounds can then be used in several ways; one novel application proposed in the paper is to detect approximation errors in USFAs. Maybe spelling this out in the abstract and introduction would help the reader to quickly understand the message of the paper.

3. It seems like the $\max$ in the upper bound (Eq. 10) will always be resolved based on the sign of $\alpha_{\mathbf{w}}$, since the first term will always be larger when $\alpha_{\mathbf{w}} > 0$ and the second term will always be larger when $\alpha_{\mathbf{w}} < 0$. This seems to be the ``trick'' used to improve upon  Nemecek and Parr's bound. The authors should consider adding this comment to the paper. 

I hope the constructive feedback is useful to improve your paper.

[1] Nemecek, M. Parr, R. Policy Caches with Successor Features. ICML, 2021.


confidence: Certain
Decision: Accept