Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors extend the Successor Features (SF) framework to allow for a quadratic  relationship between the features and rewards, rather than the strictly linear one given in the initial formulation. The derivation is relatively clear and appears to be correct, with the nice result being that the additional term needed to account for the non-linearity can still be estimated via a Bellman equation.

One fact that the authors fail to address is that the linearity between the features and rewards doesn't limit the expressiveness of value functions computed by SF, since the features can be an arbitrarily non-linear function of the observations. Now, it might be the case that the ability to generalize to novel tasks is greater when making the rewards be non-linear in the features, but the author should make this distinction.

Indeed, this ties into my larger issue when the paper: the Axes task (and maybe the Reacher task depending on the dynamics) is very simple, to the point that it is very unclear why the linear SF formulation does worse than not only the quadratic case, but than a random agent. As per my argument about expressivity, both SF agents should be equally able to solve the training tasks, with the only space for an advantage for the non-linear version coming from generalization.

But the Axes task appears too simple to even be used for comparing generalization performance (unless I'm mistaken about the dynamics). If the tasks terminate upon reaching the goal, and the start states are all in the middle of the goals, then couldn't an agent linear in the *observations* solve the tasks optimally? e.g. map the difference between the current and goal observations to the action most closely corresponding to the resulting direction.

The Reacher task might also suffer from these problems, but I'm unsure without further details. Do the actions act on the torques of the robot motors, or do they directly impact end-effector space? If it is the latter, then Reacher appears to just be Axes with an addition spatial dimension, which would still be readily solved without any non-linear function approximation.

I apologize if I sound overly harsh. The extension is interesting, but currently its unclear if these experiments provide any support to your claims about its advantages. Unless I'm mistaken about the dynamics of these environments, I'd suggest showing results on tasks of known complexity, like the Doom game Deep SR uses or the Scavenger environment from the SF paper.

The exploration angle is quite interesting, but currently its hard to understand the motivation. Why is perturbing this expected variance-like object a good idea? Expanding upon that would be greatly appreciated, though for it to be the main claim of the paper there would also have to be some comparison to alternative exploration methods (e.g. Noisy Nets or Bootstrap DQN).
review_title: Sound extension to Successor Features, but empirical implications unclear

final_score: rating: 4: Ok but not good enough - rejection
review_content: Successor representations are an old idea that has seem recent interest in the ML community. The idea is conceptually straightforward, by assuming the rewards are linear in some space $r = \vect{\phi}(s, a) \cdot \vect{w}$ then we learn something analogous to an action-value function for the discounted expected features under a policy so that the action-value on task $\vect{w}$ is $Q^\pi_{\vect{w}}(s, a) = \psi^\pi(s, a) \cdot \vect{w}$. This allows computing the action value for the policy under a new task $\vect{w}'$ straight.

One limitation is the assumption that there is some feature space where the rewards of all tasks can be linear in this space.

The key idea is to extend the idea of successor features to the 2nd order relation between features and reward so now the assumption is the reward for all tasks is defined by eq 1 in the paper
so in addition to computing the expected features in the future $\psi^\pi(s, a)$ (a vector value) the autocorrelation of the features $\Lambda^\pi(s, a)$ must also be track (a matrix).

They construct a method for learning features as an autoencoder of the state and experiment on some simple tasks such as reaching to different locations comparing this ``second order'' method against successor features.

Strengths:
- This does seem like the natural ``second order'' version of successor features. I'm not aware of any closely related prior work to extend successor features in this way and it clearly allows for a more expressive description of rewards.

- Paper is mostly well-written and communicated, including try to help develop an intuition for the new quantities introduced.

- The interpretations of $\Lambda$ are interesting and ideas around how to use it for exploration are intriguing.

Weakness:

The primary weakness of this work is the experimental section. There seems to a number of different issues all conflated into one set of experiments.

Firstly, whether using successor features (SF) or 2nd order successor features (SF^2), $\psi^\pi$ and $\Lambda^\pi$ are a function of a particular policy. It is unclear if, for each training task a different $\mathbf{w} = \mathbf{o} + \mathbf{A}$ was learned (as one might expect in a SF setup), there is never any indices indicating these are task specific so perhaps there is only one version of these learned for all training tasks? It seems that only a single policy is learned for all training tasks. The setup should be clarified and the reasoning for these choices explained.

This means that over both training and test tasks only a single policy is available. Therefore, any hope of solving an individual task must be due to computing the advantage under this policy which is quite limiting. Both Barreto et al., and Ma et al. estimate the successor features for a set of policy (indeed the main contribution of Barreto is to show how to use Generalized Policy Improvement (GPI) to construct a policy for a new task defined by $w'$ from a set of existing policies, typically one per training task, there seems no reason GPI cannot be used here).

Finally, it would be helpful when introducing a conceptual idea as here to have a simple version where only that idea is needed. In this case, by (as in Barreto) experiments with fixed, pre-defined features. Here, in both experiments the features are learned using an autoencoder of the state space (which is not guaranteed to learn a representation which is ideal for use in successor features).

These limitations make it hard to interpret the experiments or compare with other work. For example, the two tasks appear quite similar to Barreto et al., yet here the SF baseline ($\beta=0$ in the paper) performs worse than random, even on the baselines. It is hard to interpret why this: is it due to the learned features not being good for this task, using only one policy for all tasks or the non-linearity of reward? It seems clear that e.g. for the Axes task if there was a feature for being located on each potential goal and a policy per task then, at least on the training tasks, SF should perform much better than random (by learning to reach for the targets).

Ideally, SF would also be compared against other methods for transfer such as meta-learning (e.g MAML). The existence of such methods, along with model-based approaches such as used in Go mentioned in the introduction probably means the claim in the introduction that "current algorithms cannot transfer a learned policy between related tasks" is too strong.

This issues are fixable by more careful experimentation and clarity on the exact setup of the problem. As part of that the captions for figures 3 and 4 could be extended.

A more general weakness is that, while this paper improves the expressiveness of SF by allowing a 2nd order relationship between features and reward, it is not clear if this is the key limitation of successor features. Namely, it still does not allow transfer between tasks where the transition function has changed ([1] should probably be cited as an attempt to extend SF in this direction) and only supports estimating the feature occupancy under existing policies. I personally would regard these are the more limiting factors in SF rather than the linearity of reward. These issues should be discussed and the limitations and weakness compared to e.g. meta-learning approaches should be discussed (and as mentioned above, ideally compared).

Just to be explicitly clear the rating given to the paper is the paper in it's current state. I think if the issues above are addressed this is an interesting paper and I would rate higher.

[1] Zhang, Jingwei, et al. "Deep reinforcement learning with successor features for navigation across similar environments." 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.
review_title: The natural extension of successor features to 2nd order, but let down by experimental section

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a quadratic reformulation of successor features (SF), in which rewards are given by $r=\phi^\top \mathbf{o} + \phi^\top A \phi$ instead of the usual $r = \phi^\top w$. This generalization leads to learning a second-order term $\Lambda = \mathbb{E}[\sum_{t} \gamma^t \phi^\top \phi]$ to augment the expected featurization $\psi = \mathbb{E}[\sum_{t} \gamma^t \phi]$.

While this is a more expressive parametrization, learning $\Lambda$ is difficult due to its dimensionality. One of the strengths of SF is that it turns a high-dimensional prediction problem into a set of independent scalar Q-learning problems; incorporating $\Lambda$ arguably misses out on that important strength by no longer being able to treat the dimensions of the state featurization as independent. It seems that the learning problem can be made tractable by a low-rank factorization, but this does somewhat call into question whether a quadratic form is really the benefit or just expressivity from having more parameters. This could be answered by an experiment controlling for the number of parameters in $\phi$ (for the linear SF baseline) to match that in the quadratic variant.

The addition of $\Lambda$ is motivated by the limitation of a linear reward model:
> *The factorization follows from the assumption that reward can be predicted as the dot product between a state representation vector and a learned reward vector.*

This is not exactly an assumption as stated, since the featurization could be the reward itself ($\phi(s) = r(s), w = 1$), meaning that this decomposition is possible for any reward function. (See the discussion under Equation 2 in [Barreto, 2016].) That is not to say there is no limitation from linearity: it is true that for a *given* featurization, it may not be possible to linearly solve for any reward function, and that it may not be possible to find any suitable featurization for a set of many reward functions, but this is not discussed in enough depth to know exactly what issue is being referenced.

The experimental evaluation shows that the quadratic variant outperforms linear SF in two toy tasks. This is explained by the fact that these environments require a nonlinear model:
> *Finding a solution to the environmental reward structure is difficult as the reward is a
non-linear function of the features; in this case, the agent’s coordinates and the current goal location.*

However, even in the case of SF, the reward should not depend linearly on the observations themselves, but on a featurization of the observations. This is why a linear parametrization is suitable for any single reward function: the featurization can be arbitrarily complicated. Are you using observations (coordinates and goal location) instead of $\phi$ here?

Other experiments focus on exploration by adding noise to $\Lambda$ (instead of actions) and the structure of the learned $\Lambda$. The exploration idea is interesting, but somewhat underdeveloped, with there not being much justification and no clear empirical win. For what it is worth, the two environments studied might be simple enough that naive $\epsilon$-greedy exploration is good enough, and it is difficult to squeeze out much improvement without considering an environment which poses more of an exploration challenge. I did not understand the $\Lambda$ visualization in figure 5.

Though the major comparison of interest is between SF and the quadratic variant, it would be nice to explore a few more ablations of the method (like controlling for number of parameters, discussed above). It would also relieve a little bit of concern to have one or two more baselines on the standardized environments; the results in figure 3 look surprisingly sample-inefficient compared to modern Q-learning methods, so it could be worth figuring out why or showing that this is not the case by including another baseline.

**Questions:**
1. Where exactly does $\beta$ come from? It seems to appear between Equations 21 and 22 in Appendix A, but I cannot find the reason. Can the constant just be subsumed into $A$ in the reward regression in Equation 8? Is there ever a reason to set $\beta$ to anything besides 0 or 1?
2. What happens when you remove the reconstruction objective on $\phi$? Another advantage of the SF decomposition is that it can learn features informative for predicting values without having to rely on reconstruction, so can discard irrelevant parts of the state which would normally thwart a reconstruction objective. Incorporating a reconstruction term seems to miss out on that benefit. I realize prior work like [Kulkarni 2016] has used a similar auxiliary objective, so this isn't really a negative so much as a question as to why you think it is necessary.

**Copy-editing:**
1. Section 1.1 *The object* —> *the objective*

**Summary:**
Though this is an interesting generalization of SF, it does not seem quite ready for publication. I encourage the authors to more precisely motivate the quadratic form, since even after reading this paper its disadvantages (can no longer treat the dimensions of $\phi$ as independent, intractable due to dimensionality so requires approximation anyway) seem to outweigh potential advantages due to the increased expressivity (especially given the note above about a linear model being sufficient for any single reward).

review_title: An interesting generalization of successor features that requires a bit more development

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to extend successful features by learning the second moments of cumulants in addition to the cumulants. They demonstrate that the resulting method performs better on 2D and 3D goal-reaching tasks (without obstacles) when the reward is the squared distance.

The paper is clearly written and extending successor features to non-linear rewards is an interesting problem. However, the framework does not so much provide a “novel formulation of successor features” but rather presents a specific instantiation. While this maybe be an interesting observation, the resulting experiments fail to compare to more appropriate baselines and are tested on rather simple domains that can be solved by a quadratic policy that is greedy with respect to the reward.

In detail:

The paper is effectively an instantiation of the successor feature that adds structure to the cumulants. In particular, it adds features that are products of other features, i.e. phi_new = [phi_1, phi_2, phi1 * phi_2, etc.]. Thus, the claim that this is a “novel formulation of successor features with a non-linear reward” seem inaccurate since the original successor feature formulation already handles the case where the reward is a non-linear function of the state.

Given the goal-directed nature of the tasks, the authors should compare to goal-conditioned methods [1,2,3] or at least demonstrate the method on tasks that cannot be solved by goal-conditioned methods. It is also an overstatement to say that the 3D reaching task is a particularly “difficult control task” given that much more challenging control tasks have been solved [4].


Minor comments:

Is it actually torque control? If so, how is it possible for the policy to learn given that it only has the XY location of the end effector? I’m under the impression that the meta-world tasks use end-effector velocity control.

“as the dimensionality of z grows, the number of parameters needed by Lambda grows exponentially” Do you mean that as the dimensionality of phi grows (or as z grows) the dimensionality of Lambda grows quadratically?

[1] Schaul, Tom, et al. "Universal value function approximators." International conference on machine learning. 2015.
[2] Andrychowicz, Marcin, et al. "Hindsight experience replay." Advances in neural information processing systems. 2017.
[3] Nair, Ashvin V., et al. "Visual reinforcement learning with imagined goals." Advances in Neural Information Processing Systems. 2018.
[4] Plappert, Matthias, et al. "Multi-goal reinforcement learning: Challenging robotics environments and request for research." arXiv preprint arXiv:1802.09464 (2018).

--- Post Rebuttal ---

I've read the author response. However, I do not plan on changing my score as my main concerns have not be addressed. In particular:

> the framework does not so much provide a “novel formulation of successor features” but rather presents a specific instantiation... Thus, the claim that this is a “novel formulation of successor features with a non-linear reward” seem inaccurate since the original successor feature formulation already handles the case where the reward is a non-linear function of the state.

The author response is

> There is no guarantee that the learned state features are able to find appropriate values. In theory yes, we agree but practically we found this not to be the case if we explicitly test for it. As our environments do, where the reward is a non-linear function of the state.

I understand that perhaps existing methods are not capable of learning good state features, and presenting a method for finding better state features (not the weights) would be interesting. However, the current paper simply hard-codes good features, which I do not find compelling.


review_title: Proposes to train successor features to predict [phi, phi^2] rather than only [phi], but does not demonstrate clear benefit

Meta-review: This paper extends the idea of successor representations. Typically the reward is compute linearly on top of states in this setting but the authors relax it to have a quadratic form. 

${\bf Pros}$:
1. A novel formulation of the successor representation where the reward does not follow the linearity assumption
2. The idea of using a second order term for the reward branch is interesting and could have meaningful implications for learning and exploration. 

${\bf Cons}$:
1. All authors agree that the experimental results do not clearly validate the advantage of this method. More work is needed to establish the effects of using this particular reward structure on a wide variety of tasks

2. Both R2 and R4 were unconvinced of the limitations of the linearity assumptions in the original successor representation formulation -- especially in the case when the state is represented by a non-linear function approximator.

The ideas presented in this paper are quite interesting and promising. But more experimental work is needed to show the benefits of this approach. 
Decision: Reject