Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new successor feature learning algorithm, called \xi-learning. Previous SF assumes reward function is a linear composition of SF and reward weights, this paper extends previous SF to a setting with a general reward function over SF. Based on this, \xi-learning learns to estimate the cumulative discounted probability of SF, and provides two update operators, model-free \xi-learning and model-based \xi-learning. The convergence of \xi-learning is proved, and empirical results on environments with discrete and continuous state spaces show it outperforms previous SF methods.
main_review: The motivation of this paper is not described and supported well. Why the assumption of linear composition is bad, or not applicable to what domains? From the results, previous SF methods perform not that badly on the problem with a general reward function.
It is more suitable to say two environments with discrete state space and continuous state space.


For experiments, 
Why not compare with [1], and some SOTA meta RL algorithms?

For the second environment, why discretize the state space, since deep SF can be applied to continuous domains?

The reason that why MF \xi-learning outperforms MB \xi-learning is not discussed. Since MF \xi-learning is superior, what is the need to propose MB \xi-learning? 

Results of MB \xi-learning in the second environment are not provided.

Some descriptions need to be improved. MB \xi-learning refers to Model-based \xi-learning, not Modle \xi-learning.


[1] Fast reinforcement learning with generalized policy updates.

summary_of_the_review: I think this paper contributes to the SF research areas, but the motivation is not well discussed. Also, there are some details, analysis of results should be well discussed. Based on this, I give a borderline reject currently.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors extend the successor features (SF) linear formulation to general non-linear formulation. The new method allows the reward to be arbitrary composition of features. 

The propose method is based on learning cumulative discounted probability of SF rather than cumulative discounted sum of SF as in linear SF framework. 
The authors provide theoretical proofs about convergence and propose two practical methods. Experiments are conducted in two toy environments.

main_review: The authors propose an interesting extension to linear SF framework. The derivations look correct to me and are overall well written. 

Just a few suggestions on the presentation of the method. 
- It would be helpful if you reduce the new formulation to the special case linear SF around equation (13) and illustrate the key difference. 
- discuss the difficulty of simultaneously learning features and estimating the density of features.

Kudos to the authors conducted u-test for the significance of the results. Appreciated the effort in making more rigorous results. 

The primary weakness of this work is the experimental section. 
There seems to a number of different issues all conflated into one set of experiments.

Firstly, in both collection and racer environments, the features are manually defined instead of learning from observations. This brings up a major concern that it hurts the flexibility and expressiveness of linear SF. 
With learned features, even if we assume reward is a linear function of features, the resulting value function is not hindered and in fact the expressiveness of value function is not limited by the linearity, as features can be an expressive and arbitrarily non-linear function of observations. 

Secondly, it seems in all experiments the evaluation tasks set is the same as the training tasks set. This makes make it hard to interpret the experiments or understand how effective is the transfer. I encourage the authors to divide the tasks into training set and hold out test set, and show the average return on the hold out set. 

Lastly, I believe the authors should compare again random feature baselines, where a neural network is randomly initialized and fixed during training, and then running linear SF and xi-learning on top it. Similar to Hansen et al. This would provide insights on how the SF works in these domains. 

These issues are probably fixable by more careful experimentation and clarity on the exact setup.

A general weakness is that the tasks are very simple, to the point that it is very unclear why the linear SF formulation does worse than xi-learning. I encourage the authors to try to evaluate the method at least on the Mujoco tasks from Barreto et al 2018. 

A question about the reward definition. The rewards for collection environment is defined as “The general reward functions are sampled by assigning a different reward to each possible combination of object properties ..., such that picking up an orange box might result in a reward of ...”, I don’t fully understand the rationality behind the particular choice, could the authors elaborate more? 
summary_of_the_review: In summary, the paper is an interesting effort in extending the linear SF framework. 

But why the linear assumption in SF framework is bad is not well supported, especially when the features can be an expressive and arbitrarily non-linear function of observations. 

The experiments failed to show very convincing results due to simple environments and problematic experimentation setup.  

The authors are encouraged to address these concerns. It would be a nice contribution if the concerns are resolved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses a limitation of the successor features (SF) which were introduced as a mechanism for transfer learning in reinforcement learning when the reward function changes across tasks. The authors claim that the original SF framework will only provide a good approximation of the true Q-function if the reward function in a task can be represented using a linear decomposition, and propose to address this linear decomposition of the reward function requirement by introducing a novel SF mechanism, Xi-learning, based on learning a cumulative discounted probability of successor features. The paper includes theoretical proofs of the convergence of xi-learning as well as transfer learning guarantees under generalized policy improvement (GPI). The authors compare the performance of their methods against standard Q-learning and SFQL in two different domains. 
main_review: On the writing:
The paper is nicely written and well structured, making it a pleasant read. It also appears to be technically sounds. 

On the proposed method:
My main concern is that authors didn't acknowledge previous efforts by Barreto et al. 2018 to address the problem of general reward functions. The authors state that the upper-bound in (7) is only interesting if the reward function can be decomposed linearly with respect to state-action features. Barreto et al. 2018 proposed a principled way of learning features (or even the reward function) such that the approximation error on the reward remains low, and the upper-bound on the optimal Q-value error remains low as well.

Without a comparison with Barreto et al. 2018 method, it is difficult to understand the necessity of introducing a new quantity to learn (the expected cumulative discounted probability of SF). While I like the formalism and the theoretical guarantees, to recommend the paper to be accepted, I would need more convincing on the necessity of the xi-function, specially given how difficult it will be to learn in large and stochastic environments.  

On the experiment:
- Authors should specify how the weights \tilde{w_i} were learned for SFQL in the general case. 
- Results including the learning of the reward function - or weights w - would be very interesting to appear in the main text, as in practice we rarely have access to the reward function. It would help build a case for practical use of the proposed method. 
- It seems to me that the continuous environment is rather a discretized environment? 
- I did not understand the proposed justification for xi-learning improving on SFQL (even slightly) in environments with linear reward function. What did the authors mean by "xi-learning reduces the complexity for the function approximation of the xi-function compared to the phi-function in SFQL"?

Minor comment: I found the second sentence of Theorem 1 difficult to understand. 

summary_of_the_review: Despite enjoying the reading of the paper, the main reason for my recommendation is that due to the lack of discussion / comparison with another method that adresses the non-linearity of the reward function to learn accurate SF, I am not convinced by the necessity of the xi-function, which is the novelty of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a variant of successor features that represents the unnormalized cumulative discounted distribution of state features $\xi(\mathbf{s}, \mathbf{a}) = \sum_t \gamma^t p(\phi_t \mid \mathbf{s}_0 = \mathbf{s}, \mathbf{a}_0 = \mathbf{a})$ instead of the standard expectation $\psi(\mathbf{s}, \mathbf{a}) = \mathbb{E} [ \sum_t \gamma^t \phi_t \mid \mathbf{s}_0 = \mathbf{s}, \mathbf{a}_0 = \mathbf{a}]$. It proposes both a model-free and model-based variant for training $\xi$ and evaluates the effect of non-linear reward functions in two diagnostic environments.
main_review: This paper tackles the important problem of transfer in reinforcement learning. However, the generalization it proposes is oddly positioned. The main premise is described as:

> Nonetheless, this assumption also restricts successful application of
SF&GPI only to problems where such a linear decomposition is possible. This paper investigates the
application of the SF&GPI framework to general reward functions: $r_i = R_i(\phi)$.

This is true in the sense that the linearity of successor features prevents them from representing arbitrary sets of reward functions. (Any individual reward function is representable by setting $\phi_t = r_t$.) One easy example of a reward set that SF could not represent (in continuous spaces) is that of Dirac delta functions over all states, as this would require feature vectors $\phi$ of the same cardinality as the state space. While the proposed $\xi$ variant is indeed more general than the linear SF, these rewards are still functions of $\phi$, so the expressivity of $\xi$ depends on the expressivity of the featurization $\phi.$

One possible response to that criticism is to simply set $\phi_t = \text{concat}(s_t, a_t, s_{t+1})$, which really would fulfill the promise of handling general reward functions. However, this raises a serious practical issue, in that we must now recover the discounted occupancy of a policy in the original state space. This is a pretty difficult combination of dynamic programming and generative modeling (each in their own right hard problems), so is not straightforward to scale (though the paper cites a few attempts). I did not find the discussion of this difficulty too convincing; the main strategy here seemed to rely on a coarse discretization of continuous spaces. I would expect this to work well in the diagnostic environment studied, but mostly because the underlying state is low-dimensional enough that discretizing it and treating it as tabular is a reasonable strategy. (The actual observation is higher-dimensional, but the underlying state looks like position and velocity in $\mathbb{R}^2$.)

I think these two issues put the paper in a strange position:
1. If the feature vectors $\phi$ are unrestricted, then $\xi$ really is more expressive than linear SF, but the method requires solving a long-horizon generative modeling problem to represent the state occupancy and does not propose a viable way of doing this.
2. If the feature vectors $\phi$ are restricted in such a way that $\xi$ is learnable in practice, then $\xi$ likely ends up in the same spot as SF: able to represent any _individual_ reward function, but not any arbitrary set of rewards due to limitations on the featurization itself.

If there were evidence presented that $\xi$-learning is more scalable than I am giving it credit for (_e.g._, compared to SF, in environments for which the linearity assumption is limiting without artificial constraints on the featurization or observation space), then of course I would revise my review.

I want to make sure to list the good too: the paper positions itself well in the SF and GPI literature. While the analysis mostly follows from analogous results in $Q$-learning, it is thorough. There is clean code provided.

**Copy-editing**

*Introduction*:
1. "intput" --> "input"
2. "an cumulative" --> "a cumulative"

*Background*
1. "following following the Bellman equation ": repeated "following"
2. "a MDP" --> "an MDP"
3. "proofed" --> "proved
4. "error in the approximation increase" --> "increases"

*Method*
1. (Prop 1) "quotien space" --> "quotient space"
2. (Corr 2) "supreme norm" --> "supremum norm"?
3. (Prop 2) "well-define" --> "well-defined"

*Experiments*
1. "which object as been already collected" --> "which object has"
2. "number of Gaussian’s": no apostrophe
summary_of_the_review: I am in the borderline mode for this paper. Due to the difficulty of the generative modeling problem associated with $\xi$-learning, I am not quite convinced that the generalization presented in this paper would actually lead to something more expressive than SF in the types of setting where the linearity of SF proves limiting. That said, this paper does have merits and adds something new to the SF literature, as mentioned in the main review above.
review_title: null

Meta-review: The reviewers uniformly suggested rejecting the current paper.

I concur and remain especially somewhat unconvinced by the authors comments on learning features.  In particular, any argument based simply on (current) performance seems rather weak.  There are methodological reasons one might want to keep features fixed, and there are a small subset of problems with well-defined known useful features.  But in the long term surely we should want to be able to learn the features, and efficiently and elegantly handle the case where they are learnt continually.

I want to thank the authors for engaging.  This work has the potential to be improved and I would encourage the authors to carefully consider and incorporate the provided feedback by the reviewers into their work.
Decision: Reject