Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: Paper shows a systematic study of different strategies for approximate zero-shot RL including successor features (SFs) and forward-backward (FB) representations. It introduces the concepts clearly and discusses the advantages and limitations of each method in the two categories. Paper introduces new SF models and compares them in zero-shot RL tasks from the Unsupervised RL benchmark. 
strength_and_weaknesses: Strength
- This work systematically tested forward-backward representations and many new models of successor features on zero-shot tasks. The evaluation on the ExoRL dataset and unsupervised RL benchmark is very extensive and across multiple different qualities of datasets. It shows that FB representation performs best and consistently across the board in a zero-shot manner. 
- The writing is well-written and well-organized, the authors introduce prior work and compare them both in both methodology and experimental aspects. The discussion of limitations is thoughtful and contains interesting points. 
- This work systematically evaluates prior zero-shot RL methods in the same setting, and then proposes a novel and effective method that consistently outperforms prior works, reaching 85% of supervised RL performance with a good replay buffer, in a zero-shot manner.
- The results are significant and will facilitate future research on unsupervised zero-shot RL. 

Weaknesses
- The biggest weakness seems to be that it assumes there is offline diverse data. One one hand, this is a limitation since unlike NLP there is a large unlabeled dataset naturally available, in RL, collecting diverse dataset is still not yet solved. On the other hand, this work focuses on evaluating learning zero-shot RL from pre collected unlabeled dataset, which is an important research question and it’s reasonable to have such assumptions in order to make the evaluation easier. 

clarity,_quality,_novelty_and_reproducibility: Quality
- Paper shows a systematic comparison of prior unsupervised RL methods, provides methodology and empirical comparison and discussions. 
- The findings are mostly interesting and could be helpful for future research in building zero-shot RL models from large unlabelled datasets.

Clarify
- Mostly very clear.
- The key concepts are introduced clearly and the main claims are well supported.

summary_of_the_review: The authors study zero-shot RL that pretrained on large unlabeled datasets for different downstream tasks, they conduct a systematical extensive evaluation of various unsupervised RL methods, and reveal algebraic links between SFs, FB, contrastive learning, and spectral methods. Seems to me that the findings in this paper are important for future research.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies zero-shot RL in the sense of "RL that does not require optimization of a policy when presented with a task (reward model)". It covers the literature on successor representations (SR) and forward-backward (FB) representation, proposes an unifying view and improved training losses. Then it empirically compares FB and many formulations of state features for SR, to assess which one provides (offline) representations which are suited for zero-shot deduction of (quasi-)optimal policies.
strength_and_weaknesses: Disclaimer: given the very short timeline for reviewing all assigned papers, and to my great frustration, I didn't have time to check the appendix. Still, I browsed through it and it seems correct and (very) interesting. I did check the rest of the paper in detail and am confident in my evaluation.

### Strengths

This paper is very well written, easy to follow despite the technicality. One quality I find in the paper (which the authors do not claim) is that it provides a unified presentation of SR and FB. I believe this is incredibly useful for the community and is one important strength of such a paper. Because of this, I view Section 4 as a valuable contribution in itself, even though its contents may not be drastically novel (some of them actually are: the last paragraph of Section 4 on the link between FB and SF is very important, especially the argument explaining the avoidance of representation collapse).  
Then, the formulations of Section 5 are useful and partially novel, especially those of Section 5.2 which are interesting reformulations of that of Touati & Ollivier (2021).  
The empirical evaluation is thorough, well-conducted and well-discussed.  
The main conclusion of the paper is that FB representations and Laplacian eigenfunctions for SR are more suited to zero-shot transfer. Limitations of the present evaluation are explicitly mentioned (although not really discussed). This conclusion, backed by convincing empirical evidence and relevant discussion is a nice contribution to the RL community.

### Weaknesses

This paper has a catchy name, but the name (although funny and I appreciate the pun) does not necessarily reflect the contribution: it is not really about existence of zero-shot RL but rather about the ability of neural networks to learn successor representations or measures. In the interest of readers in a few years, I would suggest keeping the catchyness but maybe focusing more on the ability to learn successor representations.  

At the bottom of page 4, the authors present the fact that FB does not need state featurization as an advantage. This seems true from an abstract point of view, but I believe it deserves a bit more discussion. In very unbalanced datasets, or with very large variance transition models, or in the regime of very scarce data, fully relying on data to estimate occupancy measures boils down to discarding whatever expert prior knowledge might be available for state representations. In this case, the advantage of FB representations is less obvious and this begs for deeper evaluation (the discussion at the end of Section 7 somehow discards this question a bit too fast). 

The AE features presented in Section 5.3 incur the risk of representation collapse in the sense that they might map to a set of linearly redundant features. From this perspective, I would have liked to have more insight as to the effective dimension of the learned feature space, for example as was done in the work of Lyle et al. (2022). There is a clear difference here since these features are not learned as solutions to a non-stationary regression problem, but still, the diversity of features could (should?) be made explicit.  
In the same line of thought, only the latent transition model incorporates a term which discourages representation collapse (with a BYOL-like auxiliary loss). Maybe the other AE features would have benefited from such a term encouraging feature diversity (like regularizing features towards fixed or slowly moving random projections of the input state).  
Lyle, C., Rowland, M., and Dabney, W. (2022). Understanding and preventing capacity loss in reinforcement learning. In 10th International Conference on Learning Representations.

Somehow connected to the previous remark, I would have appreciated a discussion on the latent spaces learned. Beyond the suggestion to evaluate the effective dimension of features (previous paragraph), I am curious as to what a tSNE of the replay buffer projected in the latent space would look like. Alternatively, the set of singular values from a PCA on the same set would be informative about the ability of the learned representation to shatter points. Since, in the discrete action case, the policy is eventually a linear classifier in the representation space, this latent space could (should) be better illustrated and discussed.

I slightly disagree with the idea that if the encoder-based losses learn nothing, it is because $(x,y)$ is already a good representation and they need to do nothing. They still have the freedom to learn bad (or collapsed) representations, which might be an alternative explanation. Also, in the same paragraph, it seems somehow counter-intuitive that SFs on (encoded) $(x,y)$ cannot learn the task: I am not convinced that planning requires "specific representations", I'd rather conclude that planning requires representations which preserve key information; this information is in the state $(x,y)$ but maybe not in its encoded version. The authors reiterate the claim that planning-specific features are necessary, in the paper's conclusion, and while I agree in the general case, I seem to fail to understand why the raw state is not a good planning specific feature since it "contains" the full state information necessary to build a policy. Is it because we need features which allow linear classification of actions? Maybe this deserves a clarification.

The paragraph describing how the scores of Figure 2 are computed is a bit unsettling. Since scores are normalized as a percentage of offline TD3, shouldn't the red dashed line stand at 100 on all figures? Is there a re-multiplication by the (average?) score of offline TD3?

I did not find a mention of the experimental code. Given the amount of experiments, releasing the code, the results and the training logs seems a crucial pre-requisite for fair dissemination of these results.

### Minor discussion elements 

No need to address these in detail but I believe they could contain ideas for future work and I'm happy to share them and discuss them.

I am always a bit skeptical with the inclusion of $\gamma$ in the definition of an MDP. And even more skeptical when it comes to a reward-free MDP. This might sound a bit knitpicking but an MDP is primarily a stochastic process, whose definition does not include any notion of discount. Even the introduction of a reward model over the Markov chain conditioned by $s$ and $a$ does not require $\gamma$. It is the definition of the discounted return which involves $\gamma$ and this is quite independent of the MDP's dynamics or rewards: it is intrinsic to the criterion being maximized. Another criterion (average reward for instance) would not require $\gamma$ and the definition of the successor measure would not involve it either, while the reward-free MDP remains the same. Besides me being picky, this opens the door to alternative FB schemes for other criteria than the expected discounted sum of rewards.

There seems to be a striking connection between $z_r$ with successor features and the expression of policy values in LSTD / LSPI. This might sound a bit naive (and there are clear differences too) but I'm not sure this connection was made explicit before in prior work on SR.
clarity,_quality,_novelty_and_reproducibility: The paper is flawlessly written. All claims appear correct and well-defended. The work is novel and very significant.

Although details on the experimental setup are provided in the appendix, it is never the same as looking at the implementation choices and some hyperparameters are not mentioned. Releasing the code, the trained networks and the training logs would really make this paper reproducible and its results exploitable by others.
summary_of_the_review: Overall, this paper is a strong contribution to learning successor representations and measures. The key finding is the general applicability of FB representations as successor measures and Laplacian eigenfunctions as successor features. Along the way, the paper provides a very appreciable unified view of learning representations for zero-shot RL, with insightful new formulations for FB representations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper conducts an empirical study of existing unsupervised pre-training methods for RL. Specifically, this paper focuses on successor features (SFs) and forward-backward representation (FB). The paper uses a variety of existing unsupervised representation learning methods as a way to provide state features for SFs. The empirical results on Unsupervised RL and ExORL benchmarks show that FB performs best. 
strength_and_weaknesses: [Strength]

* Provides a nice survey of successor features, forward-backward representation, and unsupervised representation learning methods. 

[Weakness] 

* The definition of zero-shot RL in this paper is quite different from conventional zero-shot RL, which can be confusing.
* Although the evaluation of many methods is valuable, there is not much analysis or insight from the experiments. 

clarity,_quality,_novelty_and_reproducibility: **Clarity**

- The problem setting (called zero-shot RL in this paper) is unclear and inconsistent with what the actual experiment does. This paper defines zero-shot RL as follows: "The promise of zero-shot RL is to train **without rewards or tasks**, yet immediately perform well on any reward function given at test time, **with no extra training, planning, or finetuning**, and only a minimal amount of extra computation to process a task description (Section 2 gives a more precise definition)." I found this very confusing because of the following reasons.
  - Typically, zero-shot learning (or generalization) in RL has nothing to do with training in the absence of rewards or tasks. This setting is closer to **unsupervised** learning in RL but not zero-shot RL. While this paper indeed conducts unsupervised RL in the experiment, it is important to make it clear that this aspect is not about zero-shot RL. 
  - "no extra training at test time" is consistent with typical zero-shot generalization in RL. However, the experiment in the paper violates this restriction by using 10,000 reward samples to train the reward-relevant component in SF and FB (or directly provide ground-truth $z_r$). Strictly speaking, this is not zero-shot generalization. 
  - To my understanding, the actual problem setting considered in this paper can be described as "unsupervised representation learning with a offline RL dataset followed by fast adaptation to test tasks with a few samples of interactions". This setting is recently introduced by [Liu et al.] (cited in the paper). Although this is an interesting setting, I believe that this is not really about zero-shot learning. 
- For the above reason, I think the current title is quite misleading. 

**Quality**
- This paper provides a nice summary of successor features and forward-backward representation with useful insights. 
- While this paper provided an extensive evaluation of existing representation learning methods for the specific problem, there is no further analysis showing why some methods are better or worse, other than several speculative hypotheses without empirical evidence in Section 7. It would be more convincing to show more in-depth analysis. 

**Novelty**
- The empirical study conducted by this paper is new. 
- There is no novel algorithm or method other than some interesting tricks to existing methods. 

**Reproducibility**

While the paper provides hyperparameters in the appendix, there is not much detail for all of the unsupervised representation learning methods, which makes it hard to reproduce the result. 
summary_of_the_review: Although this paper presents an interesting empirical study, the presentation of the paper needs to be significantly revised due to 1) the inconsistency between the problem definition and the experiment, and 2) the unconventional use of the term "zero-shot". In addition, the paper would benefit a lot from more in-depth analysis of existing feature learning methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on the empirical study of zero-shot RL, whose goal is to learn a set of policies and/or representations during training so that they can be adapted to solve unseen tasks in a zero-shot manner. Specifically, the authors focus on the successor features (SFs) framework and forward-backward (FB) representations, suggesting ten diverse choices for SF representations ($\phi$) as well as an improved mechanism to train FB representations. They test the SF/FB methods on 13 tasks from URLB with ExORL replay buffers and find that FB representations show consistently superior performances on the benchmark.
strength_and_weaknesses: **[Strengths]**
- To my knowledge, this is the first paper that systematically studies various choices for SF representations ($\phi$), which would be especially useful for practitioners who try to use SFs without relying on hand-crafted representations.
- The experiments are exhaustive and statistically sound (with 10 seeds).
- The claims in the paper are theoretically well-supported.

**[Weaknesses]**
- One thing that I do not fully agree with the authors is the distinction between SFs and FB. For me, the FB representation can be interpreted as a special case of USFs [1] with the tasks being all possible goal-reaching tasks (ignoring the subtlety in continuous domains). The authors distinguish FB from SFs in that SFs require predefined features and/or tasks (i.e., $\phi$), but I believe this is not accurate. For example, Ma et al. [1] use an end-to-end training scheme that learns a feature vector $\phi$ and a (learnable) task vector $w(g)$ by minimizing both the scalar loss (as in Eq (6)) and the vectorized loss (as in the equation above Eq (6)). If we assume that the source tasks consist of all the goal-reaching tasks to reach any states (and also assume a discrete state space), $w(g)$ becomes $B(s)$ and $\psi(s, a, z)$ becomes $F(s, a, z)$, and thus we exactly recover the FB objective. Hence, I believe the distinction between FB and SFs lies *not* in the way it automatically builds feature vectors ($\phi$) but rather in the loss function it uses (especially Eq (8)), which can handle continuous spaces as well, unlike USFs with all the goal-reaching tasks.
- Due to the reason above, I believe *learned* USFs (i.e., learnable $\phi(s)$ and $w(g)$ with the Q loss and/or vectorized loss [1]) trained with implicit goal-reaching tasks (from the dataset) should also be included as baselines. Also, some strong statements like "SFs must be provided with basic state features $\phi$" should be adjusted accordingly. I hope that the authors include detailed discussions and comparisons between FB and learned USFs.
- Since this paper focuses on the empirical study of SF representations, I believe the code should be made publicly available and hope that the authors release their code during the discussion period.

**[Additional questions]**
- What makes $\phi$ necessarily learn something in Eq (11) and (12)? It seems that $g$ or $f$ can capture everything with $\phi$ being the identity function.
- I didn't understand the following sentence in Sec 7: "Any low-rank model of $P_\pi$ will be poor; actually $P_\pi$ is better modeled as Id − low-rank, thus approximating the Laplacian.". Why does this amount to approximating the Laplacian? Also, it would be better to use parentheses around "low-rank" to avoid confusion.

[1] Chen Ma, Dylan R. Ashley, Junfeng Wen, and Yoshua Bengio. Universal Successor Features for Transfer Reinforcement Learning. 2020.

clarity,_quality,_novelty_and_reproducibility: **[Clarity and Quality]**
- The paper is easy to follow and generally well-written.
- The second bullet point in Sec 1 is an incomplete sentence.

**[Novelty]**
- The technical novelty of the paper is limited as it focuses on empirical analysis of various SF/FB representations.
- The empirical results are novel to my knowledge.

**[Reproducibility]**
- Their code is not released.
summary_of_the_review: While there is a point (the distinction between FB and SFs) that I do not fully agree with the author's claim, I believe this paper is of high quality and will be helpful to the community. I thus recommend acceptance.

(Post-rebuttal update) Given the additional results and clarification, I raised my score to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper defines the problem of "zero-shot RL", in which an agent should be able to solve any reward function in an environment with no further learning or planning, given (1) unsupervised (reward-free) experience in that same environment and (2) access to the true reward function at inference time. The paper examines two approaches for zero-shot RL, one based on successor features and one based on forward-backward representations. The paper empirically evaluates these approaches on the unsupervised RL benchmark suite, demonstrating that while SF approaches perform inconsistently, the FB approach performs well on all tasks and approaches performance of an agent trained from scratch with rewards.

The reviewers were generally positive about the paper, citing the systematic, extensive evaluation of different SF representation learning methods plus FB and the strong, significant results. The reviewers initially had requests for clarifications which were addressed in the rebuttal both via explanation and the addition of some further experiments and analysis. I found the results very impressive and believe this will be an impactful paper if it is presented at ICLR; I therefore recommend acceptance.
justification_for_why_not_higher_score: While the results in the paper are great, the experiments are only conducted on a specific class of methods (Mujoco continuous control in deterministic environments). The paper would be more impactful if it could also demonstrate similar results in other types of environments (e.g. game-like environments). Additionally, it assumes access to samples of the reward function, and a good pre-training distribution, which somewhat limit the generality.
justification_for_why_not_lower_score: The results in the paper are very impressive (85% of what supervised offline TD3 achieves, *without* requiring rewards during pretraining), and it will no doubt be an impactful paper for the RL community.
Decision: Accept: notable-top-25%