Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a behavior similarity-based representation learning method that improves the generalization performance in an offline RL setting. Two key ideas are presented here: 
- (1) the proposed behavior similarity is based on the distance of GVF predictions between two states.
- (2) to get a robust estimation of the distance between two GVF signals, it is formulated as a cumulative distribution function under assumptions, which is further converted to K quantile bins and can be learned by self-supervised learning methods in multi-label classification tasks, as defined in Eq. (6).

Procgen and DM Control suite experiments were conducted to demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths
---------------------------------
- The problem setting of studying generalization using an offline dataset, unlike RAD, CURL, and bi-simulation Metric has significant importance in real-world problems.
- The authors demonstrate sufficient knowledge in generalizable RL literature. I really enjoy reading the introductory part.
- The proposed behavior similarity based on GVF predictions makes sense to me that should demonstrate better generalization performance.
- The conversion of robust GVF distance estimation problem to a multi-label (as K bins) self-supervised learning task, is quite nice.

Weaknesses
------------------------------
- The generality of GVF functions. I am concerned about how to specify the cumulant of the GVF function, as this is the key part that determines the generalization performance (as the authors mentioned in lines 314 ~ 319). After checking the paper, specifying the cumulants seem to still require pre-defined using some kind of human heuristic knowledge. Will the authors explain how to resolve the concern of generality of hand-specified cumulants?
- The performance gain (Fig. 8 in Appendix, page 21) seems not significant. When comparing the proposed method to a vanilla PPO in the Procgen benchmark seems very small, and most tasks do not see a significant performance improvement, except for Caveflyer, Leaper, and plunder. Will the authors help to explain any reasons for that?

questions: - (1) Cumulant specification question as I raised in the Weakness part.
- (2) The performance gain issue in Fig. 8, as I raised in the Weakness part.
- (3) Besides the cumulant function definition, we know that the GVF function is also dependent on a target policy, could the authors explain to me how the target policy is defined?
    - If it is different from the behavior policy that collects samples to compose the offline dataset, can the authors explain to me further how to tackle the off-policy learning issue here when you use the TD loss, as shown in Fig. 2?

- (4) I can see the full loss function contains both TD loss and GSF loss in Fig. 2, but there is no section, nor an algorithm that gives a complete description of how exactly the learning process happens. I can find the TD loss in Algorithm 1, line 6, but I am still wondering what is the complete algorithm solution as you summarized in Fig. 2, "Train Agent using L_TD + lambda * L_GSF". Could the authors help me to understand the complete training process?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a framework called Generalized Similarity Functions (GSF), which estimates the similarity between observations by the various future behaviors. The previous works tried to calculate the similarity based on the future action sequences, state distribution, value, etc; but the adequateness of those methods depends on the problem, and no method is optimal in all circumstances. To this end, GSF learns from some arbitrary c(s,a) function and results in a value functions of those functions. Authors argue that the proposed framework can include the previous methods by using appropriate c(s,a). After learning GSF, the paper augments the data by additional labels, i.e. quantile of GSF, and learns the latent space by minimizing InfoNCE loss.
strengths_and_weaknesses: - This paper proposes a framework that can capture any signal and use it to train the representation. It is a generalized framework that can recover a number of previous frameworks. In that sense, it is somewhat original but not something that is completely new.

- The paper seems to have OK-ish technical quality, but some stuff is not clearly written and is hard to follow (e.g. definitions of $c$, $G$, $F$). The experiment session is also somewhat not easy to understand (what is the optimality gap in figure 4?)

- This paper will have some impact on the researchers in this field, due to the good performance and the proposal of the benchmarks.
questions: - It seems $c(s,a)$ is proposed to be a vector-valued function to recover some cases, e.g. successor features, CSSC. $G$ is a cumulative sum of $c$, so it will be vector-valued when $c$ is vector-valued. However, below definition 1, it is explained that $G_i$ is learned for each distinct POMDP. Is it only the case when $G$ is learned based on reward? Or is $G$ matrix-valued when $c$ is vector-valued? Either way, the description does not seem very clear.

- The aggregated version $I(k)=\cup_{i=1}^m I_i(k)$ does not seem to be used. The definition of $l_i(o)=\max_k \mathbb{1}_{o\in I_i(k)}$ is also strange; is it $\arg \max$? Based on a number of quantiles based on different dimension of $G$, how exactly is $l_i$ computed?

- It is hard to understand how CSSC is recovered (mainly due to the lack of understanding on $l_i$). Can it be exactly recovered when we have large enough $K$? Can you also elaborate more on how to recover PSE precisely?

- In the experiment section, the proposed framework shows much performance improvement on the Procgen benchmark (almost double the performance of CQL), whereas other algorithms perform much worse than CQL, or are barely comparable. Is GSF the only algorithm that is based on CQL? What happens if we use other representation learning algorithms with CQL? (or is it already done in that way?)

- On the other hand, in distracting control suite, GSF only shows a very small improvement (~=4%). What is the main reason for being this bad compared to the Procgen benchmark?

- What happens if we use all these different similarity metrics at the same time? Can the algorithm find out the best one to use? Can we know what similarity metric to use in advance?
limitations: The authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a novel representation learning method Generalized Similarity Functions (GSF) to improve the generalization ability of offline RL. GSF clusters latent states with similar future behaviors quantified by the order statistics of Generalized Value Function (GVF). Empirical results on the offline Procgen benchmark and the offline Distracting Control Suite show that GSF successfully enhances the performance on unseen tasks.
strengths_and_weaknesses: **Strengths:**
* The problem of zero-shot generalization in offline RL is significant, and the idea of GSF seems novel.
* The choice of hyper-parameters has some theoretical analysis.
* The paper conducted experiments in both discrete and continuous benchmarks, and the experimental results in both offline and online settings demonstrate the effectiveness of the proposed method.

**Weaknesses:**
* Some related works are missing. [1] and [2] both introduce representation clustering methods for offline multi-task/meta RL and can generalize to unseen tasks.
* There is no visualization of the learned representations of different training and testing tasks, which could help readers better understand the effectiveness of GSF.
* The paper does not discuss the computation cost of GSF.

[1]  Li J, Vuong Q, Liu S, et al. Multi-task batch reinforcement learning with metric learning[J]. NeurIPS 2020.
[2] Li L, Yang R, Luo D. Focal: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization. ICLR 2021.
questions: * Q1: The authors may need to spend more effort on the related works such as [1][2].
* Q2: The authors are suggested to add a TSNE plot of the latent states of different training and testing tasks.
* Q3: A comparison of the computation cost is needed.


**Typos:**
1. line 185: you may need to delete "the" in "by its the value function". 
2. line 233: two repeated words "used".

limitations: The authors state that the limitation of the proposed method is the search of hyper-parameters.
review_title: null

Meta-review: metareview: This paper studies an interesting problem, and overall the reviewers agreed the exposition and validation are sufficient. We encourage the authors to consider the issues raised by the reviewers and further improve the work in the final version.
confidence: Certain
Decision: Accept