Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes improving the generalization performance offline RL algorithms by using a new approach to aggregate observations based on the similarity of their expected future behavior.
main_review: The idea of aggregating observations based on behavior similarity has already been explored in some papers, but the approach proposed in this paper seem to be new.
While the paper discusses its relation to CSSC and PSE and compares with these two methods in the experiments too, it misses a discussion or empirical comparison with CTRL. In addition, it is not clear whether the proposed algorithm's advantage over CSSC and PSE in the experiments is due to the use of a better aggregation method, which is the main claimed novelty. Can other aggregation methods be combined with CQL?

The problem setup and the motivation are not clear and often confusing. 
* There is a consistent confusion of MDP and POMDP throughout the paper. For example, in the introduction, the paper claims improving zero-shot generalization across a family of POMDPs, but the cited reference is Puterman's article on MDP. It then sometimes called the models POMDPs or MDPs. 
* In the historical dataset ${\cal D}^{\mu}$, what is required for $\mu$?
* What is a concrete example of the problem setting described in Section 3.1? In the motivating example, it seems the first images for Level 1 and Level 2 should be considered similar, but it is not clear why. The main paper states that they have near identical true latent states and value functions, but didn't explain why.
* It's not clear what the different MDP/POMDP models are in the experiments.

The technical writing for the proposed approach requires a careful revision too. Here are some issues:
- The term cumulant function is confusing as cumulant function is a well-known concept in statistics.
- Section 3.1: for POMDP, a policy is usually maps a belief to an action, not an observation to an action. Also note that the Q-function in Eq. (1) is a function of the state and the action, and thus not directly computable because the state is not observed.
- $c$'s output may be a vector, and $|c(o, a)|$ and $\sup_{o, a} c(o, a)$ need to be defined.
- Alg. 1: D[i] in line 1 is undefined; line 6 works only for scalar c; line 7 is not clear.
- Eq. (3): RHS depend on $i$ and $j$, but LHS does not. Where is the $d_{\mu}$ used?
- $c_{1,\max}^{\mu}$ and $c^{\mu}_{2, max}$ are undefined.
- In $F_{i}(g) = \mathbb{P}[G_{i}(o_{t}) \le g]$, is $o_{t}$ treated as a random variable? What is the underlying distribution used? Is $G_{i}$ the same as $G_{i}^{\mu}$?
- Eq. (4): what does $G_{i} \sim {\cal D}^{\mu}$ mean? Note that ${\cal D}^{\mu}$ is defined as a set of $(o, a, r)$ triplets.
- Eq. (6): $i$ appears on the RHS but not on the LHS.
- Eq. (7): $l(z)$ and $o \sim {\cal D}$ undefined. Does ${\cal D}$ refer to ${\cal D}^{\mu}$?
- Eq. (8): $(s, a, r) \sim {\cal D}^{\mu}$? 
- Alg. 1: should line 10 be inside the for loop? Line 11 is not clear.

The representation learned by DeepMDP is closely connected to bisimulation, but it doesn't learn bisimulation relations.

**Post-rebutall**

I appreciate the clarification and the additional results. I've raised my score in view of these changes. It'd be helpful to further stengthen the experiments and improve the writing. The writing is generally clear now, but there are still quite a few issues (e.g., $o_{1}$ and $o_{2}$ in Eq. (3) now have superscripts, but this now is inconsistent with other ocurrences of $o_{1}$ and $o_{2}$ around Eq. (3)), and a careful revision will help readers to better understand the idea. For the experiments, it seems the baselines are not fine-tuned, and it's also not clear how the online algorithms are set-up for the offline setting. Providing details on how CSSC/PSE/CTRL is combined with CQL will also be helpful. In addition, comparison to existing offline algorithms will be helpful (e.g. refer to Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems by Levine et al.).


summary_of_the_review: The proposed approach seems novel, but the motivation and presentation of the idea is confusing. It is not clerar whether the reported performance improvement is due to the proposed aggregation method contributes, and further discussion or experiments with related works may be needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors build on previous work which regularized states with similar value-function or future action sequences to have similar representations, by instead looking at successor features, i.e. the accumulated value of future cumulants. To deal with continuous value function whose scale can vary accross environments, they apply a quantile-binning technique. They also propose an offline version of Procgen as benchmark.
main_review: The proposed method is an interesting method for defining 'similarity' which can then be used for contrastive learning. The paper is well written and does not appear to have any technical or theoretical flaws. The topic of generalization in RL is relevant and the proposed method is a valuable contribution to this area. 

One disadvantage of this method is that it requires the ability to pre-traing the generalized value function, hence the authors' decision to only apply it to offline RL. This leads to the, imo, main weakness of the paper: While the authors evaluate the method against a range of relevant baselines, they only do so in one domain, namely their newly proposed offline procgen benchmark. Hence the results will depend heavily on the authors effort to not only tune the hyperparameters of their own method, but also those of the baselines. Unfortunately, there is no information in the paper how the hyperparameters were chosen (they do provide the final hyperparameters of their own method only), so it's hard to evaluate how much one should trust the experimental evidence.

Hence, I think the paper could be strengthend significantly by evaluating on additional domains and/or providing detailed information about they applied hyperparameter search.

Question/Nitpick: 
* Can you still call it contrastive loss? Eq. (7) just seems to be classification? 
summary_of_the_review: Overall interesting method, but more experimental detail needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the problem of generalization and representation learning in reinforcement learning.
Concretely, the authors propose an approach of self-supervision for improving offline-RL based on similarity learning over the considered state space.
In this context, the concept of generalized value function is introduced.
The similarity measure is defined as a classification problem over quantized latent space.
The paper is evaluated in an offline setting of Procgen which is a dataset for multi-task reinforcement learning generalization evaluation.
More particularly, the proposed approach claims to be particularly efficient in improving zero-shot generalization performance on one offline RL benchmark, offline Procgen.
The offline Procgen dataset is a second contribution of the paper.
main_review: Strenght:
* The paper is clearly written and the various choices are reasonably justified
* The proposition of an offline version of Procgen is a valuable asset for the offline RL research community
* the approach of quantization in this context is novel to the best of my knowledge
* The experimental results seem significant for the considered benchmark dataset
* the introduction of generalization value function seem novel and pertinent

Weakness:
* The comparison to cURL (https://arxiv.org/abs/2004.04136) is missing, especially considering the fact that this approach is also leveraging metric learning in latent state space.
* The D4RL dataset would have been useful to compare too, especially to evaluate over the latent recent corpus of approaches of offline RL.
* The approach relies on PPO, which is an online RL algorithm, and CQL which is a conservative approach of Q-Learning. Maybe it would have been interesting to evaluate how the method behaves with SAC also.
summary_of_the_review: The paper is addressing an important problem of sequential decision learning research.
The proposing is interesting while novelty to existing approach could be improved.
The proposition of a new dataset for zero-shot generalization of sequential decision-making in an offline context is valuable.
The comparison on another popular benchmark like D4RL and against the cURL approach would have been appreciated.
The introduction of generalized value function seems novel and pertinent in the context of self-supervised reinforcement learning and is clearly illustrated in the proposal of this work and could be valuable to the state of the art.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studied  zero-shot generalization in an offline reinforcement learning setting. The authors proposed to improve the generalization via a better representation learning. Specifically, the authors hypothesized that observation with similar future behaviors should be assigned to similar representations. To this end, the generalized similarity function (GSF) that aggregates latent representations with the future behavior is proposed. The proposed approach is evaluated on an offline version of Procgen. The proposed approach outperforms baselines across different tasks in Procgen. 

main_review: Strength:

- RL generalization in an offline setting is an important and interesting topic for our community. 
- The approach is well-motivated by the example in Section 4. 
- The proposed GVF is a general framework that can recover objectives of existing works. 
- The proposed approach outperforms baselines such as CQL, CCSC and PSE. 

Weakness: 

- The reviewer found the approach section is not easy to follow. Some technical details seem missing. Please see detailed comments below:  
    - The latent state decoder f seems to be a critical part of the proposed pipeline. Could you elaborate on how you learn this decoder?
    - Definition 1 states that the cumulant function c can be any bounded function over R^d. It is unclear to the reviewer how to select the cumulant function c. What cumulant function is used in the experiments. Does each task require a different cumulant function?   


- In addition, the reviewer found the notations confusing. Please see detailed comments below:
   - In line 2 of Algorithm 1, what does D[i] represent? What is the difference between D[i] and D^{\mu}
   - In line 6 of algorithm 1, does G represent GVF? If yes, why does it take a latent representation as input? According to Definition 1, shouldn’t GVF take an observation o as input?
   - "Target parameter" and "online parameter" are not formally defined
   - $\beta$ in line 7 of algorithm 1 is undefined?     


- Experiments  
  - The authors discussed F-BRC (Kostrikov et al., 2021) and MOReL (Kidambi et al., 2020), which are state-of-the-art offline RL approaches, in Section 2. However, there is no comparison with the aforementioned existing works provided. The experimental section could be more convincing if the comparison with F-BRC and MOReL is provided.   

  - Recent works [a] show that data augmentation only could significantly improve the generalization capability of RL agents. The proposed approach also uses data augmentation during training (Line 4 of Algorithm 1).  It would be interesting to see an ablation study on the data augmentation and the proposed GVF.  

[a] Reinforcement Learning with Augmented Data, Laskin et al. NeurIPS 2020

summary_of_the_review: The reviewer thinks the topic of this paper is significant. The approach seems interesting. However, the reviewer has some concerns about the experimental results and the clarity of the presentation.
review_title: null

Meta-review: The paper proposes a new offline RL technique to generalize across domains. The paper was initially confusing (i.e., MDP vs POMDP) and weak empirically.  The authors greatly improved the paper.  However, a the end of the day, it is still not clear why the proposed approach performs better than existing techniques.  We can think of the cumulant function with the discrete labels as essentially computing some statistics of future actions, observations and rewards.  This is what every self supervised technique does.  They differ in terms of their particular choice of statistics and architecture.  The paper does not sufficiently motivate the particular architecture.  Interestingly, in the experiments, the best statistics are cumulative rewards, which are closely related to the Q-values. In that case, it is even less clear why the approach should be beneficial since RL techniques that generalize across domains by learning state representations to predict Q-values seem very closely related.  

Despite the updates to the paper, the POMDP references are still confusing.  The issue is that the paper embeds observations as if they were sufficient to predict future observations and rewards.  This corresponds to the memoryless approach where a policy is optimized based on the last observation instead of the history of past actions and observations.  Memoryless strategies are effective only when the last observation is a sufficient statistic, meaning that we really have a (near) fully observable MDP.  The paper should discuss this and acknowledge that the approach will suffer in domains where memory of past actions and observations is critical.
Decision: Reject