Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel state distance metric to regularize the latent space of model-free reinforcement learning agents. The proposed RAP metric accounts for the variance of the rewards received in a pair of states under a stochastic policy. This enables the metric to be an upper bound on the difference between the values of a pair of states. The metric is added as a latent space regularizer to the soft actor-critic and evaluated in multiple high-frequency continuous-control environments.
strengths_and_weaknesses: Strengths:
* The paper contains a rigorous analysis of previously proposed behavioral metrics to motivate the proposed RAP metric.
* Excellent evaluation in multiple continuous control environments. Strong results in favor of the proposed metric in all domains.

Weaknesses:
* The derivation and interpretation of some equations could be made clearer.
* No analysis of the learned models. E.g. a visualization of the learned latent space should be included if the main contribution of this paper is a regularizer on the latent space.

Comments:

– It is not completely clear how we went from equation 12 to 13. Equation 13 should be described in more than one sentence.

– I believe the term \hat{\sigma(s_i)} is dependent on the policy \pi. Equation 13 might be misleading if the policy is not included in the notation.

– Equations 11 and 12 should be cleaned up. Using colors or other annotations to match the text and the terms in the equation would be helpful. 

– Line 193: bad spacing.

questions: 1. Is it straightforward to implement the RAP metric compared to MICo? Could you include a code snippet to compare the implementation of the two metrics?
2. I would like to see an ablation where the correction for the variance of the rewards in states s_i and s_j is removed. Would the ablation be similar / identical to MICo?
limitations: Limitations were not addressed. I would like to see a discussion of using the RAP metric with agents other than the soft actor-critic.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors focus on the behavior metric design problem so as to learn an informative and robust representation. To this end, three theoretical issues of the current behavior metrics are analyzed, including loss function mismatch, relaxation of dynamics model divergence, and limitation of the L1/L2 norm. Then a new metric named Reducing Approximation Gap (RAP) distance is proposed, and a practical representation learning algorithm is proposed based on RAP. Empirical results on three benchmarks show the effectiveness of the proposed method.
strengths_and_weaknesses: **originality**

The main contributions of this paper are clear. The authors carefully analyze the theoretical issue of current behavior metrics. Motivated by the above issues, a novel behavior metric for reducing the approximation gap from the perspective of the dynamics model. A thorough theoretical proof is given for the value function difference bound.

**quality**

The empirical results on three different domains, i.e. DM Control, Robot control, and autonomous driving are given. The performance curves validate the effectiveness of the proposed method. But For some environments, e.g. cheetah_run in DMC, the baseline curves are not consistent with the original paper.

**clarity**

The paper is well-written and easy to follow. Although this is a theoretical paper, the authors give a very clear background and the main problems of the current behavior metrics. These details help to understand what is the main contribution of the proposed RAP and why it gives a better theoretical property. But for the experimental part, more visualization of the learned latent space property will help to understand what is the difference between RAP over $\pi$-bisimulation metric and MICo.

**significance**

The representation learning plays an essential way in reinforcement learning (RL) for achieving generalizable and robust capacity. The behavior metrics-based research has drawn large attention in the RL community. The proposed method not only gives the theoretical shortcomings of the previous metrics but also provides a new metric with the better theoretical property.
questions: 1. What is the tightness of RAP bound in Eq. 10 compared with $\pi$-bisimulation metric [1] and MICo [2]? Can RAP bring a tighter bound?
2. Why the performance curves of baselines like DrQ in Figure 3 are lower than their original papers'? For example, DrQ can get 800 score in the original DrQ paper (Figure 3).
3. More visualization of the latent space property of RAP over $\pi$-bisimulation and MICo is preferred.

[1] Castro P S. Scalable methods for computing state similarity in deterministic markov decision processes[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(06): 10069-10076.

[2] Castro P S, Kastner T, Panangaden P, et al. MICo: Improved representations via sampling-based state similarity for Markov decision processes[J]. Advances in Neural Information Processing Systems, 2021, 34: 30113-30126.
limitations: No.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a representation learning approach based on behavioral metric. The core idea is to project states into an embedding space such that the distance between them w.r.t behavioral similar is preserved. Therefore any distractors or variability in the environments could be potentially soaked due to the flexibility in the loss function. This is contrary to L2 reconstruction based objectives as auxiliary tasks which will learn to capture task irrelevant details in each variable environmental condition. 

The core technical idea is to formulate the behavioral metric based approach as a new loss function. This new loss function is added as an auxiliary loss within an actor critic framework. In order to make the loss function tractable, two core assumptions are made: (1) reward expectations in the loss function of the behavioral metric are simplified to absolute errors in the rewards between two sampled states and (2) a new distance metric (RAP) is proposed, which leverages expectations over future states under the policy instead of expensive wasserstein based metrics over the transition model. 

The approach is validated on a variety of continuous control tasks. The main results is that the expected returns is higher with this method over baselines.
strengths_and_weaknesses: Strengths:

This method uses behavioral metrics for representation learning which can be more robust to environmental variations. It is an interesting and well grounded approach to handle all sorts of variabilities, especially visual distractors. Instead of hand crafting augmentations, this approach could in principle learn more robust policies under arbitrary pixel variations without apriori knowing the set of invariances and equivariances in the environment. 

So it might be a better approach where practitioners don't have good intuitions about the domain and therefore making it hard to build good data augmentations.

Weakness:
1. "Our method achieves comparable scores with data augmentation method DrQ, and its score is higher than behavioral metric DBC and MICo." -> This empirical finding suggests to me that the authors should try this method on domains where it is very difficult to design data augmentations manually. 

2. While the empirical performance is shown to be better on some environments, it is not clear why it works so well in some cases and not in others. There is no qualitative analysis so it is difficult to understand what this method is really learning.

questions: - Figure 3 and 4 have mixed results. It might be helpful to run on a more diverse variety of tasks to learn about strengths and weaknesses

- It might be helpful to analyze and show state similarities in a navigation or some 2D projected view of an environment to better understand behaviorally induced state similarities. Currently it is all hidden under a single empirical performance metric and it is very difficult to understand what is happening. 

- "we can learn a neural network approximator to estimate it by assuming that the reward rs on state s is Gaussian distributed.". This seems like a major assumption for many environments with discrete reward functions. What type of environments and problems would break under these assumptions? 

- "Similarly, in order to estimate the expected next states... we learn a dynamics model Pb taking input of state ... outputs a Gaussian distribution over the next state embedding". Is this assumption necessary? Can a neural network be used to estimate the successor representation trained via TD learning?

limitations: Discussed above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is about behavioral metrics and state representation. This paper proposes a behavioral distance, the RAP distance, and develops a practical representation learning algorithm on top of it. Experiments on DeepMind Control Suite with distraction, Robosuite, and autonomous driving simulator CARLA demonstrate promising results.
strengths_and_weaknesses: Post rebuttal: I increase my score to 6 (see discussion below)

---
Strengths:
- The results seem to be promising
- The paper is globally well written.

Weaknesses:
- Contributions: the method and the algorithm do not seem very clear because there are large discrepancies between the few theoretical results and the actual algorithm (see questions and limitations). The claim in the abstract "We provide theoretical analysis on the proposed algorithm." does not seem very accurate.
- Reproducibility: Concerning the code, is it proprietary or do you want to publish it? It seems that at least one of these two claims is wrong (1) "Did you include the license to the code and datasets? [No] The code and the data are proprietary." and (2) "Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We will publish code once paper is accepted."
- Presentation: The approach is only presented starting from page 6 and it is basically presented in two pages. It would be useful to go more directly and more in depth into the actual contributions of the paper.
- (minor) Typos/grammar: "we randomly sample a weather at each episode starts.", "Deep RL algorithms and attracts more and more attraction in the RL community".instead of "expensive" experiments, I suppose you meant "extensive"? (even though both might be true).

questions: In Equation 9, d is defined as a distance from a space of distance functions. 
- Can you confirm that some distance functions d wouldn't make a lot of sense? 
- Theorem 5.2 and theorem 5.3 uses the l_\infty loss but then in Section 5.2, a different distance function is used. How are the theorems useful for the practical algorithms?

limitations: As acknowledged by the authors, the limitations of the work are not discussed: "Did you describe the limitations of your work? [No]".

In addition to the lack of connexion between the theorems and the actual algorithm due to the metric d, an important characteristic/potential limitation of the RAP distance is that it depends on the policy. The explicit dependance on the policy is removed in Equation 13, why? How do you train the algorithm in practice given that samples might not be all obtained with one given known policy?
review_title: null

Meta-review: metareview: Unanimous accept from 4 reviewers

This paper was initially divisive among reviewers (scoring 7744), now 7756 post rebuttal

While the consensus of this representation learning work had initial strengths of clear analysis of previous metrics, clear evaluations, writing, and good experimentation; the prior weaknesses were mainly confusion about derivations and equation interpretations, no analysis of learned model, no visualizations of the latents space, lack of explanation why the method only does well in some environments and not others. These confusions have since been cleared up, in particular the authors have since added additional analysis in the appendix is useful (e.g. Table 4 and Fig 15) to satisfy reviewer's nvyx's concerns over clarifications about equations
confidence: Certain
Decision: Accept