Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces Contrastive Value Learning (CVL) for offline reinforcement learning (RL). Unlike traditional model-based or model-free methods, CVL estimates Q-values through a contrastive learning process. It leverages a contrastive loss function with a geometric distribution over time offsets to enable implicit modeling of the value function. To achieve computational efficiency, the algorithm utilizes random Fourier features. The experiments show that the proposed method obtains promising results on large benchmarks. It is also claimed that the proposed method can benefit from pretraining on unlabeled data.


review: ## Strengths

The paper is well-structured, provides a clear explanation of the proposed method, and presents good experimental results. I appreciate that there are many discussions on the relationship between the proposed method and prior works. 

## Weaknesses

Please see my questions for rebuttal in the below section.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well-written and easy to follow. However, I have questions about the problem statement and experiment settings listed in the above section. I gave my recommendation based on the current version of the paper. However, I will be happy to update it after my questions are resolved.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: Inspired by the large data regime, offline reinforcement learning (RL) changes the way the agent is being learned via trial and error by learning directly from existing data as in supervised learning. One way to find a policy that well-support the training data is by applying model-based RL approaches. Learning a dynamical system that predicts a one-step state prediction is usually challenging, especially with high dimensional inputs. 

In this work, an alternative model is proposed to support implicit multi-step predictions while modeling the value of each action. Hence, the approach is not suffering from the challenge of predicting a high-dimensional state. This approach decouples the learning of value-like quantity into learning the future state occupancy measure using contrastive learning and then re-weighting it with the reward samples.

This approach outperforms other offline RL baselines on complex robotics benchmarks (MetaWorld).
review: One of the strength points of this work is that learning the future state occupancy measure does not require any TD learning. Second, this approach releases the burden of predicting high dimensional states by learning a quantity proportional to the likelihood of visiting future states. Moreover, the pre-training study shows the capability of CVL for learning some common representations across related tasks. 

As mentioned, this approach is similar to the successor representation in the sense of decoupling the learning of the value function in terms of some fixed features and weighted by the reward signals. However, an ablation study that compares the two approaches does not exist. Also, to verify the claim mentioned in the paper "While CVL is similar in spirit to the successor representation [14, 13], the occupancy measure learned by CVL is much richer than that of SR, as it captures the entire distribution over future states instead of only the first moment".
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: To summarize, I have only one recommendation for comparing CVL with SR if applicable. I am really excited to see how well CVL can outperform the capabilities of SR. 

UPDATE: After discussion, I believe the paper should be accepted.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a novel way, named contrastive value learning, to learn the Q function, in which they decompose the Q function by first approximating the occupancy measure using an implicit function, then representing the Q-value as the occupancy measure weighted by the reward signal. The effectiveness of the proposed method is demonstrated on multiple complex robotics benchmarks.
review: Strengths: 
- This paper proposes a novel way to learn the Q function. It decomposes the Q function in a two-step way, first, approximating the occupancy measure using an implicit function, then representing the Q-value as the occupancy measure weighted by the reward signal. This decomposition is novel and has benefits in offline settings where the Q value is learned without using TD learning.
- The proposed method shows obvious improvement over baselines on a set of offline RL tasks. 

Weakness:
- For the writing, I think there are several questions that could be clarified:
    1. The usage of critic/optimal critic causes unnecessary ambiguity. The critic is often used to represent value function in RL, for example, in SAC, but the "critic/optimal critic" in this paper differs. e.g., in Line 152, "Precisely, Equation (2) corresponds to using an importance-weighted estimator, where an optimal critic that minimizes Equation (6) approximates the density ratio from Equation (7)".
    2. Line 260: “In Section 6.3, we qualitatively assess the similarity between contrastive and true Q-values on the continuous Mountain Car environment”. There is no section 6.3 available. 
    3. Might be shown in the appendix, but I haven’t found Table 5 (Line 249) in your paper,
    4. Line 258: “Results presented in Section 5 show that CVL is also able to learn meaningful Q-values and achieve good empirical performance on hard image-based tasks”. From section 5, I can only find results on offline RL with images, while the analysis of the ability to learn meaningful Q-values is missing. 
    5. In Eq. (12), the equation is shortened by using $\xi(\pi)$, it would be better to give an intuitive explanation. Also, the transpose of $F_{w, b}(\phi(s_t, a_t))$ is missing.
 
- The paper proposes a way to estimate the Q values, but the analysis of the ability to learn meaningful Q-values is missing. Actually, from the paper, I can not find a thorough discussion of the quality of the learned Q compared to the Q function trained in a regular way.
- To calculate Eq. (6), the negative samples are very important according to previous results, however, the way to draw negative samples is not very clear in the paper.
- In Eq.(12), I can understand that the usage of RFF approximation is to achieve efficient estimation, however, the paper doesn't offer enough details on how to implement it and the benefits or drawbacks of using this approximation.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes a novel way to learn the Q function, which especially fits the offline RL setting and offline-to-online RL. The proposed method demonstrates good performance on a set of offline RL tasks. I like the presented idea and the paper is generally easy to follow, however, the writing could be improved to minimize ambiguity and several important discussions or experiments are missing in the current version. Based on these concerns, I would suggest Weak Reject.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors propose CVL, an offline RL algorithms based on contrastive learning. An implicit dynamics model is learning via contrastive learning, which is then used to fit a contrastive Q function. This Q function is then used to update a policy. 
review: The paper poses an interesting idea of using contrastive learning in the offline RL setting.

However, I lean towards rejection for the following reasons:
- The paper does not have a clear robotics focus and would, therefore, be better suited for a different conference.
- I think the paper really lacks in clarity which for me makes it hard to follow and check the claims. 
   - Statements like "Solving Eq. 4 yields a critic" are just stated and not well explained. Personally, for me the connection is not obvious. This really is the crucial connection between how contrastive learning and RL can be connected and it should be made more clear.
   - At least in the preliminaries, the notation seems to slightly change, e.g. sometimes the length-K trajectory uses parenthesis, sometimes it uses curly brackets, sometimes the index k is used and other times it's t or delta t, ... . These inaccuracies make it harder to follow straight away.
   - The paper ends somewhat abruptly, where the discussion is very short and it feels like the authors just ran out of space and pushed a whole chunk to the appendix.
- Given that the CVL is presented as an offline-RL algorithm, I think it should be more rigorously benchmarked on standard offline RL tasks with more baselines, e.g. TD3+BC, IQL. Table 2, is an attempt at that, but it only compares against CQL on a small subset of the environments, which makes me somewhat suspicious. Further, there are no uncertainty quantifications in Table 2.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper has an interesting idea, but the presentation is unclear and confusion in a lot of places. The experiments the authors show are promising, but the authors created their own offline-RL benchmark and the evaluations on the classical benchmarks and baselines are lacking and, therefore, I have some suspicions.


=== After Rebuttal ===

I thank the authors for addressing and answering my questions. Based on this I am changing my score to a weak accept.
review_title: null

Meta-review: The paper introduces Contrastive Value Learning (CVL) for offline reinforcement learning of the Q-function. The ideas are interesting and experimental results are strong. The paper does not have real world robotics experiments. However, while some contributions in reinforcement learning are not necessarily directly applicable to robotics, offline reinforcement learning is one domain with a connection with real-world robots where offline data is often available, and, online data collection is expensive. Reviewers highlighted places where the writing could be clarified. The authors already modified the paper but I suggest to go through the paper again for the final version.
Decision: Accept (Poster)