Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors have proposed a well-motivated contrastive learning method for solving goal-conditioned RL tasks, thus removing the need for any auxiliary losses or data augmentations for representation learning. The authors then show how this contrastive objective leads to a critic implicitly learning the Q-value function, and empirically demonstrate the effectiveness of their simple approach on many tasks. 
strengths_and_weaknesses: # Strengths

1. The proposed method simplifies the representation learning + planning in RL problem, by using the inner product of learnt representations as a correlate for the Q-value function.
2. This is especially useful for the pixel-based environments, for which the authors have shown performance gains in many continuous-control tasks.
3. I also appreciate that the authors mentioned certain failed experiments in Appendix H, which help in better understanding of the proposed method.
4. The paper presents theoretical motivation for their method and show how this contrastive objective relates to a critic in deep RL methods. 
5. Thorough comparison with several related works.

# Weaknesses

1. It is unclear why *random goals* are sampled to train the actor loss? Can the authors shed more intuition on this decision?
2. Can the authors also comment if there’s some minimum goal dimension, after which contrastive RL starts performing better than the model-based baseline?
3. In Figure 5, while contrastive RL and its variants surely perform better in simpler state-based tasks, however, looking at the plots for image-based observation tasks (Fig 5(b)), it seems that C-learning alone does quite well, especially in the challenging tasks of Sawyer bin. Also given that in fetch push (Fig 5(b)), contrastive RL (NCE + C-learning) gets a huge boost over other the baseline, I think that’s also because of the addition of C-learning pipeline only. 
questions: 1. typos:
    - line 141: “The ~~objective~~ expected reward objective”
    - line 188: “critic is parametrized as an *inner* product” - but in Algorithm 1, the comment says that you performed an outer product.
    - line 268: “One” → “On”
    - line 879: “~~has~~ uses”
2. For the fetch and sawyer tasks, what is the dimension of goals (state-based)? I am assuming that for image-based, it is 64x64x3, right?
3. Could the authors change the color palette for the plots in Figure 5? Right now, it’s extremely difficult to judge which line corresponds to which baseline, especially on print.
limitations: Yes, the authors have addressed the major limitations of their method.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes leveraging contrastive learning to action-labeled trajectories, where the learned representation will connect to the goal-conditioned value functions. Specifically, they utilize contrastive learning to estimate the Q-function. The critic function (parameterized by the inner-product between representation) in contrastive learning can also be applied as the critic function in actor-critic algorithms in the RL context. Theoretical analysis of the convergence guarantees has been provided in the meantime. The empirical results suggest that the proposed framework can lead to better performances on goal-conditioned benchmarks compared with prior methods, including methods using goal-conditioned RL and different representation learning strategies. 
strengths_and_weaknesses: ### *Strengths*

#### **1. Originality and significance**

Although this work is built upon existing literature on goal-conditioned RL and contrastive learning (in RL), it has enough novelty since it provides a clear direction to link the reward maximization and learning representation using contrastive learning. The way offered in the paper to directly use contrastive learning to perform goal-conditoned RL is very simple and efficient. This work may have the potential to benefit the RL, representation learning, and robotics communities. 

#### **2. Relevance**

The authors discuss most related works, including goal-conditoned RL, representation learning for RL, and contrastive learning (for RL). Detailed comparison has also been given in the associated sections. 

#### **3. Algorithms, theories, and evualation**

The designed algorithm is decent, and I feel this can broadly apply to other RL regimes. The convergence guarantee makes it more solid. The experiments part is also very solid, including most relevant baselines and benchmarks. The appendix H on failed experiments is also very helpful for the community. 

### *Weaknesses*

Please note that below are only for questions and potential discussions. There is no need to rerun experiments during the rebuttal phase.

#### **1. About experiments**

It seems that, in some cases, improvements compared with C-learning are minor (Fig. 5). Can the authors give some explanation. Meanwhile, why not compare with other contrastive learning approaches in RL mentioned in the early sections?

#### **2. About other common challenges in goal-conditioned RL**

Can this approach tackle common challenges in goal-conditioned RL, especially for off-line datasets? e.g., the generalization ability to different distributions.

#### **3. Minor typos**

-> Line 107: $\pi\left(\tau \mid S_{t}\right)$ -> $\pi\left(\tau \mid S_{g}\right)$;

-> Eq. 4: consider using another color, the current $v^{+}$ is hard to recognize when print out the paper to read. 

questions: Please note that below are only for questions and potential discussions. There is no need to rerun experiments during the rebuttal phase.

**1. About the required goal**

In this work, the goal is the state/observation given by the task. Can this requirement be looser?

**2. About the generalization abilities**

Can the learned representation be generalized to a set of diverse tasks with only a smaller number of interaction data?


limitations: The authors mentioned the limitations in the paper, mainly about how to generalize the work into other RL settings. I think this would be the potential direction for future works. I raise a few questions, which are more like vague points instead of limitations. I think this is a decent paper and I would vote for acceptance.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a novel view of how contrastive learning can be used by itself, and without any additional RL training on top, to learn goal conditioned policies from pre-collected data. This is achieved by using the similarity learned during contrastive learning as a Q-function, that is then used to improve a policy in the policy gradient step. It is also shown, via a proof, that the contrastive objective estimates the Q-function, and some additional convergence guarantees are provided. The authors well motivate their approach, compare it extensively with related work on learning goal-conditioned policies, as well as works that use an additional representation learning objective on top of a RL algorithm. The paper shows a good comparison between prior work and the proposed method on a range of goal-conditioned RL tasks that have been used in these prior works. The overall conclusion is that a much simpler method, like the one presented, can perform similarly well, and in some cases even better, than more complicated baselines.
strengths_and_weaknesses: ### Strengths
1. The paper presentation and clarity is excellent and of high quality. The authors have performed an extensive evaluation of their method and compared it against two sets of baselines, one based on representation learning methods and one based on learning goal-conditioned policies. The authors describe their method, motivation and results very clearly and do a very good job at relating it to prior work.
2. Their method is grounded in theory, though I cannot claim I could follow 100% of their proofs and arguments as I am not an expert on the theoretical side.
3. Simplicity of the method, while simultaneously performing similarly well to other methods, and in some cases outperforming strong and more complicated baselines.
4. The authors present an original perspective on how contrastive learning can be used to learn goal-conditioned policies directly.

I did not find any noteworthy weakness to the paper other than some typos and some more questions about the limits of this method that I have included in the next section with Questions.

questions: One question I would be interested in, is how well the method would perform in partially observable, large environments where the agent has a first-person view.Goal conditioning from pixels seems to be much easier in situations with a fixed camera covering the majority of the possible states, but it seems that the difficulty of the problem quickly grows when we move to first-person views in large environments with partial observability.
I’m also wondering how well the trained method generalizes zero-shot to changes in the color of the background, the setup of the hand (i.e. having the base of the sawyer arm on the left side of the image) or to new types/colors of objects. 

In Figure11 in the appendix, the colour seems to be wrong way round in the plot of the maze, as the positions in the centre have the colour corresponding to the highest "steps to centre". The colour for the tSNE plots seems to be correct though.

Two typos:
Line 154: “The negative” -> For the negative
Line 215: “interation” -> iteration
limitations:  The limitations have not been added to the author’s submission checklist yet, but are addressed in the actual paper.
review_title: null

Meta-review: metareview: How to design RL algorithms that directly acquire good representations? This paper gives an answer that contrastive representation learning can be cast as a goal-conditioned RL using the inner product of learned representations.
The technical novelty of this paper is sound, with the thorough theoretic motivation of the proposed method and solid experiments. The presentation of this paper is also satisfactory.
All the reviewers provided positive feedback on this paper. I also enjoy reading this paper.

confidence: Certain
Decision: Accept