Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is in the realm of generalizable reward learning and the aim is to learn value-discriminative visual representation from a diverse set of human manipulation demonstrations by contrasting conditioned on goals. The learned representation is used for constructing reward as similarity in embedded space between adjacent steps. The proposed time contrastive method differs from standard unidirectional contrastive learning, it attracts initial and final frames while repelling intermediate frames, thus presenting smoothly decreased embedding distances. The method is evaluated on trajectory optimization, online RL, and real-world offline RL tasks to demonstrate effectiveness of the visual representation over other pretraining baselines.
strength_and_weaknesses: Strength:
1. The motivation of learning visual reward functions from human demonstrations without action annotations and then generalizing it to unseen robotic control tasks is intriguing.
2. The proposed dual-objective goal-conditioned contrastive learning enables more accurate and smooth embedding distance through full demonstration.
3. The paper is well-written and easy to follow.
Weakness:
1. In Section 5.2, Table 1 shows that VIP in-domain representation learning is worse than Scratch-BC and even has no task progress due to overfitting. Is VIP easily affected by the quality of the demonstration? What if sub-optimal or not diverse enough, will it hinder RL update? 
2. Does VIP exceed in-domain imitation learning metrics? It would be more convincible if you compare your metric with IL baselines trained on in-domain demonstrations in Kitchen benchmarks, for example, behavior transformer, Implicit Behavioral Cloning.
3. Is your model able to handle non-Markovian demonstrations where different actions concatenate sequentially, instead of single clips?
4. On your project website, robot motion in MPPI trajectory seems to be fluctuating and unstable, is there any explanation or improvement for this problem?
clarity,_quality,_novelty_and_reproducibility: - Clarity: The proposed method and evaluation are clearly clarified.
- Quality: The paper is well-written and structured.
- Novelty And Reproducibility: The paper solves the problem of reward designing in complex robotic control domains by leveraging out-of-domain human demonstrations and goal frame instructions, contrastively learning a visual representation that implicitly encodes value function. The idea is interesting, significant, and new. Details of method implementation are clear and thus make it reproducible.
summary_of_the_review: The paper proposes an efficient method for visual reward and representation learning from out-of-domain data, which highly benefits downstream tasks. I would say it is an improvement to existing work but can be a good inspiration on how to contrastively learn a representation of demonstrations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents Value-implicit pretraining (VIP) which is a method to train representations from images for use in robotic reinforcement learning.  The method uses contrastive learning to generate smooth representations of trajectories, where the start and goal states of trajectories are encouraged by the loss to be close in the representation space and intermediate observations are encouraged to interpolate smoothly between the two.  Because the representation is trained to interpolate between start end goal states, the authors argue that this interpolation is equivalent to a value function when the reward is a goal-reaching reward.  The representation can thus be used to assess distance to the goal state, and thus can be used as a dense reward.  The paper shows strong empirical results generalizing representations learned in egocentric human video data sets to robotic tasks.
strength_and_weaknesses: The paper presents very compelling empirical results, and a clear derivation and description of the method.  The method is novel, and takes large steps toward solving very difficult problems in reinforcement learning and robotics.  The literature is carefully and extensively surveyed, and the contributions of VIP are clearly laid out in contrast to the relevant prior methods.  The method is also straightforward, and can be summarized by a three step algorithm, which suggests it can be easily reproduced.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, striking a good balance between technical detail and understanding for a broader audience.  The work is novel to the best of my knowledge, despite some similarities between VIP and other recent papers showing that goal reaching RL is related to contrastive objectives (and those approaches are cited and discussed in the paper).  The code is provided and although I have not combed it in fine detail, at first glance it seems clean and approachable.
summary_of_the_review: This is a very strong paper with an interesting approach to a difficult and important problem.  The empirical results are strong and I think this is a clear accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a method, VIP, to learn both an observation representation and reward using a self-supervised objective over an out-of-domain dataset. The representation learning problem is cast as offline goal-conditioned reinforcement learning for the tasks covered by the out-of-domain datasets, i.e. human activities. The reward used to learning the visual representation is 0 when the observed and goal states are the same otherwise it is -1. The reward function is selected to facilitate learning the rough distance between the current and goal frames. The representation learning objective is then cast as implicit time contrastive learning and roughly resembles InfoNCE, but without explicit negative examples. The value function for a given task is then the embedding distance between the current state and the goal state. The authors assess the impact of VIP against several representation learning baselines on trajectory optimization tasks, online RL tasks, and few-shot learning tasks to real-world robot manipulation tasks. Across experimental conditions, VIP outperforms the baselines.
strength_and_weaknesses: Strengths:
- The authors conduct extensive experiments to assess the performance of VIP and to understand the benefits of each of the proposed components. In addition to conducting experiments in simulation, the authors assess the performance of VIP in a real world robot setting.
- The authors evaluate useful and relevant baselines for learning observation representations. 
- The results demonstrate strong gains on prior work at the task of learning from videos of humans, which is the majority of activity-specific data available.
- Although in the appendix, the authors provide proofs alongside their proposed VIP components.
- The paper is well written.

Weaknesses:
- Large portions of the paper are punted to the appendix, include two-thirds of the related work section, two whole experiments, limitations, and future work. Despite working in a goal-conditioned RL setting, the authors punt the goal-conditioned RL related work to the appendix which makes it difficult to place the proposed method within the context of the relevant goal-conditioned RL prior work.
- It would be helpful to see how VIP performs against goal-conditioned RL where the observation encoding/representation is learned online alongside policy learning. 
- For the experiments in Section 5.2, how were the hyper-parameters selected? This is important to specify to make it clear that hyper-parameters favorable to VIP were not selected. 
- The need to not assess VIP under the closed-loop reinforcement learning setting is not well motivated, especially as goal-conditioned RL is deployed in such a setting. 
- Online RL results for VIP (sparse) where the policy is trained using the VIP state representation and the environment's reward do not appear to be included in the paper. There is no reference to where the results can be round. Additionally, it would be nice to have more of a discussion about why it is okay for VIP's state representation to be ineffective when learning from the ground truth reward function. This is where a comparison against an online, goal-conditioned RL algorithm would be beneficial to see. Are no methods able to solve the task in the online setting? Or only those methods evaluated?
- It would be helpful to specify the length of video sequences used to train the state representations and to assess the impact of video sequence length on performance. 
- It is not clear if contrastive RL (https://arxiv.org/pdf/2206.07568.pdf) is evaluated against and it would be interesting to see a discussion about the similarities and differences included in the paper. 
clarity,_quality,_novelty_and_reproducibility: In terms of clarity/quality:
- The paper is well written and easy to follow.
- It would have been helpful to include the rest of the related work section in the main body of the paper.

In terms of novelty:
- Overall the method seems novel, although an understanding of how VIP compares against contrastive RL would be helpful to make the exact novelty more clear. 
- The bulk of the method's novelty and contribution appears to come from its ability to enable to learning observation representations from out-of-domain datasets. 

In terms of reproducibility:
- The authors do not provide code, but extensive implementation details are provided in the appendix. The amount of information in the appendix should make reproduction possible, however I did not try and cannot say for certain.
summary_of_the_review: Overall the paper is well written, the results demonstrate clear gains on prior work, and both empirical and theoretical evaluation is technically sound. The ability to learn representations that enables few shot learning from out-of-domain, offline data can have a large impact on the field. The largest concern with the paper is the amount of information that was punted to the related work section. The related work section and limitations are key portions of a paper to appropriately contextualize and understand the method's contribution and, therefore, that should be included in the paper's main body.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a self-supervised visual backbone pretraining method for reward learning in control. The key idea is to formulate representation learning from egocentric videos as an offline goal-conditioned reinforcement learning problem, whose dual-form indicated a way to implement Value-Implicit Pre-training (VIP) on unlabeled human videos. In each training iteration, VIP samples a minibatch of sub-trajectories from video, and the loss function has two components, where the first one minimizes the L2 distance of the representation of initial and goal frames, and the last one optimize the difference of L2 distance of two consecutive intermediate frames (e.g. $\cdots \gamma (F(o_{k+1})-F(o_k))$, $F$ measures the L2 distance to the goal frame embedding.). In another word, this implicit time contrastive
learning push together the feature of start/end frame, and pull away the intermediate frames implicitly. 
Besides the elegant theory, the author pretrained their visual representation model on large-scale Ego4D human videos, and show the proposed pretraining idea can be combined with various on-domain solutions and significantly outperform all prior pre-trained representations on a set of simulated and real-robot tasks. A partial PyTorch code is provided in Appendix. 
strength_and_weaknesses: (+) The main logic of this paper is to derive a reinforcement learning target as self-supervised pre-training tasks, and use it for reinforcement learning. Therefore, it gives less domain gap in visual representation learning than in traditional CV-pretraining solutions. 

(+) The method is able to compare with R3M, which also works on visual representation for robot learning. VIP does not require extra annotations such as narration and text alignment, so it is more flexible and generic. 

(+) Experiments on visual trajectory optimization and online RL aggregate show VIP outperforms prior pre-trained representations, on both easy and hard cases.

(+) On a suite of real-world robot tasks, VIP can support few-shot offline reinforcement learning with smoother embedding distance curves.

(-) Although the derived pre-training target has a complicated form to implicitly optimize the value function, it is better to explain the physical meaning of each term. For example, the exponential term in log-expectation can be viewed as $$exp(||\phi(o)-\phi(g)||_2-\tilde \delta_g(o)-\gamma||\phi(o')-\phi(g)||_2)=\frac{exp((1-\gamma)||\phi(o)-\phi(g)||_2-\tilde \delta_g(o))}{exp((\gamma)(||\phi(o')-\phi(g)||_2-||\phi(o)-\phi(g)||_2))}$$
then maximizing the denominator is to maximize the one-step temporal difference, which is contradictory to the experimental observation. 
clarity,_quality,_novelty_and_reproducibility: This paper has good clarity and great quality and novelty. They also promise to release the pre-trained VIP code for reproducibility. 
summary_of_the_review: It is a novel idea to derive a self-supervised pre-training target from a goal-conditioned value function from RL, and the experiments are clear and supportive. I vote to accept this good paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: About generalizable reward learning and its aim is to learn value-discriminative visual representation from a diverse set of human manipulation demonstrations by contrasting conditioned on goals. The method uses contrastive learning to generate smooth representations of trajectories, where the start and goal states of trajectories are encouraged by the loss to be close in the representation space and intermediate observations are encouraged to interpolate smoothly between the two. The paper shows strong empirical results generalizing representations learned in egocentric human video data sets to robotic tasks.

+ compelling empirical results on challenging benchmarks;
+ The method is also straightforward, and can be summarized by a three-step algorithm, which suggests it can be easily reproduced.

A few technical concerns raised by reviewers and the authors addressed them to a certain extent. 
justification_for_why_not_higher_score: Although a very strong paper, I feel the scope of the work should still be considered incremental (a significant increment). Not ground-breaking. 
justification_for_why_not_lower_score: As mentioned, it is a strong paper with both theoretical support and thorough empirical studies.
Decision: Accept: notable-top-25%