Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper aims to train vision-based policies that can master a wide range of skills by learning from a large amount of diverse experience. The paper claims that learning a latent representation of the environment dynamics by modeling the predictive information (introduced by a prior work) result in policies that generalize better (zero-shot) to unseen tasks and for sim to real settings. The experimental results with up to 297 vision-based manipulation tasks (kitchen environment) support their claim. 

strengths_and_weaknesses: Strengths:
* The experimental results seem to support that modeling the predictive information can improve generalization. 
* It is a very well written and easy to read paper

Weaknesses:
* The theoretical contributions of the paper are minor. 
* The improvement compared to QT-Opt is not significant. Here is a summary for figure 3:
Move skill: QT-Opt: 25%, PI-QT-Opt: 30%
Pick skill:  QT-Opt: 43%, PI-QT-Opt: 48%
Knock skill: QT-Opt: 19%, PI-QT-Opt: 38% - It is not convincing why QT-Opt performs so poorly compared to Pick skill since the Knock skill must be much easier than pick. 
* It is not really clear how the outcome of this research can help other researchers to build more general policies in multi-task settings. It is difficult for other researchers to reproduce the setup or to have access to the computation power used in this experiment. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: It is still hard to believe that modeling a latent representation using predictive information can help since the success rate is still very low for both PI-QT-Opt and QT-Opt.
issues: Figure 1: using s to denote both proprioceptive state and image observation can be misleading.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work studies the effect of improving RL training performance with an auxiliary loss for the task of finding the mutual information between past and future states. The specific technique for the auxiliary task is using a Conditional Entropy Bottleneck, which is a representation learning task previously shown to improve RL training in "predictive information accelerates learning in RL" [2]. Unlike in [2], this paper's focus is on augmenting QT-Opt with the auxiliary loss (as opposed to augmenting SAC) for the purpose of vision-based task-conditioned robotic object manipulation (as opposed to simpler simulated continuous tasks), with task conditioning allowing a single agent to be trained for hundreds of different tasks. The main contribution of this work are the empirical results of applying the ideas from [2] to this different setting, as well as the demonstration of multi-task learning by task condition through images or language. 
strengths_and_weaknesses: Strengths:
* The paper is well written and is easy to follow. For the most part there is no missing information in the explanation of related work, technical details, and experimental setup details.
* The dual contributions of extending the findings of [2] to complex task-conditioned manipulation and of demonstrating a new technique for task-conditioning QT-Opt are of significant value to the field. 
* The set of experiments and the results observed convincingly demonstrate this auxiliary loss does help with training, and more significantly appears to improve real-world performance by a large margin. While this primarily shows 
* The papers notes some significant limitations and how they might be addressed in future work

Weaknesses:
* Results for the knock skills are confusingly different from the other ones, with lesser granularity and significantly larger benefits from using the auxiliary loss. 
* Apart from section 5.3, there is not much analysis of the results. Section D from the appendix does present some ablation results, which I feel would have been better in the main paper. 
* The (e), (f), and (g) rows of Figure 2 have really small images that are hard to understand. 
* It is stated that "With our best effort, we were unable to get SAC and PI-SAC working on our manipulation tasks." Given this work specifically builds on [2], I would have liked to see more discussion or analysis of why QT-Opt with PI worked better than PI-SAC. 
* There is very little discussion of task-conditioning, and no citations of prior works on the topic. Given the second claimed contribution is "We show that simple forms of task conditioning are sufficient to allow QT-Opt", it feels like more discussion of related ideas and evaluation of how well task conditioning actually works are needed.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe this work presents valuable empirical results that involved a substantial amount of experimentation that shows the the key idea from  [2] applies to the robotics domain. However, given the contributions are primarily about empirical results rather than technical ideas, I would have liked to see significantly more analysis of the results that could provide further insights similar to the one in section 5.3.
issues: As per my summary and noted weaknesses, I think presenting further analysis with additional insights would strengthen this work. Beyond that, more discussion of task-conditioning and some iteration on Figure 2 would be benficial.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper improves the QT-Opt algorithm by adding a predictive information axillary loss. The proposed algorithm PI-QT-Opt achieves better performance (task success rate) over various tasks in both simulator and real-world robotic testbed. The extensive experiments validate the advantage of the predictive information axillary loss.
strengths_and_weaknesses: Strengths
- The extensive experiments validate the advantage of the predictive information axillary loss.
- The predictive information axillary loss shows advantages in all the tasks in the figure. 3, 4. This indicates the benefit of predictive information exits in vast robotic tasks.

Weaknesses
- The proposed algorithm PI-QT-Opt is very similar to [2]. If it is true, this paper will be more about an empirical study of applying [2] to robotic manipulation. (Please correct me if I am wrong)
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper proposed an algorithm PI-QT-Opt that incorporates a predictive information axillary loss. The benefit of predictive information axillary loss is been validated by extensive experiments in the paper. Whoever, the proposed algorithm PI-QT-Opt is very similar to [2]. The similarity reduces the novelty of the paper.
issues: - Please clarify the difference between the proposed PI-QT-Opt algorithm and [2].
- Please provide more explanation of the tasks in line 232  "(2) not only the task but the object is never seen during training.". Please define "the task ... is never seen during training".
- Please point out what could limit PI-QT-Opt to achieve better results in table 1 in the limitation section. Though PI-QT-Opt is better than QT-Opt, there is room for improvement.
- How do you think what are the advantages/ disadvantages of the predictive information, compared with model-based RL algorithms? Model-based RL algorithms also utilize the information between the current state action tuple and the future state reward tuple.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to add an auxiliary loss of predictive information to a model-free RL algorithm, QT-Opt. This additional loss helps shape the representations, which leads to efficient training and better transfer. The exhaustive simulated and real-world experiments demonstrate that QT-Opt combined with the auxiliary predictive information loss learns faster, generalizes better, and zero-shot transfers to the real world by training with sim-to-real translated images.
strengths_and_weaknesses: ### Strenghts

* Even though the proposed method is built upon existing methods (QT-Opt, multi-task learning, predictive auxiliary loss, and CycleGAN), the paper proposes a practical integration of these components to achieve a multi-task policy that transfers to the real world.

* The experiments in simulation and the real world demonstrate the effectiveness of using predictive information for representation learning over QT-Opt.


### Weaknesses

* The baseline performance of QT-Opt shows comparable results with the proposed method, PI-QT-Opt, except for Knock skills. 

* The idea of supervising representations with predictive information itself is not new, which is one of the major contributions of the paper.

* The task definitions in the experiments are limited to simply specifying the target objects and the target regions. It is questionable whether this approach is scalable to more complex tasks.

### Questions and suggestions

* In Figure 2e, the robot seems to reach the orange, but the robot does not seem to move the orange near the purple area. What is the robot supposed to do? Similarly, Figure 2g is also unclear.

* One of the main claims of the paper, PI-QT-Opt solves up to "297 tasks", can be misleading. Before reading the details of the tasks, these tasks are likely to be interpreted as a lot of diverse types of manipulation tasks (e.g. pushing, inserting, pouring, folding, screwing) rather than pick-and-place with 200+ variations. This should be written more clearly.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The proposed method integrates many existing techniques to improve a multi-task robotic manipulation agent, which also zero-shot transfers to the real-world robot. However, the proposed method is incremental to the prior work PI-SAC and QT-Opt and the improvement is marginal compared to the baseline. Thus, I stand for weak rejection.


**After rebuttal**

I appreciate the authors' detailed answers. I appreciate that this paper demonstrates the effectiveness of "predictive information loss" for multi-task RL and sim-to-real transfer, and the experiments are impressive. Although this reassures the effectiveness of the predictive model loss for representation learning, the technical novelty seems incremental. Also, it could be better if the authors provided an updated paper addressing all the reviewers' comments, which may help understand the paper better. Thus, I am not confident to recommend accepting this paper.
issues: * The proposed approach shows marginal improvements over the baseline algorithm. Also, it is not clear whether the proposed predictive information loss is critical for representations necessary for learning diverse manipulation tasks. It could be more convincing if the proposed PI-Qt-Opt is compared against other well-known representation learning approaches (e.g. pre-trained R3M networks [1], Transporter networks [2], ATC [3]).


[1] Nair et al. R3M: A Universal Visual Representation for Robot Manipulation, 2022

[2] Zeng et al. Transporter Networks: Rearranging the Visual World for Robotic Manipulation, CoRL 2020

[3] Stooke et al. Decoupling Representation Learning from Reinforcement Learning, ICML 2021
review_title: null

Meta-review: metareview: Strengths:
- All the reviewers have found that the experimental results of the paper is convincing. 
- All the reviewers have found the technical approaches in the paper making sense. 
- The paper is well-written and well-structured. 


Weaknesses: 
- Most reviewers have pointed out limited theoretical contributions as this article basically deals with practical applications of an existing methodology in [2]. => Emprical contributions have been strenghthened during the rebuttal phase. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)