PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale
Authors: Kuang-Huei Lee
Ted Xiao
Adrian Li
Paul Wohlhart
Ian Fischer
Yao Lu
Venue: CoRL 2022 Poster (2022)
Keywords: deep reinforcement learning
robot manipulation
multi-task learning

Abstract: The predictive information, the mutual information between the past and future, has been shown to be a useful representation learning auxiliary loss for training reinforcement learning agents, as the ability to model what will happen next is critical to success on many control tasks. While existing studies are largely restricted to training specialist agents on single-task settings in simulation, in this work, we study modeling the predictive information for robotic agents and its importance for general-purpose agents that are trained to master a large repertoire of diverse skills from large amounts of data. Specifically, we introduce Predictive Information QT-Opt (PI-QT-Opt), a QT-Opt agent augmented with an auxiliary loss that learns representations of the predictive information to solve up to 297 vision-based robot manipulation tasks in simulation and the real world with a single set of parameters. We demonstrate that modeling the predictive information significantly improves success rates on the training tasks and leads to better zero-shot transfer to unseen novel tasks. Finally, we evaluate PI-QT-Opt on real robots, achieving substantial and consistent improvement over QT-Opt in multiple experimental settings of varying environments, skills, and multi-task configurations.

PI-QT-Ope Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale
Kuang-Huse Lee"
Ted Xiao
Adrian Li Google Research
Paul Wohlhart
Ian Fischer
Yao Lu
[leekh, tedxiao, alhli, wohlhart, iansf, yaolug] @google.
The predictive information, the mutual information between the past and future, has been shown to be a useful representation learning auxiliary loss for training reinforcement learning agents, as the ability to model what will happen next is critical to success on many control tasks. While existing studies are largely restricted to training specialist agents single-task settings simulation, in this work, we study modeling the predictive information for robotic agents and its im- portance for seneral-purposs agents that are trained master a large repertoire of diverse skills from large amounts of data. Specifically, we introduce Predictive Information QT-Opt (PI-QT-Opt), a QT-Öpt agent augmented with an auxiliary loss that learns representations of the predictive information to solve up to 297 vision-based robot manipulation tasks in simulation and the real world with sin- gle set of parameters. We demonstrate that modeling the predictive information significantly improves success rates on the training tasks and leads to better zero- shot transfer to unseen novel tasks. Finally, we evaluate PI-QT-Opt on real robots, achieving substantial and consistent improvement over QT-Opt in multiple exper- imental settings of varying environments, skills, and multi-task configurations.
Abstract:
Keywords: deep reinforcement learning, robot manipulation, multi-task learning
Introduction
Real robotic control systems are often partially observable and non-Markovian, and include high- dimensional observations, such as pixels. In such systems, we can learn representations by explicitly modeling the mutual information between consecutive states and actions the predictive informa- tion [1] to facilitate policy and value learning [2, 3]. When learning a predictive information representation between state, action pair and its subsequent state, the learning task is equivalent to modeling environment dynamics [4]. In this work, we are interested training multi-task gen- eralist agents [5, 6, 7] that can master wide range of robotics skills in both simulated and real environments by learning from a large amount of diverse experience. We hypothesize that modeling the predictive information will give latent representations that capture environment dynamics across multiple tasks, making it simpler and more efficient to learn generalist policy. We also hypothe- size that such generalist agent may do better job of transferring to real-world environments and novel tasks unseen during training. While existing studies of learning predictive information repre- sentations in RL [2. 3, 4, 8, 9] have largely been limited learning single-task specialist agents in simulated environments such as DM-Control [10] and Atari [11], our hypotheses can be seen as ex- tending the generalization results Lee et al. [2]. which showed both more sample-efficiee learning and better fine-tuning on unseen tasks. We investigate both of these hypotheses in this work.
We combine predictive information auxiliary loss with QT-Opt [12], model-free off- off-policy inforcement learning method that been shown to work well on vision-based continuous control problems. QT-Opt is able to leverage large-scale, multi-task datasets in simulation and the real world [12, 13, 14, 15]. We train a task-conditioned QT-Opt agent [12] with a predictive information auxiliary loss similar Lee et al. [2], which we refer to as Predictive Information QT-Opt (PI-QT- Opt). We study various simulated and real robot environments using an Everyday Robots manipu-
"qqual contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

lator arm [16]. including a large diverse set of real-world environments with up to 297 challenging dision-based manipulation tasks in a kitchen setting [7] (See Section 3.2 for task definitions).
Lee et al. [2] showed the benefits of predictive information regularization for accerlerating policy learning and quickly finetuning a policy trained on one task to solve related task in the same environment. Our experiments show that predictive information regularization additionally gives substantial benefits in two challenging zero-shot settings: from simulation to real-world robots, and from one set of training objects to novel objects in both simulation and real environments. We demonstrate that PI-QT-Opt significantly outperforms QT-Opt terms of success rate on training tasks in simulation. When evaluated on tasks that are unseen during training, modeling the predictive information increases the zero-shot success rate substantially. We verify these improvements on real robots via sim-to-redl transfer, and observe that PI-QT-Opt significantly outperforms QT-Opt.
Our primary contributions are:
We validate that modeling the predictive information an effective auxiliary task for learning multi-task generalist robot control agents.
We show that simple forms of task conditioning are sufficient to allow QT-Opt learn to solve large numbers of tasks simultaneously, avoiding some of the complexity of earlier multi-task approaches to QT-Opt [13].
We verify the improvements through large-scale real-world experiments, including training a reward-based agent that performs well on 297 real robotic control tasks.
We show that the predictive information helps zero-shot generalization to unseen tasks.
We demonstrate that PI-QT-Opt can train only on simulated environments and transfer to real robots more effectively than our QT-Opt baseline.
2 Related Work
Predictive Information Representations. Previous studies [17, 2, 4, 9, 18, 8, 19] have shown that predictive information [1] is an effective auxiliary or representation learning objective for RL agents planning. This result can be connected to findings in neuroscience that suggested that the brain maximizes predictive information at an abstract level [20, 21]. Broadly, our work differs from those approaches by focusing on multi-task, real-world environments. This enables us to verify improvements and study broader generalization properties in realistic settings. Additionally, with the exception of [2), most such works do not explicitly learn a compressed representation of the predictive information. Finally, because the predictive information can model the underlying environment dynamics, which are stationary, its use as an auxiliary loss can also be seen as a representation regularizer for RL agents, possibly helping avoid overfitting to any specific value function during training [22].
basond robot learning in both simulated and
Robot Manipulation. We focus on two main categories of related work in robot manipulation: multitask robot learning methods and real world data-driven robotics methods. One family of ap- proaches for multitask robot learning focuses on supervised learning of multitask control policies to maximize few-shot performance via meta-learning [23, 24, 25] direct zero-shot performance via behavioral cloning [26, 27, 28, 29]. However, methods based on imitation learning require ex- pensive expert demonstrations and struggle to improve autonomously with on-policy learning. On the other hand, reinforcement learning (RL) based approaches are able to bootstrap without a large amount of expert data and are able continuously improve from their own experience. While some RI. methods combine subtasks [30, 31] or map tasks to individual policies [32, 33), we wish to learn single shared policy. Recent work focuses on learning a single multi-task policy with both on- policy [34] and off-policy RL methods [35, 36, 37], and have shown promising results on a variety of robot manipulation tasks in simulation [34]. Some approaches real-world robot learning for manipulation utilize RL with real robot data collection or simulation with domain randomization [15, 29, 12, 13, 38, 39, 40, 41, 42, 43], while other approaches focus on imitation learning from ex- pert demonstrations [27, 44, 45, 46, 47, 48). MT-Opt [13] is closely related to our approach; it also extends QT-Opt [12] to real world multitask robot learning setting. While MT-Opt suggests that data routing and data sharing is very important, it is quite challenging to scale this approach to the hundreds of tasks that we consider. Thus, we take different approach from MTT-Opt, focusing on learning better representations for a single fask-conditionel critic that does not use any data routing.

---

Conditional Q-Function Qe(s, a, Ctask)
MLP
Conv
Conv
Conditional Forward Encoder Eg [z|s, a, Ctask)
MLP
8y 0320, 256,
Conditional Target Q-Function
Action
argmax Q3 (s' a', Crask)
Cross-Entropy Loss
Crank
Task Context
Conditional Backward Encoder
Sp
Proprioceptive State
bg s',, a Crasks
Contrastive
Figure I: An overview of the PI-QT-Opt system for multi-task robotic reinforcement learning. The contrastive CEB loss is an auxiliary objective to Q-learning. = 8p). See Section 3 for details.
3 Methods
We describe the details of Predictive Information QT-Opt (PI-QT-Opt) in Section 3.1, and task con- text conditioning for enabling multi-task robot RL at scale in Section 3.2. More implementation details are described in Appendix B. Figure presents an overview of the our system.
3.1 Predictive Information QT-Opt
Predictive Information. The Predictive Information [1] is the mutual information between the past and the future, futuur) has been shown that the predictive information an effective auxiliary loss for RL agents [2, 3]. From here on, we will denote the past by X and the future by Y. Lee al. [2] argues that a learned representation Z of the predictive information should be compressed with respect to X, based on the observation in Bialek and Tishby [1] that ((X) (X), the entropy of the past, grows more quickly than Y(:;Y.. Following [2], we use the Conditional Entropy Bottleneck (CEB) [49] to learn the representation Z, utilizing the same variational bound on CEB:
CEB min BI(X; I(Y 2
min Beg.srp(e log e(z(zr b(=ly)
b(2ly) Σ b(21)))
log
where (x, 3) are sampled from the data distribution, p(x, y). e(2 (x) is the learned forward encoder distribution, b(2lg) is the learned variational backward encoder distribution, is Lagrange mul- tiplier that controls how strongly compressed the learned representation Z is, with smaller values corresponding to less compression, and K is the number of examples in mini-batch during training. The second term of Equation (2) corresponds to the contrastive InfoNCE bound [3, 50] on mutual information J(Y; Z). Following the CEB implementation of Lee et al. [51], we choose e(2(x) and b(=ly) to be parameterized von Mises-Fisher distributions. Details are described in Appendix B.
QT-Opt. QT-Opt is an offline actor-critic algorithm where only the critic is explicitly learned. learns the Q-function (or critic) by minimizing Bellman errors:
{(0)
[Qo(s, a). Qr(s,a,
(3)
where 0 is the of model parameters, and s' state observations, is the action taken, D is some divergence (QT-Opt uses cross-entropy). Qo is the learned state-action value function, and Qr r(s.a) a) <V(')) gives larget value for the given transition (s, a, s'). V(s)) min Qø, (s', Te, (s')) is Double DQN state value function [52, 53, 54], which for QT-Opt is computed by using two lagged versions of the parameters 6, 0 and 6z, with different lagging methods. The QT-Opt policy, पह, (+) arg max, Qū, (8, a). is optimized directly each envi- ronment step using the cross-entropy method (CEM) [55]. In CEM, N actions are sampled from Guassian over the action space, the best M < N actions as measured by Qa, are used to estimate the mean and variance of new Guassian, from which another N samples are drawn. This is repeated

---

fixed number of steps, converging towards a narrow Gaussian over the part of the action space that the critic believes will perform best at the current state. See [12] for further details.
PI-QT-Opt. As shown in Figure 1, PI-QT-Opt combines a predictive information auxiliary simi- lar to that introduced in Lee al. [2] with the QT-Opt architecture. We define the past (X) to be the current state and action, (s, (s,a), and the future (Y) ) to be the next state, next optimal action, and reward, (s', a', r). A state s includes an RGB image observation and proprioceptive information. Image observations are processed by a simple conv net. the output of which is mixed with action, proprioceptive state, and the current task context (described in Section 3.2) using additive condition- ing. A second simple conv net processes the combined state representation. All of the convolutional parameters are shared by both the forward encoder 6g for modeling the predictive information (as in Equation (2)) and the Q-function Qo (as in Equation (3)), but the shared representation output from this is further processed by separate ML.ss, to allow each loss to specialize its representation as needed, while still allowing the predictive information loss to influence the shared convolutional representation. Not shown in Figure is that the target Q-function Qa, and backward encoder bg, for modeling the predictive information also share the same base lagged and non-trainable convo- lutional representation, but the backward encoder has its own trainable MLP, in order to learn any differences in dynamics when trying to predict the past from the future, rather than predicting the future from the past, as the forward encoder does. In addition, we concatenate the convolutional representation with observed reward r(s, a) as the input to the backward encoder MLP head.
We find that adding a predictive information auxiliary loss is an easy way to give substantial perfor- mance improvements to our chosen RL algorithm, as in Lee et al. [2] which introduced Predictive Information Soft Actor-Critic (PI-SAC). However, we note that PI-SAC on its own was unable to solve our tasks, yielding close-to-zero success rates, and neither was SAC [56]², which may indicate that the choice of base RL algorithm is still critical.
3.2 Multi-Task Context Conditioning
In order to learn one general- purpose agent for multiple tasks, we condition the Q-functions and the Predictive Information auxiliary on task context, which describes the specific task that we wish the agent to perform, as illustrated in Figure 1. In our setting, task involves robot skill and a set of objects that the robot should interact with. We use two practical implementations of task context in different robot manipulation settings (Section 4. One is image-based, where task is specified with the initial image, the initial object locations, and the skill type as depicted in Figure 2. It only considers locations and skill types and thus could enable good generalization across different and even novel objects. The other one is language-based, where tasks are specified with natural language, similar to Ahn et al. [7]. Details of these two implementations are described in Appendix B.2. A generalization of the language-base based approach is considered in Appendix F.
An alternative approach to extending QT-Opt for multi-task learning is having one critic head per task with shared base encoder, as used in MT-Opt [13]. However, MT-Opt found that multi- headed architecture performed worse than a single-headed architecture; in addition, we note that a melti-headed strategy is practically prohibitive scale to the order of hundreds of tasks.
4 Experimental Setup
To analyze how PI-QT-Opt compares with QT-Opt across different multi-task robotic learning sce- narios, we explore variety of challenging simulation and real vision-based robotic manipula- tion environments. While prior results on large-scale robotic grasping focused on limited set of tasks [12, 13], we verify the robustness and scalability of PI-QT- by studying many different environments across hundreds of different tasks in the real world.
4.1 Robot Manipulation Tasks
We study 6 different multi-task, vision-based robotic manipulation settings in 3 different environ- ments in simulation. Four of the manipulation settings have the corresponding hardware setup per- mitting real-world evaluation.
With our best effort, were unable to get SAC and PI-SAC working on tasks (See Appendix B.4)

---

(a) SayCan (sim)
(b) SayCan (real)
(c) Instance Grasping
(d) 6-Obj Manipulation
SayCan Skills
(c) Move
Move orange
near Zup con
(i) Pick
Pick Popsi
(g) Knock
bottle down
Figure 2: Example tasks in simulation and real-world environments using the Everyday Robots manipulator arm. See Section 4.1 for details. The first images of (e), (f) and (g) show target object locations used in the image-based task context conditioning method we designed. In real-world environments, the object locations are detected using a vision model (Section 5.2). An overlay image is created based on the object locations and the skill type, where we use different colors to indicate different skills. This overlay image is only created for the first image of an episode. Then, the first image and the overlay image are used task context throughout the whole episode.
Experiment 1: (Sim & Real) SayCan Move skill (Figure (()). We use the kitchen envi- ronment from SayCan [7] which contains 17 objects that spawn countertop. The simulation environment is shown in Figure 2(a) and the real countertop is shown in Figure 2(b). The Move skill contains 272 tasks of the form "move object A near object B". 246 tasks are used during training and the 26 remaining tasks are held out. amage-based task conditioning is used.
Experiment 2: (Sim & Real) SayCan Pick skill (Figure (f). Same environment as Experiment 1. The Pick skill encompasses picking up each of the individual objects, for total of 17 tasks. 12 tasks are used during training and the remaining 5 tasks are held out. Image-based task conditioning is used, as described in Section 3.2.
Experiment 3: (Sim & Real) SayCan Knock skill (Figure (b) Same environment as Experiment 1. The Knock skill contains 8 tasks testing knocking over a can or bottle. 7 tasks are used during training and task is held out. Image-based task conditioning used.
Experiment 4: (Sim & Real) SayCan 297 tasks, All skills (Figure (t)) Same environment as Experiment
This task includes all SayCan skills (297 tasks). 265 tasks are used during training and the 32 remaining tasks are held out. Image-based based task conditioning is used. Experiment (Sim Only) Instance Grasping (Figure 2(c)). A sampled subset of 37 different trash objects are placed randomly in bins. A vision model provides a target object for the robot to grasp. The episode is successful if the robot lifts the target object. Image-based task conditioning used. Experiment 6: (Sim Only) 6-Object Manipulation (Figure 2(d)). A fixed set of 6 objects are placed randomly on a table, similar to [27]. There are 30 separate tasks comprising the manipulation skills of picking, pushing, and pick and place. Language- based task conditioning used.
In all of the experiments, we use an Everday Robots manipulator robot [16] with parallel-jan grip- pers, an over-the wer-the-shouldes camera, and 7-DoF arm. The robot has a proprioceptive observation space that includes the RGB camera image, the arm pose, and the gripper angle. For the action space, learned policies control the robot via relative position control of the end effector. In simulation-only environments, we utilize blocking control, where the policy waits until the previous action completes before planning the next action. Motivated by faster and more reactive robot mo- tions for real world evaluations, we utilize concurrent control [ [14] in all SayCan experiments, which
5

---

00-
0.30
success rat
0002911 min
0.25
C4-
04
R-0T-ppt
P-OT-P
0m
01-
4I ogt
000
120000
Xc0000
training steps
50000 amii
10000
c00
130003
training staps
205006
360002
Jaaiing stegs
(a) Move Skill
(b) Pick Skill
(c) Knock Skill
09
04
0.30
0.7
Success
R-O7-Opt
PHOT-ppt
007
OF-GPT
U) spt
01-
0.0
maaining stops
220000
000
40200
training steps
20100
120000
training stops
(d) SayCan 297 Tasks (All Skills)
(e) Instance Grasping
(f) 6-Obeect Manipulation
Figure 3: Performance (success rate) tasks that used for agent training.
means that the current action is computed while the previous action is still executing, We provide additional implementation details in the Appendix B.
4.2 Reducing the Sim-to-Res Gap with CycleGAN
We train our PI-QT-Opt and QT-Opt models in simulated environments that roughly match the real world evaluation environments. In order to reduce the simulation-to-real (sim-to-real) gap when deploying policies on real robots, we train a RetinaGAN [57] model, version of CycleGAN [58], to transform simulated robot images to look more realistic while preserving general object structure, following the sim-to-real setup in [14]. This enables our method to train purely on RetinaGAN- transformed simulation images and directly transfer to the real world. We apply this image transfor- mation all the SayCan models, which we evaluate both simulation and the real world.
4.3 Training and Evaluation Protocols
For each of the experimental settings in Section 4.1, we create sparse binary reward (success or failure) in simulation based on ground truth object poses. For each environment, we follow the large scale asynchronous distributed training procedure in [12] and train QT-Opt agents and PI-QT-Opt agents with the same hyperparameters with batch size of 4096 using 16 TPUv2. In simulation, we evaluate models with 700 episodes and compute their success rates. For the real-world evaluations, we test each policy on episodes of standardized starting scenarios for a fair comparison between policies. More details are described in Appendix B.6 and Appendix B.7.
5 Experimental Results
We discuss experimental results on the manipulation tasks introduced in Section 4.1 in Section 5.1 and Section 5.2. We report mean and one standard deviation of success rate over 3 training runs with different random seeds for each model. We analyze the relationship between predictive information and agent performance in Section 5.3 and information compression in Appendix E.
5.1 Evaluation in Simulated Environments
Performance on training tasks. We learn PI-QT-Opt and QT-Opt models for each of exper- iment settings introduced in Section 4.1. Figure 3 shows evaluation results tasks that are used for agent training in simulation. We can see that PI-QT-Opt consistently outperforms QT-Opt in all settings throughout training, improving the move model by 20% and the 297-tasks model by 25%

---

relatively for example. This empirically validates our hypothesis that training with the predictive information auxiliary loss leads to better and more efficient learning of general-purpose agents.
Zero-shot transfer unseen tasks. We use the held-out SayCan tasks to evaluate zero-shot trans- fer of PI-QT-Opt and QT-Opt models. A SayCan task is composition of a skill and set of target objects (1-2) that the robot should interact with, as described in Section 1. Here, we consider two types of zero-shot transfer: (1) the task is never seen during training, but the target objects have been seen during training in other tasks, and (2) not only the task but the object is never seen during train- ing. For (1), we evaluate the move skill models on held-out move tasks, and the 297-tasks models on held-out tasks of all skills?. For (2), we evaluate the pick/knock skill models on held-out pick/knock tasks. As demonstrated in Figure 4, PI-QT-Opt outperforms QT-Opt in all the settings, showing that using the predictive information auxiliary loss leads to better zero-shot transfer to novel task compositions and unseen objects. The 297-tasks model, for instance, is improved by 28% relatively.
0.35
0.30
0.4
0.25
0.3
0.20
0:35
0:10
R-QT-Opt QT-Opt
R-QT
0.05
0.00 0
100000
200000
300000
400000
500000
0
100000
200000 training steps
300000
training steps
(a) Move Skill
(b) Pick Skill
0.45
0.40
M
0.5
success
0.4
0.25
0.3
0.20
0.5
0.2
010
0.1
P.QT-Opt
Opt
0.05
P-QT-Opt Op-tpt
0.0
100000
200000 training steps
300000
400000
0.00
0
100000
200000
300000
400000
training steps
(c) Knock Skill
(d) SayCan 297 Tasks (All Skills)
Figure 4: Performance (success rate) on unseen novel tasks.
5.2 Evaluation in the Real World
We directly deploy our PI-QT-Opt and QT-Opt models trained in simulation for SayCan skills on real robot in the SayCan kitchen environment. As described in Section 4.1, these models use image-based task context conditioning that requires information about the initial object locations (Section 3.2). Unlike the simulated environments, the ground truth locations are not available in the real environments. Therefore, we use a VILD [59] model to detect objects conditioned on object names to locate the target objects associated with the names. When the VILD model fails detect the target objects, we adjust the scene slightly until VILD succeeds and then continue with the evaluation. This allows us to evaluate the policy performance exclusively. We evaluate the SayCan Pick, Move, and Knock models; the results are presented in Table 1, showing a clear advantage for PI-QT-Opt. This empirically supports our hypothesis that using the predictive information auxiliary loss enables better performance when transferring to the real world.
'The Move Skill training task set contain 246 tasks and each involves 2 objectives. Due to the nature of this task set, it is difficult to avoid seen objects in held-out tasks. Therefore, we only use it for type (1) shot transfer but not type (2).
7

---

Table Evaluations on the real robot (mean and standard deviation over evaluations)
Task
PI-QT-Opt Success Rate
QT-Opt Success Rate Relative Change
SayCan Move
22.9 + 8.4%
13.93 + 3.2%
+64.4%
SayCan Pick
42.0 9.9%
28.7 = 8.0%
+46.6%
SayCan Knock
54.6 + 2.4%
36.2 + 11.6%
+50.7%
5.3 How does Predictive Information Relate to Agent Performance?
A core hypothesis of this work is that the ability to model what will happen next is critical to success on control tasks. This ability can be quantified by the amount of predictive information, I (XX, Y). the agent's s representation captures. In this section, we analyze the SayCan 300-task PI-QT-Opt model.
We compare the estimates of [(X, Y), E Eloog b(zly) log 1 Σx= b(2(yx)] (from Equation (2)) in successful and failed episodes versus TD-error in Figure 5t We can observe that the amount of predictive information is generally higher in successful episodes, and that episodes with high TD-errors have much lower predictive information and are always failures.
4.5
Failure Success
4.0
MI Estimates
3.5
3.0
2.5
2.0
1.5
0.1
0.2
TD-Error
Figure 5: Predictive information estimate versus TD-error. Each point is averaged over episode.
The predictive information is the mutual information (MI) between the past and future.
6 Conclusion
We have shown that using the predictive information auxiliary with QT-Opt agent, i.e. PI-QT- Opt, results in faster training, higher final performance, and better generalization unseen tasks for single generalist agent trained on hundreds of tasks. We have also shown that PI-QT-Opt and QT-Opt can made to support multiple tasks by adding simple task conditioning. Our system, PI-QT-Opt, is generalist agent capable of solving hundreds of real-world tasks in a simple kitchen environment, in spite of having only been trained simulation.
Limitations. We focused on a single robotic arm and gripper, so we cannot speculate on how well our approach would work on different robotic setups. We limited our real-world experiments to simple, carefully-controlledd "kitchen" settings, and can say nothing about performance on different types of environments. Most importantly, our models do not have any safety guarantees, and we did not attempt evaluate how they would perform in the presence of other agents, such as humans or animals. Using these models in settings where there are other agents could lead to injury or death. These limitations may be mitigated in future research by working with range of robot platforms, expanding the breadth of tasks considered, including other agents in the environments during training and evaluation, and integrating safety systems.
*For these predictive information analyses, we collect data for each task with a converged policy, and use a batch size of 128, which corresponds to an [(X, Y) upper bound of log 128 = 4.852 in order to fit these analyses into machine, while the distributed training batch size is 4096.

---

Acknowledgments
The authors would like to thank Alex Herzog, Mohi Khansari, Daniel Kappler, Peter Pastor for adapting infrastructure and algorithms for the image-based task context from generic to instance specific grasping. Sangeetha Ramesh for leading robot operations for data collection and evaluations for the VILD model training, and Kim Kleiven for leading the waste sorting service project that constitutes the framework for training and deployment of the instance grasping task set, including defining benchmark and protocol. We thank Jornell Quiambao, Grecia Salazar, Jodilyn Peralta, Justice Carbajal, Clayton Tan, Huong T Tran, Emily Perez, Brianna Zitkovich and Jaspiar Singh for helping administrate real-world rohot experiments. We would also like to thank Sergio Guadarrama and Karol Hausman for valuable feedback.
References
[1] W. Bialek and N. Tishby. Predictive information. arXiv preprint cond-mat/9902344, 1999.
[2] K.-H. Lee, I. Fischer, A. Liu, Y. Guo, H. Lee, J. Canny, and S. Guadarrama. Predictive infor- mation accelerates learning in RL. Advances in Neural Information Processing Systems, 33: 11890-11901, 2020.
[3] A. v. d. Oord, Li,, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 18X7:
2018
[4] A. Anand, E. Racah, S. Ozair, Y. Bengio, M.-A. Côté, and R. D. Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems, pages 7766-8779, 2019.
[5] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175. 2022.
[6] K.-H. Lee, O. Nachum, M. Yang. L. Lee, D. Freeman, W. Xu, S. Guadarrama, I. Fis- cher, Jang. H. Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205. 15244, 2022.
[7] M. Ahn, A. Brohan, Brown, Chebotar, O. Cortes, B. David, Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, F. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Ser- manet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, Xu, S. Xu, and M. Yan. Do can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.0I1 01691, 2022.
[8] A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from inforcement learning. International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[9] X. Chen, S. Toyer, C. Wild, S. Emmons, I. Fischer, K.-H. Lee, N. Alex, S. H. Wang, P. Luo, S. Russell, al. An empirical investigation of representation learning for imitation. arXiv preprint 2rXiv:2205.07888 2022.
[10] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li. D. d. L. Casas, D. Budden, A. Abdolmaleki. J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801000690 2018.
[11] M. Bellemare, Y. Naddaf, Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, jun 2013.
[12] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.

---

[13] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021.
[14] T. Xiao, E. Jang. D. Kalashnikov, S. Levine, J. Ibarz, K. Hausman, and A. Herzog. Think- ing while moving: Deep reinforcement learning with concurrent control. arXiv preprint arXiv:2004.0600 06089, 2020.
[15] Y. Lu, K. Hausman, Y. Chebotar, M. Yan, E. Jang, A. Herzog, T. Xiao, A. Irpan, M. Khansari, D. Kalashnikov, and S. Levine. AW-Opt: Learning robotic skills with imitation andreinforce- ment at scale. In 5th Anmual Conference on Robot Learning, 2021.
[16] Development, LLC. Everyday Robots.
com, 2022. Ac-
cessed: 2022-06-15.
[17] K.-H. Lee, O. Nachum, T. Zhang, S. Guadarrama, J. Tan, and W. Yu. PI-ARS: Accelerat- ing evolution-learned visual-locomotion with predictive information representations. arXiv preprint arXiv:2207.133: 13224, 2022.
[18] K. Rakelly, A. Gupta, C. Florensa, and S. Levine. Which mutual-information representation learning objectives are sufficient for control? Advances in Neural Information Processing Systems, 34, 2021.
[19] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto. deformable objects using contrastive estimation. arXiv preprint arXiv:2003.05436. 2020.
Learning predictive representations for
[20] K. Friston, A theory of cortical responses. Philosophical transactions of the Royal Society B: Biological sciences, (55 2005.
[21] R. P. Rao and D. H. Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
[22] W. Dabney, A. Barreto, M. Rowland, R. Dadashi, J. Quan, M. G. Bellemare, and Silver. The value- inlro-improvemmnt path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7160-7168, 2021.
[23] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. arXiv preprint arXiv:1709.0 04900, 2017.
[24] R. Rahmatizadeh, Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. arXiv preprint arXiv:1707.02920. 2017.
[25] S. James, M. Bloesch, and A. Davison. Task-embedded control networks for few-shot imi- lation learning. arXiv preprint 1.XX:v:1800003377. 2018.
[26] R. Fox, R. Berenstein, Stoica, and K. Goldberg. Multi-task hierarchical imitation learning for home automation. In 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), pages 1-8. IEEE, 2019.
[27] E. Jang, A. Irpan, M. Khansari, D. Kappler, Ebert, C. Lynch, S. Levine, and C. Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference Robot Learning, 2021. URL
[28] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2 03299, 2021.
[29] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[30] A. G. Barto, S. Singh, N. Chentanez, et al. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning, pages 1 112-19. Piscataway, NJ, 2004.
10

---

[31] T. G. Dietterich. Hierarchical reinforeement learning with the maxq value function decompo- sition. Journal of artificial intelligence research, 13:227-303, 2000.
[32] B. Da Silva, G. Konidaris, and A. Barto. Learning parameterized skills. arXiv preprint VrXiv:/200.6333 2012.
[33] M. P. Deisenroth, P. Englert, J. Peters, and D. Fox. Multi-task policy search for robotics. 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3876-3881, 2014.
[34] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100 110.. PMLR, 2020.
[35] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning. arXiv preprini arXiv:2001.6677 2020.
[36] R. Yang, H. Xu, Y. WU, and X. Wang. Multi-task reinforcement learning with soft mod- ularization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4767-4777. Curran Associates, Inc.. 2020. URL https: pprrcee//proceeiingg nwwrips https/ccppppp/p//Eaa.... pdf.
[37] Sodhani, A. Zhang, and Pineau. Multi-task reinforcement learning with context-based representations. arXiv preprint wxXi:22102.0//777,,222 2021.
[38] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, In 2017 JEEE/RSJ international conference intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[39] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint
2018.
[40] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[41] Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's 's cube with a robot hand. arXiv preprint arXiv: 1910.09111.. 2019.
[42] James, P. Wohlhart, M. Kalakrishnan, Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via mandonized-to-canonie adaptation networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12627-12637, 2019.
[43] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004.12570 2020.
[44] M. Lopes, F. S. Melo, and L. Montesano. ffordance-based imitation learning in robots. In 2007 IEEE/RSJ international conference on intelligent robots and systems, pages 1015-1021. IEEE, 2007.
[45] Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. Advances in Neural Information Processing Systems, 32, 2019.
[46] N. Ratliff, J. A. Bagnell, and S. S. Srinivasa. Imitation learning for locomotion and manipula- tion. In 2007 7th IEEE-RAS International Conference on Humanoid Robots, pages 392-397. IEEE, 2007.
[47] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3(4):362-363) 2019.
11

---

[48] M. Khansari. D. Kappler, J. Luo, J. Bingham, and M. Kalakrishnan. Action image repre- sentation: Learning scalable deep grasping policies with zero real world data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3597-3603. IEEE, 2020.
[49] 1. Fischer. The conditional entropy bottleneck. Entropy, 22(9):999, 2020.
[50] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. 2019.
[51] K.-H. Lee, Arab, S. Guadarrama, J. Canny, and I. Fischer. Compressive visual representa- tions. Advances in Neural Information Processing Systems, 34, 2021.
[52] H. van Hasselt. Double Q-learning. Advances in neural information processing systems, 23, 2010.
[53] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
[54] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[55] R. Y. Rubinstein and D. P. Kroese. The cross-entropy method: unified approach to combi- natorial optimization, Monte- Carlo simulation, and machine learning, volume 133. Springer, 2004.
[56) T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, el. Soft actor-critic algorithms and applications. arXiv preprint (a0i:::88///59900 2018.
[57] D. Ho, K. Rao, Z. Xu, E. Jang. M. Khansari, and Y. Bai. Retinagan: An object-aware approach to sim-to-real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10920-10926. IEEE, 2021.
[58] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedingy of the IEEE international conference on com- puter vision, pages 2223 2232,, 2017.
[59] X. Gu. T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. arXiv preprint arXiv:2104.13921 2021.
[60] K. Fang, Y. Bai, S. Hinterstoisser, S. Savarese, and M. Kalakrishnan. Multi-task do- main adaptation for deep learning instance grasping from simulation. arXiv preprint arXiv: 1710.0642, 2017.
[61] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo- Cespedes, S. Yuan, C. Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803 11175, 2018.
[62] Ioffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[63] S. Guadarrama, A. Korattikara, O. Ramirez, Castro, Holly, S. Fishman, K. Wang, E. Gonina, N. Wu, E. Kokiopoulou, L. Sbaiz, J. Smith, G. Bartók, J. Berent, C. Harris, V. Vanhoucke, and F. Brevdo. TF-Agents: A library for reinforcement learning in tensor- flow. https://github. www./enoorflww///eeeeeeoo aggntss 2018. URL https: htt///gttutt /tt.thb.ccco///// tensorflos/agents. [Online; accessed 25-June-2019],
12


Reviews:
[{"review_id": "wW5JsGE_6UW", "reviewer": "Reviewer_G4sr", "summary": "This paper improves the QT-Opt algorithm by adding a predictive information axillary loss. The proposed algorithm PI-QT-Opt achieves better performance (task success rate) over various tasks in both simulator and real-world robotic testbed. The extensive experiments validate the advantage of the predictive information axillary loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The extensive experiments validate the advantage of the predictive information axillary loss.\n- The predictive information axillary loss shows advantages in all the tasks in the figure. 3, 4. This indicates the benefit of predictive information exits in vast robotic tasks.\n\nWeaknesses\n- The proposed algorithm PI-QT-Opt is very similar to [2]. If it is true, this paper will be more about an empirical study of applying [2] to robotic manipulation. (Please correct me if I am wrong)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposed an algorithm PI-QT-Opt that incorporates a predictive information axillary loss. The benefit of predictive information axillary loss is been validated by extensive experiments in the paper. Whoever, the proposed algorithm PI-QT-Opt is very similar to [2]. The similarity reduces the novelty of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Please clarify the difference between the proposed PI-QT-Opt algorithm and [2].\n- Please provide more explanation of the tasks in line 232  \"(2) not only the task but the object is never seen during training.\". Please define \"the task ... is never seen during training\".\n- Please point out what could limit PI-QT-Opt to achieve better results in table 1 in the limitation section. Though PI-QT-Opt is better than QT-Opt, there is room for improvement.\n- How do you think what are the advantages/ disadvantages of the predictive information, compared with model-based RL algorithms? Model-based RL algorithms also utilize the information between the current state action tuple and the future state reward tuple."}}, {"review_id": "ulKpE4J_Y7yv", "reviewer": "Reviewer_QJ7c", "summary": "The paper aims to train vision-based policies that can master a wide range of skills by learning from a large amount of diverse experience. The paper claims that learning a latent representation of the environment dynamics by modeling the predictive information (introduced by a prior work) result in policies that generalize better (zero-shot) to unseen tasks and for sim to real settings. The experimental results with up to 297 vision-based manipulation tasks (kitchen environment) support their claim. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The experimental results seem to support that modeling the predictive information can improve generalization. \n* It is a very well written and easy to read paper\n\nWeaknesses:\n* The theoretical contributions of the paper are minor. \n* The improvement compared to QT-Opt is not significant. Here is a summary for figure 3:\nMove skill: QT-Opt: 25%, PI-QT-Opt: 30%\nPick skill:  QT-Opt: 43%, PI-QT-Opt: 48%\nKnock skill: QT-Opt: 19%, PI-QT-Opt: 38% - It is not convincing why QT-Opt performs so poorly compared to Pick skill since the Knock skill must be much easier than pick. \n* It is not really clear how the outcome of this research can help other researchers to build more general policies in multi-task settings. It is difficult for other researchers to reproduce the setup or to have access to the computation power used in this experiment. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is still hard to believe that modeling a latent representation using predictive information can help since the success rate is still very low for both PI-QT-Opt and QT-Opt.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Figure 1: using s to denote both proprioceptive state and image observation can be misleading. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "QRu3DkLngmF", "reviewer": "Reviewer_jMwU", "summary": "This work studies the effect of improving RL training performance with an auxiliary loss for the task of finding the mutual information between past and future states. The specific technique for the auxiliary task is using a Conditional Entropy Bottleneck, which is a representation learning task previously shown to improve RL training in \"predictive information accelerates learning in RL\" [2]. Unlike in [2], this paper's focus is on augmenting QT-Opt with the auxiliary loss (as opposed to augmenting SAC) for the purpose of vision-based task-conditioned robotic object manipulation (as opposed to simpler simulated continuous tasks), with task conditioning allowing a single agent to be trained for hundreds of different tasks. The main contribution of this work are the empirical results of applying the ideas from [2] to this different setting, as well as the demonstration of multi-task learning by task condition through images or language. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and is easy to follow. For the most part there is no missing information in the explanation of related work, technical details, and experimental setup details.\n* The dual contributions of extending the findings of [2] to complex task-conditioned manipulation and of demonstrating a new technique for task-conditioning QT-Opt are of significant value to the field. \n* The set of experiments and the results observed convincingly demonstrate this auxiliary loss does help with training, and more significantly appears to improve real-world performance by a large margin. While this primarily shows \n* The papers notes some significant limitations and how they might be addressed in future work\n\nWeaknesses:\n* Results for the knock skills are confusingly different from the other ones, with lesser granularity and significantly larger benefits from using the auxiliary loss. \n* Apart from section 5.3, there is not much analysis of the results. Section D from the appendix does present some ablation results, which I feel would have been better in the main paper. \n* The (e), (f), and (g) rows of Figure 2 have really small images that are hard to understand. \n* It is stated that \"With our best effort, we were unable to get SAC and PI-SAC working on our manipulation tasks.\" Given this work specifically builds on [2], I would have liked to see more discussion or analysis of why QT-Opt with PI worked better than PI-SAC. \n* There is very little discussion of task-conditioning, and no citations of prior works on the topic. Given the second claimed contribution is \"We show that simple forms of task conditioning are sufficient to allow QT-Opt\", it feels like more discussion of related ideas and evaluation of how well task conditioning actually works are needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe this work presents valuable empirical results that involved a substantial amount of experimentation that shows the the key idea from  [2] applies to the robotics domain. However, given the contributions are primarily about empirical results rather than technical ideas, I would have liked to see significantly more analysis of the results that could provide further insights similar to the one in section 5.3.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "As per my summary and noted weaknesses, I think presenting further analysis with additional insights would strengthen this work. Beyond that, more discussion of task-conditioning and some iteration on Figure 2 would be benficial."}}, {"review_id": "F1aSQ54bi6", "reviewer": "Reviewer_UcuR", "summary": "This paper proposes to add an auxiliary loss of predictive information to a model-free RL algorithm, QT-Opt. This additional loss helps shape the representations, which leads to efficient training and better transfer. The exhaustive simulated and real-world experiments demonstrate that QT-Opt combined with the auxiliary predictive information loss learns faster, generalizes better, and zero-shot transfers to the real world by training with sim-to-real translated images.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strenghts\n\n* Even though the proposed method is built upon existing methods (QT-Opt, multi-task learning, predictive auxiliary loss, and CycleGAN), the paper proposes a practical integration of these components to achieve a multi-task policy that transfers to the real world.\n\n* The experiments in simulation and the real world demonstrate the effectiveness of using predictive information for representation learning over QT-Opt.\n\n\n### Weaknesses\n\n* The baseline performance of QT-Opt shows comparable results with the proposed method, PI-QT-Opt, except for Knock skills. \n\n* The idea of supervising representations with predictive information itself is not new, which is one of the major contributions of the paper.\n\n* The task definitions in the experiments are limited to simply specifying the target objects and the target regions. It is questionable whether this approach is scalable to more complex tasks.\n\n### Questions and suggestions\n\n* In Figure 2e, the robot seems to reach the orange, but the robot does not seem to move the orange near the purple area. What is the robot supposed to do? Similarly, Figure 2g is also unclear.\n\n* One of the main claims of the paper, PI-QT-Opt solves up to \"297 tasks\", can be misleading. Before reading the details of the tasks, these tasks are likely to be interpreted as a lot of diverse types of manipulation tasks (e.g. pushing, inserting, pouring, folding, screwing) rather than pick-and-place with 200+ variations. This should be written more clearly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed method integrates many existing techniques to improve a multi-task robotic manipulation agent, which also zero-shot transfers to the real-world robot. However, the proposed method is incremental to the prior work PI-SAC and QT-Opt and the improvement is marginal compared to the baseline. Thus, I stand for weak rejection.\n\n\n**After rebuttal**\n\nI appreciate the authors' detailed answers. I appreciate that this paper demonstrates the effectiveness of \"predictive information loss\" for multi-task RL and sim-to-real transfer, and the experiments are impressive. Although this reassures the effectiveness of the predictive model loss for representation learning, the technical novelty seems incremental. Also, it could be better if the authors provided an updated paper addressing all the reviewers' comments, which may help understand the paper better. Thus, I am not confident to recommend accepting this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {"issues": "* The proposed approach shows marginal improvements over the baseline algorithm. Also, it is not clear whether the proposed predictive information loss is critical for representations necessary for learning diverse manipulation tasks. It could be more convincing if the proposed PI-Qt-Opt is compared against other well-known representation learning approaches (e.g. pre-trained R3M networks [1], Transporter networks [2], ATC [3]).\n\n\n[1] Nair et al. R3M: A Universal Visual Representation for Robot Manipulation, 2022\n\n[2] Zeng et al. Transporter Networks: Rearranging the Visual World for Robotic Manipulation, CoRL 2020\n\n[3] Stooke et al. Decoupling Representation Learning from Reinforcement Learning, ICML 2021"}}]

Author rebuttal:

Decision: Accept (poster)