VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning
Authors: Che Wang
Xufang Luo
Keith W. Ross
Dongsheng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep reinforcement learning
visual control
data-driven framework
pretraining

Abstract: We propose VRL3, a powerful data-driven framework with a simple design for solving challenging visual deep reinforcement learning (DRL) tasks. We analyze a number of major obstacles in taking a data-driven approach, and present a suite of design principles, novel findings, and critical insights about data-driven visual DRL. Our framework has three stages: in stage 1, we leverage non-RL datasets (e.g. ImageNet) to learn task-agnostic visual representations; in stage 2, we use offline RL data (e.g. a limited number of expert demonstrations) to convert the task-agnostic representations into more powerful task-specific representations; in stage 3, we fine-tune the agent with online RL. On a set of challenging hand manipulation tasks with sparse reward and realistic visual inputs, compared to the previous SOTA, VRL3 achieves an average of 780% better sample efficiency. And on the hardest task, VRL3 is 1220% more sample efficient (2440% when using a wider encoder) and solves the task with only 10% of the computation. These significant results clearly demonstrate the great potential of data-driven deep reinforcement learning. 

VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning
Che Wang 1.2+
Xufang Luo
Keith Ross!
Dongsheng Li
New York University Shanghai
New York University
Microsoft Research Asia, Shanghai, China
Abstract
We propose VRL3, powerful data-driven framework with simple design for solving challenging visual deep reinforcement learning (DRL. tasks. We analyze a number of major obstacles in taking data-driven approach, and present a suite design principles, novel findings, and critical insights about data-driven visual DRL. Our framework has three stages: in stage 1, we leverage non-RL datasets (e.g. ImageNet) to learn task-agnostic visual representations; stage 2, we use offline RL data (e.g. a limited number of expert demonstrations) to convert the task-agnostic representations into more powerful task-specific representations; stage 3, fine-tune the agent with online RL. On set of challenging hand manipulation tasks with sparse reward and realistic visual inputs, compared to the previous SOTA, VRL3 achieves average of 780% better sample efficiency. And on the hardest task, VRI.3 is 1220% more sample efficient (2440% when using a wider encoder) and solves the task with only 10% of the computation. These significant results clearly demonstrate the great potential of data-driven deep reinforcement learning.
Introduction
Over the past few years, the sample efficiency of Deep Reinforcement Learning (DRL) has signifi- cantly improved in popular benchmarks such as Gym 4 82) and DeepMind Control Suite (DMC) 80 However, the environments in these benchmarks often look different from the real world (see discussion in section 5.8, page 8 in (73)). It remains unclear whether the sample-efficient methodolo- gies developed for these benchmarks can be successfully extended to more practical tasks that rely on realistic visual inputs from cameras and sensors, such as in robotic control and autonomous driving.
promising direction is to take data-driven approach, that is, try to use all the data available that might contribute to the learning process. Indeed, in the past few decades, the most important advances in deep learning have been facilitated by the of large amounts of data and computation [78] 47). Such a data-driven approach is natural and mature for supervised learning, but not as much so for RL, where learning most commonly performed in an online fashion.
Although recent advances representation learning for RL and offline RL reveal the potential of such approach, most cxisting offline RL methods focus tasks with proprioceptive ("raw-state") input and not the challenging visual control tasks. There yet to be a method that can fully utilize data from multiple different sources to boost learning efficiency. In this work, the goal is to develop a data-driven framework that fills this role. And this framework is designed to be as simple as
"hii work was done when Che Wang was interning with Microsoft Research Asia.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Non-RL Data
Offline RL Data
Online Data
data
GENET
Buffer
Env
Put data buffer
Critic
Critic
Action
environment
Encoder
Encoder
Encoder
Actor
Actor
Contioue
VRL3 Agent
VRL3 Agent
from last
stage
Stage encoder with large image dataset
Stage 2
Stage
Initialize ageat stage encoder, train with offline RL.
Initialize replay builer with stage continue to train with online RT. and collect more data
The design of VRL3. In stage 1, pretrain an encoder using large image datasets to obtain task-agnostic visual representation. In stage 2, we initialize an actor-critic RI. agent with the pretrained encoder, and then use offline RI techniques to train on offline RI. data. Here we finetune the encoder and also learn the actor and critic. stage 3, we initialize a buffer with the offinee data from stage 2, and further train the entire agent with online RL. The idea is simple: to address challenging visual control tasks, the goal is to fully exploit all the available data.
Figure 1:
possible while being sample efficient. As discussed in a number of critique papers 27 28
simplicity in algorithm design is preferred because the performance of RL algorithms be heavily affected by hidden details in the code, and a simpler design often leads to cleaner analysis and better reproducibility. Note that VRL3 is clearly not simpler when compared only trains stage and does not take advantage of data from other sources. However, VRL3 is simpler than other alternative ways utilize data from 3 separate stages. the later sections, provide extensive discussions and ablations to show that VRL3 indeed has simple and intuitive design that can fully exploit all data sources to reach superior performance.
purely online method that
We also make an effort to maximize robustness and reproducibility, by comparing to stronger baselines that re-implemented and improved, presenting extensive ablations and hyperparameter robustness study, listing full technical details, providing open source code, and more. (Source code is being reviewed and cleaned and will be put Github soon).
Figure shows the proposed framework, Visual Deep Reinforcement Learning in 3 Stages (VRL3)). VRI.3 is designed with data-driven mindset, and each stage utilizes particular type of data source.
In stage 1, we learn from large, existing non-RL datasets such as the ImageNet dataset. These datasets might not be directly relevant to the RL task, but they provide task-agnostic visual representations that might be helpful, especially if the RL task also has realistic visuals. A major challenge here is that the non-RL data can come from a different domain with different input shapes. Special care is required make sure the representations obtained here can be smoothly transferred the RL task.
In stage 2, we learn from offline RL data, such as a small number of expert demonstrations, or data collected by previous learning agent. This offline data is related to the RI. task and contains important information about solving the task. We initialize an actor-critic RI. agent with the pretrained encoder and then employ offline RL techniques which not only train the actor-critic RL agent but also fine-tune the encoder, thereby creating task-specific representation.
In stage 3, we learn from online RL data. Our main goal is sample efficiency (achieve high perfor- mance with as little online data as possible). We, therefore, use off-policy training. which uses and re-uses all available data during training. Note that naively transitioning into standard online RL can lead to training instability and destroy what we learned from the previous stages. We avoid this instability by constraining the maximum Q target value during off-policy updates.
Although each stage brings number of unique challenges, we show that VRL33 can address all of them and take full advantage of the different data sources. Our contributions follows:

---

1. Novel framework: we present a data-driver framework that can fully exploit non-RL, offline RL, and online RL data to solve challenging tasks with sparse reward and realistic visual input. Most existing visual methods do not utilize offline data, and most offline RI. methods do not work with difficult visual tasks, making our framework a novel contribution. We provide source code and a full set of technical details to maximize reproducibility.
2. Novel technical contributions: we propose convolutional channel expansion (CCE) to enable an ImageNet-pretraing encoder to work with RL. tasks that take in multiple input images. We also propose the Safe Q target technique for minimal-effort offline to online transition.
3. Novel insights: we show that offline RL data (expert demonstrations) can provide a benefit that cannot be replaced by other stages, and that conservative RL might be a superior option when exploiting offline data, compared to contrastive learning. We also show that a reduced encoder learning rate helps learning a multi-stage setting. And interestingly, representation pretrained with non-RL data, despite the large domain gap, can give better performance compared to representation learned from limited RL data, but is outperformed when RL data is ample. We also find stage pretraining helps more in more difficult tasks.
4. Novel results: We present extensive ablations that reveal the effect of each component of VRL3, and conduct a large hyperparameter study on tasks with different difficulties, providing useful guidelines on how to deploy and tune our framework to new tasks.
5. Significant improvement over the previous SOTA: VRL3 achieves new level of SOTA sample efficiency (780% better on average), parameter efficiency (3 times better), and computation efficiency (10 times faster to solve hardest task) on the challenging Adroit benchmark while also being competitive on DMC. When using a wider encoder, we can even reach 2440% better sample efficiency on the hardest Relocate task compared to the previous SOTA.
2 Related Work
VRL3 has a straightforward design and effectively combines representation learning, offline RL, and online RL. We now discuss related work and how contribution is novel and different from prior works, with a focus on model-free methods. Additional related work is discussed in Appendix D
2.1 Learning visual tasks with online RL
With the help of data augmentation, many visual RL tasks can be learned in online fashion. Typically, agents trained with such methods use convolutional encoders to extract lower-dimensional
features from the visual input, and train in end-to-end fashion [35 93 4 of methods uses contrastive representation learning to solve visual tasks learning can be used to learn useful representations without gradients from the RI. objectives. These methods work well in environments with simple visuals, such as in the DMC environments, but can fail entirely if naively applied to challenging control tasks with sparse rewards and more realistic visuals such as Adroit, due to the difficulty in both exploration [66] and learning representations for realistic visual inputs 773. Different from these works, VRL3 can fully exploit offline RL data and even non-RL data and can achieve SOTA performance in both Adroit and DMC.
94). Another class
77 99 Contrastive
2.2 RL with ImageNet pretraining
number of previous works have used a pretrained encoder, such as a ResNet trained on ImageNet to help convert visual input into oowddimesiional features, the encoder can be further finetuned or
64 72). However, such pretraining alone might not be sufficient to achieve the best performance or computation efficiency. Naively initializing the agent encoder might not provide a significant advantage over training from
kept frozen during training [73 16 31
22
48
with an ImageNet-pretrainee scratch [31]. Most of these works do not incorporate offline RL data. RRL [73] uses offline RL data but does not exploit them to the full extent. Different from these works, we show how ImageNet pretraining can be best combined with offline and online RL and provide a series of novel analyses, interesting results, and important insights. Other pretraining settings are also explored, [90] show
ctmpgle..cmm//aeee edu/vr13
3

---

masked image pretraining can lead to strong performance on a new suite of robotic tasks. 662 and [59] show that pretraining on different datasets and with more sophisticated methods can be combined with imitation learning. Compared to them, we study RI., which is different subject, and we also achieve better success rate on Adroit (95%, while they have 85% and <70%, respectively).
2.3 Offline RL, and data-driven RL
In the past few decades, some of the most important advances in Al research have been based on data- driven methods, so it is natural to ask whether we can also utilize similar mindset for DRL 47 78
Although off-policy algorithms do not naively work in offline settings [8] 49), many specialized
algorithms are proposed to allow effective offline training [18 37 8 63 88 36 17
87
34
92 2 20
[8]. Recent works also study the transition from offline to online [58
46 102 46
49 However,
most of these works are tested on tasks with proprioceptive (positions & velocities) input, and they do not consider pretraining from non-RL. data. Our work is different and focused on challenging control tasks with visual input, sparse reward, and limited demonstrations.
Note that Adroit has often been studied with proprioceptive and not the more challenging visual input. RRL 773 and FERM [98] are the most relevant to our work. RRL uses an ImageNet-pretraing encoder, performs behavioral cloning (BC) offiin RL data, and then performs on-policy learning. Prior to our work, RRL is the SOTA on visual Adroit and is the only method that achieves non-trivial performance on the hardest Relocate task. FERM performs contrastive learning on offline RL data and then performs off-pooicc online RL with data augmentation. FERM can learn simple robotic tasks quickly but has brittle performance in Adroit 773.
In most prior works, the different training stages have been studied separately. What missing is a framework that can combine all stages effectively and efficiently. A major novel contribution of this work is such a framework and a comprehensive study on how its components interact and build up towards superior performance.
3 VRL3: Visual DRL in 3 stages
We now present the design our framework VRL3. We focus on the high-level ideas and provide additional technical details in Appendix and pseudocode in appendix [C. Our agent has a convolu- tional encoder, two Q networks, two target Q networks and a policy network. Note the encoder can be trained separately from the policy and Q networks.
Stage We follow the standard supervised learning routine and train a convolutional encoder fe on 1000-class ImageNet classification task. We follow the standard training procedure of a typical ResNet model. During training, we shrink the images to the size of 84x84 so that they match the input dimension of our downstream RL task. In our main results, also lightweight encoder with 5 convolutional layers, with batch norm after each layer. In appendix A.7 we show additional results with deeper ResNet models.
Transition to multi-frame RL input After stage 1, we expand the first convolutional layer of the encoder so the input channel size matches the RL task which can take in multiple frames, and the weight values this layer are divided accordingly to maintain features to later layers, we refer to this convolutional channel expansion (CCE) (Further discussed in Section 5
Stage We first initialize an RL agent with the pretrained encoder and put the offline RL data in an empty replay buffer D. Note that for Adroit, we have standard 25 expert demonstrations per task (collected by human users with VR) (66). We build our implementation on top of DrQv2 because is simple off-policy actor-critic algorithm with SOTA performance and clean and efficient codebase 93.. Here we also try to follow the notations in DrQv2 for better consistency.
Input and image augmentation Let aug denote random shift image augmentation, and cat for concatenation. Let æ be the visual input from the camera (or cameras). the Adroit environment, in addition to the visual observation, we are also given sensor values from the robotic hand, which is a vector of real numbers, denoted by z. The augmented visual input (image frames) put into the encoder, giving domer-dimensional representation h fe (aaggx)). We then concatenate it with the

---

sensor values (this is skipped in DMC). For simplicity, let s cat(h, z) denote the concatenation; the input to the Q and policy networks, which are multi-layer perceptrons.
Offline RL update For each update, we sample a mini-batch of transitions (ri, Ze, Ot, -tt--n-1; Pt+n Zt+-a) from the replay buffer D. Note have t + here to compute n-step returns [93]. Let 8, cat(h,, zt). Let ār ~ (s()s) be actions sampled from the current policy. The Q target value is: y Ei=o r're- +.2 m Q8, (st, āctn).
The standard Q loss Lo is (for k 1) (D) EE [Q0, (h,, az) - y)
Following 33), and build on top of our framework, we use additional conservative Q loss Loc term: (oc E [log Ea. exp(Q(s;, a,)) - Q( az)] On high-level, this loss essentially reduces the Q value for actions proposed by the current policy, and increase the Q values for actions in the replay buffer (37].
The standard policy (actor) loss L is: (D) = -EyD min Qo, (st. at)]
Let a be the learning rate for the policy network and the Q networks. Let Bene be the encoder learning rate scale, so that the encoder learning rate C'enc = Be For each offiine RL update in stage 2, we perform gradient descent (D) + (Dc(D) to update the encoder Se and the Q networks Qo Qay and gradient descent on L. to update the policy π.. Note that we do not use the gradient from the policy to update the encoder, following previous work [94] 93). We also use Polyak averaging hyperparameter T to update target networks, as is typically done.
Safe Q target technique When computing the Q target in stages 2 and 3, we propose a simple yet effective solution to prevent potential Q divergence due to training instability and distribution shift. For all tasks, we set a maximum Q target value. To decide this threshold value, we reshape the maximum per-step reward Tmax to be 1 and estimate what the maximum Q value should be for an optimal policy (Details in Appendix A.2). To be concrete, we can compute the sum of this infinite geometric series, Qmax = >t=n r
maxx 100 when 7 0.99 (7 discount factor). Empirical results show that this computed threshold is robust and does not require tuning (as shown in Figure le in the appendix [ slightly lower or higher threshold values such as 50 and 200 also give similar performance). Now if during any Q update, Q target value y exceeds this threshold value. then we reduce it to be closer to the threshold value: if y > Q maz + 1, then f Qmax + (y - Qmax)", " where 0 S 17 < 1 is the safe Q factor. Notice the Qmax 1 here is a technical detail to address edge cases.
Reduced encoder learning rate Due to the domain gap between ImageNet and the RL task, we apply a reduced encoder learning rate to avoid the disturbance of pretrained features in stages 2 and
We continue to the replay buffer D, which already contains the offline RL data. We now perform standard online RL, and newly collected data are added to D. The update is the same as in stage 2, except we remove the conservative Q loss LQo(D).
Stage 3
4 Results
Figure 2a shows success rate comparison averaged over all 4 Adroit tasks, Figure shows results on the most challenging Relocate task, and Figure 2c shows average performance comparison on all 24 DMC tasks. We use the following methods as our baselines. DrQv2 is an off-policy online RL method that has the SOTA performance on DMC. RRL is an on-policy method that uses pretrained ImageNet features and utilizes demonstrations with behavioral cloning and has SOTA performance on Adroit. FERM is a 2-stage method that first trains an encoder with contrastive learning on offline RL data and then perform online RL. Note that currently there are not many successful multi-stage stage visual RL frameworks like VRL3, and as discussed in section [ most existing offline-online methods are not designed to work with the more challenging visual inputs, so they are not included.
Getting stronger baselines Naively, DrQv2 does not use offline data and does not work well in Adroit due to the difficulty in exploration. And FERM has brittle performance on Adroit [73. To create stronger baselines, we introduce variants of DrQv2 and FERM, which are both built on top of our framework. Specifically, DrQv2FD(VRL3 does not have stage 2 training but starts stage training with demonstrations the buffer. FERM(VRLL3) our re-implementation based on VRL33 and has much stronger performance compared to the results in [7] (more robust, is about 2 times
5

---

1.0
800
Neluute
0.8
0.8
VRL3
Guocess R30
600
0.5
0.4
RRRL
Success
0.6
0.4
FERMVRRL3)
FERM+S1
DrOvZIDVRL3)
400
VRL3/S123)
HERMOAALS)
VRL3+FERM
DrOv2
0.2
FERM-S1
0.2
200
DrOv2TD
VRL34FERM
FERMVRLL)
0.0
0.0
Frames
(a) 4 Adroit tasks
(b) Relocate
(c) 24 DMC tasks
(d) Adroit Suite
66
Figure 2: (a) Average success rate over 4 visual Adroit tasks. The performance for VRL3 is averaged over 10 seeds (seeds 0-9). To ensure consistency with prior work, results for RRL (Adroit SOTA) are provided by the authors. VRL3 achieves the strongest performance with a simple design. (b) Success rate for the hardest Relocate task. (c) Average performance over 24 DMC tasks. VRL3 is still the best, though the performance gap is smaller. Results for DrQv2 are provided by the authors.
faster, and reaches a 1.25x average success rate). For Adroit, we also compare with FERM+S1, which FERM(VRL3) plus S1 pretraining, and VRL3+FERM, which is VRL3 with additional contrastive encoder learning in stage 2. For these improved baselines, we use the hyperparameters we searched for VRL3 for maximum performance. Doing so helps ensure our comparison actually reflects the differences in algorithmic design and not hidden implementation details 17 27
Figure 2a shows that interestingly, for Adroit, both FERM(VRL3) and DrQv2ID(VRL3) outperform RRL in the early stage, showing that with the right implementation, these off-policy agents can learn quickly even without stage 1 representations. Figure 2 shows that the most challenging Relocate task, DrQv2fI and FERM variants fail to learn entirely because they do not exploit stage 2 data sufficiently. VRL3 greatly outperforms all other methods.
Significantly stronger SOTA sample efficiency on Adroit: Compared to RRL, VRL3 achieves an average of 780% better sample efficiency over all tasks and is 1220% faster (2440% when finetuned) to solve the most difficult Relocate task (in terms of the data required to reach 90% success rate). VRL3+FERM has the same performance VRL3, showing that additional contrastive learning is not necessary. FERM+S1 is stronger than FERM, showing that stage pretraining also works well with contrastive stage learning. For parameter and computation efficiency, we also much smaller encoder compared to RRL. We are 50 times more parameter efficient the encoder and times more parameter efficient for all networks in the agent. VRL3 is also more computationally efficient, solving the hardest task with just 10% of the computation (Detailed comparison in Appendix B
For DMC, we collect 25K data with fully trained DrQv2 agents to enable stage 2 training. As shown in Figure 2c, VRL3 has the best performance in the early stage, FERM(VRLS and DrQv2ID are slightly weaker, but achieve the same performance at 1M data. All three of them outperform DrQv2 (results from authors). Note that DMC is typically treated as an online RL benchmark, and DrQv2 is an online method, we do not claim can work this popular benchmark and can utilize offline data to learn faster. Compared to DMC, Adroit is more challenging due to its sparse reward setting and more complex, more realistic visuals with more depth, lighting, and texture information. This might explain the larger performance gap in Adroit. This shows the advantage of VRL3 is more evident in more challenging visual tasks.
SOTA. These results are only to show that VRL3
5 Challenges, Design Decisions, and Insights
In addition to the strong performance on Adroit, another advantage of VRL3 is its simplicity. The simple design of VRL3 not only makes it much faster and more lightweight than the previous SOTA but also allows us to easily ablate and study its core components. In this section, we discuss some technical challenges and provide a comprehensive analysis of how the different components of VRL3 work together towards superior performance. We focus on Adroit and all experiment results are averaged over 4 Adroit tasks (per-task plots in Appendix A).

---

5.1 Stage 1: Pretraining with non-RL data
The first challenge we face in stage 1 is input shape mismatch. In stage 1, the input to the encoder is a single image with RGB channels. For the downstream RL task, however, can be beneficial for the agent to take in multiple consecutive frames of images as input, or frames from multiple different cameras (sometimes even a combination of both), so that the agent can learn to use temporal and spatial information in these frames for better performance.
naive solution is to put each frame separately into the pretrained encoder, as is done in 73 This naive approach has two downsides: 1) it requires more computation, and 2) even if we finetune the encoder, it cannot learn important temporal and spatial features from the interaction of frames.
Advantage of convolutional channel expansion (CCE): We propose the CCE technique to address the above issue. Let m be the number of input frames for the RL task, With CCE, we expand the first convolutional layer of the stage I encoder so that it can now take in inputs of 3m channels instead of 3. We repeat the weight matrix m times and then scale all the weight values 1/m of their original values. This allows us to modify the input shape without disrupting the learned representations. Such simple method (can be done in one line of code, with no additional computation overhead, ablation study against 4 other variants in Appendix A.9) can effectively solve the mismatch problem when transitioning a visual encoder from the non-RL. RL pretraining stage to the RL, stages.
To enable faster experimentation and avoid difficulties in training deep networks in DRL 3 61,, we use lightweight encoder with 5 convolutional layers, which is 50 times smaller than the ResNet34 used in RRL. Quite surprisingly, VRL3 can achieve superior performance with such a tiny encoder.
5.2 Stage 2 and Stage
In stage 2, why use conservative RL updates instead of BC or contrastive representation learn- ing? We use conservative RL for the following reasons: a) we can finetune the encoder to get task-specific representations with RL signal b) comparing with contrastive learning methods which only train the encoder, we can train the encoder as well as the policy and Q networks, giving a jump-start in stage 3. c) With the safe Q technique, this also brings us a minimal-efioo design that allows us to transition effectively from stage 2 to stage 3. We can simply remove the conservative term without any elaborate modifications. d) this fits well with an off- off-policy backbone, allowing maximum data reuse. (BC such as in [66] only works with on on-policy methods). Next, we discuss extensive empirical evidence that supports our design.
Figure 3al shows that for VRL3, only using BC on expert demonstrations in stage 2 does not give a good performance. In fact, using BC in stage 2 gives a similar performance to when stage is entirely disabled. Figure 3a shows that for VRL3, only using BC on expert demonstrations in stage leads to the same performance compared to when stage 2 is entirely disabled. Additional ablation can be found in Appendix A..8 This problem caused by a distribution mismatch of actor and critic at the beginning of stage 3 since BC only trains the actor and not the critic. For contrastive learning, we observe from Figure? 2 that FERM has slightly worse performance in DMC and much worse performance in Adroit. This is because Adroit has a more challenging sparse reward setting and FERM fails to fully exploit offline RL data by only updating the encoder. These results show that when tackling challenging tasks, learning only the representation in stage 2 can be useful, but is not enough to fully exploit offline RL data. This a novel and important finding that has never been carefully studied or emphasized in the literature.
Effect of safe Q target for stage 2-3 transition: Special care is required effectively perform off- oif-policy learning in stage 2. In VRL3, we perform conservative updates in stage 2 and remove the conservative term in stage When we transition from stage 2 to stage 3. the new online data will cause a distribution shift that can lead to instability in Q values (38), The safe Q target technique is a simple and efficient solution this issue, Figure and and show the effect of these design decisions: updating with naive RL in stage 2 (S2 Naive) or removing the safe Q technique (No safe Q) lead to severe overestimation in stage and cause performance drop. Updating with conservative RL in stage 3 (S3 Cons) leads to severe underestimation and even worse performance. Naive RI in stage 2 with safe Q in stages and (S2 Naive Safe) can mitigate overestimation but is still weaker than VRL3, which takes conservative updates in stage 2 and use safe Q in stages 2 and 3.
7

---

1.0
1.0
1000
1.0
0.8
0.8
0.8
600
Sucoess Ralo
0.5
0.6
Valuu
0.6
Sucosss
VRLS
0
VRL3
Sucosss
52 Naiva
52 Neiva
0.001
Natve Safe
-500
Natve Safe
0.01
Stage 2 BC
53 Cons
53 Cons
0.1
Stage dieabled
No dafe
No Q
0.0
-1000
0.0
2 Frames
2 Frames
2
Frames
1eB
1e8
1eB
Frames
1e6
(a) Stage BC or disable
(b) S2-S3
(c) S2-S3 Q value
(d) Encoder Ir
Figure 3: a) Effect of BC and offline RL updates in stage 2. Only applying BC disabling stage 2 training entirely leads to poor performance. b) and c) Taking naive RL updates in stage 2 (S2 Naive) leads to severe overestimation, which can be mitigated by safe (S2 Naive Safe). Taking conservative updates in stage 3 (S3 Cons) leads to underestimation. VRL3 gives the best performance with conservative update only stage 2 and safe Q. Note in our setting, the maximum reasonable Q value should be around 100. d) Effect of different encoder learning rates (e.g., 0.01 means the encoder's learning rate 100 times slower than that for the policy and Q networks.)
When we fine-tune the encoder in stages and 3, a reduced encoder learning rate can prevent pretrained representations from getting destroyed by noisy RL signals, especially in the early stage of RL training. This problem has also been observed in other environments [69]. Figure [3] shows the effect of encoder learning rates. These results show that even though there is a large domain gap between ImageNet and RL tasks, a frozen stage I encoder can still provide useful features, and finetuning the encoder can further improve performance. Although it not new that transfer finetuning can give the best result [95] it has not been adequately studied in the RL setting with large domain gap and noisy signals.
Reduced encoder learning rate for better finetuning:
5.3 Contribution of each stage
VRL3 achieves strong performance by fully exploiting data from 3 different stages. We now try to understand how much each stage contributes the final performance. We perform 2 sets of comparisons, for each VRI.3 variant in Figure 4a we disable one or more stages of training entirely (e.g. S13 means stage training is removed), and for Figure 44b we disable the encoder training in one or more stages, but always maintain the updates to actor and critic networks (e.g. S12 means encoder frozen in stage 3).
0.8
Sucoss Rate
Sucoosa Rale
0.6
$123
51
0.4
0.4
5521
0.2
$13
0.2
0.0
Framess
168
Frames
168
(a) Stage ablation
(b) Encoder ablation
Figure 4: (a) Effect of each training stage. The numbers after "S" indicates the stages that are enabled. (b) Effect of enabling encoder training in each stage.
Here we summarize number of insights we can obtain from these results. In Figure 4a Stage I
pretraining improves sample efficiency (S123> > S23); stage 2 is critical and its contribution cannot be replaced by the other stages (S123> S13, S23 S13). Note that since Adroit is challenging with limited offline data, S3 is always enabled to obtain non-trivial performance.
In Figure 4bb when the encoder is only trained in one stage, stage encoder training gives much stronger performance than the other two, showing the importance task mask-specific representations (S3 > SI and S2). Note that interestingly, task-specific features are more useful, however, when RL
8

---

data is scarce, non-RL pretraining provides better results despite the large domain gap (S3> SI S2). Training the encoder in more stages always leads to improved performance (S123> S12 $1).
An important finding is that results indicate offline RL is more beneficial than just contrastive representation learning in stage 2, note that S3 in Figure 4b (perform offline RL update but freeze encoder in stage encoder trained stage 3) has much stronger performance than S3 in Figure 4a (no stage 2 training at all). This is consistent with results in Figure [ the performance gap between VRL3 and FERM+S1 is caused by VRL3 doing offline RL updates in stage 2, and thus learning full agent, instead of just the encoder features.
With the above insights, Table I summarizes fundamental differences between VRL3 and some other popular visual RL methods.
Table 1: Comparison of core design decisions. VRL33 fully utilizes non-RI, offline RL and online RL datasets. DA refers to data augmentation, Con for contrastive learning, Offline for offline RL. Note that offline RL data can be used to learn representation (encoder), or the task (actor, critic), or both.
Characteristics
DrQ/RAD CURL/ATC
FERM
RRL
VRL3
Stage 1
None
None
None
Pretrain
Pretrain
Stage 2
None
None
Con
BC
Offiie
Stage 3
DA
Con
DA
No DA
DA
Leverage large visual datasets
X
Task-specific representations
X
Offline RL data for representations Offline RL data for task learning High data reuse rate (off- ooff-policy)
X
X
X
X
X
X
V
J
X
Prevent Q network divergence
X
X
X
5.4 Additional Studies and Analysis
We also perform extensive study hyperparameter robustness and sensitivity on each of the Adroit tasks (with varying difficulty). Due to limited space, we only show a small portion of our study in Figure and put the rest of the results in Appendix AA] This provides additional insights and helps us understand what hyperparameters should be tuned with high priority when applied to a new task, for example, results show that SI pretraining is useful for harder tasks but for the simplest Door task; data augmentation is always critical; the encoder learning rate is important and should be tuned difficult tasks; even a small number of S2 offline updates can greatly help learning. In total, we studied 16 different components and hyperparameters, and identified 9 of them as important to performance, out of these 9, we further show that of them are robust, and 3 of them are sensitive (learning rate, action noise std, encoder learning rate scale). Note that the encoder learning rate scale the only important and sensitive hyperparameter introduced by VRL3, while the other are from the backbone algorithm. These important studies show that our framework is quite robust, we believe these results and insights will provide a better understanding on data-driven DRL methods, and we hope they will make it casier for other researchers to deploy this framework to other tasks.
08
06
Doot
0.6
08
08
02
Pin
02
02
02
Relocate
pretrain
random
augment
sugment
0.001 0.01 0.1
5000 30000 100000
(a) S1 pretrain (R)
(b) S2,3 augmentation (R) (c) S2,3 encoder Ir scale (S) (d) S2 num of update (R)
Figure Part of the hyperparameter sensitivity study. For each figure, X-axis shows different hyperparameter values, Y-axis shows the average success rate in the first 1M training frames, over three seeds. Each dot is the average success rate of one seed, the error bar shows one standard deviation. (S) means critical and relatively sensitive. (R) means critical but is robust or easy to tune.

---

Discussion, Limitations and Future Work
In this paper, we propose 3-stage data-driven framework for solving challenging visual control tasks. We perform extensive experimentation to ablate and analyze our framework to identify the most important technical components. We arrive at method that is simple, intuitive, and much more sample efficient and computationally efficient than the previous SOTA on Adroit. Although this a significant result, we have some limitations: for example, Adroit has more realistic visuals compared to DMC, but it is still a simulated benchmark. Experiments on robotics hardware are required to understand how VRL3 should be deployed the real world. Other limitations include: it remains unclear whether stage training is still beneficial if the domain gap between non-RL data and RL data is too great (RL task visuals different from the real world). We also did not experiment extensively model-based method. These can be exciting future research topics. Despite these limitations, VRL3 brings a number of novel and interesting insights, in addition the strong empirical results. We believe VRL3 is important step toward practical data-driven DRL for real world visual control tasks.
Acknowledgments and Disclosure of Funding
The authors would like thank Kan Ren and Tairan He for number of interesting discussions on the effect of representation learning and pretraining. Thanks to Dongqi Han for helpful discussion and insights on potential encoder design and on the pretanh issue. Thanks Yue Wang for an early discussion on the structure of the paper. Thanks to Han Zheng for insights on offline RL issues. Thanks to Kaitao Song. Xinyang Jiang, Yansen Wang. Caihua Shan. Lili Qiu for helpful feedback and discussions related to supervised learning and pretraining. Thanks to Zhengyu Yang, Kerong Wang, Ruoxi Shi, Bo Li, Ziyue Li, Yezhen Wang discussion and feedback group meetings. Thanks to Kexin Qiu for providing critical feedback on the naming of the framework. Thanks to Denis Yarats for the super clean DrQv2 codebase and answering some related questions. Thanks to Rutav Shah and Vikash Kumar for helping with the RRL codebase and Adroit on GitHub. Thanks to Suraj Nair for answering a question related to R3M computation speed. The authors also want to thank our reviewers and meta-reviewers for providing all the valuable feedback and suggestions.
10

---

References
[1] Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. arXiv preprint arXiv:2008.0553 2020.
[2] Chenjia Bai, Lingxiao Wang. Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202. 1566, 2022.
[3] Johan Bjorck, Carla Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning. arXiv preprint 1rXiv:2106.011 051, 2021.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 015400 2016.
[5] Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample- efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, pages 8234, 2018.
[6] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch, Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
[7] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. International Conference on Learning Representa- tions, 2021.
[8] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 18353-18366 2020.
[9] Hyesong Choi, Hunsang Lee, Wonil Song, Sangryul Jeon, Kwanghoon Sohn, and Dongbo Min. Self-supervised structured representations for deep reinforcement learning. 2021.
[10] Ignasi Clavera, Violet Fu, and Pieter Abbeel. Model-augmented actor-critic: Backpropagating through paths. arXiv preprint arXiv:2005. 08066. 2020.
[1]] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020-2027. PMLR, 2021.
[12] Yuchen Cui, Scott Niekum, Abhinav Gupta, Vikash Kumar, and Aravind Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? Learning for Dynamics and Control Conference, pages 893-905. PMLR, 2022.
[13] Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. International Conference on Machine Learning (ICML), 2020.
[14] Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, and Marc Toussaint. Reinforcement learning with neural radiance fields. arXiv preprint arXiv:2206. 01634, 2022.
[15] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pages 1329-1338, 2016
[16] Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519, IEEE, 2016.
[17] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2 06860, 2021.
[18] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[19] Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: a8Xiv: 2018.
[20] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.

---

[21] Baan-Battien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- sepe-supe learning. Advances in neural information processing systems, 33:21271-21288 2020.
[22] Abhinav Gupta, Adithyavairavan Murali, Dhiraj Prakashchand Gandhi, and Lerrel Pinto. Robot learning in homes: Improving generalization and reducing dataset bias. Advances in neural information processing systems, 31, 2018.
[23] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 2018.
[24] Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and match: Su- percharging imitation with regularized optimal transport. arXiv preprint arXiv:2206. 15469, 2022.
[25] Dongqi Han, Kenji Doya, and Jun Tani. Variational recurrent models for solving partially observable control tasks. arXiv preprint arXiv:1912.1070: 2019.
[26] Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. arXiv preprint 1511v45511044::
2015.
[27] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[28] Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reprodueibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.0413: 2017.
[29] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pages 12519-12530, 2019.
[30] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
[31] Ryan Julian, Benjamin Swanson, Gaurav S Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. arXiv preprint arXiv:2004.10199 2020.
[32] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:200.005551, 2020.
[33] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.690
2014.
[34] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offfline reinforcement learning with implicit q-learning. arXiv preprint arXiv: 10.06169, 2021.
[35] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regulariz- ing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[36] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint 1eti::990.009.// 2019.
[37] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 2020.
[38] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.11590 2018
[39] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. arXiv preprint arXiv:2005.00 02000, 2020.
[40] Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimiza- tion. arXiv preprint arXiv:2007.0199 2020.
[41] Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin geleannin:: Control- ling the estimation bias of q-learning arXiv preprint 27X1v:2002.06448, 2020.
12

---

[42] Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.0000 02057, 2019.
[43] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 2020.
[44] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised rep- resentations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[45] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint arXiv:2007.0492:3 2020.
[46] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
[47] Sergey Levine. Understanding the world through action. In Conference on Robot Learning, pages 1752-1757. PMLR, 2022.
[48] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abheel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-137, 2016.
[49] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.00164 2020.
[50] Xiang Li, Jinghuan Shang, Srijan Das, and Michael Ryoo. Does self- self-supervised learning really improve reinforcement learning from pixels? arXiv preprint arXiv:2206.05266, 2022.
[51] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wwt.vv:550000099 02977................. 2015.
[52] Nan Lin, Yuxuan Li, Keke Tang. Yujun Zhu, Xiayu Zhang, Ruolin Wang. Jianmin Ji, Xiaoping Chen, and Xinming Zhang. Manipulation planning from demonstration via goal-conditioned prior action primitive decomposition and alignment. IEEE Robotics and Automation Letters, 7(2):1387-1394, 2022.
[53] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.0956/ 2022.
[54] Lingheng Meng, Rob Gorbet, and Dana Kulić, The effect of multi-step methods on overesti- mation in deep reinforcement learning. arXiv preprint arXiv:2006. 12692, 2020.
[55] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.55002, 2013.
[56] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[57] Yao Mu, Shoufa Chen, Mingyu Ding, Jianyu Chen, Runjian Chen, and Ping Luo. Ctrlformer: Learning transferable state representation for visual control via transformer. arXiv preprint arXiv:2206.0888 2022.
[58] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. 2020.
[59] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
[60] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, pages 16828- 16847. PMLR, 2022.
[61] Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. arXiv preprint arXiv:2102.0799 2021.
13

---

[62] Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsur- prising effectiveness of pre-trained vision models for control, arXiv preprint 2:Xx::2003.33880, 2022.
[63] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintage-weighted regres- sion: Simple and scalable off off-policy reinforcement learning. arXiv preprint arXiv: 2019.
[64] Lerrel Pinto and Abhinav Gupta. Supersizing sulf-suuee Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406-3413. IEEE, 2016.
[65] Senthil Purushwalkam Sh. Visual Representation and Recognition without Human Supervision.
PhD thesis, Carnegie Mellon University, 2022.
[66] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrationn.. arXiv preprint arXiv: 10087, 2017.
[67] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. arXiv preprint arXiv:2004.07884, 2020.
[68] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforce- ment learning? arXiv preprint XrXiv22201.12122 2022.
[69] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, Devon Hjelm, Philip Bachman, and Aaron Courville. Pretraining representations for data- efficient reinforcement learning. arXiv preprint arXiv:2106.04477 2021.
[70] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. arXiv preprint arXiv:2206. 14244, 2022.
[71] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, pages 19561-19579, PMLR, 2022.
[72] Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXiv: (eti::161.00000 2.69699, 2016.
[73] Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. In Self-Supervision for Reinforcement Learning Workshop-ICLR 2021, 2021.
[74] Wenling Shang. Xiaofei Wang, Aravind Srinivas, Aravind Rajeswaran, Yang Gao, Pieter Abbeel, and Misha Laskin. Reinforcement learning with latent flow. Advances in Neural Information Processing Systems, 34:22171-2211 221133, 2021.
[75] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak. Robotic telekinesis: learning a robotic
hand imitator by watching humans on youtube. arXiv preprint arXiv:2202. 2022.
[76] Maks Sorokin, Jie Tan, C Karen Liu, and Sehoon Ha. Learning navigate sidewalks in outdoor cnvironments. IEEE Robotics and Automation Letters, 7(2):3906-311,, 2022.
[77] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint 2xXv:20099088811 2020.
[78] Richard Sutton. The bitter lesson. http::/v/n hatps//tttt/iiiiccoooleeeeeeeee http://wwww.ccooppeeeeeadaaaaeea httpppp/lleeee.aaae..eeeee/dddad
BitterLesson. html 2021. Accessed: 2022-01-24.
[79] Allison C Tam, Neil C Rabinowitz, Andrew K Lampinen, Nicholas A Roy, Stephanie CY Chan, DJ Strouse, Jane X Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. arXiv preprint arXiv:2204. 05000000000000, 2022.
[80] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki. Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: /001. 006900 2018.
[81] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
[82] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026-5033. IEEE, 2012.
14

---

[83] Ikechukwu Uchendu, Ted Xiao, Yao Lu. Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma. Jiantao Jiao, et al. Jump-start reinforcement learning. arXiv preprint arXiv:2204.02372 2022.
[84] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In AAAI, volume 2, page 5. Phoenix, A7, 2016.
[85] Homer Walke, Jonathan Yang, Albert Yu, Aviral Kumar, Jedrzej Orbik, Avi Singh, and Sergey Levine. Don't start from scratch: Leveraging prior data to automate robotic reinforcement learning. arXiv preprint 220i::22000040033. 2022.
[86] Che Wang, Yanqiu Wu, Quan Vuong, and Keith Ross. Striving for simplicity and performance in off-policy drl: Output normalization and non-uniform sampling. In International Conference on Machine Learning, pages 10070-10080. PMLR, 2020.
[87] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. In Advances in Neural Information Processing Systems, pages 6288-6297, 2018.
[88] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911.11311, 2019.
[89] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhut- dinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08144 2021.
[90] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06/73, 2022.
[91] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXi::2/02.005815 2021.
[92] Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201 13425, 2022.
[93] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint 21Xiv:2107.09645. 2021.
[94] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[95] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv preprint 1x1.v:441.11992. 2014.
[96] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
[97] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022.
[98] Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin. A framework for efficient robotic manipulation. arXiv preprini arXiv:2012.07975. 2020.
[99] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine, Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint rrXiv:2006.1072. 2020.
[100] Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor K Prasanna. Maximum entropy model rollouts: Fast model based policy optimization without compounding errors. arXiv preprint arXiv:2006. 04802, 2020.
[101] Qihang Zhang. Zhenghao Peng, and Bolei Zhou. Action-conditioned contrastive policy pretraining. arXiv preprint urXiv:2204.02.99, 2022.
[102] Han Zheng, Xufang Luo, Xuan Song, Dongsheng Li, Jing Jiang, et al. Adaptive q-learning for interaction- limited reinforcement learning. 2021.
15


Reviews:
[{"review_id": "l1cJRGS0DgD", "reviewer": "Reviewer_i5GH", "summary": "This paper outlines a 3-stage process to train policies. Stage 1 involves training an encoder using large image datasets to obtain task-agnostic visual representation. Stage 2 involves training a policy using offline RL and the pre-trained encoder. This serves a as a good initialization for the final stage 3 that involves further training the agent with online RL.", "questions": "Please see strengths and weaknesses section for details and context. My main questions/concerns are related to:\n\n1. Novelty of VRL3 and its relationship to prior work (that appeared sufficiently before NeurIPS deadline) -- MVP, PVR, R3M etc.\n\n2. Lack of rigorous comparisons to substantiate utility of CCE and SafeQ over current SOTA baselines (as opposed to naive straw-man baselines).", "limitations": "The limitation section in the paper is too generic (e.g. evaluation in sim vs real robots). ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The overall pipeline appears obvious and straightforward. In my view, the main contribution of the paper is putting together the components and demonstrating empirical success. \n\nWeaknesses: While the overall approach and results look promising, the current submission unfortunately lacks technical depth and rigor. In the introduction section, the authors outline their main contributions. Unfortunately, I do not believe there is sufficient evidence to rigorously substantiate these claims.\n\n1. \"Novel framework...\" -- I believe there is limited novelty in this submission, especially in light of several recent papers (most of which are published by now). For example, the current submission does not cite or discuss MVP[1], PVR[2], R3M[3] which all explore training encoders from out of domain data.\n\n2. \"Novel technical contributions...\" the main technical contribution claims are: (a) CCE or convolutional channel expansion; (b) safe Q technique. As for CCE, there are no experiments to benchmark it against simple alternatives like contatenation of framewise embeddings [1,2,3] or their latent differences [4], which have been successfully used in several prior works. The safeQ update does seem to help compared to a naive update rule. However, the challenges in transitioning from offline RL to online RL has been well-documented in prior work, along with different solutions (e.g. AWAC [5]). The authors should consider comparisons with such SOTA baselines as opposed to naive straw-man baselines. Overall, due to the lack of rigor in comparisons and baselines, it is hard to judge the importance or utility of the claimed technical contributions.\n\n\"VRL3 achieves an entirely new level of SOTA sample efficiency, parameter efficiency, and computation efficiency on the highly challenging Adroit benchmark while also being very competitive on DMC.\" -- I would encourage the authors to rephrase such statements to more technically rigorous and falsifiable claims. \n\n[1] Xiao et al. Masked Visual Pre-training for Motor Control. arXiv 2022.\n\n[2] Parisi et al. The Unsurprising Effectiveness of Pre-Trained Vision Models for Control. ICML 2022.\n\n[3] Nair et al. R3M: A Universal Visual Representation for Robot Manipulation. arXiv 2022.\n\n[4] Shang et al. Reinforcement Learning with Latent Flow. NeurIPS 2021.\n\n[5] Nair et al. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hmu_hDPixAF", "reviewer": "Reviewer_1wyj", "summary": "This paper proposes a 3-stage visual RL framework, consisting of encoder pretraining with ImageNet, offline finetuning, and online finetuning. The encoder pretraining stage leverages an off-the-shelf large-scale dataset to learn a powerful representation. Then, the offline pretraining uses a conservative method to transfer the learned representation into task-specific embeddings. Finally, the online rl achieves high sample efficiency in accomplishing adroit  tasks. The core contribution of this paper is the usage of imagenet dataset and offline  method to perform pretraining.  The authors also conduct detailed ablative experiments to understand the challenges and important design choices. ", "questions": "Per the previous section, could you re-write some key sections in your paper?\n\nCan you explain the sensitivity caused by Q threshold? If other people want to adopt this framework, how they can find a good Q threshold？\n\nAre the data augmentations chosen randomly or on purpose? Why do you choose this augmentation?", "limitations": "The authors mention real robot experiment and large domain gap as their limitations. These are reasonable. Thank you for being straightforward.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \nAlthough all the components are not new, the combined framework is powerful. Within this framework, the authors conduct extensive experiments to understand possible alternatives and analyze the challenges. This shed light on future researchers who is willing to jump into this field. The sample efficiency on adroit benchmark is much higher than previous baselines and their forged stronger baselines with a large margin.\n\nWeakness:\nThe overall writing quality is slightly concerning. This is an issue throughout the whole paper. For example, when the authors are describing their experimental results, they use very subjective words such as “good “, “very” without supporting evidence. In the introduction, the authors also describe the necessity of their method using the word “we want to …”. These subjective tones heavily reduce the scientific value of this paper. Moreover, there are many grammatical issues.\n\nSlight overclaim, over-selling, and abuse of terms. The author emphasizes their method is a “minimalist” approach; a 3-stage approach with tons of tunings can hardly be regarded as minimalist. When the authors mention offline RL, they should be more careful when they are only using expert demonstrations. This needs clarification somewhere in the paper. The overall paper quality, especially in terms of the experimental aspects, is ok. That being said, I think over-selling the paper is not a good idea. The authors mentioned that it is a ResNet architecture. However, if it does not have skip connect, the naming is inaccurate.\n\nAccording to the supp material, the framework is brittle to many hyperparameters. For example, the Q threshold is extremely important to tune. What would happen if other people want to use this paper? Does the tuning take a lot of effort?\n\nThe data augmentation technique seems important. More reasoning and explanation are needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E_peXAqNjdS", "reviewer": "Reviewer_5Wwb", "summary": "The authors tackle the problem of improving the efficiency of online learning for visual RL tasks. The main idea of the paper is to pre-train an encoder on non-RL data (Imagenet), then train a value function and policy network on top of the shared encoder with offline RL, and finally fine-tune with online data. With this simple framework and a small number of technical tweaks (e.g. how to pre-train the encoder with single-image input but transfer to tasks with multi-image input, how to stabilize the value function), the authors show that their method achieves SOTA sample efficiency on the Adroit benchmark.", "questions": "- Lines 66-68 - I missed if the authors ablate CCE? How does performance compare if the first layer is simply randomly initialized for stage 2?\n- Lines 249-251 - a 5-layer CNN seems quite limiting in terms of the range of tasks that such a network can solve. What's the Imagenet accuracy of this network after Stage 1 pre-training? Is the framework the authors propose stable for deeper networks?\n- Figure 2 - these curves would be easier to read if the legend was moved outside the plot. I also recommend reordering the labels in the legend to match the ordering of the curves in the graph.\n- Algorithm 1 - For me at least I found the algorithm box too confusing to be helpful for understanding the method. I personally feel it distracts from the clarity of description in Section 3 and Figure 1.\n- Lines 173 - 177 - \"To allow more flexibility in the Q network, we can also set the threshold to be higher than this value\". Can the authors elaborate more on why it makes sense to allow the Q network to output larger values than what is already a very high upper bound on the range of Q? I'm also wondering why the authors don't parameterize Q with tanh or sigmoid in order to enforce the correct range and instead use the technique they describe in line 176.\n- Line 307 - \"to\" I believe should be \"than\"", "limitations": "Yes - the authors discuss that experiments on simulated benchmarks do not imply success in the real world.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found this paper a breeze to read. The method is very practical and leverages popular tools in the contemporary ML toolbox (pre-training and fine-tuning, offline RL) in order to achieve impressive sample efficiency on synthetic visual-input robotic tasks. I especially appreciate the careful ablations the authors performed in Section 5 which I believe gives insight into the contribution of each of the 3 stages that the authors propose for their method. I think this paper provides valuable insight for the ML community on the relative importance of leveraging non-RL, offline, and online data for RL tasks.\n\nThe main weakness I see for this paper is I think the authors make some claims about the importance of offline RL vs. BC in section 5.2 that I think need an additional experiment to verify. In lines 263-265, the authors claim that BC is the same as entirely ignoring stage 2 altogether \"since BC only trains the actor and not the critic\". I think BC should provide a stronger baseline than this - maybe the authors could try a stage 2 in which the policy is trained with BC and the value function is trained in a naive way to predict cumulative reward of the offline trajectories? My takeaway from Figure 4 is that as long as we have a decent policy-value match going into stage 3, the tasks will get solved even if the encoder hasn't even been trained yet (with smoother more efficient learning when the encoder is pre-trained on Imagenet). It would be great to disentangle the intricacies of DrQv2 for offline data from the goal of pre-training both the policy and value function in stage 2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1l75npr4EvQ", "reviewer": "Reviewer_vuXw", "summary": "The authors propose VRL3, a 3 stage data driven technique for pretraining convolutional encoders, performing offline RL and fine tuning with online RL. The authors demonstrate their approach in a number of vision based control benchmarks, include Adroit and DMC. \nThe approach greatly improved sample efficiency and outperforms end-to-end methods. \n\nThe usage of offline datasets and pretrained encoders if an interesting field of work, but I am concerned that the problems this framework is evaluated on are somewhat trivial and I would be interested to see the results after more iterations of fine tuning.", "questions": "* The authors retrain their autoencoder on 84x84 images, given they are using the convoluational part, why could they not take the weights from an already existing pretrained resnet 18, for example?\n* What is the benefit of a more convoluted method if I can train my baseline for more iterations and achieve comparable performance?\n* How applicable is this process to other RL algorithms, could you have made comparisons to on-policy training and memory-based agents?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper and methodolgy is clear and easy to read.\n* Improved sample efficiency.\n* The authors share their codebase and plan to release it as open source, with the data, checkpoints etc.\n* This method, could potentially be applied to a large variety of benchmarks and achieve SOTA in terms of sample efficiency.\n\nWeaknesses:\n* The method requires training an encoder at the observation size of the environment.\n* I am not convinced that this approach outperforms pure online methods if they are left to convergence, figure 2 (a) show the performances of RRL still increasing, has this baseline actually converged\n* I am not convinced the approach would transfer well to POMDP settings and more challenging benchmarks (DeepMind Lab, Habitat-lab ...)\n* It would have been interesting to see comparisons of other pretraining regimes for the encoder, (self-supervised, BOYL, contrastive.. etc)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept