FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing
Authors: Kyle Stachowicz
Dhruv Shah
Arjun Bhorkar
Ilya Kostrikov
Sergey Levine
Venue: CoRL 2023 Poster (2023)
Keywords: reinforcement learning
offroad driving
vision-based navigation

Abstract: We present a system that enables an autonomous small-scale RC car to drive aggressively from visual observations using reinforcement learning (RL). Our system, FastRLAP, trains autonomously in the real world, without human interventions, and without requiring any simulation or expert demonstrations. Our system integrates a number of important components to make this possible: we initialize the representations for the RL policy and value function from a large prior dataset of other robots navigating in other environments (at low speed), which provides a navigation-relevant representation. From here, a sample-efficient online RL method uses a single low-speed user-provided demonstration to determine the desired driving course, extracts a set of navigational checkpoints, and autonomously practices driving through these checkpoints, resetting automatically on collision or failure. Perhaps surprisingly, we find that with appropriate initialization and choice of algorithm, our system can learn to drive over a variety of racing courses with less than 20 minutes of online training. The resulting policies exhibit emergent aggressive driving skills, such as timing braking and acceleration around turns and avoiding areas which impede the robot's motion, approaching the performance of a human driver using a similar first-person interface over the course of training.

FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing
Kyle Stachowiczi, Dhruv Shaht, Arjun Bhorkari, Ilya Kostrikov, Sergey Levine UC Berkeley
Abstract: We present system that enables a 1/10th autonomous car to drive at high speeds from visual observations using reinforcement learning (RL). Our system, FastRLAP (faster lap), trains autonomously in the real world, without human interventions, and without requiring any simulation or expert demonstra- tions. FastRLAP integrates several components to facilitate the learning process: we initialize low-dimensional visual representations from similar reinforcement learning objective applied to large offline navigation dataset from other robots, providing a navigation-relevant representation. Given series of checkpoints rep- resenting driving course, we then use sample-efficient online RL to learn fast driving policy, resetting automatically on collision or failure. Perhaps surpris- ingly, our system can learn to drive over a variety of racing courses with less than 20 minutes of online training. The resulting policies exhibit emergent aggressive driving skills, such as timing braking and acceleration around turns and avoiding areas which impede the robot's motion, approaching the performance of human driver using similar first-person interface over the course of training.
Keywords: reinforcement learning, offroad driving, vision-hased navigation
Progression
Stage 1: Learning
Stage 2: Online Learning
Encoder
Critic
Encoder JPrusn
RLPO
Will Clec Time inni
Checkpcitts
Racng Lsp
Figure 1: Fast reinforcement learning via autonomous practicing. By pre-training to learn task-relevant visual features (Stage 1), and deploying our autonomous practicing framework for continuous online improve- ment (Stage 2), the robot can autonomously navigate between sparse checkpoints (blue). recover from collisions (red) and improve its driving behavior to maximize speed (yellow magenta). FastRLAP learns fast driving policies little 20 minutes. Videos available https //sites .ww.oooooo/o.iioeeeoeeeo.ooce com/viien/astrrapp
Introduction
High-speed vision-based navigation presents a range of challenges, requiring a policy that can account for both the vehicle's dynamics and interactions with the terrain and obstacles (Fig. D). Learning-based methods offer a particularly appealing approach to such challenges, they in principle capture arbitrary high-performance driving behaviors while accounting for visual indica- tors. Some prior work has approached similar problems via imitation learning, acquiring end-to-end skills from expert demonstrations [1, 2]. However, if we aim to maximize performance, we might
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Figure 2: FastRLAP learns fast driving policies for a 1/10" vehicle operating in diverse indoor and outdoor environments with challenging terrain in tens of minutes, using offline pre-training and online RI.
instead prefer to directly adapt the driving strategy to the vehicle autonomously. In principle, rein- forcement learning allows an agent to continually improve based on its experience, as shown previ- ously in board games and robot manipulation where RL can even exceed human performance [3-6].
However, in practice, learning autonomous navigation with RL presents major challenges. Because we cannot reset the system to a random state, the learning process is highly dependent on the sys- tem's ability to continually reach new states without human intervention. Instead, the RL-based system should train without supervision, while smoothly recovering from failures or collisions. Furthermore, directly learning from high-dimensional observations in the real world can be pro- hibitively slow. Because features are learned from very weak signal (reward), RL often requires huge number of interactions with the environment to learn a robust policy. Alternatively, we could learn entirely from offline data [7, 8], but this yields suboptimal policies when the desired behavior (aggressive driving) is not included in the dataset (low-speed navigation). The goal of this paper is to address these challenges and understand how RL can be applied to learn high-speed driving from vision in the real world. We design a system for Fast Reinforcement Learning via Autonomous Practicing (FastRLAP) which mitigates the sample complexity challenges by first learning a low- dimensional representation of driving-related features such as free space and obstacles from offline data, and then applying online RL to these features learn a fast driving policy. The online RL phase proceeds autonomously, automatically recovering from failures and improving with each lap.
We demonstrate FastRLAP in challenging environments on a custom 1/10m-scale RC car modified for real-world online RI. FastRLAP can autonomously practice and learn aggressive maneuvers over time, improving by up 40% over the demonstration lap and achieving performance close to human expert. Notably, the online training phase typically takes less than 20 minutes (and as little as 5 minutes), depending on the size of the environment. During this time, the robot learns complex maneuvers such as drifting, avoiding low-speed or bumpy areas, and maintaining a racing line, without requiring high-speed human demonstration or explicit reward for these behaviors. The training requires no human interventions and is fully autonomous. To the best of our knowledge, LAP is the first instantiation of vision-based mobile robotic system that uses model-free RI. to autonomously practice high-speed driving maneuvers and improve online in the real world.
2 Related Work
Leveraging prior data to bootstrap online learn- ing has been widely studied in the con- text of supervised learning [9], representation learning [10], continual learning [11-15], and RL [8]. Offline RL in particular has proven powerful due to its ability to directly learn policies from large datasets, which can be fine-tuned through online interaction [16-19]. This has enabled variety of robotic systems leveraging com- bination of offline data and online interaction to perform real-world manipulation tasks [20-22], typically in controlled spaces in which the offline data consists of many high-performance demon- strations with the same robot in the target environment. In contrast, FastRLAP operates with only low-performance (slow) data primarily from other robots and environments, and the majority of behaviors in the resultant policy do not appear in the original dataset.
Figure 3: High-speed visual navigation faces chal- lenges including: (a) noisy odometry and localiza- tion, (b) overexposure and motion blur, and (c) terrain- dependent overr/undrr-steer.
Existing approaches for learning high-speed driving typically rely either highly accurate position information to define states [23-26], localize visual observations relative to high-fidelity global

---

map [27, 28], or operate via behavioral cloning against some privileged expert [29]. This is pro- hibitive in unstructured environments, where (i) onboard state estimates can be highly inaccurate, and (ii) generating a high-fidelity map is difficult or impossible. FastRLAP learns high-speed driving directly from vision, and improves its behavior by self-practice without using privileged state.
Prior success in learning visual navigation policies typically requires large-scale simulated data [30- 33], passive data [34], human interventions [35, 36] or real-world data from other robots [37]. While these modalities (in particular, simulation) are typically used to overcome the high number of sam- ples often required by RI. algorithms, we demonstrate that it is possible to train such a policy in reasonable time with only real-world interactions, opening the door to policies reflecting complex relationships between vision, dynamics, and terrain (Fig. 3) that might be difficult to simulate.
Several works have studied autonomous real-world RL via safety or reset-frce training [38-43] with applications in robotic manipulation, locomotion, and mobility [20, 44-46]. We draw inspiration from these works to build a high-speed navigation system that uses a finite state machine to practice driving around a circuit. FastRLAP can drive diverse courses 100+ meters in length and continually improves its performance over the course of minutes rather than hours or days.
3 Autonomous Practicing with RL
The objective of our high-speed visual navigation task is to drive through a race course, defined by sequence of position checkpoints {ez}, in the minimum possible time. We assume access to two sources of offline data, neither of which contains the desired high-speed behavior: a large-scale dataset representing common navigation behaviors executed on a different robot, and small dataset including a single lap around the course at low speed. Our system aims to enable efficient end-to-end RL in the real world. FastRLAP has three components (see Fig. 1): high-level finite state machine (FSM) for autonomous practicing (shown in blue), representation of visual observations learned via offline RL (purple), and a sample-efficient RL algorithm for online learning (orange).
3.1 Problem formulation
We frame this task Markov decision process M(S,A,p; with state (V, v, Ot, 9, (pre) S. Here, V E [g128x128x9 128x9 is sequence of 3 RGB images; v, W. R3 denote the robot's body-frame linear velocity, angular velocity, and linear acceleration; the goal g is a body-frame vector to the next checkpoint, expressed as a unit vector and distance; Aprev is the previous action.
In order to align the visual representations learned offline with those most useful for the online task, both oflline learning and online training phases are structured to maximize the same reward, ensuring optimal transfer between the two settings. To this end, we define the reward as the weighted sum of three components: wweed-made-goww which is the dot product of the current velocity with the unit vector pointing towards the current goal; collision penalty proportional to the magnitude of the collision (measured by lateral acceleration) applied only when a collision is detected; and a fixed stuck penalty applied whenever the robot determined to "stuck" by the practicing system.
3.2 Autonomous Practicing and Goal Checkpoint Selection
In the autonomous learning setting, the robot is expected learn in the environment without any episodic resets human interventions. Early in training, the policy may reach irrecoverable states, such as collisions, or otherwise become stuck. Without reset, the learning algorithm may fail due to collapse in the state distribution [41]. To overcome this, we use a simple FSM that switches between a simple collision recovery policy and the learned policy.
When the RL policy reaches checkpoint, the FSM selects new goal corresponding the next checkpoint in the course sequence {c:}. foreing the learner to practice reaching all of the checkpoint goals in sequence. The goal checkpoints are typically beyond linc-of-sight (c.g., Fig. 1, blue), up
3

---

to 40 meters away. If the RL policy reaches an inrecoverable state (see Sec. 4), the FSM commands an automatic recovery policy to provides "pseudo-reset."
3.3 Online RL Training
Algorithm 1: FastRLAP
Data: Navigation dataset D, slow demo Bstow
To maximize reward and continually improve lap times, we use off-policy RL [3, 47]. Off-
1 Keys: Pre- Pre-Training, Practicing, Online RL
while Encoder is not converged do
policy algorithms benefit greatly from perform- ing many training steps for each environment step, known as the update-to-data ratio (UTD):
8, a, s', idx t- (aaaDDaa(())
g + LoadFFtureDDta(( idx Rand(H))
< ComputeReward(s, g)
Trainiu. a, (s'.g)
high UTD leads to efficient learning, but suffers from overfitting [48]. To overcome this limita- tion, we use RLPD [49], which trains an en- semble of crities to avoid catastrophic overesti- mation and overfitting [50] and learns quickly using combination of online interactions and a small amount of suboptimal, on-task data.
while True do On Robot
s +-bservee) if s near g then
10
11
Lg g- NexiCheckpoint(g)
r r -CoomuteReward Speev, Opees 9)
13
SendToWorsstaionn Spree. apeev. T. 8, g)
14
a ~v =<((Siimgg)) Sprapeis 9)
We obtain this on-task data by collecting a sin- gle slow lap in the target environment. While this data is very limited (under minute in most environments) and does not contain fast driv- ing behaviors, even suboptimal demonstations can significantly accelerate online learning by avoiding critic collapse in early stages of train- ing [49]. During online training, we sample 50% of each training batch from this low-speed data, interleaved with 50% of data collected on- line. We found this to be critical to the efficiency of our system in our evaluations (Sec. 5.1).
15
Actuate(u)
if Collision Stuck then
17
Execute recovery policy
18
On Workstation
19
ReceiveFromRohon(3)
20
b Sample(B), by <- Sample(Baw)
21
x, Q <- TrainerrooT. Q, ba)
3.4 Representation Learning with Offline RI.
When training image-to-action policies, end-to-end RI. allows gradients from the control objective to optimize the encoder. This results in a task-specific encoder that produces features that are most relevant to the agent's 's task, rather than general features (e.g.. features necessary for classification or video prediction). Unfortunately, training directly on full images is very computationally expensive and unacceptably reduces the UTD ratio. Ideally, we would prefer to pre-train some encoder to produce task-relevant features offline, and then freeze the encoder during online training.
We address this by training the encoder with offline RI. on an existing large-scale dataset with a similar objective in a different setting. In particular, we use RECON [51], large-scale navigation dataset collected by manually driving a Clearpath Jackal UGV outdoors at low speeds. This dataset contains navigation trajectorics from many environments and an entirely different robot, but impor- tantly does not include aggressive high-speed driving. Thus, the role of pre-training is not teach the robot how to drive quickly, but only to extract a relevant representation simplify the online learning problem. The high-speed driving behaviors necessary to solve the desired task must be learned through practice in the real world, building on this pre-trained foundation.
We apply goal-conditioned offline RL by selecting a 1:1 mixture of random goals and goals from the robot's future trajectory this dataset, and use Implicit Q-Learning [52] to train a critic network (illustrated in Fig. 1, purple). We then take the learned encoder, which now encodes features relevant to the navigation task, and freeze it for training the policy and critic (orange) as illustrated in Sec. 3.3.
4 System Design for Online Learning
We instantiate FastRLAP 1/10"h autonomous [53, 54]. Our system is based on Traxxas Slash 4x4 modified to facilitate online learning. See the appendix for full parts list.

---

200
Indoor-A
Outdoor-D
10
Indoor-B
200
230
Outdoor-B
202
342
Indoor-C
Sim-F
3.0
32.0
13.3
18.2
10
20
25
Collisions
Figure 4: FastRLAP achieves high-speed driving diverse environments via autonomous practicing. FastR- LAP improves lap times (best-so-far shown black) given a slow demo lap (green) and achieves near-expert lap times (red) in under 40 minutes.
Since our high-speed system operates directly visual observations, we use forward- facing fisheye camera to obtain low-latency stream of 128x 128 RGB images. The policy also depends on IMU data and motor speed as well as relative checkpoint position from state estimator.
Sensing:
Indoor state estimation: Indoors, we use a RealSense T265 tracking camera, mounted facing the ceiling, to estimate the robot's pose and velocity.
Outdoor state estimation: We mount GPS receiver onboard the robot. To estimate absolute heading 0, we use an extended Kalman filter to fuse wheel odometry Yar, absolute GPS velocity and angular velocity w with dynamics 0,41
0g + wAt and measurement model ū = (Vw cos 0, Uw sin 0).
Compute: We
NVIDIA Jetson Orin NX for onboard compute. We process visual obser- vations onboard using pre-trained encoder (Sec. 3.4), and offload training to workstation with GTX 1080ti GPU. We implement our algorithm in JAX [55] and compile several training steps into single function, allowing ~800 actor-critic updates per second (vs. 80 without any optimizations).
Actuation: Standard RC motors exhibit "cogging"", stuttering behavior depending heavily the (unobserved) rotor position. We instead use sensored motor provide closed-loop sequencing.
Action space: The action space consists steering angle and a target speed, which is limited to 4.5m/s across most our experiments to avoid damaging the robot. This limitation addressed in Sec. 5.2 safely learn policy driving at higher speeds. To ensure smooth actions, (i) we use shifted tanh linearized around Oprev constrain the action to (Aprev 5, aprev + 8). and (ii) append the previous action to the observation. See the appendix for details.
Detecting blocked states: The state machine triggers pseudo-reset and delivers negative reward when the robot collides, detected by high lateral acceleration, or is not moving for seconds.
Faster Lap Times with FastRLAP
In this section we present an experimental evaluation of FastRLAP in a variety of real-world and simulated environments. We consider several metrics analyze the peak performance, as well as cumulative metries during practice. The time-to-first-lap (T2F) represents the time taken complete the first full collision-fres free lap, starting from scratch. We track the best lap time achieved during training well as the median time of last five laps completed capture the converged behavior.
5

---

(a] Sim-G
b Outdoor-D
(d] Sim-F
(c) Indoor-B
Figure 5: Emergent behaviors with FastRLAP. Maximizing speed with RI. results in "racing line", hraking as late as possible to maintain speed in and out tight comnrr (a) and drifting on slick surfaces (c). Outdoors, tall grass slows the robot's motion, promoting driving on paths (b). In Sim-F, the robot infers that the bridge is faster than driving through mud via visual correlation (d).
T2F
Lap Times
Collisions
Eav.
(min)
Best
Median
Demo Expert"
Median
Indoor-A
4.07
32.7
39.0
54
25
Indoor-B
12.41
44.2
65.7
70
43
Indoor-C
7.27
11.7
17
Outdoor-D
11.04
22.7
18
Outdoor-E
19.29
62.1
94.0
160
40
Sim-F Sim-G
8.11
104.1
107.0
286
112
41.54
18.0
18.1
36
19
Outdoor-D
21.5
13.1
23.4
43
18
(Schedule)
Figure 6: Sample trajectories of FastRLAP practic- ing in Indoor-C. FastRLAP recovers from collision (green) and learns collision-fis -feee navigation (orange). Maximizing speed, FastLLAP discovers a smooth racing line (purple). The 3D scan is shown only for illustration.
Table I: Summary of experiments: FastRLAP rapidly learns fast driving policies in environments of varying difficulty levels, improving over the demo lap by over 40% and achieving lap times within 5% of the expert, using only egocentric observations.
Additionally, we list the median collisions in the last five laps to capture safety. To contextualize our results, we provide timing for laps driven each environment by human drivers watching the robot from a third- seird- view ("Human Expert"), as well as the duration of the "slow demo" lap.
We used the same hyperparameters (network architecture, learning rate, etc.) for all experiments, both in the real-world and simulation. See the appendix for a full list of hyperparameter values, detailed laptime plots for all baselines, and additional qualitative analysis.
5.1 Real-World Deployment
We deploy FastRLAP in several diverse environments to demonstrate autonomous practicing. Be- fore training, we manually drive the robot around the course for a slow lap to define the rough layout of the track. This lap used in two ways: (i) to generate sequence of sparse checkpoints (s for the practicing FSM described in Sec. 3.2, and (ii) provide low-speed demonstration for off-policy actor-critic updates as described in Sec. 3.3.
We test in five real- weal-world environments: three indoor and two outdoor, labeled A-E, shown in Fig. 4. We also test in two simulated environments, Sim-F and Sim-G. Environments include challenging features such as large scale (Indoor-C. Outdoor-E, Sim-F), tight or cluttered navigation (Indoor-B, Outdoor-E, Sim-G), and highly terrain-dependent speed that must be inferred by correlating visual observations to proprioceptive speed measurements during training (Outdoor-D, Outdoor-E, Sim- F). All environments are described in detail in the appendix.
Table and Fig. summarize the performance of our system in these environments. FastRLAP is able to consistently improve over the low-speed demonstration lap in a handful of laps, and nearly match human performance in Indoor-B and Outdoor-D in 30 minutes of real-world practice, with- out any human interventions. As training progresses, the achieved lap times continue to decrease, with the robot's 's path becoming smoother secondary effect of optimizing speed Fig. 6.

---

Emergent behaviors: Maximizing the reward for reaching checkpoints quickly leads to several emergent behaviors. The system learns a racing line, optimizing speed through corners and chicanes (a). In Fig. 5(a), the robot maintains speed through the apex of a tight corner, braking sharply to turn and accelerating out to minimize driving time. On a low-friction surface (c), the poliey over- steers slightly, achieving fast rotation without braking. Outdoors, the learned policy prefers smooth, high-traction areas and around concrete paths (b), avoiding tall grass that hinders motion.
5.2 Even Faster Laps: Scheduling Speed Limits
Lap time (seconds)
80 00 40
Expert FastRLAP, fixed limit
The action space described in Sec. 4 allows limits to be adjusted arbitrarily during training. Leverag- ing this property, we push the limits FastRLAP in Outdoor-D, increasing the action bounds linearly over time from an initial maximum speed of 2.5m/s
30
20
20
30
40
50
60
to final speed of 6.75m/s. By increasing the max- imum commanded speed smoothly from slow ini- tial limit, FastRLAP first learns basic behavior in the low-speed setting that then transfers to high-speed driving, without causing crashes at high speeds. We see that in the Outdoor-D setting FastRLAP is able to learn a much more aggressive policy than with the original limits, both quantitatively (Fig. 7) and qualitatively (see the videos on our website).
Training time (minutes)
Figure 7: Lap time progression with scheduled increases in the speed limit. Note log scale.
5.3 Comparative Analysis
We compare the performance of FastRLAP against several baselines and ablations in Indoor-C and Sim-G to demonstrate the importance of each of the components of our method: pre-trained visual representations, online RL starting from a slow demo lap, and autonomous recovery behaviors to handle the reset-free environment. Specifically, we consider the following variations:
No Demo Lap: Ablate the demonstration lap and only online data.
No Pre- Tre-Training: Ablate pre-training and use DrQ [56] to learn the encoder from scratch.
No Pseudo-Resets: Ablate the scripted pseudo-resets, requiring the robot to learn recovery behavior.
ImageNet Pre-Training Ablate task-specifie pre-training and instead train the encoder for image classification on ImageNet [57, 58) (with the same encoder structure).
DINOv2 Pre-Training: Uses DINOv2 VIT-S [59], self-supervised vision transformer, as the en- coder in place of RL pre-training. The much larger model introduces roughly 40ms of actor latency.
Offline RL: Ablate online learning and use a policy trained purely offline with 15 minutes of expert data from the replay buffer of a successful run of FastRLAP using IQL [52].
State-Based: Replaces visual observations with privileged state (absolute x; y, from VIO).
In both Indoor-C and Sim-G, FastRLAP outperforms the ablation with no demo lap in both time- to-first lap (T2F) and best lap time while causing fewer collisions (Tab. 2). The demo lap helps the robot make progress early in training, enabling broad state coverage and better final performance. Removing pseudo-resets causes the robot to become stuck, causing similarly poor performance.
While initializing FastRLAP with general-purpose pre-trained visual encoder (DINOv2 and Im- ageNet) gives reasonable performance in simulation, its performance is comparatively poor in the real-world Indoor-C. This suggests that while general-purpose visual features are sufficient for low speeds, high-speed navigation requires task task-spec features. Training the encoder online achieves good asymptotic performance, but takes a long time complete its first collision- free lap and im- proves relatively slowly due reduced UTD ratio (Fig. 8). Our approach also outperforms the variant with access privileged state information, suggesting that the pre-traue pre-trained features generalize better than simple localization estimate.
7

---

Indoor-C
Sim-G
Name
T2F
Best
Median
Collisions
T2F
Best
Median
Collisions
State Based
11,07
12.7
18.8
4
9.5
21.111.7
26.2
No Demo Lap
14.64
16.0
62.6
12
9.8 2.2
20.5 = 0.9
22.2
Oflline
No Pre Training
10.34
12.7
20.0
1
10.3 1: 2.2
19.3
18.4
ImageNet Encoder
19.7
29.7
21.0
22.1
DINOv2 Encoder [59]
16.09
17.0
34,8
4
8.6 + 1.7
20.6 = 1.3
25.6
FastRLAP (Ours)
7.27
10.9
13.3
6.9:1:099
19.3::0.1
18.1
Human FPV
11.1
14.4
2
18.6
18.9
Human Oracle?
7.3
8.8
0
Table 2: Comparing to baselines: In real and simulated environments, FastRLAP has faster time to first lap (T2F), best/median lap times. and median collisions, and achieves near-human performance. Offline RL does not complete a collision-fres lap. T2F listed minutes; other times seconds; lower is better for all metries. Simulation results are reported as mean dev over 3 scods.
6 Discussion
We presented system for learning high-speed driving with reinforcement learning from rich observations, practicing autonomously real world. Our approach uses representations from prior data initialize the policy, followed by sample-efficient online RL and checkpoint- based navigation strategy recover autonomously from collisions and continue practicing. Al- though deep RL is often believed to be inefficient and difficult use in the real world, demon- strate that with appropriate pre-training and system design it is possible to learn effective driving strategies in less than 20 minutes of real-world training. This result may seem quite surprising when viewed in contrast to prior work that uses simulated data [30] or hundreds of hours of training [61], and it provides strong validation that deep RL, in conjunction with task-specific pre-training and approximate resets, can indeed be viable tool for learning real-world policies from raw images.
the
Figure 8: Lap times for baselines in Sim-G.
A qualitative investigation of the policies learned by our system also reveals interesting emergent be- havior. Although we bootstrap training with prior data (in other domains and from other robots) and a single slow demonstration lap, the learned policies exhibit behaviors deviating significantly from the dataset including drifting, selecting for high-speed terrain, and maintaining a racing line. Thus, the online RL process not only robustifies existing behavior, as observed in prior work [21], but also acquires new emergent behaviors by building on the foundation established by the prior data. Our ablations establish the importance of task-relevann pre- training, supporting the notion that represen- tations learned from diverse robot navigation data serve as effective foundation for downstream skill learning much as pre-trane veetrrainnng enables efficient fine-tuning for vision and NLP [62, 63].
Limitations and future work: While our system enables highly effective image- based driving, does have several limitations. First, the current implementation requires coarse state estimator to provide vector to the next checkpoint. This could be addressed in future work by specifying future goals in another format, such as images [64]. Second, our system does not explicitly account for safety during the training process: the agent will learn to avoid collisions because they lead to task failure, but high-speed collisions during training could cause damage. Future work could include a conservative or risk-aware formulation to counteract this effect. Nevertheless, we believe that our work represents a step towards RL-based systems that can autonomously learn highly performant navigation skills wide range of domains.
Acknowledgments
This research was partially supported by DARPA RACER, ARL DCIST CRA W911NF-17-2-0181. the National Science Foundation through I1S-2150826. and the Office of Naval Research. The au- thors would like to thank Alejandro Escontrela, Noriaki Hirose, and Philippe Hansen- Hansen-Estruch, for their help with running experiments and providing baseline implementations.

---

References
[1] M. Bojarski et al. End to end learning for self-driving cars, 2016.
[2] M. Bansal, A. Krizhevsky, and A. Ogale. ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst. Roboties: Science and Systems, 2019.
[3] V. Mnih, K. Kavukeuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arxiv, 2013. 2, 4
[4] Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 2020.
[5] Gu, E. Holly, Lillicrap, and Levine. Deep reinforcement learning for robotic manip- ulation with asynchronous off- off-policy updates. In IEEE International Conference on Roboties and Automation (ICRA), 2017.
[6] O. Kroemer, S. Niekum, and G. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. JMLR, 2021. 2
[7] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. Reinforcement learning: State-of-the-Art, 2012.
[8] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020.
[9] S. Ross, G. Gordon, and D. Bagnell. A Reduction of Imitation Learning and Structured Pre- diction to No-Regret Online Learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. 2
[10] D. Yarats, A. Zhang, Kostrikov, B. Amos, Pineau, and R. Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI Conference Artificial Intelligence, 2021. 2
[11] S. Thrun and T. M. Mitchell. Lifelong robot learning. Robotics and Autonomous Systems, 1995.
[12] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In 26th Annual International Conference on Machine Learning, 2009.
[13] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, 2018.
[14] T. Matüisen, A. Oliver, T. Cohen, and J. Schulman. Teacher-student curriculum learning. IEEE Transactions on Neural Networks and Learning Systems, 31(9):3732-3740, 2020. doi: 10.1109/TNNI.S.2011 N.W.NNN.S.201..95.5. 2934999...
[15] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play elf-play. In Inil. Conf. Learning Representations
(ICLR), 2018. 2
[16] A. Nair, A. Gupta, M. Dalal, and S. Levine, Awac: Accelerating online reinforcement learning with offline datasets. arXiv, 2020. 2
[17] A. Villaflor, J. Dolan, and J. Schneider. Fine-tuning offline reinforcement learning with model- based policy optimization, 2021.
[18] T. Xie. N. Jiang, H. Wang, C. Xiong, and Y. Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning, In Neural Information Processing Systems, 2021.

---

[19] S. Lee, Y. Seo, K. Lee. P. Abbeel, and J. Shin. fifine-to--nling reinforcement learning via balanced replay and pessimistic q-ensemble. In Conf. on Robot Learning, 2021. 2
[20] H. R. Walke, H. Yang, A. Yu, A. Kumar, J. Orbik, Singh, and Levine. Don't start from scratch: Leveraging prior data automate robotic reinforcement learning. In OpenReview, 2022. 2.3
[21] A. Kumar, A. Singh, F. Ebert, Yang, C. Finn, and S. Levine. Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful Trials. arXiv, 2022. 8
[22] N. Gürtler, S. Blaes, P. Kolev, F. Widmaier, M. Wuthrich, S. Bauer, B. Schölkopf, and G. Mar- tius. Benchmarking offline reinforcement learning real-robot hardware. In Intl. Conf. Learning Representations (ICLR), 2023. 2
[23] Funke, P. Theodosis, R. Hindiyeh, G. Stanek, K. Kritatakirana, C. Gerdes, D. Langer, M. Heranndez, Müller-Bessler and Huhnke. Up to the limits: Autonomous audi tts. In IEEE Intelligent Vehicles Symposium, 2012. doi:10. 1A..1199/VVS200 (09//VS.2012.6222222.
[24] N. Keivan and G. Sibley. Realtime simulation in-onn--ii--ie- in-the-loop control for agile ground vehicles. In Towards Autonomous Robotic Systems, 2013.
[25] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou. Aggressive driving with model predictive path integral control. IEEE International Conference Roboties and Automation (ICRA), pages 1433-1440, 2016
[26] U. Rosolia and F. Borrelli, Learning How to Autonomously Race Car: A Predictive Control Approach. IEEE Trans. on Control Systems Technology. 2020. 2
[27] P. Drews, G. Williams, B. Goldfain, E. A. Theodorou, and J. M. Rehg. Aggressive deep driving: Combining convolutional neural networks and model predictive control. In Conf. on Robot Learning, 2017.
[28] Drews, Williams, B. Goldfain, Theodorou, and J. M. Rehg. Vision-based high- speed driving with deep dynamic observer. IEEE Roboties and Automation Letters, 2019. 10:10..109/LRA.20111 20019.899449.9
[29] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. A. Theodorou, and B. Boots. Imitation learning for agile autonomous driving. The International Journal of Robotics Research, 2020.
[30] A. Loquercio, E. Kaufmann, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza. Deep drone racing: From simulation to reality with domain randomization. IEEE Transactions on Robotics, 2020. 3,
[31] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza. Learning high-speed flight in the wild. Science Robotics, 2021.
[32] F. Fuchs, Y. Song, E. Kaufmann, D. Scaramuzza, and P. Dürr Super-human human performance in gran turismo sport using deep reinforcement learning. IEEE Robotics and Automation Letters, 6(3):42574424, 2021. doi: 10. 10w.1199/RRAA.. 100.1
[33] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating objects in the real world. ArXiv, abs/2212.00922. 2022. 3
[34] M. Chang, A. Gupta, and S. Gupta. Semantic visual navigation by watching youtube videos. Advances in Neural Information Processing Systems, 2020. 3
[35] A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. Allen, V. Lam, A. Bewley, and A. Shah. Learning to drive in day. CoRR, 2018. 3
10

---

[36] G. Kahn, P. Abbeel, and S. Levine. Land: Learning to navigate from disengagements. IEEE Robotics and Automation Letters, 2021. 10. 109/LRAA.0221330644004
[37] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: General Navigation Model to Drive Any Robot. In arXiV, 2022. 3
[38] W. Han, S. Levine, and Abbeel. Learning compound multi-step controllers under unknown dynamics. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2015. 3
[39] C. Richter and N. Roy. Safe visual navigation via deep learning and novelty detection, In Robotics: Science and Systems, 2017.
[40] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine. Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning. In Intl. Conf. on Learning Representations (ICLR), 2018.
[41] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The Ingredients of Real World Robotic Reinforcement Learning. In Intl. Conf. on Learming Repre- sentations (ICLR), 2020. 3
[42] K. Lu, Grover, Abbeel, and Mordatch. Reset-free lifelong learning with skill-space planning. In Intl. Conf. Learning Representations (ICLR), 2021.
[43] A. Sharma, K. Xu, N. Sardana, A. Gupta, K. Hausman, Levine, and C. Finn. Autonomous reinforcement learning: Formalism and benchmarking. In Intl. Conf. on Learning Representa- tions (ICLR), 2022.
[44] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. In IEEE International Conference on Robotics and Automation (ICRA), 3021. 3
[45] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort. In Conference on Robot Learning, 2020.
[46] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine. Fully autonomous real-world reinforcement learning with applications to mobile manipulation. In Conf. on Robot Learning, 2022. 3
[47] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference Machine Learning, 2019. 4
[48] D. M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample- efficient reinforcement learning by breaking the replay ratio barrier. Deep Reinforcement Learning Workshop NeurlPS 2022.
[49] P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data, 2023.
[50] X. Chen, C. Wang, Z. Zhou, and K. Ross. Randomized Ensembled Double Q-Learning: Learn- ing Fast Without Model, Mar. 2021. arXiv:2101.05982 [cs]. 4
[51] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine. Rapid exploration for open- world navi- gation with latent goal models. 2021.
[52] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arxiv, 2021. 4.
[53] MIT RACECAR, 2014, URL https://racecaat mit. edu. 4

---

[54] M. O'Kelly, H. Zheng, D. Karthik, and R. Mangharam. FITENTH: An Open-source Eval- uation Environment for Continuous Control and Reinforcement Learning. In NeurlPS 2019 Competition and Demonstration Track, 2020. 4
[55] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milnn and Q. Zhang. JAX: composable transfor- mations of Python+NumPy programs, 2018. URL http: /tte://githbbb.t. com/ haal//oool/aaa 5
[56] I. Kostrikov, D. Yarats, and R. Fergus. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixcls, Mar. 2021. 7
[57] Deng, W. Dong, Socher, L.-J. Li, K. Li, and L.. Fei-Fei. Imagenet: large-scale hier- archical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[58] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Leurning, 2022.
7
[59] M. Oquab et al. Dinov2: Learning robust visual features without supervision, 2023. 7, 8
[60] D. Shah, A. Bhorkar, H. Leen, Kostrikov, N. Rhinehart, and S. Levine. Offline reinforcement learning for visual navigation. In Conf. Robot Learning. 2022. 8
[61] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD- PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. In Intl. Conf. on Learning Representations (ICLR), 2020.
[62] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018.
[63] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arxiv, 2020.
[64] D. Shah and Levine. ViKiNG: Vision- Based Kilometer- Scale Navigation with Geographic Hints. In Robotics: Science and Systems XVIII, 2022.
12


Reviews:
[{"review_id": "uxOatB4jTc", "reviewer": "Reviewer_q79b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the question of training a vision-based policy for racing by blending offline and online data.  The offline pre-training phase is used primarily in learning a vision representation, while the online fine tuning phase allows training and adapting a policy.   Overall, the contribution of the paper lies in addressing some practical problems of sample efficiency in offline RL applied to the racing domain in various realistic environments.  The procedure assumes access to, and observability of, a series of checkpoints along the course, a set of recovery policies, and a set of offline data: one with a diverse set of views along the course (not necessarily taken from the target robot), and another with the target robot taken at slow speed.", "summary_of_recommendation": "Overall, the paper is well written, and the approach itself is sound.  The paper addresses a long-standing issue in RL in vision-based planning, and narrows the gap in applying offline RL to vehicle planning and control in real-world settings.  While the factorization of the problem into pre-training and online fine-tuning is not a new idea, the evaluation in a diverse set of experiments makes for a strong contribution.  I lean towards acceptance.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Could the authors explain the motivation for the data choices: in particular why the choice of training the actor from scratch and freezing the encoder during the online phase make sense in this setting?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is clear in stating its contribution, and is generally attacking a challenging problem.  Because the paper is largely concerned with the practical aspects of RL in racing environments, the novelty is largely in the implementation details.  However, the extensive evaluation conducted in indoor, outdoor and simulated environments greatly strengthen its contributions.  Nonetheless, there are flaws in the paper that should be addressed.\n\nThe authors could clarify what properties the offline dataset is needed to freeze the encoder for online use.  It seems that the dataset would need to have complete coverage of the entire environment, and the online-trained policy could catastrophically suffer if it encountered a new state not included in the offline data.  What are the consequences of updating or fine-tuning the encoder online?  The authors should explain such design rationales, and shoud consider including alternative amounts of data as ablations – in the extreme, no small-scale dataset, and different training splits between online and low-speed offline data.\n\nI wonder why, in the offline step, only the encoder is carried forward into the online step, and why it is useful to collect large-scale offline data using a different robot.  This seems overly restrictive.  In offline RL, it seems natural that the actor policy could be warm-started using the offline one, given that they are both learned from the same reward in the same environment. Relatedly, if there are two robots, does the encoder suffer from any camera pose or other perception differences when going from one robot to the next?\n\nThe “state-based” ablation seems a bit of an unfair comparison.  The authors seem to compare against a policy trained on instantaneous pose observations.  Other approaches that use proprioceptive observations assume awareness of a local map (obstacles, road bounds, a linear fit of the checkpoints, etc), and including velocity as an observed state would seem more sensible and straightforward.  The authors should consider updating this ablation to include such information, or discuss why this was not used.\n\nThe nomenclature used in Algorithm 1 are not properly defined.  For instance $\\mathcal{B}$, $b_d$, $s_{slow}$, $s_{prev}$, etc. are not defined at all.  In Table 2, the bolding of the 18.0 for FastRLAP under “Sim-G”, “Best” seems incorrect, as No Pretraining seems to outperform it.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "WzGnM1WdHX", "reviewer": "Reviewer_1JBu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors introduce FastRLAP, a method for making online autonomous reinforcement learning (with images) more performant by pre-training on low-quality offline data that is task-relevant, but not in the exact domain / with the desired target behavior. They specifically apply this to task of fast, aggressive racetrack driving, and show experimental results both in simulation and in indoor and outdoor real world environments. ", "summary_of_recommendation": "Overall, I really liked this paper - the experiments and ablations were very thorough, the results impressive, and the method well-motivated and clearly described. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "\n- How familiar were the human experts with the task? Were they \"experts\" at the task, or just random human drivers? Given the specialized nature of the task, I was actually surprised that they outperformed the current method -- what were some qualitative differences between FastRLAP and human experts? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I found the paper very interesting, an exciting application of autonomous RL, and the experiments quite thorough. \n\nMain Comments\n- Especially since the same architecture and hyperparameters were used throughout, it would be valuable to have that information in the main paper. \n- The discussion of emergent aggressive driving behavior could be more rigorous. Perhaps some quantification of how often such behavior appears in the offline data (is it really 0?), or a more precise description of how to measure such behavior, would be useful. \n- It would have been great if there could be additional insights into the enviornment similarity required between the pre-training data and the test domain. While the ImageNet encoding baseline is interesting, I think ImageNet is too far away visually, and this baseline muddles both change in environment and change in training task (image classification vs. IQL). It would be great to understand why these baseliens do well in Sim but not Real more as well. \n- Was the Offline RL baseline properly trained? What if more data was used? More discussion on the failure modes of this baseline would be really useful, as it is the baseline that directly shows what happens if we relax the assumption that we have no expert demonstrations. \n-  I think the paper could benefit from a more prescriptive limitations section, including describing (i) why specifying goals as images was not already done in the work, (ii) describing precisely how safety considerations could be integrated into the current method, and (iii) further description of what other domains the author thinks their method could work on (e.g. what are other examples where there is sufficient offline data to use for pre-training, but the desired behavior is unlikely to be included in this set?).   \n\nVery Minor\n- In 5.3, it seems like Real-C and Indoor-C are used to refer to the same thing. \n- Line 48: \"Without *us* explicitly demonstrating these behaviors\"? On first read, it seems like it states the robot is not demonstrating these behaviors. ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "TnaeaUc1KTr", "reviewer": "Reviewer_ieSQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors contributions can be summarized as follows: \n\n- Offline pretraining of an image encoder on a task-relevant, autonomous driving dataset\n\n- Finite state machine that provides high level checkpoints to facilitate online learning and a collision-recovery subroutine that facilitates a reset-free learning paradigm \n\n- “low quality” initial demonstrations that stabilize and accelerate the learning process combined with online data to learn emergent behaviors and driving policies that close the gap with expert-driver results. \n", "summary_of_recommendation": "The authors propose a system in which classical components such as FSM are combined with a modern offline-RL algorithm in the \"offline-pretraining online fine-tuning\" paradigm. The system shows generalization capability to various indoor and outdoor environments, and the ability to learn autonomously in a reset-free manner, both of which are impressive. However, the novelty aspect of the contributions seems somewhat limited, especially when placed with in the broader context of the \"offline pretraining + online fine-tuning\" or IL + RL literature. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "n/a", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and easy to follow.\n\n# Strengths\n\n- The use of an FSM to provide intermediate checkpoints and a collision-recovery subroutine to facilitate “in the wild” reinforcement learning in a sample-efficient, reset-free manner is impressive, novel as far as I’m aware, and brings the field closer to making RL more reliable in real-world settings. \n\n- The ablations are comprehensive and have convinced me that the authors’ design decisions wrt encoder pretraining, the use of “low quality” demonstrations to facilitate learning, and the FSM are important aspects of the algorithm. \n\n- The learned policies demonstrate real-world driving capability in indoor and outdoor environments, under different terrains, lighting and motion blur conditions, and surfaces. \n\n# Weaknesses \n\n- The algorithm learning emergent, aggressive behaviors not present in the initial dataset, while impressive, is not entirely as novel as the authors make it seem, and should be situated within the broader context of prior methods. Broadly speaking, the training paradigm laid out by the authors can be described as \"offline pre-training plus online fine-tuning\" or \"imitation plus reinforcement learning\". The former has been studied in methods such as [1, 2], and the later has been documented and well-studied within the context of autonomous driving in papers such as [3]. The question that then arises is, how does this approach compare to these prior methods and, how is the current approach situated in the broader space of offline pretraining with online fine-tuning? The authors could have discussed this in greater detail in the paper. \n\n- (Section 6) *This result may seem quite surprising when viewed in contrast to prior work that uses simulated data [30], or hundreds of hours of training [61],   and we believe it provides strong validation that deep RL can indeed be a viable tool for learning real-world policies even from raw images*. The authors make misleading claims that RL itself can be used to train real world policies in under 20 minutes, whereas in reality the FSM, prior demonstrations, and pre-training of an encoder on a task-specific dataset are all employed to address the well-known weaknesses of RL without which the algorithm would work poorly, if at all, as demonstrated by the ablations. Specifically, the FSM and demonstrations provide sufficient exploration, help the RL agent recover from terminal states, and provide an initial manifold of driving policies for the RL component that helps stabliize training. To be clear, I am not against utilizing classical or hand-crafted systems to aid RL in addressing its weak points, but the authors should be honest that their results stem from a combination of classical and learned components, rather than claiming RL itself yielded these results. \n\n[1] Nakamoto, Mitsuhiko, et al. \"Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.\" arXiv preprint arXiv:2303.05479 (2023).\n\n[2] Walke, Homer Rich, et al. \"Don’t start from scratch: Leveraging prior data to automate robotic reinforcement learning.\" Conference on Robot Learning. PMLR, 2023.\n\n[3] Lu, Yiren, et al. \"Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios.\" arXiv preprint arXiv:2212.11419 (2022).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "LBA-iCbaC_g", "reviewer": "Reviewer_gJxi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper describes a method for learning fast navigation through a combination of offline data (consisting of previously-released data collected by another robot in other train environments, as well as a slow lap through the test environment) and online learning. The paper evaluates this method by demonstrating its ability to learn these navigation tasks and by ablating each component of the method. \n", "summary_of_recommendation": "I am unfortunately not an expert in the navigation subfield, so I will weight my review lesser than other reviewers with more direct expertise. \n\nI found this paper to be well written and reasonable in its approach. I have some concerns about reproducibility and some requests for edits (e.g., a new related work section). On balance, I think the work is well suited to the CoRL audience and is sufficiently polished to recommend its acceptance.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "How was the reward function designed? Why was the reward function structured this way? \n\nLine 208 refers to 'maximizing the reward for reaching checkpoints quickly' but this is not part of the reward fn described earlier? (It is implicit in the speed-made-good term, but this is weighted against two other terms. It is also implicit, to some extent, in the discount factor.)\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well written and the accompanying video is impressive. The paper is also practical: I appreciate the switching between a recovery policy and a learned policy. I also appreciate the inclusion of ablation studies. \n\nI am not an expert in the precise area of focus of this paper. My first thought on reading this paper was that it seemed like there might be little novelty to it (especially on reading the last sentence of the introduction, \"FastRLAP is the first instantiation of a vision-based mobile robotic system that uses model-free RL to autonomously practice high-speed driving maneuvers and improve online in the real world\"; this seemed unbelievable). However, I did a small literature search, and I didn't find anything exactly comparable. Two papers which seem useful: \n- 1, Xiao, Xuesu, et al. \"Autonomous Ground Navigation in Highly Constrained Spaces: Lessons Learned From the Benchmark Autonomous Robot Navigation Challenge at ICRA 2022 [Competitions].\" IEEE Robotics & Automation Magazine 29.4 (2022): 148-156. \n- 2, This paper was put on ArXiv after the author’s CoRL submission, so I am only listing it as a potential resource and not asking for a comparison: Shah, Dhruv, et al. \"ViNT: A Foundation Model for Visual Navigation.\" arXiv preprint arXiv:2306.14846 (2023). \nAnother related work is GT-Sophy, though of course that takes place entirely in simulation. \n\nThe paper's results do not list any kind of error bounds - i.e., it would be nice to have the system evaluated ~10 times and provide the variance of the completion times to help us assess whether the FastRLAP is a significant improvement. For example, the No Pre-Training numbers in Table 2 look relatively close. \n\nI find the related work section of the paper to be weak; it does not help me position this work in the larger research context. If this paper is to be accepted, I would ask the authors to rewrite this section with this goal of positioning the paper in the larger research context in mind. The related work should, in my opinion, help an expert in an adjacent research area assess where to learn more about this specific research problem. Broad citations to supervised learning, representation learning, continual learning, etc do not help with this. For this paper, I would like to know: is this reset mechanism and policy switching approach new (how do citations 38-43 relate, and what exactly is relevant about them)? Are there different settings where others have successfully trained a policy first with offline data and then with a small number of imperfect online samples? What is unique about this racing problem? Etc. \n\nThis paper mentioned that it uses offline data to first learn a navigation-ready representation, but exactly what this data is composed of is not stated until Section 3 (and not precisely explained until Section 3.4). The precise composition of this data should be discussed in the introduction. \n\nAs written, the codebase is uncommented and very hard to reuse, so work would need to be done to improve this resource. I am concerned about reproducibility of this paper. This is always hard with robotics settings, but a well-commented codebase would significantly help. \n\nThe graphs in figure 4 need more explanation and axis labels. \n\nMinor comments: \n3.1 should define each term of the MDP and explain how they are modeled for this problem. Also missing the mention of the discount factor.\n\nFigure 4's horizontal order should be image-A graph-A | image-B graph-B (instead of current order image-A graph-A | graph-B image-B).\n\nLine 257 typo \"before before\"\n\nTable 2 has an incorrectly highlighted value (FastRLAP, Best for Sim-G). \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Summary of Paper
The paper proposes a method to speed up RL by using "low quality" initial data. The approach is evaluated in a racing task both in simulation and the real world.

## Summary of Reviews
The reviewers found the paper interesting and well written. The experiments are extensive and clearly show the benefits of the method. Some inaccurate claims were pointed out, some clarifications and improvements int he writing are needed, the embedding in the state-of-the-art should be improved.

## Influence of Rebuttal
The rebuttal addressed all major concerns of the reviewers. Reviewer ieSQ still has some concerns about the positioning of the paper.

## ToDos for Final Version
Please implement all promised improvements (limitations section, adjusted claims, error-bars on simulation results, etc.) and include the clarifications.
Decision: Accept (Poster)