Learning Visual Parkour from Generated Images
Authors: Alan Yu
Ge Yang
Ran Choi
Yajvan Ravan
John Leonard
Phillip Isola
Venue: CoRL 2024 (2024)
Keywords: Generative AI
Simulation
Legged Locomotion
Sensory Motor-learning
TL;DR: Train a State-of-the-art quadruped parkour policy on generated data

Abstract: Fast and accurate physics simulation is an essential component of robot learning, where robots can explore failure scenarios that are difficult to produce in the real world and learn from unlimited on-policy data. Yet, it remains challenging to incorporate RGB-color perception into the sim-to-real pipeline that matches the real world in its richness and realism. In this work, we train a robot dog in simulation for visual parkour. We propose a way to use generative models to synthesize diverse and physically accurate image sequences of the scene from the robot's ego-centric perspective. We present demonstrations of zero-shot transfer to the RGB-only observations of the real world on a robot equipped with a low-cost, off-the-shelf color camera.

Learning Visual Parkour from Generated Images
Alan Yu², Ge Yang's, Ran Choi, Yajvan Ravan, John Leonard, Phillip Isola MIT CSAIL, 'Inttitute for Al and Fundamental Interactions
*Equal contribution.
Learning On-Policy in LucidSim
Zero-shot Transfer to Real
Diverse generated images
Robust RGB-oal policy in the wild
Thousands of LLM- generated prompts
LucidSian
-10 million images
nelien
Visual Policy
Expert Labeler
Figure 1: Learning a real-world policy from generated images. Left: we source structured image prompts from LLM, which are combined with a depth map and semantic masks produce diverse and on-policy visual data. Right: the policy is sufficiently robust to transfer to a variety of challenging terrains in the real world, despite never having seen real data during training.
Abstract: Fast and accurate physics simulation is an essential component of robot learning, where robots can explore failure scenarios that are difficult to produce in the real world and learn from unlimited on-policy data. Yet, remains challenging to incorporate RGB-color perception into the sim-to-real pipeline that matches the real world in its richness and realism. In this work, we train a robot dog simulation for visual parkour. We propose way to use generative models synthesize diverse and physically accurate image sequences of the scene from the robot's ego-centric perspective. We present demonstrations of zero-shot transfer to the RGB-only observations of the real world on robot equipped with low- cost, off-the-shelf color camera. project website: http://lucidsim.githee
Introduction
The success of robot learning system depends largely on the realism and coverage of its training data. Real-world data, though inherently realistic, limited in its coverage over the diverse scenarios robot might encounter upon deployment. Real training data typically only includes small number of environments and is not reliable source for failures that cause injury harm. As robot improves throughout training, the data it needs to improve its skills further also evolves. Getting the right data is critical for improving the robot's performance, but in current practice this is manual process that needs to repeated from scratch for new scenes and new tasks.
The alternative is to train in simulations, where we can sample a greater diversity of environmental conditions, and our robots safely explore failure cases and learn directly from their own actions. Despite substantial investment into simulated physics and rendering, our best efforts at achieving
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Dreams In Motion
Expert Labeler
Visual Policy
Depth Imuge
1
Simple Terrain from
MuJoCo
Optiol
+
Thousands of prompts from
Semantic
Ganemted Fraane
Evaluation Digital Twin
Figure 2: The LucidSim graphics pipeline. We use the same parameterized terrain geometry as [5]. We use MuJoCo to simulate the physics, and render semantic masks and the depth image that then fed into a ControlNet trained with MiDAS depth maps. The generated image is then combined with the dense optical flow to generate short videos via Dreams In Motion (DIM, see Sec. 2).
realism retain reality gap [1, 3, 4]. This is because rendering a realistic image means making de- tailed and realistic scene content. Trying to hand-craft such contents at scale to obtain the diversity required by our robots for sim-to-real transfer is prohibitively expensive. Without diverse and high quality scene content, robots trained in simulation are too brittle to transfer to the real world. There- fore, how to match the real world in its infinite jest, and integrate color-perception into sim-to-real learning, a key challenge.
The purpose of this work is develop a solution. We turn generative models promising new data source for robot learning and use visual parkour as a testbed, where robot dog equipped with single color camera is tasked to scale tall obstacles at a fast speed. Our ultimate vision is to train robots entirely in generated worlds. At the heart of this is finding ways to exert precise control over the semantic composition and scene appearance to align with the simulated physics-while while maintaining the randomness critical for sim-to-real generalization.
Our method works follows (Figure 2): take popular physics engine, MuJoCo [2], and render the depth image and semantic masks at each frame, that together are used as input to depth- conditioned ControlNet. We then compute the ground-truth dense optical flow from the known scene geometry and changes in the camera poses, and warp the initial generated frame for the following six timesteps to produce a temporally consistent video sequence. On the learning side, we train the visual policy two stages: first, we optimize the policy imitate expert behavior from rollouts collected by privileged teacher. The policy performs poorly after this pre-training step. The post- training step involves collecting on-policy data from the visual policy itself, interleaved with learning on all data aggregated far. Repeating this step three times makes the visual policy significantly more performant. This policy is sufficiently robust to transfer zero-shot to color observations in the real world throughout our test scenes.
Our contributions are threefold: First, we provide a scalable recipe to translate compute into real- world capabilities by producing geometrically and dynamically aligned visual data for robots. Sec- ond, propose outo-prompting technique significantly increase data diversity, which in prac- tice also enables tailored data synthesis. Finally, we provide the first demonstration of robust, visual parkour policy trained entirely in simulation that has seen zero real-world data.
LucidSim: Generating Diverse Visual Data with Physies Guidance
We consider a sim-to-real setup, where the robot trained in simulation and transferred to the real world without further tuning. We have partial knowledge about the environments we intend to deploy robot in perhaps rough description a reference image. Since this information is incomplete, rely on prior knowledge within generative models to fill the gap. We refer to this

---

Mara
Sex
Grazry Coutyyud Biu Carpet Coblladone
LLM Koars
Avrlasiaa kfnca-kappia axzid-
Exd Exdipon noliectve sirunaace procape maae
Figure We solicit batches of 20 30 image prompts in JSON format from chatGPT. Each task requires 10 generated prompts.
Figure CLIP embeddings of images gen- erated by three sets of meta-prommts. Images from the same prompt (o) are diverse.
guided process as Prior-Assisted Domain Generation (PADG), that begins with an auto- promoooroopingg technique critical for synthesizing diverse domains.
Sourcing diverse, structured prompts from LLM. We observed early on that repeatedly sam- pling from the same prompt (Figure 4) tends to reproduce similar-looking images. To obtain diverse images, we first generate batches of structured image prompts by prompting chatGPT with a "meta" prompt that contains title block, details of the request, and ends with question asking for strue- tured outputs in JSON (3). Our request includes particular weather, time of day, lighting conditions, and cultural sites. It is impractical to edit the generated image prompts by hand. Instead, we tweak the meta prompts by generating a small number of images, and iterate until they consistently pro- duce reasonable images. Examples of diverse samples from the same meta-prompt, but different image prompts, are shown on the bottom row of Figure
Generating images with geometry and physics guidance. We augment vanilla text-to-image model [6] with additional semantic and geometric control that aligns it with simulated physics. First, we replace the text prompt for the image with pairs of prompts and semantic masks that each correspond to a type of asset (Figure 2). In the stairs scene, for instance, we spec- ify the material and texture the steps inside a coarse silhouette via text. To make the im- ages geometrically consistent, we take an off-the-shelf ControlNet [7] trained on monocular depth estimates from MIDAS [8]. The conditioning depth image is com- puted by inverting the bufffe and normalizing it within each image. It is important adjust the control strength to avoid losing image details (discussed in depth in Section 4.6). Our scene geometry is simple terrain sourced from prior work [5, 9, 10] that optionally includes side walls. We
Generation Speed
3.0
Frame Rate (ips).
No DIM
2.0
DIM
avoid randomizing the terrain geometry to focus our analysis on visual diversity.
1.0
To produce short videos, developed Dreams In Motion (DIM), which warps generated image into subsequent frames using the ground- truth optical flow computed from the scene geometry and the change in
Warp Interval
Figure 6: A 6.5x speed up in vidco generation.
Chinese Alley
Prompt #2
Prompt #3
Prompt #4
Echos Industry
Tuscan Alley
German Town
South Street
Figure 5: LucidSim image samples from the stairs environment. Top row: images generated from different prompts produced by the same meta prompt; bottom row: different meta prompts.
3

---

the camera perspective between two the frames (see Figure 2). The resulting image stack con- tains timing information critical for parkour. Generation speed is also important. DIM significantly improves the rendering speed (Figure 6) because computing the flow and applying the warping is significantly faster than generating images.
3 Learning Robust Real-world Visual Policy from On-Policy Supervision
Our training procedure has two phases: a pre-training phase that bootstraps the visual policy by imitating privileged expert that has direct access to a height map, trained via RL following the procedure in [5]. We collect rollouts from the expert and its imperfect earlier checkpoints, and query the expert for action labels to supervise the visual policy. The visual policy performs poorly after pre-training, but it makes sufficienlly reasonable decisions for collecting on-policy data in the phase (Figure 7). We follow DAgger [11] and combine the on-policy rollouts with the teacher rollouts from the previous step. We collect action labels from the expert teacher and run seventy epochs of gradient descent with the Adam optimizer [12] under a cosine learning rate schedule. We repeat the DAgger iteration three times. This second stage is responsible for the majority of the final performance, approaching expert-level.
second,
simple transformer policy. We present a simple transformer architecture that reduces the number of moving parts for working with multi-modal inputs (Figure 8). Prior work in quadruped parkour uses composite architecture that first processes the depth image into compact latent vector using ConvNet, followed by recurrent backbone [5]. We
five-layer transformer backbone with the multi-query attention (MQA [13]). The imput camera feed is diced into small patches and processed in parallel by convolution layer. We then stack these tokens together with linear embedding of the proprioceptive observation of the same time step. We repeat this for all timesteps and add a learnable embedding at the token level. We find that for RGB images, it is helpful also to include a batch normalization layer before the convolution. We compute the action output via an additional class token (cls) stacked the end of the input sequence, followed by a ReLU latent layer and linear projection.
Supervised Pre-Training
On-Policy Learning Sim
five layers
at
Parallel Transformer
Expert Labeler
learned pos. emb
t
Export Data
0
Visual Policy
cls.
Visual Policy
prop.
actios
LucidSim
image patches
one timestep at f = 0 Figure 8: We treat proprioceptive ob- servations and image patches as to- kens, plus an additional class (cls) to- ken for the action output.
Figure 7: Pre-training: Expert rollouts are labeled offline using LucidSim's 's graphics pipeline. Post-Training in Sim: Data is collected on-policy with the visual policy, super- vised by action labels from the expert.
This five-layer transformer policy can process seven input frames while maintaining 50 Hz frame rate when running on the Nvidia AGX Orin. This memory span is rather short (140 ms). Common ways to speed up L.M inference do not apply due to the use of a rolling input window. This is a key bottleneck that affects skills that require a longer memory. For instance, the robot needs a memory span closer to 400 ms order to jump over wide gaps.
4 Results
We consider the following tasks: tracking soccer ball (chase-soccer); tracking orange traffic cone (chase-cone), climbing over hurdles (hurdle); and traversing stairs featuring various material

---

Figure 9: Rendered views of subset of environments used for evaluation. Each scene is modeled using 3D Gaussian Splatting. The first-person view from the robot's perspective is highly realistic.
types (stairs). We evaluated the performance of our learned controller in both the real world and on a small set of real-world scenes modeled in simulation using 3D Gaussian Splatting [14], recent graphics technique for building complex and photo-realistic environments transferred from the real- world. Examples these benchmark environments are shown Figure 9.
In chasing tasks, randomly sampled locations for the target objects within the robot's camera frustum. For hurdle and stairs, manually placed orange cones to visually indicate the waypoints. Each task was evaluated in three replica scenes with 50 trials each, randomizing both the starting pose and waypoint location offsets. We report the following metrics in Table and 6: fraction of goals reached (FGR) computed via the ratio guals reached and normalized forward displacement towards the final goal, computed via Taist.
goals total
Amasstd
distance goal
We consider the following baselines: an expert policy that requires privileged terrain data (the ora- cle); depth student policy trained using the identical pipeline; an RGB student policy trained using classical domain randomization frame stacks using DIM. We also provide the simulated performance of the Extreme Parkour [5] depth policy for calibration, which is trained on magnitude more data. Details the distinction between the depth baselines can be found in Section 4.4.
textures, and method, LucidSim, trained with generated
4.1 Learning from Generated Images Out-Performs Domain Randomization
our simulated evaluations, we observed that LucidSim outperformed classical domain random- ization [15] in almost all evaluations, as seen in Tables I and 6. The domain randomization baseline was able to climb stairs quite effectively in simulation, likely due the repetitive gait that in- duced after recognizing the first step. However, it struggled to perform on hurdles, where the timing of the jump was critical. The depth student suffered from subtle but common sim- sim-to-real gaps in the 3D scene. For instance, the oracle policy struggled in one of the stairs environments (Marble) due to the presence railing, which it had never seen before in its training environment. We found that the LucidSim policy was less affected by Similar phenomena also affected the depth student,
Hunfle
Hundle
100%
100%
1007%
100%
Coals Roaclied
10%s
Goals Roasted
50%5
X Diopacement
Disglaceeet
50%
60%-
6055
60%
50%
40%
4055-
40%
Expen
LasidSies
200g
2056
200g
20%
Doman Ranl
092
0%
3000
Figure 10: On-policy supervision significantly boosts performance. Each data point represents a new DAgger step. Increasing the number of DAgger iterations improves performance the simulated benchmark environments. Evaluation includes 50 unrolls on three environment instances for each task. Gray dotted line indicates the performance of the expert teacher.
5

---

which was distracted by features such as chairs, walls, and railings in the benchmark environment. Past work used aggressive clipping to mitigate this type of sim-to-real gap, which we elaborate on in Section 4.4.
Table 1: Success Rate (Fraction of Goals Reached) Real-to- Benchmark Environments.
Chase-Cone
Chasc- Soccer
Hurdie
Stairs
Method
Space
Lawn
Lab
Urban
Lawn
Urban
Lab
Concrete
Previleged Expert
state=terrain
98.6
96.2
97.9
98.6
96.2
97.9
95.8
100.0
99.0
97.0
100.0
73.4
Extreme Parkour [5]
elipped depth depth
84.0
98.5
97.0
93.5
Depth
80.7
80.7
80.7
80.7
84.7
80.0
78.3
56.0
$4.0
93.0
86.0
72.9
Depth
elipped depith
98.7
87.3
98.7
98.7
92.7
98.7
70.7
83.7
84.7
94.0
85.0
85.4
Domain Rand.
color
81.9
50.4
66.7
97.3
76.7
78.0
56.5
52.5
44.0
95.5
81.5
71.7
LucidSim
color
96.7
84.0
98.0
88.7
90.7
94.7
90.7
93.5
96.5
87.0
81.0
83.7
4.2 Transfer Zero-shot to The Real World
Experiment setup: We deployed LucidSim on Unitree Gol equipped with budget RGB webcam and ran inference on the Jetson AGX Orin. Each task was evaluated in multiple scenes, and recorded whether the robot reached the target object (chase) or successfully traversed the obstacle.
We compare LucidSim to Domain Ran- domization (DR) and present the results in Figure 11. In the chasing tasks, we able
Task
Trials
LucidSim
Domain Rand.
Depth
20
100.0%
70.0%
80.0%
observed that Domain Rand. to identify color well (orange cones), but struggled with recognizing the patterns of the soccer ball. On the other hand, Lu- cidSim was not only able to recognize the classic black and white soccer ball, but also generalized to different colored soc- balls due to the rich diversity of the generated data it had seen before. For hurdles and stairs, Domain Rand. failed to consistently recognize the obstacle in front of it, often resulting in head-on collision. while LucidSim was able to consistently anticipate the obstacle and successfully traverse it. We additionally evaluated the clipped depth baseline and show that LucidSim achieves comparable results.
soccer
20
85.0%
35.0%
65.0%
dark hurdle
15
86.7%
26.7%
86.7%
light hurdles
15
73.3%
40.0%
80.0%
stairs
10
100.0%
50.0%
100.0%
Figure 11: We measured the success rate of the Lu- cidSim, Domain Rand., and Depth student in a variety real-world scenarios. Each task was evaluated over multiple environments, diverse in appearance.
4.3 Learning On-Policy Out-Performs Naïvely Scaling Expert Data
We compare learning from on-policy data against naïvely scaling expert data collection in Figure 12. The performance gain from training on additional expert-only data saturated quickly. In both do- mains, on-policy learning through DAgger was necessary for producing sufficiently robust policy. The discrepancy was especially apparent on hurdles, where the student struggled to make any mean- ingful progress through the terrain. We observe the benefit of DAgger with both LucidSim and the domain randomization baseline in Figure 10, with LucidSim reaching higher overall performance.
4.4 Depth-Only Policy Overfits to Training Geometry
Besides Extreme Parkour [5], consider two depth policies, both trained identically to LucidSim, but with different depth inputs. The first (row three in Tables 1 and 6) receives depth up to far clip of 5m and shares the same 120" FoV as the color methods. The second (row four in Tables I and 6, indicated by the clipped depth observation space) receives depth clipped to 2m, as in [5]. We also reduce the FoV and set a near clip of 0.28m to agree with the properties of the Realsense camera. In our simulated evaluation (Tables 1 and 6), we observed that the policy trained with unclipped depth overfits to the minimal and simple geometry of the training scenes, getting thrown off by distractors in the background of the evaluation scenes. The depth policies with limited vision did not suffer as much from the diversity in the test scenes and achieved significantly higher performance.

---

Hurdle
Stairs
DIM Ablation
100%
100%
DIM
DIMI
1.0
Goals Reachel
Goals Reached
0.5
50%
607
0.6
40%
4075
0.4
20%
LacidSim Dagger
0.2
056
0%
1000
0.0
Rollouts
Rollouts
Hurdle Scene
We compare learning policy against naïve scaling of expert data. Increasing the amount of expert data does not improve performance significantly.
Figure 12:
Figure 13: Warped images (blue) increase speed without degrading policy performance.
We interpret this as failure mode for training on policy, the diversity the test scenes confuses the student. LucidSim is less susceptible to this, adds diversity the policy's experience by hallucinating the background. Although limiting vision beneficial here for parkour tasks, there general way know which part of the visual input to censor for which task, and coming up with effective censoring strategies currently requires hand design. In particular, if the task required tracking more distant objects, then the 2m clip would be inappropriate.
limiting the vision of the depth
4.5 Understanding the Speed and Performance of Dreams in Motion (DIM)
Image generation is bottleneck in our pipeline. DIM greatly accelerates each policy unroll, while also providing synamically-consiste
frame stacks by trading off diversity. We study how the stu- dent's performance is affected by generating every frame independently, opposed to warping batches of images with DIM. We consider the Hurdle domain as is the most challenging. As illus- trated by Figure 13, the performance was similar, yet DIM was able to achieve the same results in a fraction of the time (Figure 6).
4.6 Strong Conditioning Decrease Diversity and Image Details
There exists a trade-off between the accuracy of the geometry versus the richness of the details in the generated images. When the conditioning strength is too low, the image deviates from the scene geometry (left, Figure 14). When it too strong, the image loses the diversity and rich details (right side of Figure 14) and becomes highly reduced due cver-constraints.
(0) Depth Conditioning
(b) Depth Conditioning
(c) Depch
Conditioning
Figure 14: Stronger geometric control reduces the level of detail. As we increase the control strength from left to right, the geometry becomes better aligned with the depth map. However, this comes at the cost of reducing the amount of details in the image. Prompts in Appendix A.7.
Related Work
Generative AI for task-creation and world-building. LucidSim is part of growing line of work in robot learning that uses generative AI systems to automatically design parts of the learning setup,

---

including the task specification [16, 17], and the reward function [18]. This is distinct from work that augments real-world data with image in-painting [19, 20], or style transfer [21]. Works that learn aetion-conditioned video generation models [22, 23] are also being considered as potential source of unlimited data [24, 25, 26). In general, however, it is challenging for video models to learn accurate physics [27]. LucidSim's 's hybrid approach aligns with growing trend in frontier video generation models that use shysics-based image manipulation for guidance, followed by generative in-painting [28].
Robot parkour. Recent work in agile locomotion uses deep reinforcement learning and supervised distillation simulated environments to achieve impressive levels of agility in quadrupeds [5, 29, 9] and humanoid robots [30]. These methods all rely on depth images as imput. contrast, our work does not depend depth and uses the color image from low-cost, off-the-shelf webcam instead. We choose visual parkour because a blind teacher can not accomplish this task [31]. To our knowledge, LucidSim is the first reported result of visual parkour using a color camera and the first that is trained entirely in simulation with generated images.
Robot learning from demonstrations. Recent work in robot learning leverages low-cost hardware and expressive new policy classes borrowed from language modeling and image generation to pro- duce increasingly capable visuomotor controllers [32, 33, 34]. More recent work further lowers the barrier-to-selt by removing the robot, retaining just the end effector [35, 36, 37]. The remaining cost is dominated by the need to visit and set up physical scenes and the human effort in tailoring data collection to the evolving robot. In contrast, LucidSim aims to move data collection from the physical world into software.
Real-to-sim and learning from digital twins. New techniques [38, 14] in computer graphics have made it easy to build high-fidelity digital twins in sim. Efforts in drone-racing [39], autonomous- driving [40]. and humanoid soccer [41] take advantage of this to produce robust but highly spe- cialized controllers. In contrast, we employ these techniques for evaluation only, where targeted assessment via a few high-quality digital scans can be highly effective.
6 Conclusion
We present a technique to synthesize unlimited, geometrically, and dynamically correct, multi-frame image stacks for robot learning. We also provide the first empirical demonstration of a visual parkour policy on a quadruped robot trained entirely using generated data. Although preliminary, we con- sider these results a promising proof-of-concept that points towards a more common common-place place usage of generative AI in producing learning data for difficult robotic tasks.
Key Assumptions and Limitations. Enforcing the geometry through conditioning degrades the richness of the generated images (Section 4.6). A way to apply loose control [42] would be helpful. We also made terrain randomization outside the scope of this work and opted to use simple hand- designed geometries inherited from prior work [43, 44, 10]. Ultimately, we believe the 3D assets and the scene layout will also be generated autonomously [43, 44], where AI models amplify human creativity and effort.
Acknowledgments
This work was partially supported by a Packard Fellowship and a Sloan Research Fellowship to P.I., by ONR MURI grant N00014-22-1-2740,, by Amazon.com Services LLC Award #2D-063 10236, and by the Defence Science and Technology Agency, Singapore. This work was also partially sup- ported by ONR grant N00014-19-1-2571 (Neuroautonomy MURI) and by the MIT Lincoln Lab- oratory. G.Y. was partially supported by the National Science Foundation Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) under the Cooperative Agreement PHY-2019786.
8

---

References
[1] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martin-Martin, C. Wang, G. Levine, M. Lingelbach, Sun, M. Anvari, M. Hwang. M. Sharma, A. Aydin, D. Bansal, S. Hunter, K.-Y. Kim, A. Lou, R. Matthews, Villa-Renteria, J. Tang, C. Tang, Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and L. Fei-Fei. Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 80-93. PMLR, 14-18 Dec 2023.
[2] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012. 1oi:10.1 101.1111111O..S... 201.R006.20..6686005
[3] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[4] S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y. Liu, T. kai Chan, Y. Gao, X. Li, T. Mu, N. Xiao, A. Gurha, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint
10.00425, 2024.
[5] X. Cheng, K. Shi. A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv:2309. 14341, 2023.
[6] S. AI. Sdxl turbo, 2024. URL htt.hhtts//ttabiiiitttttt
[7] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[8] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623- 1637, 2020.
[9] D. Hoeller, N. Rudin, D. V. Sako, and M. Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9, 2023. URL https://api Cemanticscholl
[10] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Conference on Computer Vision and Pattern Recognition 2023, 2023. URL https:// openreview.ne net/ forum? wotooummiidyyylieeee. id=JYYWCCnwDS.
[11] S. Ross, G. J. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2010.
[12] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.666 412.,9980. 2014.
[13] N. M. Shazeer. Fast transformer decoding: One write-head is all you need. ArXiv, abs/1911.02150,
2019. URL
2 207880429.
[14] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), July 2023. URL https: //repo- sam. inria. a.ff fr haingraph/3d-qausstee aasaussinn-splatnngg//

---

[15] J. Tobin, R. Fong, A. Ray. J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017.
[16] L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang. Gensim: Generating robotic simulation tasks via large language models. Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[17] A. M. L. W. H. L. W. Pu Hua, Minghuan Liu. Gensim2: Scaling robot data generation with multi- multi-modal and reasoning llms. In Conference Robot Learning. 2024.
[18] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Arxiv-2310. 12931, 2023.
[19] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, Singh, Tan, M. Dee, J. Per- alta, B. lchter, K. Hausman, and F. Xia. Scaling Robot Learning with semantically imagined experience. arXiv.org, 2023.
[20] Z. Chen, Z. Mandi, H. Bharadhwaj, M. Sharma, S. Song, A. Gupta, and V. Kumar. Seman- tically controllable augmentations for generalizable robot learning. ArXiv, abs/2409.00951, 2024. URL https: /api. weeantticccoolll......... cetscholar.ore www.aarorra.eeeeeeeeeeeeeeoeeeee org. wotproooo/oooooo////.. 2723377777
[21] W. Xu, N. Souly, and P. P. Brahma. Reliability of gan generated data to train and validate perception systems for autonomous vehicles. In Proceedings of the ieee/cvf winter conference on applications of computer vision, pages 171-180, 2021.
[22] C. Finn, J. Goodfellow, and S. Levine. Unsupervised learning for physical interac- tion through video prediction. ArXiv, abs/1605.07157, 2016. URL https://api. org/ semanticscholes wwt.ooog//////eoee /CopppoID:2269115:555
[23] C. Finn and S. Levine. Deep visual foresight for planning robot motion. 2017 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 2786-2793, 2016. URL https:/ /tt::///api.. sewanticscoolar..c.coooooocc
Cttp:/CrrssII:I:: 278066999
[24] W. AI. Introducing gaia-l: A cutting-edge generative ai model for autonomy. 2023. URL https: h/wpyss//wwve..../ a htt.://hhhnnnnnnnntonoooo....... roduiing-- a Accessed: 2024-10-15.
[25] W. AL Scaling gaia-l: 9-billion parameter generative world model for autonomous driving, 2023. URL hwtpl//pp../eeeeeaaaa///////a/aa 10-15.
Accessed: 2024-
[26] X. Technologies. Ix world model, 2023. URL
1x-world-model.. Accessed: 2024-10-15.
[27] OpenAl. Video generation models as world simulators, February 2024. URL https:// openai. com/ index/viidie http:/vvdooggeeeee http:////i.eeeeeeeeettoooooooooo -tw.ee---a-oooo--sssmaaoooooo Accessed: 2024-10-15.
[28] Pika.art. Pika.art: Video on command, 2024. Utthttts::/piiaa.rrrt 2024-10-15.
Accessed:
[29] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference Robot Learning (CoRL), 2023.
[30] Z. Zhuang, Z. Fu, Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference Robot Learning, 2023. URL https: : : /ppi. semanticscholar: www.laa...rrggoooooooeeeee wwtp/oooppsD:
10

---

[31] A. Loquercio, A. Kumar, and J. Malik. Learning visual locomotion with cross-modal supervi- sion. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7295- 7302, 2022. URL https:/ / aapi. semant cescholar.....
[32] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[33] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Apr. 2023.
[34] Z. Fu, T. Z. Zhao, and C. Finn. Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation. Jan. 2024.
[35] C. Chi, Xu, C. Pan, Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and Song. Universal Manipulation Interface: In-The-Wile Robot Teaching Without In-The Wild Robots. Feb. 2024.
[36] Wang. Dasari, M. K. Srirama, S. Tulsiani, and Gupta. Manipulate by seeing: Creating manipulation controllers from pre-trained representations. 2023.
[37] N. M. M. Shafiullah, A. Rai, H. Etkkrr, Y. Liu, I. Misra, Chintala, and L. Pinto. On bringing robots home. ArXiv, abs/2311.16099 2023. URL https: http://///a..sssattiissiseeeeeee semanticcchho
org/Corpus ID: 25455972.
[38] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Commun. ACM, 65(1): 99-106, Jan. 2022.
[39] M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwa- ger. Vision only robot navigation in neural radiance world. IEEE Robotics and Automa- tion Letters, 10,1-1,0021, URL htsps: : api. remanticschol 238253331.
CorppssD
[40] Z. Yang, Chen, Wang, Manivasagam, W.-C. Ma, Yang, and R. Urtasun. Unisim: neural closed-loop sensor simulator. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1389-1399, 2023. URL https://api. semanti www.ccschllaroooooooooocccccaa.e org
[41] A. Byravan, J. Humplik, Hasenclever, A. Brussee, Nori, T. Haarnoja, B. Moran, S. Bohez, F. Sadeghi, B. Vujatovic, and N. Heess. Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields, 2022.
[42] S. F. Bhat, N. Mitra, and P. Wonka. Loosecontrol: Lifting controlnet for generalized depth conditioning. In ACM SIGGRAPH 2024 Conference Papers, SIGGRAPH '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400705250. 1oi:10.1145/ 3641519.3657525, URL http:////...........1111///..... 3657525
org /cc. 10. 11w...111111111......
[43] L. Höllein, A. Cao, A. Owens, J. Johnson, and M. Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEF/CV International Conference on Computer Vision (ICCV), pages 7909-7920, October 2023.
[44] 7. Chen, G. Wang, and 7.. Liu. Scenedreamer: Unbounded 3d scene generation from 2d image collections. IEEE Transactions Pattern Analysis & Machine Intelligence, 45(12):15562- 15576, dec 2023. ISSN 9939. 101:10.110///PAMM22233.3333
[45] Comfyanonymous. Comfyui, 2023. URL https::/s///t wwt.:oocccccc// com.eno///mmmm//aaymmoooss
ComfyUI. GitHub repository.
[46] S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference. In International Conference Learning Represen- tations (ICLR), 2024. URL https: http///pppnweeeeeeeeee/ee////eep net. wtt.ee//oormmmmdddddc/oeoooo

---

[47] G. Yang. Zaku task queue: A fast, scalable task queue for machine learning workloads, 2024. URL https::/p//.........m. zaku. readthedocs, /lo//n/latesss Light-weight, scalable task queue for machine learning workloads. Accessed: 2024-10-15.
[48] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Martin, A. Joshi, S. Nasiriany, and Y. Zhu. robo- suite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. /2293, 2020.
[49] L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference Computer Vision (ECCV), 2016.
[50] L.. Schönberger and J.-M. Frahm. Structurefrrom-motion revisited. Conference on Com- puter Vision and Puttern Recognition (CVPR), 2016.
[51] M. Turkulainen, X. Ren, I. Melekhov, O. Seiskari, E. Rahtu, and J. Kannala. Dn-splatter: Depth and normal priors for gaussian splatting and meshing, 2024.
[52] M. Tancik, E. Weber, E. Ng. R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin,
K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
12

---

Appendix
A.1 Generative Workflow
We developed our image generation workflows with ComfyUl [45], popular software tool with a graphical user interface that facilitates rapid prototyping. Figure A15 contains a screenshot of the stairs scene. The supplementary material includes a copy of the Python implementation this workflow that we use in production to generate images for this paper.
Figure A15: Generative workflow. This example is for the stair scene.
A key feature of this workflow is Area Composition, where cross-attention with text prompt specific regions. The base generative model, Stable Diffusion XL Turbo [6], when used in combination with the LCM sampler [46], is very fast and can generate reasonable images with single diffusion step. Image quality improves with more diffusion steps, up to eight or ten, beyond which it starts to degrade. We run six diffusion steps production to balance between speed and image quality.
semantic masks limit the
A.2 System Design Strategies for Scaling Image Generation
Image generation contributes the bulk of each experiment's wall-clock time. We accelerate data generation distributing trajectory sampling and image generation parallel workers. We up two Zaku task queues [47]. The first is responsible for dispatching unroll requests for each trajectory to the trajectory sampling workers. The image generation node is called "dream-weaver." or weaver in short. The second Zaku task queue is responsible for dispatching image rendering requests to these weaver workers. We present an overview the system design Figure A16.
The system design is different between the pre-training phase using expert unrolls, and the on-policy learning step that samples from the visual policy. In the pre-training phase (see Figure A16 a), render images after each rollout. The unroll workers upload the semantic masks, optical flow, and depth image to the weaver queue, which triggers the weaver workers to generate and upload the requested image to a centralized data server.
In the on-policy learning phase, we need to sample actions from the visual policy, which requires im- age observations as input. We implemented pub-sub and remote procedural calls (RPC) in Zaku (see documentation), backed by redis replica set. At the start of each flow stack (of seven frames), the unroll worker dispatches request for the first frame to the weaver RPC queue. Once the generated
13

---

Experiment Config Swerpe
Tescher Unnoll Task Quene
Experiment Coodig Sweaps
Quere
stacks
peivileged ols.
Unnall Worker
Unall Rereer (ditzec))
daptii
Image Frames
Queue
Wenver RPC Quene
Images
Unroll Buffer (datasee)
GPU Worker
GPU Worker
(a) Generating Visual Data Off-line.
Collecting On-Policy Unrolls Requires RPC.
Figure A16: System Architecture. Left: collecting unroll data from the expert teacher does not require generating the images online. We populate the weaver queue once after collecting a hatch of teacher unroll, and the weaver workers upload the generated and downsized images the mi- logger service. Right: collecting unroll from the visual policy required building remote-procedural call (RPC) into the task queue. The unroll buffer is a centralized ml-logger server that stores and serves the unroll data for the learning step.
frame has been returned it is used input the visual policy. During the subsequent six timesteps, the unroll worker computes the optical flow and uses warp the initial frame into those following frames. This process restarts after six steps, with new weaver RPC request.
A.3 Domain Randomization Baseline
Our domain randomization pipeline [15] is adapted from Robosuite [48]. We randomize the appear- ance the terrain by sampling textures (solid, checker, noisy, gradient), color, and material
Hurdle
Stairs
Chase Soccer
Chase Cones
Sample images from the Domain Randomization baseline. Textures, colors, material properties, and lights are randomized every seven steps. We do not randomize the cone so that the policy can learn to it as a landmark. This makes it fair comparison to LucidSim.
Figure A17:
14

---

properties (reflectance, shininess, specular). We also randomize the lighting parameters of each light in the scene. Just as with LucidSim, we sample a new appearance every seven frames. Figure A17 includes samples of the rendered image from the domain randomization baseline on all four domains.
A.4 Training Details
The expert teacher is trained according to the procedure described in [5]. We reproduce the reward used during training in Table 2; the PPO training parameter in Table 3; and the domain randomiza- tion in Table 4.
For imitation learning (both phases), we use the same training hhper--yramannn parameters presented in Table 5. In the pre-training phase, we sampled the teacher trajectories from a mixture of expert teachers of multiple random seeds and their intermediate checkpoints. The action labels used to supervise the visual policy were always computed by the expert teacher.
Table 2: Expert Reward Terms
Table 3: Expert Training Parameters
Term
Symhol
Scale
Hyperparameter
Value
parkour velocity tracking [5]
min ((0, dre)s tema)
1.5
value loss coefficient
1.0
yaw tracking
exp(
Iw2
wand
0.5
clip range entropy coef Icarning rate
0.01
velocity roll-pitch velocity base orientation
-1.0
2c-4
Wayy
-0.05
# minibatches per epoch
1l nar proj
-1.0
# epochs per rollout timesteps per rollout discount factor
hip position collision
qqip
Qup
-0.5
1 collision
-10.0
0.99
GAE parameter max grad norm optimizer
0.95
action rate
la.
-0.1
1.0
joint accelerations
-2.5e-7
Adam
delta joint torques
Tr T--1|
-1.0c-7
joint torques
|T|
-1e-05
joint stiffness
20
joint error
1 4012
-0.04
joint damping
foot vertical contact [5]
Σ,
(5)
-1.0
vertical
foot clearance [5]
Σ,
(1)
contact -1.0
Table 4: Expert Randomization Parameters
Table 5: Behavior Cloning Parameters
Term
Min
Max
Unit
Hyperparameter
Value
friction range
0.6
2.0
max. timesteps per rollout
600
added mass
0.0
kg
rollouts per DAgger Iteration
1000
Body Center of Mass
-0.20
0.20
learning rate
5e-4
push velocity (vr, My)
0.0
0.5
m/s
Motor Strength
80
120
%
timesteps optimizer
Forward Velocity Command (v.)
0.3
0.8
m/s
Adam
weight decay
5e-4
momentum
0.9
dropout
0.1
A.5 Real-to-Sim Evaluation Environments
Figure A18 provides an overview of our process for constructing the evaluation environments. For each task, we select a few scenes that differ in appearance (c.g. red bricks, pavement, grass, indoors). We report the results from three different scenes on each task in Tables and 6. We capture 500 images for each scene, and extract the resulting collision mesh from Polycam. For appearance, COLMAP [49,50] 50] to obtain camera pose estimates, and reconstruct the scene using 3D Gaussian Splatting (3DGS) [14, 51, 52].
15

---

3D Mesh
(c) 3DGS aligned
Scan seene
(c) 3D Gaussian Splatting
(d) Align 3DGS and mesh
Add waypoints and
Figure A18: Process for making the benchmark environments. (a-c) Scan the scene to collect the 3D mesh and the 3D Gaussian Splats. (d-e) These two are initially unaligned, so we manually scale and align them. (f) add the markers for the waypoints.
Figure A19: Learning Visual Parkour from Generated Images. Top bottom: (1.2) robot climbing stairs. (3) robot climbing hurdles on a stone ground (4) on a grassy courtyard. Notice the different box color. For more experiment videos, visit https:/lucidsimm.ginn iithuu.
We use our custom viewer to align the collision mesh with the gaussian splat. For the hurdle and stairs scenes, we manually label 3-5 waypoints along the course that appear as orange traffic cones. We use the collision mesh as the terrain, and the 3DGS render as visual observation to the robot. For objects that are not present in the initial scan (i.e., soccer ball, traffic cones), we apply the mask rendered by the physies engine insert them into the robot's ego view.
A.6 Prompt Used in Figure 4
Foreground prompt "Cool, gray slabs of granite that are flecked with darker mineral deposits. The polished finish is unmarred but faintly glistening under the ambient light, revealing a durable, ancient presence."
Background prompt "An ancient alley lined with tea houses and small, quaint shops, each dis- playing traditional ornaments and calligraphy. The walls are adorned with ivy and red paper deco- rations, while overhead, strings of lanterss sway gently in the breeze."
16

---

A.7 Prompts Used in Figure 14
Hurdle scene "Sunny Afternoon"
foreground "A ceramic box with colorful, intricate patterns."
background : "The sun illuminates somewhat unkempt lawn dotted with dry patches. Gravel paths crisscross the grass, leading to sunlit, red-brick buildings with large, gleaming windows."
Stairs scene "Cemented Courtyard Pathway"
foreground "Composed of a myriad of small, smooth pebbles incorporated within, creating uniquely textured appearance. Discoloration and cracks reflect its rich history of countless treaders."
"Walls of natural stone, varying from reds yellows, provide a strikingly authentic atmosphere. Wrought iron railings draped with climbing roses frame the scene. The melody of nearby street musician overlays the hum of quiet conversations, while the occasional cyclist clacks down the cobbled thoroughfare."
background ":
A.8 X-Displacement
We report the corresponding X-Displacement metric for Table 1 in this section for completeness.
Table 6: X-Displacement Real-to-Sim Benchmark Environments.
Chasc-Cone
Chase-Socorr
Hurdie
Stairs
Method
Obs. Space
Lawn
Lab
Urban
Lawn
Lab
Urban
Lab
Urban
Bricks
Conerete
Marble
Privileged Expert
state+terran
99.6
99.1
98.7
99.6
99.1
98.7
96.3
100.0
99.0
91.2
100.0
76.0
Extreme Parkour
elipped depth depth
88.8
98.8
96.6
95.5
83.2
Depth
95.8
93.6
93.8
95.0
92.9
92.9
80.7
70.4
59.
93.5
88.8
76.5
Dopth
elipped depth
99.9
94.7
99.8
99.8
97.9
99.9
75.4
88.5
89.5
94.0
88.7
86.1
Domain Rand.
color
91.6
81.2
84.9
99.3
89.5
61.6
LncidSim
color
99.5
92.7
99.7
92.3
96.8
98.0
92.6
94.0
97.8
88.8
85.6
83.6
17


Reviews:
[{"review_id": "WrabIO4bTo", "reviewer": "Reviewer_xbt2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper demonstrates how to use generative AI to generate a diversity of experience to support RL-based training of a walking robot. It works with chatGPT to generate a generator of environment description which is then used to generate physically consistent sequence of rendering.", "summary_of_recommendation": "This paper presents a very interesting new idea that could prove to be a major contribution to the RL community. Nevertheless, it is not presented with the thoroughness expected from a scientific contribution.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "The ablation study and the limitation section are both only a few lines and most of the pipeline is described in the appendices. Because CoRL is a scientific conference, I feel that the large space allocated to figures should be moved to the appendices, and the paper itself should focus on the scientific contribution, its algorithms, its detailed analysis and justification, with a real ablation study.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQeKVW4Gwx", "reviewer": "Reviewer_di9g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a set of approaches that can be used to generate realistic RGB observations from simulated depth images to use when training a RGB-input policy, helping avoid the low real-world accuracy of policies trained directly on simulated RGB images.", "summary_of_recommendation": "The paper introduces a clever new approach to bridging the sim-to-real gap caused by poor rendering. However, it has some strong assumptions (the scene geometry needs to be modeled ahead of time) which lowers the significance of the contribution. Nonetheless, it is an interesting contribution that would be nice to see at CoRL.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Can you describe the exact simulation setup (e.g. what simulator, how did you design the geometry, etc.)?\n\nCan you also describe some possible approaches for how the need for hand-designing the scene geometry be avoided?\n\nHow do you think the policy performs compared to a policy that is trained and tested with depth images instead of RGB?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hRhUQZAXHl", "reviewer": "Reviewer_n6VN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work attempts to learn visual parkour by training with a teacher-student method and performing behavior cloning using generative AI. To do so, the work proposes using a meta-prompt to utilize ChatGPT to generate prompts to create diverse images for training. Dreams-in-motion is then proposed to generate short videos for training, used in conjunction with DAgger. Results are demonstrated on multiple tasks, where notable baselines include a standard augmentation to RGB as well as a commonly utilized depth-based model. Results are shown in both real-world and real-to-sim experiments.", "summary_of_recommendation": "This work demonstrates a shift from recent work that has performed parkour abilities through depth, instead a RGB-based method is proposed to do this. The proposed method outperforms a domain randomization and depth-based model as baselines. With the combination of adding scaling through generative AI, this method is promising for improving related locomotion works.", "technical_quality": 4, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1) Add the depth model as a baseline in the real-world experiments table (Figure 11).\n2) Add the requested technical details to the methods and experiments section.\n\n(Post-Rebuttal). I thank the authors for their improvements and additional experiments to the manuscript. It appears that the table pasted below may have some inaccuracies, as the table for reviewer di9g seems to have more accurate numbers based on the reply. It might also be beneficial if the methods section variables were utilized more explicitly, rather than just introduced and condensed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: High level summary of reviews (PRE REBUTTAL):

Strengths:

- Reviewer di9g highlights that the paper is a novel and interesting idea. Other reviewers rate originality quite high (if not explicitly calling out originality in the review); there are no obvious issues around lack of technical contribution.
- The proposed method is shown to be effective in generating diverse and realistic training data for RL-based learning of walking robots. It is clear that the proposed method works well.
- The paper is well-written, with generally good figures and reasonably clear explanations of the proposed method and experimental results (with some caveats around completeness and clarity - see below).

Weaknesses:

- Some reviewers found that the paper lacks sufficient technical details and could benefit from a more thorough description of the pipeline and design choices. Improving the clarity of writing (see reviewer suggestions for specifics) will help with this.
- The level of details for the ablation study is considered insufficient and could be improved to provide more insights into the contribution of different components of the proposed method.
- One reviewer suggests that the paper could be improved by comparing the proposed method with relevant literature.
- The significance of the contribution is questioned by one reviewer, who argues that the approach has strong assumptions (e.g., the need for pre-modeled scene geometry) that limit its usefulness.

POST REBUTTAL:

The authors reworked the introduction, clarified a number of key items requested by the reviewers and added additional experiments. While reviewer ratings didn't change, the paper is still quite a strong submission and is improved following the edits during rebuttal.
Decision: Accept