Learning Robotic Locomotion Affordances and Photorealistic Simulators from Human-Captured Data
Authors: Alejandro Escontrela
Justin Kerr
Kyle Stachowicz
Pieter Abbeel
Venue: CoRL 2024 (2024)
Keywords: Navigation
Dataset
Real2Sim
TL;DR: A data capture system and dataset that enable the training of locomotion affordance models and fast, photorealistic simulators.

Abstract: Learning reliable affordance models which satisfy human preferences is often hindered by a lack of high-quality training data. Similarly, learning visuomotor policies in simulation can be challenging due to the high cost of photo-realistic rendering. We present PAWS: a comprehensive robot learning framework that uses a novel portable data capture rig and processing pipeline to collect long-horizon trajectories that include camera poses, foot poses, terrain meshes, and 3D radiance fields. We also contribute PAWS-Data: an extensive dataset gathered with PAWS containing over 10 hours of indoor and outdoor trajectories spanning a variety of scenes. With PAWS-Data we leverage radiance fields' photo-realistic rendering to generate tens of thousands of viewpoint-augmented images, then produce pixel affordance labels by identifying semantically similar regions to those traversed by the user. On this data we finetune a navigation affordance model from a pretrained backbone, and perform detailed ablations. Additionally, We open source PAWS-Sim, a high-speed photo-realistic simulator which integrates PAWS-Data with IsaacSim, enabling research for visuomotor policy learning. We evaluate the utility of the affordance model on a quadrupedal robot, which plans through affordances to follow pathways and sidewalks, and avoid human collisions. Project resources are available on the [website](https://pawslocomotion.com).

Learning Robotic Locomotion Affordances and Photorealistic Simulators from Hunan-Captured Data
Alejandro Escontrela
Justin Kerr
Kyle Stachowicz
Pieter Abbeel
University of California, Berkeley
Original Dataset
Augmented Views
Affordance Labels
Figure 1: PAWS expands the affordance model's training data well beyond the original images by using the learned radiance field generate novel views of the scene. Affordance labels are generated by comparing DINOv2 features near foot poses to other parts of the scene.
Abstract: Learning reliable affordance models which satisfy human preferences often hindered by lack of high-quality training data. Similarly, learning visuomo- tor policies in simulation rendering. We present PAWS: comprehensive robot learning framework that uses a novel portable data capture rig and processing pipeline to collect long-horizon trajectories that include camera poses, foot poses, terrain meshes, and 3D radiance fields. We also contribute PAWS-Data: an extensive dataset gathered with PAWS containing over 10 hours of indoor and outdoor trajectories spanning a variety of scenes. With PAWS-Data we leverage radiance fields' reato-realistic rendering to generate of thousands of viewpoint-augmennted images, then produce pixel af- fordance labels by identifying semantically similar regions to those traversed by the user. On this data we finetune navigation affordance model from pre- trained backbone, and perform detailed ablations. Additionally, We open source PAWS-Sim, a high-speed photo-realistic simulator which integrates PAWS-Data with IsaacSim, enabling research for visuomotor policy learning. We evaluate the utility of the affordance model on a quadrupedal robot, which plans through af- fordances to follow pathways and sidewalks, and avoid human collisions. Project resources available the website.
be challenging due to the high cost of photo-realistic
Keywords: Navigation, Dataset, Real2Sim
Conference on Robot Learning (CoRL 2024), Munich, Germany.

---

Capture
SLAM
Gaussian Splat
IsaacSim Photorealistic Rendering
Locomotion Affordance Model
Figure 2: PAWS proposes a data capture pipeline consisting of three cameras positioned on the shoulders (1, and facing down (3), as well as fiducial marker attached to each foot (4). RGB and depth streams are processed with SLAM to estimate camera poses while fiducial markers provide foot poses. The processed data is then used to extract terrain meshes and train 3D Gaussian radiance fields with Splatfacto. We utilize the processed dataset to train a powerful locomotion affordance model and develop fast and photo-realistic simulator.
1 Introduction
Ensuring that robots navigate in the wild while satisfying human preferences is crucial to ensure safe deployment. Without considering navigation affordances, our robots may perilously ignore cross- walks, stray off of sidewalks, trespass on forbidden ground, or damage themselves on hazards. How- ever, current algorithms for learning affordances and training visuomotor policies often suffer from a lack of high-quality training data. Additionally, defining effective affordance labels from human data is challenging, as the naive approach of assigning high affordances solely to traversed areas leads to sparse labels with many false negatives. Further, simulators for training visuomotor policies tend to provide low-fidelity renders.
In this work we propose PAWS, a method for instrumenting a person with cameras to capture ego view of natural human locomotion the wild for affordance model training. PAWS can record long-horizon trajectories of indoor outdoor scenes and process them obtain camera poses, foot positions, radiance field reconstructions, and terrain meshes. Using this method contribute PAWS- Data, a dataset of over ten hours of the aforementioned data spanning diverse scenes including hikes, cityscapes, and indoor environments. We also integrate PAWS-Data with IsaacSim (Mittal et al., 2023) to develop a fast and realistic simulator which can generate thousands of photo-realistic images per second across thousands of environments.
Neural Radiance Fields have exploded in popularity method (Mildenhall et al., 2020), however they are prohibitively expensive to train and render at scale. Recently, Gaussian Splatting (Kerbl et al., 2023) significantly reduces the cost of rendering by elim- inating the need for expensive ray casting operations, opening the door for real-time reconstruction in robot data pipelines. We take advantage of this fact to reconstruct photo-realistically renderable environments from human locomotion data.
photo-realistic reconstruction
We utilize the PAWS-Datt radiance fields to train a Locomotion Affordance Model (PAWS-LAM) large augmented dataset by rendering from novel views while retaining visual fidelity. Affordance labels are generated using DINO features to label parts of the scene which are semantically similar to areas traversed by the user. our experiments, we demonstrate that viewpoint augmentation significantly enhances the prediction accuracy for trained models, while semantic- labels accurately capture human preferences for navigation. We also find that foundation model backbone enables generalization to different environments beyond the training data. We apply PAWS- LAM to a quadruped navigation task, and combine PAWS-LAM with a simple planner to produce robot trajectories which are safe and efficient.
affordance
2 Approach
In this section we discuss the three contributions that make up our method: PAWS: data collection system for capturing and processing large quantity of trajectories. PAWS-LAM: A robotic locomo- tion affordance model that leverages learned radiance fields and semantic features to produce high

---

Figure 3: PAWS-Data provides many data products derived from the original RGB, Depth, and IMU streams, including scene meshes (1), pointclouds (2), camera poses (3), and foot positions (4).
quality affordance predictions, and PAWS- Sim: which leverages the PAWS-Date dataset and a mas- sively parallel simulators to achieve fast photo-realistic rendering across thousands of environments.
2.1 Data Collection
PAWS uses three cameras mounted on the user's torso to capture three stereo camera streams. Two cameras are mounted on the shoulder and angled such that they capture a wide field view while attending to the terrain ahead. A third camera is pointed directly down towards the user's feet, and is used for two functions: track the fiducial markers placed the user's shoes, and to provide clear top down view of the terrain for later when training 3D Gaussian radiance fields. Figure 2 demon- strates overview of the camera setup and our data processing steps. We use DROID-SLAM (Teed and Deng, 2021) to provide an estimate of the camera poses and to generate a colored point cloud of the environment. Importantly, we find that initializing DROID-SLAM disparities with stereo depth values significantly improves the quality estimated poses. We also compute human segmentation masks with Mask2Former (Cheng al., 2022) to mask out humans for later processing steps. Given the estimated camera poses, we can estimate foot poses in the global frame using the relative poses provided by the fiducial markers (Olson, 201 1). Re- Re-projecting depth from the known camera poses allows us to generate an accurate mesh of the environment, capturing details such stairs and rough terrains. Finally, the RGB, Depth, camera poses, and point cloud are used to train 3D Gaussian splatting model of the scene using Splatfacto (Ye et al., 2023; Tancik et al., 2023). Figure 3 visualizes the data products produced by PAWS. PAWS-Data consists of 10 hours of diverse indoor and outdoor trajectories, subset of which are visualized in Figure 6.
2.2 Locomotion Affordance Model
Dataset Augmentation Given a dataset consisting radiance fields and camera poses, we signifi- cantly expand the dataset size by rendering from novel viewpoints, a process we refer as Viewpoint Augmentation. Namely, we sample batch of new camera poses by perturbing the positions and orientations of the original camera poses by up meters and 30 degrees, respectively. We then ras- terize images from the sampled camera poses using the learned 3D Gaussian radiance fields. which allow for fast batch rendering. Figure 1 visualizes a radiance field along with the original camera poses, augmented poses, and affordance labels defined in the next section.
Affordance Label Generation Given dataset camera images, camera poses, and foot positions, naive approach to generate labels would be to simply project the foot position into the camera image, While these labels accurately capture where the human navigated, they sparse and falsely label possibly navigable areas as having low affordance. Instead, we propose to label regions which are

---

Gaussian Splatting Render
IsaacSim
RGB
Depth
Figure PAWS-Sim a fast, photo-realistic environment suite which leverages the 3D Gaussian radiance fields, terrain meshes, and poses from PAWS-Data to generate thousands of high-fidelity renders per second.
semantically similar to those traversed by the human. We do so by computing DINOv2 features (Caron et al., 2021) for the M patches the human stepped on in the scene. We then compute a M N x N cosine similarity matrix, comparing features in the input image. We take the minimum along the footstep feature dimension and threshold by CDINO get affordance patches, and upsample with bi- bi-linear sampling to obtain affordance labels. Figure 5 compares the naive dilated sparse affordance labels to the affordance labels produced by PAWS.
M traversed ground features the N x N
Model Architecture Given the affordance dataset from the previous section, fine-tune foundation model backbone the task of affordance prediction. Specifically, attach simple token projection head to pre-trained DINOv2 (Caron et al., 2021) model and train the model to minimize the cross entropy loss against the affordance labels. Section 3.1 we evaluate different choices of backbone and model size.
Figure Naive v.s. PAWS affor- dance labels
2.3 Fast and Photo-realistic Simulation
PAWS-Datt provides camera poses, foot poses, meshes, and radiance fields that accurately match the scale of objects in the real world. Additionally, 3D Gaussian radiance fields allow for fast rendering, allowing for thousands of renders per seconds. We integrate these features into IsaacSim (Mittal et al., 2023) to develop an open source photo-realistic gym environment for running robots which can be adapted to any desired reward signal or user-captured scene. Namely, we create a convex decomposition of the 3D terrain mesh to allow for stable physics simulation. In env env.step, we query the 3D Gaussians for each scene with a batch containing each robot's camera pose, camera intrinsics, and image resolution to obtain thousands of renders per second. While 3D Gaussian splats provide
renders for a large portion poses within the scene, deviating significantly from the original camera poses can lead to highly distorted and unrealistic renders. To combat this, we initialize the robots near the original camera poses, and terminate the episode once the camera pose has deviated significantly from the set of original camera poses, while still bootstrapping values.
3 Experiments
In this section evaluate PAWS-LAM and assess it's utility in practical robotics setting. Specifi- cally, we aim to answer the following research questions:

---

2
Figure 6: Example scenes in PAWS-Data. Aside from radiance fields, the PAWS dataset also contains terrain meshes, foot positions, and affordance labels. Artifacts in acrial renders can be attributed to the extreme viewing angle.
5

---

How does PAWS-LAM compare to alternative approaches to affordance learning?
How well does PAWS-LAM generalize to unseen environments?
Does PAWS-LAM learn affordances that facilitate safe quadruped navigation?
3.1 Locomotion Affordance Model Evaluation
We ablate different design decisions, including training with viewpoint augmentation and different backbone sizes. In Table Figure 7b, we ablate PAWS-LAM trained with and without viewpoint augmentation. We find that disabling viewpoint augmentation drastically reduces performance the validation set, as measured by pixel accuracy and mean intersection over union (mloU). We attribute this to the decreased dataset size, well as a smaller set of distinct camera poses, which both contribute to overfitting. Figure 7a visually demonstrates the failure modes of training without viewpoint augmentation. Specifically, models trained without viewpoint augmentation fail to capture the nuanced human preferences encountered in complex scenarios, such as predicting the entire road as navigable while only sidewalks or crosswalks should be traversed. Additionally, increasing the size of the pre-trained backbone from 21M parameters (ViT-S/14) to 86M parameters (ViT-B/14) significantly enhances model performance.
Additionally, in Table Figure 7e we evaluate PAWS-LAM's ability to generalize to environments unseen during training. We evaluate model trained solely on outdoor data on held-out dataset of indoor trajectories and find that PAWS-LAM generalizes favorably, displaying slight decrease in performance. Further, models trained without viewpoint augmentation generalize poorly to the indoor dataset, which we hypothesize is due to the model overfitting to small set of DINOV2 features which do not generalize to indoor environments,
Original Image
LAM Viewpoint Aug
Locomotion Affordance Model
Method
mloU
Pix Acc 0.72 = 2.9
LAM w/o View Aug
0.61 + 2.6
LAM (ViT-S/14)
0.76 + 1.5
0.84 + 1.5
LAM (ViT-B/14)
0.83 1.8
0.91 1.7
(b) Locomotion affordance model ahlations.
Method
mloU
Pix Acc
LAM w/o View Aug
0.38 + 1.2
0.44 + 1.4
LAM (ViT-S/14)
0.69 2.3
+ 2.1
LAM (ViT-B/14)
0.73 + 1.9
0.81 + 2.1
(c) Locomotion affordance model generalization perfor- mance on held-out indoor dataset. Model was only trained outdoor data.
(a) Locomotion Affordance Model (LAM) predic- tions exemplary held-out test images.
Figure 7: Locomotion Affordance Models trained with viewpoint augmentation more accurately cap- ture human navigation preferences such as staying on crosswalks and sidewalks and not hitting ob- stacles.
3.2 Robotic Navigation
We now propose one potential downstream application of our data collection interface and splat- based rendering pipeline: learning affordance models that transfer across embodiments, from human data robot locomotion. We propose a simple pipeline using our affordance model to optimize robot's high-level plan ensure that it is physically reasonable and matches human affordances such as sticking to marked paths or avoiding other humans.
We use Unitree GO1 quadruped as our target platform, equipped with two depth cameras to capture wide field of view the scene. For each camera, we compute visual affordances using PAWS-LAM, then project the predicted affordance probabilities directly onto point clouds from the depth cameras. This map is then flattened and used as 2-dimensional cost map, through which any planner can be

---

used to to generate a high-level plan. For simplicity, we use grid base- Dijkstra's 's algorithm, but the generally proposed method is applicable to any sufficiently flexible planning mechanism. Figure 8a visualizes an example plan through the affordance map.
As shown in Figure 8, we find that the resulting system is able to match human affordances in an in- tuitive manner, sticking to paths whenever possible and following forks in the road. Importantly, the robot's camera perspective is not contained in the initial dataset. Suggesting that the viewpoint aug- mentation used for PAWS-LAM helps to obtain high downstream performance on non-humanoid embodiment. Please see our supplementary materials for a reference implementation of the planning algorithm and videos of our robot following human-inte cenerrnnterpretable affordances.
RGB
(a) Fork in the road
(b) Human avoidance
(c) Turn
Figure Planning through learned affordances enables the quadrupedal robot to navigate safely without violating human preferences over navigation.
4
Related Work
4.1 Large Scale Photo-realistic Reconstruction
Scene reconstruction has long been studied, classically in the context of SLAM or SfM reconstruc- tion (Newcombe et al., 2011; Schönberger and Frahm, 2016; Mur-Artal et al., 2015). More recently, advances in neural rendering have drastically improved the capabilities of these systems for rendering visual outputs. Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) can render highly photo- realistic images from novel views, and an explosion of extensions expand on the speed (Müller et al., 2022), quality (Barron et al., 2021; 2022; 2023), scene scale (Tancik al., 2022), and tooling (Tan- eik et al., 2023). Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) proposes method of representing a scene with a collection of oriented 3D Gaussians, which can be rasterized onto image quickly with modern GPU hardware. This work uses 3DGS to reconstruct the surroundings of human capture trajectories for fast rendering at volume.
4.2 Learning Affordances for Robot Locomotion
For mobile robots to efficiently navigate the world while respecting both the laws of physics and so- cial rules, it is important that they have some notion of affordances describing which states they can enter and which actions they can perform. Traditional legged locomotion policy learning architec- tures learn these affordances in simulation with domain randomization (Feng al., 2023; Lee et al., 2020; Margolis al., 2023) relying either blind models (Kumar al., 2021) 1) or depth information (Agarwal et al., 2023). However, many more subtle cues cannot in fact be recovered by depth infor- mation alone: concrete road and a sand pit might appear identical when viewed as depth image (Loquercio et al., 2023), and they have difficulty resolving transparent or translucent objects. Further- more, using RGB cameras instead of depth can allow pociall--copplaie
behavior rather than purely physical behavior: walking on paths and in crosswalks rather than through grass field, for example. Terrestrial navigation research has also studied the problem of traversability estimation from visual data, particularly in the off-road driving (Jung et al., 2024). In contrast these works, we do not require hand-tuned heuristics nor our learned affordances limited to only physical parameters:
7

---

PAWS can learn rich, interpretable human affordances from features in in both dense urban environ- ments and outdoor settings, and can be used to provide a dense traversability score rather than just a 0-1 traversability map if desired.
4.3 Photo-realistic Simulators
Developing high-fidelitty simulators is a large and active area of research, spanning applications in autonomous driving (Amini et al., 2021; Dosovitskiy et al., 2017), UAV research (Shah et al., 2017; Guerra et al., 2019), and robotics (Mittal et al., 2023; Gan et al., 2020), While these simulators can achieve high-quality renders, they often require high compute budgets which decreases sample throughput. Additionally, designing simulation can be time consuming, and even the highest qual- ity simulators may not fully overcome the simulation to reality (Sim2Real) gap. These challenges have spurred researchers find new solutions to the challenges of overcoming the Sim2Real gap. Bousmalis et al. and Shrivastava et al. explored pixel-level domain adaptation by training networks to post-process the simulated render to appear more realistic. While exciting, these approaches are difficult to train due to lack of paired training data and can suffer from unwanted hallucinations. Tirumala et al. leverage Neural Radiance Fields (NeRF) to generate photo-realistic renders for an indoor robot soccer environment. While promising, rendering with NeRFs requires expensive ray tracing operations, which can be prohibitively slow when compared to rasterization, and NeRFs to not allow for efficient batch rendering. Additionally, they limited their focus to one particular environ- ment. In contrast, PAWS-Sim utilizes 3DGS rendering backbone, which enable batch rendering at speeds that are multiple orders of magnitude higher, while retaining high visual fidelity. Addition- ally, PAWS-Sin support the over 10 hours of scenes present in PAWS-Datt -Daa,, spanning wide variety of environments.
5 Discussion
We present PAWS, self-supervised data collection method for obtaining large datasets of scenes from human captured data. Accompanying PAWS PAWS-Data, dataset of over 10 hours of in- door and outdoor trajectories gathered with PAWS, containing RGB + Depth, camera poses, foot poses, terrain meshes, and radiance fields. We also introduces PAWS-LAM, which leverages PAWS- Data to train a locomotion affordance model capable of modeling the nuanced human preferences of outdoor navigation. We ablate PAWS-LAM and show that it benefits from our novel affordance dataset augmentation approach, our proposed affordance label generation method which drastically reduces false negatives, and the use of a foundation model backbone. Additionally, PAWS-LAM generalizes well to unseen environments, which we showcase by evaluating on indoor scenes using model trained solely outdoor data. We show how PAWS-LAM can be used for robotic navigation by using the affordance model optimize high-level plan ensure the robot's trajectory satisfies human navigation preferences. PAWS-LAM generalizes the robot's distribution of observed im- ages, showcasing favorable generalization across embodiments. Finally, we introduce PAWS-Sin -imm. a fast and photo-realistic simulator which integrates PAWS-Data IsaacSim to enable fast visuomotor policy learning.
Limitations & Future Work While PAWS can gather complete trajectories for the camera pose, the foot goes out of frame for roughly half the user's gait, thereby limiting foot position estimation for that duration. Additionally, fast motions can produce motion blur, making it difficult to detect the fiducial marker. Approaches such as SelfPose (Tomè et al., 2020) estimate body positions from headmounted camera and a pose detection network. Such approaches would potentially increase foot position availability and decrease hardware complexity. Further, integrating terrain dynamics estimation with our method would increase the physical fidelity of the simulator, further reducing the RimlReal gap.
8

---

Acknowledgments
We thank Philipp Wu for his assistance the design and manufacture of the data capture rig. We also thank Oleh Rybkin and Carmelo Sferrazza for providing insightful feedback that helped guide the project. This work was supported in part by an NSF Fellowship, NSF GRF #2146752. Pieter Abbeel holds concurrent appointments Professor at UC Berkeley and an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.
References
M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, L. Yuan, R. Singh, Y. Guo, H. Mazhar, A. Mandlekar, B. Babich, G. State, M. Hutter, and A. Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8(6):3740- 3747, 2023. 111.1111119/A...22.......
B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
B. Kerbl, G. Kopanas, Leimkühler, and Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https://repo-san inria. inria. wwwfffrrwfpf/ffaaae http:///uuuraaaodo.aaaaaaaaaaaaa
Z. Teed J. Deng. DROID-SLAM: deep visual SLAM for monocular, stereo, and RGB-D cameras. CoRR, abs/2108. 10869, 2021. URL https: /attps://arxiv.. wwwp:/iivvrrr....oooe arg/abs/2108 10869.
B. Cheng, Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1290-1299, June 2022.
E. Olson. Apriltag: robust and flexible visual fiducial system. In 20/1 IEEE International Conference on Robotics and Automation, pages 3400-3407, 2011. 101:10.1109/IRA2220111 5979561.
V. Ye. M. Turkulainen, and the Nerfstudio team. praiito-prooct/gsppla.e 2023.
gsplat.
https://github. com/
M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, et al. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.04264.222 2023.
M. Caron, Touvron, Misra, H. Jégou, Mairal, Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers, 2021.
R. A. Newcombe, S. Lovegrove, and A. J. Davison. Dtam: Dense tracking and mapping in real- time. In 2011 International Conference Computer Vision, pages 2320-2327, 2011. doi:10. 1109/1CCV2001 1.6626111..611133
J. L. Schönberger and J.-M. Frahm. Stracturefrrom-motion revisited. In Conference Computer Vision and Pattern Recognition (CVPR), 2016.
R. Aurrlrtal, M. M. Montiel, and J. D. Tardós. ORB-SLAM: a versatile and accurate monocular SL.AM system. IEEE Transactions on Robotics, 3165):1147-1163, 2015. doi: 1000/T00.... 2463671.
T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphies primitives with multiresolution hash encoding. ACM Trans. Graph., 41(4):102:11022115 July 2022. doi: 10.1145/5222233333127, URL https / //w..//iii/// org/10. 0145/3528822 3530127. 25555/332222...11177....

---

J. T. Barron, B. Mildenhall. M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip- nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3855-5864, 2021.
T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf' 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid- based neural radiance fields. arXiv preprint arXiv:2304.06706, 2023.
M. Tancik, Casser, X. Yan, S. Pradhan, B. Mildenhall, P. Srinivasan, T. Barron, and H. Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248-82555 2022.
M. Tancik, E. Weber. E. Ng, R. Li, B. Yi, J. KerI, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
G. Feng, H. Zhang, Z. Li, X. B. Peng, B. Basireddy, L. Yue, Z. Song, L. Yang, Y. Liu, K. Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference Robot Learning, pages 1893-1903. PMLR, 2023.
J. Lee, J. Hwangbo, L.. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47))eabc5986, 2020.
G. B. Margolis, X. Fu, Y. Ji, and P. Agrawal. Learning to see physical properties with active sensing motor policies. In J. Tan, M. Toussaint, and K. Darvish, editors, Proceedings of The 7th Conference Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2537-2548. PMLR, 06-09 Nov 2023. URL
www.m.rrs.......m..mm hir.press/v229/
margolis23a. html.
A. Kumar, Z. Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. ArXiv, abs/2107.04034, 2021. URL attps://api. semanticscholar org/CorpuuID::2566550
A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In K. Liu, D. Kulic, and J. Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 403-415. PMLR, 14-18 Dec 2023. URL sttps://procceedings .ml.. /tr.p/////205// agarwal23a.htan htmml
A. Loquercio, A. Kumar, and Malik. Learning visual locomotion with cross-modal supervision. In 2023 IEEE International Conference Roboties and Automation (ICRA), pages 7295-7302. IEEE, 2023.
S. Jung, J. Lee, X. Meng, B. Boots, and A. Lambert. V-strong: Visual self-supervised traversability learning for rof-road navigation, 2024.
A. Amini, T. Wang, Gilitschenski, W. Schwarting, Z. Liu, S. Han, S. Karaman, and D. Rus. VISTA 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles. CoRR, abs/211 11.12083,, 2021. URL peattps://axxiiie org/ aabs/2111 12083.
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
S. Shah, D. Dey, C. Lovett, and A. Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Roboties, 2017. URL https: //aaiiv org/abs/ 1705. 05065.
10

---

W. Guerra, E. Tal, V. Murali, G. Ryou, and S. Karaman. Flightgoggles: Photorealistic sensor simulation for perception-driven robotics using photogrammetry and virtual reality, 2019.
C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. D. Freitas, Kubilius, A. Bhandwaldar, N. Haber, M. Sano, K. Kim, E. Wang, D. Mrowca, M. Lingelbach, A. Curtis, K. T. Feigelis, D. M. Bear, D. Gutfreund, D. D. Cox, J. J. DiCarlo, J. H. McDermott, J. B. Tenenbaum, and
D. L. K. Yamins. Threedworld: A platform for interactive multi-modal physical simulation. CoRR, 0hs/2007.04954 2020. URL https: /t////axxii wttpirivivv////a///eaa///aaaaaaa wwwpsii..rrg//oo.e org/abs wwrg/abs//000 http://0000...900.
K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, S. Levine, and V. Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 IEEE International Conference on Robotics and Automation (ICRA), page 4243-4250. IEEE Press, 2018.
URL wwtph::t/p/oottooo org/10///// /0. 119////IAAAA 2018. 8460875.
A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb. Learning from simulated and unsupervised images through adversarial training. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2242-2251, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. 11t.111////....11...... URL //doi wtw.:liieeeeoooopeeeocci... org/10. 10. 1109/CVPR. 2017 241.
D. Tirumala, M. Wulfmeier, B. Moran, S. Huang, J. Humplik, G. Lever, T. Haarojja, L. Hasenclever, A. Byravan, N. Batchelor, N. Sreendra, K. Patel, M. Gwira, F. Nori, M. Riedmiller, and N. Heess. Learning robot soccer from egocentric vision with deep reinforcement learning, 2024.
D. Tomè, T. Alldieck, P. Peluse, G. Pons-Moll, L. Agapito, H. Badino, and F. D. la Torre. Selfpose: 3d egocentric pose estimation from a headset mounted camera. CoRR. abs/2011.01519, 2020. org/abs///a/eeeeeeee/..oo abs// 01519.
URL
O. Seiskari, J. Ylilammi, V. Kaatrasalo, P. Rantalankila, M. Turkulainen, J. Kannala, and A. Solin. Gaussian splatting on the move: Blur and rolling shutter compensation for natural camera motion, 2024.
L. Zhao, Wang, and P. Liu. BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting. 2024.
Z. Ye, W. Li, S. Liu, P. Qiao, and Y. Dou. Absgs: Recovering fine details for 3d gaussian splatting, 2024.
L. Lipson, Z. Teed, and J. Deng. Raft-stereo: Multilevel recurrent field transforms for stereo matching. CoRR, abs/2109.07547, 2021. URI. httpt::/s///aii...... wttp://wwwwrrraa///....... ht0t:////////////oooo/////////// 07547.

---

A Gaussian Splatting Details
We initialize the 3DGS start points with points deprojected from DROID-SLAM disparity, de- duplicated. To accomodate small amounts of SLAM drift, we enable camera optimization, similar to (Seiskari et al., 2024; Zhao et al., 2024), which refines poses during 3DGS training. deal with the large scene scale our captures, we need modify few important hyperparameters inside Splatfacto, which was primarily tuned for with object-centric captures. We the AbsGS split- ting heuristic introduced by (Ye et al., 2024) with gradient threshold of 0.0006 well lowered learning rate on the Adam optimizer controlling 3DGS means to 5e 5 as recommended by the orig- inal implementation. We also extend the length of the culling/splitting portion of 3DGS training from 15000 to 25000 steps, and lower the densify size threshold to 0.1% from 1,, ensuring fine details are reconstructed.
B Hardware Specifications
The PAWS data collection system consists of three cameras, one on each shoulder, and third camera pointed the feet. The shoulder-mounted cameras are stereo ZedX Mini cameras, while the down- ward facing camera is stereo Zed Mini. For depth estimation opt for the provided neural depth model, which is slightly less accurate than RAFT-Stereo (Lipson et al., 2021) yet considerably faster. The three cameras are connected to a rigid plate attached to the chest, and their precise transforms are derived from the CAD model. All three cameras are connected to a NVIDIA Jetson AGX Orin Developer Kit which hardware encodes and saves the three camera streams for later processing.
C Rendering Performance Analysis
3D Gaussian Splatting models support fast batch rendering, capable of rendering thousands of frames per second certain resolutions.
Resolution
FPS (RGB + Depth)
64 64
1487 + 12
96 96
1023 + 9
256 256
756 9
512 x 512
431 + 6
1024 1024
185 + 3
Figure C.I: Rendering frames per second for various resolutions. Reported times are for computing both RGB and Depth values from the 3DGS.
12


Reviews:
[{"review_id": "w69cqg6cvo", "reviewer": "Reviewer_ZQSK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel pipeline for locomotion data collection", "summary_of_recommendation": "This paper is making solid contributions, but its scientific contribution is rather low, and it could benefit from more analysis of the data.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I do not have specific questions. I encourage the authors to provide additional analysis of the semantic similarity-based labeling procedure.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9v6BACeQT2", "reviewer": "Reviewer_gXMo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a pipeline for collecting human-captured real-world data and affordances. Such data can be used for affordance prediction and building a photo-realistic simulator. The learned affordances can be used for guiding a navigation policy.", "summary_of_recommendation": "Overall a good paper -- experimental validation could be further improved.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The photo-realistic simulator is proposed but it is unclear how useful it might be for downstream tasks such as policy learning. It would be great (but not necessary, in this reviewer's opinion) to have experiments to validate if the simulator could be used for a downstream task.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMhRkhIInL", "reviewer": "Reviewer_V5mu", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents PAWS, a comprehensive robot learning framework designed to enhance robot navigation by leveraging human-captured data. The framework encompasses three main contributions: (1) PAWS Data Collection System: A portable data capture rig and processing pipeline for collecting long-horizon trajectories; (2) PAWS-Sim: Using Gaussian Splatting to construct a high-speed, photo-realistic simulation in IsaacSim using the PAWS-Data; (3) PAWS-LAM: A robot locomotion affordance model, which is fine-tuned on a dataset enriched through novel viewpoint augmentation with the DINOv2 model and viewpoint projection.", "summary_of_recommendation": "I recommend Weak Reject because of the lack of technical novelty in the method, however I am willing to change my recommendation either way depending on the author responses and reviewer comments", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. Please provide a detailed explanation of the process used to convert the \"affordance\" (the output of the model) into a score for planning.\n2. Could you elaborate on how motion blur affects the quality of reconstruction? Did you explicitly select keyframes to mitigate this issue?\n3. The assumption underlying the data collection method requires further clarification, especially considering the differences in dynamics between humans and legged robots.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The focus of this paper is on the development of a data collection and simulation pipeline for legged locomotion.

-- The strength of this paper is on the data collection pipeline. Generally the reviewers were favorable about that. One reviewer thought the data collection rig was too complex.

-- There was a question about whether the human footstep affordances collected in the data would generalize well to different robot morphologies. 

-- There was also a question about how well DINO v2 features would adequately capture footfall affordances and transfer them to novel scenes (reviewer ZQSK). Perhaps this is something that can be validated experimentally.
Decision: Accept