STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience
Authors: Haresh Karnan
Elvin Yang
Daniel Farkash
Garrett Warnell
Joydeep Biswas
Peter Stone
Venue: CoRL 2023 Poster (2023)
Keywords: Vision-based Navigation
Representation Learning
Learning from Experience

Abstract: Terrain awareness, i.e., the ability to identify and distinguish different types of terrain, is a critical ability that robots must have to succeed at autonomous off-road navigation. Current approaches that provide robots with this awareness either rely on labeled data which is expensive to collect, engineered features and cost functions that may not generalize, or expert human demonstrations which may not be available. Towards endowing robots with terrain awareness without these limitations, we introduce Self-supervised TErrain Representation LearnING (STERLING), a novel approach for learning terrain representations that relies solely on easy-to-collect, unconstrained (e.g., non-expert), and unlabelled robot experience, with no additional constraints on data collection. STERLING employs a novel multi-modal self-supervision objective through non-contrastive representation learning to learn relevant terrain representations for terrain-aware navigation. Through physical robot experiments in off-road environments, we evaluate STERLING features on the task of preference-aligned visual navigation and find that STERLING features perform on par with fully-supervised approaches and outperform other state-of-the-art methods with respect to preference alignment. Additionally, we perform a large-scale experiment of autonomously hiking a 3-mile long trail which STERLING completes successfully with only two manual interventions, demonstrating its robustness to real-world off-road conditions.

STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience
Haresh Karnan haresh. miresh. utexas.edu
Elvin Yang! elvin.yang@uur uueexas.edu
Daniel Farkash² dmf248@cornell @ocaolllllleddu
Garett Warnell garrett.a. warnett...wrreellco @ @army.mil
Joydeep Biswas'
Peter Stone 1,4 pstone @ uduu
Abstract: Terrain awareness, i.c., the ability identify and distinguish differ- ent types of terrain, is critical ability that robots must have succeed tonomous off-road navigation. Current approaches that provide robots with this awareness either rely on labeled data which is expensive to collect, engineered fea- tures and cost functions that may not generalize, or expert human demonstrations which may not be available. Towards endowing robots with terrain awareness without these limitations, we introduce Self-supervised TErrain Representation LearnING (STERLING), novel approach for learning terrain representations that relies solely easy-to-collect, unconstrained (e.g., non-expert), and unlabeled robot experience, with additional constraints on data collection. STERLING employs novel multi-modal self-supervision objective through non-contrastive representation learning to learn relevant terrain representations for terrain-aware navigation. Through physical robot experiments in off-road environments, we evaluate STERLING features on the task of preference-aligned visual navigation and find that STERLING features perform on par with sully-super approaches and outperform other state-of-the-art methods with respect to preference align- ment. Additionally, perform large-scale experiment of semi- mutimoutonomouslyy hiking 3-mile long trail which STERLING completes successfully with only two manual interventions, demonstrating robustness to real-world off-road conditions. Robot experiment videos and more details can be found in the appendix and the project website https//haareskkarnan http://hww.aaaniiiu...ooo/ttiiit gittubbio/sterlingg
Keywords: Vision-Base Bason- Navigation, Representation Learning.
Introduction
Off-road navigation is emerging as a crucial capability for autonomous mobile robots envisioned for use in a growing number of outdoor applications such as agricultural operations [1], package delivery [2], and search and rescue [3]. Endowing robots with this capability has, however, proved to be challenging and remains active area of research.
One particularly difficult challenge in off-road autonomous navigation is that of providing the robot with terrain awareness, i.e., the ability to identify distinct terrain features that are relevant to a wide
variety of downstream tasks (e.g., changing preferences over terrain types) [4, 5, 6, 7, 8, 9, 10, 11]. While litany of prior work has attempted address this challenge [12, 13, 14, 15], existing approaches typically rely difficult-to-colleet curated datasets [16, 17, 18, 19, 20] has been
'The University of Texas at Austin, Austin, TX, USA
*Comell University, Ithaca, NY, USA "Ammy Research Laboratory, Austin, TX, USA *Sony AL, North America
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

focused on particular tasks [21, 22, 23, 24, 25, 9] and is not amenable to downstream task changes [26, 25, 7]. These limitations prevent existing approaches from appropriately scaling to the vast distribution of terrains and navigation tasks in the real world.
Toward overcoming the scalability challenges in terrain awareness, introduce Self-supervised TErrain Representation LearlNNG (STERLING)), novel approach to learning terrain representa- tions for off-road navigation. STERLING learns an encoding function that maps high-dimensional, multi-modal sensor data to www-dimesiional, verrain-award representations that amplify differences important for navigation and attenuate differences due to extraneous factors such as changes in view- point and lighting. Importantly, STERLING works with casy-to-collect unconstrained and unlabeled robot data, thereby providing scalable pathway to data collection and system improvement for the wide variety of terrain and downstream tasks that off-road robots must face.
To evaluate STERLING, we apply it to the problem of preference-aligned off-road navigation and provide a detailed comparison to existing approaches to this problem, including RCA [7], GANav [19], SE-R [8], and a sully-supervised oracle. We find that STERLING enables performance on par with or better than these existing approaches without requiring any expert labels or demonstrations. Additionally, we report the results of large-scale qualitative experiment which STERLING en- abled semi-autonomous robot navigation
3-mile long hiking trail.
The key contributions of this paper are- 1) Self-supervised TErrain Representation LearnING (STERLING), a novel approach that learns terrain representations from easy-to-collect unconstrained robot experiences, 2) Detailed evaluation of STERLING against baseline methods on the task of op- erator preference-aligned off-road navigation, and 3) A large-scale qualitative experiment of semi- autonomously hiking 3-mile long trail, demonstrating the effectiveness of STERLING-features.
2
Related Work
In this section, we review related work on terrain-aware visual off-road navigation. We specifically focus approaches that learn to navigate off-road conditions using supervised and self- self-supervised learning.
2.1 Supervised Methods
Several approaches in the past have proposed using supervised learning from large-scale data to navigate off-road environments. We divide them into two categories as follows.
End-to-End End Learning: The initial success of applying dearning-based solutions to off-road terrain- aware navigation was by LeCun et al. [28] who used a convolutional network to learn to drive in off-road conditions. More recently, Bojarski et al. [21] trained a deep neural network end-to-end using several miles of driving data collected on vehicle in the real world. While both approaches were promising in urban and off-road environments, end-to-end methods require large amounts of data and are well-known to suffer from domain and covariate shifts [29, 30, 31].
Image Segmentation: Unlike end-to-end approaches that learn behaviors, segmentation-based ap- proaches seek to characterize terrain using a set of known semantic classes, and the resulting semantic features are consumed by downstream planning and control techniques for navigation [32, 19, 33]. Guan al. [19] propose GANav, a transformer-based architecture pixel-wise seg- ment terrains, trained RELLIS [17] and RUGD [16] datasets, with manually assigned terrain costs. While effective at terrain awareness, regmentation-based methods are fixed to the specific terrain types available in the datasets and require additional labeling effort to generalize to novel ter- rains. In STERLING, we do not require semantically labeled datasets and learn terrain representations from unconstrained experience collected onboard a mobile robot.
A preliminary version of this work was presented at the PT4R workshop ICRA 2023 [27]

---

2.2 Self-Supervised Learning
To alleviate the need for extensive human labeling, self-supervised learning methods have been proposed either learn terrain representations or costs from data gathered onboard a mobile robot.
Representation Learning: Brooks et al. [34] utilize contact vibrations and visual sensors to clas- sify terrains via sulf--uueevision. Loquercio et al. [35] use proprioceptive supervision to predict extrinsic representations [36] of terrain geometry from vision, used as inputs to drive a Reinforce- ment Learning- based locomotion policy. this work, we do not learn robot-specific locomotion policy and instead learn relevant representations for off-road terrain awareness. Zürn et al. [8] introduce SE-R which utilizes acoustic and visual sensors on the robot segment terrains using self-supervised triplet-contrastive learning framework. Using triplet-based contrastive learning methods requires negative samples which may not be available when learning using unlabeled data. In STERLING, we use recently proposed non-contrastive unsupervised learning approaches such as VICReg [37] that do not require any negative samples and instead rely on correlations between data modalities to learn relevant terrain representations.
Cost Learning: Several methods have applied self-supervision to assign traversability costs for the downstream off-road navigation task [7, 38, 26, 39, 40, 41, 42]. Specifically, these methods rely inertial spectral features [7], future predictive models [26], inertial-odometry errors [38], force- torque values from foothold positions [39, 43] as self-supervision signals to learn a traversability cost map, used to evaluate candidate actions. More recently, Frey et al. [44] have proposed an online traversability estimation approach inspired by the above self-supervision schemes. Instcad of inferring costs or rewards using self self-supervision for fixed task, in this work, we focus on learning relevant visual features from unconstrained robot experiences that could be used in downstream tasks. This framework allows designer reuse features across tasks without retraining entirely from scratch.
The approach closest to ours is VRL-PAP [6] which requires human expert tele- operated demonstrations of a particular trajectory pattern to both explicitly learn visual terrain rep- resentations as well as to infer terrain preference costs. However, in this work, we focus on learning terrain features from unconstrained robot experiences without requiring human experts in the field for demonstrations, which is general problem than the one considered by VRL-PAP.
Hybrid Methods:
3 Approach
In this section, we introduce the supe-suue terrain representation learning approach, STERLING, proposed in this work. We first describe the offline pre-processing performed on unconstrained robot data and then summarize the self-supervision objectives. Finally, describe the problem formu- lation for preference-alignee off-road navigation and present how features learned using STERLING can be utilized within planner for terrain-awar and preference-aligned navigation.
3.1 Data-Collection and Pre-Processing
STERLING learns terrain representations from unconstrained, unlabeled robot experiences collected using any navigation policy. This policy may be, for instance, non-expert human teleoperation, curiosity-driven exploration [45], or point-to-point navigation using any underlying planner. Com- pared to requiring a human expert to provide telcoperated demonstrations and labels, collecting this type of robot experience is cheap and easy, thereby providing scalable pathway to data collection and system improvement. We additionally assume that the robot is equipped with multiple sensors, e.g., an egocentric RGB camera, odometry sensors, onboard IMU, proprioceptive, and tactile sen- sors, that together provide rich multi-modal observations as the robot traverses over different terrains collecting experience. STERLING leverages this multi-modal data by using the correlation between different modalities inform the learned terrain representations.
3

---

In order to learn terrain representations using STERLING, we begin by pre pre-processing the visual and non-visual observations, which are explained detail below.
Visual Patch Extraction: The egocentric camera observations are homooraphy--pojjeceed into a vir- tual bird's eye view (BEV) frame, assuming that the ground is a flat plane, using the intrinsic and extrin- sic camera matrices. As shown in Fig. 1, we project
sertoure
transter
the robot's trajectory onto the BEV frame and ex-
Proproo
tract 64-by-64 pixels (equivalent to the robot's foot- print of 0.5-by-0.5 meters) square visual patches of terrain along with the corresponding inertial, pro- prioceptive, and tactile observations at the same lo- cation, along the trajectory. Since the terrain at $ę is unobservable when the robot itself is at Sj (i.e., underneath the robot), we extract terrain im- age patches corresponding to 8k from BEV obser- vations at previous locations 8g-1; 2... along its trajectory. Fig. illustrates the offline patch extraction process from two previous viewpoints, however, we extract patches from up to 20 previ- ous viewpoints within 2 meters. Although just one viewpoint is sufficient to learn the correlation be- tween visual and other sensor observations, when planning to navigate, the robot will need visu- ally evaluate terrain at future locations, and therefore STERLING also seeks representations that are invariant patch differences due viewpoint, also known viewpoint invariance.
8k-2
An illustration of the pre- pre-oessesing performed on unconstrained robot experience. Image patches of traversed terrain at location 8ą are extracted from bird's eye view observa- tions at prior locations 8 8 along the trajectory. The corresponding IPT observa- tions at 8 are transformed from time series to PSD signals. Note the visual artifacts caused by noise homography transformation from viewpoints farther away from 8).
Figure I:
ITT Pre-Processings For the inertial, proprioceptive, and tactile (IPT) observations, we retain to 2- second history and convert the time-series signals into power-spectral density (PSD) representation in the frequency domain. This ensures the IPT time-series data representations used as input to STERLING are invariant to differences in length and phase in the recorded signals. Additional details are provided in Supplementary Section 9.5.
3.2 Non-Contrastive Terrain Representation Learning
It is desired for learned representations of terrains to be such that representations of similar terrain are close together in the embedding space and that representations of different terrains are suffi- ciently far apart. Although we do not possess privileged information such as semantic labels of terrains for training, the visual and kinodynamic observations experienced by the robot reflect sim- ilarities and differences between terrain samples. For instance, traversing smooth terrain that human may refer to cement sidewalk may lead to relatively smooth motion by the robot's joints, whereas rough terrain such as what might be referred to as marble rocks may correspond to jerkier motion. STERLING leverages this multi-modal experience observed by the robot and com- putes a correlation objective between visual and pertiial-pooroo-tttee rain representations. Additionally, STERLING uses viewpoint invariance as an objective unique to the visual component of the experience to learn viewpoint- viewpoint-invarian invariant terrain representations.
signals to learn desired ter-
Fig. 2 provides overview of the self-supervised representation learning framework adopted in STERLING. A parameterized visual encoder (4-layer CNN with 0.25 million parameters) encodes terrain image patch observations vj and U2 of the same location s into visual representations Φ., and 6uz, respectively, collectively referred to as Φ0,2 for brevity. Similarly, an inertial mnertial-proprio-tee proprio-tactile encoder (4-layer MLP with 0.25 million parameters) encodes frequency domain IPT observations of the robot at that location to peetial-proprio-ttcttee representation ói. We follow the framework of prior self-supervised representation learning algorithms from the computer vision community such VICReg [37], and utilize parameterized projector network (2-layer MLP with 0.25 million

---

local parameters) that maps encoded visual and non-visual representations independently to a higher- dimensional feature space Ve and W, respectively, over which the self-supervision objectives are computed. The STERLING objective composed of the multi-modal correlation LA ( 4/s) and viewpoint-invarianee Lv (1/2 (hg)) objectives are defined as:
LSTERLING = Lvr(Um Vr2) + LMM th)) LVI(Nors Do2) = LvicReg (Weer ve2))
(I)
(Var vi) [Lviceeg (W : Ui) + Sonccee(Voos 12)
Lvieneg is the VICReg loss that is composed of variaace-iivvarince--vvai representation learn- ing objectives, as proposed by Bardes et al. [37]. Given two alternate projected representations Z and Z' of a data sample (in STERLING, Z and Z' are projected representations of the visual and non-visual sensor modalities), the VICReg loss defined as Lvicreg (Z, Z') As (s, ++(((((()) ezZ)) + vlc(Z) + c(z'). Note that while Bardes al. use VICReg to learn representations from visual inputs using artificial image augmentations, in this work, we extend VICReg multi-modal inputs and use real-world augmentations via multi-viewpoint image patches as described in Sec. 3.1. A. ns, and v are hyper-parameters and the functions v, 8, and are the variance, invariance, and covariance terms computed on a mini-batch of projected features. We refer the reader to Bardes et al. [37] for additional details on the individual terms and also define them here for completeness.
The variance term a hinge function defined as v(z) = Σ
,e), where the standard deviation, and the dimensionality of the projected feature space. c is the covari- ance term, defined as o( Σ ((())) where C(Z) the covariance matrix of Z. the 1Ø)
max S(=),e)), y=1
invariance term defined as s(Z = - - z/1l- More details on the individual terms in the
loss function are provided in Sec. 9.5. We apply an 12 norm on the visual and non non-visual features to ensure they are on a hypersphere, which helped improve the quality of learned representations. On mini-batch of data containing paired terrain image patches and IPT observations, we compute the EsterIig loss and update parameters of the two encoder networks and the shared projector network together using Adam optimizer.
3.3 Preference-Alligned Off-Road Navigation
In this subsection, we describe the downstream navigation task of preference-aligned visual naviga- tion that we focus on when evaluating STERLING.
Preliminaries: We formulate the task of preference-aligned terrain-aware navigation moth-planning problem, where the robot operates within a state space S, action space A, and deterministic transition function T S x A - S in the environment. The state space consists of s [xc y, 0, (u). where [x. y, 0] denote the robot's position in SE(2) space, and Φ, denotes the visual features of the terrain at this location. Given a goal location G, the preference-alignee navigation task to reach this goal while adhering to operator preferences over terrains. We assume access sampling-baid based planner, the details of which are provided Supplementary Sec.
Learning the preference utility: Following Zucker et al. [46], we learn the utility function u. 4, + + using human queries. From the predicted terrain features on data samples in our training set, we cluster the terrain representations using k-means with silhouette-scor elbow criterion, and sample candidate terrain patches from each cluster, which is presented to the human operator using GUI. The human operator then provides a full-order ranking of terrain preferences over clusters, which is utilized to learn the utility function u(.), represented by 2-layer MLP. While recovering absolute cost values from ranked preference orders is under-constrained problem, find that this approximation provided by Zucker et al. [46] works well in practice.
5

---

Experiments
In this section, we describe the experiments performed to evaluate STERLING. Specifically, the experiments presented in this section tai- lored to address the following questions:
Vasue
Represencations
Eys
Visual Encoder
Projector
Vay
(1) How effective are STERLING features in comparison to baseline approaches at enabling terrain awareness in off- road navigation?
(lace buato
Robches
P-nor
Ciaai
Tratly Parmr Sadle
Inertia
Bhoode
(Q2) How effective are the proposed STER- LING objectives in learning discrimi- native terrain features in comparison to other representation learning objec- tives?
Proprto
Projector
Uhee AOHOIN Dandin)
Inertis--
Figure 2: Overview of the training architecture in STERLING. Terrain patches V1 and V/2 from different viewpoints of the same location coded as o,, and Φe respectively, and mapped into embeddings V'y and dey- Similarly, iner- tial, proprio, tactile signals are encoded as Φs, and mapped 1/s, Self-supervision objectives Lv for viewpoint-invariance and LM for multi- modal correlation are computed on the minibatch to perform gradient descent.
We investigate Q1 through physical robot ex- periments the task of preference-aligned off- road navigation. We perform quantitative eval- uations in six different outdoor environments, and then further perform a large-scale qualita- tive evaluation by semiiauuton hiking a 3-mile long off-road trail using preference costs learned using STERLING features. To compare various methods, use the success rate of preference alignment tory followed by any algorithm fails to reach the goal, at any time traverses over any terrain that is less preferred than any traversed by the dreontredeemonstratee trajectory, we classify the trial as failure. We additionally investigate Q2 by comparing STERLING against other unsupervised terrain representation learning methods and perform an ablation study on the two STERLING objectives. Additional experiments are provided in Supplementary Sec. 9.2.
metric. If trajec-
Baselines: To perform quantitative evaluations for Q1, compare STERLING with SE-R [8], RCA [7], GANav [19]. geometric-only planning [47], and fully-supervised baseline. SE-R and RCA perform self-supervised learning from unconstrained robot experience to learn terrain representa- tions and traversability costs respectively, making them relevant baselines for this problem. Since there is no open-source implementation geometric-only approach ignores terrain costs (Cterrain) and plans with geometric cost (Cgeom) only, making it relevant ablation the cost formulation for preference-aware planning. GANav² [19] is segmentation- based approach trained the RUGD [16] dataset. We additionally train the fully-supervised baseline in which the terrain cost function is learned end-to-end using supervised leamning from linear extrapolation of operator preferences. GANav and the fully-supervised baseline require supervision via terrain labels to learn and hence serve as references for comparison. We normalize the terrain cost predicted by all methods to between 0 and I for a fair comparison.
RCA, we replicate it to the best of our abilities. The
4.1 Evaluating Terrain-Awareness via Robot Experiments
In this subsection, report experiments to investigate the effecctiveness of STERLING features in enabling terrain awareness during off-road navigation. We quantitatively compare the performance of STERLING with baselines RCA [7], GANav [19], SE-R [8] and the fully-supervised baseline, on the task of greference-aligned navigation. We identify six environments within the university campus, with eight different terrain types, as shown in Fig. For this study, we use the same data col- lected on the robot to train RCA. SE-R, fully- superviede baseline, and STERLING, and the operator

---

Operator Preferences: bush < sazble Fooks grass < molch < pebble sidevack coneict sidekack yellow beicka
red bzicks
Env 1
Env 2
Eni
6
100
Rully-Speeriiee
Annex
STERLING
RCA SEIR
Env
Env
1.7ebble eidewalk 2.Graae
4.Hecble Rocke
3.5ed Beicka
6.Cement aidevalx
7.Huich
B.Yallow Beicks
Figure 3: Trajectories traced by different approaches in environments containing 8 different terrains. The operator preferences are shown above. We see that STERLING navigates in an operator-peeference aligned manner, by preferring cement sidewalk, red bricks, pebble sidewalk, and yellow bricks over mulch, grass, marble rocks, and bush, outper- forming other baselines and performing on-aar with the Fully- Supervised approach.
provides the rankings for all methods during training. Note that the encoder and utility function across all environments and do not retrain/linetune to each environment to prevent environment--peccfiic overfitting.
Fig. shows the operator's 's (first author) terrain preferences for all Envs. to 5, and the performance of baseline approaches, including an operator-demmostratee trajectory for reference. In all environ- ments, we that STERLING navigates in a terrain-aware manner while adhering the operator's preferences. Note that although Fully-Supervised also completes the task successfully, it requires privileged information such as terrain labels during training, whereas STERLING does not require such supervision, and can potentially be used on large datasets containing unlabeled, unconstrained robot experiences. GANav, trained on the RUGD dataset fails to generalize to unseen real-world conditions. RCA uses inertial spectral features to learn terrain traversability costs and hence does not adhere to operator preference. SE-R does not address viewpoint invariance which signifi- cant problem vision-based off-road navigation and hence performs poorly in Envs. and We perform additional experiments in outdoor environment (Env. 6) to study adherence operator preferences, detailed in Supplementary Sec. 9.1.
Table shows the success rate of preference alignment for all approaches all environments, over five different trials. STERLING outperforms other self-supervised baselines and performs on par with the fully-supervised approach. summary, the physical experiments conducted six environments quantitatively demonstrate the effectiveness of STERLING features in enabling terrain awareness during off-road navigation.
4.2 Evaluating Self-Supervision Objectives
In this subsection, we investigate the effectiveness of STERLING at learning discriminative terrain features and compare with baseline unsupervised terrain representation learning methods such as Regularized Auto-Encoder (RAE) and SE-R [8] and large pretrained networks such as a ResNet- 50 pretrained on ImageNet. STERLING uses multi-modal correlation (CMM) and viewpoint in- variance (Lvr) objectives for self-supervised representation learning. whereas, SE-R and RAE use soft-triplet-conrrsttie loss and pixel-wise reconstruction loss, respectively. Additionally, we also perform ablation study the two objectives in STERLING to understand their contributions to learning discriminative terrain features. To evaluate different visual representations, we perform

---

unsupervised classification using k-means clustering and compare their relative classification accu- racies with manually labeled terrain labels. For this experiment, we train STERI.ING, SF-R, and RAF on our training set and evaluate on a held-out validation set. Fig. 4 shows the results of this study. We see that STERLING-features using both the self-supervision objectives perform the best among all methods. Additionally, we see that using a non-contrastive representation learning approach such VICReg [37] within STERLING performs better than contrastive learning methods such as SE-R, and reconstruution-bassed methods such as RAE. This study shows that the proposed self-supervision objectives in STERLING indeed help learn discriminative terrain features.
Environment
Approach
4
(a)
(b)
Geometric-only
0/5
0/5
0/5
0/5
0/5
0/5
5/5
RCA[7]
2/5
4/5
2/5
0/5
175
5/5
075
GANav[19)
5/5
0/5
0/5
5/5
0/5
4/5
5/5
SF-R[[]
1/5
0/5
5/5
1/5
3/5
5/5
4/5
SupervSuee
5/5
5/5
5/5
5/5
5/5
5/5
5/5
STERLING (Ours)
5/5
5/5
5/5
575
575
575
575
Table 1: Success rates of different algorithms on the task of preference-aligned off-road navigation
5 Limitations and Future Work
STERLING requires traversing over terrains in order to learn representations, which may be unsafe in certain situations. Uncertainty-awar safe exploration and exploration focusing on in-
sTorand
formative and diverse terrains for data collec- tion is a promising direction for future work. Extending STERLING features to work on un- structured non-flat environments such as stairs [48] and boulders [49] is another promising di- rection for future work. Extending STERLING by pretraining with large-scale off-road datasets using modern architectures such as transform- ers that are known to scale well with large-scale data an exciting direction for future work.
0,575
Spprrated 10:97
Prageter
52
3.0
Classification Accuracy
Figure 4: Ablation study depicting classification accuracy (value closer to 1.0 is better) from terrain representations learned using different approaches and objectives. The combined objective (VI MM) proposed in STERLING achieves the highest accuracy, indicating that the learned representa- tions are sufficiently discriminative of terrains.
Conclusion
In this paper, we introduce Self-supervised TEr- rain Representation LearnING (STERLING), novel framework for learning terrain repre- sentations from easy-to-collect, unconstrained
(e.g., non-expert), and unlabeled robot experience. STERLING utilizes non-contrastive representa- tion learning through viewpoint invariance and multi-modal correlation self-supervision objectives to learn relevant terrain representations for visual navigation. We show how features learned through STERLING can be utilized learn operator preferences over terrains and integrated within a plan- ner for preference-aligned navigation. We evaluate STERLING against state-of-the-art alternatives on the task of preference-aligned visual navigation performs other methods and performs par with fully-supervised baseline. We additionally per- form a qualitative large- scale experiment by successfully hiking a 3-mile-long trail using STERLING, demonstrating its robustness to off-road conditions in the real world.
Spot robot and find that STERLING out-

---

Acknowledgments
This work has taken place in the Learning Agents Research Group (LARG) and Autonomous Mo- bile Robotics Laboratory (AMRL) at UT Austin. LARG research is supported in part by NSF (CPS-1739964, IIS-1724157, NRI-1925082), ONR (N00014-18-2243), FLI (RFP2-000), ARO (W91INF19-2-0333), DARPA, Lockheed Martin, GM, and Bosch. AMRL research is supported in part by NSF (CAREER2046555, IIS-1954778, SHF-2006404), ARO (W911NF-19-2- 0333, 1911NF-21-20217) DARPA (HR001 120C0031), Amazon, JP Morgan, and Northrop Grumman Mission Systems. Peter Stone serves the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and ap- proved by the University of Texas Austin in accordance with its policy on objectivity in research.
References
[1] T. Wang, B. Chen, 7. Zhang, H. Li, and M. Zhang. Applications of machine vision in agri- cultural robot navigation: A review. Computers and Electronics in Agriculture, 198: 107085, 2022.
[2] Agility robotics launches next generation of digit: World's first human-centric, multi-purpose robot made for logistics work, 2023. URL https://aagllityroociccc com/news//0//// ruture-robotics-111e
[3] X. Xiao, J. Dufek, and R. R. Murphy, Autonomous visual assistance for robot operations using a tethered uav, 2019.
[4] H. Karnan, E. Yang, G. Warnell, J. Biswas, and P. Stone. Wait, that feels familiar: Learning to extrapolate human preferences for preference aligned path planning, 2023.
[5] E. Yang, H. Karnan, G. Warnell, P. Stone, and J. Biswas. Wait, that feels familiar: Learning to extrapolate human preferences for preference-aligee aligned path planning. In ICRA2023 Workshop on Pretraining for Robotics (PT4R), 2023.
[6] K. Sikand, S. Rabiee, Uccello, X. Xiao, G. Warnell, and J. Biswas. Visual representation learning for preference-aware path planning. In 2022 International Conference on Robotics and Automation (ICRA), pages 1300-11309 IEEE, 2022.
[7] X. Yao, J. Zhang, and J. Oh. Rea: Ride comfort-aware visual navigation via self-supervised learning. In 2022 JEFF/RS International Conference on Intelligent Robots and Systems (IROS), pages 7847-7852. IEEE, 2022.
[8] Zürm, W. Burgard, and A. Valada. Self-supervised visual terrain classification from unsuper- vised acoustic feature learning. IEEE Transactions Robotics, 37(2):466 2020.
[9] H. Karnan, S. Sikand, Atreya, S. Rabiee, X. Xiao, G. Warnell, Stone, and J. Biswas. Vi-ikd: High-speed accurate off-road navigation using learned visual-inertial inverse kinody- namies. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3294-3301. IEEE, 2022.
[10] Atreya, H. Karnan, K. S. Sikand, X. Xiao, Rabiee, and J. Biswas. High-speed accurate robot control using learned forward kinodynamics and non-linear least squares optimization, 2022.
[11] H. Karnan, A. Nair, X. Xiao, G. Warnel, S. Pirk, A. Toshev, J. Hart, Biswas, and P. Stone. Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. IEEE Robotics and Automation Letters, ((4):11807-1184) 2022.
[12] S. Desai, H. Karnan, J. P. Hanna, G. Warnell, and Stone. Stochastic grounded action trans- formation for robot learning in simulation. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 6106-6111. IEEE, 2020.

---

[13] J. P. Hanna, S. Desai, H. Karnan, G. Warnell, and P. Stone. Grounded action transformation for sim-to-real reinforcement learning. Machine Learning, 110(9):249--249, 2021.
[14] H. Karman, S. Desai, J. P. Hanna, G. Wamell, and P. Stone. Reinforeed grounded action trans- formation for sim-to-real transfer. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4397 4402. IEEE, 2020.
[15] S. Desai, I. Durugkar, H. Karnan, G. Warnell, Hanna, and P. Stone. An imitation from observation approach to transfer learning with dynamics mismatch, 2020.
[16] M. Wigness, S. Eum, Rogers, D. Han, and H. Kwon. rugd dataset for autonomous navigation and visual perception in unstructured outdoor environments. In 2019 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 5000-5007. IEEE, 2019.
[17] Jiang, Osteen, M. Wigness, and Saripalli. Rellis-3d dataset: Data, benchmarks and analysis, 2020.
[18] S. Sharma, L. Dabbiru, T. Hannis, G. Mason, D. W. Carruth, M. Doude, C. Goodin, C. Hudson, Ozier, J. Ball, al. Cat: Cavs traversability dataset for off-road autonomous driving. IEEE Access, 10:24759-24768, 2022.
[19] T. Guan, D. Kothandaraman, R. Chandra, A. J. Sathyamoorthy. K. Weerakoon, and
D. Manocha. Ga-nav: Efficient terrain segmentation for robot navigation in unstructured out- door environments. IEEE Robotics and Automation Letters, 7(3):8138 111181111111, 2022.
[20] S. Triest, M. Sivaprakasam, S. J. Wang, W. Wang, A. M. Johnson, and S. Scherer. Tartandrive: large-scale dataset for learning off-road dynamics models, 2022.
[21] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to end learning for self-driving cars, 2016.
[22] Y. LeCun, U. Muller, J. Ben, E. Cosatto, and B. Flepp. Off-road obstacle avoidance through end-to-end learning. In Proceedings of the 18th International Conference on Neural Informa- tion Processing Systems, NIPS 05, page 739-746, Cambridge, MA, USA, 2005. MIT Press.
[23] M. Wulfmeier, P. Ondruska, and I. Posner. Maximum entropy deep inverse reinforcement learning, arXiv preprint arXiv:1507.044 007.04888, 2015.
[24] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. A. Theodorou, and B. Boots. Imitation learning for agile autonomous driving. The International Journal of Robotics Research, 39 (2-3):286-300) 30266-3022222 2020.
[25] G. Kahn, Abbeel, and S. Levine. Land: Learning to navigate from disengagements. JEEE Roboties and Automation Letters, 6(2):1872-1879, 2021.
[26] G. Kahn, P. Abbeel, and S. Levine. Badgr: An autonomous self-supervised learning-based navigation system. IEEE Robotics and Automation Letters, 6(2):1312-119,, 2021.
[27] H. Karnan, E. Yang, D. Farkash, G. Warnell, J. Biswas, and P. Stone. Self-supervised ter- rain representation learning from unconstrained robot experience. In ICRA2023 Workshop on Pretraining for Robotics (PT4R), 2023.
[28] Y. LeCun, U. Muller, J. Ben, E. Cosatto, and B. Flepp. Off-road obstacle avoidance through end-to-end learning. In Proceedings of the 18th International Conference on Neural Informa- tion Processing Systems, NIPS 05, page 739-746. Cambridge, MA, USA, 2005. MIT Press.
[29] X. Xiao, B. Liu. G. Warnell, and P. Stone. Motion planning and control for mobile robot navigation using machine learning: survey, 2020.
10

---

[30] H. Karnan, G. Warnell, X. Xiao, and P. Stone. Voila: disual-observation-oo only imitation learn- ing for autonomous navigation, 2021.
[31] X. Xiao, Z. Xu, Z. Wang. Y. Song, G. Warnell, P. Stone, T. Zhang, Ravi, G. Wang, H. Kar- nan, J. Biswas, Mohammad, L. Bramblett, Peddi, N. Bezzo, Z. Xie, and P. Dames. Autonomous ground navigation in highly constrained spaces: Lessons learned from the barn challenge at iera 2022, 2022.
[32] F. Schilling, X. Chen, J. Folkesson, and P. Jensfelt. Geometric and visual terrain classification for autonomous mobile navigation. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2678-2684. IEEE, 2017.
[33] M. Wigness, J. G. Rogers, and L. E. Navarro-Serment Robot navigation from human demon- stration: Learning control behaviors. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1150-1157. IEEE, 2018.
[34] C. A. Brooks and K. D. lagnemma. Sepe-supe classification for planetary rover terrain sensing. In 2007 IEEE aerospace conference, pages 1-9. IEEE, 2007.
[35] A. Loquercio, A. Kumar, and J. Malik. Learning visual locomotion with cross-modal supervi- sion, 2022.
[36] H. Karnan, F. Torabi, G. Warnell, and Stone. Adversarial imitation learning from video using a state observer. In 2022 International Conference on Robotics and Automation (ICRA), pages 2452-2458. IEEE, 2022.
[37] A. Bardes, J. Ponce, and Y. LeCun. Vicreg: Variane-iivaraane--ovariaaa regularization for self-supervised learning, arXiv preprint arXiv:2105.04900 2021.
[38] A. J. Sathyamoorthy, K. Weerakoon, T. Guan, J. Liang, and D. Manocha. Terrapn: Unstruc- tured terrain navigation using online self-supervised learning, 2022.
[39] L. Wellhausen, A. Dosovitskiy, R. Ranftl, K. Walas, C. Cadena, and M. Hutter. Where should i walk? predicting terrain properties from images via supersuue learning. IEEE Robotics and Automation Letters, 122:1509-1 2019.
[40] M. G. Castro, S. Triest, W. Wang, J. M. Gregory, F. Sanchez, J. G. R. I. au2, and S. Scherer. How does it feel? self-supervised costmap learning for off-road vehicle traversability, 2023.
[41] S. Triest, M. G. Castro, P. Maheshwari, M. Sivaprakasam, W. Wang, and S. Scherer. Learning risk-aware costmaps via inverse reinforcement learning for off--rod navigation, 2023.
[42] E. Chen, C. Ho, M. Maulimov, C. Wang. and S. Scherer. Learning-on-thers the-drive: Self- supervised adaptation of visual offroad traversability models, 2023.
[43] L. Wellhausen, R. Ranfil, and M. Hutter. Safe robot navigation via multi-modal anomaly detection. IEEE Robotics and Automation Letters, 5(2): 1333, apr 2020. doi:10. 109//ra. 2020.2967706. URL https: //tp://oo...oooo ottp:/...//1/1..1.0....1.. http://....19922211.. 2020. 2967706.
[44] J. Frey, M. Mattamala, N. Chebrolu, C. Cadena, M. Fallon, and M. Hutter. Fast traversability estimation for wild visual navigation, 2023.
[45] D. Pathak, P. Agrawal, A. A. Efros, and Darrell. Curiosity-driven exploration by self- supervised prediction, 2017.
[46] M. Zucker, N. Ratiff, M. Stolle, Chestnutt, J. Bagnell, G. Atkeson, and Kuffner. Optimization and learning for rough terrain legged locomotion. The International Journal of Robotics Research, 1002):155-191,000 2011.

---

[47] J. Biswas. Amrl autonomy stack. 2013.
[48] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62), jan 2022. doi: 10.1 12///ccrrootiissabk2222.
[49] A. Datar, C. Pan, M. Nazeri, and X. Xiao. Toward wheeled mobility on vertically challenging terrain: Platforms, datasets, and algorithms, 2023.
12

---

7 Data Collection
In all experiments, we use a legged Boston Dynamics Spot robot and collect robot experi- ences on eight different types of terrain around the university campus that we labeled as mulch, pebble sidewalk, cement sidewalk, grass. bushes, marbled rock, yellow bricks, and red bricks. The data is collected through human teleoperation (by the first and second authors) such that each trajectory contains unique terrain throughout, with random trajectory shapes. Note that STERLING does not require a human expert to teleoperate the robot to collect robot experience nor does it require the experience to be gathered on a unique terrain per trajectory. We follow this data collection approach since it is easier to label the terrain for evaluation purposes. STERLING can also work with random trajectory lengths, with multiple terrains encountered along the same trajectory, without any semantic labels such terrain names, and any navigation policy can be used for data collection. We record 8 trajectories per terrain, each five minutes long, and use trajectories for training and the remaining for validation.
8 Sampling-hased Planning
Sampling-based planning: We assume access to receding horizon sampling based motion planner with a fixed set of constant-curvate ares (To, (To,Ti:- I....,, I hs r E S. which solves
Brrd' (dbV) Oiserrution
for the optimal are T [(T
minimizing the objective function J[T, GT, GT - R+. For the task of preference- aligned off-road navigation, we assume the ob- jective function is composed of two components Igeom (T', G) and Jterrain (T), and can de- fined as JTT, a) Fterrainn (I'). Jgeom G) is the geometric cost that deals with progress towards the goal G and avoiding geometric obstacles, whereas Jterrain (T) is the terrain cost associated with preference-alignment. We utilize the geomet- ric cost as defined in AMRL'S graph navigation stack 3. The multiplier a E [0, 1] trades off relative contributions of the geometric and rain preference components of the path planning objective. A ID time-optimal controller trans- lates the sequence of states in the optimal tra- jectory m* to a sequence of receding horizon ac- For given are T {50, .1,..., ......,8y. ). such that state 8g is closest to the robot, the ceraain-preferee cost can be com- puted follows.
Resresiniztion Pv
Vitial
Profer
Visual Encader
geoom (T, (T,G) G) + (1
(Prwnn))
nurction
G(x)
Cost
Ireed cred esed CoeeMap
Figure
overview of the cost infer- ence process for local planning at deployment. The constant-curvature ares (yellow) are over- layed the BEV image, and the terrain cost Tterrain (T) is computed on patches extracted along all ares. White is high cost and black is low cost.
tions (ao, (co,at,...,.x))
(C(x(f.(x)))
Tterrain (T)
Σ U,
N+
The function f.).) maps from RGB space of a visual patch terrain v; at specific state s;, to its visual representation Φ. D,. For instance, fa can be the visual encoder learned using STERLING, as described in Section 3.2. The utility function u(.) maps the visual representation de of patch of terrain to real-valued utility of preferences. We follow the utility function formulation of Zucker al. [46] and assume the terrain preference cost follows multiplicative formulation such that given
13

---

Cenmetricconly
RCA
STERLING
Full-Superviedd
Human Demo
GANAe
ft
START
START
GOAL
GOAL
Cideve3k Zeah
Oudraalc grass IMIS
Env (a)
Env
Figure 6: Trajectories traced by different approaches for the task of preference- alligne off-road navigation. Shown here are two different preferences expressed by the operator the same environment- 6 (a), sidewalk is more preferred than grass which is more preferred than bush, and in (b), grass and sidewalk are equally preferred and bush is least preferred. We see that without retraining the terrain features, in both cases (a) and (b), STERLING optimally navigates to the goal while adhering to operator preferences.
utility value E IR+, the traversability cost is C(x) c= The discount factor y weighs the terrain cost proportional to its proximity to the robot. We set to 0.8, which we find to work well in practice.
Planning at Deployment: Fig. 5 provides an overview of the cost inference process for local
planning at deployment. To evaluate the terrain cost Jterrain (T) for the constant-curvature ares, overlay the arcs on the bird's eye view image, extract terrain patches states along the arc, and compute the cost according to Eq. 2. We compute the visual representation, utility value, and terrain cost of all images at once a single batch inference. Since the visual encoder and the utility function are relatively lightweight neural networks with about 0.5 million parameters, we are able to achieve real-time planning rates of 40 Hz using a laptop-grade Nvidia GPU.
Additional Experiments
In this section, we detail additional experiments performed to evaluate STERLING-features against baseline approaches.
9.1 Preference Alignment Evaluation
In addition to the evaluations STERLING-features with baseline approaches in five environments as shown in Sec. 4, utilize Env. to further study adherence operator preferences. We hypoth- esize that the discriminative features learned using STERLING is sufficient to learn the preference cost for local planning. To test this hypothesis, in Env. 6 containing three terrains shown in Fig. 6, the operator provides two different preferences 6(a) and 6(b). While bush is the least preferred in both cases, in 6(a), sidewalk is more preferred than grass and in 6(b), both grass and sidewalk are equally preferred. We see in Fig. 6 that using STERLING features, the planner is able to suffi- ciently distinguish the terrains and reach goal while adhering to operator preferences. Although SE-R [8] adheres to operator preference in 6(b), incorrectly maps grass bush assigning higher cost and taking longer route reach the goal. On the other hand, RCA [7] fails adhere to operator preferences since it directly assigns traversability costs using inertial features.
9.2 Large-Scale Qualitative Evaluation
In this subsection, we perform a qualitative evaluation of STERLING by reporting large-scale study of semi-autonomously hiking a 3-mile-long off-road trail using the Spot robot.
We train STERLING using unconstrained robot experience collected within the university campus and train the preference utility function using operator-provided preferences: marble rocks grass dirt cement. The task is to navigate the trail without global map while adhering to operator preferences all times. Since we do not use global map, visual terrain awareness is
14

---

Start & End
1000
Figure 7: A large-scale qualitative evaluation of STERLING on 3-mile outdoor trail. STERLING features successfully complete the trail with only two manual interventions (shown in red).
necessary to navigate within the trail and avoid catastrophic events such falling into the river next to the trail. We set a moving goal of six meters in front of the robot, updated every second. While the robot navigates autonomously, the operator walks behind the robot and takes manual control only to correct the robot's path during forks, or to yield to incoming pedestrians and pets. The attached supplementary video shows the rohot navigating the trail successfully while avoiding less preferred terrains. The robot needed two manual interventions while traversing along the trail. Fig. shows the 3-mile trajectory traced by the robot and the two failure cases that required manual intervention. This large-scale qualitative experiment demonstrates the reliability of STERLING during real-world off-road deployments.
9.3 Experiments on a Wheeled Mobile Robot
STERLING is intended as a general algorithm to learn relevant terrain representations for off-road navigation. Towards demonstrating the versatility of STERLING to heing applied to robots of differ- ent morphology, we conduct two additional experiments on the Clearpath Jackal, a wheeled mobile robot.
Learning Representations Wheeled Robots: We utilize unconstrained data collected on the Jackal consisting of multi-modal visual and inertial sensor data and learn terrain representations using STERLING followed by a utility function of operator preferences. Fig. (STERLING-Jackal) shows the path traversed by the Jackal in Env. 6, following the human preference sidewalk > grass > bush. This experiment demonstrates the applicability of sterliNG on wheeled robots with inertial sensors, as against legged robots that have access to additional sensors such as joint encoders and tactile information.
Zero-Shot Cross-Morphology Transfer: In noteworthy experiment to evaluate the transfer- able property of terrain representations across robot morphologies, we utilized the visual encoder trained on data from the legged Spot robot and applied it on the wheeled Jackal robot without additional fine-tuning. Fig. 8 (STERLING-Spot) showcases the Jackal's trajectory, leveraging STER- LING representations learned from Spot's data, and adhering to the operator's terrain preference: sidewalk grass > bush. Fig. 10 shows costmaps generated using STERLING features, used by the sampling-base planner to navigate in operator-aligned manner. This demonstrates STER-
15

---

STERLING-Spor
STERLING-lacka
START
GOAL
Fidexe1k g2255 > bush
Figure 8: Experimental study of STERLING Clearpath Jackal-a wheeled mobile robot in En- vironment 6. STERLING-Spot shows the trajectory traced using STERLING trained on data collected on the Spot, deployed zero-shot the Jackal robot, whereas STERLING-Jackal shows the trajectory traced by STERLING trained on data collected on the Jackal, deployed also on the Jackal robot. In both experiments, see the robot reach the goal successfully while adhering human operator's preferences over terrains (sidewalk > grass bush).
T'L va
Figure 9: Clearpath Jackal, a wheeled robot navigating using STERLING features trained on uncon- strained data collected on the Jackal robot (STERI.NNGJaccal) We see here in Env. that the robot reaches the goal while adhering to operator preferences Sidewalk >> Grass >> Bush. This experi- ment demonstrates the versatility of STERLING in being applied robots of different morphology.
LING'S capability generalize across diverse robotic platforms, emphasizing its adaptability and broad applicability.
Fig. 9 shows a third-person view of the deployment of STERLING-Jackal Env. In both ex- periments above, we see the Jackal robot reaches the goal successfully while adhering to human operator preferences, terrain-aves aware manner, highlighting STERLING's adaptability regardless of robot morphology.
9.4 On the Efficacy of Multi-Modal Data Over Vision Alone
While it might seem that visual cues are sufficient for distinguishing terrains, evidenced in Fig.3, the reality is more complex. Variations in lighting, shadow, color, texture, and other artifacts may lead to inconsistent representations for the same terrain type and can render visually distinctive terrains deceptively similar. For instance, while six different visual patches of the terrain "sidewalk" as shown in Fig. 11 might each exhibit unique visual characteristics because of these variations, they all denote the same terrain category and evoke similar mertial-priopriiotactiee (IPT) response on the robot (feel similar to the robot). Solely relying vision may lead to overlooking underlying commonalities between terrains, resulting in inconsistent terrain representations.
16

---

Figure 10: Visualizing the costmaps from the Jackal robot when traversing Env 6., trained using data from the Jackal (STERLING-JaCkal)
Six distinct instances of sidewalk terrain, showcasing the variability in visual appear- ance due to factors such as lighting, texture, shadows, and other artifacts. Despite the visual differ- ences, each patch represents the same terrain type.
Figure 11:
Another concrete example is the scenario of fallen leaves. A sidewalk, a grass patch, and forest trail could be covered with fallen leaves, making them visually similar. However, underneath those leaves, the actual terrain properties and the robot's interaction with them vary significantly. While the leaves might visually mask the terrain differences, the robot would feel different terrain responses when moving over them due to differences in underlying ground properties.
Furthermore, visual similarity is not a conclusive indicator of identical terrains. Consider four im- ages as case in point, as shown in Fig. 12. Though the first two and the last two images might seem visually similar, they represent distinct terrains: the first image depicts "bush", the second and third denote "grass", and the fourth is "sidewalk". These three terrains induce different inertial, proprioceptive, and tactile responses in robot. Thus, the mere semblance of appearance does not capture the relevant features of terrain.
STERLING's approach integrating additional modalities allows for more precise terrain identifi- cation by accounting for these subtleties. By considering variations and similarities among terrains across different modalities, we ensure relevant terrain representations for off-road navigation. In all examples shown in Figs. 11 and 12, STERLING correctly associates the samples with the right cluster for each terrain.
9.5 Experimental Setup and Methodological Details
In this subsection, outline the specifics of our experimental setup, detailing hyperparameters, architectural decisions, data, and sensory inputs. These insights ensure clarity and reproducibility of our experiments.
In all experiments in STERLING, including the baselines RCA and SE-R, we shallow 4-layer CNN with kernel size of 3 and stride of 1. Our choice of shallow 4-layer CNN driven by the specific need for lightweight and efficient model that could operate in real time at 40Hz
17

---

Bush
Sidews
Figure 12:
collection of four terrain patches, illustrating the challenge of terrain representation learning based solely on visual cues. From left to right: bush, grass, grass, sidewalk. Despite visual similarities (and differences), the terrains can elicit different non-visual IPT responses on a robot.
Table 2: Hyperparameter Choices for STERLING Experiments
Hyperparameter
Value/Range
Learning Rate
3 x 10
Batch Size
128
Number of Epochs
50
Optimizer
Adam
Weight Decay
5 10
Activation Function
ReLU
Kernel Size
3
Stride
on laptop GPU, requirement that was effectively met with this simple architecture of 0.25 M parameters, which we found was sufficient for the problem. While modern architectures like vision transformers / Mobile ViT could be applied with larger scale data, the primary concern was real-time performance and compatibility with our robot's hardware. Our experiments and results demonstrate that the selected architecture was sufficient for the purpose, and do not find evidence that our approach's effectiveness is constrained by this architectural choice.
To train STERLING, SE-R, and RCA, we used a total of 1 117,644 data samples for all terrains com- bined. Example raw time-series sensor data shown Fig. 14. Each data sample contains a minimum of visual patches and maximum of 20 visual patches of the same location from mul- tiple different viewpoints from which we randomly sample 2 patches per location during training. We convert the time-series IPT signals into their corresponding Power Spectral Density PSD values. Power spectral density describes the power of a signal across different frequency components. To compute this, we perform fast-fourier transform over the time-series signal imerrial/proorriccettvv/// lactile) and compute the PSD defined as (SD(W) = E[| (w) across each frequency component w.
On the Spot robot, we use a VectorNav IMU record the inertial signals (angular velocities in the and y-axis and linear acceleration in z-axis) 200Hz, the joint angles and velocities of the legged robot, referred proprioceptive feedback in this work recorded at 25 Hz, and the feet contact measurements (contact booleans and estimated feet depth from ground) collectively referred to as tactile feedback in this work are recorded at 25Hz. An Azure kinect camera is mounted on the Spot, used for visual sensing of the terrain. On the wheeled Clearpath Jackal robot, we use a 7ed2 camera for visual sensing, and utilize the internal IMU sensor for inertial feedback. Fig. 13 depicts the two robots, sensor mounts, and other sensors used in this work.
Note that in all experiments, to prevent overfitting to specific environment, pretrain the visual encoder and utility function once and deploy them in all environments. The encoders and utility functions are not being retrained/finetuned per environment, including the large-scale outdoor trail.
More details on the loss function: STERLING extends VICReg algorithm initially proposed by Bardes al. [37] for self-supervised learning from vision-only data. While the foundational work by Bardes al. uses image augmentations to learn visual representations in self-supervised way, we utilize images from multiple viewpoints and multi-modal inputs such as vision, inertial, proprio-
18

---

Ouster
Onboard IMU
Velodyne LIDAR
VectorNaav IMU
Zed Camera
Azure Kinect (Vision)
Joint Encoder
(Propricception)
Feet Depth Estimate (Tactile)
JACKAL Side
View
SPOTT
Figure 13: Figure depicting the legged Spot and the wheeled Jackal robot, along with other sensors used in this work.
Deat
Time
Visual depiction of 1-second of time-series features of inertial- pwwpppoccppioonttaceeeee tactile data and visual patches of three representative terrains (Bush, SideWalk and Grass).
Figure 14:
ceptive, tactile using novel formulation learn relevant terrain representations in self-supervised way. The STERLING loss based VICReg is defined in Section 3.2. The VICReg loss is defined as Lvickeg (2, 3") = As(Z, Z') slv(2 Z) + v(Z') + vle(Z) + c(Z')), where X, and are hyper- parameters. We use the values 25.0, 25.0. 1.0 for these hyperparameters respectively, suggested by Bardes et al. [37]. s(Z,2' denotes the invariance between the two inputs. In STERLING, this computed across the two image patches from different viewpoints, and also between the visual and non-visual (IPT) projections. u()) denotes the variance across the batch dimension, which we compute for the projections of individual patches and the IPT signals. c(2) denotes the covariance across the feature dimension which encourages distinct, non-correlative features which again compute for the projections of individual patches and the IPT signals. We refer the reader to Bardes et al. [37] for additional details regarding individual terms in the loss function. We compute the loss provided in Eq. across a mini-batch of samples and use the Adam optimizer for gradient-based optimization of the visual encoder, IPT encoder and the common projector network.
9.6 Visualizing the terrain representations learned using STERLING
Fig. 15 depicts t-SNE visualization of terrain representations learned using STERLING. Individual patches are color-coded by their ground truth semantic terrain label. We see that STERLING learns relevant features for terrains, given their unique clustering in this latent space.
9.7 Visualizing the costmaps
Fig. 16 shows cost visualizations of baseline approaches- RCA [7]. SE-R [8], GANav [19] and Fully- Supervised in comparison with STERLING. Fig. 16 shows that RCA and SE-R exhibit issues with
19

---

T-SNE plot of terrain representations learned using STERLING
Marble Rocks
Yellow Bricks
Grass
Cement Sidewalk
Pebbled Sidewalk
Red Bricks
Bush
Figure 15: t-SNE visualization of terrain representations learned using STERLING. Each data point represents a terrain example, color-coded by its ground truth label. The clustering of colors show- cases the efficacy of STERLING in capturing meaningful and distinctive terrain features.
visual artifacts due to homography transformations. GANav, trained on the RUGD [16] dataset, fails generalize novel real-world situations. In contrast, costmaps from both the fully-supervised model and STERLING efficiently guide planning, demonstrated by quantitative results in Section 4 and results in behaviors that align with operator preferences, prioritizing sidewalks over terrains like rocks or bushes.
9.8 Generalization to Unseen Terrains
During autonomous off-road navigation, generalization to novel terrains paramount. Although difficult comment on generalizability, we document an instance during the large-scale deployment where STERLING navigates around an unseen terrain "water puddle", as shown Fig. 17.
20

---

RCA
SER
Full-Supernied
STERLING
Figure 16: Comparative visualization of the costmaps generated by STERLING (this work) and other baseline algorithms (RCA [7], SE-R [8], GANav [19], Fully-Supervised) for given scene. Paired with each costmap is bird's-eye view image of the corresponding terrain. In the costmaps, white regions indicate high traversal cost, black signifies low cost, and areas red are ignored or non- observable regions. We see that compared to other approaches, using STERLING features results in costmaps that align with operator preference of Sidewalk Rocks Bush.
2
3
5
6
8
Figure 17: STERLING navigating around an unfamiliar terrain, specifically "water puddle", during the qualitative 3-mile rof- deployment.
21


Reviews:
[{"review_id": "yv19EF4DJ2b", "reviewer": "Reviewer_f3rt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors propose a novel terrain representation learning method based on multi-modal self-supervised learning. Self-supervised TErrain Representation LearnING (STERLING) learns terrain-aware representation with a non-contrastive objective on visual and inertial-proprioceptive-tactile representations extracted from unlabelled robot experience. Features learned by STERLING are evaluated on human preference-aligned navigation, and the experiments along with a qualitative demonstration show that the proposed method outperforms several baselines in navigating through various scenarios with specified preference adherence.", "summary_of_recommendation": "The proposed approach STERLING is a novel non-contrastive, non-reconstructive self-supervised learning framework that learns more general representation than prior work for off-road navigation. Comparison of STERLING against several baselines demonstrates the effectiveness of STERLING features for the downstream task of preference-aligned navigation while the qualitative evaluation results are impressive. However, the proposed approach also has some limitations, and the paper has some missing details on analyzing experiment results as well as the method’s overall limitations. Concerns raised by the reviewer are described in Questions for Rebuttal. Lastly, the authors are suggested to consider a code release, and/or adding details on the methodology for reproducibility.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. L139: The paper does not describe in detail how image patches from various timestamps are aligned to extract several viewpoints for a particular location $s_{k}$. Is this done by combining depth and odometry?\n2. L143: Could the authors provide more detail on power-spectral-density representation or cite relevant work? It seems this preprocessing step may be challenging for other researchers to follow/reproduce if unfamiliar with the topic.\n3. Given that the set of input sensor measurements used for STERLING is similar to [19], would it have been possible to include this work as another baseline? (I would suspect this method would fail to capture human preferences similar to RCA).\n4. An open question is the generalizability of features learned by STERLING. Would STERLING still be able to perform the evaluated scenarios if its data was less diverse? Based on the information from the supplementary, it seems like the data was collected carefully to ensure the robot traverse over a diverse set of terrains.\n5. Sec. 4.1 mainly describes the reasons for the failure of RCA and SE-R. Why does GANAV fail? The method seems to perform fairly well on environments 1, 4, and 6 but fails on the rest. Is the model experiencing a generalization issue?\n6. Table 1 shows STERLING surpassing all methods with a 100% succession rate, while the qualitative evaluation required two manual interventions. Could the authors provide details on what these failures were, and whether they are related to the performance of STERLING?\n7. Besides uncertainty-aware safe exploration, explorations that focus on discovering diverse and informative trajectories also seem quite important. The data collection process involved teleoperation “such that each trajectory contains a unique terrain throughout,” which perhaps is a strong prior. The authors may consider including this as another potential limitation or discuss this point.\n8. Was VICReg chosen as the non-contrastive objective for a specific reason? Could SimSiam[1] or BYOL[2] also be used here? It may be beneficial for authors to briefly describe their basis in choosing VICReg. The authors are also suggested to consider adding definitions of $\\lambda, s, \\mu, v, c$ from $\\mathcal{L}_{VICReg}$ for completeness of the paper.\n9. L200 Typo: Sec.6 instead of 7.\n10. L219: Authors may consider adding a brief description of what the geometric cost includes.\n11. L290: Typo: Sec. 8.1 instead of 9.1.\n\n\n[1]Chen, Xinlei, and Kaiming He. \"Exploring simple siamese representation learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2]Grill, Jean-Bastien, et al. \"Bootstrap your own latent-a new approach to self-supervised learning.\" Advances in neural information processing systems 33 (2020): 21271-21284.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Post-rebuttal comment**\nThe authors provided a modified version of the paper following reviewer feedback, and I believe it now presents a more convincing claim of STERLING's capabilities for learning generalizable terrain features. I'm confident that most of my original concerns have been addressed in the modified version. I suggest the authors take their time until the final paper submission to ensure good readability as the paper now contains much more details and experimental results than before.  Overall, I am willing to raise my rating from weak accept to strong accept.\n\nRegarding the generalization of STERLING, additional tests on Jackal showed STERLING may generalize to other robot platforms, while the evaluation of the 3-mile test run showed some generalization capabilities to unseen features, while the two interventions may also be seen as failures due to unseen features. The authors should try to mention the two failures to clarify this point in Supplemental Section 9.8.\n\n--\nThe paper is overall easy to follow and demonstrates impressive results but holds some limitations. See the strengths and weaknesses below.\n\n**Strengths:**\n1. The authors were able to use the proposed approach to navigate a 3-mile-long trail with two manual interventions. They quantitatively showed improvement over several baselines in various scenarios as well, which is overall an encouraging demonstration of the effectiveness of the method.\n2. Evaluation of STERLING is done with quantitative comparison against GANAV, SE-R, and RCA, which is a diverse set of various prior works on supervised and self-supervised traversability estimation. Against these baselines, STERLING shows better performance in real robot experiments.\n3. STERLING relies on self-supervision with some manual preference ranking to generate traversability cost, which is more annotation-efficient than curating annotated datasets. It’s also to be noted that the manual ranking is only done in the context of finetuning STERLING features to a downstream task, and the feature learning itself does not require any annotation. Furthermore, the framework learns more general features that allow downstream tasks compared to prior methods that directly predict traversability.\n4. STERLING also does not heavily rely on the actual behavior performed during data collection i.e. it can learn from non-expert demonstrations, and therefore can learn from data that is easier to collect.\n\n**Weaknesses:**\n1. Experiments are conducted on relatively simple environments (all six environments are within the university campus with mostly flat terrain). Traversability estimation for off-road navigation would likely require navigating more complex, uneven terrain most of the time. STERLING also uses images projected to BEV with a flat-plane assumption. While this is a reasonable choice in order to focus on the task of preference-aligned navigation, in the context of traversability analysis this is a significantly limiting assumption.\n2. One drawback to STERLING is that it focuses on learning from only positive samples i.e. it is only able to learn features from locations it traveled on. It is difficult to comment on the generalizability of the features with respect to new environments and/or new terrain. It seems then that STERLING would require data collection with some prior e.g. novel terrain discovery, and may not perform so well with random experiences.\n3. The method requires multi-modal robot experience collected with various sensors, which can be a limitation compared to uni-modal methods such as GANAV.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "tpcg7-JvOTn", "reviewer": "Reviewer_jqM5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a method that enables a robot to learn terrain representation in a self-supervised fashion from unconstrained experience. The key idea is to leverage multi-modal sensor data (color, proprioceptive and tactile) for representation learning. It uses VICReg framework that learns a joint embedding over multiple modalities. For planning, it learns a utility function that maps embeddings to costs from human preferences, and then combines it with a geometric cost with a receding horizon planner. Experiments show that the proposed approach adheres to human preferences better than the baselines, while not requiring expert demonstrations during training.", "summary_of_recommendation": "This seems a promising approach of learning terrain representations using non-expert data. The experiments are thorough and convincing. I vote for weak accept. But it would be great if the authors could address my questions since they are important to understand the effectiveness of the approach on real data.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why does vision alone not produce good enough terrain features? In Figure 3, it seems that different terrains have quite distinctive appearances. \n- The model uses a shallow CNN. Is there any reason not to use a better vision backbone? In the supplementary, it mentions that a laptop GPU is used for inference, which should be able to run modern neural nets in real time. This may also be related to the first question about why vision alone does not produce good terrain features.\n- Since the proposed approach requires collecting paired sensor data, it cannot leverage large amounts of image data that is readily available. I wonder how a vision-only model would work in this context. For example, using features from DINO, how much accuracy can you achieve? And how much multi-modal sensor data would help? This does not diminish the benefit of using multi-modal sensor data, but it would help us to understand where multi-modal sensor is helping.\n- SE-R does not address viewpoint invariance. Would it be possible to add viewpoint invariance to SE-R? Would it perform comparably to STERLING?\n- The supplementary mentions that 4 trajectories for training, each being 5 minutes long. How many total frames were used for training? It will be good to show the four trajectories along with the example sensor data as well.\n- When you compare with the baselines, did you use the same network architecture and training data?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n- Using VICReg for learning terrain representation is clever and makes a lot of sense, given that the strength of VICReg is that it can learn from multi-modal inputs.\n- The experiment is thorough with comparison to multiple baselines.\n- The paper is well-written and easy to follow.\n\nWeaknesses\n\n- The paper misses some important details. \n    - Since the approach is about using multiple sensor modalities, it is important to visualize the different modalities and discuss how they are correlated for different types of terrains. Visualizing the embedding space would also be helpful.\n    - There is no visualization of the costmap. There is one figure that shows the BEV RGB projection and the costmap in the supplementary, but it will be more informative to compare the costmaps produced by different methods. This would help the readers to better appreciate the effectiveness of the proposed approach. \n- The proposed approach uses a rather simple CNN for processing images. It is unclear how a modern vision backbone would change the results. \n\nComments:\n- The proposed idea seems like a good approach for self-supervised terrain representation learning from multiple sensors. My main concern is that it uses a very outdated vision backbone and the terrains being visually distinctive, thus making the effectiveness of multi-modal learning less apparent. More visualization, discussion and analysis could help with strengthening the paper.\n- Some of the ablation results in supplementary are important for understanding the contribution of each component. I suggest moving them into the main paper.\n- Line 291 references Sec 9.1 in the supplementary, but I cannot find it.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "nyvEhcXOBs", "reviewer": "Reviewer_DHLH", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes \"STERLING\" to allow terrain understanding for autonomous off-road navigation without relying on expensive labeled data, engineered features, or expert human demonstrations. The approach takes unlabelled robot experience to learn relevant terrain representations. Physical robot experiments have been carried out to show performance similar to fully-supervised methods and outperform other chosen baselines.", "summary_of_recommendation": "Without positioning and comparing with the relevant work in legged robotics the main contribution can not be assessed.\n\n**Post rebuttal **\n\nI thank the authors for the big effort during the rebuttal. And I raise my score to reflect that.\n\nHowever, I am still on the rejection side as I am not convinced with major points:\n1. More general than legged robots: then why to use the  “feet ground depth penetration estimate\" for spot?\n2. Continual Learning: several times the authors have used this term as if this is a trivial thing to do. There is a vast amount of work in this topic that shows how is easier said than done.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "No questions, see weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper is mostly clear and easy to read.\n- The tackled problem is of relevance and active in the legged robots community\n\nWeaknesses:\n- The paper has failed to position itself in the legged robotics state of the art. [A] [B] and more recently (no mandatory as it was arXiv by the CORL submission date [C]).\n- The chosen baselines where not targeted to legged robots ([A] and [B] must be included, and why not using [28] as well? ).\n- The 3 listed contributions (1. Sterling, 2. evaluations, 3. large scale) get diminished without the proper comparisons.\n\nLegged systems:\n[28] Wellhausen L, Dosovitskiy A, Ranftl R, Walas K, Cadena C, Hutter M. Where should i walk? predicting terrain properties from images via self-supervised learning. IEEE Robotics and Automation Letters. 2019 Jan 27;4(2):1509-16.\n[A] Wellhausen L, Ranftl R, Hutter M. Safe robot navigation via multi-modal anomaly detection. IEEE Robotics and Automation Letters. 2020 Jan 20;5(2):1326-33.\n[B] Loquercio A, Kumar A, Malik J. Learning visual locomotion with cross-modal supervision. In2023 IEEE International Conference on Robotics and Automation (ICRA) 2023 May 29 (pp. 7295-7302). IEEE.\n[C] Frey J, Mattamala M, Chebrolu N, Cadena C, Fallon M, Hutter M. Fast Traversability Estimation for Wild Visual Navigation. arXiv preprint arXiv:2305.08510. 2023 May 15.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "dKqMpK-l9xt", "reviewer": "Reviewer_oD4L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a self-supervised terrain representation framework, which takes both proprioceptive and image data as inputs. The terrain representation network is combined with a terrain-aware planning scheme proposed by [29] for an off-road navigation task. The experiments using a Boston Dynamics Spot robot show the trained network allows the robot to follow user-preferred terrain during autonomous navigation.", "summary_of_recommendation": "The experimental results show promising results on the real robot. However, I'm concerned about the retraining for each experiment, as the network might overfit specific applications instead of learning actual terrain representations. More evidence is needed to support the claims, and there're several issues that need to be addressed. As such, I'm recommending a weak reject.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1. The description of the input signals is a bit obscure. Proprioceptive observations often refer to a boarder collection of sensory data, including IMU, joint encoders, kinematics, and contact sensors [1*]. It is difficult to assess what sensory data are being input into the system by \"inertial, proprioceptive, and tactile observations.\" (line 143). I recommend the authors mention the sensors, e.g., joint encoders, force sensors on foot, etc, in the paper. \n2. The limitations of the proposed method are not addressed. The one point listed in the limitation section is not the limitation of the proposed work. Please discuss the limitations of the proposed framework.\n3. From the paper, it looks like the proposed method is trained separately for different experiments (lines 277, 299, 488). Is it possible that the network is overfitted to specific tasks? For example, since the sidewalk and bricks are the highest priority in experiment 4.1, is it possible that the network only learns to have a low utility cost for sidewalks/bricks and cannot differentiate the rest? What happens if there is no sidewalk/brick in front of the robot? Will the proposed method search for and follow the second preferred terrain? \n4. The paper claims the proposed framework allows the Spot to autonomously hike a 3-mile-long off-road trail (line 297). However, it also states, \"The operator takes manual control only to correct the robot's path during forks, or to yield to incoming pedestrians and pets.\" (line 312) In my opinion, it cannot be claimed as \"autonomous\" if human intervention is required to avoid obstacles and change directions. Furthermore, the paper later reports that only \"2\" human interventions are needed, but from the video and the map, there're multiple turns and dynamic obstacles (human/pets), which according to line 312, require human intervention. \n5. I understand that the lost function is detailed in [26]. However, loss function design is crucial for a self-supervised learning network. I recommend the authors illustrate what the variance, invariance, and covariance function are. \n6. In line 138, the paper states, \"we extract patches from up to 20 previous viewpoints within 2 meters.\" However, in Figure 2 and line 174, it seems only 2 images are used as inputs. It would be great if the authors can clarify this. \n7. The details of the network hyperparameters are not mentioned. (kernel size, step size, etc.) It'll be difficult for future researchers to reproduce the result if they are not provided. \n\n[1*] Barfoot, Timothy D. State estimation for robotics. Cambridge University Press, 2017.\n\n\n----------------------------------------------------------------------------------------------------------------------------\nPost Rebuttal\n\nI thank the authors for their explanation. The new results in Figure 15 addressed my main concern. As a result, I'm updating my ratings to weak acceptance. \n\nHowever, the limitation of this work is still not well stated. Instead of simply stating potential future directions, it is more important to point out the limitation of this work. Under what condition does this method fail? What are the angles that are not covered by this method? What are the things this method can/cannot do?\n\nI strongly recommend the authors to address the limitation thoroughly. This can help the future reader to better understand the potential of this paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strength\n1. The paper is well-organized and easy to follow.\n2. The proposed framework is self-supervised, which reduces the need for manual labeling.\n3. With retraining, the network outperforms baselines in sidewalk/brick following planning. \n4. The proposed method is deployed and tested on a real robot. \n\n## Weakness\n1. The limitations of the proposed work are not well addressed. \n2. Some of the terminologies are not well-defined. For example, the term \"visual invariance\" is repetitively used without referencing what it is invariant to. Is it invariant to $SO(2)$? Or is it invariant to time? Another example is the input IPT data. \n3. It seems the user preference is encoded during training. (Because of the preference utility function.) If the user preference changes, a new training process need to be conducted.\n4. The main contribution is the self-supervised terrain representation learning module. However, no in-depth evaluation of the terrain representation is provided in the main paper. I recommend replacing the large-scale evaluation with the experiments in Appendix 8.2 and adding more in-depth evaluations on the terrain representation learning part. \n5. The network is retrained for each experiment. This reduces the credibility of the results. The network can just be overfitted to a specific tasks/scenarios. \n6. The long-distance experiment doesn't give too much insight into the proposed terrain representation network, as it depends on many different factors outside of the contribution of the proposed framework (planning, human intervention, etc.).\n\nAlso, see questions for rebuttal.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a relevant robotics problem and, after engaging positively with the reviewers, is persuasive in its arguments.  The authors should consider textual revisions to the paper to improve readability, as their (impressive) responsiveness to reviewers has left the work technically dense and not easy to digest.
Decision: Accept (Poster)