Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes "STERLING" to allow terrain understanding for autonomous off-road navigation without relying on expensive labeled data, engineered features, or expert human demonstrations. The approach takes unlabelled robot experience to learn relevant terrain representations. Physical robot experiments have been carried out to show performance similar to fully-supervised methods and outperform other chosen baselines.
review: Strengths:
- The paper is mostly clear and easy to read.
- The tackled problem is of relevance and active in the legged robots community

Weaknesses:
- The paper has failed to position itself in the legged robotics state of the art. [A] [B] and more recently (no mandatory as it was arXiv by the CORL submission date [C]).
- The chosen baselines where not targeted to legged robots ([A] and [B] must be included, and why not using [28] as well? ).
- The 3 listed contributions (1. Sterling, 2. evaluations, 3. large scale) get diminished without the proper comparisons.

Legged systems:
[28] Wellhausen L, Dosovitskiy A, Ranftl R, Walas K, Cadena C, Hutter M. Where should i walk? predicting terrain properties from images via self-supervised learning. IEEE Robotics and Automation Letters. 2019 Jan 27;4(2):1509-16.
[A] Wellhausen L, Ranftl R, Hutter M. Safe robot navigation via multi-modal anomaly detection. IEEE Robotics and Automation Letters. 2020 Jan 20;5(2):1326-33.
[B] Loquercio A, Kumar A, Malik J. Learning visual locomotion with cross-modal supervision. In2023 IEEE International Conference on Robotics and Automation (ICRA) 2023 May 29 (pp. 7295-7302). IEEE.
[C] Frey J, Mattamala M, Chebrolu N, Cadena C, Fallon M, Hutter M. Fast Traversability Estimation for Wild Visual Navigation. arXiv preprint arXiv:2305.08510. 2023 May 15.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Without positioning and comparing with the relevant work in legged robotics the main contribution can not be assessed.

**Post rebuttal **

I thank the authors for the big effort during the rebuttal. And I raise my score to reflect that.

However, I am still on the rejection side as I am not convinced with major points:
1. More general than legged robots: then why to use the  “feet ground depth penetration estimate" for spot?
2. Continual Learning: several times the authors have used this term as if this is a trivial thing to do. There is a vast amount of work in this topic that shows how is easier said than done.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work proposes a method that enables a robot to learn terrain representation in a self-supervised fashion from unconstrained experience. The key idea is to leverage multi-modal sensor data (color, proprioceptive and tactile) for representation learning. It uses VICReg framework that learns a joint embedding over multiple modalities. For planning, it learns a utility function that maps embeddings to costs from human preferences, and then combines it with a geometric cost with a receding horizon planner. Experiments show that the proposed approach adheres to human preferences better than the baselines, while not requiring expert demonstrations during training.
review: Strengths

- Using VICReg for learning terrain representation is clever and makes a lot of sense, given that the strength of VICReg is that it can learn from multi-modal inputs.
- The experiment is thorough with comparison to multiple baselines.
- The paper is well-written and easy to follow.

Weaknesses

- The paper misses some important details. 
    - Since the approach is about using multiple sensor modalities, it is important to visualize the different modalities and discuss how they are correlated for different types of terrains. Visualizing the embedding space would also be helpful.
    - There is no visualization of the costmap. There is one figure that shows the BEV RGB projection and the costmap in the supplementary, but it will be more informative to compare the costmaps produced by different methods. This would help the readers to better appreciate the effectiveness of the proposed approach. 
- The proposed approach uses a rather simple CNN for processing images. It is unclear how a modern vision backbone would change the results. 

Comments:
- The proposed idea seems like a good approach for self-supervised terrain representation learning from multiple sensors. My main concern is that it uses a very outdated vision backbone and the terrains being visually distinctive, thus making the effectiveness of multi-modal learning less apparent. More visualization, discussion and analysis could help with strengthening the paper.
- Some of the ablation results in supplementary are important for understanding the contribution of each component. I suggest moving them into the main paper.
- Line 291 references Sec 9.1 in the supplementary, but I cannot find it.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This seems a promising approach of learning terrain representations using non-expert data. The experiments are thorough and convincing. I vote for weak accept. But it would be great if the authors could address my questions since they are important to understand the effectiveness of the approach on real data.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: In this paper, the authors propose a novel terrain representation learning method based on multi-modal self-supervised learning. Self-supervised TErrain Representation LearnING (STERLING) learns terrain-aware representation with a non-contrastive objective on visual and inertial-proprioceptive-tactile representations extracted from unlabelled robot experience. Features learned by STERLING are evaluated on human preference-aligned navigation, and the experiments along with a qualitative demonstration show that the proposed method outperforms several baselines in navigating through various scenarios with specified preference adherence.
review: **Post-rebuttal comment**
The authors provided a modified version of the paper following reviewer feedback, and I believe it now presents a more convincing claim of STERLING's capabilities for learning generalizable terrain features. I'm confident that most of my original concerns have been addressed in the modified version. I suggest the authors take their time until the final paper submission to ensure good readability as the paper now contains much more details and experimental results than before.  Overall, I am willing to raise my rating from weak accept to strong accept.

Regarding the generalization of STERLING, additional tests on Jackal showed STERLING may generalize to other robot platforms, while the evaluation of the 3-mile test run showed some generalization capabilities to unseen features, while the two interventions may also be seen as failures due to unseen features. The authors should try to mention the two failures to clarify this point in Supplemental Section 9.8.

--
The paper is overall easy to follow and demonstrates impressive results but holds some limitations. See the strengths and weaknesses below.

**Strengths:**
1. The authors were able to use the proposed approach to navigate a 3-mile-long trail with two manual interventions. They quantitatively showed improvement over several baselines in various scenarios as well, which is overall an encouraging demonstration of the effectiveness of the method.
2. Evaluation of STERLING is done with quantitative comparison against GANAV, SE-R, and RCA, which is a diverse set of various prior works on supervised and self-supervised traversability estimation. Against these baselines, STERLING shows better performance in real robot experiments.
3. STERLING relies on self-supervision with some manual preference ranking to generate traversability cost, which is more annotation-efficient than curating annotated datasets. It’s also to be noted that the manual ranking is only done in the context of finetuning STERLING features to a downstream task, and the feature learning itself does not require any annotation. Furthermore, the framework learns more general features that allow downstream tasks compared to prior methods that directly predict traversability.
4. STERLING also does not heavily rely on the actual behavior performed during data collection i.e. it can learn from non-expert demonstrations, and therefore can learn from data that is easier to collect.

**Weaknesses:**
1. Experiments are conducted on relatively simple environments (all six environments are within the university campus with mostly flat terrain). Traversability estimation for off-road navigation would likely require navigating more complex, uneven terrain most of the time. STERLING also uses images projected to BEV with a flat-plane assumption. While this is a reasonable choice in order to focus on the task of preference-aligned navigation, in the context of traversability analysis this is a significantly limiting assumption.
2. One drawback to STERLING is that it focuses on learning from only positive samples i.e. it is only able to learn features from locations it traveled on. It is difficult to comment on the generalizability of the features with respect to new environments and/or new terrain. It seems then that STERLING would require data collection with some prior e.g. novel terrain discovery, and may not perform so well with random experiences.
3. The method requires multi-modal robot experience collected with various sensors, which can be a limitation compared to uni-modal methods such as GANAV.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The proposed approach STERLING is a novel non-contrastive, non-reconstructive self-supervised learning framework that learns more general representation than prior work for off-road navigation. Comparison of STERLING against several baselines demonstrates the effectiveness of STERLING features for the downstream task of preference-aligned navigation while the qualitative evaluation results are impressive. However, the proposed approach also has some limitations, and the paper has some missing details on analyzing experiment results as well as the method’s overall limitations. Concerns raised by the reviewer are described in Questions for Rebuttal. Lastly, the authors are suggested to consider a code release, and/or adding details on the methodology for reproducibility.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a self-supervised terrain representation framework, which takes both proprioceptive and image data as inputs. The terrain representation network is combined with a terrain-aware planning scheme proposed by [29] for an off-road navigation task. The experiments using a Boston Dynamics Spot robot show the trained network allows the robot to follow user-preferred terrain during autonomous navigation.
review: ## Strength
1. The paper is well-organized and easy to follow.
2. The proposed framework is self-supervised, which reduces the need for manual labeling.
3. With retraining, the network outperforms baselines in sidewalk/brick following planning. 
4. The proposed method is deployed and tested on a real robot. 

## Weakness
1. The limitations of the proposed work are not well addressed. 
2. Some of the terminologies are not well-defined. For example, the term "visual invariance" is repetitively used without referencing what it is invariant to. Is it invariant to $SO(2)$? Or is it invariant to time? Another example is the input IPT data. 
3. It seems the user preference is encoded during training. (Because of the preference utility function.) If the user preference changes, a new training process need to be conducted.
4. The main contribution is the self-supervised terrain representation learning module. However, no in-depth evaluation of the terrain representation is provided in the main paper. I recommend replacing the large-scale evaluation with the experiments in Appendix 8.2 and adding more in-depth evaluations on the terrain representation learning part. 
5. The network is retrained for each experiment. This reduces the credibility of the results. The network can just be overfitted to a specific tasks/scenarios. 
6. The long-distance experiment doesn't give too much insight into the proposed terrain representation network, as it depends on many different factors outside of the contribution of the proposed framework (planning, human intervention, etc.).

Also, see questions for rebuttal.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The experimental results show promising results on the real robot. However, I'm concerned about the retraining for each experiment, as the network might overfit specific applications instead of learning actual terrain representations. More evidence is needed to support the claims, and there're several issues that need to be addressed. As such, I'm recommending a weak reject.
review_title: null

Meta-review: This paper addresses a relevant robotics problem and, after engaging positively with the reviewers, is persuasive in its arguments.  The authors should consider textual revisions to the paper to improve readability, as their (impressive) responsiveness to reviewers has left the work technically dense and not easy to digest.
Decision: Accept (Poster)