Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to leverage autonomous robots to explore and collect self-supervised data for training neural fields. Specifically, the authors show that neural fields can be self-supervised to predict materials and force distribution in a real-world setting.
strengths_and_weaknesses: ### Strengths
 
**Originality**

I think the paper has high originality. The prediction of force distribution is novel and interesting.
 
**Quality**

The proposed method is technically sound and applied appropriately. The experiments are thorough and the video results look promising. 
 
### Weaknesses

**Clarity**

Although I think the paper is novel, several contributions listed in the Introduction are overclaimed, and not enough related works are cited:
- Contribution 1 “the first fully autonomous, real-time neural scene labeling robot capable of operating in the real-world” This claim is simply not true. Both [2] and [3] and earlier works they cited have applied autonomous, interactive perception to various problems such as articulated parts discovery. I recommend authors read [4] and refine the claim.
- Contribution 3 “dynamic-keyframe optimization” basically means discarding all the frames after a mode of interaction.
- Contribution 4 “prediction of continuous-valued semantics using a neural-field model” I am not sure why this is a contribution. The most popular values predicted by neural fields such as RGBs and signed distances are all continuous.

**Citation**

For the paragraph starting at line 78, I suggest authors also cite NICE-SLAM [1].

**Typo**

- Line 57: interactive perception, affordance learning and online scene understanding. -> interactive perception, affordance learning, and online scene understanding.
- Line 136: The robot to focuses on -> The robot focuses on
[1] NICE-SLAM: Neural Implicit Scalable Encoding for SLAM, Zhu et al.
[2] Structure From Action Learning Interactions for Articulated Object 3D Structure Discovery, Nie et al.
[3] Ditto Building Digital Twins of Articulated Objects from Interaction, Jiang et al.
[4] Interactive Perception: Leveraging Action in Perception and Perception in Action, Bohg et al.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend Weak Accept but wish the authors to consider my opinion in the Weaknesses section and rephrase the contributions.
issues: I wish the authors address my concerns about **Clarity** within the Strengths And Weaknesses section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper shows a robot manipulator system equipped with a camera, force sensors at the joints, and a single-pixel multi-band spectrometer, that aims to learn physical properties of the table-top scene through active probing and interaction. The inferred properties include the material type of the object (from the spectrometer), the static friction force threshold required to move an object by lateral pushing, a rigidity prediction based on top-down poking. The paper builds on top of iMAP [1] and iLabel [2] for real-time neural scene representation and interactive labeling from few demonstrations. 

[1] iMAP: Implicit Mapping and Positioning in Real-Time
[2] ILabel: Interactive Neural Scene Labelling
 
strengths_and_weaknesses: Strengths:

S1: In my opinion, the paper's main contribution comes from showing that fully autonomous interactive perception can be very effective in figuring out object properties with few interactions, for both rigid and deformable objects, such as clothes, even when neural scene representations are used. Active sensing is often tricky because random queries seem to catch up to active queries faster than hoped for, in some domains. It is informative to know that this is not the case here. 

S2: The use of the spectrometer is something that rarely appears in robotics literature, but seems to be very helpful in categorizing material types. Although the material-type classifier was pretrained on all materials, I think it enables a range of possible applications in manipulation, and more roboticists should be aware of these possibilities.

S3: The paper is clearly written, with many descriptive figures, and an informative video. 

S4: The limitation section is thoughtful, honestly written, and extensive. I applaud the authors for doing this well.

Weaknesses:

W1: The paper claims that its neural representation can handle dynamically changing scenes, for example "learning neural field representations in dynamic environments using dynamic-keyframe optimisation." This claim is slightly misleading, as the paper clarifies later on, as the history of old keyframes from iMAP gets dropped, and the neural representation is only valid if the changes in the scene are small, as is the case usually in the lateral pushing experiments. This is indeed clarified in section 3.6, but there is an incongruence between the claimed contributions in the introduction and the strict conditions under which they hold. This is a hard problem, understandably, but I would have appreciated more clarity about assumptions in the claimed contributions. 

W2: Interaction queries are generated only from the first viewpoint, and are not updated as the robot acquired more measurements. This slightly undermines the goal of the paper to perform fully active perception, since dynamic scene changes are not accounted for, but it is not a deal-breaker. It just means that the active query selection is not performed optimally.   
 
W3: Entropy-based active perception is claimed as a contribution, but entropy has been used as an acquisition function for many learning systems and labeling tasks, including in robotics. While this is a key component of the paper, and seems to work well, I don't think it has to be claimed as a novel contribution. The fact that it works well is informative enough for the reader.  

W4: Clarification needed, what do the error bars in Fig 5 represent? And why are the error bars for the random selection strategy not decreasing much over time?

W5: Nitpick, but, it would be better if the 3d plots in Fig 1 were rotated to match the orientation of the scene.

W6: It would be useful to mention the exact model of the spectrometer used in the paper. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I find this paper very exciting, and I am recommending it be accepted because of strengths S1 and S2. I also very much enjoyed the fact that the paper was not limited to rigid objects. 
issues: I am hoping that the weaknesses I mentioned can be addressed easily during the revision period, as they are mostly related to presentation and writing, and do not require additional experiments.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an approach for letting a robot learn about its surroundings through physical interaction. The approach builds upon an existing work for interactive 3D scene labeling ( Reference [2] in the manuscript). The method is largely agnostic to the segmentation task, as long as the property of interest can be efficiently sensed.
strengths_and_weaknesses: The main strength of this paper is its underlying idea: using proprioceptive experience to label physical properties of the world is much more scalable than having human annotators collect data. Some previous works (e.g. [1] and other references in the paper) proposed a similar idea, but the proposed setup differs in several aspects. 
The proposed approach is also very interesting: connecting the two worlds of assisted labeling and learning by interaction is very interesting and well-suited to the problem. Of course, several technical changes have been required to make the approach work for the current problem. However,  I guess that several works in the future can benefit from this connection.
Finally, the work is quite well executed: results are provided on multiple segmentation tasks, showing the generality of the idea. 

I see two main weaknesses in the manuscript. First, the information about the differences between the original Ilabel work and the proposed method is scattered and difficult to grasp. For example, the entropy-based formulation was already proposed by it, but it is not clearly mentioned. I would propose an overview paragraph explaining in detail the required changes that made the system work.
Related to this limitation, not enough ablation studies have been conducted. Ideally, each of the changes required by IMAP should be ablated with controlled experiments. This would give the reader a good grasp of what pipeline parts need improvement.

Overall, I think this is an interesting paper that could be of interest to the community. I also appreciate the extensive limitation section, which gives an honest overview of the current status of the idea of learning by interaction. 

[1] Learning Depth With Very Sparse Supervision, Loquercio et al.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My recommendation is based on the interesting idea and approach. I could increase my score given more controlled experiments on the different aspects of the methodology.
issues: I think the paper would benefit from a better discussion of previous work and the differences between the proposed approach and the iLabel work, upon which it builds. In addition, more ablation studies would improve the quality of the experimental section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes autonomous interactive experimentation with a scene in order to learn its properties, leveraging the ability of 3D neural field representations to generalise given as few as tens of examples. Experimental design is guided by entropy, maximising confidence under the constraints of the platform and environment. A workaround is included for the dynamics resulting from interactive experimentation. A generalisation of neural semantic labelling to continuous-valued semantics is included. The approach is demonstrated estimating rigidity, material, and frictional force distribution.
strengths_and_weaknesses: The paper makes important and timely advances in the joint representation of geometry and semantics using neural fields. Several contributions combine to yield an impressive demonstration: experimental design under constraints, dealing with dynamics, and encoding continuous semantics work together to enable a novel autonomous robotic explorer. This is a clever use of point sensors, allowing complex properties like hyperspectral signatures to be sensed without need for more expensive snapshot hyperspectral cameras, or more time-consuming pushbroom scanning.

The literature review covers the main related works well and relates the contributions of the present work. The paper is well represented and contains sufficient detail to allow replication by a skilled practitioner.

Visual results are included that qualitatively demonstrate and validate the approach. Quantitative results establish the efficacy of the experimental design with respect to naive approaches for segmentation and classfication accuracy. Force estimation is quantitatively compared to an analytic approach with privileged information, comparing favourably.

While some limitations are discussed, the paper misses discussion of some key limitations of the method. The underlying 3D representation discards the directional dependence captured by NeRF, for example, meaning the method should be expected to fail around visually challenging reflective and refractive objects. A robot experimenter can accumulate a vast and always-growing repertoire of experimental results... how to deal with this growing dataset is not discussed.

The paper would be strengthened through release of source code and datasets.

The force estimation results show only one example, a more comprehensive demonstration with multiple object types and aggregate statistics would be more convincing.

The references incorrectly render at least two acronyms in lowercase.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper is well presented and makes multiple important and timely advancements that will be of interest to a substantial part of the robotics community.
issues: The authors may wish to expand on the limitations section as per above comments, and consider releasing code and exemplar datasets. It would have been stronger to include more experimental results around force estimation, this could perhaps form the basis for an extension of the work.
review_title: null

Meta-review: metareview: This paper proposes an approach leveraging 3D neural implicit scene representations to autonomously infer rigidity, material, and frictional force distributions in real settings.

Overall, the reviews are positive. Reviewers agree that the presented ideas are compelling, but point to several areas (e.g. assumptions, limitations, ablations) that need additional clarification – particularly with respect to overclaiming and consolidating contributions. Authors are encouraged to respond to reviewers' comments and questions.

Update:

All reviewers agree that the paper is well-presented and should be accepted; the ideas are timely, and will be interesting to the robotics community. The authors did not submit a rebuttal – several reviewers re-raised concerns that were not addressed during the rebuttal phase, including a discussion on key limitations (visually challenging reflective and refractive objects: a key advantage of NeRF vs classic RGB-D fusion), as well as providing more clarity on assumptions being made for claimed contributions with respect to prior work. Authors are strongly encouraged to address the concerns raised in the original reviews for the camera-ready version.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)