Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter
Authors: Iain Haughton
Edgar Sucar
Andre Mouton
Edward Johns
Andrew Davison
Venue: CoRL 2022 Oral (2022)
Keywords: Neural field
robot experimentation
physical properties

Abstract: Neural fields can be trained from scratch to represent the shape and appearance of 3D scenes efficiently. It has also been shown that they can densely map correlated properties such as semantics, via sparse interactions from a human labeller. In this work, we show that a robot can densely annotate a scene with arbitrary discrete or continuous physical properties via its own fully-autonomous experimental interactions, as it simultaneously scans and maps it with an RGB-D camera. A variety of scene interactions are possible, including poking with force sensing to determine rigidity, measuring local material type with single-pixel spectroscopy or predicting force distributions by pushing. Sparse experimental interactions are guided by entropy to enable high efficiency, with tabletop scene properties densely mapped from scratch in a few minutes from a few tens of interactions.

Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter
Jain Haughton Edgar Sucar
Andre Mouton Edward Johns
Andrew J. Davison
Dyson Technology Ltd.
Dyson Robotics Lab, Imperial College
3 Robot Learning Lab, Imperial College
iain. haa..hauggttndddyon..comm
Abstract:
Neural fields can be trained from scratch to represent the shape and appearance of 3D scenes efficiently. It has also been shown that they can densely map correlated properties such semanties, via sparse interactions from human labeller. this work, we show that robot can densely annotate scene with arbitrary dis- crete or continuous physical properties via its own fully-autonomous experimental interactions, it simultaneously scans and maps it with an RGB-D camera. A variety of scene interactions are possible, including poking with force sensing to determine rigidity. measuring local material type with single-pixt piggell spectroscopy or predicting force distributions by pushing. Sparse experimental interactions are guided by entropy to enable high efficiency, with tabletop scene properties densely mapped from scratch in a few minutes from a few tens of interactions.
Keywords: neural field, robot experimentation, physical properties
Rigidity classification
Material classification
Force distribution
Figure 1: Over a few minutes our robot makes sparse, automatic physical seene interactions, such as touching to test rigidity, sampling local material type with spectroscopy pushing to determine frictional force distribution. The interaction results used sparse labels the output channels of joint neural-field model of 3D shape and appearance, trained real-time. Model coherence allows the measured physical properties be efficiently densely propagated whole scene, without the need for prior training data.
Introduction
An autonomous agent performing complex tasks in unpredictable environments (e.g. tidying home), must build a rich internal representation its environment. This representation should extend
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

beyond geometry, colour and traditional semantics to include physical properties from which robotic affordances may be inferred. Like its biological counterparts, robotic agent cannot infer these properties from vision alone [1]. In this work, we present real-world robot that learns about its environment by autonomous exploration and experimentation. The robot incrementally discovers physical scene properties by performing targeted physical interactions with the scene, measuring their effects and feeding the results into a joinly-optimised neural-field MLP. With only a handful of these autonomous robotic experiments, task-driven internal representation of the scene is built from scratch, without any pre-training or external guidance.
We build upon the observation that the joint internal representation of shape and appearance learned by implicit scene representation models, and the smoothness and compactness priors present in these models, allow for ultra- -ettaa--ffiiien,, wene propagation of user- -ssr-prrvided labels [2]. We remove the human from the loop entirely, presenting the first fully-autonomous, 3D scene understanding robot that combines active exploration and physical experimentation with a unified neural-field representation as its underlying computational model.
The robot explores a scene with an RGB-D camera and simultaneously builds a map of semantic entropy, representing confidence in its current predictions. Considering this entropy, as well as kinematic feasibility and collision avoidance, the robot selects the optimal point of interaction and proceeds take a physical measurement used for optimising the semantic head of the underlying MLP. The real-time rendering of entropy guides the robot in selecting efficient and beneficial interactions, allowing for dense scene labelling with only a handful of experiments. In contrast iLabel [2], which assumes a static scene, and can therefore rely on growing collection of keyframes over which to optimise semantics, the interactive nature of our framework results a dynamic scene. In addressing this, we make the observation that neural representations possess temporal memory, and thus propose new strategy that uses only the latest keyfram keyframe during optimisation. In this way, the neural representation remains current, allowing for continuous exploration in scenes with limited dynamics.
Autonomous discovery of physical scene properties potentially facilitates a broad range of practical downstream robotic tasks. For example, automated tool exchange in a curface-cleaiin robot: object and material sorting for automated sorting of recycling or laundry: improved task planning enabled by understanding object affordances and physical properties of objects; and improving the safety of autonomous robots by mitigating interactions with potentially hazardous objects or materials.
In summary, the key contributions of this paper labelling robot, trained from scratch, in real-time, and capable of operating in the real-world; 2) active robotic experimentation achieved using a single MLP drive entropy-guided automatic query generation, kinematic feasibility checking, collision avoidance and physical interaction planning and execution; 3) demonstration of the temporal memory characteristics of neural field representations via single- frame optimisation and robustness to small temporal scene changes and 4) prediction of dense, continuous- valued semantics using neural-field model optimised with respect to sparse ground truths. We substantiate these contributions with a series of real-world experiments, demonstrating the capability of our system to produce variety of perceptual maps or scene labellings, based on the chosen sensor and action primitives.
1) the first aully-autonomous, neural-scene
2
Related work
We address the problem of fully-autonomour unsupervised scene understanding via active experimen- tation. Our solution operates at the intersection of several research domains: instance segmentation, interactive perception, affordance learning, and online scene understanding.
Instance segmentation: Unseen Object Instance Segmentation (UOIS) is a necessary capability for robots operating in unstructured environments. Xie et al. [3] propose a class-agnostic RGB- D instance segmentation method, using only simulated training data with reasonable sim-to-real generalisation. Xiang et al. [4] improve sim-to-real transfer by learning RGB-D feature embeddings from synthetic data using a contrastive loss, which allows their Unseen Clustering Network (UCN) to capitalise on both synthetic depth and non-photor RGB data. Gouda et al. [5] fine-tune a Mask R-CNN [6] model to perform class-agnostic instance segmentation. The current state-of-the-art in UOIS is achieved by refining the outputs of the UCN method with the graph-based RICE method [7].

---

Interactive object segmentation: Bohg et al. [8] first provided a detailed definition of Interactive Perception (IP) and an analysis of its benefits in solving robotic perception tasks. common approach in interactive object segmentation is to exploit motion cues induced by physical interactions [9, 10, 11, 12]. While our work shares the broad objective of segmentation through interaction, is not limited to rigid objects or segmenting based on geometric extent. Instead, we demonstrate segmentations that are optimised for given downstream tasks, dictated by the chosen action primitive and sensor.
Interactive affordance learning: Le Goff et al. [13] learn affordance-specife relevance maps. describing which actions can be applied at which locations. The method relies on an off-the- shelf point-cloud segmentation [14] and an online classifier [15]. Nagarajan and Graumann [16] simultaneously train an interaction exploration policy and an image-based affordance segmentation model mapping image regions to the likelihood that they permit an action. Both these prior works limit experimentation to simulated environments. Although we do not consider affordances explicitly in this work, it the ultimate objective of performing task-driven segmentation the way that we do.
Online scene understanding and labelling: The iLabel framework [2] (built upon iMAP [17]), demonstrated that online, scene-specific training of a compact MLP model, which jointly encodes scene geometry, appearance and semantics allows for ultra-sparse interactive labelling and produces accurate dense semantic segmentations outperforming existing pre-trained methods. We remove the human-inn numan--n-thee -n--tie--ooog entirely. demonstrating the first aullooouto neural-scene labelling robot. To achieve this, we introduce several sigmificant contributions allow for operation on real-world robot, where scene dynamics, kinematics, constraints and interaction feasibility must be considered.
3 Method
We represent 3D scenes similarly to iMAP [17], with an MLP that maps a 3D coordinate to colour and volume density. We build sparse set of keyframes, selected incrementally based on information gain, and whose viewpoints span the scene. The MLP parameters and camera poses are jointly and continually optimised via differential volume rendering of actively-sampled sparse pixels in this set of keyframes. il.abel [2] adds semantic head the neural scene MI.P in iMAP, allowing users to provide pixel-level annotations in the keyframes. The joint optimisation in iMAP is then extended to include scene semantics, which are optimised through semantic rendering of the user annotations. Both models are trained from scratch, in real-time and without any prior data and have demonstrated the power of the joint internal representations learnt by the neural-field MLP.
3.1 Autonomous robot experimentation
The robot builds an internal representation of its environment via a series of autonomous experiments. First, it actively selects interaction locations that are both feasible and information-ris (based on semantic entropy). Second, the selected 2D image locations are mapped to the real-world coordinate system of the robot, and a physical interaction with the scene is planned and executed. Third, the resulting measurement is processed and/or classified (see specifics in Section 3.5) to obtain the ground-truth semantic label (akin to the user-provided object class in [2]). Finally, using the labels obtained in this manner, scene semantics are optimised through semantic rendering of the robot- selected keyframe pixels. As demonstrated previously [2], the resulting joint internal representation of shape, appearance and semantics of the neural-field allows for the sparsely-annotaal semantics to be propagated efficiently and densely throughout the scene.
3.2 Modes of interaction
Our framework facilitates the autonomous discovery and mapping of any measurable characteristic of a scene, provided that a suitable measurement sensor and interaction protocol can be defined. We demonstrate three particular interaction types: 1) predicting rigidity by top-down poking: 2) predicting material type using a single-pixel multiband spectrometer ;; and 3) predicting frictional force distributions by lateral pushing. These modes constitute the basis our fully automatic system described in the following sections.
'SparkFun Triad Spectroscopy Sensor- AS7265x (Qwiic)
3

---

3.3 Entropy-guided interactions
In [2], qualitative experiments demonstrated that well-placed user clicks, especially in regions where the model is performing poorly, are the most beneficial in terms of improving segmentation quality. This observation was exploited in an automatic query generation framework, whereby an uncertainty- based sampling was used to actively propose pixel positions for the user to label. Similarly, we utilise softmax entropy to guide the physical interactions that the robot makes with the scene, encouraging interactions that are optimal in terms of information gain and thereby minimising the number of interactions required to produce optimal segmentations. Softmax entropy, us, defined as [18]:
Ug Σ
Se [u, w/ log (S. [u, (l,
(1)
with Se(u,, o] the rendered semantic distribution and C the number of categories.
3.4 Interaction feasibility
In order to integrate uncertainty-bast based sampling into fully-autonomous robotic behaviour, kine- matic feasibility and the particular mode of interaction have to be considered. Queries deemed to be kinematically infeasible and/or unsuitable given the mode of interaction, are added to a query mask, preventing repeated testing. For example, top-down interactions require masking of candidate points not approximately perpendicular to the surface normal, while the converse is true for lateral interactions. Similarly, sensor type impacts interaction feasibility. For example, when performing spectroscopic measurements, we mask regions of high curvature to encourage measurements perpen- dicular to the target surface. More generally, we have found that interactions object boundaries are generally undesirable, owing to a higher incidence of failures associated with misclassification and object tipping. We have therefore investigated two approaches to reduce the probability of bound- ary interactions: 1) Gaussian filtering of the uncertainty maps and 2) uncertainty clipping. While Gaussian filtering yields reliable behaviour in scenes containing coherent objects, it performs poorly with overlapping objects (e.g. fabrics/clothing). In contrast, clipping the uncertainty map by setting values above threshold zero produces more reliable results. The robot focuses on regions of high uncertainty but ignores problematic regions of maximal uncertainty. There is a trade-off between the degree of boundary suppression and uncertainty map degradation. We determined empirically an optimal clipping threshold of 0.7 and kernel size of 41.
3.5 Semantic representation
In contrast to [2], the inputs to the semantic head of our neural-field MLP can be one of several physical properties, measured via apposite affordances and modes of interaction. Raw sensor measurements acquired by the robot need to be post-processed or converted into the target variable being predicted by the semantic head of the MIP. For binary prediction tasks (e.g. rigidity), this may be as simple as applying a threshold the raw measurement. Multi-class target variables may require additional processing. For example, when predicting material type from a multidimensional spectrometer reading, we use pretrained multiclass SVM classifier which outputs predefined material classes, which are then fed to the semantic head. In both scenarios, as in [19, 2] the semantic head of the MLP predicts categorical value and can be optimised using cross-entropy loss.
We additionally demonstrate for the first time the prediction of continuous- target variables in the semantic head of the MLP, where the ground-truthe are sparse, in contrast to the dense ground- truths used in the optimisation of the colour and density heads. For example, when predicting frictional force distributions, we feed the minimum (stiction) force required to move an object, directly to the semantic head and optimise using an L11 loss.
3.6 Single frame optimisation
During lateral pushing, interactions between the robot and the scene may introduce object displace- ments. This violates the static-scene assumption common to prior work neural radiance fields. While this assumption allows [17, 2) to optimise over an expanding set of keyframes, dynamic scene potentially invalidates historie keyframes, ultimately leading to errors in the reconstruction. We therefore clear the keyframe history and corresponding labels after each interaction in this mode. Our

---

experimentation has suggested that neural radiance representations possess some form of temporal memory characteristic over the labelled properties, whereby network weights adapt over time and maintain consistency with the dynamic scene, provided scene changes are comparatively small.
4 Experiments
A fundamental contribution of work is the realisation of a autloouu robot that operates in a real-world environment. We demonstrate the ability of our system to perform series of autonomous experiments, using the aforementioned interactive modes, to discover and predict variety of physical scene properties. We demonstrate the quantitative benefits of entropy- the case of rigidity and material-type classification, compare segmentation performance against two state-of-the-ane class-agnostic segmentation techniques (see Sec. 4.2 for details). Finally, we refer the reader supplementary video for additional results.
experimentation and, in
4.1 Experimental setup
We Franka Emika Panda robot, anchored to table which variety of objects arranged (Figure 1). The robot is equipped with a Realsense D435 RGB-D sensor [20], tracked using the forward kinematics of the arm, which is controlled using ROS [21]. Prior to physical experiments, the robot builds a geometric reconstruction of the scene, to allow for collision-fre motion planning. For this purpose, set of RGB-D keyframes is captured over series of random motions in order to optimise the 3D neural field, and subsequent querying of the network produces a collision mesh and normal map. All objects of interest placed within reach of the robot arm and any points located beyond this range, or on the plane of the table, are automatically labelled 'table".
4.2 Quantitative results
Rigidity and material-type prediction may be viewed as segmentation problems. While training any popular instance segmentation technique (e.g. Mask R-CNN [6]) the object classes present in our scenes (e.g. material types), is likely to produce high-quality segmentations, one would need to repeat this training for each scenario. Therefore, instead of comparing against closed-set segmentation class- agnostic instance segmentation approaches: I) Mask R-CNN trained to perform class-agnostic segmentation [5] and 2) Unseen Clustering Network (UCN) [3] with RICE refinement [7] (see Sec. for details). For each method, we perform instance segmentation the keyframe and use the resulting instance mask to guide the robot-scene interaction. In particular, the robot takes a single sensor reading as near to the centre of each instance in the mask feasible and propagates the measurement to the rest of the region. The measurements converted to categorical labels (binary rigidity or material-type) in the same manner as described in Sec. 3.5. We report the mean Intersection over Union (mloU) averaged over the ground-truth labels.
techniques, we consider two state-of-the.
Table shows the quantitative performance comparison against the Mask R-CNN and UCN base- lines for each scene. As expected, the baselines perform well for scenes 1 and which contain geometrically-coheeee
objects and strong colour and depth cues. Scene 2, however, is considerably more challenging for the colour and/or depth-based baselines, characterised by significant drop in performance. In contrast, our autonomous approach performs well for all three scenes, with comparable results to the baselines in Scenes I and 3 and significantly superior results in Scene 2.
4.3 Entropy exploration ablation study
Figure 2 illustrates material discovery in complex scene containing wool (blue), cotton (yellow) and synthetic (green/pink) materials. Prior to physical measurements, there is high uncertainty (red) across the entire scene, while the final uncertainty map has high confidence (blue) throughout. We show the evolution the uncertainty map through three consecutive interactions in Figure 3. The robot is guided to high-entropy pixel (unfilled circle). On completion of the experiment, there is clear, localised uncertainty reduction surrounding the target region (filled circle).
We observe that while the uncertainty in the localised region of measurement decreases, it often increases in more distant regions. As the model accumulates information, it continuously adapts its predictions and corresponding confidence, ultimately converging accurate representation. This
5

---

observation motivates the use of uncertainty an exploration metric in neural implicit representations. To substantiate the benefits of entropy-driven exploration quantitatively. we conducted an ablation study comparing performance to random exploration. We compare the evolution of mloU and false- confidence (where the model produces high-confidence but incorrect predictions) with an increasing number of interactions for each technique. Figure 5 demonstrates superior convergence rates for catropy-guided interaction all three benchmark scenes Figure across both metrics.
4.4 Force measurement analysis
Figure 6 illustrates the stiction force maps produced by our framework for objects with uniform (top row) and non-uniform (bottom row) mass and friction distributions. Note that each scene objects are displaced following the pushes performed by the robot. The scene in the row contains three cylindrical containers of varying mass and material. As desired, the resulting force renderings match the varying masses. This is potentially valuable information when planning for downstream manipulation tasks (c.g. distinguishing between full and empty containers). A key observation in Figure is that the renderings for objects remain consistent despite displacement, demonstrating for the first time a memory quality in neural field representations.
In the bottom row of Figure 6, we demonstrate the ability of the robot to predict stiction force values reliably, even for complex geometries, with non-uniform mass and friction distributions. We substantiate this quantitatively in Figure 7, where the robot interacts with non-uniform rectangular box containing a 5 kg weight at one end. We show that the output of our model, after three pushes, comparable to that of simple analytical physics model [22] with access to privileged information, including the contact surface area, mass distribution and friction coefficient.
Initial keyframe and uncertainty map
Final segmentation and uncertainty map
Example material type segmentations using spectrometer. 46 interactions were required to separate the pile of laundry into wool (blue), cotton (ycllow) and synthetic (green/pink) materials. Red/blue signifies high/low uncertainty the uncertainty map.
Figure 2:
Interaction 25
Interaction 26
Interaction 27
Figure 3: Demonstration of autonomous guidance over consecutive interactions in Figure 2. From left to right, the interactions (unfilled markers) follow the highest uncertainty pixel. After interaction (filled marker). there is localised reduction in uncertainty.
Table 1: Classification performance for different types of scene, (examples in Figure 4).
Segmentation
Example
Ours
Mask R-CNN
UCN + RICE
Material
Scene
0.91
0.02
0.92 0.02
0.90 0.02
Material
Scene 2 0.89 1 0.03
0.56 1 0.11
0.56 1: 0.10
Rigidity
Scene
0.91
0.04
0.92 0.02
0.91 0.02

---

ROSE
Scene material
Scene 2 material
Scene rigidity
Figure 4: Our system can interact with and segment varicty of scenes.
1.0
1.0
1.0
0.4
222
0.4
0.2
Entropy
0.2
Entropy
0.2
Entropy Random
Random
Random
0.00
25
0.0g
15
30
0.0g
Interactions
Interactions
Entropy
Entropy
Entropy
Faroo CorEdonce
Random
cortoonce
Random
Ragge Corsoonce
Random
0.8
0.5
0.3
0.4
0.4
0.4
meg
0.0g
0.0g
0.0
Scene
Scene
Scene
Figure 5: Comparisons of mean IoU (top) and false-confidence (bottom) entropy- based and random exploration approaches.
number of interactions for
Limitations
Several assumptions made during the experimentation procedure: 1) rigid objects are never placed on top of flexible objects; 2) the SVM classifier has been trained on all materials and 3) pushed objects are both rigid and free to move across the tabletop.
The main limitations of our system relate its self-awareness of errors. We only include examples in which all measurements and inferences are correct. In reality, this is not always the case, with each mode of measurement having an associated accuracy, and incorrect measurements impacting model performance. We observed that, although the model recovers after an erroneous measurement, it involves significantly more interactions. An alternate solution would be to associate a likelihood with each measurement and allow the model to adapt accordingly. Furthermore, with all scene interactions, there is a possibility of catastrophic disturbance, preventing the model from propagating labels effectively. This scenario was observed several times during experimentation, requiring manual reset. Ideally, failures of this nature should be detected autonomously based on the corresponding surge in reconstruction loss.
Although several keyframes are captured during robot motion, interaction queries were only ever generated from the first keyframe, which in our setup was predefined the initial viewpoint of the robot. This has the advantage limiting the attention of the robot to what we deemed to be the most interesting part of the scene. However, it also meant the robot was unable to conduct

---

Scene
Normals rendering
Force rendering
Figure 6: Stiction force mapping. Top row: three cylindrical objects with uniform mass of (from left to right) 0.5 1.5 1.5 kg and 0.1 kg. Guided by entropy, the robot applies single push to each object measuring, stiction forces of N. 3.0N and N. Bottom row: power drill with non-uniform mass distribution. The final rendering produced after sequence of 3 pushes.
Ours
10
Physics
[N]
Force [N]
kg
0.5
0.2m
Physics
0.1
0.2
0.3
0.4
0.5
0.6
8.00
0.05
0.30 Witth [m]
0.35
0.20
Longth [m]
Crossection
Length distribution
Width distribution
Figure 7: Rendered stiction force distribution, compared analytical approach with privileged information, along length (blue) width (red) with non uniform density.
wider investigation; in particular, the back surfaces of objects or those regions occluded in the initial viewpoint. This was most evident lateral pushing, where pushing from multiple angles could only be achieved with careful choice of viewpoint and object positioning. A future improvement would address this by allowing queries across multiple keyframes, though robot will still constrained by its kinematic limits.
The sensitivity of force measurement is limited by the robot joint torque sensors and subsequent estimation of external torques and wrenches. This could be improved by using an external force- torque or tactile sensor. Using spectrometer with greater bandwidth would allow for enhanced differentiation between material types. Finally, while we propose that our system facilitates the discovery of large variety of physical scene properties, we only consider three specific scenarios in this paper.
6 Conclusions
By taking advantage of the unsupervised decomposition capabilities of 3D neural field trained on single scene, have demonstrated fully-automatic, real-world robotic platform that is able to build dense, accurate and physically meaningful representations its environment. The fact that this be achieved from scratch and in real-time, without any pre-training or external guidance, gives the method potentially wide applicability in robotic scene learning and understanding. In the future, excited extend the system with new interactions, such probing to build a grasping map, or using alternative sensors to measure properties such as temperature or surface roughness.
with gripper

---

Acknowledgments
Research presented in this paper has been supported by Dyson Technology Ltd. We would like to thank Charles Collis at Dyson Technology Ltd. for his support and in enabling the secondment of lain Haughton from Dyson Technology Ltd. to the Dyson Robotics Lab, Imperial College. We also wish to thank Barry Beard at Dyson Technology Ltd. for his help with using the spectroscopy sensor for material classification. Finally, we would like to thank Tristan Laidlow, Daniel Lenton, Kentaro Wada and Shuaifeng Zhi who engaged in research discussions as well as the reviewers who provided valued feedback work.
References
[1] M. Egelhaaf, N. Boeddeker, R. Kern, R. Kurtz, and J. P. Lindemann. Spatial vision in insects facilitated by shaping the dynamics of visual imput through behavioral action. Frontiers in neural circuits, 6:108, 2012.
[2] S. Zhi, E. Sucar, A. Mouton, I. Haughton, T. Laidlow, and A. J. Davison. iLabel: Interactive neural scene labelling. arXiv preprint urXiv:2111.14637 2021.
[3] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics, 37(5):1343-1399 2021.
[4] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning RGB-D feature embeddings for unseen object instance segmentation. In Conference on Robot Learning CoRI., 2021.
[5] A. Gouda, A. Ghanem, and C. Reining. Category-agnostic segmentation for robotic grasping. arXiv preprini urXiv:2204. 2222004,16133 13613, 2022.
[6] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In Proceedings of the Interna- tional Conference on Computer Vision (ICCV), 2017.
[7] C. Xie, A. Mousavian, Y. Xiang, and D. Fox. RICE: Refining instance masks in cluttered environments with graph neural networks. In Conference on Robot Learning, pages 1655-1665. PMLR, 2022.
[8] J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. S. Sukhatme. Inter- active perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6):123-1129, 2017.
[9] J. Kenney, T. Buckley, and O. Brock. Interactive segmentation for manipulation in unstructured environments. In 2009 IEEE International Conference on Robotics and Automation, pages 1377-1382. IEEE, 2009.
[10] K. Hausman, F. Tracking-based interactive segmentation of textureless objects. In 2013 IEEE International Conference Robotics and Automation, pages 1 1122-11 1129, IEEE, 2013.
D. Pangercic, Z.-C. Marton, R. Ueda, K. Okada, and M. Beetz.
[11] H. Van Hoof, O. Kroemer, and J. Peters. Probabilistic segmentation and targeted exploration of 199-1199 2014.
objects in cluttered environments. IEEE Transactions on Robotics,
[12] C. D. Singh, N. J. Sanket, C. M. Parameshwara, C. Fermüller, and Y. Aloimonos. NudgeSeg: Zero-shot object segmentation by repeated physical interaction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2714-2712. IEEE, 2021.
[13] L. K. Le Goff, O. Yaakoubi, A. Coninx, and S. Doncieux. Building an affordances map with interactive perception. Frontiers in Neurorobotics, page 68, 2022.
[14] J. Papon, A. Abramov, M. Schoeler, and F. Worgotter. Voxel cloud connectivity segmentation- supervoxels for point clouds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2027-2034, 2013.
[15] L. K. L. Goff, G. Mukhtar, A. Coninx, and S. Doncieux. Bootstrapping robotic ecological perception from limited set of hypotheses through interactive perception. arXiv preprint 1xXiv:1901.10968, 2019.

---

[16] T. Nagarajan and K. Grauman. Learning affordance landscapes for interaction exploration in
3D environments. Advances in Neural Information Processing Systems. 33:2005-2015, 2020.
[17] E. Sucar, S. Liu, J. Ortiz, and A. Davison. iMAP: Implicit Mapping and Positioning in
Real-Time. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[18] V. ll,, J. Porta, and J. Andrade-Cette Information-Based Compact Pose SLAM. IEEE Transactions on Robotics (T-RO), 26(1):78-92)) 93, 2010.
[19] S. Zhi, T. Laidlow, S. Leutenegger, and A. J. Davison. In-Place Scene Labelling and Under- standing with Implicit Scene Representation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[20] L. Keselman, J. Iselin Woodfill, A. Grunnet-Jepsen, and A. Bhowmik. Intel Realsense stereo- scopic depth cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1-10, 2017.
[21] Robot Operating System. ROS. URL https://www.ti. org.
[22] M. T. Mason. Mechanics and planning of manipulator pushing operations. The International Journal of Robotics Research, 5(3):53-71, 71, 1986.
10


Reviews:
[{"review_id": "ojCLkb36EG7", "reviewer": "Reviewer_yfuZ", "summary": "This paper proposes an approach for letting a robot learn about its surroundings through physical interaction. The approach builds upon an existing work for interactive 3D scene labeling ( Reference [2] in the manuscript). The method is largely agnostic to the segmentation task, as long as the property of interest can be efficiently sensed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper is its underlying idea: using proprioceptive experience to label physical properties of the world is much more scalable than having human annotators collect data. Some previous works (e.g. [1] and other references in the paper) proposed a similar idea, but the proposed setup differs in several aspects. \nThe proposed approach is also very interesting: connecting the two worlds of assisted labeling and learning by interaction is very interesting and well-suited to the problem. Of course, several technical changes have been required to make the approach work for the current problem. However,  I guess that several works in the future can benefit from this connection.\nFinally, the work is quite well executed: results are provided on multiple segmentation tasks, showing the generality of the idea. \n\nI see two main weaknesses in the manuscript. First, the information about the differences between the original Ilabel work and the proposed method is scattered and difficult to grasp. For example, the entropy-based formulation was already proposed by it, but it is not clearly mentioned. I would propose an overview paragraph explaining in detail the required changes that made the system work.\nRelated to this limitation, not enough ablation studies have been conducted. Ideally, each of the changes required by IMAP should be ablated with controlled experiments. This would give the reader a good grasp of what pipeline parts need improvement.\n\nOverall, I think this is an interesting paper that could be of interest to the community. I also appreciate the extensive limitation section, which gives an honest overview of the current status of the idea of learning by interaction. \n\n[1] Learning Depth With Very Sparse Supervision, Loquercio et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation is based on the interesting idea and approach. I could increase my score given more controlled experiments on the different aspects of the methodology.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I think the paper would benefit from a better discussion of previous work and the differences between the proposed approach and the iLabel work, upon which it builds. In addition, more ablation studies would improve the quality of the experimental section."}}, {"review_id": "bDF4yw2WH9n-", "reviewer": "Reviewer_beEy", "summary": "This paper shows a robot manipulator system equipped with a camera, force sensors at the joints, and a single-pixel multi-band spectrometer, that aims to learn physical properties of the table-top scene through active probing and interaction. The inferred properties include the material type of the object (from the spectrometer), the static friction force threshold required to move an object by lateral pushing, a rigidity prediction based on top-down poking. The paper builds on top of iMAP [1] and iLabel [2] for real-time neural scene representation and interactive labeling from few demonstrations. \n\n[1] iMAP: Implicit Mapping and Positioning in Real-Time\n[2] ILabel: Interactive Neural Scene Labelling\n ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1: In my opinion, the paper's main contribution comes from showing that fully autonomous interactive perception can be very effective in figuring out object properties with few interactions, for both rigid and deformable objects, such as clothes, even when neural scene representations are used. Active sensing is often tricky because random queries seem to catch up to active queries faster than hoped for, in some domains. It is informative to know that this is not the case here. \n\nS2: The use of the spectrometer is something that rarely appears in robotics literature, but seems to be very helpful in categorizing material types. Although the material-type classifier was pretrained on all materials, I think it enables a range of possible applications in manipulation, and more roboticists should be aware of these possibilities.\n\nS3: The paper is clearly written, with many descriptive figures, and an informative video. \n\nS4: The limitation section is thoughtful, honestly written, and extensive. I applaud the authors for doing this well.\n\nWeaknesses:\n\nW1: The paper claims that its neural representation can handle dynamically changing scenes, for example \"learning neural field representations in dynamic environments using dynamic-keyframe optimisation.\" This claim is slightly misleading, as the paper clarifies later on, as the history of old keyframes from iMAP gets dropped, and the neural representation is only valid if the changes in the scene are small, as is the case usually in the lateral pushing experiments. This is indeed clarified in section 3.6, but there is an incongruence between the claimed contributions in the introduction and the strict conditions under which they hold. This is a hard problem, understandably, but I would have appreciated more clarity about assumptions in the claimed contributions. \n\nW2: Interaction queries are generated only from the first viewpoint, and are not updated as the robot acquired more measurements. This slightly undermines the goal of the paper to perform fully active perception, since dynamic scene changes are not accounted for, but it is not a deal-breaker. It just means that the active query selection is not performed optimally.   \n \nW3: Entropy-based active perception is claimed as a contribution, but entropy has been used as an acquisition function for many learning systems and labeling tasks, including in robotics. While this is a key component of the paper, and seems to work well, I don't think it has to be claimed as a novel contribution. The fact that it works well is informative enough for the reader.  \n\nW4: Clarification needed, what do the error bars in Fig 5 represent? And why are the error bars for the random selection strategy not decreasing much over time?\n\nW5: Nitpick, but, it would be better if the 3d plots in Fig 1 were rotated to match the orientation of the scene.\n\nW6: It would be useful to mention the exact model of the spectrometer used in the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I find this paper very exciting, and I am recommending it be accepted because of strengths S1 and S2. I also very much enjoyed the fact that the paper was not limited to rigid objects. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I am hoping that the weaknesses I mentioned can be addressed easily during the revision period, as they are mostly related to presentation and writing, and do not require additional experiments. ", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "JOMss1ooFFS", "reviewer": "Reviewer_CQYj", "summary": "The paper proposes autonomous interactive experimentation with a scene in order to learn its properties, leveraging the ability of 3D neural field representations to generalise given as few as tens of examples. Experimental design is guided by entropy, maximising confidence under the constraints of the platform and environment. A workaround is included for the dynamics resulting from interactive experimentation. A generalisation of neural semantic labelling to continuous-valued semantics is included. The approach is demonstrated estimating rigidity, material, and frictional force distribution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper makes important and timely advances in the joint representation of geometry and semantics using neural fields. Several contributions combine to yield an impressive demonstration: experimental design under constraints, dealing with dynamics, and encoding continuous semantics work together to enable a novel autonomous robotic explorer. This is a clever use of point sensors, allowing complex properties like hyperspectral signatures to be sensed without need for more expensive snapshot hyperspectral cameras, or more time-consuming pushbroom scanning.\n\nThe literature review covers the main related works well and relates the contributions of the present work. The paper is well represented and contains sufficient detail to allow replication by a skilled practitioner.\n\nVisual results are included that qualitatively demonstrate and validate the approach. Quantitative results establish the efficacy of the experimental design with respect to naive approaches for segmentation and classfication accuracy. Force estimation is quantitatively compared to an analytic approach with privileged information, comparing favourably.\n\nWhile some limitations are discussed, the paper misses discussion of some key limitations of the method. The underlying 3D representation discards the directional dependence captured by NeRF, for example, meaning the method should be expected to fail around visually challenging reflective and refractive objects. A robot experimenter can accumulate a vast and always-growing repertoire of experimental results... how to deal with this growing dataset is not discussed.\n\nThe paper would be strengthened through release of source code and datasets.\n\nThe force estimation results show only one example, a more comprehensive demonstration with multiple object types and aggregate statistics would be more convincing.\n\nThe references incorrectly render at least two acronyms in lowercase.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well presented and makes multiple important and timely advancements that will be of interest to a substantial part of the robotics community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The authors may wish to expand on the limitations section as per above comments, and consider releasing code and exemplar datasets. It would have been stronger to include more experimental results around force estimation, this could perhaps form the basis for an extension of the work."}}, {"review_id": "7jtYLhJ5cS6", "reviewer": "Reviewer_jBNQ", "summary": "This paper proposes to leverage autonomous robots to explore and collect self-supervised data for training neural fields. Specifically, the authors show that neural fields can be self-supervised to predict materials and force distribution in a real-world setting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n \n**Originality**\n\nI think the paper has high originality. The prediction of force distribution is novel and interesting.\n \n**Quality**\n\nThe proposed method is technically sound and applied appropriately. The experiments are thorough and the video results look promising. \n \n### Weaknesses\n\n**Clarity**\n\nAlthough I think the paper is novel, several contributions listed in the Introduction are overclaimed, and not enough related works are cited:\n- Contribution 1 “the first fully autonomous, real-time neural scene labeling robot capable of operating in the real-world” This claim is simply not true. Both [2] and [3] and earlier works they cited have applied autonomous, interactive perception to various problems such as articulated parts discovery. I recommend authors read [4] and refine the claim.\n- Contribution 3 “dynamic-keyframe optimization” basically means discarding all the frames after a mode of interaction.\n- Contribution 4 “prediction of continuous-valued semantics using a neural-field model” I am not sure why this is a contribution. The most popular values predicted by neural fields such as RGBs and signed distances are all continuous.\n\n**Citation**\n\nFor the paragraph starting at line 78, I suggest authors also cite NICE-SLAM [1].\n\n**Typo**\n\n- Line 57: interactive perception, affordance learning and online scene understanding. -> interactive perception, affordance learning, and online scene understanding.\n- Line 136: The robot to focuses on -> The robot focuses on\n[1] NICE-SLAM: Neural Implicit Scalable Encoding for SLAM, Zhu et al.\n[2] Structure From Action Learning Interactions for Articulated Object 3D Structure Discovery, Nie et al.\n[3] Ditto Building Digital Twins of Articulated Objects from Interaction, Jiang et al.\n[4] Interactive Perception: Leveraging Action in Perception and Perception in Action, Bohg et al.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend Weak Accept but wish the authors to consider my opinion in the Weaknesses section and rephrase the contributions.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I wish the authors address my concerns about **Clarity** within the Strengths And Weaknesses section.\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)