Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presents data-driven exploration strategies for 3D reconstruction using visual and tactile data, a haptic simulator and a mesh-based 3D shape reconstruction model.

main_review: The paper addresses an interesting problem. However it can be improved considering the following points. Some of the listed previous works also focus on improving 3D reconstruction through tactile exploration, e.g. choosing where to touch next without exhaustive exploration also demonstrating experiments using real data. Please clarify and highlight the differences of your work in comparison. Another related work that can be included: Active tactile exploration with uncertainty and travel cost for fast shape estimation of unknown objects. The experiments are only conducted in simulation, how would the results be affected in terms of noise and uncertainty which would be increased in case of real data. The difference in performace in Table 1 looks quite small. Why is [48] chosen as a baseline among all related works, please motivate the choice, also motivate the choice of baselines for the exploration experiments. Could you report the value of K (number of touches/grasps) in experiments? Could you summarise the main findings from the experiments based on all the exploration strategies?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents an approach for active 3D reconstruction of objects based solely on touch, and optionally together with vision. The main idea is active perception -- estimate where to touch/see the object in order to better reconstruct it. The proposed method to solve this involves a neural network that fuses touch and vision signals to estimate 3D shape, and a policy that predicts where to touch next for best reconstruction performance. Data comes from a simulator that produces both touch and image signals (from the ABC dataset).
main_review: I would like to begin by listing several positives with this paper:

- It addresses an important problem (3D reconstruction) and tries to do so using a new active perception paradigm.
- The paper is well written and motivated.
- The experiments are extensive and convincing.

I like the ideas in the paper and the proposed approach. In the following, I will focus on comments, questions, and suggestions for improvement.

- It would be beneficial to differentiate between related previous work [48,59] and the proposed work. The main difference is active perception, but this can be made clearer.

- Figure 2 is missing some labels. What are the top and bottom rows?

- Regarding related work, there is some existing literature on the idea of "Shape from interaction" which would have been nice to discuss. In general, the paper could do a better job of crediting previous papers that present similar ideas. Here's a start:

Shape from interaction, Michel, Zabulis, Argyros, 2014

The paper uses charts for reconstruction but does not cite the most relevant paper:

A papier-mâché approach to learning 3d surface generation, Groueix et al. 2018

- Line 133 mentions 4 touch readings, but it isn't immediately obvious why it's 4 (mention 4 fingers of the robot).

- The simulator sounds great. Will this also be released publicly?

- Is the figure 4 legend correct? Red is denoted as vision and touch but takes only vision signals in the top part.

- I generally like the proposed approach and do not have too many complaints.

- Regarding results and experiments:

    - The results in table 1 show only a marginal improvement over [48], could the authors comment on why?
    - I would have liked to see results on real data, but I understand this is a hard problem.

Overall, this is a great paper that I would like to see published. If the authors can propose a plan to fix the issues above in the rebuttal, that would be great.
limitations_and_societal_impact: I appreciated the paper allocating significant space to discuss limitations and broader impacts. Not many of the papers I have seen do this.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces and explores the task of active 3D reconstruction using visual and haptic data. Instead of relying on static data to perform reconstruction, the learning process is guided by exploration, from which new data can be accounted for in the reconstruction process by the proposed model. In order to enable this setting/task, the authors develop a simulator that provides both visual and touch signals that are fed to the model, and an environment for training and evaluating policies.
main_review: Strengths:
- The authors introduce the task of active 3D reconstruction, which presents a different setting for reconstruction by leveraging exploration rather than static data, and develop a simulator and environment to enable the task.
- The authors extend the previously proposed chart-based reconstruction method to their setting to better leverage positional information, as well as incorporate additional information collected from exploration.
- The authors use 4 learning settings across a number of data-driven policies to evaluate their approach, including an oracle that represents an upper bound on the performance of the method. Experiments demonstrate the usefulness of active touch through the improved performance on 3D shape reconstruction of the learned policies.
- The paper is well-written, especially when providing an overview of all the policies and baselines used in the experiments.

Weaknesses:
- There are a few design decisions that were skimmed over (e.g., How was 50 selected to be the size of the action space? How was N (max # expected touch charts) determined?)
- The authors use mesh representations, in contrast to point cloud representations used by Smith et al. [48], which introduces additional parameterization with needing to learn an autoencoder to reduce the dimensionality of the features used in learning the policies. Since the autoencoder's objective is to minimize the CD between an input mesh and output point cloud, did the authors try directly using point clouds in their pipeline instead of meshes (i.e., the main difference from [48] being the composition of 2 GCNs for the mesh deformation model)?
- While Table 1 demonstrates better reconstruction in terms of CD, the variance is noticeably higher. Is there any intuition for why this may occur?

----------

Thank you to the authors for addressing and clarifying the concerns raised. My rating remains unchanged.
limitations_and_societal_impact: Yes, the authors have addressed the limitations and societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a novel active search on touch parameters to improve the 3D shape reconstruction with single (touch) or dual (touch & vision) modalities. The proposed reconstruction network and the active policy show good performance on simulated objects.
main_review: 1.	On the method originality
This paper mainly has two contributions which are the modified reconstruction network based on [48], and the active search policy of grasp parameters. The whole design is reasonable and novel to me. However, since the modification on the network and the policies are a little direct and intuitive to me, which makes the technical contribution a little limited.
2.	On the experiments
My major concern on the experiments is that they are all experimented in simulation. Since the object is easy to find or 3D printed in the real world, the robot hand (Allegro) is not that expensive (compared to Shadow robot hand), the tactile sensor is open-source for manufacturing with low-cost. And after [48] which has already put forward the discussion of vision-touch learning, I think at this moment, a real-world experiment is needed. I wonder if the simple policy will also work in the real world, the evident challenges would be natural occlusion of object shape, and the displacement after touching.

If the training requires fairly large amount of data, how about apply some simple sim2real strategy? As touch may involves many contact-rich processes, I am suspicious of how well the simulation can mimic the behavior of real touching.

If the pipeline only works with simulation, the applicability will be largely limited.

3.	On the writing
The writing of this paper is quite clear, especially the description about the difference on the reconstruction network. The whole work is easy to read and follow. 
4.	On the significance
The integration of vision and touch modalities are important for many applications, especially on robot. 3D shape reconstruction is a key problem when revealing the underlying 3D geometric information from the observations. I think it is a rather under explored area.
5.	Conclusion
Though I have some concerns about the technical contribution and the experiments, I also admit I may set a rather high bar. As the vision-touching learning problem is interesting, important and less explored, I am positive on this work.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: The paper proposes a learning-based approach to 3D reconstruction using a combination of (optional) visual and tactile measurements. A modified neural network takes as input visual and tactile observations and outputs an estimate of the object's 3D shape. This network is coupled with an active perception policy that chooses the next contact point to facilitate tactile-based reconstruction. The method is evaluated through a newly proposed simulator that provides synthetic tactile and visual measurements.

The reviewers agree that the idea of combining tactile observations with visual measurements in an active fashion is a compelling approach to object-scale 3D reconstruction. The paper is well written and easy to follow, and provides an extensive set of experiments that demonstrate the effectiveness of the approach. However, the experiments are limited to simulated data and thus it is not clear how well the framework would perform when operating with real-world visual and tactile measurements. The authors are encouraged to provide a more compelling discussion of the generalizability of the method and, ideally, include real-world experimental results.
Decision: Accept (Poster)