Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper describes a method for next-best view prediction for reconstructing large-scale 3D scenes with a depth sensor.  They derive a formula to estimate the surface coverage gain for any potential camera pose given a camera pose history and a probabilistic occupancy map.  They use one neural network to predict probabilistic occupancy map based on a point cloud input, and a second network to predict the visibility gain, which is used in the calculation of surface coverage gain.  The first network is trained to match the ground truth occupancy map, and the second is trained to match the ground-truth surface coverage gain.  Extensive experiments on synthetic datasets demonstrates an improvement in performance over SOTA.  
strengths_and_weaknesses: They provide a thorough derivation an approximation of surface coverage gain that they prove asymptotically approaches the true value.  This formulation is novel to the best of my knowledge.

The structure of the neural network and the loss functions is interesting.  They use separate networks to predict the probabilistic occupancy  map and the visibility gain functions.  The loss function for the visibility gain network is not visibility gain itself but the surface coverage gain.  They also use attention mechanisms to model occlusion effects.

They provide a thorough set of experiments to demonstrate their method leads to an improvement on ShapeNet following a standard protocol.

They also provide some ablation studies to establish the usefulness of various aspects of their proposed approach.

Overall, they have some interesting new ideas which lead to an increase in performance for NBV selection, and their work also can lead to new research such as handling noise in the depth map and selecting optimal paths rather than single viewpoints.






questions: L89: the expression (1- \lambda) c_pos + \lambda x seems to be interpolating between the camera position and the point, but it doesn't take the camera's viewing direction into account?
L101: "tubular" -- is this the right word?  or should it be "spherical"?  I couldn't figure out why the neighborhood region would be tubular.  Based on the expression in L105 it seems to be spherical (all points within a distance of \mu_o from x.


limitations: Limitations are discussed but not potential negative societal impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to solve the problem of next best view from partial point cloud towards a complete reconstruction. The method uses the principle of SDF  and define a way of computing the maximum coverage gain if a camera is at position "c" given the history of camera position, the visibility of the surface on those historical position. This in turn  expected to maximise the total coverage. The authors derive the relations of the incremental coverage gain mathematically and use a neural network to realise those relation to produce the coverage gain given a camera position "c". They show results on various dataset including large scale reconstruction.
strengths_and_weaknesses: Strengths:
The paper has a strong mathematical foundation on defining the coverage gain. The use of spherical harmonics for visibility gain is nice, though SH is being used in recent volume rendering work which authors also referred. Supplementary method provides all the derivation required for the proof of the relations used in the main paper. The paper shows interesting results.

Weaknesses
The paper is well grounded with the theoretical formulations. But the readability of the  paper is not very good. It is relatively difficult to relate the equations with the neural architecture for a reader. The concepts are philosophically mapped but the derivation is not clearly mapped with the neural method or in the inductive bias design. The paragraph from line 132 to136 attempted to give some clarity but this needs to be expanded for a general reader. Its not clear if there were neural aggregator instead of Monte Carlo, then what could happened? In table 1 the numbers are very close. What is the sensitivity of these numbers w.r.t the reconstruction?

questions: 1. Isn't the surface point could be obtained using ray intersection from camera in line 100? Or am I missing something?
2. Its not clear if the point cloud has a noise how it impact the visibility gain and occupancy function? Though authors mention this in the limitation, but the discussion on sensitivity of noise could have been good.
3. If the ablation is given with the reconstruction quality then it will be more clear to the reader about the significance of the numbers.
4. How the authors create the ground truth of the surface coverage gain?
5. If the explanation from Eq. 6 to the neural design is more lucidly written, then the readability will increase. 
6. Is the method tried in indoor? How this method is positioned w.r.t https://arxiv.org/abs/1805.07794
7. Is it possible to use the Pointnet features for encoding the 3D point and its neighbourhood in Fig 2?
8. How the occlusion is handled?
limitations: The limitation of this method is narrated by authors and some of the insight we can get from the above questions. For reproducibility, the crucial information regarding the training needs to be there in the main paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method for next-best-view prediction during three-dimensional surface reconstruction using depth sensors. A characteristic feature of the method is its scalability: most of the state-of-the-art methods are able to work only with artificial models from the ShapeNet dataset of very limited dimensions, while the proposed method is aiming to tackle large-scale structure-from-motion reconstructions such as the one of Colosseum.  The paper introduces a formal theoretical framework explaining that it is possible to estimate surface visibility metrics by sampling points in the volume around the surface, not necessarily strictly on the surface. The proposed method build on this framework. It is evaluated on ShapeNet and on a proposed dataset of large-scale models.
strengths_and_weaknesses: The paper develops a new theoretical framework, that explains an approach chosen by the authors to sample points for surface visibility prediction. In particular, the paper shows that it is possible to sample  points in the volume around the surface rather than on the surface directly. 

The paper proposes to use two models: one for occupancy prediction, and another one for point visibility estimation for a particular camera viewpoint. The first model for occupancy prediction is elegant and scalable, reminding of fully convolutional models for images, but working with point clouds. It shows an approach to build spatially localized models for occupancy prediction effectively working with local sub-sets of input points.

The approach with decomposing the problem into two models seems to be the key to achieving scalability of prediction. It is a distinct step forward in this field, because previous models assumed small object-like structures and simply predicted probabilities of choosing the next view from a pre-defined set of viewpoints evenly distributed on a sphere. The new method is significantly more powerful both in tackling arbitrary geometry and large scale models.


One of interesting questions possibly weakening the paper is that it is rather uncommon to use depth sensors for reconstructing large-scale objects. Usually people focus on Structure-from-motion or photogrammetry tools in these cases, and these tools rely on stereo algorithms rather than on the depth sensors. However, one may argue that a stereo method applied to a pair of images can be understood as a depth sensor.

In general, the theoretical framework developed in the paper is concerned with point sampling. However, when the method itself is explained, a particular approach to point sampling used in the method is not described well. This way, the theory leaves an impression of being rather disconnected from practice in this particular paper.

I see a conceptual difficulty embedded in a formula (8). However, the formula (8) defines dependency of point visibility prediction on point-camera direction. First of all, camera orientation does not enter the formula at all. It is unclear how to find the best orientation of the camera. Next, an infinite number of camera viewpoints lying on a ray have the same camera-point direction. It is unclear which viewpoint should one choose. Importantly,  occupancy status of the point with respect to the cameras lying on this ray may change. One may move along the ray further away, and if the ray intersects some surface, then the point will become occluded starting from that intersection. This seems not to be addressed by the current model. The method, as I see it, is based on a significant simplification: it downscales the space of camera viewpoints from the 5-dimensional space down to two dimensions. It would be very interesting to clarify this point.

It remains unclear, what kind of input does the method expect. Sometimes, we see a reference to a ‘depth sensor’. In the ShapeNet experiment, it remains unclear, what is given as input. I would really like to see some clarification in the paper, what particular type of depth sensor should to be used with this method.
questions: L67: Could you please explain the meaning of “volumetric methods are less efficient ..because they dilute information in the 3D space”

L75 this maps is not -> this map could be

Rephrase introduction and abstract to emphasize the input precisely (e.g., ‘a method taking a point cloud of points observed so far as input’). Right now the introduction only tells about a probabilistic occupancy map, referring that the perfect map is not known at run-time, but not describing, what is exactly considered to be known.

L78 ‘predict unknown geometry’ - What do these words refer to? 

L93: Maybe rename ‘knowledge factor’ to ‘knowledge indicator’ as it can only  be equal to 0 or 1.

In (4) dx -> dx_0

L122: the volume is supposed to be opaque - what property exactly is expected here, in more precise terms?

L177: How does the model sample 3D points using the predicted occupancy probability function?

In experiment 3.1, how are the partial point clouds generated?

L256-257: how can ray tracing be used to render a point cloud? What is done to make this rendering approximate specifically the LIDAR sensor, but not time-of-flight or active stereo depth sensors?

L292: ‘Model suffers ... to compute coverage gain’ - does it mean that the second model suffers to predict coverage gain from such a set of points?
limitations: The prediction of coverage gain is done in the 2D camera viewpoint space rather than in the 5D space

The input to the method is not clearly formalized

It is difficult to judge about the limitations of the current method in terms of accuracy. Would be nice to illustrate some cases when the method does not perform well, and reason about why it happens so.

Would be nice to see how accurate the occupancy prediction model is, how far can it extrapolate geometry, as I think it is the main property this model should have. The second model should decide where can it expect to see more geometry that was previously unseen. Essentially it means that the first model should do occupancy map completion. How well does the model perform in this taskm is a good question for an additional ablation study.
review_title: null

Meta-review: metareview: The paper describes an approach to next-best-view (NBV) planning for the reconstruction of large-scale 3D scenes using depth sensors. The proposed framework models the scene using a probabilistic occupancy map and chooses the next-best-view as the free camera pose that maximizes the gain in surface coverage. Integral to the approach's ability to handle large-scale scenes is the paper's formulation of surface coverage estimation as sample-based volumetric integration. Based on this formulation, the approach employs one neural network to predict the visibilities that are used to calculate surface coverage gain, and a second network to estimate the probabilistic occupancy map from the point cloud input. The paper presents experimental evaluations on the benchmark ShapeNet dataset as well as a proposed large-scale dataset, demonstrating gains over contemporary methods.

The paper was reviewed by three reviewers who read the author response and discussed the paper with the AC. The reviewers agree that the proposal to approximate surface coverage via sample-based volumetric integration, which is integral to the approach, is both novel and principled. To that end, the reviewers appreciate that the proposed architecture is well grounded in rigorous theoretical foundations. The experimental evaluation is thorough, with ablations that clearly demonstrate the advantages of the proposed architectural components. A key concern raised by several reviewers is that the readability of the submission is poor, which makes it difficult to relate the formal derivations to the neural network architecture. This lack of clarity lead to notable misunderstandings on the part of at least two reviewers. During the discussion phase, the reviewers acknowledged that the author response largely resolves this concern, but it is critical that the paper be updated to address these issues as well.
confidence: Certain
Decision: Accept