SCONE: Surface Coverage Optimization in Unknown Environments by Volumetric Integration
Authors: Antoine Guedon
Pascal Monasse
Vincent Lepetit
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computer Vision
3D reconstruction
Next Best View
3D scene exploration
Deep Learning

Abstract: Next Best View computation (NBV) is a long-standing problem in robotics, and consists in identifying the next most informative sensor position(s) for reconstructing a 3D object or scene efficiently and accurately. Like most current methods, we consider NBV prediction from a depth sensor like Lidar systems. Learning-based methods relying on a volumetric representation of the scene are suitable for path planning, but have lower accuracy than methods using a surface-based representation. However, the latter do not scale well with the size of the scene and constrain the camera to a small number of poses. To obtain the advantages of both representations, we show that we can maximize surface metrics by Monte Carlo integration over a volumetric representation. In particular, we propose an approach, SCONE, that relies on two neural modules: The first module predicts occupancy probability in the entire volume of the scene. Given any new camera pose, the second module samples points in the scene based on their occupancy probability and leverages a self-attention mechanism to predict the visibility of the samples. Finally, we integrate the visibility to evaluate the gain in surface coverage for the new camera pose. NBV is selected as the pose that maximizes the gain in total surface coverage. Our method scales to large scenes and handles free camera motion: It takes as input an arbitrarily large point cloud gathered by a depth sensor as well as camera poses to predict NBV. We demonstrate our approach on a novel dataset made of large and complex 3D scenes.

SCONE: Surface Coverage Optimization in Unknown Environments by Volumetric Integration
Antoine Guédon Pascal Monasse
Vincent Lepetit
LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, France
www.oinenggeenn,nnno www.acca..onaase..mmm monasse, vincent.
Abstract
Next Best View computation (NBV) is long-standing problem in robotics, and consists in identifying the next most informative sensor position(s) for reconstruct- ing a 3D object or scene efficiently and accurately. Like most current methods, we consider NBV prediction from a depth sensor like Lidar systems. Learning- based methods relying a volumetric representation of the scene are suitable for path planning. but have lower accuracy than methods using a surface-based representation. However, the latter do not scale well with the size of the scene and constrain the camera to a small number of poses. To obtain the advantages of both representations, we show that Carlo integration over a volumetric representation. In particular, we propose an approach, SCONE, that relies occupancy probability in the entire volume of the scene. Given any new camera pose, the second module samples points in the scene based their occupancy probability and leverages self-attention mechanism to predict the visibility of the samples. Finally, integrate the visibility evaluate the gain in surface coverage for the new camera pose. NBV is selected as the pose that maximizes the gain total surface coverage. Our method scales large scenes and handles free camera motion: It takes as input an arbitrarily large point cloud gathered by depth sensor well as camera poses predict NBV. We demonstrate our approach on novel dataset made of large and complex 3D scenes.
maximize surface metrics by Monte
neural modules: The first module predicts
Introduction
Next Best View computation (NBV) is long-standing problem in robotics 66 29 which consists in identifying the next most informative sensor position(s) for reconstructing 3D object or scene efficiently and accurately. Typically, a position is evaluated on how much it can increase the total coverage of the scene surface. Few methods have relied on Deep Learning (DL) for the NBV problem, even though DL can provide useful geometric prior to obtain a better prediction of the surface coverage 33 16 25 Like most current methods, we consider NBV prediction from a depth sensor. Existing methods based on a depth sensor rely either on volumetric or on a surface- based representation of the scene geometry. Volumetric mapping- -aased methods can compute collision efficiently, which is practical for path planning in real case scenarios 20
24 14 7 However, 5 for the scene, which results in poor accuracy in reconstruction and poor performance in NBV selection for complex 3D objects. the contrary, surface mapping-base
23
they typically rely on voxels or global embedding 12] 3 21 22
27
methods that process directly a dense point cloud of the surface gathered by the depth sensor are efficient for NBV prediction with digh-detailed geometry. They are however limited to very specific cases, generally a single, small-scale, isolated object with the camera constrained stay a sphere centered the object [[5] 24 Thus, they cannot be applied to the exploration of 3D scenes.
16 25
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Our Next Best View (NBV) method SCONE can handle unknown large-scale 3D scenes to produce accurate 3D reconstruction inside given 3D bounding box. Here, we call SCONE iteratively within naive path planning algorithm compute complete camera trajectory avoiding collisions and obtain a complete 3D model. Despite being trained only on ShapeNet 3D models, it generalizes to complex scenes as shown above. (input 3D model courtesy of Brian Trepanier, under CC License. Downloaded from Sketchfab.)
Figure 1:
As shown in Figure D we introduce volumetric DL method that efficiently identifies NBVs for unknown large-scale 3D scenes in which the camera can move freely. Instead of representing the scene with a single global embedding, we choose to use grid of local point clouds, which scales much better to large and complex 3D scenes. We show how to learn to predict the visibility of unseen 3D points in all directions given estimate of the 3D scene geometry. We can then integrate these visibilities in the direction of any camera by using Monte Carlo integration approach, which allows us optimize the camera pose to find the next most informative views. We call our method SCONE, for Surface Coverage Optimization in uNknown Environments.
In this respect, we introduce theoretical framework to translate the optimization of surface coverage gain, a surface metric on manifolds that represents the ability of a camera to increase the visible area of the surface, into an optimization problem volumetric integrals. Such formalism allows to use a volumetric mapping of geometry, which is convenient not only to scale the model to exploration tasks and scene reconstruction, but also to make probabilistic predictions about geometry.
In particular, given a partial point cloud gathered by a depth sensor, our model learns to make predictions with virtually infinite resolution about the occupancy probability in the scene volume by learning a deep implicit funetion [1
19]. Such predictions scale to very large point clouds since they depend only on neighborhood geometry. Then, our model leverages self-attention mechanism (26 to predict occlusions and compute informative functions mapped on continuous sphere that represent visibility likelihood of points in all directions. The occupancy probability field is finally used as a basis to sample points and compute Monte Carlo integrals of visibility scores.
28 18
Since NBV learning-based methods are mostly limited to single, small-scale, centered object recon- struction in literature, we first compare the performance of our model to the state of the art on the ShapeNet dataset (2), following the protocol introduced in [33]. While our method was designed to handle more general frameworks such as 3D scene reconstruction and continuous cameras poses in the scene, it outperforms the state of the art for dense reconstruction of objects when the camera is constrained to stay on a sphere centered on the object. We then conduct experiments in large 3D environments using a simple planning algorithm that builds a camera trajectory online by iteratively selecting NBVs with SCONE. Since, to the best of our knowledge, we propose the first supervised Deep Learning method for such free 6D motion of the camera, we created a dataset made of several large-scale scenes under the CC License for quantitative evaluation. We made our code and this dataset available for allowing comparison of future methods with SCONE on our project webpage: https://gi htt.b/b.tttuuut..// www./oot//t/oeeee Anttt//tttt//EEE
2
Approach
Let us consider depth sensor exploring 3D scene, at time step 2 0. Using its observations at discrete time steps j with 0 < j < t, the sensor has gathered a cloud of points distributed on the surface of the scene. We refer to this cloud the partial surface point cloud, it describes the part of the surface seen -or covered- by the sensor in the scene. To solve the NBV problem, we want to identify a camera pose that maximizes the coverage of previously unseen surface.
To this end, our method takes as input the partial surface point cloud as well as the history of 6D camera poses time steps j < t (i.e.all previous positions and orientations of the sensor). Our

---

Fartla Serfare
Presy Potete
Peoty Puttos
Smeed
Car au!
Pres der (h Hakey Roataor
chop hilacc
Gutad Companc Protuiiiles
Sacaaa You Billy to
af (S
Thene
Tranderfie
Trandhefane
Figure 2: The main steps of our method SCONE. At time t, the depth sensor has visited camera poses H = {oj.j S f and gathered a partial surface point cloud Pи on the true surface, shown in red in the left image. Using its first neural module, our model predicts from the real surface points PH an occupancy probability distribution over proxy points (I1,..., n,) shown in the middle. The points (Z1, ) are sampled uniformly in the scene; refer them as proxy points because use them to encode the volume. For readability, the figure does not show the proxy points with an occupancy value under 0.5. To compute the coverage gain of any next camera pose 1 the model samples a subset of proxy points x; in the field of view of c--1 and uses its second module to predict the visibility gain in direction C++1 2; for each point 2y, as shown the right. The proxy points sampled with probabilities proportional to their occupancy value. Moreover, for each proxy point 2i, the second module encodes the relative positions of previous cameras with specific features hr (z:)). We finally integrate visibility gains over proxy points in the field of view of c++1 to approximate the volumetric coverage gain integral appearing in Equation 6
approach is built around two successive steps, each relying on dedicated neural module as shown in figure First, we make a prediction about the geometry of the scene, to estimate where the uncovered points could be. Then, we predict the visibility gain of uncovered points from any new camera pose; The NBV is finally selected as the camera with the most new visible points in its field of view.
Although we seek to maximize a surface metric such as surface coverage gain, our method relies on volumetric representation of the object or scene reconstruct. In this regard, we show that we can maximize a surface metric by integrating over a volumetric representation with virtually infinite resolution. As we argue below, such a representation is not only useful for collision-fre robot navigation but is also much more efficient for optimizing surface coverage gain than the alternative of identifying the 3D points lying on the surface, which is difficult in an unknown and occluded environment. More exactly, we derive volumetric integral that is asymptotically proportional to the surface coverage gain metric, which is enough for our maximization problem.
In the following subsection, we first present this derivation considering volume represented as perfect binary occupancy map. We then present the two neural modules of SCONE and explain how we use them to predict all terms involved in the volumetric integral, by leveraging neural models and self-attention mechanisms to predict occupancy and occlusions.
2.1 Maximizing Surface Coverage Gain Binary Occupancy Map
Here, we consider a binary occupancy map a : R3 - (0. 1) representing the volume of the target object scene. We will relax our derivations a probabilistic occupancy map when looking for the next best view in the next subsections. From the binary map a, we can define the set of occupied points, i.e., the of points verifying (xx 1, its surface the boundary 0x, and the surface coverage C(c) achieved by a camera pose e = (Cposs Gra)) E C. R x SO(3) as the following surface integral:
Taxls Jo we(x) dr,
1
C(c)
where laxls == -Jox dr is the area of surface Ox. Xe C C is the subset of occupied points contained in the field of view of camera e, and v.(xr) is the visibility of point from camera e, iee, ve(x) = xx (x) 1 (a (1(1- A)epos + Ar such that À E [0,1)]) {0}).
3

---

Since we want to maximize the total coverage of the surface by all cameras during reconstruction. we are actually interested in maximizing the coverage of previously unobserved points rather than the absolute coverage. Given set of previous camera poses, which we call the camera history H C C, and a 3D point æ; we introduce the knowledge indicator ?H RR3 + (0, 1] such that TH (z) = max (v) c E H.. We then define the coverage gain G (c)) of camera pose c as:
/ Bx
GH(C)= - Texls Toxls
voa (x) dr,
(2)
where (x) (1 (1-21) (x)) (x) is the visibility gain of x in Xc. for camera history H. This function is equal to 1 iff x is visible at pose but was not observed by any camera pose in H. Given a camera history H, our goal to identify a pose e that maximizes (c)
Given an occupancy map a, we could evaluate the integral in Eq. 2 by simply sampling points on surface 0x. However, in practice we will estimate the occupancy map iteratively in an unknown environment, and we will only have access to an occupancy probability distribution. Extracting surface points from such a probabilistic occupancy map gives results that can differ lot from the true surface: Indeed, in 3D, a surface acts as a very concentrated set with zero-measure, and requires high confidence to give meaningful results. Instead of extracting surface points, we extend the properties of such points to a small spherical neighborhood of the surface. This will allow us to replace the maximization of a surface metric by the maximization of volumetric integral, which is much easier to compute from our volumetric representation.
More exactly, we assume there exists a quantity Mo O such that any volume point in the spherical neighborhood T(xx, 14) == {p E IR3 Ar E ox, || pll2 < hO} keeps the same visibility property as its neighboring surface points. With such hypothesis, we give a thickness to the surface, which makes sense when working with discrete points sampled in space to approximate a volume.
To this end, we introduce new visibility gain function 9. to adapt the definition of the former visibility gain ve" For any < μ < Mo:
H ge (1:: x) = {
if Exo E 0x À such that = xo + AN and uf (ro) = 1, otherwise.
where N is the inward normal vector field. With further regularity assumptions about the surface that are detailed in the appendix, such quantities are well defined. Assuming p4 is small enough, the following explicit formula translates the surface approach into a volume integral for any camera pose e E C and j < Pn:
S ge (u;z)dx= 1 J"h -H ge (Jz; 20 + XN(r))) det (T AW., dd dro,
(4)
with Wzo the Weingarten map at 2o, that is, the Hessian of the signed distance function on the boundary of X. which is continuous on the scene surface, assumed to be compact [10]
By developing the determinant, we find that det(1 - >,,.) = + Ab(A, z0) where b is a bounded function on the compact sp
i-H, μ4| x 0x. Moreover, for all zg 0x. have by definition
9 (nt; Do + AN(x))) = -: (p:: rD) = 1 (Io) when 0 < X < μ. and " (p: + AN(z))) = 0
when -M X < O. It follows that, for every 0 < p no:
S Lon Be De (15200( + Ab(A, dA
ge (1;=)do=
H
= J get gf (u5 (oo dro / 1" dg (4; di (5)
Jox
= + J dge (1naa))(A) 2o) dd ddd
The complete derivations are given in the appendix.
Function g.d (μ:: -) is naturally equal to 0 for every point outside T(ax, H). Moreover, considering the regularity assumptions we made on the compact surface, if po is chosen small enough then for all zy e 0x, u < no. the point zo + pNN(z)) is located inside the volume, such that ST(ox of (μ;z:) dx
J - (ii x)) dr. Since ! (p:: -) & for all c E C and > O, we deduce the following theorem by
bounding [b] on [-H, m] x 0x:

---

Pir
Self-Attention Unit - Pooling glPnr
f(x]
Proling ma
Concatecation
INON
SA + Poodiig
(2, Ply
MLP
((R)
Figure 3: Architecture of the first module of SCONE, which predicts the occupancy probability field ö. This module predicts the occupancy probability of a point x from several inputs: The point æ after transformation by MLP fxx); A coarse global encoding g(PH) ) of the point cloud obtained by applying self-attention units on the sequence of points followed by a pooling operation; Multi-scale neighborhood features (x, Px), ) 0, .... N computed by down-sampling multiple times the point cloud, and encoding the & nearest neighbors of x with self-attention units after each sampling.
Theorem 1. Under the previous regularity assumptions on the volume x of the scene and its surface 0x there exist Po > and M > O such that for all μ < po. and any camera e E C:
1 xxlv / dl (1:z)dr - laxls Gn(() S SM2²
(6)
Ixlv
where !xlv is the volume of X-
This theorem states that, asymptotically for small values of jr, the volume integral fx 9e (1; dr gets proportional to the surface coverage gain values GH (c) that we want to maximize. This result is convenient since volume integral can be easily approximated with Monte-Carke integration on the volume and a uniform dense sampling based on the occupancy function a. Consequently, the more points we sample in the volume, the smaller μ integral of spherical neighborhood visibility gain gets to maximizing the surface coverage gain.
choose, and the closer maximizing the volume
2.2 Architecture
To approximate the volumetric integral in Equation 6 for any camera pose c, we need to compute Xe as well as function gf In this regard, we need to compute both the occupancy map and the visibility gains of points for any camera pose. Since the environment is not perfectly known, we predict each one of these functions with dedicated neural module. The first module takes as input the partial point cloud gathered by the depth sensor to predict the occupancy probability distribution. The second module takes as input a camera pose, a feature representing camera history as well as a predicted sequence of occupied points located in the camera field of view to predict visibility gains.
Predicting the occupancy probability field ō. The occupancy function a is not known perfectly in practice. To represent occupancy, most volumetric NBV methods rely on memory-heavy repre- sentations (like an occupancy 3D-grid or volumetric voxelization), that are generally less efficient for encoding fine details and optimizing dense reconstructions, and will necessarily downgrade the resolution compared to a point cloud directly sampled on the surface. To address this issue while still working with a volumetric representation of the scene, use a deep implicit function to encode the 3D mapping of occupancy efficiently. Such a function has a virtually infinite resolution, and prevents us from saving large 3D grid in memory. We thus approximate with the first module of our model, which consists of a neural network ē P P(R3) x R - [0. 1] that takes as inputs a partial surface R and outputs the occupancy probability ò (Prr; x) of &. Pir is obtained by merging together all depth maps previously captured from cameras in H.
point cloud PH R² and query points
As shown in Figure 3 rather than using a direct encoding of the global shape of Ри as input to é, we take inspiration from [ to achieve scalability and encode PH using features computed from the points' neighborhoods. The difference with [9] is that rely given query 3D point x, these features are computed from the nearest neighbors r computed at different scales. For each scale s, we downsample point cloud Pи around into a sparser point cloud pl+) before recomputing the nearest neighbors p. (x), = 1,... & of z: In this way, the size of the neighborhood increases with scale.
multiscale approach: For
5

---

Next, for each value of we use small attention units 226 I on the sequence of centered neighbor-
hood points (pi (+) (x)) -2, ( (x) x) and apply pooling operations encode each sequence of k neighbors into a single feature that describes the local geometry for the corresponding scale. We finally concatenate these different scale features with another uncentered global feature as well the query point 2, and feed them to an MLP to predict the occupancy probability. The last global feature aims to provide really coarse information about the geometry and the location of x in the scene.
This model scales well to large scenes: Adding points from distant views to the current partial point cloud does not change the local state of the point cloud. To avoid computing neighborhoods on the entire point cloud when reconstructing large scenes, we partition the space into cells in which we (5) store the points in PH. Given query point x, we only use the neighboring cells to compute p. (z).
Predicting the visibility gain s² To maximize surface coverage gain, we need to compute the volumetric integral of visibility gain functions gf We do this again by Monte Carlo sampling, however, in unknown environments functions Hf since the geometry, represented as a point cloud, is partially unknown and sparser than true surface. We thus train the second module of our model to predict visibility gain functions by leveraging a self-attention mechanism that helps to estimate occlusion effects in the point cloud Ри-
cannot compute explicitly occlusions to derive visibility gain
In particular, for any camera pose E C and 3D point I E Xe. the second module derives its prediction of visibility gains from three core features: (i) The predicted probability (PP; x)) of x to be occupied, (ii) the occlusions on by the subvolume X and (iii) the camera history H. To feed all this information to our model in an efficient way, we follow the pipeline presented in Figure 4 The model starts by using the predicted occupancy probability function ē to sample 3D points in the volume X. These samples will be used for Monte Carlo integration. We refer to these points as proxy points
use them to encode the volume in the camera field of view, i.e., in a pyramidal frustum view. We write X as the discrete set of sampled proxy points, and X as the set of proxy points located in the field of view of the camera c.
We first encode these proxy points individually by applying a small MLP on their 3D coordinates and their occupancy probability value concatenated together. Then, our model processes the sequence of these encodings with a self-attention unit to handle occlusion effects of subvolume Xe on every individual point. Note there is pooling operation on the output of this unit: The network predicts per-point features and does not aggregate predictions, since we do it ourselves with Monte Carlo integration. Next, for each proxy point 2 E X we compute an additional feature h (1) that encodes the history of camera positions H with respect to this point as a spherical mapping: It consists in sphere centered on z of all camera positions for which was in the field of view. These features are concatenated to the outputs of the self-attention unit.
the projection
Our model finally uses an MLP on these features to predict the entire visibility gain functions of every point J as a vector of coordinates in the orthonormal basis of spherical harmonics. With such a formalism, the model is able to compute visibility gains for points inside a subvolume in all directions with single forward pass. In this regard, we avoid unnecessary computation and are able to process a large number of cameras in the same time when they share the same proxy points in their field of view (e.g., reconstruction of a single object centered in the scene, where Xe = X for all c, or when several cameras observe the same part of the 3D scene, i.e., Xe = X' c X for several c).
Formally, if we denote by Y,M : S2 + R the real spherical harmonic of rank (4, m) and 67" (Xei x. hus (x)) the predicted coordinate of rank (I, m) for proxy point x E Xe with attention to subset Ea and camera history feature h (x), the visibility gain of point æ in direction d E 5 is defined as
Σ of Zeizz (z())) Y (d) I.m
so that the coverage gain GH(e) of a camera pose c e is proportional to
(c(()= =
Σ TE
1 (x:) Σ
Lor of (oi (x)) ym
x Gpas
(8)
||
Cpos l12
1,
The next best view among several camera positions is finally defined as the camera pose e* with the highest value for I (c"). Equation
8 is a Monte Carlo approximation of the volumetric integral in

---

Tebllt Gis
Tiskord Derapesd Frehesdig
h
Druaveetice
mosaliy Carare faltery thoy Pein
Selection
MEPP
NLPa
NI Iulegd
Ki
Za
Photo Peat
Octapuncy Pribubiitt
Caneagg Gate
Figure 4: Architecture of the second module of SCONE, which predicts a visibility gain for each proxy point. To make this prediction, the model encodes the proxy points concatenated with their occupancy probability ā (x). We use an attention mechanism to take into account occlusion effects in the volume between the proxy points and their consequences the visibility gains.
Equation 6 where the occupancy map and the visibility gains are predicted with neural networks. We choose to use a Monte-Carke integral rather than a neural aggregator because this approach is simple, fast, makes training more stable, has good performance, better interpretability, and can handle sequences of arbitrary size. In particular, it implicitly encourages our model to compute meaningful visibility gains for each point since there no asymmetry between the points.
We also use spherical harmonics to encode the camera history encoding h#(2) makes a homogeneous input the predicted output. Consequently, this input comes the end of the architecture, and aims to adapt the visibility gain according to previous camera positions. This convenient representation, inspired by 32). allows us handle free camera motion; On the contrary,
h (x) ( of each point which
several models in the literature encode the camera directions on a discrete sphere B3 16 25].
Training. We train the occupancy probability module alone with a Mcan Squared Error loss with the ground truth occupancy map. We do not compute ground- -ruth visibility gains to train the second module since it would make computation more difficult and require further assumptions: We supervise directly on I (c) by comparing
the ground-trui surface coverage gain for multiple cameras, with softmax normalization and Kullback-Leible divergence loss. Extensive details about the training of our modules and the choices we made are given in the appendix.
3 Experiments
As discussed in the introduction, deep learning-based NBV models for dense reconstruction are currently limited to single, scall-scale, centered object reconstruction. To compare SCONE to these previous methods in this context, we first constrain the camera pose to lie on sphere centered on an object. We then introduce our dataset made of 13 large-scale 3D models that we created to evaluate SCONE on free camera motions (3D models courtesy of Brian Trepanier, Andrea Spognetta, and 3D Interiors, under CC License; models were downloaded from the website Sketchfab).
3.1 Next Best View for Single Object Reconstruction
We first compare the performance of our model to the state of the art on subset of the ShapeNet dataset (2) introduced in [33 and following the protocol of [33 We sample 4,000 training meshes from specific categories of objects, 400 validation meshes and 400 test meshes from the same categories, and 400 additional test meshes from 8 categories unseen during training.
The evaluation on the test datasets consists of 10-view reconstructions of single objects. Given mesh in the dataset, camera positions are discretized on a sphere. We start the reconstruction process by selecting a random camera pose, then we iterate NBV selection 9 times in order to maximize coverage with a sequence of 10 views in total. The evaluation metric is the area under the curve (AUC) of surface coverage throughout the reconstruction. This criterion not only evaluates the quality of final surface coverage, but also the convergence speed toward satisfying coverage. Results are presented Table [ Further details about the evaluation, the estimation of ground truth surface coverage gains, and the metric computation available in the appendix.
7

---

Table 1: AUCs of surface coverage for several NBV selection methods for dense object re- construction, as computed on the ShapeNet test dataset following the protocol of [33]. and after averaging over multiple seeds in the case of our method. For this experiment, we constrain the camera to stay on sphere centered on the objects in order to compare with previous methods. Even if our model designed scale entire scene reconstructions with free camera motion, it is still able to beat other methods trained for the specific case of dense object reconstruction with constrained camera motion.
Categories seen during training
Method
Airplane
Cabinet
Car
Chair
Lamp
Sofa
Table
Vessel
Mean
Random
0.745
0.545
0.542
0.724
0.770
0.589
0.710
0.674
0.662
Proximity Count S
0.800
0.596
0.591
0.772
0.803
0.629
0.753
0.706
0.706
Area Factor 23
0.714
NBV-Net
0.778
0.576
0.596
0.743
0.791
0.599
0.693
0.667
0.680
PC-NBV
33
0.799
0.612
0.612
0,782
0.800
0.610
0.760
0.719
0.716
(Ouns)
0.734
Categories
training
Method
Bus
Bed
Bookshelf
Bench
Guitar
Motorbike
Skateboard
Pistel
Mean
Random
0.609
0.619
0.695
0.795
0.795
0,672
0.768
0.614
0.694
Proximity Count
0.646
0.645
0.749
0.829
0.854
0.705
0.828
0.660
0.740
Area Factor
0.629
0.631
0.742
0.827
0.852
0.718
0.799
0.660
0.732
0.710
0.751
PC-NBV
0.667
0.662
0.740
0.845
0.849
0.728
0.840
0.672
0.750
SCONE (Ours)
0.694
0.689
0.746
0.832
0.860
0.728
0.845
0.717
0,764
(a) Dunnottar Castle
(b) Pantheon
(c) Statue of Liberty
(d) Leaning Tower, Pisa
(e) Fushimi Castle
(f) Alhambra Palace
(g) Eiffel Tower
(h) Natural History Museum
Application of our NBV approach to the reconstruction of large 3D structures. The path, constructed iteratively in real time with our method, is shown in red (3D models courtesy of Brian Trepanier, Andrea Spognetta, and 3D Interiors, under CC License; all models were downloaded from the website Sketchfab).
Figure 5:
3.2 Active View Planning in 3D Scene
To evaluate the scalability of our model to large environments as well as free camera motion in 3D space, we also conducted experiments using a naive planning algorithm that incrementally builds a path in the scene: We first discretize the camera poses in the scene on a 5D grid, corresponding to coordinates Cpos = (xe: Ve, Ze) of the camera as well as the elevation and azimuth to encode rotation Crot. The number of poses depends on the dimensions of the bounding box of the scene. This box is an input to the algorithm, as way for the user to tell which part of the scene should be reconstructed. In our experiments, the number of different poses is around 10,000. Note we did not retrain our model for such scenes and use the previous model trained on ShapeNet as explained in Section 3.1
The depth sensor starts from a random pose (from which, at least, part of the main structure to reconstruct is visible). Then, at each iteration, our method estimates the coverage gain of direct neighboring poses in the 5D grid, and selects the one with the highest score. The sensor moves to this position, captures a new partial point cloud and concatenates to the previous one. Since we focus on online path planning for dense reconstruction optimization and designed our model to be scalable
8

---

(a) Dunnottar Castle
(b) Pantheon
(c) Statue of Liberty
(d) Leaning Tower, Pisa
(c) Fushimi Castle
(f) Alhambra Palace
(g) Eiffel Tower
(h) Natural History
Museum
Figure 6:
Reconstruction large 3D scenes with our NBV approach SCONE. We show the
partial point cloud gathered by the depth sensor for each scene.
3D scene
Statee of Libert
Manhuttas Bridge
Fudini Cadle
Duntotu Casée
Noochwarstii Cusll:
Method
Coloszun
Mean
Random Walk
0.308
0.469
0.405
0.584
0.355
0.403
0,421
SCONE-Entropy
0.512
0.681
0.361
0.802
0.456
0.538
0.558
Renden Mark
SCONE
0.571
0.693
0.685
0.841
0.739
0.653
0.697
Nucder mande
(a)
(b)
Figure 7: (a) AUCs of surface coverage for several NBV selection methods during active view planning experiments on several 3D scenes from our dataset. Results are averaged on several trajectories. (b) Surface coverage curves for the different NBV selection methods, averaged on all 13 3D scenes. Despite being trained only on centered ShapeNet 3D models, the second module of SCONE is able to generalize complex scenes when predicting the visibility scores.
with local neighborhood features, the size of the global point cloud can become very large. We iterate the process either 100 times to build a full path around the object in around 5 minutes on a single Nvidia GTX 1080 GPU, and recovered partial point cloud with up 100,000 points.
Contrary to single, small-scale object reconstruction where the object is always entirely included in the field of view, simulated limited range in terms of field of view and depth for the depth sensor, so that the extent of the entire scene cannot be covered in a single view. Thus, taking pictures long range and going around the scene randomly is not efficient strategy; the sensor must find path around the surface to optimize the full coverage. We use ray-casting renderer to approximate a real Lidar. More details about the experiments can be found in the appendix.
We compared the performance of SCONE with simple baselines: First, random walk, which chooses neighboring poses randomly with uniform probabilities. Then, we evaluated an alternate version of our model, which we call SCONE-Entropy, which leverages the first module our full model to compute occupancy probability in the scene, then selects the next best view as the position that maximizes the Shannon entropy in its field of view. The comparison is interesting, since SCONE- Entropy adapts classic NBV approaches based on information theory deep learning framework. Figures and provide qualitative results, Figure 7 shows the convergence speed covered surface by SCONE and our two baselines, averaged on all scenes of the dataset.
Despite being trained only on ShapeNet 3D models, our method able to compute meaningful paths around the structures and consistently reach satisfying global coverage. Since we focused on building a metric for NBV computation, this experiment is simple and does not implement any further prior from common path planning strategies: For instance, we do not compute distant NBV, nor optimal paths to move from a position to distant NBV with respect to our volumetrie mapping. There no doubt the model could benefit from further strategies inspired by the path planning literature.

---

Architocture
Mcan Squared Error
IoU
Architecture
Kullback-Leibler Divergence
Base Archilecture
0.0397
0.843
Base Architecture
0,00039
Neighbothood Feature
0,0816
0,702
No prediction
0.00051
With Camera History
0,0386
0,844
No Camera History
0.00045
(a) Occupancy probability
(b) Coverage gain
Figure 8: (a) Comparison of Mean Squared Error and IoU for variations of occupancy probability prediction model. (b) Comparison of Kullback-Leible divergence for variations of our coverage gain computation model. Thanks the volumetric prediction o, the full model predicts a better distribution of coverage gains on the whole space as it is indicated by KL divergence, which is convenient for full path planning in a 3D scene.
3.3 Ablation Study
We now provide an ablation study for both modules of our full model: The prediction of occupancy probability, and the computation of coverage gain from predictions about visibility gains. To evaluate both modules separately, we compared their performance on their training critera: MSE for occupancy probability, and softmax followed by KL-Div on a dense set of cameras for coverage gain. The values are reported in Figure 8 We provide extensive details and analysis in the appendix.
Occupancy probability. Apart from the base architecture presented in Figure 3 trained the first module under two additional settings. First, we removed the multi-scale neighborhood features com- puted by downsampling the partial point cloud PH, and trained the module to predict occupancy probability value ā only from global feature g(Pr) and encoding fx)) As anticipated, the lack of neighborhood features not only prevents the model from an efficient scaling large scenes, but also causes huge loss in performance. We also trained a slightly more complex version of the module by feeding it the camera history harmonics h (ar) an additional feature. It appears this helps the model to increase its confidence and make better predictions, but the gains are quite marginal.
Visibility gain. We trained two variations of our second module. First, completely removed the geometric prediction: We use directly the surface points as proxy points, mapped with an occupancy probability equal to 1. As consequence, the model suffers from significant loss in performance to compute coverage gain from its predicted visibility gain functions. Thus, we can confirm that the volumetric framework improves the performance of SCONE. This result is remarkable, since surface representations usually make better NBV predictions for dense reconstruction of detailed objects in literature. On the contrary, we show that our formalism is able achieve higher performance by leveraging a volumetric representation with a high resolution deep implicit function ã. We trained second variation without the spherical mappings hg of camera history. We verified that such additional features increase performance, especially at the end of 3D reconstruction.
Limitations and Conclusion
Beyond the prediction of the Next Best View, method SCONE is able to evaluate the value of any camera pose given a current estimate of the scene volume. We demonstrated this ability with a simple path planning algorithm that looks for the next pose around the current pose iteratively. However:
Like current methods, SCONE relies on a depth sensor and assumes that the captured depth maps are perfect. In practice, such a sensor is not necessarily available and can also be noisy. It would very interesting to extend the approach to color cameras.
Also like current methods, it is limited to the prediction of the camera pose for the next step. This is greedy, non-optimal as multiple poses are required anyway for a complete 3D reconstruction. Thanks to its scalability, believe that our method could be extended to the prediction of the "Next Best Path", where future camera poses would be predicted jointly for their complementarity.
Acknowledgements
This work was granted access to the HPC resources of IDRIS under the allocation 2022-AD01 1013387 made by GENCI. We thank Renaud Marlet, Elliot Vincent, Romain Loiseau, and Tom Monnier for inspiring discussions and valuable feedback.
10

---

References
[1] Fredrik Bissmarck, Martin Svensson, and Gustav Tolt. Efficient Algorithms for Next Best View Evaluation. In International Conference Intelligent Robots and Systems, 2015.
[2] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rics 3D Model Repository. Technical report, Stanford University, Princeton University, Toyota Technological Institute at Chicago, 2015.
[3] Benjamin Charrow, Gregory Kahn, Sachin Patil, Sikang Liu, Kenneth Goldberg, Pieter Abbeel, Nathan Michael, and Vijay Kumar. anformation-Therretiie Planning with Trajectory Optimiza- tion for Dense 3D Mapping. In Robotics: Science and Systems, 2015.
[4] S.Y. Chen and Youfu Li. Vision Sensor Planning for 3-D Model Acquisition. IEEE Transactions on Systems, Man, and Cybernetics, 2005.
[5] Titus Cieslewski, Elia Kaufmann, and Davide Scaramuzza. Rapid Exploration with Multi- Rotors: Frontier Selection Method for High Speed Flight. In International Conference on Intelligent Robots and Systems, 2017.
[6] C. Ian Connolly. The Determination of Next Best Views. In International Conference Robotics and Automation, 1985.
[7] Jonathan Daudelin and Mark Camphell. An Adaptable, Probabilistic, Next Best View Algorithm for Reconstruction of Unknown 3D Objects. IEEE Robotics and Automation Letters, 2017.
[8] Jeffrey Delmerico, Stefan Isler, Reza Sabzevari, and Davide Scaramuzza. A Comparison of Volumetric Information Gain Metries for Active 3D Object Reconstruction. Autonomous Robots, 2018.
[9] Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Michael Wimmer, and Niloy Mitra. Points2Surf: Learning Implicit Surfaces from Point Cloud Patches. European Conference on Computer Vision, 2020.
[10] David Gilbarg and Neil S. Trudinger. Elliptic Partial Differential Equations of Second Order. Classics in Mathematies. Springer Berlin Heidelberg. 2001.
[11] Meng-Hao Guo, Junxiong Cai, Zheng-Ning Liu, Tai-Jian Tai-Jiang Mu, Ralph Martin, and Shi-Min Hu. PCT: Point Cloud Transformer. Computational Visual Media, 2020.
[12] Benjamin Hepp, Debadeepta Dey, Sudipta N. Sinha, Ashish Kapoor, Neel Joshi, and Otmar Hilliges. Learn-to-Score: Efficient 3D Scene Exploration by Predicting View Utility. In European Conference Computer Vision, 2018.
[13] Simon Kriegel, Tim Bodenmüller, Michael Suppa, and Gerd Hirzinger. A Surface-Based Next-Best-View Approach for Automated 3D Model Completion of Unknown Objects. In International Conference on Robotics and Automation, 2011.
[14] Simon Kriegel, Christian Rink, Tim Bodenmüller, Alexander Narr, Michael Suppa, and Gerd Hirzinger. Bext-Best-Scan Planning for Autonomous 3D Modeling. International Conference on Intelligent Robots and Systems, 2012.
[15] Inhwan Dennis Lee, Ji Hyun Seo, Young Min Kim, Jonghyun Choi, Soonhung Han, and Byounghyun Yoo. Automatic Pose Generation for Robotic 3-D Scanning of Mechanical Parts. JEEE Transactions on Robotics and Automation, 2020.
[16] Miguel Mendoza, Juan Irving Vasquez-Gomez, Hind Taud, Luis Enrique Sucar, and Carolina Reta. Supervised Learning of the Bext-Best-View for 3D Object Reconstruction. Computing Research Repository, 2019.
[17] Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space, International Conference Computer Vision and Pattern Recognition, 2019.
11

---

[18] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. European Conference on Computer Vision, 2020.
[19] Michael Oechsle, Songyou Peng. and Andreas Geiger. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-Viee Reconstruction. International Conference on Computer Vision, 2021.
[20] Christian Potthast and Gaurav Sukhatme. A Probabilistic Framework for Next Best View Estima- tion a Cluttered Environment. Journal of Visual Communication and Image Representation, 2014.
[21] Soohwan Song and Sungho Jo. Online Inspection Path Planning for Autonomous 3D Modeling using Micro-Aerial Vehicle. In International Conference on Robotics and Automation, 2017.
[22] Soohwan Song, Dackyum Kim, and Sungho Jo. Online Coverage and Inspection Planning for 3D Modeling. Autonomous Robots, 2020.
[23] Juan Vasquez-Gomes Luis Sucar, Rafael Murricta-Gid, and Efrain Lopez-Damian. Damian. Volumetric Next-Best-Vii View Planning for 3D Object Reconstruction with Positioning Error. International Journal of Advanced Robotic Systems, 2014.
[24] Juan Vusquez-Gomez, Luis Sucar, and Rafael Murrieta- Murrieta-Cid. View/State Planning for Three- Dimensional Object Reconstruction under Uncertainty. Autonomous Robots, 2017.
[25] Juan Vasquez-Gomez, David Troncoso Romero, Israel Becerra, Luis Sucar, and Rafael Murricta- Cid. bext-best-view Regression using a 3D Convolutional Neural Network. Machine Vision and Applications, 2021.
[26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances in Neural Information Processing Systems, 2017.
[27] Chaoqun Wang, Han Ma, Weinan Chen, Li Liu, and Max Meng. Efficient Autonomous Exploration With Incrementally Built Topological Map in 3-D Environments. Transactions on Instrumentation and Measurement, 2020.
[28] Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomír Mech, and Uirich Neumann. DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction. Advances in Neural Information Processing Systems, 2019.
[29] Brian Yamauchi. A Frontier-Basse Approach for Autonomous Exploration. In International Conference on Robotics and Automation, 1997.
[30] Lior Yariv, Matan Atzmon, and Yaron Lipman. Universal differentiable renderer for implicit neural representations. arXiv Preprint, 2020.
[31] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume Rendering of Neural Implicit Surfaces. Advances in Neural Information Processing Systems, 2021.
[32] Alex Yu, Sara Fridovich-Ken Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. International Conference on Computer Vision and Pattern Recognition, 2022.
[33] Rui Zeng, Wang Zhao, and Yong-Jin Liu. PC-NBV: A Point Cloud Based Deep Network for Efficient Next Best View Planning. In International Conference on Intelligent Robots and Systems, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the last section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] A
discussion can be found in the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] Not yet, but we
will release our source code and dataset.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] Our data do not have such issue.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yuBntlB-rcc", "reviewer": "Reviewer_CVjA", "summary": "This paper aims to solve the problem of next best view from partial point cloud towards a complete reconstruction. The method uses the principle of SDF  and define a way of computing the maximum coverage gain if a camera is at position \"c\" given the history of camera position, the visibility of the surface on those historical position. This in turn  expected to maximise the total coverage. The authors derive the relations of the incremental coverage gain mathematically and use a neural network to realise those relation to produce the coverage gain given a camera position \"c\". They show results on various dataset including large scale reconstruction.", "questions": "1. Isn't the surface point could be obtained using ray intersection from camera in line 100? Or am I missing something?\n2. Its not clear if the point cloud has a noise how it impact the visibility gain and occupancy function? Though authors mention this in the limitation, but the discussion on sensitivity of noise could have been good.\n3. If the ablation is given with the reconstruction quality then it will be more clear to the reader about the significance of the numbers.\n4. How the authors create the ground truth of the surface coverage gain?\n5. If the explanation from Eq. 6 to the neural design is more lucidly written, then the readability will increase. \n6. Is the method tried in indoor? How this method is positioned w.r.t https://arxiv.org/abs/1805.07794\n7. Is it possible to use the Pointnet features for encoding the 3D point and its neighbourhood in Fig 2?\n8. How the occlusion is handled?", "limitations": "The limitation of this method is narrated by authors and some of the insight we can get from the above questions. For reproducibility, the crucial information regarding the training needs to be there in the main paper. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper has a strong mathematical foundation on defining the coverage gain. The use of spherical harmonics for visibility gain is nice, though SH is being used in recent volume rendering work which authors also referred. Supplementary method provides all the derivation required for the proof of the relations used in the main paper. The paper shows interesting results.\n\nWeaknesses\nThe paper is well grounded with the theoretical formulations. But the readability of the  paper is not very good. It is relatively difficult to relate the equations with the neural architecture for a reader. The concepts are philosophically mapped but the derivation is not clearly mapped with the neural method or in the inductive bias design. The paragraph from line 132 to136 attempted to give some clarity but this needs to be expanded for a general reader. Its not clear if there were neural aggregator instead of Monte Carlo, then what could happened? In table 1 the numbers are very close. What is the sensitivity of these numbers w.r.t the reconstruction?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "VrvROZ7Mt7c", "reviewer": "Reviewer_tPcs", "summary": "The paper proposes a method for next-best-view prediction during three-dimensional surface reconstruction using depth sensors. A characteristic feature of the method is its scalability: most of the state-of-the-art methods are able to work only with artificial models from the ShapeNet dataset of very limited dimensions, while the proposed method is aiming to tackle large-scale structure-from-motion reconstructions such as the one of Colosseum.  The paper introduces a formal theoretical framework explaining that it is possible to estimate surface visibility metrics by sampling points in the volume around the surface, not necessarily strictly on the surface. The proposed method build on this framework. It is evaluated on ShapeNet and on a proposed dataset of large-scale models.", "questions": "L67: Could you please explain the meaning of “volumetric methods are less efficient ..because they dilute information in the 3D space”\n\nL75 this maps is not -> this map could be\n\nRephrase introduction and abstract to emphasize the input precisely (e.g., ‘a method taking a point cloud of points observed so far as input’). Right now the introduction only tells about a probabilistic occupancy map, referring that the perfect map is not known at run-time, but not describing, what is exactly considered to be known.\n\nL78 ‘predict unknown geometry’ - What do these words refer to? \n\nL93: Maybe rename ‘knowledge factor’ to ‘knowledge indicator’ as it can only  be equal to 0 or 1.\n\nIn (4) dx -> dx_0\n\nL122: the volume is supposed to be opaque - what property exactly is expected here, in more precise terms?\n\nL177: How does the model sample 3D points using the predicted occupancy probability function?\n\nIn experiment 3.1, how are the partial point clouds generated?\n\nL256-257: how can ray tracing be used to render a point cloud? What is done to make this rendering approximate specifically the LIDAR sensor, but not time-of-flight or active stereo depth sensors?\n\nL292: ‘Model suffers ... to compute coverage gain’ - does it mean that the second model suffers to predict coverage gain from such a set of points?", "limitations": "The prediction of coverage gain is done in the 2D camera viewpoint space rather than in the 5D space\n\nThe input to the method is not clearly formalized\n\nIt is difficult to judge about the limitations of the current method in terms of accuracy. Would be nice to illustrate some cases when the method does not perform well, and reason about why it happens so.\n\nWould be nice to see how accurate the occupancy prediction model is, how far can it extrapolate geometry, as I think it is the main property this model should have. The second model should decide where can it expect to see more geometry that was previously unseen. Essentially it means that the first model should do occupancy map completion. How well does the model perform in this taskm is a good question for an additional ablation study.\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper develops a new theoretical framework, that explains an approach chosen by the authors to sample points for surface visibility prediction. In particular, the paper shows that it is possible to sample  points in the volume around the surface rather than on the surface directly. \n\nThe paper proposes to use two models: one for occupancy prediction, and another one for point visibility estimation for a particular camera viewpoint. The first model for occupancy prediction is elegant and scalable, reminding of fully convolutional models for images, but working with point clouds. It shows an approach to build spatially localized models for occupancy prediction effectively working with local sub-sets of input points.\n\nThe approach with decomposing the problem into two models seems to be the key to achieving scalability of prediction. It is a distinct step forward in this field, because previous models assumed small object-like structures and simply predicted probabilities of choosing the next view from a pre-defined set of viewpoints evenly distributed on a sphere. The new method is significantly more powerful both in tackling arbitrary geometry and large scale models.\n\n\nOne of interesting questions possibly weakening the paper is that it is rather uncommon to use depth sensors for reconstructing large-scale objects. Usually people focus on Structure-from-motion or photogrammetry tools in these cases, and these tools rely on stereo algorithms rather than on the depth sensors. However, one may argue that a stereo method applied to a pair of images can be understood as a depth sensor.\n\nIn general, the theoretical framework developed in the paper is concerned with point sampling. However, when the method itself is explained, a particular approach to point sampling used in the method is not described well. This way, the theory leaves an impression of being rather disconnected from practice in this particular paper.\n\nI see a conceptual difficulty embedded in a formula (8). However, the formula (8) defines dependency of point visibility prediction on point-camera direction. First of all, camera orientation does not enter the formula at all. It is unclear how to find the best orientation of the camera. Next, an infinite number of camera viewpoints lying on a ray have the same camera-point direction. It is unclear which viewpoint should one choose. Importantly,  occupancy status of the point with respect to the cameras lying on this ray may change. One may move along the ray further away, and if the ray intersects some surface, then the point will become occluded starting from that intersection. This seems not to be addressed by the current model. The method, as I see it, is based on a significant simplification: it downscales the space of camera viewpoints from the 5-dimensional space down to two dimensions. It would be very interesting to clarify this point.\n\nIt remains unclear, what kind of input does the method expect. Sometimes, we see a reference to a ‘depth sensor’. In the ShapeNet experiment, it remains unclear, what is given as input. I would really like to see some clarification in the paper, what particular type of depth sensor should to be used with this method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "V40GNEqt3r1", "reviewer": "Reviewer_DgoJ", "summary": "This paper describes a method for next-best view prediction for reconstructing large-scale 3D scenes with a depth sensor.  They derive a formula to estimate the surface coverage gain for any potential camera pose given a camera pose history and a probabilistic occupancy map.  They use one neural network to predict probabilistic occupancy map based on a point cloud input, and a second network to predict the visibility gain, which is used in the calculation of surface coverage gain.  The first network is trained to match the ground truth occupancy map, and the second is trained to match the ground-truth surface coverage gain.  Extensive experiments on synthetic datasets demonstrates an improvement in performance over SOTA.  ", "questions": "L89: the expression (1- \\lambda) c_pos + \\lambda x seems to be interpolating between the camera position and the point, but it doesn't take the camera's viewing direction into account?\nL101: \"tubular\" -- is this the right word?  or should it be \"spherical\"?  I couldn't figure out why the neighborhood region would be tubular.  Based on the expression in L105 it seems to be spherical (all points within a distance of \\mu_o from x.\n\n", "limitations": "Limitations are discussed but not potential negative societal impacts.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "They provide a thorough derivation an approximation of surface coverage gain that they prove asymptotically approaches the true value.  This formulation is novel to the best of my knowledge.\n\nThe structure of the neural network and the loss functions is interesting.  They use separate networks to predict the probabilistic occupancy  map and the visibility gain functions.  The loss function for the visibility gain network is not visibility gain itself but the surface coverage gain.  They also use attention mechanisms to model occlusion effects.\n\nThey provide a thorough set of experiments to demonstrate their method leads to an improvement on ShapeNet following a standard protocol.\n\nThey also provide some ablation studies to establish the usefulness of various aspects of their proposed approach.\n\nOverall, they have some interesting new ideas which lead to an increase in performance for NBV selection, and their work also can lead to new research such as handling noise in the depth map and selecting optimal paths rather than single viewpoints.\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept