MeshMVS: Multi-view Stereo Guided Mesh Reconstruction
Authors: Rakesh Shrestha
Zhiwen Fan
Siyu Zhu
Zuozhuo Dai
Qingkun Su
Ping Tan
Venue:  (2021)
Keywords: Mesh Reconstruction
Multi-view Stereo
Deep Learning

Abstract: Deep learning based 3D shape generation methods generally utilize latent features extracted from color images to encode the objects' semantics and guide the shape generation process. These color image semantics only implicitly encode 3D information, potentially limiting the accuracy of the generated shapes. In this paper we propose a multi-view mesh generation method which incorporates geometry information in the color images explicitly by using the features from intermediate 2.5D depth representations of the input images and regularizing the 3D shapes against these depth images. Our system first predicts a coarse 3D volume from the color images by probabilistically merging voxel occupancy grids from individual views. Depth images corresponding to the multi-view color images are predicted which along with the rendered depth images of the coarse shape are used as a contrastive input whose features guide the refinement of the coarse shape through a series of graph convolution networks. Attention-based multi-view feature pooling is proposed to fuse the contrastive depth features from different viewpoints which are fed to the graph convolution networks.
We validate the proposed multi-view mesh generation method on ShapeNet, where we obtain a significant improvement with 34% decrease in chamfer distance to ground truth and 14% increase in the F1-score compared with the state-of-the-art multi-view shape generation method.

Under review as a conference paper at ICLR 2021
MESHMVS: MULTI-VIEW STEREO GUIDED MESH RE- CONSTRUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning based 3D shape generation methods generally utilize latent features extracted from color images to encode the objects' semantics and guide the shape generation process. These color image semanties only implicitly encode 3D information, potentially limiting the accuracy of the generated shapes. this paper we propose multi-view mesh generation method which incorporates geometry information in the color images explicitly by using the features from intermediate 2.5D depth representations of the input images and regularizing the 3D shapes against these depth images. Our system first predicts coarse 3D volume from the color images by probabilistically merging voxel occupancy grids from individual views. Depth images corresponding to the multi-view color images are predicted which along with the rendered depth images of the coarse shape are used as a contrastive input whose features guide the refinement of the coarse shape through a series graph convolution networks. Attention-basee multi-view feature pooling is proposed to fuse the contrastive depth features from different viewpoints which are fed to the graph convolution networks.
We validate the proposed multi-view mesh generation method on ShapeNet. where we obtain a significant improvement with 34% decrease in chamfer distance to ground truth and 14% increase in the Fi-score compared with the state-of-the-art multi-view shape generation method.
INTRODUCTION
3D shape generation long-standing research problem computer vision and computer graphics with applications in autonomous driving, augmented reality, etc. Conventional approaches mainly leverage multi-view geometry based on stereo correspondences between images are restricted by the coverage provided by the input views. With the availability of large-scale 3D shape datasets and the success deep learning in several computer vision tasks, 3D representations such as voxel grid Choy al. (016);; Tulsiani al. (2017); Yan et al. (2016) and point cloud Yang et al. (2018); Fan et al. (2017) have been explored for single- view 3D reconstruction. Among them, triangle mesh representation has received the most attention it has various desirable properties for wide range of applications and is capable of modeling detailed geometry without high memory requirement.
Single-view 3D reconstruction methods WWang al. (2018): Huang et al. (2015); Kar et al. (2015): Su et al. (2014) generate the 3D shape from merely a single color image but suffer from occlusion and limited visibility which leads to low quality reconstructions in the unseen areas. Multi-view methods Wen et al. )011)); Choy et al. (2016); Kar et al. 2017); Gwak et al. (2017) extend the input to images from different viewpoints which provides more visual information and improves the accuracy of the generated shapes. Recent work in multi-view mesh reconstruction Wen et al. (2019) introduces a multi-view deformation network using perceptual feature from each color image for refining the meshes generated by Pixel2Mesh Wang al. (2018). Although promising results were obtained, this method relies on perceptual features from color images which do not explicitly encode the objects' geometry and could restrict the accuracy of the 3D models.
In this work, we present novel multi-vien mesh generation method where we start by predicting coarse volumetric occupancy grid representations for the color images of each input viewpoint independently using a shared fully convolutional network which are merged into a single voxel grid in probabilistic fashion followed by cubí Gkioxari et al. (209)) operation to convert it triangle

---

Under review as a conference paper at ICLR 2021
Input Images
M|
Depths
Contrastive feature extractior
Contrastive leature extractor
Contrastive leature extractur
Depth prediction module
ResNet
Varel grid predetion
aeSwork
ResNet
grid prodictios sehvek
Probefibdis vooal grld merging
ResNet
Yonalgri perdiction
Multi-view voxel grid prediction module
Neural Renderer
Neural
Readerer
R Renderer
VGG
Attentive
Pooling
Inttial Shaps (cudified yrk0
GCN2
GCN3
Outpot M Mesh
VGG
Mesh refinement module
Rendeved Predicted Degths
Contrastive
feature extractor
Figure I: Architecture of the proposed method. The voxel grid prediction module predicts coarse voxel grid representation which is further refined by series of GCNs. The GCNs use contrastive depth features from rendered depths of the current shape and the predicted depths from MVSNet. Multi-view features are pooled using multi-head attention mechanism.
mesh. We then use Graph Convolutional Network (GCN) Scarselli al. (2008); Wang et (018) to fine-tune the cubified voxel grid in coarse-to-fine manner. The GCN refines the coarse mesh by using the feature vector each graph node (mesh vertices) obtained by projecting the vertices on the 2D contrastive depth features. The contrastive depth features are extracted from the rendered depth maps of the current mesh and predicted depth maps from a multi-view stereo network. also propose intention-basse method to fuse feature from multiple views that can learn the importance of different views for each of the mesh vertices. Constrains between the intermediate refined mesh from GCN with predicted depth maps of different viewpoints further improve the final mesh quality. By employing multi-view voxel grid generation and refining it using geometry information from both the current mesh (through the rendered depth maps) and predicted depth maps, high-quality meshes. We validate our method on the ShapeNet Chang et al. (2015) benchmark and our method achieves the best performance among all previous multi-view and single-view mesh generation methods.
able generate
RELATED WORK
2.1 TRADITIONAL. SHAPE GENERATION METHODS
3D model generation has traditionally been tackled using multi-view geometry principles. Among
them, structure-from-nnee motioon (SIM) Schonberger & Frahm (2016): Agarwal et al. (201 Cui & Tan (2015): Cui al. (2017) and simultaneous localization and mapping (SLAM) Cadena
(2016);
Mur-Artal et al. 2015); Engel et al. (2014); Whelan et al. (2015) are popular techniques that perform camera pose estimation at the same time. These methods extract local image features, match them across images and use the matches to estimate camera poses and 3D geometry. Closer to our problem setup, multi-view stereo methods infer 3D geometry from images with known camera parameters. Volumetric methods Kar et al. (2017): Kutulakos & Seitz (2000); Seitz & Dyer (1999) predict voxel grid representation of objects by estimating the relationship between each voxel and object surfaces. Point cloud based methods Furukawa & Ponce (2009): Lhuillier & Quan (2005) start with sparse point cloud and gradually increase the density of points to obtain a final dense point cloud of the object. Durou et al. (2008); Zhang et al. (1999); Favaro & Soatto (2005) reason about shading, texture and defocus to reason about visible parts of the object and infer its 3D geometry. While the results of these works are impressive in terms of quality and completeness of reconstruction,
3D reconstruction
2

---

Under review as a conference paper at ICLR 2021
they still struggle with poorly textured and reflective surfaces and require carefully selected input views.
2.2 DEEP SHAPE GENERATION METHODS
Deep learning based approaches can learn to infer 3D structure from training data and can be robust against poorly textured and reflective surfaces as well as limited and arbitrarily selected input views. These methods can be categorized into single view and multi-view methods. HHann et al. (2015); Su et al. (2014) use shape component retrieval and deformation from large dataset for single-view 3D shape generation. Kurenkov al. (2018) extend this idea by introducing free-form deformation networks on retrieved object templates from a database. Some work learn shape deformation from ground truth foreground masks of 2D images Kar et al. (2015); ; Yan al. (2016); Tuussani al. (2017). Recurrent Neural Networks (RNN) based methods Choy et al. (2016): Kar et al. (2017). 7);
Gwak al. (2017) another popular solution to solve this problem. (waak et al. (201 7): Lin et al. (2019) introduce image silhouettes along with adversarial multi-view constraints and optimize object mesh models using multi-vies photometric constraints. Predicting mesh directly from color images was proposed in Wang et al. 2018); Wickramasinghe et al.] 2019); Pan et al.) (2019); Wen et al. (2019): Gkioxari al. (2019); ; Tang al. a (2019). DR-KFS Jin al. (2019) introduces a differentiable visual similarity metric while SeqX Y Han et al. (2020) represents 3D shapes using set of 2D voxel tubes for shape reconstruction. Front2Back Yao et al. (2020) generates 3D shapes by fusing predicted depth and normal images and DV-Net Dia et al. (2020) predicts dense object point clouds using dual-view RGB images with gated control network to fuse point clouds from the two views. FoldingNett Yang et al. (2018) learns reconstruct arbitrary point clouds from single 2D grid. AtlasNet Groueix et al. 2018) use learned parametric representation while Mescheder et al. (2019); Park et al. (2019): Liu al. (209)221hh)) a): Murez et al. (2020) employ implicit surface representation to reconstruct 3D shapes.
2.3 DEPTH ESTIMATION
Compared to 3D shape generation, depth prediction is an easier problem formulation since it simplifies
the task to per- view depth map estimation. Traditional methods Campbell et al. (2008): : Galliani et al.
(2015); (ccöönbreee et al. (2016) use multi-view stereo principles for depth prediction. Deep learning
based multi-view stereo depth estimation was first introduced in Hartmann et al. (2017) where a learned cost metric is used estimate patch similarities. DeepMVS Huang al. (2018) warps multi-view images to 3D space and then applies deep networks for regularization and aggregation to estimate depth images. Learned 3D cost volume based depth prediction was proposed MVSNet Yao iet al. (2018) where a 3 dimensional cost volume is built using homographically warped 2D features from multi-view images and 3D CNNs are used for cost regularization and depth regression. This idea was further extended by Chen et al. (2019): Luo et al. (2019); Gu et al. (2019); Yao et al. (2019).
METHODOLOGY
F showss the architecture of the proposed system which takes as input multi-view color images of object with known poses and outputs triangle mesh representing the surface of the object.
3.1 MULTI-VIEW VOXEL GRID PREDICTION
The single- view voxel branch consists of a ResNet feature extractor and fully convolutional voxel grid prediction network. It generates the coarse initial shape of an object from one viewpoint as voxel occupancy grid using a color image. Here, we set the resolution of the generated voxel occupancy grid as 32 x 32 32. The voxel prediction networks for all viewpoints share the same weights.
Single-view Voxel Grid Prediction
Probabilistic Occupancy Grid Merging Voxel occupancy grid predicted from a single viewpoint suffers from occlusion and limited visibility. In order to fuse voxel grids from different viewpoints, propose probabilistic occupancy grid merging method which merges the voxel grids from each input viewpoint probabilistically to obtain the final voxel grid output. This allows occluded regions in one view be estimated from other views where those regions visible as well as increase the
3

---

Under review as a conference paper at ICLR 2021
confidence of prediction in overlapping regions. Occupancy probability of each voxel is represented by p(z) which is converted to log-odd. (logit):
p(x)
((x) = log 1 1-p(2> p(x)
(1)
Bayesian update on the probabilities reduce to simple summation of log likelihoods Konolige (197). Hence, the multi-view log-odds of a voxel is given by:
(xx) 1, (xx + (x() + In (2x)
where 1, is the voxel's log-odds in view i and n the number of input views. The final voxel probability x is obtained by applying the inverse function of Equation (I) which is sigmoid function.
3.2 MESH REFINEMENT
The cubified mesh from the voxel branch only provides a coarse reconstruction of the object's surface. We apply graph convolutional networks which represent each mesh vertex as one graph node and deforms them to more accurate positions.
GCN-based Mesh Deformation The features pooled from multi-view images along with 3D coordinates of the vertices in world frame used features of the graph nodes. Series of Graph- based Convolutional Network (GCN) blocks are applied to deform a mesh at the current stage to the next stage, starting with the cubi fied voxel grids. A graph convolution deforms mesh vertices by propagating features from neighboring vertices by applying f/ Re + EjEN() Wif) where (i) the set of neighboring vertices of the i-th vertex in the mesh, fo represents the feature vector of vertex, and Wo and W1 are learnable parameters of the model. Each GCN block utilizes several graph convolutions to transform the vertex features along with a final verter refinement operation where the features along with vertex coordinates are further transformed as v 0/2 + tanh (Weerolfi; e:]) where the matrix Weert is another learnable parameter to obtain the deformed mesh.
Contrastive Depth Feature Extraction Yao et al. 2020) demonstrate that using intermediate, image-centric 2.5D representations instead of directly generating 3D shapes in global frame from raw 2D images can improve 3D reconstruction quality. We therefore propose to formulate the features for graph nodes using 2.5D depth maps input additional inputs alongside the RGB features. Specifically, we render the meshes at different GCN stages to depth image at all the input views using Kato et al. 2018) and use them along with predicted depths for depth feature extraction. We call this form of depth input contrastin
dept h as it contrasts the rendered depths of the current mesh against the predicted depths and allows the network to reason about the deformation better than when using predicted depth or color images alone. Given the 2D features, corresponding feature vectors of individual vertices can be found by projecting the 3D vertex coordinates to the feature planes using known camera parameters. We use VGG-16 Simonyan & Zisserman (2014) as our contrastive depth feature extraction network.
Multi- View Depth Estimation We extend MVSNet Yao et al. (2018) and predict the depth maps of all views since the original implementation predicts depth of only one reference view. This is achieved by transforming the feature volumes to each view's coordinate frame using homography warping and applying identical cost volume regularization and depth regression on each view. Detailed network architecture diagram of this module is provided in the appendix.
Attention-based Multi- View Feature Pooling In order to fuse multi-view contrastive depth fea- tures, we formulate an attention module by adapting multi-head attention mechanism originally designed for sequence to sequence machine translation using transformer (encoder-decoder) archi- tecture Vaswani et al.) (2077). In transformer architecture the encoder hidden state is mapped to lower dimension key-value pairs (K, V) while the decoder hidden state is mapped to a query vector Q using independent fully connected layers. The encoder hidden state in case is the multi-view features while the decoder hidden state is the mean of the multi-view features. The attention weights computed using scaled-dot product:
QKT JV
Attention(Q. K, V) = soofmaa
(3)
4

---

Under review as a conference paper at ICLR 2021
Figure 2: Attention weights visualization. From left right: input images from viewpoints, corresponding ground truth point clouds color-coded by their view order and the predicted mesh vertices color-coded by the attention weights of the views. Only the view with maximum attention weight is visualized for each predicted points for clarity.
where is the number of input views.
Multiple attention heads are used which are concatenated and transformed to obtain the final output
head; = Attention Kw VW!) M MultiHead(( Q2 K, V) [head] ...; heads. W
(4)
(5)
where multiple W are parameters be learned, h is the number of attention heads and i [1, ht).
We choose multi-head attention as our feature pooling method since it allows the model to attend information from different representation subspaces of the features by training multiple attentions in parallel. This method is also invariant to the order and number of input views. We visualize the learned attention weights (average of each attention heads) in Figure where we can observe that the attention weights roughly takes into account the visibility/ocelusion information from each view.
3.3 LOss FUNCTIONS
Mesh losses The losses which are derived from Wang et al. (2018) to constrain the mesh predicted
by each GCN block (P) resemble the ground truth (Q) include Chamfer distance Lehamfer (P. Q)
|P-- E(p.))EA,, 1pp 4 + |Q| - (q.p)) llg pl and surface normal loss Loormal (P. Q) =
-IPI-' (p.g)eAr |up ugl IQ 1 (ax)esgr [Mg Upl with additional regularization in the form of edge length loss Ledge E) [हe (1,
1/l r2112 for visually appealing results.
Depth loss Our depth prediction network is supervised using adaptive reversed Huber loss Zwald (2016). Ldepth = |z| [ s otherwise #'t2" where is the depth error of a pixel and is constant set to 0.2. Note that the original M MVSNet uses I.1-loss, but we used BerHu loss since it gave slightly higher accuracy. Intuitively, this is because BerHu provides good balance between L.1 and I.2 loss and has shown similar improvement in Laina et al. (2016).
(also known as BerHu criterion) Laabert-Lacroix
Contrastive depth loss BerHu loss is also applied between the rendered depth images different GCN stages and the predicted depth images. Lcontrastive = | if [x]| S c, otherwise =²+0²
2c
Voxel loss Binary cross-entropy loss between the predicted voxel occupancy probabilities and the ground truth occupancies is used as voxel loss to supervise the voxel predictions Lvoxel - (p(x)))g(p(()) + (1 pox) 100-pppe (1-p()))
Final loss We the weighted sum of the individual losses discussed above as the final loss to train our model in an end-to-end fashion. = X Cehamfer ChormalCoommi + Aedge + Atepth deeph
Avomtustive Lcontrastive Commtrsii A where L. is the final loss term.

---

Under review as a conference paper at ICLR 2021
Input image
Ground
truth
MVP2M
P2M++
Ours
Figure Qualitative evaluation on ShapeNet dataset. From top bottom: one of the input images, ground truth mesh, multi-view extended Pixel2Mesh, Pixel2Mesh+++ and ours. Our predictions are closer to the actual shape, especially for the objects with more complex topologies.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Comparisons We evaluate the proposed method against various multi-view shape generation methods. The state-of-the-art method is Pixel2Mesh++ Wen et al. (2019) (referred as P2M++), Wen
et al. (2019) also provide baseline by directly extending Pixel2Mesh Wang et al. (2018) to operate on multi-viow images (referred as MVP2M) using their statistical feature pooling method to aggregate features from multiple color images. Results from additional multi-view shape generation baselines 3D-R2N2 Choy et al. (016) and Lar et al. (017) are also reported.
Dataset We evaluate our method against the state-of-the oftttr--tttttttto methods on the dataset from Choy et al. (2016) which is subset of ShapeNet Chang et al. (2015) and has been widely used by recent 3D shape generation methods. It contains 50K 3D CAD models from 13 categories. Each model is rendered with transparent background from 24 randomly chosen camera viewpoints to obtain color images. The corresponding camera intrinsies and extrinsics are provided in the dataset. Since the dataset does not contain depth images, render them using custom depth renderer at the same viewpoints as color images and with the same camera intrinsics. We follow the rraining/esting//valdatioon split of Gkioxari et al. (2019).
Implementation For the depth prediction module, we follow the original MVSNet Yao et al. (018)) implementation. The output depth dimensions reduces by factor of 4 56x56 from the 224x input image. The number of depth hypotheses is chosen as 48 which offers a balance between accuracy and running/training time efficiency. These depth hypotheses represent values from 0.1
to 1.3 m at an interval of 25 mm. These values were chosen based on the range of depths present in the dataset.
The hicrarchical features obtained from "Contrastive Depth Features Extractor" are of total 4800 dimensions for each view. The aggregated multi-view features are compressed 480 dimensional after applying attentive feature pooling. attention heads are used for merging multi-view features.
The loss function weights are set as Achumfer = 1, Anormaa 1.6 x 10-4 Atepth = 0.1, Acvntrastive = 0.001 and Avoxel = 1. Two settings of Aedge were used, Xadge 0 (referred as Best) which gives better quantitative results and Aedge 0.2 (referred as Pretty) which gives better qualitative results.
Training and Runtime The network is optimized using Adam optimizer with learning rate of 10 The training is done on 5 Nvidia RTX-2080 GPUs with effective batch size 5. The depth prediction network (MVSNet) trained independently for 30 epochs. Then the whole system is

---

Under review as a conference paper at ICLR 2021
trained for another 40 epochs with the weights of the MVSNet frozen. Our system is implemented in PyTorch deep learning framework and takes around 60 hours for training.
Evaluation Metric Following Wang et al. (018); Wen et al. (2019). we use Fl-score as our evalu- ation metric. The F1-score is the harmonic mean of precision and recall where the precision/recall are calculated by finding the percentage of points in the redicteddgrounnd truth that can find a nearest neighbor from the other within a threshold. We provide evaluations with two threshold values: 7 and 2T where 7 = 10 m²
4.2 COMPARISON WITH PREVIOUS MULTI-VIEW SHAPE GENERATION METHODS
We quantitatively compare our method against previous works for multi-view shape generation in Table and show the effectiveness of our methods in improving the shape quality. Our method outperforms the state-of-the- distance to ground truth by 34% and 15% increase in Fi-score at threshold Note that in Table I the same model is trained for all the categories but accuracy on individual categories as well as average over the categories are evaluated. We provide the chamfer distances in the appendix.
method Pixel2Mesh++ Wen et al. (2019) with a decrease in chamfer
F-score (
F-score (2F)
Category
3D.R2N2
LSM
MVP2M
P2M++
(pretty)
Ours
(best)
Ours
3D-R2N2
L.MM
MVP2M
P2M++
(pretty)
Ours
(best)
Ours
Couch
45,47
43.02
53.17
57.56
71.63
73.63
59.97
55.49
73.24
75.33
85.28
88.24
Cabinet
54.08
56.85
75.91
76.39
81.57
87.61
88.84
Bench
44.56
49.33
60,37
66.24
81.11
83.76
62.47
65 62.92
75.69
79.67
90.56
92.57
Chair
37.62
48.55
54.19
62.05
77.63
78.69
64.95
90.02
Monitor
36.33
43.65
53.41
60.00
74.14
76.64
48.65
56.33
70.63
75.42
86.04
88.89
Firearm
55.72
56.14
79.67
80.74
92.92
94.32
76.79
73.89
89,08
89.29
96.81
97.67
Speaker
41.48
45.21
48.90
54.88
66.02
67.83
52.29
56.65
68.29
71.46
79.76
82.34
Lamp
32.25
45.58
50.82
62.56
72.47
75.93
49,38
64.76
65.72
74.00
82.00
85.33
Cellphone
58.09
60.11
66.07
74.36
85,57
86.45
69.66
71.39
82,31
86.16
93.40
94,28
Plane
47.81
55.60
75.16
76.79
89.23
92.13
70.49
76.39
86.38
86.62
94.65
96.57
Table
48.78
48.61
65.95
71.89
82.37
83.68
62.67
62.22
79.96
84.19
90.24
91.97
Car
59.86
51.91
67.27
68.45
77.01
80.44
78.31
68.20
84.64
85.
89.99
92.33
Watercraft
40.72
47.96
61.85
62.99
75.52
80.48
63.59
66.95
77.49
77.32
86.77
90.35
Mean
46.37
49.73
61,05
78.58
77.10
80.30
90.72
Table I: Qualitative comparison against state-of-the-art multi-view shape generation methods. We
report F-score each semantic category along with the mean over all categories using two thresholds T and 27 for nearest neighbor match where r=10 m²
We also provide visual results for qualitative assessment of the generated shapes by our Pretty model in Figure which shows that it is able to more accurately predict topologically diverse shapes.
4.3 ABLATION STUDIES
Contrastive Depth Feature Extraction We evaluate several methods for contrastive feature ex-
traction (Sub-section 3.2 These methods are 1) Input Concatenation: using the concatenated rendered and predicted depth maps input to the VGG feature extractor, 2) Input Difference: using the difference of the two depth maps as input to VGG, 3) Feature Concatenation: concatenating features from rendered and predicted depths extracted by shared VGG, 4) Feature Difference: using difference of the features from the two depth maps extracted by shared VGG, and 5) Predicted depth only: using the VGG features from the predicted depths only. 6) Rendered depth only: using the VGG features from the rendered depths only. The quantitative results are summarized in Table 2 and shows that Input Concatenation method produces better results than other formulations.
Accuracy with different settings Table [ shows the contribution of different components towards the final accuracy. Naively extending the single-view Mesh R-CNN Gkioxari et al. (2019) to multiple views using statistical feature pooling WWnn et al. (2019) for mesh refinement (row gives an Fl-score of 72.74% for threshold - which is 6.26% improvement over Pixe/2Mesh+++ We further extend the above method with our probabilistic multi-view voxel grid prediction in row 2 and get a 4.23% improvement.
In row 3 of Table 3 we use our contrastive depth features instead of RGB features for mesh refinement and get 2.7% improvement. We then replace the statistical feature pooling with the proposed attention method and get 0. 0.19% improvement. The improvement not significant on our final architecture but we found the multi-hen head attention to perform better more light-weight architectures. We also evaluate the effect of using additional regularization from contrastive depth losses: rendered depth vs predicted depth in the 5th rows of which improves the score by 0.98%. In row we use ground truth
7

---

Under review as a conference paper at ICLR 2021
Fl-7
F1-2r
(1) Input Concatenation
80,80
90.72
(2) Input Difference
80.41
90.54
(3) Feature Concatenation
80.45
90.54
(4) Feature Difference
80.30
90.40
(5) Predicted Depth only
79.40
89.95
(6) Rendered Depth only
78.20
88,9
Table Comparisons of different contrastive depth formulations. In 1st and 2nd rows, concate- nation and difference of the rendered and predicted depths are fed to VGG feature extractor while in 3rd and 4th rows, concatenation and difference the VGG features from the depths is used for mesh refinement. uses VGG features from predicted depths only while uses VGG features from rendered depths only.
instead of predicted depths on our final model which gives the upper bound on our mesh prediction accuracy in relation to the depth prediction accuracy as 84.58%.
F1-7
F1-2r
(I) Naive multi-view Mesh R-CNN
72.74
84.99
(2) + Multi-view voxel grid prediction
76.97
88.24
(3) + Contrastive depth input
79.63
90.10
(4) + Multi-head attention pooling
79.82
90.18
(5) + Contrastive depth loss (final model)
80.80
90.72
(6) Using GT depth (final model)
84.58
92.86
Table 3: Comparison of shape generation accuracy with different settings of additional con-
trastive depth losses, multi-view feature pooling. The Baseline framework uses multi-head attention mechanism without any contrastive depth losses.
Number of View We test the performance of our framework with respect to the number of views. Table shows that the accuracy of our method increases as we increase the number of input views for training. These experiments also validate that the attention-base feature pooling efficiently encode features from different views to take advantage of larger number views.
Table shows the results when using different number of views during testing with views which indicates that increasing the number views during testing does not improve the accuracy while decreasing the number of views cause significant drop in accuracy.
model trained
Metric
2
3
4
6
Metric
3
4
6
FI-T
73.60
80.80
82.61
83.76
84.25
FI-T
72.46
80.80
80.98
80.94
80.85
F1-27
85.80
90.72
91.78
92.73
93.14
F1-27
84.49
90.72
91.03
91.16
91.20
Table 4: Accuracy w.rt the number of views during training. The evaluation was performed on the number of views training.
Table 5:
Accuracy w.rt the number of views during testing. The same model trained with views was used in all of the cases.
CONCLUSION
We propose neural network based solution to predict 3D triangle mesh models of objects from images taken from multiple views. First, we propose a multi-view voxel grid prediction module which probabilistically merges voxel grids predicted from individual input views. We then cubify the merged voxel grid to triangle mesh and apply graph convolutional networks for further refining the mesh. The features for the mesh vertices are extracted from contrastive depth input consisting of rendered depths at each refinement stage along with the predicted depths. The proposed mesh reconstruction method outperforms existing methods with large margin and is capable of reconstructing objects with more complex topologies.
REFERENCES
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in day. Communications of the ACM, 54(10):105-112 2011.
8

---

Under review as a conference paper at ICLR 2021
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, lan Reid, and John Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-pereee perruttpercettion age. IEEE Transactions on robotics, 32(6):1309-1333 2016.
Neill DF Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla. Using multiple hypotheses to improve depth-maps for multi-view stereo. In European Conference on Computer Vision, pp. 766-779. Springer, 2008.
Angel Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 211..30002222 2015.
Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1538 1538-1547, 2019.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision, pp. 028-644. Springer, 2016.
Hainan Cui, Xiang Gao, Shuhan Shen, and Zhanyi Hu. Hsfm: Hybrid moucture-from-iie motton.ion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1212-1221, 2017.
Zhaopeng Cui and Ping Tan. Global kiructre-frrommmotiie by similarity averaging. In Proceedings of the IEEE International Conference Computer Vision, pp. 864-872, 2015.
Deans Durou, Maurizio Falcone, and Manuela Sagona. Numerical methods for shape-from: shading: A new survey with benchmarks. Computer Vision and Image Understunding, 109(1): 22-43, 2008.
Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference on computer vision, pp. 834 849. Springer, 2014.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Paolo Favaro and Stefano Soatto. A geometric approach to shape from defocus. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2763):406-417, 2005.
Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence, 32(6): 2009.
Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 873-881, 2015.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. In Proceedings of the IEEE International Conference Computer Vision, pp. 9785-9799 2019.
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier-måché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 216-224, 2018.
Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. arXiv preprint arXiv:1912.06377. 2019.
Jun Young Gwak, Christopher B Choy, Manmohan Chandraker, Animesh Garg, and Silvio Savarese. Weakly supervised 3d reconstruction with adversarial constraint. 2017 International Conference on 3D Vision (3DV), pp. 263-272. IEEE, 2017.
Zhizhong Han, Guanhui Qiao, Yu-Shen Liu, and Matthias Zwicker. Seqxy2seqz: Structure learning for 3d shapes by sequentially predicting 1d occupancy segments from 2d coordinates. arXiv preprint ar arXiv:2003.00553 2020.

---

Under review as a conference paper at ICLR 2021
Wilffied Hartmann, Silvano Galliani, Michal Havlena, Lue Van Gool, and Konrad Schindler. Learned multi-patch similarity. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1584-1594, 2017.
Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2821-28300 2018.
Qixing Huang, Hai Wang, and Vladlen Koltun. Single-view reconstruction via joint analysis of image and shape collections. ACM Transactions on Graphics (TOG), 34(4):1-10, 2015.
Xin Jia, Shourui Yang. Yuxin Peng, Junchao Zhang, and Shengyong Chen. Dv-net: Dual-view network for 3d reconstruction by fusing multiple sets of gated control point clouds. Pattern Recognition Letters, 131:376-382, 2020.
Jiongchao Jin, Akshay Gadi Patil, Zhang Xiong, and Hao Zhang. Dr-kfs: A differentiable visual similarity metric for 3d shape reconstruction, 2019.
Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik. Category-specine object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1966-1974, 2015.
Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. In Advances in neural information processing systems, pp. 365-376, 2017.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Kurt Konolige. Improved occupancy grids for map building. Autonomous Robots, 4(4):351-367, 1997.
Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, Jun Young Gwak, Christopher Choy, and Silvio Savarese, Deformnet: Free-form deformation network for 3d shape reconstruction from a single image. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 858-866. IEEE, 2018.
Kiriakos N Kutulakos and Steven M Seitz. A theory shape by space carving. International journal of computer vision, 38(3);199-25) 228, 2000.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth international confer- ence on 3D vision (3DV), pp. 239-248. IEEE, 2016.
Sophie Hamberrt-acrolfe and Laurent Zwald. The adaptive berhu penalty robust regression. Journal of Nonparametric Statistics, 28(3):487-514, 2016.
Maxime Lhuillier and Long Quan. A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE transactions on pattern analysis and machine intelligence, 27(3):418-433 2005.
Chen-Hsuan Lin, Oliver Wang, Bryan C Russell, Eli Shechtman, Vladimir Kim, Matthew Fisher, and Simon Lucey. Photometric mesh optimization for video-aligned 3d object reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 969-978, 2019.
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. arXiv preprint 1rXiv:1911.1322. 2019a.
Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3d supervision. In Advances in Neural Information Processing Systems, pp. 8293-8300 2019b.
Keyang Luo, Tao Guan, Lili Ju, Haipeng Huang, and Yawei Luo. P-mvsnet: Learning pasch-wise matching confidence aggregation for multi-view stereo. In Proceedings of the IEEE International Conference Computer Vision, pp. 10452-10461, 2019.
10

---

Under review as a conference paper at ICLR 2021
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1460-4477, 2019.
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robolics, (1(5): 13:511147-1116, 2015.
Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-ent 3d scene reconstruction from posed images. arXiv preprint arXiv:2003.10432, 2020.
Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9964-9973, 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Trunsactions Neural Networks, 20(1):61-80 2008
Johannes L Schonberger and Jan-Michael Frahm. Stracture-from-mottio revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104-4113, 2016.
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view
selection for unstructured multi-view stereo. In European Conference on Computer Vision, pp. 501-518. Springer, 2016.
Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. Interna- tional Journal of Computer Vision, 35(2):151-173, 1999.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1xXiv:140911556. 2014.
Hao Su, Qixing Huang, Niloy J Mitra, Yangyan Li, and Leonidas Guibas. Estimating image depth using shape collections. ACM Transactions on Graphics (TOG) 33(4):1-11, 2014.
Jiapeng Tang, Xiaoguang Han, Junyi Pan, Kui Jia, and Xin Tong. A skeleton-bridged deep learning approach for generating meshes complex topologies from single rgb images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4541-4550. 2019.
Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2626 2634,, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Nanyang Wang. Yinda Zhang. Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference Computer Vision (ECCV), pp. 52-67. 2018.
Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. Pixel2mesh++: Multi-view mesh generation via deformation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1042-1051, 2019.
Thomas Whelan, Stefan Leutenegger, R Salas-Moreno, Ben Glocker, and Andrew Davison. Elastic- fusion: Dense slam without pose graph. Robotics: Science and Systems, 2015.
Udaranga Wickramasinghe, Edoardo Remelli, Graham Knott, and Pascal Fua. Voxel2mesh: 3d mesh model generation from volumetric data, 2019.
11

---

Under review as a conference paper at ICLR 2021
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without supervision. In Advances neural information processing systems, pp. 1696-1704 2016.
Yaoqing Yang, Chen Feng. Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference Computer Viston and Pattern Recognition, pp. 206-215, 2018.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 767-783, 2018.
Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mvsnet for high-resolution multi-view stereo depth inference. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5525-5534, 2019.
Yuan Yao, Nico Schertler, Enrique Rosales, Helge Rhodin, Leonid Sigal, and Alla Sheffer. Front2back: Single view 3d shape reconstruction via front to back prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 531-540, 2020.
Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. Shape-from-shading survey. IEEE transactions on pattern analysis and machine intelligence, 2168):690-706, 1999.
A APPENDIX
NETWORK ARCHITECTURE
MVSNET ARCHITECTURE
Varioace metri:
Argmin
Shared Weights
Shared Weights
Vartance metric
Soft Argmin
Shared W Weights
Shared W Weights
Soft Argmin
Input RGB Images
Differentiable
Output Depth
Extraction
Homography
Regularization
Images
Figure 4: Depth prediction network (MVSNet) architecture
Our depth prediction module is based on MVSNet Yao et al. (2018) which constructs regularized 3D cost volumes to estimate the depth map of the reference view. Here, we extent MVSNet to predict the depth maps of all views instead of only the reference view. This is achieved by transforming the feature volumes to each view's coordinate frame using homography warping and applying identical cost volume regularization and depth regression each view. This allows the reuse of pre- prerrauularization feature volumes for efficient multi-view depth prediction invariant to the order of input images. Figure shows the architecture of the depth estimation module.
12

---

Under review as a conference paper at ICLR 2021
PROBABILISTIC OCCUPANCY GRID MERGING
We single-view voxel prediction network from Gkioxari et al. (2019) to predict predicts voxel grids for each of the input images in their respective local coordinate frames. The occupancy grids transformed to global frame (which is set to the coordinate frame of the first image) by finding the equivalent global grid values the local grids after applying bilinear interpolation on the closest matches. The voxel grids in global coordinates then probabilistically merged according to Sub-section of the main submission.
EXPERIMENTS
We quantitatively compare method against previous works for multi-view shape generation in Table and show effectiveness of our proposed shape generation methods in improving shape quality. Our method outperforms the state-of-the-art method Pixel2Mesh++ Wen al. (2019) with decrease in chamfer distance to ground truth by 34%, which shows the effectiveness of our proposed method. Note that in Table same model trained for all the categories but accuracy on individual categories well as average over all the categories are evaluated.
Chamfer Distance (CD)
1
Category 3D-R2N2
LSM
MVP2M
P2M++
Ours
Couch
0.806
0.730
0.534
0.439
0.220
Cabinet
0.613
0.634
0.488
0.337
0.230
Bench
1.362
0.572
0.591
0.549
0.159
Chair
1.534
0.495
0.583
0.461
0.201
Monitor
1.465
0.592
0.658
0.566
0.217
Firearm
0.432
0.385
0.305
0.305
0.123
Speaker
1,443
0.767
0.745
0.635
0.402
Lamp
6.780
1.768
0.980
1.135
0.755
Cellphone
1.161
0.362
0.445
0.325
0.138
Plane
0.854
0.496
0.403
0.422
0.084
Table
1.243
0.994
0.511
0.388
0.181
Car 0.358
0.326
0.321
0.249
0.165
Watercraft
0.869
0,509
0.463
0.508
0.175
Mean
1,455
0.664
0.541
0.486
0.211
Table 6: Qualitative comparison against state-of-the-art multi-view shape generation methods. Following Wen et al. (2019), report Chamfer Distance in m 1000 from ground truth for different methods. Note that same model is trained for all the categories but accuracy on individual categories well average over all the categories are evaluated.
ABLATION STUDIES
Coarse Shape Generation We conduct comparisons on voxel grid predicted from our proposed
probabilistically merged voxel grids against single view method Gkioxari et al. (2019). As is shown in Table 7 the accuracy of the initial shape generated from probabilisticaly merged voxel grid is higher than that from individual views.
Accuracy at Different GCN Stages We analyze the accuracy of meshes at different GCN stages Table 8 The results validate that our method produces the meshes coarse-to-fine manner and multiple GCN refinements improve the mesh quality.
Resolution Depth Prediction We conduct experiments using different numbers of depth hy- potheses in depth prediction network (Sub-section A), producing depth values different resolutions. A higher number of depth hypothesis means finer resolution of the predicted depths. The quantitative results with different hypothesis numbers are summarized in Table We set depth hypothesis as 48 for our final architecture which is equivalent to the resolution of 25 mm. We observe that the mesh accuracy remain relatively unchanged if we predict depths finer resolutions.
13

---

Under review as a conference paper at ICLR 2021
Metric
Single-view
Multi-view
Metric
Cubified
Stage-1
Stage-2
Stage-3
Fl-7
25.19
31.27
Fl-r
31.48
76.78
79.88
80.80
Fl-2r
36.75
44,46
F1-2r
44.40
88.32
90.19
90.72
Table 7: Accuracy of predicted voxel grids from single-view prediction compared against the pro- posed probabilistically merged multi-view voxel grids. The voxel branch was trained separately without the mesh refinement and evaluation was performed on the cubified voxel grids. We three views for probabilistic grid merging.
Table 8: Accuracy of the refined meshes at dif- ferent GCN stages. 1, 2 and 3 indicate the per- formance at the corresponding graph convolution blocks while Cubified for the cubified voxel grids used as input for the first GCN block. All the stages, including the voxel prediction, were trained jointly and hence the accuracy of voxel predictions varies from that in Table
Metric
24
48
72
96
FI-7
80.29
80.80
80.69
80.34
F1-2r
90.43
90.72
90.74
90.47
Table 9: Accuracy w.r.f the number of depth hypothesis. A higher number of depth hypothesis increases the resolution of predicted depth values the expense of higher memory requirement. The range of depths for all the models are same and based on the minimun/maximum depth in the ShapeNet Chang al. 2015) dataset.
Generalization Capability We conduct experiments to evaluate the generalization capability of our system across the semantic categories. We train our model with only 12 out the 13 categories and test the category that was left out. Table TO shows that the accuracy generally does not decrease significantly when compared with the model that was trained on all 13 categories when using 27 threshold for the F-score.
F-score (
F-score (27)
Category
Excluding
Including
Excluding
Including
Couch
63.29
73.63
80.79
88,24
Cabinet
68.26
76.39
83.10
88.84
Bench
76.08
83.76
87.42
92.57
Chair
60.60
78.69
75.93
90.02
Monitor
67.26
76.64
81.57
88,89
Firearm
78.59
94.32
86.28
97.67
Speaker
62.39
67.83
77.77
82.34
Lamp
63.50
75.93
74.66
85.33
Cellphone
67.24
86.45
80.54
94.28
Plane
57.48
92.13
67.27
96,57
Table
76.41
83.68
86.86
91.97
Car
59.08
80.43
75.58
92.33
Watereraft
64.97
80.48
78.95
90.35
Table 10: Accuracy when a category is excluded during training and evaluation performed on the category verify how well training other categories generalizes to the excluded category.
B APPENDIX
BEST PRETTY MODELS
We provide qualitative comparison between the our models trained with best and pretty configurations
in Figure 5 The best configuration refers to our model trained without edge regularization while pretry refers to the model trained with the regularization (Sub-section 4.I. We observe that without the regularization we get higher score on our evaluation metrics but get degenerate meshes with self-intersections and irregularly sized faces.
14

---

Under review as a conference paper at ICLR 2021
Input image
Ground truth
Pretty
Best
Figure 5: Qualitative evaluation: best vs pretty wireframe models. The best models while being preferred by the evaluation metrics lead to degenerate meshes, with irregularly sized faces and self-intersections
FAILURE CASES
Some failure cases of model (with pretty setting) shown in Figure 6 We notice that the rough topology of the mesh is recovered while we failed to reconstruct the fine topology. We can regard the recovery from wrong initial topology as promising future work.
Input image
Ground truth
Ours
Figure 6: Failure Cases. Our system can struggle to roughly reconstruct shapes with very complex topology while some fine topology of the mesh is missing.
15


Reviews:
[{"review_id": "tygzj8aP2wd", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes to first predict a coarse (32^3) voxel grid by aggregating independent predictions from individual views. Then, it translate it into a mesh and refine it using deepMVS predictions (using each view in turn as a reference view), and a GCN architecture on the mesh.\n\nOn the positive side:\n- I like the idea of using MVS-Net, but why not use it from the start (before the single view voxel prediction). \n- I think this paper is going toward a render-and-compare approach for 3D shape prediction, which I think is a good idea.\n- the boost in the results seems impressive compared to P2M++\n\nThere are however several things I don't like or that worry me about this paper:\n- the pipeline presented in this paper is extremely complicated, and has many different parts. After reading it, I have no idea what really makes the improvement compared to P2M++. It uses voxels, mesh and depth maps, Graph convolution networks, attention-based architecture, SVR and deepMVS, the training loss has 5 balancing hyperparameters, between things as different as cross-entropy and chamfer distance.\n- To me, the ablation studies (Table 2 and 3) show clearly that the most complex parts of the pipeline (3.2, contrastive depth and attention based aggregation) only provides very minor improvements (~1%). Given their complexity and number of hyper parameters, I do not think these can be considered as significative. Given these results, it is completely unclear to me how the proposed approach can lead to a ~14% improvement over pixel2Mesh. I thus think the approach should be strongly simplified (maybe loosing 1% in final performance), but the paper should provide a clear ablation that actually explain why their framework is so much better than P2M++ and this is interesting. Right now, I believe it could be for a bad reason (for example DeepMVS could give excellent results on synthetic data because it is too simple - note I realize that Table 3 shows it is not perfect since there is a further 3.5% boost using GT depth, but it could still be unrealistically good for synthetic data)\n- Related work is lacking discussion of important references, namely all classical references for point-based SfM in 2.1 , foldingNet and AtlasNet for mesh generation in 2.2, all implicit volumetric works also in 2,2 (deepSDF, OccupancyNetworks…), the most classical deep depth prediction works in 2.3 (Eigen and Fergus…)\n\nTo summarise, despite its impressive numbers, I think this paper cannot be accepted as is, mainly because of its complexity, lack of clear explanation for its huge performance boost, and the only marginal/not significative boosts given by the most complexe parts of the pipeline.\n\nSome additional notes on presentation:  \n- I am not sure “contrastive depth” is a good choice of name since contrastive feature learning is a popular but unrelated research direction.\n- I found 3.2 very hard to parse/re-order. I could only do it with the help of fig. 1 which is itself hard to parse and does not represent e.g. how the attention-based pooling happens\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XQgmDB-zFba", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview\n\nThis paper proposes a system of reconstructing 3D objects from multi-view images. The system consists of a single-view voxel generation network, a multi-view voxel fusion mechanism, a multi-view depth estimation network, and a refinement network aggregating multi-view depth features. \nThe major contribution is in the refinement stage upon the coarse reconstruction obtained from voxel predictions, typically for the introduction of the Attention-based Multi-View Feature Pooling.\n\nMethod Novelty \nAccording to the paper and the attached code, it seems like the authors mostly utilized existing networks to build a system. The author introduces their Attention-based Multi-View Feature Pooling mechanism which is new. Despite the results, the system is rather bulky and ad-hoc. For the use of GCN in refinement, see Question 2.\n\nResults\nThe paper achieves plausible state-of-the-arts quantitate results on standard evaluation sets and metrics.\nThe visual quality is reasonable, however from Figure 3 it seems like reconstructed local surface suffers from noises. Their results struggles to getting clean surface especially when compared to implicit-based methods, such as DeepSDF. The authors did not provide more qualitative results in supplementals.\n\nClarity\nThis paper is well written and easy to understand. The attached code is well documented and can be deployed.\n\nConclusion\n\nOverall, this is a well written paper with plausible outcomes. The reviewer believes this paper carries out reasonable efforts and insights into this topic. The reviewer is marginally positive towards its acceptance due to the pleasing results, but is holding a conservative attitude towards its contribution significances. The reviewer would like to see the questions addressed in the rebuttal period, while also refer to other's reviews. \n\nQuestions:\n1. For each single-view voxel prediction, the paper did not clarify which coordinate system those voxel are in. When aggregating multi-view voxel grid, how is the coordinate transformation handled between different viewpoints? If voxel from different coordinate systems should undertake transformation, how is interpolation handled when merging to a single 32x32x32 grid? \n2. Use of GCN. As GCN only optimizes the current mesh, it cannot correct the topology error occurring after the coarse reconstruction. How would this method overcome this, especially when the cubified mesh is in wrong topology?\n3. Use of depth. From multi-view predicted depth, one can simply reconstruct from the depths, or run differentiable render for optimizing the mesh geometry directly. Why would we need contrastive depth feature extraction?", "extra_scores": {}, "extra_text": {}}, {"review_id": "KsGE3v5Zex8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Quality:**\nOverall the quality of this work is high.  The quantitative and qualitative results are impressive relative to the SoA. I would like to see the qualitative results for the Best model as opposed to just the Pretty model, and I'm curious why the best qualitative mode was not the same as the best quantitative model.  I would think analyzing this difference could give the authors insight into how to improve the model. \n\n**Clarity:**\nOverall the paper is written clearly, explaining and justifying the different components of the model clearly.  There are a few issues/questions I have:\n\n* Page 2: change \"non-reflective\" -> \"reflective\"\n* For depth estimation, I'm wondering why you changed the MVSNet loss function to use BerHu instead of L1 used in the original paper?\n* Could you define the terms in the BerHu criterion?  What are x and c? It would also be good to shed some intuition on why this criterion is the right one.\n* The mixing constants in your loss function ($\\lambda$) vary across several orders of magnitude.  How were those selected?\n* On page 6 you state that two values of $\\tau$ are used, but elsewhere in the paper $\\tau$ is defined as $10^{-4}$ and you use $\\tau$ and $2\\tau$. \n\n**Originality:**\nThe paper generally uses a mix of SoA techniques creatively woven together in a fairly sophisticated model. Oher novel aspects such as using the neural renderer to create the contrastive depth module was interesting. \n\n\n**Significance:**\nThis work is significant based on the importance of the problem - this is one of the harder and most important problems in computer vision today,  in the quality of its results and in the creative way it combines SoA methods to provide multiple semi-supervised losses. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission is an interesting case...

The method it presents appears to work quite well, achieving state-of-the-art quantitative reconstruction results (though qualitatively, the reconstructed surfaces are locally noisy).

The method is quite complex, which different reviewers saw as either a strength or a weakness ("a mix of SoA techniques creatively woven together in a fairly sophisticated model" vs. "bulky and ad hoc").

Most critically: it appears that the reasons for the method's significant (14%) improvement over the prior art for this problem (Pixel2Mesh++) are not due to the novel contributions that the paper focuses on (multi-headed attention, contrastive depth loss). Rather, it is other system design choices that are not novel research contributions that make up all but 1% of this difference (primarily, using a voxel grid predictor to get the initial mesh, as opposed to an initial ellipsoid mesh).

It might be possible for the authors to write a systems paper supporting these design decisions and showing how they lead to better results. However, this is not the paper the authors have written (the majority of the technical detail in the paper is focused on method components that make minimal impact). I would also argue that this hypothetical paper would not necessarily be appropriate for ICLR, since it does not focus on any new representations. It would be better suited to a venue such as CVPR, ICCV, or 3DV.

p.s. Reviewer 5 deserves all of the credit for noticing this major issue with the paper.
Decision: Reject