Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The paper proposes to first predict a coarse (32^3) voxel grid by aggregating independent predictions from individual views. Then, it translate it into a mesh and refine it using deepMVS predictions (using each view in turn as a reference view), and a GCN architecture on the mesh.

On the positive side:
- I like the idea of using MVS-Net, but why not use it from the start (before the single view voxel prediction). 
- I think this paper is going toward a render-and-compare approach for 3D shape prediction, which I think is a good idea.
- the boost in the results seems impressive compared to P2M++

There are however several things I don't like or that worry me about this paper:
- the pipeline presented in this paper is extremely complicated, and has many different parts. After reading it, I have no idea what really makes the improvement compared to P2M++. It uses voxels, mesh and depth maps, Graph convolution networks, attention-based architecture, SVR and deepMVS, the training loss has 5 balancing hyperparameters, between things as different as cross-entropy and chamfer distance.
- To me, the ablation studies (Table 2 and 3) show clearly that the most complex parts of the pipeline (3.2, contrastive depth and attention based aggregation) only provides very minor improvements (~1%). Given their complexity and number of hyper parameters, I do not think these can be considered as significative. Given these results, it is completely unclear to me how the proposed approach can lead to a ~14% improvement over pixel2Mesh. I thus think the approach should be strongly simplified (maybe loosing 1% in final performance), but the paper should provide a clear ablation that actually explain why their framework is so much better than P2M++ and this is interesting. Right now, I believe it could be for a bad reason (for example DeepMVS could give excellent results on synthetic data because it is too simple - note I realize that Table 3 shows it is not perfect since there is a further 3.5% boost using GT depth, but it could still be unrealistically good for synthetic data)
- Related work is lacking discussion of important references, namely all classical references for point-based SfM in 2.1 , foldingNet and AtlasNet for mesh generation in 2.2, all implicit volumetric works also in 2,2 (deepSDF, OccupancyNetworks…), the most classical deep depth prediction works in 2.3 (Eigen and Fergus…)

To summarise, despite its impressive numbers, I think this paper cannot be accepted as is, mainly because of its complexity, lack of clear explanation for its huge performance boost, and the only marginal/not significative boosts given by the most complexe parts of the pipeline.

Some additional notes on presentation:  
- I am not sure “contrastive depth” is a good choice of name since contrastive feature learning is a popular but unrelated research direction.
- I found 3.2 very hard to parse/re-order. I could only do it with the help of fig. 1 which is itself hard to parse and does not represent e.g. how the attention-based pooling happens


review_title: complicated deep 3D reconstruction pipeline with good results

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview

This paper proposes a system of reconstructing 3D objects from multi-view images. The system consists of a single-view voxel generation network, a multi-view voxel fusion mechanism, a multi-view depth estimation network, and a refinement network aggregating multi-view depth features. 
The major contribution is in the refinement stage upon the coarse reconstruction obtained from voxel predictions, typically for the introduction of the Attention-based Multi-View Feature Pooling.

Method Novelty 
According to the paper and the attached code, it seems like the authors mostly utilized existing networks to build a system. The author introduces their Attention-based Multi-View Feature Pooling mechanism which is new. Despite the results, the system is rather bulky and ad-hoc. For the use of GCN in refinement, see Question 2.

Results
The paper achieves plausible state-of-the-arts quantitate results on standard evaluation sets and metrics.
The visual quality is reasonable, however from Figure 3 it seems like reconstructed local surface suffers from noises. Their results struggles to getting clean surface especially when compared to implicit-based methods, such as DeepSDF. The authors did not provide more qualitative results in supplementals.

Clarity
This paper is well written and easy to understand. The attached code is well documented and can be deployed.

Conclusion

Overall, this is a well written paper with plausible outcomes. The reviewer believes this paper carries out reasonable efforts and insights into this topic. The reviewer is marginally positive towards its acceptance due to the pleasing results, but is holding a conservative attitude towards its contribution significances. The reviewer would like to see the questions addressed in the rebuttal period, while also refer to other's reviews. 

Questions:
1. For each single-view voxel prediction, the paper did not clarify which coordinate system those voxel are in. When aggregating multi-view voxel grid, how is the coordinate transformation handled between different viewpoints? If voxel from different coordinate systems should undertake transformation, how is interpolation handled when merging to a single 32x32x32 grid? 
2. Use of GCN. As GCN only optimizes the current mesh, it cannot correct the topology error occurring after the coarse reconstruction. How would this method overcome this, especially when the cubified mesh is in wrong topology?
3. Use of depth. From multi-view predicted depth, one can simply reconstruct from the depths, or run differentiable render for optimizing the mesh geometry directly. Why would we need contrastive depth feature extraction?
review_title: Positive towards the results, conservative towards the contribution

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: **Quality:**
Overall the quality of this work is high.  The quantitative and qualitative results are impressive relative to the SoA. I would like to see the qualitative results for the Best model as opposed to just the Pretty model, and I'm curious why the best qualitative mode was not the same as the best quantitative model.  I would think analyzing this difference could give the authors insight into how to improve the model. 

**Clarity:**
Overall the paper is written clearly, explaining and justifying the different components of the model clearly.  There are a few issues/questions I have:

* Page 2: change "non-reflective" -> "reflective"
* For depth estimation, I'm wondering why you changed the MVSNet loss function to use BerHu instead of L1 used in the original paper?
* Could you define the terms in the BerHu criterion?  What are x and c? It would also be good to shed some intuition on why this criterion is the right one.
* The mixing constants in your loss function ($\lambda$) vary across several orders of magnitude.  How were those selected?
* On page 6 you state that two values of $\tau$ are used, but elsewhere in the paper $\tau$ is defined as $10^{-4}$ and you use $\tau$ and $2\tau$. 

**Originality:**
The paper generally uses a mix of SoA techniques creatively woven together in a fairly sophisticated model. Oher novel aspects such as using the neural renderer to create the contrastive depth module was interesting. 


**Significance:**
This work is significant based on the importance of the problem - this is one of the harder and most important problems in computer vision today,  in the quality of its results and in the creative way it combines SoA methods to provide multiple semi-supervised losses. 

review_title: Solid work on multi-view mesh reconstruction

Meta-review: This submission is an interesting case...

The method it presents appears to work quite well, achieving state-of-the-art quantitative reconstruction results (though qualitatively, the reconstructed surfaces are locally noisy).

The method is quite complex, which different reviewers saw as either a strength or a weakness ("a mix of SoA techniques creatively woven together in a fairly sophisticated model" vs. "bulky and ad hoc").

Most critically: it appears that the reasons for the method's significant (14%) improvement over the prior art for this problem (Pixel2Mesh++) are not due to the novel contributions that the paper focuses on (multi-headed attention, contrastive depth loss). Rather, it is other system design choices that are not novel research contributions that make up all but 1% of this difference (primarily, using a voxel grid predictor to get the initial mesh, as opposed to an initial ellipsoid mesh).

It might be possible for the authors to write a systems paper supporting these design decisions and showing how they lead to better results. However, this is not the paper the authors have written (the majority of the technical detail in the paper is focused on method components that make minimal impact). I would also argue that this hypothetical paper would not necessarily be appropriate for ICLR, since it does not focus on any new representations. It would be better suited to a venue such as CVPR, ICCV, or 3DV.

p.s. Reviewer 5 deserves all of the credit for noticing this major issue with the paper.
Decision: Reject