Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The submission studies the problem of 3D geometry and appearance reconstruction from a sparse set of 2D images, and proposes a new neural implicit model called Neural Reflectance Surfaces (NeRS). The main difference from prior work (specifically, NeRF) is the following: 
1. using a much sparser set of 2D observations as input - typically around 10 images per instance 
2. a neural surface model for geometry instead of neural volumetric representations, 
3. a neural environment map to model lightings, conditioned only on the viewing direction,
4. BRDFs representation for surface specularity to model view-dependent effects.

In particular, the shape geometry is modeled as a continuous and implicit deformation of a unit sphere, parametrized by a multi-layer perceptron (MLP). Similarly, the diffuse color is parametrized by a second MLP with spherical coordinates as input. To render an image, a continuous unit sphere in 3D is first discretized into a set of vertices (spherical coordinates), which are then passed as input to the shape MLP to construct a triangulated surface mesh. The per-vertex diffuse colors are computed from a second MLP using the same set of spherical coordinates. Together with the neural environment map (from a third MLP), the final pixel intensity can be rendered using a Phong reflection model. These 3 different MLPs are optimized sequentially and iteratively using a combination of 2D image loss, a mask loss, and mesh regularization. The method is compared against different variants of NeRF on a new "Craigslist Multiview" (CMV) dataset. Both quantitative and qualitative comparisons show that the proposed NeRS method outperforms NeRF under this few-shot experiment setup.
main_review: #### Originality
- This submission is slightly different from NeRF in the problem setup - it focuses on the few-shot scenario, and reconstructs only a single instance of objects.
- Some of the main technical components are from Tulsiani et al. (2020), including the mesh and color parametrization and the loss function. There is a high level of similarity between the two methods, with some differences in using environmental map and BRDFs as well as the number of inputs. 
- Related Work: 
    - Please elaborate on the differences from Tulsiani et al. (2020). 
    - The idea of using surface representation is relevant to "Deferred neural rendering: Image synthesis using neural textures" (ToG, 2019). Please consider adding this (and its follow-up works) to the related work section. 
    - Some papers on modeling reflection in the context of NeRF is also relevant, including but not limited to "NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis" and "Neural Light Transport for Relighting and View Synthesis"

#### Quality
- The submission is technically sound.
- Both the quantitative and qualitative results show that the proposed method significantly outperforms NeRF in the sparse-input regime on the new dataset. 
- However, the experiment section is limited in several major ways:
    - The submission is only tested on one new dataset - it would be much more convincing if the authors can include experiments on ShapeNet and/or other datasets that are widely used in the community.  
    - There are a number of variables in the optimization problem, including geometry, textures, camera poses, and specularity. An ablation study on the contribution of these optimization variables to the final accuracy is needed - for instance, how important is it to include the camera poses and the specularity in this optimization framework, or can we remove them without hurting the accuracy? Showing the quality metrics in Table 2 (in the supplementary materials) after each optimization stage would be helpful.
    - It is unclear why the environment map requires a MLP representation. Can we achieve the same performance with bilinear sampling from an optimizable tensor? Please add an ablation study to support your design choice. 
    - Another great ablation study to include is a curve of reconstruction quality vs. number of input images. This helps readers understand how many images are needed for their use cases.
- While this submission is interesting to the community, it seems like a work-in-progress rather than a complete piece of work, especially given the limited experiments.
- Instead of directly claiming NeRF fails, the authors have made an effort to improve NeRF with several modifications to ensure a fair comparison. The extensive visualization provided by the authors is quite helpful. 

#### Clarity
- The paper is overall clearly written and well organized, and the supplementary materials are also helpful. 
- I believe it is largely reproducible based on the technical description. 
- A few comments that might help improve the writing:
    - Please consider adding new paragraph in Section I as a summary of the main contributions, as well as the differences from NeRF and Tulsiani et al. (2020).
    - It is not crystal clear from the paper whether the specular material parameters $(k_s, \alpha)$ are defined on an instance level (e.g., shared by the entire surface of an object instance) or on a per-pixel level (parametrized by another MLP). My understanding is the former, but an explicit explanation would help.
    - One missing implementation detail is how the authors sample the view directions $\omega \in \Omega$.

#### Significance
- The results are significant, since it might help enable 3D reconstruction from a sparse set of 2D images. I believe the topic is of great relevance to the machine learning / computer vision / computer graphics community and beyond. 
- I can see practitioners using this technique to build new applications that require less intensive data capturing from users. Researchers could also build on top of the submission in terms of complexity, extending from single objects to complete scenes.
- Compared to NeRF, the submission's task is more challenging in the input size but at the same time simpler in scene complexity (single object vs. entire scene). It is addressing a simpler task than Tulsiani et al. (2020) given that it is using more input images.
- It improves the state of the art on few-shot reconstruction in a clear and demonstrable way.
- It also provides a new, unique dataset for few-shot reconstruction, establishing a new benchmark for future exploration.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of recovering 3D shape, surface albedo, specular material (Phong model), and environment lighting map from a sparse set of images from ~8 viewpoints, given a rough template shape as initialization and automatically obtained masks and camera poses. This is achieved by coupling a AtlasNet-like representation for shape and texture with explicit material and lighting models in an optimization pipeline that reconstructs the input observations.

The paper shows plausible decomposition results on multi-view image sets from the Internet with rough camera estimates and mask predictions, whereas existing NeRF-based methods fail dramatically on such few inputs.

main_review: 
## Strengths
### S1 - Method
- Overall, the proposed method is carefully designed and achieves compelling decomposition results.
- It uses an AtlasNet like representation for 3D shape, texture and surface material, which is discretized and rendered into 2D images through a differentiable renderer. In this view, the proposed method essentially extends an AtlasNet-based multi-view reconstruction model with material and lighting components.
- The environment lighting map is modeled similarly with a sphere AtlasNet, and sampled for rendering.

### S2 - Results
- The decomposition results are visually appealing, and no existing methods seem to be able to achieve similar results on real world photo sets "in the wild". The comparisons against NeRF-based methods also show significant performance gap.
- The ablation results with naive NeRF-like view-conditioning texture prediction reported in the supplementary material are very insightful, confirming the effectiveness of enforcing explicit shading models for few-view reconstruction.

## Weaknesses
### W1 - Initialization
One significant limitation of the proposed method is need of a category shape template for shape initialization. It also relies on another method for estimating coarse initial camera poses. These additional information has significantly simplified the problem.

### W2 - Over-claim on shape representation
The shape representation they are using is essentially an AtlasNet over a sphere. I think its novelty has been over-claimed by focusing the comparison instead on NeRF and its rendering procedure.

### W3 - Simplistic material model
- The method assumes a Phong model with a global shininess to model specularity, which cannot model sophisticated real world illumination effects, resulting from complex spatially-varying BRDF.
- It also assumes a gray-scale environment map, which is not realistic.
- It seems such explicit material and lighting models, especially with the unrealistic Phong model, also limit the image quality, resulting in relatively low faithfulness scores in novel view synthesis as shown in Table 1.

### W4 - Limited comparison
- The method is only compared to NeRF-based models. How does the method compare to IDR and other photometric stereo methods? Can this CVPR20 work [A] handle these cases, which also recovers 3D shape and SVBRDF from a small set of images, but seems to assume collocated flash light?

[A] "Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images". Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. CVPR 2020.

## Additional comments
- How is the shape network $f_\text{shape}$ pretrained with the template mesh? I suppose the mapping from sphere to surface is unknown. Is this achieved using for example an occupancy loss?
- How well is the environment lighting estimated? There is only one single example in Fig 4. It would be helpful to show more visualizations.
- What are the resolutions of the discretized mesh and environment map?

limitations_and_societal_impact: 
A few limitations are discussed, and visual examples are provided in the supplementary material. It would also be interesting to show how the method fails without sufficiently accurate initial mesh.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introduces a method to reconstruct a scene from a small number of views. Rather than using a volumetric representation, it uses an explicit surface that is defined as a sphere with perturbed vertices. This restriction allows the model to learn to decompose the reflectance into a Phong lighting model. A contribution of the work is the introduction of a large, in-the-wild multiview dataset of cars from craigslist. 
main_review: Pros:

+ Surface representations deserve more exploration, and using the additional constraints for few view reconstruction makes sense.

+ Real world dataset with few views is a great contribution and space to work in. I really like the idea of scraping marketplaces for such data, and this work introduces a new dataset consisting of millions of multi-view captures, which is great. The main downside is that as far as I can tell, they are only for cars. 

+ BRDF factorization from sparse views is very hard.

+ Can work with only coarse cameras (by jointly optimizing camera parameters during reconstruction).

+ Formulation is clean and elegant, everything defined on the surface of a sphere (geometry, env map, texture).

+ Can use better reconstruction losses due to entire images generated at train time. 

Cons: 

- My biggest concern is that this approach is missing comparisons to other surface based models! for example: [Yariv et al] with code available, why was this not compared to? This would be key to showing that the "deformed sphere" approach is better than an SDF representation, as it comes with some limitations in terms of what kinds of models you can represent. 

- Results are only okay, they tend to look like "balloon" versions of the original, and the Phong shading model does not capture specular reflections well (at all?). 

- Requires a mask loss (like Yariv et al), but it seems like the mask loss might already get you a decent starting model simply by reconstructing the visual hull from the masks for convex objects like cars. Is it possible to compare the result to this simple solution?  

- Requires per-category initialization mesh. In fact, this part was a bit unclear to me, what do you use for cars? How do you show motorcycle results? Where do the other non-car examples and initialization meshes come from (e.g., the coffee machine)? What is the importance of the initialization mesh? Is it just for camera pose estimation, or do you use it to initialize the geometry network as well?  

Additional Questions:

When comparing to baselines, the final camera positions that are derived by optimizing this method's objective are used. In some senses, those may be optimal for this approach. How much does the camera change 1) between initialization and the pre-training, 2) between pre-training and final training? 

It is interesting that Nerf outperforms with respect to LPIPS but seems clearly worse in the results. Do you have any intuition why this is? 

How do you model shadows? This seems important given that all images are outdoor, and you often see the object shadowing itself. Does the neural renderer correctly model these? 

TL/DR: I like this work, but I think its missing some comparisons (e.g., to another surface based method), and the overhead of category specific steps may make it hard to extend to new classes in the wild.
limitations_and_societal_impact: I believe that few view 3D reconstruction has the ability to give non-professional users the ability to create 3D assets. Negative social impact is adequately discussed.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a method for surface geometry and surface reflectance estimation from multiple photographs in the wild.  The central idea can be viewed as an extension of the NerF-type neural implicit volumetric representation, here the goal is instead to learn neural implicit surface  representation for both object geometry and surface reflectance.    
main_review: This paper  extends the idea of NeRF from volumetric  model to neural surface modelling.   This idea is very similar to the work of Yariv et al in NeurlPS2020 for disentangled multi-view object geometry  recovery (also represented in the form of neural surface) and object reflectance estimation (also used a similar photometric model).    The overall computation pipeline adopts the inverse graphics rendering procedure where a pre-defined mesh (here a simple mesh) is gradually deformed from a canonical shape (a sphere)  to the target shape to conform with the multi-view image measurements.    Rather than directly predicting 3D vertex positions on the mesh,  the paper propose to predict a displacement field (i.e. mesh flows).   

Advantages:   
1. impressive experiment results: from a sparse set of multi-view images ( can be as few as 8), the method is able to recovery the shape and texture of objects with general environment lighting. In contrast, NerF often needs more images (>20~40).
 2.  Estimating displacement field rather than shape itself will in principle make the neural net more generalisable to more complex shapes.  ( though no ablation study is provided  in the paper).
3.  Contribute a new  "in the wild" multi-view object dataset (the Craiglist Multi-View dataset). 

Limitations: 

1. Limited originality:   The originality and novelty of the main idea of the paper is relatively limited.  Apart from some significant overlaps with Yariv NeurlPs 2020,  as well as with Piex2Mesh++ (in CVPR2019),  the idea of estimation mesh deformation field (i.e. Mesh Flow) rather than mesh itself was also proposed in "Kunal Gupta and Manmohan Chandraker, Neurla MeshFlow"  (Neurlps 2020) for the very same task of  image2mesh reconstruction by deforming a canonical initial mesh.    However, no comparison is presented in the paper.   The Pixel2Mesh++ in CVPR2019 also used the same deforming a sphere and rendering and comparing it with the input multi-view images, despite using an explicit surface representation (rather than implicit neural surface representation).   The NeuralMeshFlow pipeline is almost identical to the paper's, which also estimates  a deformation field for neural surface representation.   Doing the above comparisons is imo essential for demonstrating the value of this work.


2. Insufficient validation:    As shown in the figures of the paper,  most of the objects being reconstructed have rather simple  (sometimes convex) geometry shapes  ( e.g. ketchup bottle, a sedan/car,  a rice cooker, a coffee machine,  etc.).   For these simple shapes,  the method of "shape from multi-view silhouettes" may already work well to recover the shape.   Indeed,  I notice the "object mask-loss" used in your method is precisely for this purpose.   Unfortunately, there is no ablation study, nor comparison in your paper to validate this.  It is unclear whether the good result is largely due to the use of the object mask-loss alone.  Which terms plays what role ?     In light of this, the paper is clearly pre-mature, and more work is needed to  validate the main idea.   

3. Unclear analysis:  Similarly,  although the paper claims it also estimates a spherical light map (environment map), the efficacy of this env-map has never been demonstrated or validated in the paper.   After all, the Phong shading model is an overly simplified  lighting model not too far away from the dichromatic model.   The recovered texture map shown in the paper (figure-7 in the main paper, as well as in the demo videos) show  strong highlight/specularity-- suggesting  the reflectance estimation is quite off by your method.   Have you evaluated how accurate the recovered env-map, or the recovered Phong parameters ? 

More importantly,  it is unclear how this estimated spherical environment map resolves the "fundamental ambiguity" (in reflectance and lighting).  Given the env-map is an area light source,   how can the network distinguish a low-textured object illuminated by a high-frequency env-map, and a highly-textured object illuminated by a smooth env-map ?   To speak of the least,  there is no account nor experiments about how well/accurate the estimated env-map looks like or works.  

5.  The paper claims ( in line 120) the method only requires approximately calibrated camera parameters to start with.  However, based on the description in the experiment section (line 236-243), it seems the camera parameters are optimized BEFORE optimizing the other parameters (f_shape and f_env and f_tex), and kept fixed afterward.     This is strange, because if you do not have a good enough initial f_shape, f_texture, and F_env the environment map estimations,   how can you expect to get the camera calibration parameters right or accurately ?    The experiments do not seem to support your claim. 
 
6.  Most of the objects shown in the paper (as well as in the video) have the spherical topology.  While the implicit surface representation naturally accepts non-sphere topologies,  the rasterization graphics rendering step often has issue with surface with self-intersection or holes.   The paper failed to analyze this point.  Tests on more complex shapes  (e.g. with non-spherical topology) is necessary.  
  

Overall,  the paper combines multiple interesting ideas, and experiment results looks good, but its execution (of those ideas) is rather poor,  weak and insufficient.   Its theoretical novelty is limited, and experiment  are also lacking (missing necessary comparisons and ablation studies).   The quality of the paper is okay but not good enough.  It is clearly under-developed and pre-mature, making it just below the bar for NeulPS. 



limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper received borderline reject initial reviews, with the reviewers asking for comparisons with IDR etc. and more discussion on the environment maps. The authors provided significant new results which addressed most of the reviewers' concerns. Limitations remain both on the  accuracy of the illumination/reflectance decomposition and on the range of shapes that the method handles. 
However, based on the current interest in the area, the novel experimental set-up and dataset and the fact that the authors have demonstrated better results in the case of a challenging sparse view set-up, the metareviewer recommends acceptance. The authors are urged to include the comparisons from the rebuttal to the final paper.
Decision: Accept (Poster)