Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper uses a neural field-based approach to estimate the shape, BDRF parameters, and per-image camera pose and illumination from the in-the-wild image collections. It is the first method that is able to estimate all parameters simultaneously. Conventional neural field-based approaches require an almost-correct camera pose to estimate the shape and BDRF parameters. The proposed method replaces the need for accurate camera pose with a rough user-initialized camera pose quadrant. This paper introduces a novel in-the-wild image collection dataset in which COLMAP is hard to estimate the camera pose. Experimental results show that the proposed method achieves much better accuracy in the challenging dataset (pose not known).
strengths_and_weaknesses: # Strengths

- The proposed method is able to solve a highly under constrained problem with rough camera pose initialization. It estimates the BRDF parameters, unit-length surface normal, volume density, latent-per-image illumination vectors, per image camera pose and intrinsics.
- Flexible camera parameterization for varying distances. As the proposed method targets for in-the-wild image collection, the near and far bound of conventional methods cannot be applied. Thus, the proposed method places the camera in a distance where the is visible given the field of view.
- Camera multiplex optimization. It is challenging to optimize the camera pose due to local minima. The proposed methods optimize multiple camera poses with their corresponding weights based on the camera loss to find the best possible camera pose. 
- Posterior scaling of input images. Similar to camera multiple optimization, the proposed method also optimizes the image collection used for training. It gives different weights based on the noise level for each image.
- The proposed method can be applied for various applications, such as AR & VR, which makes it easy for those applications to insert the real-world objects without huge efforts. 


# Weaknesses

- The authors claim that standard pose estimation techniques fail on the challenging images, but there is no justification whether the proposed dataset is a challenging dataset. Proof of pose of estimation failure should be included.
- While the conventional methods automatically estimate the camera pose with COLMAP, the proposed method requires user interaction for each dataset to roughly annotate the position. However, there is no justification why the proposed rough pose estimation is preferred. In addition, there is no ablation study of the camera pose initialization using 3 simple binary questions.
- The intention of the proposed method is to do proof-of-concept of large-scale set objects. However, the qualitative evaluation of large-scale set objects is unavailable in the main manuscript. It is questionable whether the proposed method can be scaled to large-scale set objects. In the supplementary material, there are two online image collections, but the results are not satisfying. In addition, comparison with state-of-the-art methods is unavailable.
    - As in Modeling the World from Internet Photo Collections, the camera pose of large objects images might be estimated using correspondences.

questions: - In L157, the authors claim that they do not use the coarse-to-fine network. What if the proposed method use coarse network? Wouldn’t the accuracy be better? In addition, what is the coarse-to-fine optimization in the ablation study?
- Why couldn’t the proposed method achieve GT camera pose? What if the proposed method utilizes deep camera pose estimation (such as Wide-Baseline Relative Camera Pose Estimation with Directional Learning) to improve the accuracy? Would it be possible?

limitations: Yes, the authors has adequately addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a method that works on in-the-wild image collections of an object to estimate the shape, BRDF, per-image camera pose, camera intrinsics and illumination in a jointly optimized framework.
This problem is very challenging and under constraint as the camara parameters, illuminations and even the background of the images may vary a lot across the online image collections. To my knowledge, this is the first method that aims at solving the shape, BRDF and camera poses with this challenging setup. 

The paper proposed several components that based on the recent Neural-PIL-Rendering technique to construct the entire pipeline, including: an object-centric camera parametrizetion to learn the clipping planes per image; camera multiplex optimization to avoid local minima during the optimization; posterior scaling of input images to suppress the influence of corrupts images; a two-stage mesh extraction for refined mesh reconstruction.
strengths_and_weaknesses: Strengths 
- The paper is well written and easy to follow. The figures, tables and videos help the understanding of the paper and identify its contributions.

- The proposed camera multiplexes, although been used in mesh optimization works [22] before, is the first time being used in the context of neural volume rendering. This technique will dynamic re-weighing the each camera loss to reduce the influence of bad camera poses during the optimization of the shape and materials.

- The proposed posterior scaling of input images also makes sense in the context of suppressing the influence of corrupted images. This technique can also be used in a broader application: when dealing with online image collections and images with poor quality can be dynamically re-weigh.

- The evaluation of the method over real and synthetic datasets demonstrates the effectiveness of this method in recovering shape and material under unknown camera poses, changing illuminations and varying backgrounds. (From Table 1 and Table 3, Table A3)

- The Ablation study in table 2 also validates the effectiveness of each component of the methods. 

Weaknesses
- In table 1, the SAMURAI's performance is lower when the camera poses are available. Why?

- I have a concern about the quality of the reconstructed meshes. In Figure A4, the reconstructed mesh of "Statue of Liberty" and "Chair" is inaccurate, and the texture is also blurry. 

- The novel view synthesis quality is very low compared to the prior "fix-illumination" or "known camera pose" methods. As can be seen in Table 1, Table 3 where the reconstructed images have much lower PSNR compared to prior works. 
questions: See weaknesses part.
limitations: The most of the limitations are discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a NeRF-based single-object inverse rendering pipeline where the camera intrinsics is unknown and extrinsics are coarsely initialized. The neural implicit based reflectance (BRDF) is also optimized separately for relighting/material editing. Compared with previous methods that uses camera intrinsics and extrinsics estimated from a third-party pipeline (eg. COLMAP), the proposed method is more robust in cases where COLMAP fails to estimate accurate poses due to the camera optimization module. A small dataset consisting of images captured with different background/illumination for 8 objects is collected and used in evaluation of the proposed method. The experiment results show that the proposed method outperforms the compared state-of-the-art methods that also use unposed input images. 

strengths_and_weaknesses: Strength

+The joint optimization of the BRDF, occupancy represented as neural implicit functions and the camera parameters (poses and intrinsics) given unposed images captured under different illuminations/backgrounds is a challenging task. The recipes used in this submission can be of good reference for the following works that pushes the NeRF-based inverse rendering towards in-the-wild scenarios.

+Compared with the similar method (BARF), the performance boost of the proposed pipeline is obvious.

+A small dataset is proposed for challenging cases where the COLMAP fails (L276-L277). This dataset can be useful for testing algorithms using unposed images with varying illumination as inputs.


Weakness

-My main concern is about the limitation in the cases where the proposed method performs better than the compared method: 
Other methods, such as NeRF-W, also uses internet photos with varying background/illumination/camera intrinsics as inputs; the key difference between the work and the previous works is that this work is robust to cases where COLMAP fails to estimate good camera poses or does not work at all, versus the assumption that COLMAP works for in-the-wild dataset in other methods. Although this work has proposed a small dataset on which COLMAP is not working at all and shows that the proposed method works on this dataset, the following question is not answered: if COLMAP does estimate some noisy camera poses, can this method with optimized pose method still output perform the counterparts (either the other parts of this method + COLMAP pose, or NeRD + COLMAP pose) ? From Tab.3 of this paper, it shows that if COLMAP generates good camera pose, then the performance of the proposed method is not as good as the compared methods. This seems to reduce the application scope of this proposed method: it only works better if COLMAP does not work, which is not always the case given that COLMAP has decent robustness.

-Self occlusion/shadowing is not considered. This poses a challenge for objects with concave shapes (grooves) that generate cast shadows, which may be reconstructed as albedo variations.

-Based on L181-L199, the 'valid' sampling ranges for the rays are determined by the intersections between the predefined object-centric sphere and the camera rays, which are intern determined by the camera pose and intrinsics. As a result, the sampling location along the rays depends on the camera intrin/extrin. If that is the case, how the dependence is dealt with during the optimization process is not well explained. 

-The weighting scheme in camera multiplexes depends on the reliability of each candidate pose. How about the cases where none of the positions from one multiplex is reliable? Are the corresponding images ignored during reconstruction?

questions: Please see the questions in the weakness section.
limitations: The method does not specifically handles shadows, as the author pointed out in the paper.
review_title: null

Meta-review: metareview: This paper had notable consistent reviews.  All reviews were thoughtful, and there was a consensus that this paper tackles an important problem in a way that has not been explored.  While there were some weaknesses highlighted in the review process, discussion and the author rebuttal ameliorated all major concerns.  Therefore I am accepting this paper.
confidence: Certain
Decision: Accept