Kati8/ICLR_2023_2025 huggingface dataset:
authors: Peng Wang
authors: Hao Tan
authors: Sai Bi
authors: Yinghao Xu
authors: Fujun Luan
authors: Kalyan Sunkavalli
authors: Wenping Wang
authors: Zexiang Xu
authors: Kai Zhang
reviews: 
Comments: The paper proposes a 3D reconstruction method that focuses on shape and pose predictions given a sparse set of un-posed input images. The proposed method consists of two major components: 1) a transformer that aggregates DINO-based image tokens from multi-view images and 2) a DSAC-fashion pose estimator that predicts 3D points coordinates from aggregated features followed by a differentiable PnP refinement. While aggregating multi-view features, the transformer also enables shape reconstruction by predicting NeRF parameters in terms of Triplane features.
Strengths: 1. The paper is generally well written and easy to follow.
2. The method looks convincing and the paper is addressing a critical challenge in this field.
3. The anonymous website provides extra demos.
Weaknesses: 1. In Figure 3 and Figure 4, I am confused about why the results are comparing input images with their predictions? Shouldn’t it be comparing GT novel view images with rendered novel views, as in Figure 5? Given input images, it’s not surprising that the reconstruction of input images should be almost perfect right?
2. The first issue leads to further confusion when reading Figure 4. I am not sure I understand correctly, but it seems like pose prediction makes limited difference.
3. Missing related works. Section 3.3 is about pose predictions and PnP. This approach is closely related to the line of work in *scene coordinate regression* [a][b], which is not mentioned at all in the related work section.

**Reference**

[a] Shotton, Jamie, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. "Scene coordinate regression forests for camera relocalization in RGB-D images." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2930-2937. 2013.

[b] Brachmann, Eric, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. "Dsac-differentiable ransac for camera localization." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6684-6692. 2017.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper proposes a method to reconstruct 3d models from unposed sparse-view images. To enable this goal, the authors use a single-stream transformer to simultaneously process image patch tokens and nerf tokens, resulting in a simultaneously estimation of triplane-nerf and camera poses. The estimation of camera poses utilizes predicted per-view coarse geometry and differentiable pnp. This method shows strong cross-dataset generalization ability and outperforms baselines on various datasets.
Strengths: 1. The authors conducted rich experiments to prove the effectiveness of their method. 
2. Reconstructing 3d models from sparse-view images is an important topic in the AIGC era. 
3. Employing differentiable pnp into nerf reconstruction is a good idea to handle camera pose uncentainty.
Weaknesses: The authors mainly handled the cases of 2-4 views (e.g. Figure 1). Is there any criteria for the authors to select image views? It seems that the tested camera views have suitable (or informative) relative angles (about 30 degrees to about 120 degrees). What would happen if extreme/degenerated views are provided (e.g. the front and back views that are parallel, or views with small angles, or views with translational displacement only)?   I think explaing this would make the study stronger.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 3 good
reviews: 
Comments: The paper presents an amortized inference model for a fast NeRF reconstruction of 3D objects from a few source views. These sources are either obtained from a view-aware diffusion model such as zero123 or give a ground truth. The work trains a large transformer model i.e. 0.59B parameter model for fast (1.3s) inference on A100GPUs. The transformer model outputs coarse pointclouds as well as triplanes for differentiable PNP based pose estimation and NeRF reconstruction respectively. The model is trained on Objaverse dataset and shows generalization capability on other datasets such as ABO, GSO, Omni Objects3D etc.
Strengths: The paper proposes a technically sound amortized inference approach for 3D object reconstruction in a fast manner from a few unposed images. The strnegths of this paper are as follows:

1. Clearly better quantitative results against relevant pose predictor baselines
2. Good ablations showing how various factors affect the model performance, especially mask noise
3. A good downstream application of the method showing text-to-3D results which is an important application for this work
Weaknesses: Although the paper is nicely written and the experiments are good, I have the following questions:

1. How much of the improvement comes from the data? The pipeline is not that novel since all of which the authors have presented have been shown before, even the differentiable PnP solver comes from prior work. I would have liked to see scaling laws of the performance on the increased number of data points.

2. No comparison with sparse NeRF methods[1, 2, 3] is shown. Is this because they require accurate camera poses? I would have liked to see some comparison or discussion with these methods, some of them already show sparse novel view synthesis using triplane formulations[1]

3. Relating to point 1, how much does the network learn training data distribution? Is it possible that due to the massive scale of the objaverse dataset, some of the generalizable evaluation, might be in questions, since we don't know if the network has seen similar data before. Can the authors comment on that with some analysis/discussion?

[1] Irshad et al, NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes
[2] Niemeyer et al, RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs
[3] Truong et al, SPARF: Neural Radiance Fields from Sparse and Noisy Poses
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 4 excellent