Kati8/ICLR_2023_2025 huggingface dataset:
authors: Hanwen Jiang
authors: Zhenyu Jiang
authors: Yue Zhao
authors: Qixing Huang
reviews: 
Comments: **Summary**: The paper proposes a 3D modelling method that predicts a NeRF volume from sparse input views without requiring camera poses at inference time. 

**Method**: The main contribution is proposing an attention-based structure and a feature volume to associates image features across all views and between 2D and 3D. Essentially the method is similar to PixelNeRF and IBRNet, but does not need camera poses at inference time. 

**Evaluation**: The evaluation is comprehensive and convincing. It would be better to add some cross-dataset evaluation.
Strengths: 1. The paper is well written and easy to follow.
2. Code is provided in supplementary.
3. The proposed method is novel. The proposed method alleviates the need of camera poses while being simple and effective, by applying attention across image features from all source views and attention between a feature volume and source views.
4. The evaluation is convincing. The paper shows clear improvement comparing with previous methods and offers extensive analysis and discussions.
Weaknesses: While the experiment is convincing, it appears to me that all experiments are trained and evaluated on train/test splits of same datasets. It would be interesting to see cross dataset performance and a comparison between other method, i.e. training on dataset A and evaluating on dataset B.

A minor issue: missing a reference to *Sajjadi, Mehdi SM, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučić, and Klaus Greff. "RUST: Latent Neural Scene Representations from Unposed Imagery." In CVPR 2023*.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: The paper introduces a new methodology designed to transition Nerf from a pose-based optimization framework to a pose-independent reconstruction paradigm. Within the fusion phase, the network generates a feature volume, with the initial view serving as the canonical reference point, subsequently integrating further perspectives by means of feature matching between the 3D volume and 2D image feature similarity. This design ensures that the entire process remains agnostic to pose, obviating the necessity for explicit pose optimization.

The paper conducts an exhaustive series of experiments to substantiate the robustness and efficacy of the proposed methodology, employing both object-centric and scene-centric datasets. The findings conclusively demonstrate that the proposed approach yields results on par with previous works that rely on ground-truth poses, while exhibiting superior generalizability in comparison to its predecessors.
Strengths: Originality:
The reviewer did not identify a comparable concept within the existing literature, suggesting that the idea presented in this paper is novel and distinctive. Furthermore, the concept exhibits significant potential for broader applications across various use cases, further underscoring its relevance and practical utility.

Clarity:
The problem statement, literature review, and a portion of the methods and experimental details are clear to me.

Quality and significance:
The paper exhibits a well-structured organization that facilitates ease of comprehension. The experimental design effectively showcases the efficacy of the proposed methodology.
Weaknesses: 1. Clarity in the exposition of the method's approach to generating predicted images during the optimization process would be beneficial.
2. It would be advantageous if the paper delved deeper into the reasons behind its enhanced speed and the trade-offs involved in achieving such acceleration.
3. The proposed methodology presents certain limitations, particularly concerning relative poses. A more detailed exploration of how the network achieves accurate scale predictions without pose information would be insightful.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: This paper proposes a pose-free method for novel view synthesis in sparse-view settings. Unlike previous methods that estimate/optimise camera poses, the proposed approach uses a novel transformer-based 2D-3D mapping method to aggregate 2D image features in 3D space. After training in large-scale data with ground-truth poses, the method is able to generalize to new scenes without the pose input. It shows that the proposed method outperforms previous approaches that use estimated camera poses in both object-level and scene-level datasets.
Strengths: 1. The idea is novel. Unlike previous methods that try to predict or estimate camera poses in sparse views, the proposed method completely did not use camera pose to build the 3D volume representation.

2. The paper is well-written. It is easy to read and understand the motivation, background, problem, and high-level ideas to address the challenge.

3. The proposed multi-view encoder and the 2d-3d information mapping layers are novel, and efficacy has been demonstrated in the ablation study.

4. The experimental results are evaluated in both object-level and scene-level datasets, and it shows that the proposed method outperforms previous methods when they use estimated camera poses.
Weaknesses: 1. Unlike the traditional pose-based projection, the proposed 2D-3D mapping layers are a weighted fusion of 2D features. The mapping may be more robust than pose-based projection when the pose is inaccurate but it limits in accuracy. Consequently, the reconstruction and rendering are often blurred, as shown in Figure 6. Do authors have ideas to improve it?

2. Sparse-view reconstruction is an ill-posed problem because the input images contain incomplete scene information. Although the proposed method is better than previous approaches, the performance is still very limited. In this scenario, most of the existing methods can refine the results by using more input images. However, the proposed method only aggregates information and runs forward pass at the inference time. Is it possible to do refinement when more input images are given?

3. Does the proposed method work well with different frame numbers during inference time? As the proposed 2d-3d mapping needs aggregating information from all frames, does it consume huge memory in dense views (e.g., more than 100 frames)?
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: The paper proposed LEAP, a pose-free approach for 3D modeling from a set of unposed sparse-view images. By appropriately setting the 3D coordinate and aggregating 2D image features, LEAP demonstrates satisfying novel view synthesis quality.
Strengths: ++ As a pose-free approach, LEAP discards pose-based operations and learns geometric knowledge from data. 

++ LEAP is equipped with a neural volume, which is shared across scenes and is parameterized to encode geometry
and texture priors. For each incoming scene, it updated the neural volume by aggregating 2D image features in a feature-similarity-driven manner. The updated neural volume is decoded into the radiance field, enabling novel view synthesis from any viewpoint.

++ The experimental evaluations and ablation studies are extensive.
Weaknesses: -- Novel view synthesis is defined as rendering the images as specific camera pose and time (for dynamic scenes). When the camera poses are not avaliable or not estimated as this paper, how to deal with NVS with given camera poses, i.e., how to align the given camera poses with the training set images.

-- Essentially, the method incoorporated the feature correspondences into the overall optimization, it is thus interesting to make comparisons with estimated correspondences from optical flow where the dense matching is learned.

-- It is worth to compare related work such as Unposed NERF to verify the effectiveness of the proposed method.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 3 good
Clarity: 3 good