Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yicong Hong
authors: Jiuxiang Gu
authors: Yang Zhou
authors: Difan Liu
authors: Feng Liu
authors: Trung Bui
reviews: 
Comments: The paper proposes a large reconstruction model LRM, predicts the 3D model of an object from a single input image within 5 seconds .LRM adopts a transformer-based architecture with 500 million learnable parameters to directly predict a neural radiance field from the input image The model is trained with multi-view data containing around 1 million objects, including both synthetic renderings from Objaverse and real captures from MVImgNet. This make LRM to be highly generalizable and produce high-quality 3D reconstructions from various testing inputs including real-world in-the-wild captures and images from generative models.
Strengths: The strengths of the paper are as follows:

1. The paper designs a 3D foundation model for reconstruction, which intrinsically establishes a 3D prior knowledge model. It fills the gap between 2D and 3D foundation generative model. This contribution makes this work not only achieve SOTA 3D generation result, but advances the 3D deep learning. It has potential to influence the 3D learning community as BERT for NLP, or DALLE2/ Muse for 2D generation.

2. The paper figures out a learnable positional encoding to generalize NeRF generation conditioning on 2d image features. The architecture pave the way for other 3D generative models to scale up.

3. The paper drastically improves the generation speed for 3D assets, which will help democratize the technologies in the burgeoning fields like 2d-to-3d or text-to-3d. The potential quasi-real time applications can make 3D content generation more accessible to common users, which in turns, will benefit the development of the community.
Weaknesses: 1. As mentioned in the limitation, the back side of the model (e.g., figure 3 penguin) is blurry and besides that, the color tone/ saturation seems a bit inconsistent with the frontal view.

2. The model relies on the correct camera parameters and normalization of camera, which still based on the fact that the objects in the training dataset (objectaverse?) have a certain distribution of orientation. The more general way to formulate the scene is to assume the world coordinate is the camera coordinate (extrinsic = np.eye(4)),  which make the object always face forward to the camera. Although, from the geometric learning works in shapenet, this approach is more challenging than appropriating the object orientation.

3. The approach relies on multiview data which limits the model incorporating more diversity in data source of single view images.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper proposes the Large Reconstruction Model (LRM) for image-to-3D task. Different from previous works that distill an image-to-image model for this task, the LRM directly predict the 3D NeRF representation from the input image. In detail, authors utilize a transformer encoder-decoder architecture to predict the triplane NeRF, and trained on a large-scale multi-view dataset including Objaverse and MVImgNet. Qualitative results on images from different source show the superior performance of the proposed LRM. In addition authors also provide an extensive ablation on different design components.
Strengths: This paper is well-written and the results are promising. Although there were papers trying to train a generalizable nerf predictor, this paper proves the possibility of training on a large-scale dataset for the generalizable nerf prediction. To my knowledge, this is the first attempt to train it on scale like Objaverse + MVImgNet. The experiment part is well-organized and sufficient, provide a thorough ablation for different model components.
Weaknesses: I don't think there exists any apparent weakness in the paper. Please refer to the following questions part for my other questions regarding the details of paper.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good
reviews: 
Comments: The paper proposes LRM, a Transformer-based NeRF model that transforms an input image into its corresponding 3D triplane representation.
Compared to recent image-to-3D approaches that rely mainly on post-processing/optimization strategies to extract 3D representations from pre-trained image diffusion models, LRM stands out by directly establishing a 2D-to-3D mapping.

Qualitative results show improved performance in generating novel views and synthesizing unseen objects

----------
Post discussion phase:
I agree with the other reviewers that the paper is a good contribution to the image-to-3D problem, and the authors' responses have addressed all my questions. Therefore, I raised the score from 6 to 8 (good paper, accept).
Strengths: The paper presents an interesting system that hallucinates/synthesizes 3D appearance from DINO features. Compared to the recent methods, LRM excels at
- Directly producing 3D representations in a single forward pass, instead of running optimization to construct a 3D model for each input instance.
- LRM retains details from the input view better, possibly due to the use of image-based features.
- LRM does not require canonicalized training objects, making it easier to apply LRM to other datasets.

Additionally, the paper presents comprehensive ablation studies, showing how each design choice affects the final performance. Overall, the method is presented clearly and easy to follow, the architecture design is sound, and the qualitative results look promising.
Weaknesses: Although the results look promising, the paper has two main weaknesses:
- Insufficient quantitative comparisons: the paper does not conduct any quantitative evaluation against other methods. I believe the novel view synthesis and 3D reconstruction can be evaluated on the held-out sets for those 3D object datasets, and user study should also be possible. Even if the quantitative results may not reflect the generation quality entirely, the paper should include discussions on why these scores are not reliable/not feasible.
- The quality on the occluded side is still limited. For objects that have overall smooth/uniform textures, LRM seems to do a good job filling in the occluded side information. But for more complex, asymmetric patterns (e.g., Figure 2. Giraffe,  supp. website shoe example), LRM struggles to synthesize plausible appearances.

In summary, the absence of quantitative comparisons raises significant concerns.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper presents the first *large-scale* single-view 3D reconstruction method. Given an input image, it uses DINO to extract features from an input image, then uses a *large* transformer with cross-attention and modulation to process these features into triplanes, creating a triplane NeRF, thereby conducting novel view synthesis. The method is trained on large-scale 3D/multiview datasets Objaverse and MvImgNet, with 128 A100 GPUs.
Strengths: * To my knowledge, this paper is the first work showing the scaling ability of transformers on novel view synthesis.
* The task of single-view novel view synthesis is extremely challenging and well-motivated.
* The method is extremely efficient during inference as it only requires a single forward pass, unlike many generative models, e.g. score-based generative models and/or optimization-based methods.
* As a non-generative model, it is astounding for me to see its amazing performances. It seems to be able to handle the ill-posed one-to-many pretty well, despite being a discriminative model.
Weaknesses: * The method requires significant computational resources.
* A minor issue, but the paper shows no quantitative comparison with any prior work.
* Since the method is discriminative, it is not able to sample different realizations of an input. Additionally, the averaging of modes, even though has been weakened a lot compared with prior works such as PixelNeRF, still exists as the author mentioned.
* The task of novel view synthesis is inherently probabilistic, as the author mentioned. Even though the method shows amazing scaling-up ability, it is questionable whether solving a generative task in a discriminative way is reasonable.
* The contribution of this paper comes mainly from its presentation of the ability of large transformers and large-scale 3D data on novel view synthesis. Technically, the efforts mainly come from the engineering efforts combining different techniques and processing with the data.
Recommendation: 8: accept, good paper
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 4 excellent
Clarity: 4 excellent