GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation
Authors: Chubin Zhang
Hongliang Song
Yi Wei
Chen Yu
Jiwen Lu
Yansong Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large reconstruction model
3D generation
3D gaussian splatting
Primary area: generative_models

Abstract: In this work, we introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. This limits these methods to a low-resolution representation and makes it difficult to scale up to the dense views for better quality. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms to effectively integrate image features into 3D representations. We implement this solution through a two-stage pipeline: initially, a lightweight proposal network generates a sparse set of 3D anchor points from the posed image inputs; subsequently, a specialized reconstruction transformer refines the geometry and retrieves textural details. Extensive experimental results demonstrate that GeoLRM significantly outperforms existing models, especially for dense view inputs. We also demonstrate the practical applicability of our model with 3D generation tasks, showcasing its versatility and potential for broader adoption in real-world applications. The project page: https://linshan-bin.github.io/GeoLRM/.

GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation
Chubin Zhang 1.3
Hongliang Song Yi Wei
Yu Chen Jiwen Lu
Yansong Tang'
1 TTsinghua Shenzhen International Graduate School, Tsinghua University
DDeartmentt of Automation, Tsinghua University
Alliabaa Group
{zcb24, y-wei19 1trl119mmmmlss.ssnnnmi. edu.aua..uinnnnuuu..... edu.cn, cn,
{hongliang, shl, chenyu.cheny wwweeeeelllaaaaaaaeeeeeaaaeeaaae com.
lujivenOtsinghua. edu. cn, tang. yansongesz. tsinghua edu. cn.
corresponding author
Abstract
In this work, we introduce the Geometry- Aware Large Reconstruction Model (Ge- oLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. This limits these methods to low-resolution representation and makes it difficult to scale up to the dense views for better quality. GeoLRM tackles these issues by incorporating a novel 3D-rwa transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms effectively integrate image features into 3D representations. We implement this solution through two-stage pipeline: initially, lightweight proposal network gen- crates a sparse set of 3D anchor points from the posed image inputs; subsequently, specialized reconstruction transformer refines the geometry and retrieves textural details. Extensive experimental results demonstrate that GeoLRM significantly outperforms existing models, especially for dense view inputs. We also demon- strate the practical applicability of our model with 3D generation tasks, showcasing its versatility and potential for broader adoption in real-world applications. The project page: cittps://linshe bin, www..iiiinniibb..... io/GeeLRM/
Introduction
In fields ranging from robotics virtual reality, the quality and diversity of 3D assets can dramatically influence both user experience and system efficiency. Historically, the creation of these assets has been a abbour-intensive process, demanding the skills of expert artists and developers. While recent years have witnessed groundbreaking advancements in 2D image generation technologies, such as which iteratively refine images, their adaptation 3D asset creation
diffusion models [43
44 42
remains challenging. Directly applying diffusion models to 3D generation [20] 36 is less than satisfactory, primarily due to dearth of large-scale and high-quality data. DreamFusion (40) innovatively optimize a 3D representation pre-trained image diffusion models
2 by distilling the score of image distribution from However, this approach lacks deep integration of 3D-specific knowledge, such as geometric consistency and spatial coherence, leading to significant issues such as the multi-head problem and the inconsistent 3D structure. Additionally, these methods require extensive per-scene optimizations, which severely limits their practical applications.
44
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

number inpal views
11
21
Input Image
Rendering
Figure I: Image to 3D using GeoLRM. Initially, 3D-aware diffusion model, specifically SV3D (60, transforms an input image into multiple views. Subsequently, these views are processed by our GeoLRM to generate detailed 3D assets. Unlike other LRM-based approaches, GeoLRM notably improves as the number of input views increases.
The introduction of the comprehensive 3D dataset Objaverse [12] [12111] [1] brings significant advancements for this field. Utilizing this dataset, researchers have fine-tuned 2D diffusion models to produce images consistent with 3D structures [28] 47 (48). Moreover, recent innovations
774
64
72
655 have
18 to achieve rapid and accurate 3D image generation. These methods typically employ large transformers UNet models that convert sparse-view images into 3D representations in a single forward step. While they excel in speed and maintaining 3D consistency, they confront two primary limitations. Firstly, previous works to represent the 3D models, wasting lots of features in regions devoid of actual content and involving dense computations during rendering. This violates the sparse nature of 3D as our analysis shows that the visible portions of the 3D models in the Objaverse dataset constitute only about 5% of the overall spatial volume. Though Gaussian based methods [54] 74 655 may use pixel-aligned Gaussians for better efficiency, this representation is incapable of recovering the unseen area and thus heavily relies on the input images. Secondly, previous works tend to overlook the explicit geometric relationships between 3D and 2D images, which results in ineffective processing. The tri-plane or axileed Gaussian tokens do correspond a specific space in 3D, thus being unable to utilize the projection relationship between 3D points and images. In other words, they conduct dense attention between the 3D queries and the image keys. This leads to the fact that these methods tend to reconstruct 3D with sparse view inputs but cannot achieve better performance with denser inputs.
combined these 3D-aware models with large reconstruction models (LRMs)
utilize triplanes [8]
72 [c)
To address these challenges, we introduce the geometry-awand large reconstruction model (GeoLRM) for 3D Gaussian generation. Our method centres on 3D-aware reconstruction transformer that eschews conventional representations like triplanes or pixel-aligned Gaussians in favour of direct interaction within the 3D space. However, directly generating 3D Gaussians in the whole 3D space requires huge memory costs. To this end, we first propose specialized proposal network to predict an occupancy grid from input images. Only the occupied voxels will further processed to generate 3D Gaussian features. The proposed transformer replaces the dense cross attention with deformable cross attention [86] By projecting the input 3D tokens onto the corresponding image planes, these tokens only focus the most relevant features, which greatly improves the effectiveness.

---

We trained our GeoLRM on the Objaverse dataset rendered by [4] and tested it on the Google Scanned Objects [T3. By integrating geometric principles, our model not only outperforms existing methods with the same number of inputs but also makes it possible to work with denser image inputs. Significantly, the model efficiently handles up to 21 images (even more necessary), yielding superior 3D models in comparison to those generated from fewer images. Leveraging this capability, we integrated GeoLRM with SV3D [60] for high-quality 3D model generation.
In summary, our contributions are as follows:
We introduce two-stage pipeline that leverages the sparse nature of 3D data, resulting in sparse 3DGS token representation suitable for extension to high resolution.
We fully exploit the projection relationship between 3D points and images, significantly reducing the space complexity of attention mechanisms in LRMs, thus enabling denser image input configurations.
To the best of our knowledge, GeoLRM is the first to process dense inputs using LRM, po- tentially paving the way for integrating video generation models into 3D AIGC applications.
2 Related Work
2.1 Optimization-basse based 3D reconstruction
3D reconstruction from multi-view images has been extensively studied in computer vision for decades. While traditional methods like SIM 68 58 45 and MVS [46 [6] provide basic recon- struction and calibration, they lack robustness and expressiveness. Recent advancements leverage
learning-basee based methods for better performance. Among these methods. NeRF (33 stands out for its capability of capturing high-frequency details. Following works [2] improve its performance and speed. Though NeRF has made a great improvement, tons of points during the rendering process makes it hard for real-time applications. 3D Gaussians [2]] solves this problem by explicitly expressing scene with 3D Gaussians and utilizing an efficient rasterization pipeline. These methods involve a per-scene optimization process and require dense multi-view images for good reconstruction.
83
3 34 77
8
53 further need to query
2.2 Large Reconstruction Model
Different from optimization-bann based 3D reconstruction methods, large reconstruction models 18 22
82 62 64) are able to reconstruct 3D shapes in a feed-forward way. As the pioneer work of this area, the LRM 18 illustrates that the transformer backbone can effectively leverage the power of scale datasets and translate image tokens into implicit 3D triplanes under multi-view supervision. Beyond LRM, Instant3D [22 improves reconstruction quality with sparse- sparse-view inputs. It employs two-stage paradigm, which first generates four views with the diffusion model and then regresses NeRF 33 from generated multi-view images. Instead of NeRF, InstantMesh 772 utilizes mesh representation to reconstruct 3D objects, which adopts differentiable iso-surface extraction choose 3D Gaussians 21 as the outputs. GRM (74) proposes transformer network to translate pixels to the set of pixel-aligned 3D Gaussians while LGM [54) uses an asymmetric UNet to predict and fuse 3D Gaussians. Compared with these methods, our GeoLRM projects multi-view features to the 3D space with cross-view attention mechanisms, which explicitly explores geometric knowledge.
54 [4| 65
module. However, many works (54 82
2.3 3D generation
Early methods 6 contents. Although some methods 32 32 models for high -uality generation, their generalization ability is bounded by the limited training data. Recently, proposed in DreamFusion [40], score distillation sampling (SDS) requires no 3D data and
35
51
73
37 in 3D generation area utilize 3D GANs generate 3D-aware
30
10 49 80 replace 3D GANs with 3D diffusion
able to leverage the great power of 2D text-to-image diffusion models 44 43 42 Specifically, optimizes a randomly-inittalized 3D model and diffuses the render images with a pretrained diffusion many methods have been proposed to accelerate the optimization process improve 3D generation quality. Different
model. As the follow-up works 6 26
55
25
23
3

---

Projection C
Concatenate
Proposal Transformer
XN
Dense Tokens
Occupancy Grid
Sparse Tokens
K, V
Self-Atrention I Layer
Plicckr rays
Conv
(FT.)
(F
DINOv2
Hierarchical Image Features
Multi-view Images
Deformable Cross Attention
MLP
Feod-Forward Network
Rendered Novel Views
3D Gaussians
Gaussian Tokens
Reconstruction Transformer
Figure 2: Pipeline of the proposed GeoLRMM, a geometry-powered method for efficient image to 3D reconstruction. The process begins with the transformation dense tokens into an occupancy grid via Proposal Transformer, which captures spatial occupancy from hierarchical image features extracted using a combination of a convolutional layer and DINOv2 B8 Sparse tokens representing occupied voxels are further processed through Reconstruction Transformer that employs self-attention and deformable cross-attention mechanisms to refine geometry and retrieve texture details with 3D projection. Finally, the refined 3D tokens are converted into 3D Gaussians for real-time rendering.
with SDS-based methods, Zero-1-to-3 28 fine-tunes the 2D diffusion models on large-scale
synthetic dataset change the camera viewpoint given image. Similar to Zero- 1-to-1, many other works [7
67 31 (9) aim to synthesize multi-view consistent images. Our method can reconstruct 3D contents based on these synthesis multi-view images.
48
75 29
Methodology
3.1 Overview
Figure 2 fillustrates the pipeline of our proposed method. Our approach takes a set of images {T') N with their corresponding intrinsic [K*) and extrinsic { as input. Initially, we encode input images into hierarchical image features and predict an occupancy grid with proposal transformer. Each occupied voxel within this grid is considered a 3D anchor point. These 3D anchor points are then processed by reconstruction transformer, refining their geometry and retrieving textural details. The proposal and reconstruction transformers share the same model architecture, which is further discussed in Section The outputs the reconstruction transformer are decoded into Gaussian features with shallow MLP for rendering. Loss functions are described in Section 3.3
/i=
3.2 Model Architecture
Our model architecture features a hierarchical image encoder for extracting high and low-level image feature maps along with geometry-aware transformer for lifting 2D features into 3D representations.
Hierarchical Image Encoder Our method integrates both high and low-level features to enhance model performance. For high-level features, we utilize DINOv2 (38) which excels in single-image 3D tasks M. To capture low-level features, we combine Plücker ray embeddings and RGB values. The Plücker ray parameterizes each corresponding to pixel by (d, ox d), with d representing the ray's direction and origin (50 75). These embeddings, denoted as R² for each image T", are concatenated with the RGB values of the image. This combined data is then integrated through

---

convolution layer. The encoding processes are succinetly described by the equations:
F DINOv2(1"),
(1)
JL Conv(Conca(("", RR)),
where Fli and FL represent the high and low-level feature maps of image I", respectively.
Geometry-aware Transformer The geometry-aware transformer aims to efficiently lift image features to 3D. The proposal transformer and reconstruction transformer are both instances of this architecture. Previous methods [8]] 54
use tri-planes or pixel-aligned Gaussians to represent 3D contents. However, these data structures make it hard to utilize the projection relationships, causing dense computations. Instead, we use 3D anchor points, which serve as proxies for their surrounding points, significantly reducing the number of points we need to process. As detailed in Figure 2 each transformer block contains a self-attention layer, a deformable cross- attention layer and a feed-forward network (FFN). The model takes N anchor point features Fa {fi as input tokens. Each token f. comprises the coordinate of the corresponding point and a shared learnable feature.
74
65 82]
For the self-attention layer, a crucial problem is how to inject positional information into the sparse 3D tokens. We extend the Rotary Positional Embedding (RoPE) (52 to 3D conditions for relative positional embedding. For query Im and a key kn at absolute position m and n, we ensure that the inner product of embedded values reflects only the relative position information yet promising way is splitting the features into three parts and applying RoPE [52] on each part with y, and z positions respectively.
A direct
As we can locate each anchor point in the 3D space, a possible way lift 2D features to 3D is to project them to the feature maps with known poses and average the corresponding features. However, this method assumes an accurate anchor position, an equal contribution of all images and good 3D correspondence of input images, which is often impractical, especially in 3D generation tasks. To tackle these issues, we employ deformable attention [6 24 [6) for robust fusion of image features. Given a 3D anchor point feature f its spatial coordinate o; and multiple feature maps {F²) J==1> the deformable attention mechanism is formulated as:
DeformAttn(f: 2i
a;; (F (F') (
1c
A (P:v APiva))
where k indexes the sampled keys and K is the total sampled key numbers. Pie is the projected 2D coordinate on feature map F and Ap,.k is the sampled offset. (.) indicates the interpolation operation. A the attention weight predicted from f.- w. a per-view weight derived from the feature it weights. Notably, the prediction of Apick allows the network to correct the geometry error of anchor points and the inconsistency of input images; The w. enables different importance levels for each image. To further enhance the representation ability of the model, this mechanism is extended to multi-head and multi-scale conditions.
Given input tokens Fin, the transformer block enhances these tokens through a series of sophisticated transformations described as follows:
Fxel/ Fi + SelfAtt (F
(4)
FX f + DefommCrossAt((RMSNoom????"" A,,,F,, FDM
(5)
Fxt = FX Foross FEN(RMSNOM((FT00)
(6)
This design introduces several improvements over the original transformer architecture [59] By
incorporating RMSNorm [79] for normalization and SiLU [14] for activation, we achieve more stable training dynamics and better performance.
Post- post-processing The proposal network takes a low-resolution dense grid (162) as anchor points. The output is upsampled to a nigh-reeoluttin grid (1283) with a linear layer. This grid formulated to represent the occupancy probability of the corresponding area ([-0.5. 0.5]3). The reconstruction transformer takes occupied voxels as anchor points. Each output token f. is decoded into multiple 3D Gaussians (G Mr with multilayer perceptron. The 3D Gaussian G.y is parameterized by the offset O:j regarding the anchor points, 3-channel RGB cij. 3-channel scale 8ij. 4-channel rotation quaternion a Tig, and 1-channel opacity aij. We employ activation functions to limit the range of the
5

---

offset, scale and opacity for better training stability similar to [54]
Oty Sigmoid(o Omax
(7)
8ij Sigmoid s'yy) Smax
(8)
02j Sigmoid(a(y))
where Omax, Smax are predefined maximum values of offsets and scales. Given target camera views {er the 3D Gaussians can be further rendered into images {1.)=1> alpha masks {Me){= and depth maps { (ir through Gaussian splatting 21
3.3 Training Objectives
We employ two-stage training mechanism for our model. In the first stage, we train the proposal transformer using 3D occupancy ground truth. This stage presents challenge as it involves a highly unbalanced binary classification task; only about 5% of the voxels are occupied. To address this imbalance, we employ combination of binary cross-entropy loss and the scene-class affinity loss, as proposed in [5], to supervise the training process. For the generation of ground truth data, see A.
For the second stage, we supervise the rendered T' images, alpha masks and depth maps with corresponding ground truth:
C= Σ (Ling (Fe, It) + Lamask (Me, M,) (DD, De,
(10)
Cimal (in 12) = =11,, - 4ell2 + 2CIPIPs(4e, 4t),
(11)
mmaas (Me, M,) ||M, M, 1a,
(12)
Ldepth (Dr, 1t)
[exp(- o log(1 + +I,, D.D|I
(13)
|
where LLPIPS is the perceptual image patch similarity loss 84 [D.] is the total number of pixels
in DD), AT, is the gradient of the current RGB image and @ is the element-wise multiplication
operation. As demonstrated in [57]. applying a logarithmic penalty and weighting the per-pixel depth errors with the image gradients result a smoother geometric representation.
Experiments
4.1 Datasets
Used for training. Derived from the original Objaverse [2 dataset, GObjaverse includes high-quality renderings of albedo, RGB, depth, and normal images. These images are generated through hybrid technique combining rasterization and path tracing. The dataset comprises approximately 280,000 normalized 3D models scaled to fit within a cubic space of 1-0.5, 0.5]$. GObjaverse employs a diverse camera setup involving:
G-buffer Objaverse (GObjaverse) [1]
Two orbital paths yielding 36 views per model. This includes 24 views at elevations between 5a and 30" (incremented by 15e rotations) and 12 views at near-horizontal elevations from -59 to 5 (with 30 rotation steps).
Additional top and bottom views for comprehensive spatial coverage.
Google Scanned Objects (GSO) 13: Used for evaluation, this dataset is rendered similarly to GObjaverse. random subset of 100 objects is selected streamline the evaluation process.
OmniOhject3D [70]: Also used for evaluation, this dataset is consistently rendered like GObjaverse. random subset of 100 objects is chosen for efficient evaluation.
4.2 Implementation details
Our model features 330 million parameters distributed across two distinct image encoders and two transformers. The first encoder processes geometry with the 6-layer proposal transformer, while the

---

Table 1: Quantitative results on Google Scanned Objects (GSO) 13] where we used six views for inputs and four for evaluation. Inference time and memory usage account only for the reconstruction process. Bold and underline denote the highest and second- highest scores, respectively.
Method
PSNR f
SSIM
LPIPS 4
CD 4
FS t
Inf. Time (s)
Memory (GB)
LGM
20.76
0.832
0.227
0.295
0.703
0.07
7.23
CRM
22.78
0.843
0.190
0.213
0.831
0.30
5,93
InstantMesh
23.19
0.856
0.166
0.186
0.854
0.78
23.12
Ours
23.57
0.872
0.167
0.167
0.892
0.67
4.92
Table 2: Quantitative results on OmniObject3D [70]. highest scores, respectively.
Bold and underline denote the highest and
Method
PSNR t
SSIM
LPIPS /
CD 4
FS
LGM
21.94
0.824
0.203
0.256
0.787
CRM
23.12
0.855
0.175
0.204
0.810
InstantMesh
23.86
0.860
0.139
0.178
0.834
Ours
24.74
0.883
0.134
0.156
0.863
second focuses more on textures crucial with the 16-layer reconstruction transformer, During training. we maintain maximum number of transformer input tokens 4k and randomly select views from possible 38 for supervision. From these 8 views, we randomly select to 7 views inputs to predict the remaining views. This flexibility in view selection not only tests the robustness of our method but also mimics real-world scenarios where complete data may not always be available. Both input and rendering resolutions are maintained at 448x448 pixels. At the testing and inference stages, we use resolution of 512x512 to align with existing methods. Besides, the number of input tokens is extended to 16k during testing, showcasing its scalability without the need for fine-tuning, Detailed information on model's architecture and training procedures can be found in Section A.3
4.3 Quantitative Results
We evaluated the quality of reconstructed assets from sparse view inputs by analyzing both 2D visual and 3D geometric aspects on the GSO and OmniObject3D dataset [13. Visual quality was assessed by comparing rendered views to ground truth images using metrics such as PSNR, SSIM, and LPIPS. Geometric accuracy was evaluated by aligning our models to the ground truth coordinate systems and measuring discrepancies using Chamfer Distance and F-Score at threshold of 0.2, with point samples totalling 16,000 from the ground truth surfaces. Our method was quantitatively compared against established baselines, including LGM [54), CRM (64), and InstantMesh (72). We avoided comparisons with proprietary methods due to the unavailability of their test splits. Similarly, we excluded comparisons with OpenL 17 and TripoSR (56) these methods are tailored for single image inputs, which would be unfair to compare with.
Our approach achieved state-of oftte-of-the-an the-art performance in four out of the five metrics studied. Although InstantMesh showed slightly higher LPIPS on the GSO dataset, attributed to its mesh-based smoothing capabilities, our method demonstrated superior geometric accuracy, benefiting from explicit modelling of the 3D-to-2D relationship.
In another experiment, outlined in Table we observed a notable trend: our model's performance consistently improves with more input views while maintaining low computational costs. This indicates robust scalability, a critical feature for practical applications. In contrast, the performance of InstantMesh [72), does not follow this pattern. Specifically, InstantMesh shows decline in perfor- mance when the input views increase to 12. This degradation could be due to two primary factors. First, the low-resolution tri- tri-planes may reach their maximum capacity to represent details. Second, the model tends to oversmooth details when handling a large volume of image tokens. Our approach strategically addresses these issues. We employ an extendable sequence of 3D tokens that can be dynamically adjusted to fit the resolution requirements. Additionally, our model features deformable attention mechanisms that intelligently focus on the most pertinent information, preventing the loss of critical details.
7

---

Table 3: Quantitative results on Google Scanned Objects (GSO) with different numbers of input views. We keep the same four views for testing while changing the number of input views. Bold denotes the highest score.
PSNR
SSIM InstantMesh
Inf. Time (8)
Memory (GB)
Num Input
InstantMesh
Ours
Ours
InstantMesh
Ours
InstantMesh
Ours
4
22.87
22.84
0.832
0.851
0.68
0.51
22.09
4.30
8
23.22
23.82
0,861
0.883
0.87
0.84
24.35
5,50
12
23.05
24.43
0.843
0.892
1.07
1.16
24.62
6.96
23.15
24.79
0.861
0.903
1.30
26.69
8.23
20
23.25
25.13
0.895
0.905
1.62
1.84
28.73
9.43
Input Image
TripoSR
LGM
CRM
InstantMesh
Ours
Figure 3: Qualitative comparisons of different image-3D methods. Better viewed when zoomed in.
4.4 Qualitative Results
We conducted qualitative analysis comparing our method with several LRM-based baselines. including TripoSR [17]. LGM [54]. CRM [64], and InstantMesh [72), maintaining their original settings ensure optimal performance. In our approach, we utilized the SV3D [60] technology to generate 21 multi-view images, significantly enhancing the resolution and textural details of the 3D Gaussians produced, as illustrated in Figure Furthermore, as shown Figure 4 employing InstantMesh to reconstruct these images did not yield satisfactory outcomes, corroborating our quantitative findings. This demonstrates the superior capability of our method in handling more complex 3D reconstructions.

---

Input image
InstantMesh (with Zerol23-+)
InstantMesh (SV3D)
Ours (with SV3D)
Figure 4: Qualitative comparison concerning scalability in imput views.
4.5
Ablation Study
In this part, We provide ablation studies for the key designs of method shown in Table 4 Due to the limited computational sources, the ablation is done using a smaller reconstruction model (12 layers) and lower resolution (224x224).
Table Ablation study of some key designs. Models are tested on the GSO dataset 13]. Upper: input views and 4 testing views. Lower: different input views. Bold and underline denote the highest and second-highest scores, respectively.
Method
PSNR
SSIM
LPIPS 1
W/o Plücker rays
20.64
0.826
0.244
W/o low-level features
20.29
0.817
0.246
W/o high-level features
15.85
0.798
0.289
W/o 3D RoPE
20.52
0.827
0.224
Fixed # input views
20.97
0,839
0.220
Full model
20.73
0.831
0.216
4 Inputs
8 Inputs
12 Inputs
Method
PSNR
SSIM f
PSNR
SSIM t
PSNR
SSIM
Fixed input views
19.72 19.94
0.822
20.85
0.833
21.43 22.04
0.838
Full model
0.835
21.16
0.840
0.853
Input image
W/o high-level features
W/o low-level feature
Full model
Figure 5: Effects of excluding high-level and low-level features in the image encoder.
Hierarchical Image Encoder Our ablation study underscores the critical role of hierarchical image features in reconstruction tasks, which necessitate both high-level semantic information (e.g.. object identity and arrangement) and low-level texture information (e.g., surface patterns and colors). As illustrated in Figure[3] the absence of high-level features leads to model instability, while omitting low-level features results in loss textural detail. This dual requirement emphasizes the model's reliance comprehensive feature set for accurate image reconstruction. We also performed an ablation study regarding the Plücker ray embeddings in the low-level encoder. These coordinates assist the model in learning camera directions, contributing improved performance.

---

In transformer-basee architectures, the role of positional embeddings is critical for accu- rately interpreting sequence data positions. A key question arises: With the reconstruction transformer employing deformable cross-attention to elevate 21) features to 3D, is positional embedding still necessary? Our ablation studies confirm its necessity. Notably, 3D RoPE significantly enhances the model's s ability to handle longer sequences. For instance, increasing the sequence length from 4k to 16k elements, models equipped with 3D RoPE exhibited a PSNR improvement of 0.4, compared to a 0.2 improvement in models lacking 3D RoPE. This observation aligns with the ID RoPE [52]
3D RoPE
The ablation study demonstrates decrease in performance when employing our dynamic imput view strategy compared to the fixed 6-input view setting when the training and testing phases were consistent. Despite this, the dynamic input strategy enhances the model's ability to generalize across different input configurations. This adaptability is critical for handling more complex scenarios, aligning with our primary objectives.
Dynamic Input
Deformable attention As shown in Table 5| the ablation results indicate that increasing the number of sampling points in the deformable attention generally improves performance. Given the trade-off' between computational cost and performance gain, we find that using 8 sampling points strikes the best balance.
Table 5: Ablation study of deformable attention. '0 sampling points' means directly using the projected points without any deformation. Bold and underline denote the highest and sacond-highes scores, respectively.
Method
PSNR
SSIM
LPIPS 4
O sampling points
19.52
0.802
0.265
4 sampling points
20.21
0.819
0.238
8 sampling points
20.73
0.839
0.220
16 sampling points
20.80
0.846
0.219
Conclusion
In this paper, we present GeoLRM, a geometry-aware large reconstruction model designed to improve the efficiency and quality of 3D generation. Our approach distinguishes itself from previous methods by effectively utilizing the inherent sparsity of 3D structures and explicitly integrating geometric relationships between 3D and 2D images. The GeoLRM framework employs a 3D- transformer architecture that predicts 3D Gaussians through sophisticated coarse-to-fing methodology. Initially, proposal network estimates coarse occupancy grids, which serve as foundational 3D anchor points for subsequent refinement. The second stage leverages deformable cross-attention to enhance the structure, integrating detailed textural information. Extensive experiments validate that Geol.RM can process higher resolutions and accommodate denser image inputs, outperforming existing models in terms of detail and accuracy. This innovation demonstrates significant potential for real- world applications, particularly in domains where dense view inputs can enhance output quality and user experience. GeoLRM's ability to handle up to 21 images efficiently underscores its scalability and adaptability, paving the way for integration with advanced video generation technologies.
Limitation
While GeoLRM achieves impressive reconstruction quality, does so through two-stage process, which is not inherently end-to-end. This segmentation can lead to the accumulation of errors. The reliance on a proposal network currently indispensable due to the computational intensity of processing Gaussian points across the entire 3D space. This necessity introduces potential inefficiencies and constraints that could hinder real-time applications. Future research will focus on developing end-to-end solution that integrates these stages seamlessly, reducing error propagation and optimizing processing time. By addressing these limitations, we aim to enhance the model's robustness and applicability across broader range of 3D generation tasks.
10

---

Acknowledgement
This work supported in part by the Beijing Natural Science Foundation under Grant No. L247009 and in part by Young Elite Scientists Sponsorship Program by CAST (No. 2024QNRC003).
References
[1] Mohamed EI Banani, Amit Raj, Kevis- Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. arXiv preprint arXiv:2404.08636, 2024.
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, pages 5855-5864, 2021.
and Pratul P
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, pages 5470-5479, 2022.
[4] Jonathan Barron, Ben Mildenhall. Dor Verbin, Pratul Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In ICCV, pages 19697-19705, 2023.
[5] Anh Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In CVPR, pages 3091-9999000 100114400120000 2022.
[6] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Samch Khamis, al. Efficient geometry- 3d generative adversarial networks. In CVPR, pages 16123-16133, 2022.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. CVPR, pages 5799-5809, 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, pages 333-350. Springer, 2022.
[9] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, pages 22246-22256, 2023.
[10] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdff Conditional generative modeling of signed distance functions. In ICCV, pages 2262-2272, 2023.
[11] Matt Deitke, Ruoshi Liu, Matthew Wallingford. Huong Ngo. Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, al. Objaverse-xl: A universe of 10m+ 3d objects. NeurlPS, 36, 2024.
[12] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe annotated 3d objects. In CVPR, pages 13142 13153, 2023.
[13] Laura Downs. Anthony Francis. Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned houschold items. In ICRA, pages 2553-2560, IEEE, 2022.
[14] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. weigodd-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
[15] Jun Gao, Tianchang Shen, Zian Wang. Wenzheng Chen, Kangxue Yin. Daiging Li, Or Litany, Zan Gojeic, and Sanja Fidler. Gct3d: A generative model of high quality 3d textured shapes learned from images. NewrlPS, 55:31841-318555 2022.
[16] Michael Goesele, Brian Curless, and Steven M Seitz. Multi-view stereo revisited. In CVPR, volume 2, pages 2402-2409. IEFF, 2006.
[17] Zexin He and Tengfei Wang. Openlmm: Open-source large reconstruction models. com/ http///Doopaapppelaaa 2023.
httts://githar
[18] Yicong Hong. Kai Zhang, Jjuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2023.
11

---

[19] Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu, Selfoce: Self-supervised vision-based 3d occupancy prediction. arXiv preprint arXiv:2311.12544 2023.
[20] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.0246. 2023.
[21] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. TOG, 42(4):1-14, 2023.
[22] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d 3d with sparse-view generation and large reconstruction model. ICLR, 2023.
[23] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. In CVPR, 2024.
[24] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird s-eye-vive representation from multi-camera images via spatiotemporal trans- formers. In ECCV, pages 1-18. Springer, 2022.
[25] Yixun Liang Xin Yang, Jiantao Lin, Haodong Li. Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. In CVPR, 2024.
[26] Chen-Hsuun Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d td-3d content creation. In CVPR, pages 300 309, 2023.
[27] Fangfu Liu, Diankun Wu, Yi Wei, Yongming Rao, and Yueqi Duan. Sherpa3d: Boosting high-fidelity text-to-3d generation via coarse 3d prior. CVPR, 2024.
[28] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In JCCV, pages 9298-9309, 2023.
[29] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistee images from single-view view image. In ICLR, 2024.
[30] Zhen Liu, Yao Feng. Michael J Black. Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiftu- sion: Score-based generative mesh modeling. In ICLR. 2023.
[31] Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint 23.iv:2330.50000 2023.
[32] Luke Melas-Kyriazii, Christian Rupprecht, and Andrea Vedaldi. Pe2: Projection-commdiionedd point cloud diffusion for single-image 3d reconstruction. In CVPR, pages 12923-12932, 2023.
[33] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-1) 106, 2021.
[34] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. TOG, 4144):100:11122:55, July 2022.
[35] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsuper- vised learning of 3d representations from natural images. In ICCV, pages 7588-7597, 2019.
[36] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:22 2022.
[37] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In CVPR, pages 11453-1 11464, 2021.
[38] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-War Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
12

---

[39] Mingjie Pan. Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Li Liu, and Shanghang Zhang. Renderoce: Vision-centric 3d occupancy prediction with 2d rendering supervision. arXiv preprint arXiv:2309.0955 000002 2023.
[40] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2022.
[41] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness text-to-3d. arXiv preprint arXiv:3311.6611, 2023.
[42] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-to-image generation. In ICML, pages 8821-8831. Pmlr, 2021.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion modcls. In CVPR, pages 10684-10695, 2022.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Neur/PS, (136479-36494 2022.
[45] Johannes L Schonberger and Jan-Michael Frahm. Structurefroommmotiin revisited. In CVPR, pages 4104-4113, 13, 2016.
[46] Steven Scitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR, volume 1, pages 519-528, IEEE, 2006.
[47] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++t single image to consistent multi-view diffusion base model. arXiv preprint 2311:2310..51100 2023.
[48] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. In ICT.R, 2023.
[49] Jachyeok Shim, Changwoo Kang, and Kyungdon Joo. Diffusion--aar based signed distance fields for 3d shape generation. In CVPR, pages 00887-20897 2023.
[50] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. NewrIPS, 34:19313-19325, 2021.
[51] Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training 3d gans. NeurlPS, 35:24487-24501, 2022.
[52] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 168:1,7063, 2024.
[53] Cheng Sun, Min Sun, and Hwann-Tzoo Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. 2022 iece. CVPR, pages 5449- 5459, 2021.
[54] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054. 2024.
[55] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024.
[56] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte. Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.021 2024.
[57] Matias Turkulainen, Xuqian Ren, laroslav Melekhov, Outo Seiskari, Esa Rahtu, and Juho Kannala. Da- splatter: Depth and normal priors for gaussian splatting and meshing. arXiv preprint xxXiv:2403.1782. 2024.
[58] Shimon Ullman. The interpretation of structure from motion. Proceedings of the Royal Society of London. Series B. Biological Sciences, 203(1153) 1979.
[59] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 30, 2017.
13

---

[60] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts. David Pankratz. Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. arXiv preprint urXiv:2403.12005 2024.
[61] Haochen Wang, Xiaodan Du Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, pages 12619-12629, 2023.
[62] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. In ICLR, 2023.
[63] Zhengyi Wang, Cheng Lu. Yikai Wang, Fan Bao, Chongxuan Li. Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. NewrIPS, 36, 2024.
[64] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint 20X1::2403.00054 2024.
[65] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlmm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024.
[66] Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi-camera 3d occupancy prediction for autonomous driving. In ICCV, pages 21729-21740, 2023.
[67] Haohan Weng. Tianyu Yang. Jianan Wang. Yu Li. Tong Zhang. CL Chen, and Lei Zhang. Consistent Improve consistency for one image 3d object synthesis. In ICLR, 2024.
[68] Matthew J Westoby, James Brasington, Niel F Glasser, Michael Hambrey, and Jennifer M Reynolds. frrnctune-rrommmmotiion photogrammetry: A low-cost, effective tool for geoscience applications. Geomor- phology, 179:300-314, 2012.
[69] Sangmin Woo, Byeongjun Park, Hyojun Go, Jin- Young Kim, and Changick Kim. Harmonyview: Harmo- nizing consistency and diversity in one-image-to-3d In CVPR, 2024.
[70] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, et al. Omniohject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In CVPR, pages 803-814, 2023.
[71] Dejia Xu, Ye Yuan, Morteza Mardani, Sifei Liu, Jiaming Song. Zhangyang Wang, and Arash Vahdat. Agg: Amortized generative 3d gaussians for single image to 3d. arXiv preprint arXiv:2401 20l.:200..0409,, 2024.
[72] Jiale Xu, Weihao Cheng. Yiming Gao, Xintao Wang. Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from single image with sparsc-view large reconstruction models. arXiv preprint arXiv:2404.07119, 2024.
[73] Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou. 3d-aware image synthesis via learning structural and textural representations. In CVPR, pages 18430-18439, 2022.
[74] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang. Sida Peng. Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint 2403: 14621, 2024.
[75] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In ICLR, 2023.
[76] Taoran Yi, Jiemin Fang, Guanjun Wu. Lingxi Xie, Xiaopeng Zhang, Wenyu Liu. Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. In CVPR, 2024.
[77] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angioo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arxh::2112.05133, 2(3):6, 2021.
[78] Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, and Xiaojuan Qi. Texi-to-3d with classifier score distillation. In ICLR, 2024.
[79] Biao Zhang and Rico Sennrich. Root mean square layer normalization. NeurIPS, 32, 2019.
14

---

[80] Biao Zhang. Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. TOG, 42(4):1-16, 2023.
[81] Chubin Zhang. Juncheng Yan, Yi Wei, Jiaxin Li, Li Liu, Yansong Tang, Yueqi Duan, and Jiwen La. Occnerf: Self supervisd multi-camera occupancy prediction with neural radiance fields. arXiv preprint ..KV:2332.00224..22 2023.
[82] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting arXiv preprint arXiv:2404. 19702, 2024.
[83] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 2020.
[84] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, pages 586 595, 2018.
[85] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In ICCV, pages 5826-5835, 2021.
[86] Xizhou 7hu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
15

---

A
Appendix
A.1 Occupancy Ground Truth
have investigated the task of vision centric occupancy prediction. However, these approaches often exhibit significant performance discrepancies when compared to 3D methods. To bridge this gap, we leverage depth maps from the GObjaverse dataset to generate accurate 3D occupancy ground truths. This process begins by transforming each pixel in the depth map, represented p' [u, v, 1]², into point in world coordinates. This transformation uses both the intrinsic matrix K and the extrinsic parameters T,, consisting a rotation matrix and translation vector t, as shown in the equation:
Previous studies 33
81 19
p R(d K p')
(14)
where denotes the depth at pixel p Subsequently, these world coordinates are voxelized to pinpoint occupied voxel centres:
{| P {
V
(15)
where P includes all points in direedimmnsional space, V represents the voxel centers, and the voxel size is set at 1/128. The voxelization helps in reducing redundancy by removing duplicate entries.
Input image
Rendering
Mesh
Figure A: Image-to-3D generation with mesh extraction results.
16

---

A.2 Mesh Extraction from 3D Gaussians
We adopt the mesh extraction pipeline from [54] to derive high-quality mesh representations from 3D Gaussians. Figure illlsstraes the mesh generation results of our method, while Figure B compares our generated mesh with other techniques. The results demonstrate the effectiveness of our approach, despite some loss of detail during conversion.
CRM
InstantMest
Ous
Figure B: Comparison of the generated meshes.
A.3 More Implementation Details
We illustrate the details of network architecture and training procedure in Table A We train both the proposal transformer and the reconstruction transformer for 12 epochs on GObjaverse [41]. which takes 0.5 and days respectively on 32 110 40G. the proposal transformer, we use batch size of 2 per GPU and apply preediprecisionn training with BF16 data type. For the reconstruction transformer, we use batch size of per GPU and keep the full precision. We note that the second stage is particularly sensitive to the data type and would fail if using mixed-precisiog
17

---

Table A: Implementation details.
Image encoder
DINOv2 (ViT-B/14) + Conv
layers
Proposal Transformer
# attention head
16
deformed points
Image feature dimension
384
3D feature dimension
384
Max sequence length
4096
Image encoder
DINOv2 (ViT-B/14) Conv
# layers
Reconstruction Transformer
# attention head
16
deformed points
Image feature dimension
384
3D feature dimension
768
Max sequence length
4096
# Gaussians per token
32
Epoch
12
Learning rate
le-4
Learning rate scheduler
Cosine
Optimizer
AdamW
Training details
(Betal, Beta2)
(0.9, 0.95)
Weight decay
0.05
Warm-up
1500
Gradient accumulation
Gradient clip
# GPU
32
A.4 Social Impact
3D AIGC is transforming sectors by automating realistic 3D model creation. In entertainment, streamlines film and game production, reducing costs and enhancing experiences. Education benefits from immersive VR simulations for deeper learning. Architecture sees rapid design visualization and urban planning improvements. Challenges include job displacement and ethical concerns over content authenticity. Addressing these requires legal and policy measures, such as clear copyright laws and standards protect intellectual property. Developing advanced content moderation tools can detect false content, and enhancing Al security can prevent misuse. By focusing on these solutions, we can mitigate negative impacts and maximize the positive contributions of 3D AIGC society.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction do reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information about our methods and the results are repro- ducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We have released the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have included necessary details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The computationally intensive nature of the training procedure made imprac- tical to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reported sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification:
Our research adheres fully to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both the positive and negative social impacts of our method in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The dataset we use has no risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited used assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "UaDfkraBhb", "reviewer": "Reviewer_DmmG", "summary": "The paper proposes a geometry aware large reconstruction model, that represents the scene as 3D Gaussians. The paper proposes a novel architecture for multi view reconstruction that first generates a proposal occupancy grid for 3D gaussians with a proposal transfomer, and then refines them with a reconstruction transformer. The model uses hierarchical image encoders that encode both the semantic features (dino2), and rgb values and plucker coords. The paper uses a first proposal transformer that classifies which dense 3d tokens will have occupancy, and based on that samples sparse tokens, which are denoted as \"anchor points\", that are processed by the reconstruction transformer, to gaussian tokens that are decode a lightweight MLP. The network uses deformable cross attention to the hierarchical image features, which allows for recovering higher resolution features, and also uss a 3d version of Rotary Positional Encoding RoPE. The model is trained in two stages, first the proposal transformer, and then reconstruction model. The model is trained on Objaverse and evaluated on GSO against LGM, CRM, and Instant Mesh. The model is evaluated across different number of input views, and also some of the contributinos are ablated: types of features, 3D RoPE, and training with fixed number of views.", "questions": "- What is the feature dimension of the low-level image features (eg after the conv)?\n- The Anchor Point Decoder L125 is missing in Figure 2, and is labeled as Reconstruction Transformer.\n- How much do the plucker coords in the low-level image features matter?\n- What is the typical number of sparse tokens? Is it even an issue when there are too many?\n- L159: I did not got these subtle point. The proposal anchor points are upsampled to 128^3, right? And then each of the active becomes a token for the reconstruction transformer, with max seq length of 32k?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper proposes a novel architecture to perform 3D reconstruction from multiple views based on transformers and 3D Gaussians that improves upon previous work. The main contributions are the use of higher frequency information thanks to Deformable Cross Attention, and selective computing thanks to a proposal network that computes anchor points around which 3D gaussians are generated.\n- The paper compares results with a number of highly relevant recent papers (LGM, CRM, InstantMesh), and showcases the strength of their method.\n- The method seems to be more robust to increasing the number of input views compared to previous works, which seems like a win.\n- The paper is well written and easy to follow.", "weaknesses": "- I think the deformable cross attention is not ablated properly -- yet it seems a key to the success of this method. Similarly, understanding the sampled points would be quite interesting.\n- The paper could be a bit more clearly written. See questions. I think it it confuses the reconstruction transformer, and the anchor point decoder. I think it is also not clear the architecture of the proposal transformer, until one reads the supplementary and realizes that it's the same architecture as the reconstruction transformer (cool!). I think adding a more clear structure of when the proposal network and the reconstruction transformer are explained would make the paper more readable.\n- The paper could be a bit better if it showed that deformable attention allows the model to be robust to slight pose noise. As well, scaling experiments of the different models would be quite useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positive feedback and valuable insights. Here are our detailed responses to the comments and questions:\n\n**Response to Weaknesses:**\n\n* **About deformable cross attention:** We agree that deformable cross attention plays a crucial role in our method. To further evaluate its impact, we have conducted an additional ablation study, summarized in the table below. This experiment was performed using a smaller model configuration as described in 242 of the manuscript.\n  $$\n  \\begin{array}{|l|c|c|c|}\n  \\hline\n       \\text{Method} & \\text{PSNR} \\uparrow & \\text{SSIM} \\uparrow & \\text{LPIPS} \\downarrow \\\\\\\\\n  \\hline\n      \\text{0 sampling points*} & 19.52 & 0.802 & 0.265 \\\\\\\\\n      \\text{4 sampling points}  & 20.21 & 0.819 & 0.238 \\\\\\\\\n      \\text{8 sampling points}  & \\underline{20.73} & \\underline{0.839} & \\underline{0.220} \\\\\\\\\n      \\text{16 sampling points} & \\textbf{20.80} & \\textbf{0.846} & \\textbf{0.219} \\\\\\\\\n  \\hline\n  \\end{array}\n  $$\n  \\* 0 sampling points means directly using the projected points without any deformation.\n\n  The ablation results indicate that ***increasing the number of sampling points generally improves performance***. Given the trade-off between computational cost and performance gain, we find that using 8 sampling points strikes the best balance.\n\n* **Clarity of writing:** We apologize for any confusion caused by the presentation. To improve clarity, we will standardize the terminology for the second stage of our model as the 'Anchor Point Decoder'. Additionally, we will clarify in the figure caption that the Proposal Transformer shares the same architecture as the Anchor Point Decoder, as previously mentioned in L109.\n\n*  **More explorations:**\n    * **Robust to slight pose noise:** Thank you for your insightful advice! Given the absence of a baseline for this task, we have provided a qualitative visualization demonstrating how deformable attention responds to pose noise. Specifically, we perturbed one of the input camera poses by 0.02 along the z-axis and visualized the predicted offsets relative to the reference points. The results are detailed in the attached PDF. ***We observed that the average angle between the predicted offsets and the perturbation, when projected onto the image plane, was 31°. This indicates that the learned offsets attempt to counteract the perturbation, showcasing robustness to slight pose errors.***\n    * **Scaling experiments:** Owing to constraints on both time and computational resources, currently we are unable to scale up the model. We plan to address this as part of our future work.\n\n**Response to Questions:**\n\n* The feature dimension of the low-level image features after the convolutional layer is 384. This matches the dimensionality of the high-level features, facilitating the application of multi-level deformable attention.\n\n* As addressed in the response to weakness 2, we will correct the label in Figure 2 to 'Anchor Point Decoder'.\n\n* We performed an ablation study regarding the Plücker coordinates. ***The PSNR result without Plücker coordinates was 20.64, compared to 20.73 with the full model.*** These coordinates assist the model in learning ***camera directions***, contributing to improved performance.\n\n* During ***training***, the typical number of sparse tokens is ***4k***, while during ***inference***, this number is ***16k***. A higher number of tokens during training significantly increases memory consumption. Thanks to the 3D RoPE, our model can efficiently handle more tokens during inference to capture finer details. However, we observed that excessive tokens with a simple model might introduce artifacts or 'floaters'.\n\n* Your understanding is correct. The proposal anchor points are upsampled to a 128³ grid. Active points are then converted into tokens for the Reconstruction Transformer, with a maximum sequence length of 4k during training.\n\nWe hope these clarifications address the reviewer's concerns and provide a clearer understanding of our work. Thank you again for the valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GhMdHFMDNj", "reviewer": "Reviewer_hKRc", "summary": "This paper introduced a sparse reconstruction model based on the LRM. This paper try to use the projection between the 3D point with the  pixel position to save computational consumption and this paper use the deformable attention to lift the 2D feature to 3D, and finally propose a two-stage pipeline to generate Gaussian.", "questions": "Refer to weaknesses for details. And due to these doubts, I tend to give the borderline and hope to see the sufficient response of the author.", "limitations": "Declared in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Using the projection correspondence between 3D point with the pixel coordinate is reasonable to save consumption.\n2. Deformable attention is also a useful solution to lift 2D multi-view image features to 3D.", "weaknesses": "1. Some confusions about the method and the experiment.\n- Why cannot train the first stage and the second stage together? Is it because of the memory consumption?\n- The advantage with more input images is good but seems to be unimportant in the sparse reconstruction model, which is designed to handle the situation with only few inputs. And there also a confusion in this experiment: will the number of Gaussian primitives increase as the input increases in your method? \n- The 3DGS-based method has advantages in novel view rendering quality, but the advantage over InstantMesh is not obvious and the results shown in this paper seem to be much worse than the existing 3DGS-based methods [1,2]?\n- How does your method generate the mesh? And there seems lack this comparison experiment. Is there still advantage in the mesh generation with Gaussian representation?\n2. The contribution seems to be not novel enough.\n- The strategy of using projection strategy is a common solution, and has been widely used in existing sparse reconstruction methods like [3,4,5].\n- Using the deformable attention to lift 2D image features to 3D space is also a widely used strategy like [6,7], and the two-stage pipeline that first generating the sparse queries and then only processing in these sparse queries is also the existing solution such as [7].\n\n[1] GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation\n\n[2] GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting\n\n[3] SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views.\n\n[4] C2F2NeUS: Cascade Cost Frustum Fusion for High Fidelity and Generalizable Neural Surface Reconstruction\n\n[5] GenS: Generalizable Neural Surface Reconstruction from Multi-View Images\n\n[6] SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving\n\n[7] VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their constructive feedback. Below are our detailed responses to the specific points raised:\n\n**Response to Weaknesses:**\n\n**About the method and the experiment:**\n* **Training stages:** The primary reason for training the stages separately is due to the ***non-differentiability*** of the conversion from the occupancy grid to sparse tokens. This conversion step is necessary for processing the sparse 3D representation efficiently, but it cannot be backpropagated through directly. Moreover, the memory consumption is indeed a secondary concern that benefits from this separation.\n* **Denser inputs and Gaussian primitives:** The motivation to handle denser image input was innovated by ***the success of video diffusion models*** (Sora [1], SVD [2], etc.). Most videos naturally contain information about 3D and the diversity and quality of video datasets are much better than that of 3D. Therefore, recent works (SV3D [3], VideoMV [4], etc.) leverage video diffusion models for multi-view generation and achieved great success. The output of video diffusion models is denser than that of multi-view diffusion models and could not be efficiently processed by previous LRM methods. We are optimistic about the ability of the video diffusion model to generate multi-view consistent images and its ***great potential to be extended to scene-level 3D generation***. Therefore we propose our method to efficiently process denser inputs. The number of Gaussian primitives is set to 512k for all inputs to avoid bottlenecks in representation.\n* **Comparison with baselines:** \n  * While the improvement may appear small in terms of quantitative metrics, our method achieves significant gains in ***resolution and efficiency***. As demonstrated in Figure 3 of the manuscript, examples 3 and 4 show that our method recovers finer details from the input images. Additionally, please refer to the following table (which is an extension of Table 2 in the original manuscript) for a detailed comparison with InstantMesh, highlighting our memory efficiency and ability to process denser inputs effectively.\n  $$\n\\begin{array}{|c|cc|cc|cc|cc|}\n\\hline\n&\\text{PSNR}&\\text{PSNR}&\\text{SSIM}&\\text{SSIM}&\\text{Inf. Time (s)}&\\text{Inf. Time (s)}&\\text{Memory (GB)}&\\text{Memory (GB)}\\\\\\\\\n\\text{Num Input}&\\text{InstantMesh}&\\text{Ours}&\\text{InstantMesh}&\\text{Ours}&\\text{InstantMesh}&\\text{Ours}&\\text{InstantMesh}&\\text{Ours}\\\\\\\\\n\\hline\n4&\\textbf{22.87}&22.84&0.832&\\textbf{0.851}&0.68&\\textbf{0.51 s}&22.09&\\textbf{4.30}\\\\\\\\\n8&23.22&\\textbf{23.82}&0.861&\\textbf{0.883}&0.87&\\textbf{0.84 s}&24.35& \\textbf{5.50} \\\\\\\\\n12&23.05&\\textbf{24.43}&0.843&\\textbf{0.892}&\\textbf{1.07}&1.16&24.62& \\textbf{6.96} \\\\\\\\\n16&23.15&\\textbf{24.79}&0.861&\\textbf{0.903}&\\textbf{1.30}&1.51&26.69& \\textbf{8.23} \\\\\\\\\n20&23.25&\\textbf{25.13}&0.895&\\textbf{0.905}&\\textbf{1.62}&1.84&28.73& \\textbf{9.43} \\\\\\\\\n\\hline\n\\end{array}\n  $$\n  * ***We were unable to use the performance statistics reported by GRM/GS-LRM directly*** due to ***their undisclosed and distinct test dataset split***, as well as the fact that their model is ***closed-source***. Consequently, conducting a fair comparison with GRM/GS-LRM is currently not feasible. However, our method offers theoretical advantages in the following two key aspects:\n    * **Projection as Inductive Bias.** Unlike GRM/GS-LRM, which employs self-attention among image tokens to infer 3D structures from large datasets, our approach leverages projection as an inductive bias to directly model geometry. This design choice significantly increases the computational efficiency and facilitates the learning of 3D information, particularly critical given the scarcity of 3D data.\n    * **Flexibility of Representation.** GRM/GS-LRM utilizes pixel-aligned 3DGS, which fix the input and 3D representation resolutions during training and can lead to substantial memory consumption when scaling up. In contrast, our method employs sparse 3D grid tokens, allowing the resolution of both inputs and 3D representations to be dynamically adjusted at inference time. Moreover, pixel-aligned 3DGS are unable to reconstruct occluded regions, whereas our representation holds the potential to handle such areas effectively.\n* **Mesh generation:** The conversion from 3DGS to mesh is an active research area. We integrate the pipeline proposed by LGM for mesh generation, which currently provides the best results. ***The attached PDF includes a detailed comparison of the generated meshes with other methods***, demonstrating the effectiveness of our approach despite some loss of detail during conversion.\n\n**About novelty and contributions:**\n\nWhile the use of projection strategies and deformable attention has been explored in other domains, their application within the context of LRM and 3D AIGC is novel. ***Previous methods often suffer from high memory consumption and limited resolution due to the reliance on dense cross-attention and a lack of geometric understanding.*** So our method tackles these problems. In summary, our contributions are:\n* A set of new solutions to LRM, including ***the sparse 3DGS token representation suitable for extending to high resolution***, the two-stage pipeline to utilize the sparse nature of 3D and deformable attention for feature lifting.\n* Better ***quality and efficiency*** over previous methods.\n* The first attempt to process ***dense inputs*** using LRM, potentially paving the way for the integration of ***video generation models*** into 3D AIGC applications.\n\nWe hope these clarifications address the reviewer's concerns and provide a clearer understanding of our work.\n\n[1] Video generation models as world simulators\n\n[2] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets\n\n[3] SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion\n\n[4] Consistent Multi-View Generation Based on Large Video Generative Model", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oLp6QaDWLf", "reviewer": "Reviewer_7HGK", "summary": "The primary objective of this paper is to develop a geometry-aware large reconstruction model. Previous approaches either predict tri-planes or per-pixel Gaussians for reconstruction from multi-view images. However, these methods lack an explicit correspondence between 2D image features and 3D representations. To address this issue, the authors employ a Proposal Transformer to identify grids with valid occupancy and use these tokens to perform Deformable Cross Attention for aggregating image features and subsequently predicting Gaussians. The results demonstrate that the proposed methods achieve state-of-the-art performance and adapt well to different numbers of views.", "questions": "Questions:\n\nMy main concern is that the comparison with previous methods using different numbers of input views is unfair, and this paper employs a different rendering resolution. The experimental results do not adequately support the authors' claims.\n\n\nSuggestions:\n\n- One of the main advantages of this paper is its potential to reduce GPU memory usage as the number of input views increases. However, there is no clear comparison with other methods. Adding GPU memory and inference time comparisons in Table 2 would be helpful.\n- What if the number of input views is further increased beyond 12?", "limitations": "The authors have discussed their limitations and provided potential solutions.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- This paper presents an alternative approach for building large reconstruction models by first predicting occupancy grids. This method may be beneficial in reducing memory consumption as the number of input views increases.\n\n- The results show that the proposed method achieves state-of-the-art performance, with performance improving as the number of input views increases.", "weaknesses": "- The authors compare their model with InstantMesh under different input view settings, which is unfair. InstantMesh fixes the number of input views at six during training, so testing with varying numbers of input views may degrade its performance. In contrast, GeoLRM is trained on varying input view numbers. Therefore, the main claim of this paper is not well-verified.\n\n- The authors use a training and rendering resolution of 448x448, while other works render 512x512 images. It would be better to maintain the same settings for a fair comparison.\n\n- Introducing the Proposal Transformer complicates the process. An inference time comparison with other methods would be beneficial.\n\n- The comparison is only conducted on the GSO dataset. Evaluating and comparing with other methods on additional datasets, such as OmniObject3D, would be more convincing.\n\n- The authors claim superiority compared to per-pixel Gaussian prediction methods and compare performance with LGM. However, another method, GRM [1], released around the same time as LGM and sharing a similar architecture but showing much better performance, could also be included for a more convincing comparison.\n\n- The paper is not clearly written, especially the method section: details of the Proposal Transformer are missing; In Figure 2, the term \"Reconstruction Transformer\" is used, but in Section 3.2, \"Anchor Point Decoder\" is used, which is inconsistent.\n\n[1] Xu, Yinghao, et al. \"Grm: Large Gaussian reconstruction model for efficient 3d reconstruction and generation.\" arXiv preprint arXiv:2403.14621 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. We have addressed the specific concerns and provided additional clarification and results as follows:\n\n**Response to Weaknesses:**\n* **Number of input views:** We agree that the comparison with InstantMesh using different numbers of input views might seem unfair. The motivation of this experiment is to pave the way for ***the integration of video generation models into 3D AIGC*** applications. Because videos nauturally contain information about 3D and the diversity and quality of video datasets are much better than that of 3D. Therefore, the flexibility to handle varying numbers of input views is a significant advantage of our approach that should be considered. ***Moreover, even under conditions that are less favorable to our method (6 input views in the following table), we still achieve superior performance and efficiency.***\n* **Rendering resolution:** We chose 448 due to the requirements of our image backbone. To ensure a fair comparison, we have ***re-evaluated our method at 512*** resolution, and ***the results remain consistent with those at 448.*** The updated performance metrics are as follows:\n  $$\n\\begin{array}{|l|c|c|c|c|c|c|c|}\n\\hline\n\\text{Method}&\\text{PSNR}\\uparrow&\\text{SSIM}\\uparrow&\\text{LPIPS}\\downarrow&\\text{CD}\\downarrow&\\text{FS}\\uparrow&\\text{Inf. Time (s)}&\\text{Memory (GB)}\\\\\\\\\n\\hline\n\\text{LGM}&20.76&0.832&0.227&0.295&0.703&\\textbf{0.07}&\\text{7.23}\\\\\\\\\n\\text{CRM}&22.78&0.843&0.190&0.213&0.831&\\text{4}^*&\\underline{5.93}\\\\\\\\\n\\text{InstantMesh}&\\underline{23.19}&\\underline{0.856}&\\textbf{0.166}&\\underline{0.186}&\\underline{0.854}&\\text{0.78}&\\text{23.12}\\\\\\\\\n\\text{Ours}&\\textbf{23.57}&\\textbf{0.872}&\\underline{0.167}&\\textbf{0.167}&\\textbf{0.892}&\\underline{0.67}&\\textbf{4.92}\\\\\\\\\n\\hline\n\\end{array}\n  $$\n  \\* The 4 seconds includes the U-Net forward process (about 0.30 s) and the conversion from triplanes to meshes for real-time rendering (about 3.7 s).\n* **The Proposal Transformer:** The Proposal Transformer do introduce additional computational overhead, but ***it efficiently focuses the subsequent stages on occupied regions***. The inference time (excluding rendering) is 0.18 s for the Proposal Transformer and the total inference time is 0.67 s, as shown in the table above.\n* **OmniObject3D:** We have been conducting additional experiments on the OmniObject3D dataset and will include these results in the revised manuscript. Due to the limited time, we present the results of InstantMesh and our method, tested with 50 samples, in the following table.\n  $$\n\\begin{array}{|l|c|c|c|}\n\\hline\n\\text{Method}& \\text{PSNR}\\uparrow&\\text{SSIM}\\uparrow&\\text{LPIPS}\\downarrow\\\\\\\\\n\\hline\n\\text{InstantMesh}&\\underline{23.98}&\\underline{0.861}&\\underline{0.146}\\\\\\\\\n\\text{Ours}&\\textbf{24.65}&\\textbf{0.893}&\\textbf{0.130}\\\\\\\\\n\\hline\n\\end{array}\n  $$\n* **Comparison with GRM:** We were unable to use the performance statistics reported by GRM directly due to ***their undisclosed and distinct test dataset split***, as well as the fact that their model is ***closed-source***. Consequently, conducting a fair comparison with GRM is currently not feasible. However, our method offers theoretical advantages in the following two key aspects:\n  * **Projection as Inductive Bias.** Unlike GRM, which employs self-attention among image tokens to infer 3D structures from large datasets, our approach leverages projection as an inductive bias to directly model geometry. This design choice significantly increase the computational efficiency and facilitates the learning of 3D information, particularly critical given the scarcity of 3D data.\n  * **Flexibility of Representation.** GRM utilizes pixel-aligned 3D grids, which fix the input and 3D representation resolutions during training and can lead to substantial memory consumption when scaling up. In contrast, our method employs sparse 3D grid tokens, allowing the resolution of both inputs and 3D representations to be dynamically adjusted at inference time. Moreover, pixel-aligned 3DGS are unable to reconstruct occluded regions, whereas our representation holds the potential to handle such areas effectively.\n* **Clarity of writing:** We apologize for the confusion regarding the terminology. ***The Proposal Transformer shares the same structure with the Reconstruction Transformer as mentioned in L109***, and the detailed architecture is provided in Table 4 in Appendix A.1. We will unify the naming convention to 'Anchor Point Decoder' throughout the manuscript and add more details to Figure 2 to avoid confusion.\n\n**Response to Questions:** \n* We add ***the GPU memory and inference time comparisons*** in the original Table 2 and the revised version is given ***in the response to weaknesses 2***.\n* We have tested our method with an increasing number of input views beyond 12. ***The following table demonstrates that our method continues to improve with more input views, maintaining low memory usage and fast inference times.***\n  $$\n\\begin{array}{|c|cc|cc|cc|cc|}\n\\hline\n&\\text{PSNR}&\\text{PSNR}&\\text{SSIM}&\\text{SSIM}&\\text{Inf. Time (s)}&\\text{Inf. Time (s)}&\\text{Memory (GB)}&\\text{Memory (GB)}\\\\\\\\\n\\text{Num Input}&\\text{InstantMesh}&\\text{Ours}&\\text{InstantMesh}&\\text{Ours}&\\text{InstantMesh}&\\text{Ours}&\\text{InstantMesh}&\\text{Ours}\\\\\\\\\n\\hline\n4&\\textbf{22.87}&22.84&0.832&\\textbf{0.851}&0.68&\\textbf{0.51 s}&22.09&\\textbf{4.30}\\\\\\\\\n8&23.22&\\textbf{23.82}&0.861&\\textbf{0.883}&0.87&\\textbf{0.84 s}&24.35& \\textbf{5.50} \\\\\\\\\n12&23.05&\\textbf{24.43}&0.843&\\textbf{0.892}&\\textbf{1.07}&1.16&24.62& \\textbf{6.96} \\\\\\\\\n16&23.15&\\textbf{24.79}&0.861&\\textbf{0.903}&\\textbf{1.30}&1.51&26.69& \\textbf{8.23} \\\\\\\\\n20&23.25&\\textbf{25.13}&0.895&\\textbf{0.905}&\\textbf{1.62}&1.84&28.73& \\textbf{9.43} \\\\\\\\\n\\hline\n\\end{array}\n  $$\n\nWe appreciate the reviewer’s time and effort in providing valuable feedback, which will help us improve the quality of our manuscript. We look forward to any further comments or suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their feedback and valuable comments on our work. As suggested by Reviewer hKRc, we have further compared the mesh extraction of our method with other baseline methods. The detailed comparison is provided in the attached one-page PDF file. Additionally, following the suggestion of Reviewer DmmG, we have added a qualitative visualization demonstrating how deformable attention responds to pose noise.

Meta-review: The paper received mixed reviews. On the positive side:

- Novel architecture (reviewer DmmG) 
- Very strong performance (reviewers DmmG, 7HGK)
- More robust to a varying number of views (reviewer DmmG) 
- Less memory consumption (reviewers hKRc, 7HGK)

The AC read through all rebuttal posts and finds that the authors did a very good job at resolving the questions by providing the necessary experiments. The AC also checked the paper and finds that it is well written. 

Regarding reviewer hKRc, the question about occlusions after the rebuttal is not clear to the AC. The second point of criticism about novelty is unprecise - as reviewer DmmG mentions, the deformable attention is not novel by itself but has not been used for these types of models. Thus, designing an architecture and demonstrating the strong performance can be considered an important insight for the community. The usefulness of using deformable attention can be seen in the response to DmmG. Reviewer 7HGK in the discussion confirmed that the rebuttal addressed the concerns, even for the InstantMesh comparison that seemed to be the main point of criticism. 

A comparison to GRM should not be held against this paper, as this is concurrent (unpublished) work. 

Due to the strong performance and interesting contributions for the community, the AC decided to follow reviewer DmmG and recommends to accept the paper.
Decision: Accept (poster)