MVGamba: Unify 3D Content Generation as State Space Sequence Modeling
Authors: Xuanyu Yi
Zike Wu
Qiuhong Shen
Qingshan Xu
Pan Zhou
Joo Hwee Lim
Shuicheng YAN
Xinchao Wang
Hanwang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Generation and Reconstruction
State Space Model
Primary area: generative_models

Abstract: Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors. Current works further leverage 3D Gaussian Splatting as 3D representation for improved visual quality and rendering efficiency. However, we observe that existing Gaussian reconstruction models often suffer from multi-view inconsistency and blurred textures. We attribute this to the compromise of multi-view information propagation in favor of adopting powerful yet computationally intensive architectures (\eg, Transformers). 
To address this issue, we introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). Our Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity.
With off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only $0.1\times$ of the model size. The codes are available at \url{https://github.com/SkyworkAI/MVGamba}.

MVGamba: Unify 3D Content Generation as State Space Sequence Modeling
Xuanyu Y1 Zike Wu"*
Qiuhong Shen
Pan Zhou Xinchao Wang²
Qingshan Xu
Joo-Hwee Lim Shuicheng Yan
Hanwang Zhang'
Nanyang Technological University
National University of Singapore
University of British Columbia
Siigappore Management University
Institute for Infocomm Research Skywork Al
Single le image-to-
Input
Input
Sparse-view Reconstruction
Mesh Extraction
Input
MVGamba
Text- Text-to-30
Input
Input
Input
"Fuffy
"Colarfi butherfly'
"tyber-style kettlc
Figure 1: MVGamba is unified 3D generation framework build on Gaussian Splatting, which can generate high-quality 3D contents in a derdffowarrd manner in sub-seconds.
Abstract
Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors. Current works further leverage 3D Gaussian Splatting 3D representation for improved visual quality and rendering efficiency. How- ever, we observe that existing Gaussian reconstruction models often suffer from multi-view inconsistency and blurred textures. We attribute this the compromise of multi-view information propagation in favor of adopting powerful yet compu- tationally intensive architectures (e.g., Transformers). To address this issue, we introduce MVGamba, general and lightweight Gaussian reconstruction model featuring multi-view Gaussian reconstructor based the RNN-like State Space Model (SSM). Our Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long
'Equal Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

sequence of Gaussians for dine-detail modeling with linear complexity. With off- the-shelf multi-vier diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-an baselines in all 3D content generation scenarios with approximately only 0.1 x of the model size. The codes are available at https:/ / thub. com wtoo///yyyorAA////aaaaaaaa
Introduction
We address the challenge of crafting 3D content from single image, sparse-view images, or text input, which can facilitate a broad range of applications, e.g., Virtual Reality, immersive filming, digital gaming and animation. Previous research on 3D generation has investigated distilling 21) diffusion priors into 3D representations via score distillation sampling (SDS) [1]. Although these optimization-base approaches exhibit strong zero-shot generation capability with high-fidelity rendering quality [2-5], they are extremely time- and memory- memory-intensive, intensivee often requiring hours to produce a single 3D asset, thus not practical for a real-world scenario.
With the advent of large-scale open-world 3D datasets [6-8]. recent 3D large reconstruction models (LRMs) [9-12] integrate multi-view diffusion models [13-15] with scalable multi-view 3D recon- structor to regress a certain 3D representation (e.g. Triplane-NeRF [16, 17]. mesh) in a feed-forward manner. Specifically, current LRMs [18-20] adopt one image (or text) - mulit-view images + 3D diagram to predict 3D Gaussian Splatting (3DGS) [21] parameters, thereby ensuring the rendering efficiency while preserving fine details. Given a single image or text prompt, they first generate set of images using multi-view diffusion models, which are then fed into a multi-view reconstructor (e.g.. U-Net [22] or Transformer [23]), mapping image tokens to 3D Gaussians with superior generation speed and unprecedented quality.
However, we observe that existing feed-forward Gaussian reconstruction models typically adopt powerful yet computationally intensive architectures [23, 24] to generate long sequences of Gaussians for intricate 3D modeling. Such approaches inevitably compromise the integrity of multi-view infor- mation propagation to manage computational costs. For instance, they use local [18] or mixed [19] attention on limited multi-view image tokens or even deal each view separately and simply merge the predicted Gaussians afterwards [20]. Consequently, the generated 3D models often suffer from multi-vier inconsistency and blurred textures, as illustrated Figure 2(a). These issues indicate that current compromise strategies fail to translate into coherent, high-qualin outputs in practice. This raises a crucial question: How can we preserve the integrity of mulli-view information while efficiently generating sufficiently long sequence of Gaussians?
To address this issue, in this paper, we introduce Multi-View Gaussian Mamba (MVGamba), a general and lightweight Gaussian reconstruction model. At its core, MVGamba features a multi-view Gaussian reconstructor based on the recently introduced RNN-like architecture Mamba [25], which expands the given multi-view images into long sequence of 3D Gaussian tokens and processes them recurrently in a causal manner. By adopting causal context propagation, our approach efficiently maintains multi-view information integrity and further enables cross- view self-refinement from earlier to current views. Additionally, our Gaussian reconstructor enables the fine-detailed generation of long Gaussian sequences with linear complexity [26, 27] in single forward process, eliminating the need for any post hoc operations used in previous work.
More concretely, we first patchify the multi-view images into N tokens and rearrange them according cross-scan order [27, 28], resulting in 4 X N image tokens for selective scanning. These tokens are then processed through a series of Mamba blocks for state space sequence modeling. Subsequently, we feed the output Gaussian sequence into a lightweight Multi-Layer Perceptron (MLP) for channel- wise knowledge selection, followed by a set of linear decoders to obtain the Gaussian parameters representing high-quality 3D content (Sec. 3.2). Compared to previous LRMs [11, 29, 30], our MVGamba features many computationally efficient components: single-layer 2D convolution image tokenizer replaces the pre-trained DINO [31] transformer encoder, a lightweight MLP combined with linear decoders replaces the deep MIP decoder, and most importantly, linear complexity Mamba blocks replace quadratic complexity Transformer blocks (Figure 2(b)). Together, these designs ensure efficient training and inference while achieving higher generation quality (Sec. 4). Moreover, to

---

Soparate
Other Gausskan Recordruubl
Tansformr
Concat.
1000
800
Inpus
400
Ours
Causal Sequence
200
1024
2048
4096
8192
16384 32768
(a)
Sequence Length
(b)
Figure 2: (a) Previous Gaussian reconstruction models sacrifice the integrity of multi-view informa- tion for computationally intensive architectures, resulting multi-view inconsistency and blurred textures. (b) Comparison of FLOPs between self-attention in Transformers and SSM in Mamba. Detailed FLOPs data are provided in Table
directly convert the generated Gaussians into smooth textured polygonal meshes, alternatively incorporate 3DGS variant - 2DGS [32] - for accurate geometric modeling and mesh extraction.
We conducted comprehensive qualitative and quantitative experiments to verify the efficacy of our proposed MVGamba. The experimental results demonstrate that MVGamba (49M parameters) outperforms other latest LRMs [19, 29, 33] and even optimization-ban based methods [34, 35] on the task of text-to-3D generation, single-view reconstruction and sparse-view reconstruction with roughly only 0.1 x of the model size. The contributions and novelties of our paper are summarized follows:
We point out that directly generating a sufficiently long sequence Gaussians with full multi-view information is crucial for consistent and fine-detailes 3D generation.
We introduce MVGamba, novel fed-forward pipeline that incorporates causal context propagation for cross-view view self-refinement, allowing the efficient generation of long sequences of 2D/3D Gaussians for high-quality 3D content modeling.
Extensive experiments demonstrate that MVGamba is potentially general solution for 3D content generation, including text-to-3D, image-to-33 and sparse-vies niew reconstruction task.
Related Work
3D Generation. Previous approaches for generating high-fidelity 3D models predominantly used
SDS-based optimization techniques [1, 36] and their variants [2-5, 34, 37]. These methods yield high-quality 3D generations but require hours for the per-instance optimization process converge. Pioneered by the large reconstruction model (LRM) [29], recent works [10, 11, 30, 38) show that image tokens can be directly mapped to 3D representations, typically triplane-NeRF. in feed- forward manner via a scalable transformer-basee
architecture [23] with large-scale 3D training data [6-8]. Among them, Instant3D [1]] integrates LRM with multi-view image diffusion models [13-
15, 39, 40], using four generated images for better quality. To avoid inefficient volume rendering and limited triplane resolution, some concurrent works [18-20] follow Instant3D and introduce 3D
Gaussian Splatting [21] into sparse-view LRM variants. Specifically, GRM [18] and GS-LRM [20] use pixel-aligned Gaussian with pure transformer- based reconstruction model, increasing the number of Gaussians through image feature upsampling and per-pixel merge operations. LGM [19]
combines the 3D Gaussians from different views using convolution-basee asymmetric U-Net [22].
Our Gamba, the other hand, directly processes multi-view conditions causally, recurrently generating long sequence Gaussians for coherent and high-fidelity 3D modeling.
Mamba model for visual applications. Recent ad advancements in State Space Models (SSMs) [17, 41, 42], notably Mamba [25], have gained prominence in long sequence modeling for harmoniz- ing computational efficiency and model versatility [43-46]. Following Mamba's 's progress, there has been surge in applying this framework to critical vision domains, including generic vision backbones [26, 27, 47, 48), multi-modal streams [49, 50], and vertical applications, especially in medical image processing [51-56]. Specifically, VMMmba [27] pioneers purely Mamba-based backbone to handle intensive prediction tasks. Similarly, Vim [26] leverages bidirectional SSMs for data-dependent global visual context without image-specific biases. Subsequent works progress with advanced selective scanning algorithms [47, 48], integration with other networks [57, 58], and

---

"Ikun"
Multiview Conditions
Embedding
Positonnl
Order
Linee
Mamba Block
MVDream
IaageDream
tokens
Geliraled Marbriiew
Patchify Conv
Liner
Decoder
Multi-view Gaussian Reconstructor
Novel
Dilerertiasee randering
MLP
Gaussian
Meah
Color
TSDF
Position
(a) Multi-viow Gaussian Reconstructor
(b) Unified inference pipeline
Figure 3: (a) Multi-view Gaussian reconstructor (Sec. 3.2): Multi-view inputs with ray embedding
used for causal sequence modeling, predicting Gaussians rendered novel views and supervised with ground truth images. (b) Unified inference pipeline (Sec. 3.4); MVGamba combines multi-view diffusion models and Gaussian reconstructor to generate high-quality 3D content in sccodds..ds.
adapted structural designs [59, 59, 60]. Concurrently, Gamba [61] marries Mamba with 3DGS for single-view reconstruction with limited texture quality and generalization capacity. this paper, we explore and demonstrate the efficiency and long-sequence modeling capacity of Mamba in various 3D generation tasks with large-scale pre-training.
Method
In this section, we present our MVGamba, designed efficiently generate 3D content through two-stage pipeline. In the first stage, we utilize off-the-shelf multi-view diffusion models, including MVDream [14] and ImageDream [13], to generate multi-view images based on an input text prompt or single image. In the second stage, equipped with this multi-view image generator, we introduce an SSM-based multi-view reconstructor to generate Gaussians from multi-view images. Specifically, first provide brief overview of 3D Gaussian splatting and its variants (Sec. 3.1). Next, describe the core architecture detailed elaboration of our robust training objectives (Sec. 3.3). With above large-scale pre-training, we are able to chain these two stages to produce high-fidelity 3D content in seconds (Sec. 3.4).
our multi-view Gaussian reconstructor (Sec. 3.2), followed by
3.1 Preliminary: Gaussian Splatting
Introduced by [21], vanilla 3D Gaussian Splatting (3DGS) fits 3D scene from multi-view images with collection of 3D Gaussians. Variants of Gaussian Splatting [62, 63], typically 2DGS, leverage 2D Gaussian primitives instead, excelling in the vanilla version for more accurate geometry recon- struction. Generally, each Gaussian is composed of its 3D center E IR3 3D scale E R or 2D scale s R associated color e 3 opacity a E I, and rotation quaternion q E R These parameters can be collectively denoted by G. with G: = [Ju, 8i, C2 Osi, q: denoting the parameter of the i-th Gaussian. These Gaussians can then be splatted onto the image plane and rendered real time via the differentiable tiled rasterizer [21, 62].
3.2 SSM-based Gaussian reconstructor
The core of MVGamba is a feed-forward multi-view Gaussian reconstructor. As depicted in Fig- 3(a), reconstructor transforms multi-viens input images with camera embedding [64] into 3D> contents represented by 3D Gaussians [21] or its variants [62, 63] in feed-forward manner. This reconstructor comprises an SSM-based processor expand and process multi-view image tokens as Gaussian sequences, and a wiigh Gaussian decoder to predict attributes for each Gaussian.
"2DGS collapses the 3D volume into set of 2D oriented planar Gaussian disks to model surfaces intrinsically.

---

Expanding multi-view images as sequences. Given a posed multi-view image set {u,, T we first densely embed the camera pose π. E 4x for each view U; E RR using Plücker rays [30],
denoted as P; E R x6 The pixel values and ray embeddings are concatenated into a 9-channel fused map, which is then tokenized using a non-overlapping convolution with a kernel size of p x p:
V = Conv(Concat(vi; P.)).
(1)
where V, E RxxxxC is the tokenized feature map: h = H/p, w = W/p; C' is the embedding dimension. Note that considering the light-weight architectural design, our image tokenizer is much simpler than the pre-trained DINO [31] utilized by previous LRMs, which we empirically find to be redundant for low-level 3D reconstruction. With the tokenized multi-view features, we then adopt a cross-scan order [27] to rearrange them as sequence. Specifically, we scan the image tokens sequentially along four different directions: top-left - bot tom-ghhh,, bottom-righh ggtt + top-left, top-right + bottom- left, and bottom- -eft - top-rights which allows each token to integrate information from all adjacent tokens. This cross-scan rearrangement results in a
sequence that is 4x longer:
X P Paccn (Concat( (oncat((V/g) (V.))),
(2)
where X E IR NN denotes the expanded Gaussian sequences, N denotes the view number, and P Psean denotes the cross-scan operation on each view.
Causal sequence modeling with State Space Model. Inspired by [25, 61], we model the Gaussian sequences via an adapted SSM based processor. In detail, given the expanded Gaussian sequence X, we first add a learnable positional embedding E element-wiss to it and derive the initial Gaussian sequence Xg. Then, we feed X, into L stacked SSM layers for recurrent causal sequence modeling, formulated as:
X SSMM (X&-1; As; Ax)
where X denotes Gaussian sequence output by the k-th layer, and SSM; denotes the k-th SSM layer with vanilla Mamba [25] structure; Ag. B& and Ag are parameters of SSM layer dependent on the input sequence X Note that we are modeling Gaussian sequences rather than integrating spatial information as in existing vision Mamba models [26, 27, 47]. Therefore. we adopt ID convolution instead of 2D convolution in the Mamba blocks, similar to other sequential modeling SSMs [25, 58, 65]. Through state space sequence modeling, we successfully propagate the causal context containing the multi-view information from earlier states to later states with linear complexity. This approach efficiently incorporates multi-vier view information causally from the initial condition onward, thereby making full use of all Gaussian tokens through cross- view self-refinement. As discussed in Sec. 5, this causal sequential generation of Gaussian tokens provides the model with unprecedented robustness and self-correction abilities, even under inconsistent or noisy input conditions.
Decoding causal token sequences into Gaussians. Each token in the processed causal sequence X, is treated as a separate 3D Gaussian token. We first apply single hidden layer MLP to X where the width of the hidden layer 4C and the output channels revert to C. This process denoted as Z = MLP 2.. which aims for channel- wise knowledge selection [24, 66, 67]. We then apply sub-heads to derive each attribute of 3DGS with separate linear projections. Specifically, we predict the position [61] by discretizing the coordinates where position μı; is clamped to [-1, 1 The scale 8; is predicted with a learnable linear projection followed by a softplus activation. The opacity as is predicted with a linear projection followed by sigmoid activation. Regarding the color attribute a. we predict the RGB values instead of the spherical harmonics adopted by the original 3DGS[21], as reconstructor is mainly trained on synthetic 3D datasets free of light variation.
However, unlike other Gaussian attributes, the rotation quaternions are quite sensitive and difficult to predict directly, and hence are often set canonical isotropic and fixed in several recent works [61, 68). On the other hand, some works [19, 33] predict the rotation without any constraints, but this often causes artifacts and corrupted generations in practice. To address this issue, we design novel rotation decoder, dubbed RotNet, which balances prediction flexibility and restriction. Our RotNet consists of set of 32 pre-defined rotation quaternions, denoted as T. which forms a canonical rotation space, and a learnable linear projection matrix θ to predict the logits of these quaternions. The predicted logits are then transformed into probability distribution using the Gunbel-Softmax [69], enabling differentiation through the discrete selection process by adding noise sampled from the Gumbel distribution to the logits p E 32 before applying the softmax function:
p softmax (Z + g), where
gk log( loggo ())( u4 Uniform(0, 1).
(4)
5

---

In this way. we convert the rotation prediction into 32-class classification task in fully differentiable way, which allows for direct selection via the argmax operation during inference. We refer to Appendix D for more detailed explanations. These decoded Gaussians are finally passed into the differentiable rasterization pipeline [21] for image-level supervision.
3.3 Stable Training of MVGamba
Bridging the training-inferenee gap. In the training phase, multi-view images are collected from the ground-truth blendering of 3D objects, while they generated by diffusion models during inference. To mitigate such domain gap: (1) Following LGM [19], we leverage the grid distortion and orbital camera jitter two data augmentations with 30% probability to simulate inconsistent pixels and inaccurate camera poses, respectively. (2) We directly use ImageDream [13] as synthetic data engine generate multi-view images input and conduct joint training with the ground-truth renderings from the 3D training dataset. In practice, with a 5% chance, we train MVGamba with synthetic input mimic the inference pattern for more robust generation results.
Overall training objective. During the training phase, we differentiably render the RGB image v; and alpha mask vf of the N = 4 imput views and another six novel views for image-level supervision. Our final objective then comprises four key terms:
L=
Σ
LMSE(Es (vi, vft) + Amask LMSE ( + ALPIPS of") + Xreghreg 21 |uu|l
(5)
where LMSE and famss represent the mean square error loss in the RGB image and the alpha mask, respectively; LLPIPS represents the well-adopted VGG-based perceptual loss [70] freg is the opacity LI regularization loss 1 asl encourage more efficient use of each Gaussian by enforcing higher density. Amask, CLPPS and Areg are the trade-off coefficients that balance each loss.
3.4 Unified 3D Generation Inference
During inference (Figure 3(b),, the pre-trained reconstructor can be smoothly combined with any off- the-shelf multi-view diffusion models to efficiently predict a set of Gaussians, which facilitates both text-to-3D and image-to-3D generation. Typically, we leverage ImageDream [13] and MVDream [14] to produce 4 multi-view images with anchored poses [11] from a single image or text prompt, respectively. For mesh extraction, following Huang al. [62], utilize truncated signed distance fusion (TSDF) [71] that fuse the depth maps rendered from the output Gaussians to obtain a smooth polygonal mesh.
Experiment
4.1 Experimental Settings
Training dataset. We obtain the multi-view images from Objaverse [7] for MVGamba pre-training. Following [19, 72], we filtered 80k valid high-quality 3D objects. We then used Blender under uniform lighting to render 25 views of RGBA images with their alpha masks a resolution of 512 x 512, in the elevation range of 5" to 30 with rotation (15 r/r [0, 23] r E NJ. To align with the camera configurations ImageDream [13] and MVDream [14], at each training step, we select images of certain object input views with the same elevations, while rotations separated by 90", denoted as (ó + 90" & e 1, 3} and another random set of 6 views as supervision.
Implementation details. MVGamba is trained on 32 NVIDIA A100 (80G) with batch size 512 for about 2 days. We adopt gradient checkpointing and mixed-preeision training with BF16 data type to ensure efficient training and inference. We use the Adam W optimizer with learning rate 1 x 10-3 3 and weight decay 0.05, following linear learning rate warm-up up for 15 epochs with cosine decay to 1 x 10 The output Gaussians are rendered at 512 x 512 resolution for mean square error loss and resized to 256 x 256 for LPIPS loss for memory efficiency. The trade-off coefficients that balancing each loss were set Amask = 1.0, ALPIPS = 0.6 and X 0.001. We also follow the common practice [19] to clip the gradient with a maximum norm of 1.0. The detail of MVGamba model configuration is included in Appendix D.

---

Input
DreamGaussian
Iriplan--Gausiann
LGM
MVGamba
"Child in Furry"
"A pot of flowers"
"Armor saldier"
Figure 4: Qualitative comparison in image-to-3D and text-to-3D generation. Please refer to Ap- pendix C for more generation results.
4.2 Comparison against Baselines
In this section, we compare MVGamba with previous state-of-the-art instant 3D generation methods in image-to-3DD, text-to-3D generation, and sparse-view reconstruction tasks. For each task, we first elaborate the evaluation metrics and baseline methods, then perform extensive qualitative and quantitative comparison.
Single image-to-3D generation. We make comparisons to recent methods, including optimization- based DreamGaussian [34], Wonder3D [73]; feed-forward methods LGM [19], TripoSR [74] and Triplane-Gaussian [33] and One-2345++ [75]. We adopted the official codebase and pre-trained model weight for all the above methods and we are quite confident that the baselines presented are the finest re-implementations we have come across". We evaluated the generation quality of MVGamba with a wide range of wild images in Figure We use well-adopted PSNR, SSIM and LPIPS for quantitative measurement in the GSO [76] dataset following [18], with a total of 16 test views with equidistant azimuth and -10 maintains high fidelity and plausible generation in most scenarios. In contrast, Triplane-Gaussian severely suffers from and blurred views, which notoriously ill-posed challenge, as stated by Instant3D [1]]. Moreover, LGM frequently showcases multi-view inconsistency with transparent surface, which may be attributed to its suboptimal parameter constraints and merge operation. In
degree elevations. As illustrated Figure 4, MVGamba
Note that Instant3D [11]. GS- LRM [20] and GRM [18] are not included for comparison in the current version, code has been publicly released yet.
7

---

Inpul
Novel view synthesis
Figure 5: Qualitative results in sparse-view reconstruction. Given four views as input, MVGamba effectively reconstructs both the geometric structure and detailed textures.
Table 1: Qualitative comparison of different methods based on various metrics.
Method
PSNRT
SSIM
LPIPS
CLIP
R-Prect
INF. Time
DreamGaassian [19]
21.59
0.80
0.16
0.793
53.5
120s
Wonder3D [73]
22.92
0.84
0.19
0,850
49.8
200s
One-2-3-45++ [75]
23.10
0.85
0.15
0.869
57.1
75s
TripoSR [74]
23.76
0.91
0,09
0.815
50.1
0.5s
TrissaneGaussian [33]
21.85
0.79
0.14
0.714
28.9
0.2s
LGM [19]
22.80
0.84
0.11
0.827
43.9
5s
MVGamba (Ours)
24.82
0.90
0.08
0.895
60.4
Table MVGamba outperforms other baselines in most evaluation metrics. Note that although our inference speed (4 seconds) is relatively slower than Triplane- Gaussian, we achieve significantly better generation quality by large margin of 2.97 dB PSNR.
Text-fo-3D generation. Similar to the single image-to- saage-to--DD task, we compare MVGamba with several optimization- based methods and feed-forward models LGM, TripoSR, and Triplane-Gausiine Gaussian, using various prompts. For methods that only accept single image as input, such as TripoSR and Triplane- Gaussian, we use DALL-E 3 [77] to transform the given text into a single image. Note that for more fair comparison, MVGamba and LGM take the same generated multi-view inputs. Figure shows that MVGamba excels at generating plausible geometry and highly realistic texture. Due to its effective ray embedding and multi-view diffusion models, MVGamba also free from the multi-face problems that frequently occur in optimization-baset methods. We further randomly selected 50 prompts from the Dreamfusion [1] gallery and used CLIP Precision and CLIP scores [78-80] to measure appearance quality and alignment with the given prompt. Our high generation quality is also vividly reflected in Table where MVGamba consistently ranks the highest among all baselines.
Sparse-view view reconstruction. Given the same sparse-view inputs, we compare MVGamba with SparseNeuS [81] (trained in One-2-3-45), SparseGS [82] and LGM [19] that are capable of generating 3D Gaussians. We visualize the reconstruction results of MVGamba in Figure 5 and compare against baselines on the GSO [76] dataset with randomly selected views for each object. As shown in Figure 2 and Figure 6, MVGamba is able to faithfully reproduce high-frequeng details with accurate geometric modeling. Due to the limited space, the quantitative evaluation presented in Appendix
Ablation and Discussion
Q1: Why MYGamba outperforms other LRMs in most 3D content generation tasks?
A1: To better diagnose the progress MVGamba, we conduct two ablation experiments on G-buffer Objaverse [72] Human subset: one is counterfactual experiment simulating severe inconsistency

---

Figure 6: RGB images and normal maps rendered by MVGamba-2DGS.
Multi-view Input
Nerge Openatin
0.00
26.0
0.06
25.2
Bequenoo
0.05
24.8
(n)
Sequence Length
(b)
Figure 7: (a) Worst-case simulation of the inconsistency introduced by the multi-view diffusion model. (b) The effect of sequence length on 3D reconstruction.
during inference
robustness, and the other on the effect of Gaussian sequence length for sparse-view reconstruction. (1) In Figure 7(a), we manually perturb one of the four input images with Gaussian noise simulate the worst-case multi-view input inconsistency caused by the diffusion model. We then feed these manipulated images into the MVGamba reconstructor and merge-operation reconstructor for comparison. separately and simply merging the outputs treats the perturbed input as faithfully as the other views, resulting in inconsistency and even corrupted results. contrast, our Gaussian reconstructor generates the Gaussian sequence in causal and self-refining manner, allowing it to mitigate the effects of the perturbation by leveraging propagated multi-view information. This experiment demonstrates that MVGamba is highly resilient to inconsistencies in multi- view diffusion models used in the first stage due to its causal sequence modeling. (2) We also investigate the effect Gaussian sequence length by varying the patch size to model different sequence lengths. As shown in Figure 7(b), SSM- based Gaussian reconstructor can directly generate extremely long Gaussian sequences, and its performance improves with increasing sequence length. From these aspects, that the higher generation performance of MVGamba is indeed attributable to its self relf-refineabl multi-view modeling and efficient utilization of sufficiently long Gaussian sequences.
expected, generating 3D Gaussians conditioned on each view
conclude
Q2: What impacts performance of MVGamba terms of component-wer contributions?
A2: In Table 2, we analyze the component-wise con- tributions of MVGamba by verifying our design choices of multi-view image encoder, Gaussian decoder structure and training strategy. Note that this ablation is conducted on the filtered subset of Human category in G-buffer Ob- javerse [72] using smaller model architecture for better energy efficiency. Considering symbol simplicity, we de- note patchify convluation as PC: Gaussian Decoder with RotNet as GD; stable training strategy ST. Table 2 illustrates that the replacement or exclusion any compo- nent from MVGamba resulted in significant degradation
Table 2: Performance comparison of differ- ent model configurations.
Model
PSNR
SSIM
LPIPS
w/o PC
25.20
0.827
0.102
w/o GD
25.41
0.788
0.096
w/o ST
26.69
0.910
0.065
Ours
27.13
0.925
0.057

---

Multi-view Input
Novel View Synthesis
Front- view First
Side-view First
Figure 8: Ablation study on input order. Top: MVGamba may fail if the depth of the front-view is estimated incorrectly and the front-view is given first. Bottom: Manually changing the input order to provide the side-view first allows MVGamba to generate satisfactory 3D content, as the side-view contains sufficient depth information.
in performance. In particular, if we replace our designed Gaussian Decoder with the one in previous LRMs [11, 29] (10-layer, 64-width shared MLP), we notice large performance drop due to degraded sequential modeling ability and tendency for the deep MLP to overfit. Additionally, the lack of restrictions on position and rotation could cause more artifacts during both training and inference.
Q3: What is the limitation of MVGamba?
A3: Honestly, though MVGamba achieves promising results, it still has several limitations. (1) As we model the Gaussian sequence causally, MVGamba can sometimes fail if the depth of the front view is not estimated correctly (Figure 8 top). Fortunately, we empirically find that this limitation can be mitigated by manually changing the input order. For example, using a side-view as the first input allows our model to generate satisfactory 3D content, the side-view contains sufficient depth information (Figure bottom). In the future, we may explore automatic ways to optimize the input order to enhance robustness. (2) The generation quality of MVGamba is highly dependent the multi-view diffusion models [13, 14]. However, current
four input views provided by off-the-shee mul view diffusion models are far from perfect, known to exhibit 3D inconsistencies [9, 19, 83], and limited to a resolution of 256 x 256. We expect that our model's performance can be seamlessly boosted with the advancements in multi-view diffusion models in future work.
Conclusion
In this paper, we introduce MVGamba, a general and lightweight Gaussian reconstruction model for unified 3D content generation. MVGamba features novel multi-view Gaussian reconstructor based on state space sequence modeling, maintaining multi-view information integrity and enabling cross-view self-refinement. generates long Gaussian sequences with linear complexity in a single forward process, eliminating the need for post hoc operations. Extensive experiments demonstrate that MVGamba (49M parameters) outperforms state-of-the-ard LRMs in various 3D generation tasks with only 0.1x the model size. In general, MVGamba achieves state-of-the-arr quality and high efficiency in parameter utilization, training, inference, and rendering speed. the future, we aim to apply MVGamba to a wider range of 3D generation tasks, such scene and 4D (dynamic) generation.
10

---

7 Acknowledgments
This project is supported by Kunlun 2050 Research, Skywork Al and Agency for Science, Technology AND Research, and by the National Research Foundation, Singapore, under its Medium Sized Center for Advanced Robotics Technology Innovation, Pan Zhou was supported by the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grants (project ID: 23- SIS-SMU-028
and
References
[1] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[2] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310. 168888 2023.
[3] Zike Wu, Zhou, Xuanyu Yi, Xiaoding Yuan, and Hanwang Zhang, Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior. arXiv preprint wrXiv:2401.0905. 2024.
[4] Zhengyi Wang, Cheng Lu, Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305.. 13213, 2023.
[5] Xuanyu Yi, Zike Wu, Qingshan Xu, Pan Zhou, Hwee Lim, and Hanwang Zhang. Diffusion time-step curriculum for one image to 3d generation. arXiv preprint 24f::2004.05622 2024.
[6] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang. Jiaqi Wang. Chen Qian, et al. Omniobject3d: Large-vocatee vorabulacyy 3d object dataset for realistic perception, reconstruction and generation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 803-814, 2023.
[7] Matt Deitke, Dustin Schwenk, Jordi Salvador. Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. Ali Farhadi. Objaverse: A universe annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.
[8] Xianggang Yu, Mutian Xu. Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Tha, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9150-9161, 2023.
[9] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang. Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.0711 2024.
[10] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli. Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh arXiv preprint arXiv:2404. 12385, 2024.
[11] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.08 14, 2023.
[12] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint 200332403 050344 2024.
[13] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:
2023.
[14] Yichun Shi, Peng Wang, Jianglong Ye. Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi-view diffuuson for 3d generation. arXiv preprint ar0iv:2308. 16512, 2023.
[15] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23+H a single image to consistent view diffusion base model. arXiv preprint 2xX1v:2310.151111 15100, 2023.
11

---

[16] Ryan Shue, Eric Ryan Chan. Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 20875-20886, 2023.
[17] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-await 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16123-16133, 2022.
[18] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint 146221, 2024.
[19] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint wrXiv:2402.05055 2024.
[20] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404.1970 19702, 2024.
[21] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendcring. ACM Transactions on Graphics (ToG), 42(4):1-14, 2023.
[22] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asissed intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 20Xi::2010.11922 2020.
[25] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 20005:23/2.00752, 2023.
[26] Lianghui Zhu, Beneheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidircctional state space model. arXiv preprint MrXiv:2401.094177 2024. 2024.
[27] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401. 2024.
[28] Jinbin Bai, Tian Ye, Wei Chow, Eaxin Song, Qing-Guo Chen, Xiangtai Li. Zhen Dong. Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. arXiv preprint arXiv:2410.08261. 2024.
[29] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311. 144400 2023.
[30] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arxiv:231 1,0921,, 2023.
[31] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Yao Huang. Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Learning robust visual features without supervision, 2023.
Dinov2:
[32] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403.1788 178888 2024.
12

---

[33] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang. Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147, 2023.
[34] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei I Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023.
[35] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45; Any single image to 3d mesh in 45 seconds without per- per-shape optimization. arXiv preprint arXiv:2306. 169228 2023.
[36] Jiaxiang Tang.
Stable-dreamfusionn:
Text-to-3d
with
stable-diffusion, 2022.
[37] Yukun Huang. Jianan Wang. Yukai Shi, Xianbiao Qi. Zheng-Jun Zha, and Lei Zhang. Dreamtime: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 0e233066.2222 /2422, 2023.
[38] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint xrXiv:331..1224, 2023.
[39] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298-9309, 2023.
[40] Yuanxun Lu, Jingyang Zhang, Shiwei Li, Tian Fang, David McKinnon, Yanghai Tsin, Long Quan, Xun Cao, and Yao Yao. Direct2. 5: Diverse text-to-3d generation via muel 2.5 d diffusion. arXiv preprint arXiv:2311.159%0 2023.
[41] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrett, convolutional, and continuous-time models with linear state space layers. Advunces in neural information processing systems, 34:572-585, 2021.
[42] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971-35983, 2022.
[43] Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.079%) 2024.
[44] Rui Xu, Shu Yang, Yihui Wang. Bo Du and Hao Chen. A survey on vision mamba: Models, applications and challenges. arXiv preprint arXiv:2404.1888 2024.
[45] Xiao Liu, Chenxu Zhang, and Lei Zhang. Vision mamba: comprehensive survey and taxonomy. arXiv preprint Arxiv:2405. 044044 2024.
[46] Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges. arXiv preprint MrXiv:2404.16112. 2024.
[47] Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan, arXiv preprint arXiv:2403. 093388 2024.
[48] Chenhongyi Yang. Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition. arXiv preprint arXiv:2403.17699 2024.
[49] Zunnan Xu, Yukang Lin, Haonan Han, Sicheng Yang, Ronghui Li, Yachao Zhang, and Xiu Li. Mamhatalk: Efficient holistic gesture synthesis with selective state space models. arXiv preprint MxXix22403.09471 2024.
[50] Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. VI-mamha: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403. 13600, 2024.
[51] Jun Ma. Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722. 2024.
[52] Renkai Wu, Yinghao Liu, Pengchen I Liang, and Qing Chang. H-vmunet: High-order vision mamba unet for medical image segmentation. arXiv preprint arXiv:2403. 2024.
[53] Ziyang Wang and Chao Ma. Weak Weak-mamba-te mamba-unet Visual mamba makes enn and vit work better for scribble- based medical image segmentation. arXiv preprint 1088 2024.
13

---

[54] Wenhao Dong, Haodong Zhu, Shaohui Lin, Xiaoyan Luo, Yunhang Shen. Xuhui Liu, Juan Zhang. Guodong Guo, and Baochang Zhang. Fusion-mamba for cross-modality object detection. arXiv preprint arXiv:2404, 09140, 2024.
[55] Hongruixuan Chen, Jian Song, Chengxi Han, Junshi Xia. and Naoto Yokoya. Changemamba: Remote sensing change detection with spatio-temporal state space model. arXiv preprint arX1v:2404.03425 2024.
[56] Xuanhua He, Ke Cao, Keyu Yan, Rui Li, Chengjun Xie, Jie Zhang, and Man Zhou. Pan-mamba: Effective pan-sharpening with state space model. arXiv preprint arXiv:2402. 2024.
[57] Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding. Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. arXiv preprint MxXXv:22433115222 2024.
[58] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403. 19887, 2024.
[59] Chi-Sheng Chen, Guan- Ying Chen, Dong Zhou, Jiang, and Dai-Shi Chen. Res- Besmbaamba: Fine- graiegg food category visual classification using selective state space models with deep residual learning, arXiv preprint arXiv:2402. 15761, 2024.
[60] Haoyang He. Yuhu Bai, Jiangning Zhang. Qingdong He, Hongxu Chen, Zhenye Gan, Chengjie Wang. Xiangtai Li, Guanzhong Tian, and Lei Xie. Mambaad: Exploring state space models for multi-class unsupervised anomaly detection. arXiv preprint arXiv:2404.06566 2024.
[61] Qiuhong Shen, Xuanyu Yi, Zike Wu, Pan Zhou, Hanwang Zhang, Shuicheng Yan, and Xinchao Wang. Gamba: Marry gaussian splatting with mamba for single view 3d reconstruction. arXiv preprint MnXiv:24031187953 2024.
[62] Binbin Huang. Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In SIGGRAPH 2024 Conference Papers. Association for Computing Machinery, 2024. doi: 10.1145/3441519.367444
[63] Pinxuan Dai, Jiamin Xu, Wenxiang Xie, Xinguo Liu, Huamin Wang. and Weiwei Xu. High-quality surface reconstruction using gaussian surfels. arXiv preprint trXiv:2404.17774 2024.
[64] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering, Advances in Neural Information Processing Systems, 34:19313 193255, 2021.
[65] Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, al. Beyond efficiency: A systematic survey of resource-efffcient large language models. arxiv preprint arXiv:2401.0m 00625, 2024.
[66] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[67] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819-10829, 2022.
[68] Dejia Xu, Ye Yuan, Morteza Mardani, Sifci Liu, Jiaming Song. Zhangyang Wang. and Arash Vahdat. Agg: Amortized generative 3d gaussians for single image to 3d. arXiv preprint arXiv:2401. 0440999 2024.
[69] Eric Jang, Shixiang Gu and Ben Poole. Categorical reparameterization with gumbel-sofimax. arXiv preprint 16 1..00 144,0111111111 2016.
[70] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595. 2018.
[71] Brian Curless and Mare Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303-312, 1996.
[72] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu. Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable cormal-depth diffusion model for detail richness in text-to-3d 3d. arXiv preprint ar119:2311.16918, 2023.
14

---

[73] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song- Hai Zhang. Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusiin, arXiv preprint urXiv:2310. 15008, 2023.
[74] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte. Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint 20Xv:2243.00151, 2024.
[75] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-4544 Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint orXiv:2311.000 078885, 2023.
[76] Laura Downs, Anthony Francis. Nate Koenig. Brandon Kinman, Ryan Hickman, Krista Reymann. Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned houschold items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553. IEEE, 2022.
[77] James Betker, Gabriel Goh, Li Jing. Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyee Lee, Yufci Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e- pdf. 2(3):8, 2023.
[78] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMIR, 2021.
[79] Xuanyu Yi, Jiajun Deng, Qianru Sun, Xian-Sheng Hua, Joo-Hwee Lim, and Hanwang Zhang. Invariant training 2d-3d joint hard samples for few-shot point cloud recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14463-14474, 2023.
[80] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 15659- 15669, 2023.
[81] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision, pages 210-227. Springer. 2022.
[82] Haolin Xiong. Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Sparsegs: Real-time 360 ('deg} sparse view synthesis using gaussian splatting. arXiv preprint arXtv:2312.00206, 2023.
[83] Yuxuan Wang. Xuanyu Yi, Zike Wu, Na Zhao, Long Chen, and Hanwang Zhang. View-consistent 3d editing with gaussian splatting, arXiv preprint 2rXiv 1166,, 2024.
[84] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.0m 02643,, 2023.
[85] David Charatan, Sizhe Li, Andrea Tagliasacchi. and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312.12337, 2023.
[86] Qingshan Xu, Xuanyu Yi, Jianyao Xu, Wenbing Tao, Yew-Soon Ong, and Hanwang Zhang, Few-shot nerf by adaptive rendering loss regularization. arXiv preprint arXiv:2410. 17839, 2024.
[87] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6100 6101, 2021.
15

---

Appendices
The Appendix is organized follows:
Appendix A: claborates more details about our MVGamba pipeline. Specifically, we detailed the implementation of inference pipeline and mesh extraction.
Appendix B: discusses the comparison between concurrent transformer-based methods and the function or pixel-aligned representation.
Appendix C: showcases more qualitative and quantitative experiment results.
Appendix D: provides further details model design and configuration.
A Implementation Details
Inference. For single image input, MVGamba first segments the recentered foreground object using a pre-trained segmentation model [84], then leveraging multi-viww diffusion model, typically ImageDream, generate four posed views as input for the SSM-reconstructor. Similarly, for text prompt input, MVGamba uses MVDream to transform the given text into four posed views. For sparse-view reconstruction, MVGamba takes the given few views as the ground-truth multi-view inputs. MVGamba employs a default camera pose with both zero elevation and azimuth to produce camera tokens and Plücker ray inputs. Remarkably, this process requires only about 10 GB of GPU memory (for both the multi-view diffusion model and the Gaussian Reconstructor) and completes in less than seconds (4.5 seconds for multi-view image generation and 0.03 second for predicting Gaussians and real-time rendering) single NVIDIA A800 (80G) GPU, making it well-suited for online deployment scenarios.
Mesh extraction. For mesh extraction from reconstructed 2D splats, following [32], we render depth maps of the Gaussian rendering views by using the median depth value of the splats projected onto the pixels. We then use truncated signed distance fusion (TSDF) with Open3D to fuse the reconstruction depth maps. We reset the voxel size and the truncation threshold during TSDF fusion suitable for object mesh extraction.
B
More Discussion
B.1
Comparison with concurrent works.
We illustrate the two current sequence modeling paradigms for Gaussian LRMs: Compressed vs. Direct Sequence Modeling (ours) in Figure 9. Below, we elaborate on these two paradigms to highlight the unique approach of our proposed paradigm compared to recent transformer-based approaches, e.g., GRM, GS-LRM.
(a) Compressed Sequence Modeling: This approach tokenizes the input into a compact representation, processes it through a series of transformer blocks, and then upsamples to produce the 3DGS parameters. This paradigm represented by the concurrent GS-LRM and GRM.
(b) Direct Sequence Modeling (ours): This approach tokenizes and expands the input into sufficiently long sequence of tokens through cross-scan operations, which are then directly processed by series of mamba blocks to generate the 3DGS parameters.
Our proposed paradigm (b) offers several significant benefits:
Efficient long sequence modeling: It accommodates larger spatial dimensions to cap- ture and preserve fine-gra -raiied details while mitigating information loss typically caused by upsamplers, such as the zero zero-pade paro-iadding in de-convolution layers. This benefits accu- rate geometry and texture reconstruction. As demonstrated Table 3, Mamba's linear computational complexity allows for favorable balance between computational cost and long-sequence modeling capacity, enabling efficient processing of high-resolution inputs without prohibitive memory requirements.
16

---

1003 entedding
Ipput
Parck rabano
50GS
Paten towere
Blocks
(a)
(b)
Figure 9: The illustration of two type of Gaussian LRMs. (a) Compressed sequence modeling (b) Direct long-sequence modeling (ours).
Table 3:
Theoretically calculated FLOPs comparison between self-attention in Transformer and
SSM in Mamba. Here, dimension D 512; GFLOPS of both modules calculated according Equations (5) and (6) in Vision Mamba [26].
Model Length
1024
2048
4096
8192
16384
32768
Self-attention GFLOPs
2.15
6.44
21.47
77.31
292.06
1133.87
SSM / GFLOPs
0.07
0.13
0.27
0.54
1.07
2.15
Easy of optimization: It directly models the 3DGS sequence without any upsampler, hence establishing a more straightforward relationship between 3DGS parameters (e.g.. position, color) and the modeled sequence. This eases the non-convex optimization in inverse graphics scenario, discussed in PixelSplat, and helps learn accurate geometry and textures.
Cross-view self-refinement: It efficiently incorporates multi-view information causally from the initial condition onward, thereby enabling the refinement of inconsistent parts based earlier views and generated tokens.
B.2 The function of pixel-aligned Gaussian
We conducted an additional experiment using non-pixel-aligned transformer model for 3DGS prediction. As discussed in Appendix B.I, transformer-baset methods typically process a compressed sequence considering the computational budget. We then followed OpenLRM to model 4096 tokens, and adopt de-convolution layer up-sample the 3DGS tokens from 4096 16384, similar to TGS for intricate 3D modeling. However, as shown in Figure 11, the -on-pixel-allgned transformer yielded inferior performance, with broken geometry and extremely blurred textures after 300 epochs of training. We attribute this to the inherent non-convex optimization challenge in the inverse graphics scenario, as mentioned in pixelSplat [85], which may be further amplified by the upsampler. This preliminary result may also explain why pixel-aligned approaches are proposed and are adopted by recent transformer-based approaches with upsamplers. the other hand, our proposed direct sequence modeling paradigm allows for more fine-grained detail modeling and exhibits cross-view self-refinement ability. Morcover, our direct sequence modeling paradigm can also case optimization, which provides new feasible way for training Gaussian LRMs.
C
More Results
Table 4: Quantitative comparison on sparse view reconstruction.
Method
#views
PSNR
LPIPS
SSIMI
INF Time!
CD1
VloUt
SparseGS [82]
16
22.19
0.162
0.775
34s
SparseNeuS [81]
16
23.17
0.130
0.814
6s
0.0566
0.3479
LGM [19]
4
24.20
0.112
0.845
0.07s
0.0198
0.4410
MVGamba
4
26.25
0.069
0.881
0.03s
0.0132
0.4829
More qualitative results. We include more qualitative experiment on image-to-3D generation with rendering results in Figure 10 for novel view synthesis and in Figure 12 for normal map generation.
17

---

Input
Novel View Synthesis (Ours)
Figure 10: More qualitative results of MVGamba image-to-3D generation.
We also include qualitative comparison using samples from the GRM paper in Figure 13. indicates that MVGamba captures better geometries and details with significantly smaller model size. Note that we could not directly compare with GS-LRM as it only presents single-view results in Figure 7 of their paper.
More quantitative results. We leave the qualitative comparison sparse-view reconstruction task here. We compare the well-adpoted PSNR, SSIM, LPIPS for novel view synthesis [86] and Chamfer Distance (CD) and Volume loU (VIoU) for geometric evaluation. As shown in Table 4, MVGamba outperforms all baselines across all metrics, even though SparseNeuS require 4 times more input views, while maintaining fast inference and rendering speed.
18

---

Figure 11: The generation results of Non-pixel-alignee transformer- based Gaussian LRM, resulting in extremely broken geometry and blurred texture.
Figure 12: RGB images and normal maps rendered by AVGamba-2DGS.
D
Model Details
Network Configuration. As detailed in Table 5, the SSM-based reconstructor comprises 14 Gamba blocks, each with hidden dimensions of 512. The architecture employs RMSNorm, SSM, depth-wise convolution [87], and residual connections. In line with [24], positional embedding added rather than concatenated to the tokenized multi-view image tokens. These tokens have 512 dimensions, resulting in total of 16,384 or 32,768 tokens, which correspond to the length of Gaussian sequence. The Gaussian Decoder a compact multi-layer perceptron (MIP) with single hidden layer containing 2,048 hidden dimensions. This is followed by sub-head linear projection layers, which decode the output into 3D Gaussians for splatting.
Our RotationNet (RotNet) is designed to facilitate the prediction of rotation attributes. comprises of 32 pre-detined canonical rotation quaternions and a learnable linear projection matrix that maps input features to the logits of these quaternions. The canonical quaternions include 8 rotations around the principal axes (z,, y. 2) by or 180 degrees, and 24 rotations by +45 degrees around various axes formed by combining two principal axes. During training, RotNet uses the Gumbel-Softmal technique to enable differentiable sampling from the discrete set of quaternions. The temperature parameter is gradually decreased from to 0.01 over iterations, encouraging more confident predictions. During inference, RotNet applies the argmax operation to the logits to select the quaternion with the highest probability, operating in non-differentiabbl manner.
RotationNet.
19

---

Input
LGM
GRM
MVGamba
piy worring bedspack
Figure 13: More comparisons with concurrent state-of-the-an large Gaussian models. Note that the generation results for GRM were extracted from its paper and official project page. A direct comparison with GS-LRM is currently unfeasible, as it is not open-source, and only single view is presented in Figure 7 of the GS-LRM paper.
Algorithm 1: Gaussian parameter constraint
# Input: feats: [b, n. ...]
Output: gsparams: list of constrained parameters
constraints
"xyz": lambda v: normaliee (v. reshape (* shape [:2], 3, -1), dim=-1) coords [None, Nonel) sum (dim=-1),
"scale": lambda 0.1 * F.softplus (v),
"rot": lambda RotationNet () (v), #proposed RotNet
"opacity"; lambda v: torch signod
"rgb": lambda v: torch.sigmone sigmoid
shs": lambda V: v # No apply SHS
};
Figure 14: The pseudo code for Gaussian parameter constraint.
Gaussian Parameterization. As 3D Gaussians are an unstructured explicit representation, unlike Triplane-NeRFF structured implicit representation, the parameterization of the output parameters significantly affects the model's convergence. We provide the detailed pseudo code of Gaussian parameterization in Figure 14 for better reproduciblility.
E
Licenses
Datasets:
Objaverse [7]: ODC-By v1.0 license
Pre-trained models:
20

---

Table 5: Detailed model configuration of MVGamba.
Parameter
Value
Image Tokenizer
image resolution
448
448
patch size
14
channels
512
Backbone
Mamba layers
14
# channels
512
state expansion factor
16
local convolution width
4
block expansion factor
2
normalization
RMSNorm
Decoder MLP
width
2048
hidden layers
activation
SiLU
Training
optimizer
AdamW W
epochs
300
batch size
512
learning rate
1e-3
weight decay
0.05
gradient clipping
1.0
Adam (2)
(0.9, 0.95)
Ir scheduler
CosineAnnellingL.R
warm-up epochs
15
Amask
1.0
ALPIPS
0.6
Xreg
0.1
ImageDream [13]: Apache-2.0 license
MVDream [14]: MIT License
SAM [84]: Apache-2.0 license
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions and scope of the paper, matching the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides all necessary details to reproduce the experiments in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
The code is available online, and the dataset is open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
son.eeesssbbmissssooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //o////ooaeeeeeeeeeee
/nips wwtppbccc//aaeeccccccccccccc / ssisonolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All training and evaluation details, including dataset, hyperparameters, opti- mizers, and model configurations are listed in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because would be too computationally expensive
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information on compute resources, including type of compute workers, mem- ory and and execution time. are provided in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eerrpp.eeeee
cwwiiiiuuieeinnie dilins??
Answer: [Yes]
Justification: The research conforms to the NeurIPS Code of Ethics, with social impacts discussed the checklist.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While our work unify the 3D content generation, also raises concerns about potential negative societal effects, including the propagation of disinformation or the amplification of harmful biases with the generated contents.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We plan to ask user agree to usage guidelines before model after released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Licenses of dataset and pre-trained models are listed in Appendix.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithhaeee thcode. wwwccoo/////o//o datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "75w8DNyFC1", "reviewer": "Reviewer_XU7f", "summary": "The authors propose MVGamba, a general and lightweight Gaussian reconstruction model for unified 3D content generation. By replacing the previous LRM work’s transformer architecture with the recent model Mamba. MVGamba can generate long Gaussian sequences with linear complexity in a single forward process, eliminating the need for post hoc operations. Experiments demonstrate that MVGamba outperforms the current open-sourced LRM work in various 3D generation tasks with a smaller model size( 0.1× ). The authors also claim the proposed model has the potential to be applied to scene-level and 4D generation.", "questions": "- In lines 38-41, the authors state that previous Gaussian-based LRMs fail to achieve multi-view consistent results because they rely on local or mixed attention across limited multi-view image tokens or process each view separately before merging the predicted Gaussians. However, some prior works, like GS LRM, employ self-attention on all input image tokens. Despite merging the Gaussians afterward, they are predicted collectively. It is unclear why this approach leads to multi-view inconsistency.\n- In lines 281-286, the authors note that performance improves with increasing sequence length, and they model different sequence lengths by varying the patch size. If decreasing patch size enhances final performance without significantly impacting efficiency, as the authors claim, why didn't they choose a smaller patch size than 14, as indicated in the appendix? 14 is actually bigger than some transformer-based LRM’s patch size. (For example, GS LRM chose 8.)", "limitations": "The limitations have been well discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand, and the experiments are sufficient and well-analyzed.\n2. The motivation of this paper is sound. The current LRMs papers have achieved impressive results in 3D generation, but their architecture is highly computationally intensive, especially when handling long-sequence inputs. The recent Mama has the potential to solve it. \n3. The proposed pipeline achieves better results than the selected baselines.", "weaknesses": "1. The paper’s experiments do not align with its claims regarding the inefficiencies of transformer-based LRMs in handling long-sequence inputs. The primary issue highlighted is that these transformer-based models are not efficient for long-sequence input. However, most experiments in the paper are for single or sparse image input 3D generation. In this context, the Mamba-based architecture does not demonstrate significant speed improvements over transformer-based architectures because the input tokens are relatively short. Table 1 shows no obvious speed enhancement compared to other feedforward methods, supporting this point. Therefore, I question the necessity of Mamba for most tasks discussed in the paper. The Mamba-based reconstruction model appears to be more suitable for dense view reconstruction tasks.\n\n2. The performance of paper doesn’t achieve the state of the art. Although in Table 1, the paper achieves better quantitative results over all baselines, some previous LRMs papers have better performance according to their paper. For example, GS-LRM attains 8 dB higher PSNR than LGM according to its table. 1, while MVGamba is just 2 dB higher than LGM. I can understand this paper didn’t compare with GS LRM because GS LRM doesn’t release its code. I am just not sure if Mamba is important for quality improvement as claimed by the authors, when some transformer-based LRMs can also achieve good (or even better) quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and insightful comments! In the following, we provide our point-by-point response and look forward to the subsequent discussion.\n\n>**Q1-1: Input tokens are short.**\n\n**A1-1:  3DGS reconstruction is a long-sequence task.** We analyze the token length of two paradigms in rebuttal PDF Figure 1:\n+ Direct long-sequence modeling: Requires directly processing more than 16k tokens.\n+ Compressed sequence modeling: For a standard $384 \\times 384$ resolution input with a large hidden dimension $D=1024$ in GS-LRM, [1] proposed a ratio-based metric, where we derive $r_L \\approx 1.5 > 1$, confirming it as a long-sequence task even with compressed sequences. Note that $r_L \\approx 0.04$ in traditional image classification tasks.\n\nMoreover, we agree that Mamba-based reconstructor may potentially offer more advantages as views and sequence length increase, which we consider an interesting direction for future work.\n\n[1] Yu, W., \\& Wang, X. MambaOut: Do We Really Need Mamba for Vision?. arXiv:2405.07992.\n\n>**Q1-2: Inference speed in Table 1.**\n\n**A1-2:**\n+ In Table 1, MVGamba's inference time includes both multi-view image generation and **multi-view reconstruction (0.03s)**, as detailed in Appendix A. When considering the reconstruction module alone, **MVGamba is at least $3.6\\times$ faster** than GRM (0.11 sec), TGS (0.2 sec), and GS-LRM (0.23 sec).\n\n+ Importantly, MVGamba's direct sequence modeling achieves significant inference speed improvement even when modeling sequences that are $4\\times$ longer.\n\n>**Q1-3: Necessity of Mamba.**\n\n**A1-3: Mamba-based reconstructor is necessary.** Given the above responses in A1-1 and A1-2, we believe: 1) Sparse-view 3DGS reconstruction is a long-sequence task; 2) MVGamba achieves at least $3.6\\times$ faster inference speed while directly modeling $4\\times$ longer sequences. To our knowledge, Mamba-based reconstructor is the only experimentally validated direct sequence modeling approach. As detailed in overall response, MVGamba efficiently balances computational cost with long-sequence modeling capacity and offers additional valuable advantages, making it crucial for direct sequence modeling.\n\n>**Q2-1: Quantitative comparison with GS-LRM.**\n\n**A2-1:**\n+ **Currently, conducting a direct and fair quantitative comparison between MVGamba and GS-LRM meets significant challenges.** We believe the performance gap is largely due to **differences in experimental settings and evaluation protocols**, such as differences in test samples, rendered views, and camera trajectories.  Notably, even the number of input views for text/image-to-3D tasks differs, as GS-LRM utilizes 6 generated images in their multi-view reconstructor.  While GS-LRM is promising, it was pre-printed on arXiv near the NeurIPS deadline without released code, leaving us insufficient time for thorough analysis or reproduction. We are eager to conduct a comprehensive comparison once GS-LRM is open-sourced or re-implemented. MVGamba will also be fully open-sourced to facilitate the 3D community.\n\n+ **MVGamba's reconstruction quality can be further improved with larger model size.** Our newly trained MVGamba-B (110M) surpasses the original MVGamba-S (49M) in the manuscript by 1.9 dB in the GSO dataset for sparse-view reconstruction. We plan to further explore the potential of MVGamba with larger model size, advanced 3D representations and stronger Mamba-based architectures such as Mamba-2[2] and Samba[3].\n\n[2] Dao, T., \\& Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. ICML2024.\n\n[3] Ren, L., et. al. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling. arXiv:2406.07522.\n\n>**Q2-2: Is Mamba important for quality?**\n\n**A2-2: Mamba is crucial for performance improvement.** As detailed in A1-3, the Mamba-based reconstructor offers several unique advantages that enhances both quality and efficiency.\n\nTo further address your concerns, Figure 3 of the rebuttal PDF shows a qualitative comparison using samples from the GRM paper. It indicates that MVGamba captures better geometries and comparable details with significantly smaller model size. Note that we could not directly compare with GS-LRM as it only presents a single-view results in Figure 7 of their paper.\n\n>**Q3: Why might GS-LRM suffer multi-view inconsistency?**\n\n**A3:** Potential multi-view inconsistency in GS-LRM may stem from:\n+ **View-separated up-sampling:** GS-LRM upsamples tokens in each view separately using local deconvolution, and a large amount of Gaussians are predicted in each isolated view without 'attention' to mutual information from other views.\n+ **Merge operation:** GS-LRM merges these separately upsampled Gaussians without multi-view context modeling.\n\nNote that GS-LRM was pre-printed near the NeurIPS deadline without released code. All the analysis above is based on carefully reading their paper and our empirical knowledge of this field, acknowledging potential understanding limitations without access to GS-LRM's implementation details.\n\n>**Q4: Why choose patch size 14?**\n\n**A4:** We chose a patch size of 14 based on theoretical and empirical considerations.\n+ **Theoretical Insight:** MVGamba is a non-pixel-aligned model that processes causal context, requiring each patch to contain sufficient information, similar to traditional vision transformers like ViT-H-14. In contrast, GS-LRM is a pixel-aligned model and performs per-pixel Gaussian generation may require smaller patch.\n\n+ **Empirical Evidence:** Ablation studies with a fixed token length of 16k on the Human-Shape subset showed patch size 14 outperformed patch size 8 by 0.71 dB PSNR. The average loss was 0.03243 for patch size 14, compared to 0.03918 for patch size 8, supporting that MVGamba benefits by patches with sufficient information.\n\nBesides, MVGamba offers additional ways to increase sequence length. Please kindly refer to the response to eshR A5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lBPk3GgwzT", "reviewer": "Reviewer_eshR", "summary": "This paper introduces MVGamba, a feed-forward, sparse reconstruction model. This model takes a small number of images (e.g., 4 views) to infer 3D Gaussians. Basically, it is a Mamba version of multi-view large reconstruction model. The authors have implemented several strategies to ensure stable training and optimal performance.\n\nExperimental results demonstrate that the MVGamba model can reconstruct 3D Gaussians with higher quality than the Large Gaussian Model (LGM).", "questions": "(1)\tLine 53: Could it be more specific? What does 'post hoc' operation specifically mean?\n\n(2)\tWhat is the number of output 3D Gaussian primitives? Does it equal N (the number of tokens)? Is there any way to increase the number of Gaussians for better reconstruction quality?\n\n(3)\tGiven the advantages of Mamba in computational efficiency for long sequences, did the authors try to train an MVGamba with a larger number of views, such as 10 views?", "limitations": "The method is limited in object-level reconstruction with clean background. I encourage the authors explore on scene-level reconstruction as an interesting future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)\tThe experimental results are somewhat promising, showing better Gaussian reconstruction quality than LGM, although the quality is not as high as the concurrent GRM and GS-LRM works.\n\n(2)\tI like the experiment design in Section 5 Q1; it provides some motivation for why we need a non-pixel-aligned architecture. It seems this kind of architecture is more robust to multi-view inconsistency.\n\n(3)\tThe paper is clearly written and easy to understand.", "weaknesses": "(1)\tIn the abstract and also in Lines 40-41, the authors state that the generated 3D models often suffer from multi-view inconsistency and blurred textures, which is why MVGamba is proposed. However, it seems that this issue is more likely due to whether the architecture is pixel-aligned or not. Therefore, I believe a non-pixel-aligned transformer model can also address these issues. Did the authors try this? How does a non-pixel-aligned transformer model perform?\n\n(2)\tI cannot find any tables or numbers in the paper to support the conclusion “0.1× of the model size.” Did the authors forget to include them?\n\n(3)\tThe reconstruction results look in lower quality than the concurrent GRM and GS-LRM works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments! Below, we provide a response to address your concern. We welcome further discussion to enhance the clarity and effectiveness of our work.\n\n>**Q1: How does a non-pixel-aligned transformer model perform?**\n\n **A1:**\n\n- **Non-pixel-align is not the primary cause.** Per your suggestion, we tested a non-pixel-aligned transformer model for 3DGS prediction. As discussed in our overall response, those methods typically process a compressed sequence due to the computational budget. We followed OpenLRM[1] to model 4096 tokens, and adopt a de-convolution layer to up-sample from 4096 to 16384. However, as shown in the rebuttal PDF Figure 2, it yielded inferior performance, with broken geometry and blurred textures. We attribute this to the inherent non-convex optimization challenge in the inverse graphics scenario, as mentioned in pixelSplat[2], which may be further amplified by the de-conv upsampler. This preliminary result may also explain why pixel-aligned Gaussian are widely adopted by recent transformer-based approaches with upsamplers.\n\n- **The crux is to directly model a sufficiently long sequence of 3DGS in a causal manner.** As mentioned in the manuscript and our overall response, the direct sequence modeling paradigm allows for more fine-grained detail modeling with cross-view self-refinement . Moreover, our direct sequence modeling paradigm can also ease optimization, which provides a new feasible way for training Gaussian LRMs.\n\n[1] He, Z. and Wang, T. OpenLRM: Open-Source Large Reconstruction Models. \n\n[2] Charatan, D., et al. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. CVPR2024.\n\n> **Q2: Model size comparison.**\n\n **A2:** Thank you for pointing out! The following table discusses the model size that supports our claim. Our statement of *0.1× of the model size* refers to a comparison with open-source SOTA models TripoSR and LGM. This table will be included in the revision.\n\n\n\n| Model      | OpenLRM | TGS   | TripoSR | LGM   | GS-LRM | **MVGamba** |\n|:-----------:|:-------:|:-----:|:-------:|:-----:|:------:|:-----------:|\n| Model Size  |  366M   | 260M  |  ~400M  | 415M  |  300M  |   **49M**   |\n\n\n >**Q3: Reconstruction results comparison.**\n\n **A3:** To address your concerns, we provide an additional qualitative comparison using samples derived from the original GRM paper. As shown in the rebuttal PDF Figure 3, MVGamba accurately captures better geometric structures and comparable appearance details compared to GRM. We could not directly compare with GS-LRM as it only presents a single-view results in Figure 7 of their paper.\nNote that while GRM and GS-LRM bring impressive generation results, they were both pre-printed on arXiv close to the NeurIPS deadline without open-source code, and even **some of the multi-view diffusion models they leveraged is not available to public yet, eg., Instant3D**. Additionally, both GS-LRM, with 300M parameters, and GRM, whose parameters are not disclosed, appear significantly larger than MVGamba with 49M parameters.\n\n >**Q4: What does 'post hoc' operation mean?**\n\n **A4:** **'Post hoc' refers to the merge operation discussed in lines 39-40.** For instance, GS-LRM and LGM increase the number of Gaussians by simply merging from different views. In contrast, our MVGamba can directly generate a sufficiently long Gaussian sequence for all multi-views simultaneously without such operation.\nFurthermore, as discussed in our overall response, recent transformer-based methods such as GRM and GS-LRM involve an upsampler to increase the number of Gaussians *after modeling the patch tokens*. This type of post sequence modeling operation is also not required for MVGamba.\n\n >**Q5: What is the number of output 3D Gaussians? How to increase it?**\n\n**A5:** \n\n- As shown in Figure 1(b) in the rebuttal PDF, the number of predicted Gaussians in MVGamba *equals* the number of output tokens since MVGamba directly decodes  Gaussians from the generated long Gaussian tokens. \n\n- There are several feasible ways to increase number of Gaussians.  (1) Different scan orders: Due to the causal nature of MVGamba, we can apply various scan orders to expand patches, each increasing the number of tokens as well as the number of Gaussians by N. (2) Pixel-aligned Gaussian representation. Following the concurrent Gaussian LRMs, we could leverage upsampler to unpatchify the output tokens into per-pixel Gaussians. We believe this method might be more effective with higher image resolution and more accurate and consistent multi-view inputs.\n\n>**Q6: Larger number of input views.**\n\n**A6:** We have conducted experiments with dense view settings in the sparse view task. The Table below indicates that MVGamba effectively manages 6 input views due to its computational efficiency, achieving a notable improvement in reconstruction performance in the GSO test set. Further studies with denser input views (10 views or more) will be included in the revision.\n\n| Method  | #views |  PSNR↑  | LPIPS↓ |  SSIM↑  |\n|:-------:|:------:|:-------:|:------:|:-------:|\n| MVGamba |   #4    | 26.25   | 0.069  | 0.881   |\n| MVGamba |   #6    | 27.55   | 0.060  | 0.902   |\n\n\n\n> **Q7: Scene-level reconstruction is an interesting future work.**\n\n **A7:** Due to time limitations, we were unable to demonstrate MVGamba's capacity for scene-level reconstruction during the rebuttal period. However, we note that MVGamba's computational efficiency and cross-view self-refinement gives it potentially significant advantages  in scene reconstruction, which require more Gaussians and higher image resolution. Theoretically, with the introduced pixel-aligned Gaussian representation, MVGamba can operate at a resolution of 2560 $\\times$ 1600, compared to  512 $\\times$ 904 for GS-LRM, within a similar overall computational budget. We plan to further explore the Mamba-based reconstructor for scene reconstruction as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kL6xK9F9d", "reviewer": "Reviewer_5iTN", "summary": "This work introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). The Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity. With off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only 0.1x of the model size.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is of good written quality.\n2. The inference speed is relatively fast compared to optimization-based methods.\n3. This work introduces the new architecture, Mamba, into the field of 3D generative models, which is novel.\n4. The computation cost of the training is lower than LGM and LRM.\n5. The method is robust to multiview inconsistency.", "weaknesses": "1. In Fig. 6(b), the performance is continuously increasing as the token length increase. Although the authors' motivation is to lower the computation cost, Fig. 6(b) indicates the the performance grows as the computation cost grows.\n2. No failure cases are shown to demonstrate the limitation of this work.\n3. Fig 2(b) may be wrong. Mamba has linear complexity instead of constant complexity.\n4. The video results in the supplementary materials are only marginally better than LGM's results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and positive comments! Below, we provide a point-by-point response to address your concerns. We welcome further discussion to improve the clarity and effectiveness of our work.\n\n\n> **Q1: The performance is continuously increasing as the token length increase.**\n\n **A1:** The experimental results in Figure 6(b) confirm that performance improves as the token length increases. This is one of our motivation to *directly* process a sufficiently long token length for better performance. Consequently, we advocate for a Mamba-based architecture, which boasts linear complexity, over transformer-based architectures characterized by quadratic complexity to properly **balance** the performance and the computational cost, as illustrated in Table 1 in the rebuttal PDF. Furthermore, our architectural design incorporates various light-weight design elements, as described in the manuscript and detailed in *Appendix* C, to further reduce the computational cost and energy consumption, thereby promoting environmental sustainability in line with the growing emphasis on green AI in the Neurips society.\n\n >**Q2: No failure cases are shown to demonstrate the limitation of this work.**\n\n **A2:**\n\n- **We have illustrated the failure cases in the *Appendix* D.** As shown in Figure 8, MVGamba may sometimes fail if the depth of the front view is estimated incorrectly. Fortunately, this issue can be largely mitigated by manually changing the input order, as the side view typically contains sufficient depth information.\n\n- **Due to limited space in the rebuttal PDF, we couldn't provide additional failure cases discussed in the *Limitations* section**, particularly those caused by severely flawed input views generated by the imperfect multi-view diffusion model. It's worth noting that, compared to other baselines, our results demonstrate greater robustness against inconsistent inputs due to the self-refining nature of MVGamba, which aligns with the experimental results presented in Figure 6(a) of the manuscript. We will incorporate more failure cases and their analysis in the revision as per your advice.\n\n>**Q3: Fig 2(b) may be wrong. Mamba has linear complexity instead of constant complexity.**\n\n **A3:** We apologize for the confusion. We confirm that Figure 2(b) is correct; it indeed illustrates linear complexity. Due to the quadratic complexity of the Transformer, we had to scale the y-axis to prevent the Transformer's curve from extending outside the figure. We will revise this figure to avoid confusion in revision.\n\n> **Q4: The video results in the supplementary materials are only marginally better than LGM's results.**\n\n **A4:** We believe that our proposed MVGamba could consistently outperform LGM in most cases. To address your concerns, we have provided more qualitative results compared to LGM in Figure 3 in the rebuttal PDF, where MVGamba could generate more accurate geometry and fine-grained textures, while LGM exhibits blurred texture and severe multi-view inconsistency, eg., multiple eyes on the dragon's face.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Program Chair, Senior Area Chair, Area Chair, and Reviewers,

We sincerely appreciate the thorough review and insightful feedback provided by each reviewer. The reviewers asked perceptive questions and comments, which are answered in detail in individual responses and have improved our submission. 

**In this shared response, we have attached a rebuttal PDF containing the required additional experiments. We also illustrate the two current sequence modeling paradigms for Gaussian LRMs: Compressed vs. Direct Sequence Modeling (ours) in Figure 1 of the PDF.**

Below, we elaborate on these two paradigms to highlight the unique approach of our proposed paradigm compared to recent transformer-based approaches on arXiv.

+ (a) **Compressed Sequence Modeling:** This approach tokenizes the input into a compact representation, processes it through a series of transformer blocks, and then upsamples to produce the 3DGS parameters. This paradigm is represented by the concurrent GS-LRM and GRM.

+ (b) **Direct Sequence Modeling (ours):** This approach tokenizes and expands the input into a sufficiently long sequence of tokens through cross-scan operations, which are then directly processed by a series of mamba blocks to generate the 3DGS parameters.

We believe that paradigm (b) offers several significant benefits:
+ **Efficient long sequence modeling:** It accommodates larger spatial dimensions to capture and preserve fine-grained details while mitigating information loss typically caused by upsamplers, such as the zero-padding in de-convolution layers. This benefits accurate geometry and texture reconstruction. Table 1 in the rebuttal PDF demonstrates that Mamba's linear computational complexity allows for a favorable balance between computational cost and long-sequence modeling capacity, enabling efficient processing of high-resolution inputs without prohibitive memory requirements.

+ **Easy of optimization:** It directly models the 3DGS sequence without any upsampler, hence establishing a more straightforward relationship between 3DGS parameters (e.g., position, color) and the modeled sequence. This eases the non-convex optimization in inverse graphics scenario, as discussed in pixelSplat, and helps learn accurate geometry and textures.

+ **Cross-view self-refinement:** It efficiently incorporates multi-view information causally from the initial condition onward, thereby enabling the refinement of inconsistent parts based on earlier views and generated tokens.

We would greatly appreciate it if the reviewers could review our responses. We have addressed your concerns in detail and hope our responses have answered your questions. Please let us know at your earliest convenience if you have further questions or concerns.

Meta-review: Reviewers either recommend acceptance or leaning towards acceptance. All the reviewers appreciated the proposed the novel 3D generative model architecture which is memory efficient compared to existing feed-forward techniques with better reconstruction quality. Reviewers raised some concerns related to experiments several of which are addressed in the rebuttal. Reviewers did raise valid concerns and authors are encouraged to do the best of their abilities to address all the concerns in the final version.
Decision: Accept (poster)