MVGamba: Unify 3D Content Generation as State Space Sequence Modeling

Authors: xuanyu  yi
zike  wu
qiuhong  shen
qingshan  xu
pan  zhou
joo-hwee  lim
shuicheng  yan
xinchao  wang
hanwang  zhang
concat  patchify
  conv
mvdream  imagedream
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 11:27:02

Abstract: Recent 3D large reconstruction models (LRMs) can generate high-quality 3D content in sub-seconds by integrating multi-view diffusion models with scalable multi-view reconstructors. Current works further leverage 3D Gaussian Splatting as 3D representation for improved visual quality and rendering efficiency. However, we observe that existing Gaussian reconstruction models often suffer from multi-view inconsistency and blurred textures. We attribute this to the compromise of multi-view information propagation in favor of adopting powerful yet computationally intensive architectures (\eg, Transformers). 
To address this issue, we introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). Our Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity.
With off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only $0.1\times$ of the model size. The codes are available at \url{https://github.com/SkyworkAI/MVGamba}.

Main paper: 
 introduction: 
 We address the challenge of crafting 3D content from a single image, sparse-view images, or text input, which can facilitate a broad range of applications, e.g., Virtual Reality, immersive filming, digital gaming and animation. Previous research on 3D generation has investigated distilling 2D diffusion priors into 3D representations via score distillation sampling (SDS) [1] . Although these optimization-based approaches exhibit strong zero-shot generation capability with high-fidelity rendering quality [2] [3] [4] [5] , they are extremely time-and memory-intensive, often requiring hours to produce a single 3D asset, thus not practical for a real-world scenario.With the advent of large-scale open-world 3D datasets [6] [7] [8] , recent 3D large reconstruction models (LRMs) [9] [10] [11] [12] integrate multi-view diffusion models [13] [14] [15] with scalable multi-view 3D reconstructor to regress a certain 3D representation (e.g. Triplane-NeRF [16, 17] , mesh) in a feed-forward manner. Specifically, current LRMs [18] [19] [20] adopt a one image (or text) → mulit-view images → 3D diagram to predict 3D Gaussian Splatting (3DGS) [21] parameters, thereby ensuring the rendering efficiency while preserving fine details. Given a single image or text prompt, they first generate a set of images using multi-view diffusion models, which are then fed into a multi-view reconstructor (e.g., U-Net [22] or Transformer [23] ), mapping image tokens to 3D Gaussians with superior generation speed and unprecedented quality.However, we observe that existing feed-forward Gaussian reconstruction models typically adopt powerful yet computationally intensive architectures [23, 24] to generate long sequences of Gaussians for intricate 3D modeling. Such approaches inevitably compromise the integrity of multi-view information propagation to manage computational costs. For instance, they use local [18] or mixed [19] attention on limited multi-view image tokens or even deal each view separately and simply merge the predicted Gaussians afterwards [20] . Consequently, the generated 3D models often suffer from multi-view inconsistency and blurred textures, as illustrated in Figure 2 (a). These issues indicate that current compromise strategies fail to translate into coherent, high-quality outputs in practice. This raises a crucial question: How can we preserve the integrity of multi-view information while efficiently generating a sufficiently long sequence of Gaussians?To address this issue, in this paper, we introduce Multi-View Gaussian Mamba (MVGamba), a general and lightweight Gaussian reconstruction model. At its core, MVGamba features a multi-view Gaussian reconstructor based on the recently introduced RNN-like architecture Mamba [25] , which expands the given multi-view images into a long sequence of 3D Gaussian tokens and processes them recurrently in a causal manner. By adopting causal context propagation, our approach efficiently maintains multi-view information integrity and further enables cross-view self-refinement from earlier to current views. Additionally, our Gaussian reconstructor enables the fine-detailed generation of long Gaussian sequences with linear complexity [26, 27] in a single forward process, eliminating the need for any post hoc operations used in previous work.More concretely, we first patchify the multi-view images into N tokens and rearrange them according to a cross-scan order [27, 28] , resulting in 4×N image tokens for selective scanning. These tokens are then processed through a series of Mamba blocks for state space sequence modeling. Subsequently, we feed the output Gaussian sequence into a lightweight Multi-Layer Perceptron (MLP) for channelwise knowledge selection, followed by a set of linear decoders to obtain the Gaussian parameters representing high-quality 3D content (Sec. 3.2) . Compared to previous LRMs [11, 29, 30] , our MVGamba features many computationally efficient components: a single-layer 2D convolution image tokenizer replaces the pre-trained DINO [31] transformer encoder, a lightweight MLP combined with linear decoders replaces the deep MLP decoder, and most importantly, linear complexity Mamba blocks replace quadratic complexity Transformer blocks (Figure 2(b) ). Together, these designs ensure efficient training and inference while achieving higher generation quality (Sec. 4). Moreover, to 3 .directly convert the generated Gaussians into smooth textured polygonal meshes, we alternatively incorporate a 3DGS variant -2DGS [32] -for accurate geometric modeling and mesh extraction.We conducted comprehensive qualitative and quantitative experiments to verify the efficacy of our proposed MVGamba. The experimental results demonstrate that MVGamba (49M parameters) outperforms other latest LRMs [19, 29, 33] and even optimization-based methods [34, 35] on the task of text-to-3D generation, single-view reconstruction and sparse-view reconstruction with roughly only 0.1× of the model size. The contributions and novelties of our paper are summarized as follows:• We point out that directly generating a sufficiently long sequence of Gaussians with full multi-view information is crucial for consistent and fine-detailed 3D generation.• We introduce MVGamba, a novel feed-forward pipeline that incorporates causal context propagation for cross-view self-refinement, allowing the efficient generation of long sequences of 2D/3D Gaussians for high-quality 3D content modeling.• Extensive experiments demonstrate that MVGamba is a potentially general solution for 3D content generation, including text-to-3D, image-to-3D and sparse-view reconstruction task.2 Related Work 3D Generation. Previous approaches for generating high-fidelity 3D models predominantly used SDS-based optimization techniques [1, 36] and their variants [2-5, 34, 37] . These methods yield high-quality 3D generations but require hours for the per-instance optimization process to converge. Pioneered by the large reconstruction model (LRM) [29] , recent works [10, 11, 30, 38] show that image tokens can be directly mapped to 3D representations, typically triplane-NeRF, in a feedforward manner via a scalable transformer-based architecture [23] with large-scale 3D training data [6] [7] [8] . Among them, Instant3D [11] integrates LRM with multi-view image diffusion models [13-15, 39, 40] , using four generated images for better quality. To avoid inefficient volume rendering and limited triplane resolution, some concurrent works [18] [19] [20] follow Instant3D and introduce 3D Gaussian Splatting [21] into sparse-view LRM variants. Specifically, GRM [18] and GS-LRM [20] use pixel-aligned Gaussian with a pure transformer-based reconstruction model, increasing the number of Gaussians through image feature upsampling and per-pixel merge operations.LGM [19] combines the 3D Gaussians from different views using a convolution-based asymmetric U-Net [22] .Our MVGamba, on the other hand, directly processes multi-view conditions causally, recurrently generating a long sequence of Gaussians for coherent and high-fidelity 3D modeling.Mamba model for visual applications. Recent advancements in State Space Models (SSMs) [17, 41, 42] , notably Mamba [25] , have gained prominence in long sequence modeling for harmonizing computational efficiency and model versatility [43] [44] [45] [46] . Following Mamba's progress, there has been a surge in applying this framework to critical vision domains, including generic vision backbones [26, 27, 47, 48] , multi-modal streams [49, 50] , and vertical applications, especially in medical image processing [51] [52] [53] [54] [55] [56] . Specifically, VMamba [27] pioneers a purely Mamba-based backbone to handle intensive prediction tasks. Similarly, Vim [26] leverages bidirectional SSMs for data-dependent global visual context without image-specific biases. Subsequent works progress with advanced selective scanning algorithms [47, 48] , integration with other networks [57, 58] , and adapted structural designs [59, 59, 60] . Concurrently, Gamba [61] marries Mamba with 3DGS for single-view reconstruction with limited texture quality and generalization capacity. In this paper, we explore and demonstrate the efficiency and long-sequence modeling capacity of Mamba in various 3D generation tasks with large-scale pre-training. 
method: 
 In this section, we present our MVGamba, designed to efficiently generate 3D content through a two-stage pipeline. In the first stage, we utilize off-the-shelf multi-view diffusion models, including MVDream [14] and ImageDream [13] , to generate multi-view images based on an input text prompt or a single image. In the second stage, equipped with this multi-view image generator, we introduce an SSM-based multi-view reconstructor to generate Gaussians from multi-view images. Specifically, we first provide a brief overview of 3D Gaussian splatting and its variants (Sec. 3.1). Next, we describe the core architecture of our multi-view Gaussian reconstructor (Sec. 3.2), followed by detailed elaboration of our robust training objectives (Sec. 3.3) . With above large-scale pre-training, we are able to chain these two stages to produce high-fidelity 3D content in seconds (Sec. 3.4).Introduced by [21] , vanilla 3D Gaussian Splatting (3DGS) fits a 3D scene from multi-view images with a collection of 3D Gaussians. Variants of Gaussian Splatting [62, 63] , typically 2DGS, leverage 2D Gaussian primitives instead, excelling in the vanilla version for more accurate geometry reconstruction. Generally, each Gaussian is composed of its 3D center µ ∈ R 3 , 3D scale s ∈ R 3 or 2D scale * s ∈ R 2 , associated color c ∈ R 3 , opacity α ∈ R, and a rotation quaternion q ∈ R 4 . These parameters can be collectively denoted by G, with G i = {µ i , s i , c i , α i , q i } denoting the parameter of the i-th Gaussian. These Gaussians can then be splatted onto the image plane and rendered in real time via the differentiable tiled rasterizer [21, 62] .The core of MVGamba is a feed-forward multi-view Gaussian reconstructor. As depicted in Figure 3 (a), our reconstructor transforms multi-view input images with camera embedding [64] into 3D contents represented by 3D Gaussians [21] or its variants [62, 63] in a feed-forward manner. This reconstructor comprises an SSM-based processor to expand and process multi-view image tokens as Gaussian sequences, and a light-weight Gaussian decoder to predict attributes for each Gaussian.Expanding multi-view images as sequences. Given a posed multi-view image set {v i , π i }, we first densely embed the camera pose π i ∈ R 4×4 for each view v i ∈ R H×W ×3 using Plücker rays [30] , denoted as P i ∈ R H×W ×6 . The pixel values and ray embeddings are concatenated into a 9-channel fused map, which is then tokenized using a non-overlapping convolution with a kernel size of p × p:EQUATIONwhere V i ∈ R h×w×C is the tokenized feature map; h = H/p, w = W/p; C is the embedding dimension. Note that considering the light-weight architectural design, our image tokenizer is much simpler than the pre-trained DINO [31] utilized by previous LRMs, which we empirically find to be redundant for low-level 3D reconstruction. With the tokenized multi-view features, we then adopt a cross-scan order [27] to rearrange them as sequence. Specifically, we scan the image tokens sequentially along four different directions: top-left → bottom-right, bottom-right → top-left, top-right → bottom-left, and bottom-left → top-right, which allows each token to integrate information from all adjacent tokens. This cross-scan rearrangement results in a sequence that is 4× longer:EQUATIONwhere X ∈ R 4N hw×C denotes the expanded Gaussian sequences, N denotes the view number, and P scan denotes the cross-scan operation on each view.Causal sequence modeling with State Space Model. Inspired by [25, 61] , we model the Gaussian sequences via an adapted SSM-based processor. In detail, given the expanded Gaussian sequence X, we first add a learnable positional embedding E element-wise to it and derive the initial Gaussian sequence X 0 . Then, we feed X 0 into L stacked SSM layers for recurrent causal sequence modeling, formulated as:EQUATIONwhere X k denotes Gaussian sequence output by the k-th layer, and SSM k denotes the k-th SSM layer with vanilla Mamba [25] structure; A k , B k and ∆ k are parameters of SSM layer dependent on the input sequence X k-1 . Note that we are modeling Gaussian sequences rather than integrating spatial information as in existing vision Mamba models [26, 27, 47] . Therefore, we adopt 1D convolution instead of 2D convolution in the Mamba blocks, similar to other sequential modeling SSMs [25, 58, 65] . Through state space sequence modeling, we successfully propagate the causal context containing the multi-view information from earlier states to later states with linear complexity. This approach efficiently incorporates multi-view information causally from the initial condition onward, thereby making full use of all Gaussian tokens through cross-view self-refinement. As discussed in Sec. 5, this causal sequential generation of Gaussian tokens provides the model with unprecedented robustness and self-correction abilities, even under inconsistent or noisy input conditions.Decoding causal token sequences into Gaussians. Each token in the processed causal sequence X L is treated as a separate 3D Gaussian token. We first apply a single hidden layer MLP to X L , where the width of the hidden layer is 4C and the output channels revert to C. This process is denoted as Z = MLP(X L ), which aims for channel-wise knowledge selection [24, 66, 67] . We then apply sub-heads to derive each attribute of 3DGS with separate linear projections. Specifically, we predict the position [61] by discretizing the coordinates where position µ i is clamped to [-1, 1] 3 . The scale s i is predicted with a learnable linear projection followed by a softplus activation. The opacity α i is predicted with a linear projection followed by sigmoid activation. Regarding the color attribute c i , we predict the RGB values instead of the spherical harmonics adopted by the original 3DGS [21] , as our reconstructor is mainly trained on synthetic 3D datasets free of light variation. However, unlike other Gaussian attributes, the rotation quaternions are quite sensitive and difficult to predict directly, and hence are often set canonical isotropic and fixed in several recent works [61, 68] . On the other hand, some works [19, 33] predict the rotation without any constraints, but this often causes artifacts and corrupted generations in practice. To address this issue, we design a novel rotation decoder, dubbed RotNet, which balances prediction flexibility and restriction. Our RotNet consists of a set of 32 pre-defined rotation quaternions, denoted as T, which forms a canonical rotation space, and a learnable linear projection matrix Θ to predict the logits of these quaternions. The predicted logits are then transformed into a probability distribution using the Gumbel-Softmax [69] , enabling differentiation through the discrete selection process by adding noise sampled from the Gumbel distribution to the logits p ∈ R 32 before applying the softmax function: p = softmax(ΘZ + g), whereEQUATIONIn this way, we convert the rotation prediction into a 32-class classification task in a fully differentiable way, which allows for direct selection via the argmax operation during inference. We refer to Appendix D for more detailed explanations. These decoded Gaussians are finally passed into the differentiable rasterization pipeline [21] for image-level supervision.Bridging the training-inference gap. In the training phase, multi-view images are collected from the ground-truth blendering of 3D objects, while they are generated by diffusion models during inference. To mitigate such domain gap: (1) Following LGM [19] , we leverage the grid distortion and orbital camera jitter as two data augmentations with a 30% probability to simulate inconsistent pixels and inaccurate camera poses, respectively. (2) We directly use ImageDream [13] as a synthetic data engine to generate multi-view images input and conduct a joint training with the ground-truth renderings from the 3D training dataset. In practice, with a 5% chance, we train MVGamba with synthetic input to mimic the inference pattern for more robust generation results.Overall training objective. During the training phase, we differentiably render the RGB image v i and alpha mask v α i of the N = 4 input views and another six novel views for image-level supervision. Our final objective then comprises four key terms:EQUATIONwhere L MSE and L mask represent the mean square error loss in the RGB image and the alpha mask, respectively; L LPIPS represents the well-adopted VGG-based perceptual loss [70] ; L reg is the opacity L1 regularization loss ∥1 -α i ∥ encourage more efficient use of each Gaussian by enforcing higher density. λ mask , L LPIPS and λ reg are the trade-off coefficients that balance each loss.During inference (Figure 3 (b)), the pre-trained reconstructor can be smoothly combined with any offthe-shelf multi-view diffusion models to efficiently predict a set of Gaussians, which facilitates both text-to-3D and image-to-3D generation. Typically, we leverage ImageDream [13] and MVDream [14] to produce 4 multi-view images with anchored poses [11] from a single image or text prompt, respectively. For mesh extraction, following Huang et al. [62] , we utilize truncated signed distance fusion (TSDF) [71] that fuse the depth maps rendered from the output Gaussians to obtain a smooth polygonal mesh. 
experimental settings: 
 Training dataset. We obtain the multi-view images from Objaverse [7] for MVGamba pre-training. Following [19, 72] , we filtered 80k valid high-quality 3D objects. We then used Blender under uniform lighting to render 25 views of RGBA images with their alpha masks at a resolution of 512 × 512, in the elevation range of 5 • to 30• with rotation {15 • • r|r ∈ [0, 23] , r ∈ N}.To align with the camera configurations in ImageDream [13] and MVDream [14] , at each training step, we select 4 images of a certain object as input views with the same elevations, while rotations separated by 90 • , denoted as {ϕ + 90• • k | k ∈ 0, 1, 2, 3}and another random set of 6 views as supervision.Implementation details. MVGamba is trained on 32 NVIDIA A100 (80G) with batch size 512 for about 2 days. We adopt gradient checkpointing and mixed-precision training with BF16 data type to ensure efficient training and inference. We use the AdamW optimizer with learning rate 1 × 10 -3 and weight decay 0.05, following a linear learning rate warm-up for 15 epochs with cosine decay to 1 × 10 -5 . The output Gaussians are rendered at 512 × 512 resolution for mean square error loss and resized to 256 × 256 for LPIPS loss for memory efficiency. The trade-off coefficients that balancing each loss were set as λ mask = 1.0, λ LPIPS = 0.6 and λ reg = 0.001. We also follow the common practice [19] to clip the gradient with a maximum norm of 1.0. The detail of MVGamba model configuration is included in Appendix D.LGM MVGamba "Child in Furry" "Armor soldier""A pot of flowers" In this section, we compare MVGamba with previous state-of-the-art instant 3D generation methods in image-to-3D, text-to-3D generation, and sparse-view reconstruction tasks. For each task, we first elaborate on the evaluation metrics and baseline methods, then perform an extensive qualitative and quantitative comparison.Single image-to-3D generation. We make comparisons to recent methods, including optimizationbased DreamGaussian [34] , Wonder3D [73] ; feed-forward methods LGM [19] , TripoSR [74] and Triplane-Gaussian [33] and One-2345++ [75] . We adopted the official codebase and pre-trained model weight for all the above methods and we are quite confident that the baselines presented are the finest re-implementations we have come across † . We evaluated the generation quality of MVGamba with a wide range of wild images in Figure 4 . We use well-adopted PSNR, SSIM and LPIPS for quantitative measurement in the GSO [76] dataset following [18] , with a total of 16 test views with equidistant azimuth and -10 ∼ 10 degree elevations. As illustrated in Figure 4 , MVGamba maintains high fidelity and plausible generation in most scenarios. In contrast, Triplane-Gaussian severely suffers from flat and blurred views, which is a notoriously ill-posed challenge, as stated by Instant3D [11] . Moreover, LGM frequently showcases multi-view inconsistency with a transparent surface, which may be attributed to its suboptimal parameter constraints and merge operation. In Text-to-3D generation. Similar to the single image-to-3D task, we compare MVGamba with several optimization-based methods and feed-forward models LGM, TripoSR, and Triplane-Gaussian, using various prompts. For methods that only accept a single image as input, such as TripoSR and Triplane-Gaussian, we use DALL•E 3 [77] to transform the given text into a single image. Note that for more fair comparison, MVGamba and LGM take the same generated multi-view inputs. Figure 4 shows that MVGamba excels at generating plausible geometry and highly realistic texture. Due to its effective ray embedding and multi-view diffusion models, MVGamba is also free from the multi-face problems that frequently occur in optimization-based methods. We further randomly selected 50 prompts from the Dreamfusion [1] gallery and used CLIP Precision and CLIP scores [78] [79] [80] to measure appearance quality and alignment with the given prompt. Our high generation quality is also vividly reflected in Table 1 , where MVGamba consistently ranks the highest among all baselines.Sparse-view reconstruction. Given the same sparse-view inputs, we compare MVGamba with SparseNeuS [81] (trained in One-2-3-45), SparseGS [82] and LGM [19] that are capable of generating 3D Gaussians. We visualize the reconstruction results of MVGamba in Figure 5 and compare against baselines on the GSO [76] dataset with 32 randomly selected views for each object. As shown in Figure 2 and Figure 6 , MVGamba is able to faithfully reproduce high-frequency details with accurate geometric modeling. Due to the limited space, the quantitative evaluation is presented in Appendix C. 
ablation and discussion: 
 Q1: Why MVGamba outperforms other LRMs in most 3D content generation tasks?A1: To better diagnose the progress of MVGamba, we conduct two ablation experiments on G-buffer Objaverse [72] Human subset: one is a counterfactual experiment simulating severe inconsistency during inference to test robustness, and the other on the effect of Gaussian sequence length for sparse-view reconstruction. (1) In Figure 7 (a), we manually perturb one of the four input images with Gaussian noise to simulate the worst-case multi-view input inconsistency caused by the diffusion model. We then feed these manipulated images into the MVGamba reconstructor and merge-operation reconstructor for comparison. As expected, generating 3D Gaussians conditioned on each view separately and simply merging the outputs treats the perturbed input as faithfully as the other views, resulting in inconsistency and even corrupted results. In contrast, our Gaussian reconstructor generates the Gaussian sequence in a causal and self-refining manner, allowing it to mitigate the effects of the perturbation by leveraging propagated multi-view information. This experiment demonstrates that our MVGamba is highly resilient to inconsistencies in multi-view diffusion models used in the first stage due to its causal sequence modeling.(2) We also investigate the effect of Gaussian sequence length by varying the patch size to model different sequence lengths. As shown in Figure 7 (b), our SSM-based Gaussian reconstructor can directly generate extremely long Gaussian sequences, and its performance improves with increasing sequence length. From these two aspects, we can conclude that the higher generation performance of MVGamba is indeed attributable to its self-refineable multi-view modeling and efficient utilization of sufficiently long Gaussian sequences.Q2: What impacts performance of MVGamba in terms of component-wise contributions? Q3: What is the limitation of MVGamba?A3: Honestly, though MVGamba achieves promising results, it still has several limitations. (1) As we model the Gaussian sequence causally, MVGamba can sometimes fail if the depth of the front view is not estimated correctly (Figure 8 top). Fortunately, we empirically find that this limitation can be mitigated by manually changing the input order. For example, using a side-view as the first input allows our model to generate satisfactory 3D content, as the side-view contains sufficient depth information (Figure 8 bottom). In the future, we may explore automatic ways to optimize the input order to enhance robustness. (2) The generation quality of MVGamba is highly dependent on the four input views provided by off-the-shelf multi-view diffusion models [13, 14] . However, current multi-view diffusion models are far from perfect, known to exhibit 3D inconsistencies [9, 19, 83] , and limited to a resolution of 256 × 256. We expect that our model's performance can be seamlessly boosted with the advancements in multi-view diffusion models in future work. 
conclusion: 
 In Table 3 : Theoretically calculated FLOPs comparison between self-attention in Transformer and SSM in Mamba. Here, dimension D = 512; GFLOPS of both modules are calculated according to Equations ( 5) and ( 6) in Vision Mamba [26] .Model  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: While our work unify the 3D content generation, it also raises concerns about potential negative societal effects, including the propagation of disinformation or the amplification of harmful biases with the generated contents.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: We plan to ask user to agree to usage guidelines before use our model after released.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Licenses of dataset and pre-trained models are listed in Appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We conducted an additional experiment using a non-pixel-aligned transformer model for 3DGS prediction. As discussed in Appendix B.1, transformer-based methods typically process a compressed sequence considering the computational budget. We then followed OpenLRM to model 4096 tokens, and adopt a de-convolution layer to up-sample the 3DGS tokens from 4096 to 16384, similar to TGS for intricate 3D modeling. However, as shown in Figure 11 , the non-pixel-aligned transformer yielded inferior performance, with broken geometry and extremely blurred textures after 300 epochs of training. We attribute this to the inherent non-convex optimization challenge in the inverse graphics scenario, as mentioned in pixelSplat [85] , which may be further amplified by the upsampler. This preliminary result may also explain why pixel-aligned approaches are proposed and are adopted by recent transformer-based approaches with upsamplers. On the other hand, our proposed direct sequence modeling paradigm allows for more fine-grained detail modeling and exhibits a cross-view self-refinement ability. Moreover, our direct sequence modeling paradigm can also ease optimization, which provides a new feasible way for training Gaussian LRMs. More qualitative results. We include more qualitative experiment on image-to-3D generation with rendering results in Figure 10 for novel view synthesis and in Figure 12 for normal map generation. We also include a qualitative comparison using samples from the GRM paper in Figure 13 . It indicates that MVGamba captures better geometries and details with significantly smaller model size. Note that we could not directly compare with GS-LRM as it only presents a single-view results in Figure 7 of their paper.More quantitative results. We leave the qualitative comparison on sparse-view reconstruction task here. We compare the well-adpoted PSNR, SSIM, LPIPS for novel view synthesis [86] and Chamfer Distance (CD) and Volume IoU (VIoU) for geometric evaluation. As shown in Table 4 , MVGamba outperforms all baselines across all metrics, even though SparseNeuS require 4 times more input views, while maintaining fast inference and rendering speed. Network Configuration. As detailed in Table 5 , the SSM-based reconstructor comprises 14 Gamba blocks, each with hidden dimensions of 512. The architecture employs RMSNorm, SSM, depth-wise convolution [87] , and residual connections. In line with [24] , positional embedding is added rather than concatenated to the tokenized multi-view image tokens. These tokens have 512 dimensions, resulting in a total of 16,384 or 32,768 tokens, which correspond to the length of Gaussian sequence. The Gaussian Decoder is a compact multi-layer perceptron (MLP) with a single hidden layer containing 2,048 hidden dimensions. This is followed by sub-head linear projection layers, which decode the output into 3D Gaussians for splatting.RotationNet. Our RotationNet (RotNet) is designed to facilitate the prediction of rotation attributes. It comprises a set of 32 pre-defined canonical rotation quaternions and a learnable linear projection matrix that maps input features to the logits of these quaternions. The canonical quaternions include 8 rotations around the principal axes (x, y, z) by 0 or 180 degrees, and 24 rotations by ±45 degrees around various axes formed by combining two principal axes. During training, RotNet uses the Gumbel-Softmax technique to enable differentiable sampling from the discrete set of quaternions. The temperature parameter is gradually decreased from 2 to 0.01 over iterations, encouraging more confident predictions. During inference, RotNet applies the argmax operation to the logits to select the quaternion with the highest probability, operating in a non-differentiable manner.Figure 13 : More comparisons with concurrent state-of-the-art large Gaussian models. Note that the generation results for GRM were extracted from its paper and official project page. A direct comparison with GS-LRM is currently unfeasible, as it is not open-source, and only a single view is presented in Figure 7 of the GS-LRM paper.Figure 14: The pseudo code for Gaussian parameter constraint.Gaussian Parameterization. As 3D Gaussians are an unstructured explicit representation, unlike Triplane-NeRF's structured implicit representation, the parameterization of the output parameters significantly affects the model's convergence. We provide the detailed pseudo code of Gaussian parameterization in Figure 14 for better reproduciblility.Datasets:• Objaverse [7] : ODC-By v1.0 license Pre-trained models: Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research conforms to the NeurIPS Code of Ethics, with social impacts discussed in the checklist.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).† Note that Instant3D[11], GS-LRM[20] and GRM[18] are not included for comparison in the current version, as no code has been publicly released yet. 


Decision: Reviewers either recommend acceptance or leaning towards acceptance. All the reviewers appreciated the proposed the novel 3D generative model architecture which is memory efficient compared to existing feed-forward techniques with better reconstruction quality. Reviewers raised some concerns related to experiments several of which are addressed in the rebuttal. Reviewers did raise valid concerns and authors are encouraged to do the best of their abilities to address all the concerns in the final version.

Reviews: review: paper_summary: paper_summary: The authors propose MVGamba, a general and lightweight Gaussian reconstruction model for unified 3D content generation. By replacing the previous LRM work’s transformer architecture with the recent model Mamba. MVGamba can generate long Gaussian sequences with linear complexity in a single forward process, eliminating the need for post hoc operations. Experiments demonstrate that MVGamba outperforms the current open-sourced LRM work in various 3D generation tasks with a smaller model size( 0.1× ). The authors also claim the proposed model has the potential to be applied to scene-level and 4D generation.
strength_weakness: strength_weakness: 1. The paper’s experiments do not align with its claims regarding the inefficiencies of transformer-based LRMs in handling long-sequence inputs. The primary issue highlighted is that these transformer-based models are not efficient for long-sequence input. However, most experiments in the paper are for single or sparse image input 3D generation. In this context, the Mamba-based architecture does not demonstrate significant speed improvements over transformer-based architectures because the input tokens are relatively short. Table 1 shows no obvious speed enhancement compared to other feedforward methods, supporting this point. Therefore, I question the necessity of Mamba for most tasks discussed in the paper. The Mamba-based reconstruction model appears to be more suitable for dense view reconstruction tasks.

2. The performance of paper doesn’t achieve the state of the art. Although in Table 1, the paper achieves better quantitative results over all baselines, some previous LRMs papers have better performance according to their paper. For example, GS-LRM attains 8 dB higher PSNR than LGM according to its table. 1, while MVGamba is just 2 dB higher than LGM. I can understand this paper didn’t compare with GS LRM because GS LRM doesn’t release its code. I am just not sure if Mamba is important for quality improvement as claimed by the authors, when some transformer-based LRMs can also achieve good (or even better) quality.
questions: questions: - In lines 38-41, the authors state that previous Gaussian-based LRMs fail to achieve multi-view consistent results because they rely on local or mixed attention across limited multi-view image tokens or process each view separately before merging the predicted Gaussians. However, some prior works, like GS LRM, employ self-attention on all input image tokens. Despite merging the Gaussians afterward, they are predicted collectively. It is unclear why this approach leads to multi-view inconsistency.
- In lines 281-286, the authors note that performance improves with increasing sequence length, and they model different sequence lengths by varying the patch size. If decreasing patch size enhances final performance without significantly impacting efficiency, as the authors claim, why didn't they choose a smaller patch size than 14, as indicated in the appendix? 14 is actually bigger than some transformer-based LRM’s patch size. (For example, GS LRM chose 8.)
limitations: limitations: The limitations have been well discussed.
paper_summary: paper_summary: This paper introduces MVGamba, a feed-forward, sparse reconstruction model. This model takes a small number of images (e.g., 4 views) to infer 3D Gaussians. Basically, it is a Mamba version of multi-view large reconstruction model. The authors have implemented several strategies to ensure stable training and optimal performance.

Experimental results demonstrate that the MVGamba model can reconstruct 3D Gaussians with higher quality than the Large Gaussian Model (LGM).
strength_weakness: strength_weakness: (1)	In the abstract and also in Lines 40-41, the authors state that the generated 3D models often suffer from multi-view inconsistency and blurred textures, which is why MVGamba is proposed. However, it seems that this issue is more likely due to whether the architecture is pixel-aligned or not. Therefore, I believe a non-pixel-aligned transformer model can also address these issues. Did the authors try this? How does a non-pixel-aligned transformer model perform?

(2)	I cannot find any tables or numbers in the paper to support the conclusion “0.1× of the model size.” Did the authors forget to include them?

(3)	The reconstruction results look in lower quality than the concurrent GRM and GS-LRM works.
questions: questions: (1)	Line 53: Could it be more specific? What does 'post hoc' operation specifically mean?

(2)	What is the number of output 3D Gaussian primitives? Does it equal N (the number of tokens)? Is there any way to increase the number of Gaussians for better reconstruction quality?

(3)	Given the advantages of Mamba in computational efficiency for long sequences, did the authors try to train an MVGamba with a larger number of views, such as 10 views?
limitations: limitations: The method is limited in object-level reconstruction with clean background. I encourage the authors explore on scene-level reconstruction as an interesting future work.
paper_summary: paper_summary: This work introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor based on the RNN-like State Space Model (SSM). The Gaussian reconstructor propagates causal context containing multi-view information for cross-view self-refinement while generating a long sequence of Gaussians for fine-detail modeling with linear complexity. With off-the-shelf multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Extensive experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only 0.1x of the model size.
strength_weakness: strength_weakness: 1. In Fig. 6(b), the performance is continuously increasing as the token length increase. Although the authors' motivation is to lower the computation cost, Fig. 6(b) indicates the the performance grows as the computation cost grows.
2. No failure cases are shown to demonstrate the limitation of this work.
3. Fig 2(b) may be wrong. Mamba has linear complexity instead of constant complexity.
4. The video results in the supplementary materials are only marginally better than LGM's results.
questions: questions: See the weaknesses.
limitations: limitations: See the weaknesses.

Comments: title: Thanks for your feedback and support!
Thanks for the response and support!
Comment
Thanks for the response and support!
comment: Dear Reviewer XU7f,

Thank you very much for your comments and support! It is highly encouraging to learn that our rebuttal has successfully addressed the majority of your concerns. We would like to take this opportunity to discuss two additional points with you:

**Sequence Length and computational cost**. We emphasize that the Mamba-based reconstructor not only offers a speed advantage over GS-based methods but also, to our knowledge,  currently represents the only experimentally validated direct 3DGS sequence modeling approach capable of processing over 16K tokens, as discussed in our overall response. We also observe that with higher resolution inputs, there shows a substantial computational cost (heavy GPU memory) even for compressed sequence modeling, evidenced by the fact that with 512 resolution inputs, the batch size of GS LRM is 2, which is much smaller compared to 8 or 16 in regular LRM experimental settings, even with several efficient training strategy to save GPU memory, eg., gradient checkpointing, deferred backpropagation and mixed-precision training with BF16.

**Visual Quality**. We believe that MVGamba could achieve comparable visual quality compared to concurrent GRM and GS-LRM with extremely smaller model size. In our future work, we intend to further enhance the reconstruction quality and explore the full potential of MVGamba through several avenues: increasing the model size, integrating more advanced multi-view diffusion models, and incorporating stronger Mamba-based architectures.

We welcome any further concerns or questions you may have and are eager to engage in additional discussions or provide supplementary materials.

Once again, we express our sincere gratitude for your recognition of our work and your invaluable review feedback.

Best regards,

Authors of Submission 2178
Thank you for your rebuttal responses. Most of my concerns have been addressed.

However, the Mamba-based representation did not demonstrate impressive visual quality compared to the SOTA GS-based methods, GRM and GS-LRM. Additionally, under the current sparse input setting, the input token length is not long enough to show a significant speed advantage over the GS-based methods.

As a result, I will be maintaining my current score.
Dear Reviewer eshR,

We sincerely appreciate your  response and support. We are delighted to learn that our rebuttal has successfully addressed your concerns.

Thank you for your time and valuable feedback.

Best regards,

Authors of Submission 2178
Thank you authors! The rebuttal addressed my concerns.
Dear Reviewer eshR,

We wish to convey our sincere appreciation for your insightful and invaluable feedback, which has been of great help to us. As the discussion deadline approaches, we are keenly anticipating any additional comments or suggestions you may have. Ensuring that the rebuttal aligns with your suggestions is of utmost importance. We are deeply grateful for your commitment to the review process and your generous support throughout.

Best regards,

Authors of Submission 2178
Dear Reviewer  XU7f,

We would like to express our heartfelt gratitude for your insightful and invaluable comments, which have been of great help to us. As the discussion deadline approaches, we are eagerly looking forward to receiving your valuable feedback and comments. Ensuring that the rebuttal aligns with your suggestions is of utmost importance.
Thank you again for your dedication to the review process and your generous support.

Best regards,

Authors of Submission 2178
Dear Reviewer 5iTN,

Thank you so much for your response and support. Regarding Figure 2, we observe that as the token length increases from 1024 to 16384, the theoretical FLOPs for transformer self-attention rise dramatically from 2.15 to 292.06, while Mamba's SSM demonstrates a linear increase in FLOPs, scaling from 0.07 to 1.07 as shown in Table 1 of the rebuttal PDF. Plotting these together necessitates scaling the y-axis, which could potentially cause confusion by making Mamba's complexity **appear constant, which indeed is linear**.  Per your suggestion, we will clearly **indicate the y-axis scale** and **specific FLOPs values** to prevent misinterpretation in the revision. 

If you have any further concerns or questions, welcome to discuss with us. We are more than happy to discuss with you further and provide additional materials. 

Once again, thank you for your valuable time and sound advice!

Best regards,

Authors of Submission 2178
Thank you for your rebuttal. My concerns have been partially solved and I keep my score. I hope the Fig 2(b) can be corrected in further revision of this work.