GaussianAnything: Interactive Point Cloud Flow Matching for 3D Generation
Authors: Yushi LAN
Shangchen Zhou
Zhaoyang Lyu
Fangzhou Hong
Shuai Yang
Bo Dai
Xingang Pan
Chen Change Loy
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Object Generation
Gaussian Splatting
Flow-based Generative Models
Primary area: generative models
TL;DR: A novel interactive 3D generation framework for 3D Gaussians.

Abstract: Recent advancements in diffusion models and large-scale datasets have revolutionized image and video generation, with increasing focus on 3D content generation. While existing methods show promise, they face challenges in input formats, latent space structures, and output representations. This paper introduces a novel 3D generation framework that addresses these issues, enabling scalable and high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our approach utilizes a VAE with multi-view posed RGB-D-N renderings as input, features a unique latent space design that preserves 3D shape information, and incorporates a cascaded latent flow-based model for improved shape-texture disentanglement. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single-view image inputs. Experimental results demonstrate superior performance on various datasets, advancing the state-of-the-art in 3D content generation.

Published as a conference paper at ICLR 2025
GAUSSIANANYTHING: INTERACTIVE POINT CLOUD FLOW MATCHING FOR 3D OBJECT GENERATION
Yushi Lan', Shangchen Zhou', Zhaoyang Lyu", Fangzhou Hong!
Shuai Yang". Bo Dai Xingang Pan", Chen Change Loy?
*S-Lab, Nanyang Technological University, Singapore
"Shanghai Artificial Intelligence Laboratory, #WICT, Peking University
"The University of Hong Kong
t github. 1o/2oject////
ABSTRACT
While 3D content generation has advanced significantly, existing methods still face challenges with imput formats, latent space design, and output representa- tions. This paper introduces a novel 3D generation framework that addresses these challenges, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs Variational Autoen- coder (VAE) with multi-view posed RGB- HEB--(epth)-Noormal using unique latent space design that preserves 3D shape information, and in- corporates a cascaded latent flow-based model for improved shape-texture disen- tanglement. The proposed method, GAUSSIANANYTHING supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single image inputs. Notably, the newly proposed latent space naturally enables geometry- texture disentanglement, thus allowing 3D-aware editing. Experimental results demonstrate the effectiveness of our approach on multiple datasets, outperforming existing native 3D methods in both text- and image-conditioned 3D generation.
renderings as input,
INTRODUCTION
3D content generation holds great potential for transforming the virtual reality, film, and gaming industries. Current approaches typically follow one of two paths: either a 2D-lifting method or the design of native 3D diffusion models. While the 2D-lifting approach (Shi al., 2023b Liu et al. 2023b) benefits from leveraging 2D diffusion model priors, it is often hindered by expensive optimization, the Janus problem, and inconsistencies between views. In contrast, native 3D diffusion models (Jun & Nichol 2023 Lan et al. 2024a Zhang et al. (222bb are trained from scratch for 3D generation, offering improved generality, efficiency, and control.
Despite the progress in native 3D diffusion models, several design challenges still persist: (1) Input format to the 3D VAE. Most methods (Zhang et al. 2024b Li et al. 2024) directly adopt point cloud as input. However, it fails to encode the high-frequent
details from textures. Besides, this limits the available training dataset to artist-created 3D assets, which are challenging to collect on a large scale. LN3Diff (Lan et al. 2024a) adopt multi-view images input. Though straightforward, it lacks direct 3D information input and cannot comprehensively encode the given object. (2) 3D latent space structure. Since 3D objects set latent (Zhang et al., 2023 Sajjadi et al.] 2022: Zhang
diverse in geometry, color, and size, most 3D VAE
models adopt the permutation-invariat al. 2024b) to encode incoming 3D objects. Though flexible. this design lacks the image-latent correspondence as in Stable Diffusion VAE (Rombach et al., 2022), where the VAE latent code can directly serve as the proxy for editing input image (Mou et al. 2003h a). Other methods adopt latent tri-plane (Wu et al. 2024 Lan et al. (024a) as the 3D latent representation. However, the latent tri- plane is still unsuitable for interactive 3D editing as changes in one plane may not map to the exact
(3) Choice of 3D oulput representations. Existing solutions either output texture-less SDF (Wu et al. 2024 Zhang et al.) (024b), which requires additional shading model for proceprocessing; volumetric tri-plane (Lan al. [202aa), which struggles with high-resolution rendering due to extensive memory required by volumetric rendering (Mildenhall
part of the objects that need editing.
et al. 2020

---

Published as a conference paper at ICLR 2025
In this study, we propose novel 3D generation framework that resolves the problems above and en- ables scalable, high-quality 3D generation with an interactive Point Cloud-structure Latent space. The resulting method, dubbed GAUSSIANANYTHIN
supports multi-modal conditional 3D gener- ation, including point cloud, caption, and image. Specifically, we propose a 3D VAE that adopts multi-view posed (GGB-D((pth)-N(oomee
renderings the input, which are easy render and con- tain comprehensive 3D attributes corresponding to the input 3D object. The information of each input view is channel-wise concatenated and efficiently encoded with the scene representation trans- former (Sajjadi et al. 2022). yielding a set latent (Zhang et al.) 2023) that compactly encodes the given 3D input. Instead of directly applying it for diffusion learning (Zhang et al.) 2024b] Li et al.) 2024), our novel design concretizes the unordered tokens into the shape of the 3D input. Specif- ically, this achieved by cross-attending (Huang et al.] (024b) the sel latent via a sparse point cloud sampled from the input 3D shape, as visualized in Fig. The resulting point-cloud structured latent space significantly facilitate shape-texture disentanglement and 3D editing. Afterward, a DiT- based 3D decoder (Peebles & Xie, [023] Lan al. (024a) gradually decodes and upsamples the latent point cloud into a set of dense surfel Gaussians (Huang et al. 2024a). which are rasterized to high-resolution renderings supervise 3D VAE training.
After the 3D VAE is trained, we conduct cascaded latent diffusion modeling on the latent space through flow matching (Albergo et al., 2023] Lipman et al.) 2023: Liu et al., (023c) using the DiT (Peelles & Xie (023) framework. To encourage better shape-texture disentanglement, a point cloud diffusion model is first trained carve the overall layout of the input shape. Then, a point- cloud feature diffusion model is cascaded to output the corresponding feature conditioned on the generated point cloud. The generated featured point cloud is then decoded into surfel Gaussians via pre-trained VAE for downstream applications.
In summary, we contribute a comprehensive 3D generation framework with a point cloud structurd 3D latent space. The redesigned 3D VAE efficiently encodes the 3D input into an interactive latent space, which further decoded into high-quality surfel Gaussians. The diffusion models trained on the compressed latent space have shown superior performance in text-conditioned 3D generation and editing, as well as impressive image-conditioned 3D generation on general real world data.
2 RELATED WORK
3D Generation via 2D Diffusion Models. The success of 2D diffusion models (Song et al.
1021
Ho et al. 2020) has inspired their application to 3D generation. Score distillation sampling (ooole jet al., 2022 Wang et al. 2023) distills 3D from a 2D diffusion model, but faces challenges like expensive optimization, mode collapse, and the Janus problem. More recent methods propose learn- ing the 3D via a two-stage pipeline: multi-view images generation (Sh et al. 2023b Long et al.,
2024 Shi et al. 2023a) and feed-forward 3D reconstruction and editing (Hong et al. 2024b Xu et al. 2024a] Tang et al. 2024, Xu et al.) 2024b, Chen et al. 2024b). However, their performance bounded by the multi view generation results, which usually violate view consistency (Liu et al.)
2023b) and fail to scale up to higher resolution (Shi et al. (202a)). Moreover, this two-stage pipeline limits the 3D editing capability due to the lack of 3D-aware latent space.
Native 3D Diffusion Models. Native 3D diffusion models (Zhang et al., 2023 Zeng et al. 2022
Zhang et al. 22024b Lan et al. [2024a] Li al. 2024) are recently proposed achieve high-quality, efficient and scalable 3D generation. A native 3D diffusion pipeline involves a two-stage training process: encoding 3D objects into the VAE latent space (Kingma & Welling 2013 Kosiorek al.) 2021], and latent diffusion model on the corresponding latent codes. Though straightforward. ex- isting methods differ in VAE input formats, latent space structure and output 3D representations. While most methods adopt point alone as the VAE input (Zhang et al. 2023 2024b; Li et al.) 2024), our proposed method encodes a hybrid 3D information through convolutional encoder. Moreover, comparing the latent set (Zhang al. 2023 Sajjadi et al., (022) representation,
proposed method adopts a point cloud-structure latent space, which can be directly used for interactive 3D editing. Besides, rather than producing textureless SDF, our method directly decodes the 3D la- tent codes into high-quality surfel Gaussians (Huang et al. 2024a), which can be directly used for efficient rendering.
Point-based Shape Representation and Rendering. The proliferation of 3D scanners and RGB- D cameras makes the capture and processing of 3D point clouds commonplace (Gross & Pfister
2

---

Published as a conference paper at ICLR 2025
2011). In the era of deep learning, learning- based methods emerging for point set process- ing QQ et al. 2016 Zhao et al. 2021), up-sampling (Yu et al.
2018), shape representation Genova
et al. 2020; Lan et al. 2024b), and rendering (Phister et al.
2000 Yifan et al. 2019 Lassner &
Zoolhöfer 2021; et al. 2022 Kerbl et al., 2023). Moreover, given its affinity for modern network
architectures (Huang al. 2024bb Zhao et al. 2021), more explicit nature against other 3D repre- sentations (Chan et al. 2022 Mildenhall et al. 2020 Müller et al. (2022), efficient rendering (Kerbl ct al., 2023), and even high-quality surface modeling (Huang et al. (024a), point-based 3D repre- sentations are rapidly developing towards the canonical 3D representation for learning 3D shapes. Thus, we choose (featured) point cloud as the representation for the 3D VAE latent space, and 2D Gaussians (Huang et al. 2024a) as the output 3D representations.
Feed-forward 3D Reconstruction and View Synthesis. To bypass the per-scene optimization of NeRF, researchers have proposed learning prior model through image-based rendering (Wang) al. 2021 Yu et al. 202T). However, these methods are primarily designed for view synthe-
sis and lack explicit 3D representations. Sajjadi et al. (2022 2023) propose Scene representation transformer (SRT) to process RGB images with Vision Transformer (Dosovitskiy et al., 202T) and infers set-latent scene representation. Though benefiting from the flexible design, its geometry- free paradigm also fails to generate explicit 3D outputs. Recently, LRM-line of work (Hong et al. 2024b [Tang al. [2024] Wang al. (024) have proposed Teed-forward framework for generalized monocular reconstruction. However, they are still regression- bassed models and lack the latent space designed for generative modeling and 3D editing. Besides, they are limited 3D reconstruction only and fail to support other modalities.
GAUSSIAVANYTHING
This section introduces our native 3D flow-based model, which learns 3D-aware flow-based prior over the novel point-cloud structured latent space through dedicated 3D) VAE. The goal of training is to learn
1. An encoder Eg that maps a set of posed RGB-D-N images R. = RRi,... Ry correspond- ing to the given 3D object a point-cloud structured latent z [zz + zn.];
2. A conditional cascaded transformer denoiser e (Zi,ez Z2 L, c) o E (Zu,t, c) to denoise the noisy latent code z given the time step and condition prompt c;
3. A decoder D (including Transformer Df and a cascaded attention-bass Upsampler
Dú:) to map Zo to the surfel Gaussian ς corresponding to the input object. Moreover, our attention-based decoding of dense surfel Gaussian also provides novel way for efficient Gaussian prediction
Beyond the advantages shared by existing 3D LDM (hhng et al. 2024b Lan et al. 2024a), our design offers flexible point-cloud structured latent space and enables interactive 3D editing.
In the following subsections, we first discuss the proposed 3D VAE with a detailed framework design in Sec 3.1 Based on that, we introduce the cascaded conditional 3D flow-based model stage
in Sec. 3.2 The method overview is shown in Fig.
3.1 POINT-CLOUD STRUCTURED 3D VAE
Unlike image and video, the 3D domain is un-uniform and represented differently for different purposes. Thus, how to encode 3D objects into the latent space for flow-based model learning plays a crucial role in the 3D generation performance. This challenge is two-fold: what 3D representations to encode, and what network architecture process the input.
Versatile 3D Input. Instead of using dense point cloud (Zhang et al. 2024b Li et al. 2024), we adopt multi-view posed RGB- RGB-D(epth)-Normaa
images as input, which encode the 3D input more comprehensively and can be efficiently processed by well-established network architectures (aajadii let al.) 2022; Wu et al.) 2023a) in flexible manner. Specifically, the input is a set of multi-view renderings R of a 3D object, where each rendering within the set R = (I, A, N, TT) contains thor- ough 3D attributes that depict the underlying 3D object from the given viewpoint: the rendered RGB
3

---

Published as a conference paper at ICLR 2025
FPS Sampling
Grabke
Bijaige guarpper
Spirs Port Goud
B
Kalliann
Sateet
Coad Latext
Opsanplin
3D VIE Decoder D.
Figure 1: Pipeline of the 3D VAE of GAUSSIANANYTHING In the 3D latent space learning stage, our proposed 3D VAE &g encodes Viviews of posed RGB-D(epth)- -N(omaa)))) renderings R. into point-cloud structured latent space. This is achieved by first processing the multi-view inputs into the un-structured set latent, which further projected onto the 3D manifold through cross attention block, yielding the point-cloud structured latent code z. The structured 3D latent is further decoded by a 3D-aware DiT transformer, giving the coarse Gaussian prediction. For high-quality rendering. the base Gaussian is further up-sampled by series of cascaded upsampler Dj, towards a dense Gaussian for high-resolution rasterization. The 3D VAE training objective detailed in Eq. (5)
#xxx and the corresponding image I E #xxxx depth map A R HxW normal map N & camera pose
To unify these 3D attributes in the same format, we further process the camera into Plücker co- ordinates (Sitzmann et al. (2021) P; (o x du.v. du,o) E R', where E IR3 is the camera origin, du,; E I is the normalized ray direction, and x denotes the cross product. Thus, the Plücker embedding of a given camera æ can be expressed as E ##xWxG, Besides, following MCC (Wu al.. 2023a)), use unproject the depth map into their 3D positions X E RRHWXx x3 The resulting information is channel-wise concatenated, giving R [T x R H.W (3++3+3++-11))
Transformer-basee 3D Encoding. Given the 3D renderings R, encoding them into a 3D latent space remains a significant challenge. Independently processing each input rendering R with exist- ing network architecture (Wu et al. 2023a Dosovitskiy al. 2021) overlooks the information from other views, leading to 3D) inconsistency and content drift across views (Liu et al.) 20236).
Existing multi-view generation alleviates this issue by injecting 3D attention (Shi et al. 2023b Tang et al. 2024 Shi et al. 2023a) into the U-Net architecture. Inspired by its effectiveness, here directly adopt Scene Representation Transformer (SRT)-like encoder (Sajjadi al. 2022 2023) to process the multi-view inputs. which fully adopts 3D attention transformer block for the 3D representation learning. Specifically, the encoder first down-samples the multi-view inputs via shared CNN backbone, and then processes the aggregated multi-view tokens through the transformer encoder (Dosovitskiy et al. 2021):
2_ = eTX (EONN ({R))))
(I)
where z is the set latent corresponding to the 3D input. This can be seen as the full-attention version of the existing 3D attention- attention-augmented augmented architecture. The resulting latent codes zu fully capture the the input. Compared
existing work that adopts point clouds only as input (Zhang et al.) 2024b; Li et al. 2024), our proposed solution supports more 3D properties input flexible way. In addition, attention operations can be well optimized in modern GPU architecture (Dao et al. 2022 Dao 2024).
intact 3D information corresponding
Point Cloud-structured Latent Space. Though z, fully captures the given 3D input, it is not ideal to serve as a latent space for our task due to the following limitations: 1) The latent space is cumbersome to perform flow matching. Specifically, z, has shape of V x (H/f) x (W/f) x C. where V is the number of input views, H. W is the input resolution and f is the down-sampling factor of the CNN backbone. Given V = 8, f = 8, and = W = 512, the resulting latent codes will have the shape of 32768 x C. This latent space incurs high computation cost for multi-view attention (Shi et al. 2023b). 2) The multi-view features Z₁ are not native 3D representations and naturally suffer from view inconsistency (Liu et al. (003b) even with enough compute available (Shi et al. 2023a). 3) Since z is an un-structured ser (Lee et al. 2019) 3D latent space (Zhang et al. 2024b), it also sacrifices explicit, editable latent space (oou et al. 2023a) for flexibility.
2023
4

---

Published as a conference paper at ICLR 2025
Here, resolve these issues by proposing point cloud-structured latent space. Specifically, project the un-structured features 2 onto the sparse 3D point cloud of the input 3D shape through the cross attention layer:
//1- == (rossAttn(PEEzz.. Zu. 2z);
(2)
where (tons Q, K, V) denotes a cross attention block with query Q, key K, and value V. Ze E R3*N is a sparse point cloud sampled from the surface of the 3D input with Farthest Point Sampling (FPS) (Qi et al. 2017). and PE denotes positional embedding (Tancik et al., (220). Intuitively, we define a read cross attention block (Huang et al.) (0244) that cross attends information from un- structured representation z; into the point-cloud structured feature Nn E CCNNN with Cs < C.
In this way, we obtain the point-cloud structured latent code z = Z2 1 ZN| E R (3) for flow
matching.
High-quality 3D Gaussian Decoding. Given the point cloud-structured latent codes, how to decode them into high-quality 3D representation for supervision remains challenging. Though dense point cloud (Huang et al. 2024b) is straightforward solution, it fails to depict high-quality 3D structure with limited point quantity. Here, we resort to surfel Gaussian (Huang et al. 2024a). an augmented point-based 3D representation that supports high-fidelity 3D surface modeling and efficient render- ing. Specifically, our decoder first decodes the input through the 3D-DiT blocks (Peebles Xie)
2023 Lan et al. 2024a), which has shown superior performance against traditional transformer layer:
z = Dr- (MLP(()))
where MLP layer first projects the input latent to the corresponding dimension, and Dr is the DiT transformer. Since dense Gaussians are preferred for high-quality splatting (Kerbl et al. (003)), we gradually upsample the latent features through transformer blocks. Specifically, given learnable embedding z, E R where fu is the up- up-sampling ratio, we prepend it to each token in the latent sequence. Then, H layers of transformer blocks are used to model the upsampling process:
(k+1) = D (lz, + 2,1). z
(4)
where Dj is transformer block for predicting the h th levels of details (LoD) Gaussian as shown in Fig. and (k+1) E TR are the upsampled set of tokens. The overall tokens z (f++1) E
R(A KNTXC after up-sampling are used to predict the 13-dim attributes of surfel Gaussians.
To achieve denser Gaussians prediction, we cascade the upsampling transformer defined in Eq. 4 for K times, giving the final Upsampler Dy for high-quality Gaussian output. Note that our solution outputs a set of Gaussians that are uniformly distributed on the 3D object surface with near 100%
Gaussian utilization ratio. Existing pixel-aligned Gaussian prediction models (Tang et al., 2024 Yinghao et al.) 2024; Szymanowicz et al.] (202)), however, usually waste 50% Gaussians due to view overlaps and empty background color. Besides, our intermediate Gaussians output naturally serves as LoDs (Takikawa et al., 202I), which can be used in different scenarios to balance the rendering speed and quality.
Training, Our 3D VAE model is end-to-end optimized across both input views and randomly chosen views, minimizing image reconstruction objectives between the splatting renderings and ground- truth renderings. Besides image reconstruction loss, we also impose loss over geometry regulariza- tions, KL constraints, and adversarial loss:
C(Φ xb) = Crenter + Eyo + AuICI. dDAN
(S)
where Crender is mixture of the C and VGG loss (Zhang et al. 2018). Lyo improves geometry reconstruction (Huang et al. 2024a). LKI. is the KL-reg loss (Kinmma & Welling 2013 Rombach et al. 2022) to regularize structured latent space, and LGAN improves perceptual fidelity. All loss terms except LxL are applied over randomly chosen LoD in each iteration, and the Crender is applied to both input-view and randomly sampled novel-view images. For details of geometry loss Lgoo, please refer to Sec. Aec.
3.2 CASCADED 3D GENERATION WITH FLOW MATCHING
After training the point-cloud structured 3D VAE, we get a dataset of D shapes paired with condition vectors (e.g., caption or images), ((4, ve])) where the shape is represented by latent code
5

---

Published as a conference paper at ICLR 2025
XN
XN
Text
Stale
Conditions
PIN
PRN
Point Cloud Diffassio cG
Seals Shift
Scoile stott
CLIP
BM6 Noma
RMS Norm
Image
Toxt
Kisa
Arsention
Ood ge-leon))
Mattillea
Image
2t.8i
Scale
DING
Norm
Manit-liwal
Texture Diffusion c
Seale,
Pr.Pi
Encoder
Croos Atteation (wr (iCllmm))
Norm
MLP
MLP (stored)
[exred)
Zei P
Time
Time
(6) DIT Block (Text condition)
(b) DIT Block (Image condition)
(c) Two-slage Cascaded Diflusion
Figure 2: Diffusion training of GAUSSIANANYTHING
Based on the point-cloud structure 3D VAE, we perform cascaded 3D diffusion learning given text (a) and image (b) conditions. We adopt DiT architecture with Adngne (Chen et al. (023) and QK-Norm (Dehghani et al. [2023] Esser al. 2021). For both condition modality, we send in the conditional feature with cross attention block, but at different positions. The 3D generation is achieved in two stages (c), where point cloud diffusion model first generates the 3D layout Zg. and texture diffusion model further generates
the corresponding point-cloud features Z4.0- The generated latent code Zu is decoded into the final 3D object with the pre-trained VAE decoder.
through the 3D VAE aforementioned. Our goal is to train a flo-matching generative model to learn flow-based prior on top of Below we present how we adapt flow-based models to our case.
Cascaded Flow Matching over Symmetric Data. As detailed in Sec. A.3 flow matching involves training a neural network ee to predict the velocity of the noisy input z, with the straight- tra- jectory. After training, Eg can sample from a standard Normal prior N(0, I) by solving the reverse our case, the training data point the point-cloud structured latent code z Z Z I x3+ which is symmetric and permutation invariant (Zeng
ODE/SDE (Krrra et al. 2022).
et al. 2022 Nichol et al.
2022). Based on this property, we opt for diffusion transformer (Peebles)
Xie 2023 without positional encoding as the ce parameterization.
Here, rather than modeling and Z/: jointly, we empirically found that cascaded framework (Ho
ct al. 2021 [yu et al. 2024 2023 Schröppel et al. (024)) leads better performance. Specifically, conditioned sparse point cloud generative model ég is first trained to generate the overall structure of the given object:
C = B (M (2 -cl3]]
(6)
and point cloud feature generative model eb is cascaded leam the corresponding KL- regularized feature conditioned on the sparse point cloud:
ch (2o) - - [wi [M. Xil (2,, c) c|1
The detailed cascading process is detailed Fig. 2 (c). Our proposed design enables better geometry- texture disentanglement and facilitates 3D editing over specific shape propertics. For derivations of the flow matching training objective, please refer Sec. A.3 for more details.
Conditioning Mechanism. Compared to LRM (Hong et al. 2024b Tang et al. 2024) line of work which intrinsically relies image(s) the input, native flow-based method enables more flexi- ble 3D generation from diverse conditions. As shown in Fig. 2 (a,b), for the eoxt-conditioned model, adopt CLIIP (Radford et al. 2021) extract penaltimate tokens the condition embeddings; and

---

Published as a conference paper at ICLR 2025
Single-iew image 3D
Multi-siow Image 3D
Native 3D Diflusion Model
Imput
Open
Splatter Image One-2:3345
LGM
LIGEDU
Ours
Figure 3: Qualitative Comparison of Image-to-3D. We showease the novel view 3D reconstruc- tion of all methods given single image from unseen GSO dataset. Our proposed method achieves consistently stable performance across all cases. Note that though feed-forward 3D reconstruction methods achieve sharper texture reconstruction, these method fail to yield intact 3D) predictions un- der challenging cases (e.g.. the rhino row 2). In contrast, our proposed native 3D diffusion model achieve consistently better performance. Better zoom in.
for the image conditioned model, features. All conditions are injected into the DiT architecture through a pre-norm (Xiong et al, 2020) cross-attention block following the common practice of Zhang al. (2024b). All the models are trained with Classifier-free Guidance (CFG) Ho 2021 by randomly dropping the conditions with probability of 10%.
DINOv2 (Oquab al. 2023
extract global and patch
To cascade two flow-based models, we encode the output of stage-1 model e with PE(Zz) as in Eq. (2). and add to the first-layer features of e This guarantees that generated features are paired with the input sparse point cloud structure.
4 EXPERIMENTS
Datasets. To train our 3D VAE, we use the renderings provided by G-Objaverse (Qiu et al. 2023 Deitke et al. 2023b) and choose high-quality subset with around 176K 3D instances, where each consists of 40 random views with RGB, normal, depth map and camera pose. For text-conditioned diffusion training. we the caption provided by Cap3D (Luo al. 2023 2024) and 31Topia (Hog] jet al. (2024a). For image-conditioned training, we randomly select an image in the dataset of the corresponding 3D instance as the condition.
Implementation Details. For 3D VAE, we choose V = 8 views of RGB-D-N renderings as input to guarantee thorough coverage of the 3D object. The CNN Encoder is implemented with similar architecture as LDM VAE (Rombach et al.) 2022 with a down-sampling factor of ʃ f= 8, and the multi-view transformer has five layers as in RUST (Sajjadi al., 2023). The sparse point cloud z has a size of N 3 where N = 768, and the corresponding featured point cloud 2n, has dimension of x 10. For upsampling blocks, we employ K blocks with f 8, ft. 4, and f! 3, giving 73, 768 Gaussians in total. All transformer blocks follow a nre-norm design (Xiong et al.) 2020). During 3D VAE training, the model supervised by randomly chosen LoD renderings, with Au 2e - 6, Àa 1000, An 0.2, and AGAN 0.I. We adopt batch size 64 with both input and random novel views for training. During the conditional flow-ban bosed model training stage, we adopt batch size 256. All models efficiently and stably trained with le on 8 AxA100 GPUs for 1M iterations with BF16 and FlashAttention (Dao 2024) enabled. We use CFG=4 and 250 ODE
steps for all sampling results.
7

---

Published as a conference paper at ICLR 2025
Quantitative evaluation of image-conditioned 3D generation. Here, quality of both 2D rendering and 3D shapes is evaluated. As shown below, the proposed method demonstrates strong performance across all metrics. Although multi-view images-to-3D approaches like LGM achieves better performance the FID/KID metrics, they fall short more advanced image quality assessment metrics such as CLIP-I and MUSIQ. Besides, they perform significantly worse in 3D
Table 2:
shape quality. For nie 3D baselines, also include the number of input views (V=#).
Method
CLIP-It
FIDL
KID(%))
MUSIQI
P-FIDI
P-KID(S
COV(SM
MMD(%
OpenLRM
86.37
38.41
1.7
45.46
35.74
12.60
39.33
29.08
Splatier-mmae
84.10
48.80
3.65
30.33
19.72
7.03
37.66
30.69
One-2-3-45 (V=12)
80.72
88.39
6.34
59.02
72.40
30.83
33.33
35.09
CRM (V=6)
85.76
45.53
1.93
64.10
35.21
13.19
38.83
28.91
Lara (V=4)
84.64
43.74
1.95
39.37
32.37
12.44
39.33
28.84
LGM (V=4)
87.99
19.93
0.55
54.78
40.17
19.45
50.83
22.06
Shape-E
77.05
138.53
11.95
31.51
20.98
7.41
61,33
19.17
LN3Diff
87.24
29.08
0.89
50.39
27.17
10.02
55.17
19.94
Ours
89.06
24.21
0.76
65.17
8.72
3.22
59.50
15.48
4.1 METRICS AND BASELINES
Evaluating Image-to-3D Generation. We evaluate GAUSSIANANYTHNG on both image and text conditioned generation. Regarding image-conditioned 3D generation methods, we compare the proposed method with three lines of methods: single-image to 3D methods: OpenLRM (He &
Wang 2023 Hong et al., 2024b), Splatter Image (Szymanowicz et al. 2023). multi-view images to
3D methods: One-2 2-3-45
Liu et al. (002a)). CRM (Wang et al. (024). Lara (Chen al. 2024a).
LGM (Tang et al. 2024). and native 3D diffusion models: LN3Diff-image (Lan et al. 2024a).
Quantitatively. benchmark rendering metrics with CLIP-I Radford et al. (2021). FID (Heusel ct al. 2017). KID (Binkowski et al., 2018), and MUSIQ-konie Ke et al. 2021 Zhou et al. 2022).
For 3D quality metrics, benchmark Point cloud FID (P-FID), Point cloud KID (P-KID), Coverage
Score (COV), and Minimum Matching Distance (MMD). Following previous works Nichol et al.)
2022
Zhang al. (2023): Yariv et al. (2024), we adopt the pre-trained PointNet++ provided by
Point- E Nichol et al. (0222 for calculating P-FID and K-FID. Qualitatively, GSO (Downs et al.,
2022
Zheng & Vedaldi 2023) dataset is used for visually inspecting image-conditioned generation.
Evaluating Text-to-3D Generation. Re-
Quantitative Evaluation on Text-to-3D. The proposed method outperforms competitive alter- natives on both CLIP scores and aesthetic scores.
Table 1:
garding text-conditioned 3D generation methods, we compare against Point- (Nichol et al.) (0022, Shape-E (Jun) & Nichol, 2023). 3DTopia (Hong et al. 2024a), and IN3Diff-text (Lan et al. 2024a). CLIP score (Radford et al. 2021) is reported following the previous works (Lan et al. 2024a; Hong et al. 2024a), with aesthetic scores MUSIQ-
Method
ViT-B/321
ViT-L/141
MUSIQ-AVA
Q-Align
Point-E
26.35
21.40
4.08
1.21
Shape-E
27.84
25.84
3.69
1.56
LN3Din
29.12
27,80
4.16
2.22
3DTopia
30.10
28.11
3.31
1.42
Ours
31.80
29.38
4.99
3.13
AVA (Ke et (021) and Q-Align (Wu al.
2023b) also included.
4.2 EVALUATION
In this section, we evaluate our proposed method over image-to-3D generation, text-to-3D gener- ation, and 3D-aware editing. Please check the appendix for more visual results and point cloud- conditioned 3D generation. Image-to-3D Generation. Our proposed framework enables 3D gener- ation given single-view image conditions, leveraging the architecture detailed in Fig. ( Follow- ing current method (Chen et al. 22024a] Tang et al. 2024), we qualitatively benchmark our method in Fig. Jover the single-view 3D reconstruction task on the unseen images from the GSO dataset. Our proposed framework is robust to inputs with complicated structures (row 1,3,4) and self-occlusion (row 2,5), yielding consistently intact 3D reconstruction. Besides, shows a more natural back-view reconstruction, as opposed to regression- based methods that are commonly blurry uncertain areas.
senerative-based method
8

---

Published as a conference paper at ICLR 2025
Quantitatively, we showcase the evaluation in Tab. 2 As be seen, our proposed method achieves state-of-the-aie performance over CLIP-I and all 3D metrics, with competitive results over conven- tional 2D rendering metrics FID/KID. Note that LGM leverages pre-trained MVDream (Shi et al. 2023b) as the first-staee generation, and then maps the generated views to pixel-aligned 3D Gaus- sians. This cascaded pipeline achieves better visual quality, but prone yield distorted 3D geometry, as visualized in Fig.
Text-to-DD Generation. We demonstrate the text-to-3D generation performance in Fig. and Tab. D The flow-based model trained on GAUSSIANANYTIING latent space has demonstrated high-quality text-to-3D generation of generie 3D objects, yielding superior performance terms of object structure, textures, and surface normals. Quantitatively, our proposed method achieves better text-3D alignment against competitive baselines.
3D-aware Editing. Compared to existing methods that use unstructured tokens for 3D diffusion learning (Jun Nichol 2023), our proposed point-cloud structured latent space naturally facilitates geometry-texture disentanglement and allows for interactive 3D editing. As visualized in Fig, [ given the text-conditioned generated point cloud Zo by E we sample the final 3D objects with é with different random seed. As can be seen, the generated 3D objects maintain a consistent structure layout while yielding diverse textures. Besides, by directly manipulating the conditioned point cloud z O. our proposed method enables interactive 3D editing, as in 2D models (Pan et al. 2023 Mou et al. 2023b). This functionality greatly facilitates the 3D content creation process for artists and opens up new possibilities for 3D editing with diffusion models.
4.3 ABLATION STUDY AND ANALYSIS
Table 3: Ablation of 3D VAE Design. We ablate the design of our 3D VAE. Input-side, leveraging multi-view RGB-D-N renderings shows superior performance against dense point cloud. Besides, adding Gaussian up-sampling modules leads to consistent performance gain.
Table 4: Gaussian Utilization Ratio. We com- pare the effective Gaussians (opacity > 0.005) used during splatting here. Pixel-aligned Gaus- sian prediction methods waste a large portion of Gaussians when representing 3D object due to white background and multi-view overlap, while our proposed Gaussian predictions yields more compact reconstruction results.
Design
LPIPS @ 100K
Dense PCD as Input Multi-view RGB-D Input
0.174
0.163
Method
High-opacity Gaussians (%)
Normal Map
0.157
Splatter Image
17.14
Gaussian SR Module
0.095
LGM
52.63
x Gaussian SR Module
0.067
Ours
96.84
3D VAE Design. In Tab, benchmark each component of our 3D VAE architecture over a subset of Obiaverse with 50K instances and record the LPIPS at 100K iterations. As shown Tab. our
pazzllzze dog.
An 18 century canioon.
Topla
Shap
Hannburger.
Figure 4: Qualitative Comparison of Text-to-DD. We present text-conditioned 3D objects gener- ated by GAUSSIANNNYTHINGG displaying two views of each sample. The top section compares our results with baseline methods, while the bottom shows additional samples from our method along with their geometry maps. Our approach consistently yields better quality in terms of geometry, texture, and text-3D alignment.

---

Published as a conference paper at ICLR 2025
hydrant.
Zy
beacon.
Stage-1 Kyn generation
Slage-2 S0,0 generation
(Texture Geometry Disentanglement)
Interactive Editing
Figure 5: 3D editing. Given two text prompts, we generate the corresponding point cloud Zo.a with stage-1 diffusion model with Eg, and the corresponding point cloud features Zo,h can be further generated with e As can be seen, the samples from stage-2 are consistent in overall 3D structures but with diverse textures. Thanks to the proposed Point Cloud-structured Latent space, our method supports interactive 3D structure editing. This is achieved by first modifying the stage-1 point cloud Z0,r > z and then regenerate the 3D object with the same Gaussian noise.
popelized
Input
Editing
Latent- Eeiting (Ours)
(5] No rascaded text-to-30 generation
(b) Editing on the point cloud latent space
Figure 6: Qualitative ablation of Cascaded diffusion and latent space editing, We first show the effectiveness of our two-stage cascaded diffusion framework in (a). Compared Fig. the single- stage 3D diffusion yields worse texture details and 3D structure intactness. In (b), we disjoint the hydrant cover to demonstrate that our latent point cloud editing yields smoother and more reasonable results, while direct editing on 3D Gaussians shows tearing artifacts.
input design performs better against dense (16, 384) colored point cloud (Zhang et al. 2024b), and the reconstruction quality consistently improves by including normal map input and cascading more Gaussian upsampling blocks.
Besides, showcase high Gaussian utilization ratio of our proposed method. Specifically, we calculate the ratio of Gaussians with an opacity greater than 0.005 as effective Gaussians, they contribute well to the final rendering. We calculate the statisties over 50K 3D instances. As shown in Tab. 4 our proposed Gaussian prediction framework achieves much higher utilization ratio. On the contrary, pixel-aligned Gaussian prediction models waste noticeable portion of Gaussians on the overlapping views and white backgrounds.
Gaussian Utilization Ratio.
Effectiveness of Cascaded 3D Diffusion. We qualitatively ablate the cascaded model design in Fig. (a). where single text-conditioned DIT is trained to synthesize the 3D point cloud and features jointly. Clearly, the jointly trained model has worse texture with 3D shape artifacts to our cascaded design. Besides bringing better editing capability shown in Fig. 5 our cascaded design enables more flexible training, where the models of two stages can be trained in parallel.
3D Editing on the 3D Latent Space. Finally, we ablate the 3D editing performance in Fig. 6 (b). As can be seen, direct editing on the final Gaussians leads to 3D artifacts, while editing on our 3D latent space yields more holistic and cleaner results since suitable features are re-generated after editing. Besides, our method enables easy editing on the sparse point cloud, compared to directly manipulating dense 3D Gaussians (Dong al. 2024
CONCLUSION AND DISCUSSIONS
In this work, present paradigm of 3D generative model by learning the diffusion model over a interactive 3D latent space. A dedicated 3D variational autoencoder encodes multi-view 3D attributes renderings into point-cloud structured latent space, where multi-modal diffusion learning can be efficiently performed. Our framework achieves superior performance over both text- and image-conditioned 3D generation, and potentially facilitates numerous downstream applications in 3D vision and graphics tasks. Please check the appendix for the discussion of limitations.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement. This study is supported under the RIE2020 Industry Alignment Fund In- dustry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribu- tions from the industry partner(s). It is also supported by Singapore MOE AcRF Tier 2 (MOE- T2F:20221-0011 1) and National Research Foundation, Singapore, under its NRF Fellowship Award NRF-NRFFF16-2024-0005
REFERENCES
Michael Albergo, Nicholas Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint (.XXi:23033087977, 2023.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-ruuulla, and Pratul P. Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR, 2018.
Eric Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. CVPR, 2022.
Anpei Chen, Haofei Xu, Stefano Esposito, Siyu Tang, and Andreas Geiger. Lara: Efficient large- baseline radiance fields. In ECCV, 2024a.
Honghua Chen, Yushi Lan, Yongwei Chen, Yifan Zhou, and Xingang Pan. Mvdrag3d: Drag- based creative 3d editing via multi-view remerationn-rcoosstuuttton priors. arXiv preprint arXiv:2410. 16272, 2024b.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, Guosheng Lin, and Chi Zhang. Meshanything: Artist-created mesh generation with autoregressive transformers, 2024c.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Nießnner Scannet: Richly-annotated
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In ICLR, 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttentin:: Fast and memory-efficient exact attention with IO-awareness. In NeurIPS, 2022.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner. Mathilde Caron, Robert Geirhos. Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.0566. 2023a.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In CVPR, 2023b.
11

---

Published as a conference paper at ICLR 2025
Shaocong Dong, Lihe Ding, Zhanpeng Huang, Zibin Wang, Tianfan Xue, and Dan Xu. Interac- tive3d: Create what you want by interactive 3d generation. arXiv preprint arXiv:2404.16510, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3D shape. In CVPR, 2020.
Markus Gross and Hanspeter Pfister. Point-based graphics. Elsevier, 2011.
Zexin He and Tengfei Wang. OpenLRM: Open-source large reconstruction models. https:// github.comm ctt.:/Tom//3/mocoocoo a/OpenLRM 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge local nash equilibrium. NeurIPS, 2017.
Jonathan Ho. Classifier-free diffusion guidance. In NeurIPS, 2021.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106. 15282, 2021.
Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang. Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint arXiv:2403.022344 02234, 2024a.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2024b.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In SIGGRAPH 2024 Conference Papers. Association for Computing Machinery, 2024a.
Zixuan Huang, Justin Johnson, Shoubhik Debnath, James M Rehg, and Chao- Yuan Wu. Pointinfin- ity: Resolution-invariann point diffusion models. In CVPR, 2024b.
Heewoo Jun and Alex Nichol. Shap-E: Generating conditional 3D implicit functions. arXiv preprint arXiv:2305.02463 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
Junjie Ke, Qifei Wang, Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In ICCV, pp. 5148-5157, 2021.
12

---

Published as a conference paper at ICLR 2025
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 12(4))1-14, 2023.
Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. In Neurips, 2023.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv, 2013.
Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sovna
Mokr'a, a, and Danilo Jimenez Rezende. NeRF-VAE: geometry aware 3D scene generative model. ICML, 2021.
Yushi Lan, Fangzhou Hong, Shuai Yang, Shangchen Zhou, Xuyi Meng, Bo Dai, Xingang Pan, and Chen Change Loy. LN3Diff: Scalable latent neural fields diffusion for speedy 3D generation. In FCCV, 2024a.
Yushi Lan, Feitong Tan, Di Qiu, Qiangeng Xu, Kyle Genova, Zeng Huang, Sean Fanello, Rohit Pandey, Thomas Funkhouser, Chen Change Loy, and Yinda Zhang. Gaussian3diff: 3d gaussian diffusion for 3d full head synthesis and editing. In ECCV, 2024b.
Christoph Lassner and Michael Zollhöfer. Pulsar: Efficient sphere- neural rendering. CVPR, 2021.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based cenmuuttaioonnivariaan neural networks. In ICML,, 2019.
Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, and Xiaoxiao Long, Crafts- Man: High-fidelity mesh generation with 3D native generation and interactive geometry refiner, 2024.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In WACV, 2023.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. ICLR, 2023.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, al. One-2-3-45: Any single image to 3D mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306.16921 16928, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image 3D object, 2023b.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023c.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, al. Wonder3D: Single image to 3D using cross-domain diffusion. In CVPR, 2024.
Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3D captioning with pre- trained models. arXiv preprini arXiv:2306.07279, 2023.
Tiange Luo, Justin Johnson, and Honglak Lee. View selection ranking. arXiv preprint 00f::2244407944, 2024.
captioning via diffusion
Zhaoyang Lyu, Jinyi Wang, Yuwei An, Ya Zhang, Dahua Lin, and Bo Dai. Controllable mesh generation through sparse latent point diffusion models. In CVPR, 2023.
Zhaoyang Lyu, Ben Fei, Jinyi Wang, Xudong Xu, Ya Zhang, Weidong Yang, and Bo Dai. Getmesh: A controllable model for high-quality mesh generation and manipulation, 2024.
13

---

Published as a conference paper at ICLR 2025
Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden- and Saining Xie. Sit: Exploring flow and diffusion-bars based generative models with scalable interpolant transformers. arXiv, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Nicolas Moenne-Loccoz, Ashkan Mirzaei, Or Perel, Riccardo de Lutio, Janick Martinez Esturo, Gavriel State, Sanja Fidler. Nicholas Sharp, and Zan Gojeic. 3d gaussian ray tracing: Fast tracing of particle scenes. ACM Transactions on Graphics and SIGGRAPH Asia, 2024.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Diffeditor: Boosting accu- racy and flexibility on diffusion-based image editing. arXiv preprint arXiv:2402.02588, 2023a.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiftusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421, 2023b.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. 7OG, 112:4::02:11102:::: July 2022.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-B: A system for generating 3D point clouds from complex prompts, 2022.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision, 2023.
Xingang Pan, Ayush Tewari, Thomas Leimküüher, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag Your GAN: Interactive point-based manipulation on the generative image mani- fold. In SIGGRAPH, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, and Markus Gross. Surfels: Surface elements as rendering primitives. In PACMCGIT, 2000.
Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using diffusion. ICLR, 2022.
Charles Qi. Hao Su. Kaichun Mo, and Leonidas Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. arXiv, 2016.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: feature learning on point sets in a metric space. In NeurlPS, 2017.
Deep hierarchical
Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: generalizable normal-depth diffusion model for detail richness in text-to-3d. arXiv preprint arXiv:2311.16918 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. ICMI., 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Shunsuke Saito. Zeng Huang, Ryota Natsume. Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, October 2019.
14

---

Published as a conference paper at ICLR 2025
Mchdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry-free novel view synthesis through set-latent scene representations. CVPR, 2022.
Mehdi S. M. Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučié, and Klaus Greff. RUST: Latent Neural Scene Representations from Unposed Imagery. CVPR, 2023.
Philipp Schröppel, Christopher Wewer, Jan Eric Lenssen, Eddy 1lg, and Thomas Brox. Neural point cloud diffusion for disentangled 3d shape and appearance generation. In CVPR, 2024.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model. In arvivv 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffuuion for 3D generation. arXhv:2308. /6512, 2023b.
Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder- only transformers. arXiv preprint arXiv:2311.15473 2023.
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Du- rand. Light field networks: Neural scene representations with single-evaluation rendering. In NeurlPS, 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Stanislaw Szymanowicz. Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultraffss single-view 3D reconstruction. In arXiv, 2023.
Towaki Takikawa, Joey Litalien, Kangxue Yin. Karsten Kreis, Charles Loop, Derek Nowmuzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In CVPR, 2021.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. In NeurlPS, 2020.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In ECCV, 2024.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brunt Brualla, Noah Snavely, and Thomas A. Funkhouser. IBRNet: Learning Multi-View Image-Based Rendering. In CVPR, 2021.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. In NeurIPS, 2023.
Zhengyi Wang. Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu. Chongxuan Li, Hang Su, and Jun Zhu. CRM: Single image 3D) textured mesh with convolutional reconstruction model. In ECCV, 2024.
Chao-Yuaa Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3D reconstruction. arXiv preprint arXiv:2301.088 2023a.
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Chunyi Li, Liang Liao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guangtai Zhai, and Weisi Lin. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17099 2023b. Equal Contribution by Wu, Haoning and Zhang, Zicheng. Project Lead by Wu, Haoning. Corresponding Authors: Zhai, Guangtai and I Lin, Weisi.
15

---

Published as a conference paper at ICLR 2025
Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng. Jingxi Xu, Philipp Torr, Xun Cao, and Yao Yao. Direct3D: Scalable image-to-30 generation via 3D latent diffusion transformer, 2024.
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:24/2.01506, 2024.
Ruibin Xiong, Yunchang Yang, Di Hc, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architec- ture, 2020.
Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang. Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from single image with sparse-view large reconstruction models. arXiv preprint 19Xi4:2404.07191, 2024a.
Jiale Xu, Shenghua Gao, and Ying Shan. Freesplatter: Pose-free gaussian splatting for sparse-view 3d reconstruction. arXiv preprint arXiv:2412.09577 2024b.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neu- mann. Point-NeRF: Point- based neural radiance fields. In CVPR, 2022.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. DMV3D: Denoising multi-view diffusion using 3D large reconstruction model. In ICLR, 2024c.
Haitao Yang, Yuan Dong, Hanwen Jiang. Dejia Xu, Georgios Pavlakos, and Qixing Huang. Atlas gaussians diffusion for 3d generation. In ICLRR, 2025.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. CVPR, 2020.
Lior Yariv, Omri Puny, Natalia Neverova, Oran Gafni, and Yaron Lipman. Mosaic-sdf for 3d gener- ative models. In CVPR, 2024.
Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine- Hormnn. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (proceed- ings of ACM SIGGRAPH ASIA), 38(6), 2019.
Xu Yinghao, Shi Zifan, Yifan Wang, Chen Hansheng. Yang Ceyuan, Peng Sida, Shen Yujun, and Wetzstein Gordon. GRM: Large gaussian reconstruction model for efficient 3d reconstruction and generation, 2024.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. PixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-net: Point cloud
upsampling network. In CVPR, 2018.
Xianggang Yu, Mutian Xu. Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. MVIngNet: large-scale dataset of multi-view images. In CVPR, 2023.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3D shape generation. In NeurIPS, 2022.
Biao Zhang, Jiapeng Tang, Matthias Nießner, and Peter Wonka. 3DShape2VecSet: A 3d shape representation for neural fields and generative diffusion models. ACM Trans. Graph., 42(4), jul 2023. ISSN 0730-0301. doi: 10. 145.39442224
Biao Zhang, Jing Ren, and Peter Wonka. Geometry distributions. arXiv preprint arXiw:2411.16076 2024a.
Longwen Zhang, Ziyu Wang. Qixuan Zhang, Qiwei Qiu, Angi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. CLAY: A controllable large-scale generative model for creating high-quality 3D assets. ACM Transactions on Graphics, 2024b.
16

---

Published as a conference paper at ICLR 2025
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In CVPR, 2021.
Chuanxia Zheng and Andrea Vedaldi. Free3d: Consistent novel view synthesis without 3d represen- tation. arXiv, 2023.
Shangchen Zhou, Kelvin C.K. Chan, Chongyi Li, and Chen Change Loy. Towards robust blind face restoration with codebook lookup transformer. In NeurIPS, 2022.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
A IMPLEMENTATION DETAILS
A.1 TRAINING DETAILS
VAE Architecture. For the convolutional encoder Eg, adopt lighter version of LDM Rom- bach et al. (2022) encoder with channel 64 and 1 residual blocks for efficiency. When training on Objaverse with V 8, incorporate 3D-aware attention Shi et al. (2023b) in the middle layer of the convolutional encoder. The multi- transformer architecture is similar to RUST (Sajaadi ct al. 2023 2022). For each upsampler Dj we have 2 transformer blocks in the middle. All hyper- parameters remain at their default settings. Regarding the transformer decoder Dy, we employ the DiT-B/2 architecture due to VRAM constraints. Compared to LN3Diff (L.an al. 2024a), do not adopt cross-plane attention in the transformer decoder.
Diffusion Model. We mainly adopt the diffusion training pipeline implementation from SiT Ma et al. (2024), with pred-u objective, GVP schedule, and uniform / sampling. ODE solver with 250 steps used for all the results shown in the paper. For the DiT architecture with cross attention and zero N-rrr design, we mainly refer to PixArt Chen et al. (2023). The diffusion transformer built with 24 layers with 16 heads and 1024 hidden dimension, which result in 458M parameters. For all the diffusion models, we further add the global token to /. features as part of the condition input.
single-adaLN aaale--daL-zzzeee
Details of Geometry Loss in VAE Training, The geometry loss Lgo is composed of two regular- ization terms, including the depth distortion loss to concentrate the weight distribution along rays, inspired by Mip-NeRF (Barron et al. 2021 2022). Given a ray of pixel, the distortion loss is defined
as
La wAwy ld; dz1:
where w; = ar (I G (u(x))) the blending weight of the i-th intersection and di is the depth of the intersection points. Besides, as surfel Gaussians explicitly model the primitive normals, we encourage the splats' normal to locally approximate the actual object surface:
Cn = Σ 1o(11NNTN
(9)
where Ñ is the predicted normal maps. The final geometry loss is given by Cgm XaLo AnCn.
A.2 DATA AND BASELINE COMPARISON
Training Data. For Objaverse, use high-quality subset from the pre- precereeesed rendering from G-buffer Objaverse Qiu et al. (2023) for experiments. Since G-buffer Objaverse splits the subset into 10 general categories,
all the 3D instances except from "Poor-quality": Human-Shape, Animals, Daily-Used, Furniture, Buildings&Outdorr, Transportations, Plants, Food and Electronics. The ground truth camera pose, rendered multi-view images, normal, depth maps, and camera poses are used for stage-1 VAE training.
Details about Baselines. We use the official released code and checkpoint for all the comparisons shown in the paper. For the evaluation on the GSO dataset, we use the rendering provided by Free3D (Zheng Vedaldi 2023).
Evaluation details. For quantitative benchmark in Tab. 2 we use 600 instances from Objaverse with ground truth 3D mesh for evaluation. To calculate the visual metrics (DID/KID/MUSIO), we use the first rendered instance as the image condition and render 24 images with fixed elevation (+15 degrees) with uniform azimuths trajectory (24 x 15 degrees) with radius= 1.8. For 3D metrics, we export the extracted 3D mesh and sample 4096 points using FPS sampling on the mesh surface. The ground truth surface point cloud is processed in the same way. The pre-trained PointNet++ model from Point-E is used for P-FID and K-KID evaluation. All generated 3D models are aligned into the same canonical space before 3D metrics calculation. All intermediate results the baselines for evaluation will be released.
18

---

Published as a conference paper at ICLR 2025
A.3 MORE PRELIMINARIES
2D Gaussian Splatting (2DGS). Since 3DGS (Kerbl et al. 2023) models the entire angular radiance in blob, it fails reconstruct high-quality object surfaces. To resolve this issue, Huang et al. (2024a) proposed 2DGS (surfel-based GS) that simplifies the 3-dimensional modeling by adopting "flat" 2D Gaussians embedded in 3D space, which enables better alignment with thin surfaces.
Notation-wise, the 2D splat is characterized by its central point Pk. two principal tangential vectors ty and t.. and scaling vector S (s.: sp) that controls the variances the 2D Gaussian. Notice that the primitive normal is defined by two orthogonal tangential vectors tw tu X tz. Thus, the
2D Gaussian is parameterized with
P(u, u) Pk + sytau + Sot H u, U, 1, T)
(10)
where H
8 tu 8gtv
Pk
RS
Ps 1
(11)
0
0
Where H parameterizes the local 2D Gaussian geometry. For the point (u, v) in ue space, its 2D Gaussian value can then be evaluated by standard Gaussian ((u) exp
مياي" and the center Pk. scaling (su, Se), and the rotation (tu, t.) are all learnable parameters. Following 3DGS Kerbl jet al. (202)), each 2D Gaussian primitive has opacity a and view-dependent appearance c, and can be rasterized via volumetric alpha blending:
i-1 II a, (u(x))) j=1
c(x) = C.
(12)
where the integration process is terminated when the accumulated opacity reaches saturation. During optimization, pruning and densification operations are iteratively applied.
Flow Matching and Diffusion Model. Diffusion models create data from noise (Song et al. 2021 and are trained to invert forward paths of data towards random noise. The forward path is constructed as 2, = agTo + bic, where ~ (0, 7) Gg and b, are hyper parameters. The choice of forward process has proven have important implications for the backward process of data sampling (Lin et al. 2023).
Recently, flow matching (Liu et al. 2023c Albergo 2023 Lipman et al. 2023) has introduced
a particular choice for the forward path, which has better theoretical properties and has been verified
on the large-scale study (Esser et al. 2024). Given a unified diffusion objective (Karras al. 2022):
(xo) = - B.-ert)). [w:X]] (Z t) - -e|
(13)
2
where X; = log denotes nignall-to-niis ratio, and X, denotes its derivative. By setting wi = r-r with z, = 1)am le, flow matching defines the forward process straight path between the data distribution and the Normal distribution. The network 6g directly predicts the velocity rie and please check the following section for more detailed derivation.
Derivation of the Training Objective of Flow Matching. Since three works (Liu et al. 2023c
Albergo al. 2023 Lipman et al. 2023) proposed the flow matching idea simultaneously,
adopt the unified formulation defined in Esser et al. (2044) in Eq. 6 and Eq. [ Here we brief the background of conditional flow matching, and please read the Sec.2 of Esser et al. (2024) for depth analysis.
Specifically, consider the forward diffusion process (Ho et al. 2020
ao + bic, where I).
(14)
To express the relationship between 24, To. and e, we define the mappings 1 and un as:
< e) To + a,z0 + bie, " ()) vi l))
(15)
(16)
where 3 and vig are the inverse and derivative of v/, respectively.
19

---

Published as a conference paper at ICLR 2025
Since can be viewed solution to the ODE
2/2 = (e e), with initial condition 20 20.
(17)
the conditional vector field (, <) generates the conditional probability path pel: c).
Remarkably, one can construct a marginal vector field u, that generates the marginal probability paths Pr (Lipman et al. 2023), using the conditional vector fields ue(: Ie):
u4 (2) = (e- ue(2 (c) Pc(2 E) Pz(2)
(18)
The marginal vector field uy can be learned by minimizing the Conditional Flow Matching objective:
CCRM = pep (1e () - ux(z [)]l
(19)
To make this objective explicit, we substitute:
Vi( E) a/20 + t/E,
(20)
3 (= le) bic
(21)
at
into the expression for uz(z e):
dt
0/
1/ be )
2 we(2 c)
cbi
(22)
a₁
at
Next, consider the signal-to-noise ratio d = log oi (e e) can be rewritten as:
D..
With X
2(
1 the expression for
d/ b
de (e
X/e.
(23)
at
2
Using this reparameterization, the LCFM objective can be reformulated as a noise-prediction objec- tive:
(=( ot + dt X/6 2 (-bxx] ||ee(z,)) -
LCFM = pt.l((
(24)
2
Expr((()))p()))
(25)
where we define:
-2 60 == Xbb.
(ve
(26)
a+
Since the optimal solution remains invariant to time-dependent weighting, one can derive various weighted loss functions that guide optimization towards the desired solution. For unified analysis of different approaches, including classic diffusion formulations, we express the objective as Kingma
& Gao 2203
(xo) E Ee Wr [e)(2,,))
1
=ll3),
(27)
where w; = 1 XP corresponds to with used in Eq. and Eq.
B DISCUSSIONS OF LIMITATIONS
We acknowledge that the texture quality of our proposed method still inferior to the state-of- the-art multi-view based 3D generative models, i.e.. LGM. Besides, the visual quality of the text- conditioned 3D generation of native 3D generation methods is worse compared to SDS-based alter- natives, despite being much faster and shows better diversity. We believe our method has made step forward towards bridging the gap. To further improve the performance, we list some potential directions in the following:
20

---

Published as a conference paper at ICLR 2025
1. Enhancing the 3D VAE Quality.. The performance of the 3D VAE could be improved by increasing the number of latent points and incorporating a pixel-alignen 3D reconstruction paradigm, such as PiFU (Saito al. 2019), to achieve finer-grained geometry and texture alignment.
2. Incorporating Additional Losses Diffusion Training, Currently, the diffusion training relies solely latent-space flow matching. Prior work, such as DMV3D (Xu et al. (202cc), demonstrates that incorporating rendering loss can significantly enhance the synthesis of high-quality 3D textures. Adding reconstruction supervision during diffusion training is another promising avenue to improve output fidelity.
3. Leveraging 2D Pre-training Priors. At present, the models are trained exclusively on 3D datasets and do not utilize 2D pre- pre-training priors as effectively multi-vies (MV)- based 3D generative models. A potential improvement is to incorporate 2D priors more effectively, for instance, by using multi-view synthesized images as conditioning during training instead of single-view images.
4. Expanding Dataset Diversity, Utilizing more diverse and extensive 3D datasets, such Objaverse-XL XL (Deitke et al., (023a) and MVImageNet YYu et al. 2023). could further enhance the quality and generalizability of 3D generation.
5. Support of Physics-based Rendering. Currently, our proposed method leverages 2DGS the underlying 3D representation since it balances the 2D rendering and 3D surface qual- ity. However, incorporating more advanced 3D representations such as 3DGRT (Moenne- Loccoz al., 2024 can further support physics-based ray tracing over 3D Gaussians. Be- sides, this can be achieved by directly fine-tuning our pre-trained 3D VAE decoder only with the PBR rendering pipeline.
By addressing these aspects, the proposed method could achieve significant advancements in both the quality and versatility of its 3D generation capabilities.
C DISCUSSIONS OF CONCURRENT WORK
After the submission of this paper, several related works on 3D generation have been released. We discuss them here.
FreeSplatter (Xu et 2024b) extends the multi-view generation and 3D reconstruction pipeline for 3D object generation, eliminating the need for input poses. It also supports 3D scene generation by training on the scene-level datasets Yao et al., 2020 Dai et al.] 2017). While following a differ- ent paradigm, method could benefit from its prse-free design enable image-conditioned 3D generation from multiple input views.
Geometric Distribution ZZhang al. 2024a) introduces novel 3D representation by learning point cloud diffusion model for single-instance objects. This approach closely resembles our stage- diffusion model but focuses on encoding geometric information. While shows strong potential for downstream tasks, its efficiency remains challenge for broader applications.
AtlasGaussians (Yang et al. 2025), like our method, proposes a native 3D diffusion model over 3D Gaussians. It employs point clouds as input for 3D VAE compression and utilizes local patch fea- tures for 3D Gaussian decoding. However, it lacks an explicit 3D latent space, making unsuitable for interactive 3D editing.
Trellis (Xiang et al. 2024) adopts a similar cascaded 3D flow matching framework and achieves remarkable results in object generation. Unlike our method, which employs point cloud latent space, Trellis leverages a sparse voxel structure with efficient operators for high-quality and fast 3D generation. It also supports interactive 3D editing via its explicit latent space. We hope this design, along with our proposed method, can establish a canonical paradigm for future 3D native generative models.
D MORE VISUAL RESULTS AND VIDEOS
Please also check our supplementary video demo and attached folders for video results.
21

---

Published as a conference paper at ICLR 2025
Overview of the Qualitative Performance. In Fig. include overview of the qualitative
Astrunant Shic and Holmee
pooeteed anp.
Cload
Surtel Gausstans (Stage
Surtel Gausstans Centers
Lataut Point Editing
Strfel Gouonians Generation
Lebent Point Editing
Gaussians Gerterationn
duice with rnd woutt
Input
Initial Resulta
Inturactive 3D Biitting (3)
Interactive 3D Editing
Figure 7:
Our method generates high-quality and editable surfel Gaussians through cascaded diffusion pipeline, given single-view images texts as the conditions.
performance of the proposed method, GAUSSTANANYTHING. Here, we show the single-view con- ditioned 3D generation, text-conditioned 3D generation, and 3D-aware editing capabilities.
3D VAE Reconstruction. In Fig. we include the 3D VAE reconstruction results of our model at level of details (LoD). Thanks to the versatile multi-view 3D attributes input and transformer design, our 3D VAE enables high-quality 3D reconstruction with visually attractive textures and smooth surface. The encoded point cloud-structured latent codes, z, serves as a compact proxy for efficient 3D diffusion training. Besides, the 2D Gaussians Upsampler naturally facilitates LoD and facilitates speed/ quality trade off in practice.
More Text-to-DD results. In Fig. 9 present additional qualitative comparisons of text-to-3I) generation with GAUSSIANANYTHING For this evaluation, we use relatively complex captions input conditions and display two random samples generated by our model. As shown, GAUS- SIANANYTHING produces visually appealing results that characterized rich textures, smooth sur- face, and notable diversity. To further demonstrate the generality of proposed method, in Fig TO we include the uncurated text-to-3D results over DF-415 (Poole et al. 2022) prompts with captions and more detailed descriptions.
Point-to-3D Generation with Cascaded Point-E. Moreover, the cascaded design of our stage-2 diffusion model, </ enables flexible 3D generation given point clouds from diverse sources. To demonstrate this capability, we integrate the output of state-of-the-art 3D point cloud generative model. such as Point-E (Nichol et al. 2022). into the GAUSSIANANYTHING generation pipeline. Specifically, we first generate the point cloud using Point-E based on caption condition This generated point cloud then used as input z, to our stage-2 point coou//ext-coodiiionn model e As shown in Fig. [1 the generated surfel Gaussians exhibit significantly improved texture quality and geometry fidelity compared to the Point-E point cloud outputs. This capability broadens the applicability of method, enabling benefit from recent advances in point cloud
diffusion
22

---

Published as a conference paper at ICLR 2025
=
LoD=2
LoD =3
Figure 8: 3D VAE Reconstruction. Here, we visualize the 3D VAE reconstruction performance different level of details (LoD). As shown, higher LoD results in sharper textures and smoother surface. Better zoom in.
generation (Huang et al. 2024b) and mesh generation (Siddiqui et al. 2023 Chen et al. producing high-quality levect-level surfel Gaussians.
2024c for
Broader Social Impact. this paper, we introduce a new latent 3D diffusion model designed produce high-quality surfel Gaussians using single model. As a result, our approach has the potential to be applied to generate DeepFakes or deceptive 3D assets, facilitating the creation of falsified images videos. This raises concerns, individuals could exploit such technology with malicious intent, aiming to spread misinformation or tarnish reputations.
23

---

Published as a conference paper at ICLR 2025
with sword
handle.
cute and friendly pinke teddy bear with sitting pose.
Low poly blue chess piece
Low poly tree model with green
Low poly model of a green pine tree, resembling a Christmas tree.
White Giraffe
Caption
Point-E
Shape-E
3DTopia
(seed=1)
(seed=2)
Figure 9: More Qualitative Comparison of Text-to-3D. We present more text-conditioned 3D objects generated by GAUSSIANANYTHIN alongside comparisons with competitive alternatives, including Point-E, Shape-F,, and 3DTopia. As demonstrated, our approach consistently achieves superior quality in geometry, texture, and alignment between text and 3D content.
delicious tace.
A red eyed tree frog, low poly.
zoomed out photo pile of dice on a green tabletop.
photo of pita bread full of falafe! and vegetables.
An amigurumi bulidozer.
An orchid flower planted clay pot.
Figure 10: More Qualitative Results of Text-to-3D over DF-415 Captions, Our proposed method generalizes to long captions with detailed descriptions. All results are uncurated.
24

---

Published as a conference paper at ICLR 2025
and friendly pinke teddy bear with sitting
Low-poly tree model with green leaves.
Caption
Point-E
Point-E + GA (seed=1)
Point-E + GA (seed=2)
Figure 11: Cascaded Text-to-3D with Point-E. Thanks our cascaded 3D generation design, the stage-2 diffusion model in GAUSSIANANYTHING seamlessly integrates with other point cloud gen- erative models. To illustrate this capability, we leverage the state-of-the-art Point-E model. As demonstrated, our stage-2 diffusion model effectively transforms the point clouds generated by Point-E into diverse surfel Gaussians with more visually appealing features and enhanced geometric details.
25


Reviews:
[{"review_id": "Ru8sPhR6iH", "reviewer": "Reviewer_cogY", "summary": "This manuscript introduces a point cloud-based latent diffusion model for 3D Gaussian splatting generation. The proposed pipeline consists of a point cloud VAE and a cascaded diffusion model. The VAE encodes multiview RGBDN into a downsampled features, and employs cross attention between points sampled from the 3D surface and the features to aggregate the information into the points, yielding the featured points as the set latent. A transformer decoder is adopted to upsample the latents into 2DGS representation. The DiT based diffusion model generates first the position of the points, and then the features conditioned on the positions. Experiments have demonstrated its performance on text/image-to-3D generation and 3D editing.", "questions": "- I find L183 confusing. Flow matching models typically adopt the velocity parameterization $v=\\epsilon - x_0$. Is the network predicting $\\epsilon$ or $v$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- First of all, I think it's quite impressive to me that a 3D-native diffusion model can be trained from scratch to achieve the presented results, although my knowledge may be outdated since I'm not an expert in point cloud-based 3D generation. Many SOTAs (e.g. 3DTopia) gain advantage using 2D diffusion priors from large pretrained models, so I'm impressed that the proposed model can compete against some of those method.\n\n- Great writing overall. The network architectures are very clearly presented. \n\n- Geometry/texture disentanglement and the ability to edit the results in 3D add to the versatility of the proposed approach.", "weaknesses": "- I would disagree with the statements in L244-249. Why would a latent of shape $V\\times (H/f)\\times (W/f)\\times C$ be difficult to generate for diffusion models? Isn't this what most multi-view image diffusion models are doing? My understanding is that, such high dimensional latents have more capacity and requires a larger model and a larger dataset to unleash its potential. Choosing a more compressed latent design is reasonable for this work since the model is trained from scratch, but it may not be the optimal choice when scaling up.\n\n- Experiments are underwhelming in general, both quantitatively and qualitatively. \n  - Quantitatively, the only comparisons made are on text-to-3D with CLIP being the only metric. The details of the experiment setup, such as the number of objects tested and the source of the captions, are not specified at all. Image-to-3D and editing abilities are not evaluated quantitatively, so the real potential of its versatility is not well supported. \n  - Qualitatively, the few text-to-3D examples shown are not enough to demonstrate its generalization. The caption also appears very simple, unlike many previous approaches that adopt the more complex DreamFusion captions. Image-to-3D qualitative results are not satisfactory, given the missing divider of the first file sorter object and the slightly distorted shape of the final tower object. The competitors are also relatively weak. Many previous works (e.g. One2345++) can handle these simple objects easily. While it's understandable that many works are not open source, making fair comparison difficult, I find the performance of the proposed approach underwhelming in general.\n\n- Using Dino features for image conditioning may introduce a major limitation for image-to-3D generation due to its large patch size and thus reduced resolution. Moreover, exact image-3D alignment gets more challenging in this case since the set latent doesn't really spatially align with any view. For these reasons, I'm doubtful about whether the model could produce faithful 3D results to the image. The first example in \nFig. 3 shows that the generated file sorter has 4 dividers while the input has 5, indicating the flaw in image-to-3D generation.\n\n- The conditioning mechanism is not really unique at all. Any diffusion model can use modulated normalization and cross attention for text/image conditioning. In L363, the claim that the LRM line of work can only handle image condition is definitely wrong (Instant3D for text-to-3D).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEDQA8e1EQ", "reviewer": "Reviewer_PmZF", "summary": "The paper proposes a unified 3D generation framework using a flow-matching network to sample pre-trained point cloud structured latent in a cascaded manner to achieve geometry-texture disentanglement. The paper uses multiview RGB-DN images as input to obtain point cloud structure latent vectors. These vectors are then used to generate Gaussian Surfels through a transformer decoder and further upsampled to obtain a dense Gaussian point cloud for rasterization. This stage is trained with a VAE objective to learn a latent distribution over the latent vectors. In the second stage, a flow-matching network is used to sample the point cloud structured latent vectors in a cascading manner. This achieves geometry and texture disentanglement and allows texture variation given sampled geometry. The proposed method generates visually impressive results compared to many baselines. It also contains a rich set of ablation studies.", "questions": "1. Eq. (5) misses a parenthesis. \n2. The sentence from Ln 49 - 51 is difficult to understand. What do the authors mean by the set as the latent space? \n3.  Ln. 52-54. \"Specifically, we project the un-structured features $\\mathbf{z}_z$ onto the manifold of the input 3D shape through the cross attention layer:\" The word \"manifold\" here is unnecessary and can lead to confusion since the queries are just fixed resolution sampled points, not the actual shape.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper shows better generation results compared to competitive baselines such as GRM. The qualitative visuals also look promising in, for example, generating the thin structures in the first row of Fig. (3).  \n2. The ablation study is comprehensive. It shows that the authors clearly put lots of efforts into validating the proposed system. \n3. The proposed system enables many applications including editing and generation tasks. This is further strengthened by its ability to disentangle geometry and texture.", "weaknesses": "1. Qualitative examples. In Fig. (3) why most of the input images are very dark? It seems that perhaps the other baselines wouldn't do well because of the particular input image choice. Also, the dark image makes it hard for me to judge if the proposed method is actually better than the baselines. For example, it's hard to see the color of the cactus on row 2. Also, why do all the baselines have different poses from each other? This also makes comparison difficult.  \n2. For Fig. (6b), why editing on latent space better than directly editing on Gaussians based on this example? Both show disjoint geometry.\n3. The generation results shown in the paper do not look very complex compared to the visuals in existing works such as GRM. It would be great if authors could provide further visual comparisons on more diverse prompt and caption inputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZCw6w6zdY", "reviewer": "Reviewer_21hH", "summary": "This paper focuses on feed-forward 3D generation, introducing an elegant 3D latent diffusion framework. \n\nThe process begins by training a point cloud-structured latent representation—unlike the tri-plane representation commonly used in LRM, 3DTopia, and LN3Diff—through a VAE, using multi-view images, depths, and normals as input. The decoder of the VAE consists of a DiT transformer to predict coarse Gaussians, and a few cascaded upsamplers to predict dense Gaussians. The VAE is trained with a combination of a rendering loss, geometry regularization, KL constraints as well as an adversarial loss.\nNext, a cascaded latent diffusion model is trained using flow matching for disentangled geometry and texture generation, which enables interative 3D editing.\nThe proposed framework can handle various conditioning inputs, including point clouds, texts and images.", "questions": "What dataset is used for text-to-3D evaluation? It seems not the commonly used T^3 benchmark. Why are metrics like render-FID used in CLAY (Zhang etal. 2024) not used here?\n\nCan the method generate 3D objects with more complex geometries and higher quality?\n\nCan PBR modeling be incorporated into the proposed framework?\n\nAs mentioned in the paper, the intermediate Gaussian output naturally serves as K LoD, allowing for a balance between rendering speed and quality in various scenarios.  I would be interested in seeing visualizations of the Gaussians at different LoD learned by the decoder. Additionally, how long does it take to perform one upsampling?\n    \nI would also like to see a discussion on the limitations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is clearly presented, with a well-motivated and thoughtfully designed method. I appreciate the overall approach and its intuitive structure.", "weaknesses": "The generated 3D objects presented in the paper and video demos are relatively simple and lack high quality, which diminishes the overall effectiveness of the method.\nThere are no quantitative results on image-to-3D generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YNll3AFJf", "reviewer": "Reviewer_45wi", "summary": "- This paper introduces a text- or image-to-3D generation framework using a VAE that processes multi-view RGB, normal, XYZ, and camera embeddings to construct a \"Point-Cloud structured\" Latent space, followed by a 3D DiT to decode 3D Gaussian parameters\n- The proposed method employs cascaded geometric and texture latent diffusion models for effective shape-texture disentanglement, enhancing a \"3D-aware editing\" application.\n- Experiments on text- and image-to-3D baselines consistently outperform existing methods.", "questions": "- Are there any insights during the proposed VAE training stage (see **Sec.** 4.1) regarding model design or loss functions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes 3D-VAE and 3D-Diffusion models that are well-designed, flexibly supporting both Text- or Image-to-3D Generation and 3D Editing applications.\n- Gaussian Anything improves \"3D Gaussians Utilization\", which reduces the rendering of white backgrounds, adaptively generates compact 3D Gaussians Splatting parameters and also benefits rendering quality.\n- The paper is overall well-written, and the models (3D VAE and two-stage cascaded Diffusion model) are well-designed.\n- It surpasses existing methods like 3DTopia, LN3Diff, and Shape-E in text-to-3D quantitative metrics and Lara/LGM in image-to-3D qualitative results.", "weaknesses": "- Qualitative metrics for Image-to-3D are **omitted**. Please add appropriate experiments in **Sec.** 5.1. Since the authors have rendered GSO datasets, GaussianAnything should be compared to LGM/Ln3Diff/Lara (refer to **Fig.** 3), and metrics such as PSNR, SSIM, LPIPS, or Image-CLIP Score may be reported.\n- The claim that GaussionAnything can support point clouds or multi-view images as input lacks sufficient experimental proof (only VAE supports multi-view/point cloud as input). The authors should either abandon this claim or provide corresponding quantitative or qualitative results in **Sec.** 5.\n- The use of flow matching for \"velocity-prediction\" in **Equations** 12 and 13, as opposed to direct \"noise prediction\", warrants further explanation in **Sec.** 4.2.\n\n\nI would appreciate it if the authors could address my concerns by providing corresponding quantitative or qualitative results based on the **weaknesses** and **review feedback**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)