AGG: Amortized Generative 3D Gaussians for Single Image to 3D
Authors: Dejia Xu
Ye Yuan
Morteza Mardani
Sifei Liu
Jiaming Song
Zhangyang Wang
Arash Vahdat
Venue: Accepted by TMLR ()

Abstract: Given the growing need for automatic 3D content creation pipelines, various 3D representations have been studied to generate 3D objects from a single image. Due to its superior rendering efficiency, 3D Gaussian splatting-based models have recently excelled in both 3D reconstruction and generation. 3D Gaussian splatting approaches for image to 3D generation are often optimization-based, requiring many computationally expensive score-distillation steps. To overcome these challenges, we introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image, eliminating the need for per-instance optimization. Utilizing an intermediate hybrid representation, AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. Moreover, we propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module. Our method is evaluated against existing sampling-based 3D Gaussian frameworks and inference-based pipelines utilizing other 3D representations, where AGG showcases competitive generation abilities both qualitatively and quantitatively while being several orders of magnitude faster.

Published in Transactions on Machine Learning Research (10/2024)
AGG: Amortized Generative 3D Gaussians for Single Image to 3D
Dejia Xu University of Teras at Austin
dejio@utemas.co edu
Ye Yuan NVIDIA
yey com
Morteza Mardani NVIDIA
mmandani@mviddia.cm
Sifei Liu NVIDIA
sifeilnnviiaa.com
Jiaming Song
com
NVIDIA
Zhangyang Wang University of Teros Austin
atlaswang@uteras.co edu
Arash Vahdat NVIDIA
avahdat@mvidia.com
Reviewed OpenReview: /ttps://openreest net/ forum? id= B0q3nS
Abstract
Given the growing need for automatic 3D content creation pipelines, various 3D repre- sentations have been studied to generate 3D objects from a single image. Due to its su- perior rendering efficiency, 3D Gaussian splatting- based models have recently excelled in both 3D reconstruction and generation. 3D Gaussian splatting approaches for image to 3D generation are often optimization- requiring many computationally expensive score- distillation steps. To overcome these challenges, introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from single image, climinating the need for per-instance optimization. Utilizing an intermediate hybrid repre- sentation, AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. Moreover, we propose a cascaded pipeline that first. gen- erates a coarse representation of the 3D data and later upsamples with a 3D Gaussian supor-resolution module. Our mothod evaluated against existing optimization- based 3D Gaussian frameworks and sampling-basee pipelines utilizing other 3D representations, where AGG showcases competitive generation abilities both qualitatively and quantitatively while being several orders of magnitude faster. Project page: /ttts://irld..o io/AGG/
1
Introduction
The rapid development in virtual and augmented reality introduces increasing demand for automatic 3D content creation. Previous workflows often require tedious manual labor by human experts and need specific
OOur of the 'amortized" originates from its original meaning: to gradually reduce write off the initial cost of directly employing standard
time. To avoid potential confusion, would like clarify that amortized aptimizatiion/ineeeeee framework.

---

Published in Transactions on Machine Learning Research (10/2024)
Coarse Hybrid Generator
Gaussian Super Resolution
Figure 1: Overview of our AGG framework. We design a novel cascaded generation pipeline that produces 3D Gaussian-based objects without per-instance optimization. Our AGG framework involves coarse generator that predicts a hybrid representation for 3D Gaussians at low resolution and a super-resolution module that delivers dense 3D Gaussians in the fine stage.
software tools. A 3D asset generative model is essential to allow non-professional users to realize their ideas into actual 3D digital content.
Much effort has been put into image-to-3D generation as it allows users to control the generated content.
Early methods study the generation of simple objects Yu et al. (021)) or novel view synthesis of limited viewpoints Li et al. (2021); Shih et al. 22020 y; Xu et al. )202aa)
). Later on, with the help of 2D lext-to-image
diffusion models, score distillaton sampling Poole et al.
(2022) has enabled 360-degree object generation for
in-the-wild instances Xu al. (2022b); Mclas-Kyriazi et al. [2023]: Tang al. (2023b). More recently, the
advances in collecting 3D dalasets Deilke et. al.] (2023); Wu et al.
(2023b); Yu el al. (2023) have supported
the training of large-scale 3D-aware generative models Liu et al.
(2023b a); Gao et al. (0022) for better
image-to-3D generations.
Various 3D represenlations have been studied as the media to train 3D generative models. Many prior works have explored generating explicit representations, such as point clouds Zeng et al. (002) voxels Zhou et al. (2021); Schwarz et al. (2022), and occupancy grid Wu et al. 2023a Implicit representations like
NeRF Mildenhall al. (2020) and distance functions Lionar et al. (2023) are popular for easy optimization.
Although effective for content generation Yang et al. (2023] Erkog et al. (2023); Or-El et al.
(002)). slow
rendering and optimization hinder the wide adoption of these representations. As result, rescarchers have
looked into more elficient volumetric representations Müller et al. (2022) Chan el al. (2022) to support the
training of larger generative models Chen et al. (2023a)) Karnewar et al. (2023) and the costly optimization in
score distillation Xu et al. 2022b): Melas-Kyriazi et al. (2023). More recently, 3D Gaussian splatting Kerbl
eet al. ) Tang et al. 2002a): Chen al. 2023bb;
al. (023b) has attracted great atlention due its high-quality real-time rendering ability. This owes the visibility-aware rendering algorithm that facilitates much less optimization time cost compared with NeRFF-based variants Liu et al. (2023b); Poole et al. (022).. Such efficient renderer enables supervising the 3D Gaussians through loss functions defined the 2D renderings, which is not suitable for other representations.
Despite the exciting progress, how to properly generate 3D Gaussians Kerbl et al. 2023 remains a less studied topic. Existing works Tang et al. (2023a): Chen et al. (2023b); Yi et al. (2023b) focus on the per- instance optimization-basee seiting. Though 3D Gaussians can be optimized with adaptive density control represent certain gcometry, initialization remains critical for complex object structures. An amortized pipeline is highly desired to produce the 3D Gaussians in one shot. Such network can learn shared 3D understanding of images that generalizes to unseen objects of similar categories to the training set. This will further reduce the need for test-time optimization, trading off the computation cost of the inference stage with the training stage.
However, building such a feed-fore forward pipeline challenging for two major reasons. First, 3D Gaussians rely on adaptive density control Kerbl et al. (2023) to represent complex geometry, but this leads to a dynamic number of 3D Gaussians, making hard to predict them in an amortized training setting. Second, the
3D Gaussians require curated initialization Yi et. al. (002bb) Chen al. (2023b) to be updated properly
2

---

Published in Transactions on Machine Learning Research (10/2024)
via supervision the rendering. In the amortized setting, the 3D Gaussians are instead predicted by the neural network, which leads to the need to initialize the network well such that generated Gaussians can be supervised decently. Moreover, the optimization process often favors updating the appearance of 3D Gaussians instead of moving their positions directly to desired 3D locations.
To overcome these issues, we conduct a study on the amortized generation of 3D Gaussians through feed- forward process. As shown in Fig. | we propose AGG, a cascaded generation framework that generates 3D Gaussians from single image imput. In the first stage, employ hybrid generator that produces 3D Gaussians at coarse resolution. In this stage, decompose the geometry and texture generation task into two distinct networks. A geometry transformer decodes image features extracted from a pre-trained image feature extractor and predicts the location of 3D Gaussians. Another texture transformer similarly generates a texture field that is later queried by the Gaussian locations to obtain other point attributes. Utilizing this hybrid representation as an intermediate optimization target stabilizes our training process when jointly optimizing the geometry and texture of the 3D Gaussians. In the second stage, we leverage point-voxcl convolutional nelworks to extract. local features effectively and super-resolve the coarse 3D Gaussians from the previous stage. RGB information is further injected into the super-resolution networks to refine the texture information.
Our contributions can be summarized as follows,
We study single image-to-3D Gaussian in the amortized setting. Unlike existing works that operate on individual objects, we build novel cascaded generation framework that instantly presents 3D Gaussians in one shot.
Our AGG network first generates coarse Gaussian prodictions through a hybrid representation that decomposes geomelry and texture. With two separale transformers predicting the geometry and texture information, the 3D Gaussian attributes can be optimized jointly and stably.
A UNet-based architecture with point-voxel layers introduced for the second stage, which effec- tively super-resolves the 3D Gaussians.
Compared with existing baselines including optimization-basee 3D Gaussian pipelines and sampling- based frameworks that use other 3D representations, AGG demonstrates competitive performance both quantitatively and qualitatively while enabling zero-shot image-to-object generation, and being several orders of magnitude faster.
2 Related Works
2.1
Image-to-30 Generation
Numerous research studies have been conducted on generating 31) data from single image. Early attempts
simplify the challenges by either generating objects of simple geometry Yu et al. 2021 focusing
synthesizing novel view images of limited viewpoints T.i et al. (2021): Shih et al. 2020 Xu et al. 2022a
Later on, various combinations of 2D image encoder and 3D representations are adopted to build 3D
generative models, such as on 3D voxels Girdhar et al.] (2016) Choy et al. (2016); Yagubbayli et al. 2022
point clouds Yang et al. (2019); Fan et al. (2017): Zeng et al. 2022 Wu et al. (2023a); Lionar et al. 2023
meshes Gao et al. (2022 : Kanazawa al. (2018); Wang al. (2018). and implicit functions Erkog et al.
(023 Yang et al. (2021). More recently, with the help of score distillation sampling from pre-trained
text-to-image diffusion model, great efforts have been put into generating 3D asset from single image
through optimization Xu et al. (2022b); Melas-Kyriazi et al. (0233 Seo et all (2023); Tang et al. (2023b): Deng et al. (2023): Tang et al."
2023a Among them, Dream Gaussian utilizes 3D Gaussians as an efficient 3D representation that supports real-time high-resolution rendering via rasterization. In addition to these optimization- based methods, building feed-forward models for image-to-3D generation avoids the timo-consuming optimization process, and a large generative model can learn unified representation for similar objects, leveraging prior knowledge from the 3D dataset better. The advances in largo-scale 3D datasets Deitke et al. (2023) Wu et al. (002bb); Yu et al. 2023 largely contributed to the design of better
3

---

Published in Transactions on Machine Learning Research (10/2024)
image-to-3D generative models Liu et al. (2023b a Gao et al. (2022
; Hong et al. (2023 : Li et al. (0023)
OpenAT has trained a 3D point cloud generation model Point-F E. Nichol et al. (2022), based on million of
internal 3D models. They later published Shap-E Jun & _Nichol (2023), which is trained on more 3D models and further supports gencrating textured meshes and neural radiance fields. Our work follows the direction of building feed-forward models for image-to-3D and makes the first. attempl to construct an amortized model that predicts the 3D Gaussians instead of constructing them through optimization.
2.2 Implicit 3D Representations
Neural radiance field (NeRF) Mildenhall et al. (2020) implements a coordinate-basedd neural network to represent the 3D scenes and demonstrates outstanding novel view synthesis abilities. Many following works have attempted to improve NeRF in various aspects. For example, MipNeRF [arron et al. (2021a) and
MipNeRIF-360 [arrron et al. (2021b) introduce advanced rendering techniques to avoid aliasing artifacts. SinNeRF Xu et al. (2022a), PixeINeRF Yu et al. (2021), and SparseNeRF Wang et al. (2023) extends the application of NeRFs to few-shot input views, making single image-to-3D generation through NeRF a more feasible direction. With the recent advances in score distillation sampling Poole et al. (2022), numerous approaches have looked into optimizing a neural radiance field through guidance from diffussio priors Xu it al. 2022b); Melas-Kyriazi et al.] (2023): Seo et al. (2023) : Tang et al. (2023b); Deng et al. (2023) While suitable for optimization, NeRFs are usually expressed implicitly through MLP parameters, which makes
it challenging generate them via network Erkoç al. (2023). Consequently, many works Lorraine et al.)
(023); ;
Chan al. (2022) Bhattarai et al. 20223) instead choose to generate hybrid representations, where
NeRF MLP is adopted to decode features stored in explicit structures Chan et al. (2022); Chen et al. (2022
Müller et al.
(2022); Fridovich-Keil et al. (2023): ;
Yi et al. (2023a): Xu et al. (202cc): Barron et al. (2023
Among them, ATT3D Lorraine et. al. 2023) builds an amortized framework for Lext-Lo-3D by generating
Instant NGP Müller et al. [2022] via score distillation sampling. While focusing on generating explicit
3D Gaussians, our work draws inspiration from hybrid representations and utilizes hybrid structures as
intermediate generation targets, which can be later decoded into 3D Gaussians.
2.3 Explicit 3D Representations
Explicit 3D representations have been widely studied for decades. Many works have attempted to con-
struct 3D assets through 3D voxels Girdhar et al. (2016); Choy et al. (2016): Yagubbayli et al.] (2021), ), point clouds Yang et al. (219)) Fan ct al. (2017): Zeng et al. [2022]; Wu et al. (2023a); Lionar et al. (2023) and
meshes Gao et al. (2022) Kanazawa al. 2018); Wang al. (0018). Since pure implicit radiance fields
are operationally slow, often needing millions of neural network queries to render large-scale scenes, great efforts have been put into integrating explicit representations with implicit radiance fields to combine their advantages. Many works looked into employing lensor factorization to achieve efficient explicit represen- tations Chan et al. (2022); Chen et al. (2022): ) Fridovich-Keil et al. (2023); Yi et al. (2023a): : Müller et al. (2022). Multi-scale hash grids Müller et al. (0022) and block decomposition Tancik et al. (0022) are intro- duced to extend to city-scale scenes. Another popular direction focuses on empowering explicit structures with additional implicit attributes. Point NeRF Xu et al. (2022c) utilizes neural 3D points to render continuous radiance volume efficiently. Similarly, NU-MCC Lionar et al. (2023) uses latent point features representation, specifically focusing on completing shapes. 3D Gaussian splatting Kerbl et al. (2023)
introduces point-based a-blending along with an efficient rasterizer and has attracted great attention due to their outstanding ability in reconstruction Kerbl et al. (2023) and generation Tang et al. (2023a) : Chen
et al. (2023b;; Yi et al. (2023b) tasks. A concurrent work, Hu et al. (2024). addresses efficient point cloud rendering through generalizable neural network converting point clouds to 3D Gaussians. However, existing 3D generation works present 3D Gaussians through optimization, while our work takes one step further and focuses on building an amortized framework that generates 3D Gaussians without per-instance optimization.
3 Amortized Generative 3D Gaussians
Unlike the existing methods that optimize Gaussians, initialized from feructure-from-motte (SfM) or ran- dom blobs, our work pursues more challenging objective: generating 3D Gaussians from single image
4

---

Published in Transactions on Machine Learning Research (10/2024)
using neural network in one shot. Specifically, our cascaded approach begins with constructing a hybrid generator that maps input image features into concise set of 3D Gaussian attributes. Subsequently, a UNet architecture with point-voxel layers is employed to super-resolve the 3D Gaussian representation, improving its fidelity. In the following subsections, we first provide the background on 3D Gaussian splatting, then we delve deep into details about our proposed components, and finally, we illustraae how we address the rising challenges with amortized generation.
3.1 Background: 3D Gaussian Splatting (3DGS)
3D Gaussian splatting is a recently popularized explicit 3D representation that utilizes anisotropic 3D Gaus- sians. Each 3D Gaussian G is defined with center position μ E R covariance matrix E, color information R2 and opacity E RR. The covariance matrix E describes the configuration of ellipsoid and is implemented via a scaling matrix S E R$ and rotation matrix R E 13x3.
== RSS
(1)
This factorization allows independent optimization of the Gaussians' attributes while maintaining the semi- definite property of the covariance matrix. In summary, each Gaussian centered at point (mean) jl is defined as follows:
G(z) = (
(2)
where refers to the distance between μ and the query point. G(x) is multiplied by in the blending process to construct the final accumulated color:
C=E Ga,G(a) EE
II 2=1
An efficient tilo-based rasterizer enables fast forward and backward pass that facilitates real-time rendering. The optimization method 3D Gaussian properties is interleaved with adaptive density control of Gaussians, namely, densifying and pruning operations, where Gaussians are added and occasionally removed. In our work, we follow the convention in 3D Gaussian splatting to establish the Gaussian attributes but identify specific canonicalization designed for single image-to-3D task. We first reduce the degree of the spherical harmonic coefficients and allow only diffuse colors for the 3D Gaussians. Additionally, we set canonical isotropic scale and rotation for the 3D Gaussians, as we find these attributes extremely unstable during the optimization process.
3.2
Coarse Hybrid Generator
In the coarse stage, our AGG framework utilizes a hybrid generator to produce a hybrid representation as an intermediate generation target. Initially, the input image is encoded using a vision transformer. Subsequently, geometry and texture generator constructed to map learnable queries into location sequences and texture fields individually. This hybrid representation is then decoded into explicit 3D Gaussians, which enables elficient high-resolution rendering and facilitates supervision via multi-view images. The overall architecture of our proposed coarse hybrid generator is illustrated in Fig. 2 Our model is trained via rendcring loss defined on multi-view images and is warmed up with Chamfer distance loss using 3D Ganssian pseudo labels.
Encoding Input RGB Information Single image-to-3D is a highly ill-posed problem since multiple 3D objects can align with the single-view projection. As result, effective image encoder is greatly needed to extract essential 3D information. Previous work Liu et al. (2023blch mainly incorporate CLIP image encoder Radford et al. (2021). benefiting from the foundational training of Stable diffusion Rombach et al. 2002).. While effective for Lext-lo-image generation, CLIP encoder is not well-designed for identifying feature correspondences, which is crucial for 3D vision tasks. As a result, we use the pre-trained DINOv2 transformer as our image encoder, which has demonstratod robust feature extraction capability through self-supervised pre-training. Unlike previous works Tumanyan et al. (2022); Xu al. (2022a) that only use the aggregated global [CLS] token, choose to incorporate patch-wise features well.
5

---

Published in Transactions on Machine Learning Research (10/2024)
Queries
Location
DINOv2
Geometry Predictor
brendering
Queries
Texture Generator
Lehamfer
reshape
Opacity, Color
Figure 2: Architecture of our coarse hybrid generator. We first use a pre-trained DINOv2 image encoder to extract essential features and then adopt two transformers that individually map learnable query tokens to Gaussian locations and a texture field. The texture field accepts location queries from the geometry branch, and decoding MLP further converts the interpolated plane features into Gaussian attributes (e.g., opacity, color).
Geometry Predictor The location information of 3D Gaussian is expressed via their 3D means following 3DGS Kerbl el al. (2023). which is a three-dimensional vector for each point. Thus, we adopt a transformer- based network for predicting the location sequence. The transformer inputs are a set of learnable queries implemented with a group of Icarnable position embeddings. Each query will correspond one 3D Gaussian that we generate. Before feeding into the transformer network, the positional embeddings are summed with the global token [CLS] extracted from the DINOv2 model. The sequence of queries is then modulated progressively by series of transformer blocks, each containing cross-attention block, self-attention block, and multi-layer perception block. These components are interleaved with LayerNorm and GeLU activations. The cross-attention blocks accept DINOv2 features as conlext information. The final layer of our geometry predictor involves an MLP decoding head, converting the hidden features generated by attention modules into a three-dimensional location vector.
Texture Field Generator Joint prediction of texture and geometry presents significant challenges. A primary challenge the lack of direct ground truth supervision for texture in 3D space. Instead, texture information is inferred through rendering losses in 2D. When geometry and texture information decoded from shared network, their generation becomes inevitably intertwined. Consequently, updaling the pre- dicted location alters the supervision for rendered texture, leading to divergent optimization directions for texture prediction.
To resolve this, a distinct transformer is employed generale a texture field. Our texture field is implemented using a triplane Chan et al. (2022). complemented by a shared decoding MLP head. The triplane accepts 3D location queries from the geometry branch and concatenates interpolated features for further processing. Utilization of this texture field facilitates decomposed optimization of geometry and texture information. More importantly, texture information is now stored in structured planes. Incorrect geometry predictions do not impact the texture branch's optimization process, which receives accurate supervision when geometry queries approximate ground Iruth locations.
Supervision Through 2D Renderings Thanks to the efficient rasterizer for 3D Gaussians, we can apply supervision on novel view renderings during the training. The 3D Gaussians we generate are rendered from randomly selected novel views, and image-space loss functions are calculated against ground truth renderings available from the dataset. Concretely, render the scene into RGB images and corresponding foreground

---

Published in Transactions on Machine Learning Research (10/2024)
DINOv2
Cross
Feature
Cross
Frendering
PVCNN
PVCNN
Atin
Expansion
Atto
Figure 3: Illustration of the second-stage Gaussian super-resolution network. We first encode the original input image and the stage one prediction separately. Then, unite them through cross-attention at the latent space. We perform super-rosolutio in the latent space and decode the features jointly.
alpha masks. We utilize LPIPS Zhang et al. (0188)) and Lı loss to minimize their differences,
Lrendering frgba + wi Lipips:
(4)
where wi is weighting factor.
3.3
Gaussian Super Resolution
Although our hybrid generator is effective in the coarse generation stage, generating high-resolution 3D Gaussians requires many learnable queries which are computationally expensive due to the quadratic cost of self-attention layers. As result, we instead utilize second-stage network as super-resolution module introduce dense 3D Gaussians into generation. Since coarse geometry is obtained from the first stage, the super-resolution ion network can focus more on refining local details. For simplicity, we use lightweight UNet architecture with the efficient point-voxel layers Liu et al. (2019), as shown in Fig. 3
Latent: Space Super-Resolution As mentioned earlier, the 3D Gaussians require curated locations to update the other attributes properly. This leads to the key challenge for the super-resolution stage, which is to introduce more point locations for the super-resolved 3D Gaussians. Inspired by previous works 2D image Shhi et al. (2016) and point cloud super resolution Qian et al. (2021), we perform feature expanding as surrogate of directly expanding the number points. Through point-voxel convolutional encoder, we first convert the stage one coarse 3D Gaussians into compact latent features that can be safely expanded through feature expansion operation: rearranging tensor of shape (B, N, C r) to tensor of shape (B, N x C). The expanded features are then decoded through a point-voxel convolutional decoder that predicts the Gaussian locations and other attributes.
Incorporating RGB information While the first-stage network may capture the rough geometry of objects, the texture field may converge into blurry results due to the oscillating point locations and rough geometry. Consequently, utilizing the abundant texture information from the input image is crucial for the super-resolution network generate plausible details. For this purpose, we introduce RGB features into the bottleneck of the UNet architecture. Specifically, we adopt cross-attention layers before and after the feature expansion operation. Image features fed through cross-altention layers to modulate the latent voxet-voxel features.
3.4 Overcoming Challenges in Amortized Training
Unlike the vanilla 3D Gaussian splatting sctting, where set of 3D Gaussians are optimized specifically for each object, our amortized framework involves training generator that jointly produces the 3D Gaussians for large set of objects coming from diverse categories. As result, many specifically designed operations

---

Published in Transactions on Machine Learning Research (10/2024)
for manipulating 3D Gaussians not available in setting. Below, we illustrate how overcome these challenges in detail.
3.4.1 Adaptive Density Control
The original 3D Gaussians involve special density control operations move them toward their desired 3D locations. However, in the amortized training setting, it non-straighhfforward to adaptively clone, split, prune the Gaussians according to the gradients they receive. This is because these operations change the number of points, making it hard for an amortized framework to process.
To this end, we use a fixed number of 3D Gaussians for each object, avoiding the burden for the generator network to determine the number of points needed. Moreover, since we do not have access to clone and split operations, training the amortized generator leads to sharp 3D Gaussians with unpleasant colors that are trying to mimic fine fine-graine grained texture details. Once the predictions become sharp, they get stuck in the local minimum, and therefore, fewer 3D Gaussians blurry corrupted generations. To overcome the issue, empirically set canonical isotropic scales and rotations for the 3D Gaussians on all objects to stabilize the training process.
available represent the overall object, leading to
3.4.2 Initialization
Proper initialization plays an important role in the original aptimization-baset 3D Gaussian splatting, Though Gaussian locations jointly optimized with other attributes, it observed that optimization updates often prefer reducing the opacity and scale instead of moving the Gaussians directly. By doing so, Gaussians are eliminated from the wrong location after reaching the pruning threshold. Instcad, the adaptive density control can then clone or split Gaussians at proper locations according to their gradients. Albeit the simplicity of pruming and cloning operations for optimization algorithms, they are non-trivial to implement with amortized neural networks.
To overcome the issues, we warm up the generator with 3D Gaussian pseudo labels. We use few 3D objects and perform multi-view reconstruction to obtain 3D Gaussians for each object. Since multiple sets of 3D Gaussians can all be plausible for reconstructing the same 3D ground truth object, the 3D Gaussians are only considered pseudo labels where we use their attributes to initialize our network layers properly.
Due to these Gaussians being stored in random orders as sets, we cannot use L1 reconstruction loss as in generative frameworks such as LION Zeng et al. (2022). 1 Alternatively, we utilize the Chamfer distance loss to pre-train our network. Specifically, we first calculate the point matching correspondences according to the distance of Gaussians' 3D means and then minimize the C₁ difference of each attribute, including location, opacity, and color. The formulation can be summarized as:
Lehamfer (f) =
W1
Σ PLEPI
(If - f2ill)
P
paj min (1:-=133)
xeP2
+
222
Σ Pa
(1lf2) fuill)
Pa
Pii marg min |a [a-P2sll
11
E P:
where f refers to 3D Gaussian attributes, P1 is our generated set of 3D Gaussian, and Pa is the 3D Gaussian
pseudo label.
4 Experiments
4.1 Implementation Details
Gaussian Configuration We set the canonical isotropic scale of each 3D Gaussian to 0.03 and 0.01 when 4,096 and 16,384 3D Gaussians produced for each object, respectively. The canonical rotation for all 3D
8

---

Published in Transactions on Machine Learning Research (10/2024)
Input
Point-E
Dream Gaussian
One-2345
AGG
GT
Figure Novel view rendering comparisons against bascline methods. Our AGG model obscrves none of these testing images during training.
9

---

Published in Transactions on Machine Learning Research (10/2024)
Gaussians is set to [1, 0, O, 0] for the rasterizer. When preparing the pscudo labels for the warmup training, we use the ground truth point clouds from the OmniObject3D Wu et al. (2023b) dataset to initialize the Gaussian locations.
Network Architecture We implement our transformer blocks following the well-established practice in DINOv2 Oquab et al. (2023) with GeLU activation and Layer Normalization layers. The image features are extracted using pre-trained DINOv2- model at 256 x 256 input resolution.
Object Placement and Camera System We reconstruct objects in camera space and normalize the input view azimuth to zero. This means that the image and the 3D assets are aligned and consistent as both are rotated. This 3D augmentation during the training stage enforees rotation cquivariance and prevents the network from merely overfitting the canonical placements of objects in the dataset.
4.2 Baseline Methods
We compare with two streams of work. One involves the sampling based 3D generation that uses different 3D representations, such as Point-E and One-2345. Point-E Nichol et al. (2022) includes a large diffusion transformer that generates point cloud. Their work performs the generation in the world coordinates, where objects always canonically placed the original orientation in the dataset. One-2345 Liu et al. (0023)) leverages Zerol23 Liu et al. (2023b) and utilizes SparseNeuS Long et al. (2022) to fuse information from noisy multi-view generations efficiently. We also compare with Dream Gaussian Tang et al. (0233)) first stage, where 3D Gaussians are optimized with SDS Poole et al. (2022) from Zero123 Liu et al. (023b). Both One-2345 and DreamGaussian require the elevation of the input image at inference time. For One-2345, we use official scripts to estimate the elevation. We provide DreamGaussian with ground truth elevation extracted from the camera pose in the testing dataset.
4.3 Dataset
Our model is trained on OmniObject3D datasct Wu et al. 2023b), ) which contains high-quality scans of real-world objects. The number of 3D objects is much Tewer than Objaverse Deitke el al. (2023). Alongside point clouds, the OmniObject3D dataset provides 2D renderings of objects obtained through Blender. This allows us to implement andering-based loss functions that supervise the rendering quality in 2D space. We construct our training set using 2,370 objects from 73 classes in total. We train one model using all classes. The test set contains 146 objects, with two left-out objects per class.
4.4 Qualitative and Quantitative Comparisons
We provide visual comparisons and quantitative analysis between our methods and existing baselines in Fig. and Tab. I As shown in the novel view synthesis results, model learns reasonable geometry understanding and produces plausible generations of texture colors. Point-F, Nichol et al. (022)) produces
reasonable geometry but presents unrealistic colors, possibly because the model was trained on large-scale synthetic 3D data. Both One-2345 Liu et al. (2023a) and DreamGaussian Tang et al. (2023a) can generate corrupted shapes, which might be because the novel views synthesized from the Zero-12: diffusion model Liu et al. (202bb) are multi-view inconsistent.
We use CLIP distance defined on image embeddings to reflect high-level image similarity on multi-view renderings, well as PSNR, SSIM, and LPIPS Zhang et al. (2018a) to measure the visual quality compared to the ground truth 3D object. For Point-E's outputs, we only use CLIP distance because they are produced in canonical orientation and not aligned with the ground truth 3D object. As shown in Tab. | our method obtains competitive numbers in terms of image quality. Additionally, our AGG network enjoys the fastest inference speed.
In comparison, both Point-E and One-2344 2345 adopt an iterative diffusion process when producing their final results, while DreamGaussian leverages a costly optimization process through score distillation sampling.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Input
(o) CLIP
(b) S1, DINO
(#151, wio warmaa
Novel View GT
(d) Full
(e1 S2, wio image cond
(4) Pull
Izpu
Drennossmionn
Une1290
Our
Figure 5: Visual comparison of ablation studies. Compared to variants (a,b,c,e), our full model in stage 1(d) and stage 2(f) brings better perceptual quality.
Figure 6: Comparison in-the-wild image (a) and Google Scanned Object (b).
Table 1: Comparisons against baseline methods regarding novel view image quality and inference speed.
Method
Point-E
One-2345
DreamGaussian
Ours
PSNR1
14.73
18.87
18.36
SSIM
0.7084
0.8147
0.8027
LPIPSJ
0.3473
0.1906
0.1601
CLIP Dist L
0.5139
0.3084
0.4293
0.3458
Time (8)
78
45
60
0.19
Table 2: Ablation study of small-scale models that predict 2048 and 8192 Gaussians in stage (S1) and
stage 2 (S2). Models trained on 10 classes. Metries were calculated on 20 held-out objects. Our full model
in stagel (d) and sfage2(f) brings better perceptual quality (LPIPS).
Method (a) S1, CLIP
PSNRf
SSIMf
LPIPSI
Method
PSNR1
SSIMT
LPIPS4
15.82
0.7618
0.2440
(d) Full S1
15.84
0.7619
0.2429
(b) S1, DINO ft
16.02
0.7615
0.2437
(c) S2, w/o image cond
15.91
0.7299
0.2322
(c) S1, w/o warmup
14.95
0.7631
0.2611
(f) Full S2
16.03
0.7407
0.2217
4.5 Ablation Study
We conduct thorough experiments to validate the effectiveness our proposed components using the available 3D ground truth from the datasct. We compare with multiple variants of the model. Our full model conditions fixed DINO feature. We explored fixed CLIP features and fine-tuned DINO features in Fig. 5 a, b). In comparison, fixed DINO feature in Fig, 3 d) eases training, while other options degrade perceptual quality in the novel-view renderings. Then we study the effect of the warmup set in Fig. 5 (c). This set eases the optimization for geometry prediction. Without the warmup set, it is incffective to rely on rendering loss for accurate geometry prediction. We observe a large drop in PSNR and LPIPS in Tab. ( compared to (d). Finally, we remove the image conditioning in stage 2. As shown in Fig. (e) and Tab. e), image conditioning helps stage model overcome unexpected color shifts. As shown in Tab. 2 ff)
and Fig. ((),,
the full model delivers the best generation results both qualitatively and quantitatively when rendered from novel viewpoints.
4.6 In-the-wild Evaluation
We further test our model on image from Google Scanned Objects Downs al. (2022) and in-the-wild image. as shown in Fig. The foreground object is segmented with the help of Segment Anything Model Kirillov et al. 2023 Compared to existing baselines, our model produces more reasonable 3D objects when provided
11

---

Published in Transactions on Machine Learning Research (10/2024)
with out-of-distribution test cases. Existing works that rely the view consistency of novel-view synthesis model fail to produce consistent results, therefore their merged 3D model contains unpleasant artifacts.
4.7 Limitation
Despite the encouraging visual results produced by AGG, the number of 3D Gaussians generate is still limited to represent very complex geometry, In the future, we will further explore how to expand AGG to more challenging scenarios, such as when the input image contains multiple objects with occlusion.
5 Conclusion
In this work, we make the first attempt to develop an amortized pipeline capable of generating 3D Gaussians from a single image input. The proposed AGG framework utilizes cascaded generation pipeline, comprising a coarse hybrid generator and a Gaussian super-resolution model. Experimental results demonstrate that our method achieves competitive performance with several orders of magnitude speedup in single-image- to-3D generation, compared both optimization-baset 3D Gaussian frameworks and sampling-baee 3D generative models utilizing alternative 3D representations.
References
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5855-5864, 2021a.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. arxie preprint arXiv:2111.1007, 2021b.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti- aliased grid-based neural radiance fields. ICCV, 2023.
Ananta Bhattarai, Matthias Nießner, and Artem Sevastopolsky. Triplanenet: An encoder for eg3d inver- sion. aiiiv preprinl or 2xv: 13497, 2023.
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient. geometry-aware 3d generative adversarial networks. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123 16133, 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. European Conference on Computer Vision, pp. 333-550. Springer, 2022.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. arXiv preprint arXiv:2304.06714, 2023a.
Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. arXio preprint wrXiv:2309.16588 2023b.
Christopher B Choy, Danfei Xu, Jun JunYoung Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-viow 3d object reconstruction. In Computer Vision-ECCV 2016: 14th European Conference, Amslerdam, The Netherlands, Oclober 11-14, 2016, Proceedings, Parl VIII 14, pp. 628 644. Springer, 2016.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaversc: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13152 2023.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 20637 20647, 2023.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2553-2560. TEFE, 2022.
Ziya Erkog, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with spight-space diffusion. arXiv preprint AXi:22303.10015 2023.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605 613, 2017.
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognation, pp. 12479 12488, 2023.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 35:31841- -31844, 2022.
Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning predictable and gen- erative vector representation for objects. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pp. 484 499. Springer, 2016.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2811.044 04400, 2023.
Yueyu Hu, Ran Gong, Qi Sun, and Yao Wang. Low latency point cloud rendering with learned splatting. In Proceedings of the IEEE/CV Conference on Compuler Vision and Pallern Recognition, pp. 5761, 2024.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. ar preprint 2rXiv:2305.02403 2023.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-386, 2018
Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra. Holodiffusion: Training 3d diffusion model using 2d images. In Proceedings of the IEEE/ CEE//VV Conference on Computer Vision and Pattern Recognition, pp. 18433, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Thansaction Graphics (ToG), 42(4):1-14, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. anXio preprint 1.X.::1112.6996 2014.
Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Wen Lo, et al. Segment anything. arXiv preprint wwXiv:2304.06643 2023.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse- view generation and large reconstruction model. arXiv preprint or Xiv:23311 Xtx:22311.0021,, 2023.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lec. Mine: Towards continuous depth mpi with nerf for novel view synthesis. In Proceedings of the IEEE/ CVF International Conference on Computer Vision, pp. 12578 12588, 2021.
Stefan Lionar, Xiangyu Xu, Min Lin, and Gim Hee Lec. Nu-mec: Multiview compressive coding with neighborhood decoder and repulsive udf. ar preprinl arXiv:2307.09112 2023.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One- 2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint
arXiv:2306. 16928, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Z-ro-l- to-3: Zero-shot one image to 3d object. aixiv preprint arXiv:2909. 11228, 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from single-view image. arXiv preprinl arXiv:2309.0844 2023c.
Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel emn for efficient 3d deep learning. Advances in Newral Information Processing Systems, 32, 2019.
Xiacxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision, pp. 210-227. Springer, 2022.
Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming-Yu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. arXiv preprint arv::22306. 07349, 2023.
Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 8446 8455, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXit preprint arXiv:2201.058 05988,, 2022.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXie preprint ar arXiv:2212.08771 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Saanellmacher-Shiieemeee Stylesdf: High-resolution 3d-consistent image and geometry generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13503 13513, 2022.
Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint Xim::2200. 14988, 2022.
Guocheng Qian, Abdulellah Abualshour, Guohao Li, Ali Thabet, and Bernard Ghanem. Pu-gen: Point cloud upsampling using graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11683-11111 11692, 2021.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 11684,,65,, 2022.
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. Advonces in Neural Information Processing Systems, 55:33999-34011 2022.
Junyoung Sco, Wooscok Jang, Min-Scop Kwak, Jachoon Ko, Hyconsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint Xrx::2303. 07937, 2023.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolu- tional noural network. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1874 1883, 2016.
Meng-Li Shih, Shih-Yang Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context-aware layered depth inpainting. Tn Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 0128-803,, 2020.
Stanislaw Szymanowicz, Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d roconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11201, 2024.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8248-8258) 2022.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint ar 1xxx:2330916663, 2023a.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3de High-fidelity 3d creation from a single image with diffusion prior. ar preprint ar Xix::23003. 14184, 2023b.
Narck Tumanyan, Omer Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. Xixiu preprint 2rX:::2201.00/2, 2022.
Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. arXiv preprint arXiv:2303.1110 2023.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pp. 52 67, 2018.
Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9065-9075, 2023a.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, recon- struction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 803 814, 2023b.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from single image. In European Conference on Computer Vision, pp. 736 753. Springer, 2022a.
Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Yi Wang, and Zhangyang Wang. Neurallift-360: Lifting an wn-the-wild 2d photo to a 3d object with 360 ('degt views. arXiv preprint arXio:2211 16431, 2022b.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point- nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5438-5448, 2022c.
Farid Yagubbayli, Yida Wang, Alessio Tonioni, and Federico Tombari. Legoformer: Transformers for block- by-block multi-view 3d reconstruction. arXiv preprint Xrxi:22106. 12102, 2021.
Guandao Yang, Xun Huang, Zekun Hao, Ming-Ye Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 4541-4550, 2019.
Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34:22483-222 -22497, 2021.
Guandao Yang, Abhijit Kundu, Leonidas J Guibas, Jonathan T Barron, and Ben Poole. Learning diffusion prior for nerfs. arXiv preprint ar arXiv:2304.144 14473, 2023.
Brent Yi, Weijia Zeng, Sam Buchanan, and Yi Ma. Canonical factors for hybrid neural fields. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 3414-3426, 2023a.
Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint ar (xXi:2230008855 2023b.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578 4587, 2021.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjic Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 9150 450-9161, 2023.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis, Lion: Latent point diffuuson models for 3d shape generation. arXie preprint arXiv:2210.0688 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 186-595,72 2018b.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVE /CFF International Conference on Computer Vision, pp. 5826 5835, 2021.
Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.096. 099447,.99977.. 2018.
Zi- Xi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song- Hai Zhang, Triplanc moets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 10324- 10335, 2024.
16

---

Published in Transactions on Machine Learning Research (10/2024)
A
Implementation Details
In this section, we provide more details about the implementation of our proposed AGG method.
A.1 Training
The model is trained with Adamn optimizer Kingma & Ba (2014). Rendering loss is enforced at 128 x 128 resolution. We first train the coarse hybrid generator for ten epochs. The learning rate is set to le 4 maximum, with a warmup stage for three epochs and followed by a cosine annealing learning rate scheduler. During the warmup epochs, we the loss weight of cchamfer to ten and Crendering to one. Later, we gradually reduce the weight of Lchamfer and increase the weight of Lrendering until the weight of Crendering is ten and that of Cchamfer is one. Then, we freeze the parameters for the coarse hybrid generator and only train the Gaussian super-resolution module. We use Lrendering and optimize for five epochs, with a learning rate set to le- 4. 4. Finally, we unfreeze all the parameters in both modules and train with rendering loss Lrendering for three epochs, with a learning rate set to le 1e-5.
For Crendering, we set w1 = 2 for all epochs. The parameters DINOv2 Oquab et al. (0023) vision encoder are kept fixed in all epochs. The input image is 256 x 256 resolution, and the DINOv2 encoder produces image features of 1 x 325 768. In each iteration, we render the generated 3D Gaussians into eight novel views where Crendering are calculated against the ground truth correspondingly. The warmup set consists of ten objects per class, resulting in 730 objects in total.
A.2 Inference
At inference time, given an in-the-wild image, we first extract image features using the DINOv2 encoder. Then, we send the features sequentially into the coarse hybrid generator and the Gaussian super-resolution network. Our model does not require access to the camera pose of the input image at inference time. This is because, during training, we ensure that the object is reconstructed in view space. In other words, the 3D asset and the input image are always aligned, and the supervision signals are rotated accordingly.
B Experiment Details
In this section, provide more details about how the metrics are calculated, and results from the baselines are obtained.
B.1 Metrics Calculation
Since the objects generated by the baseline methods are not guaranteed to be aligned with the ground truth 3D assets in the OmniObject3D Wu et al. (2023b) dataset, we cannot use pixel-wise reconstruction metrics to measure the generation quality. Moreover, the ill-posed nature of image-to-3D generation makes it harder to evaluate the 3D generation results quantitatively. Therefore, we follow the practice in previous works Xu al. (022b)); TTang et al. (2023a) and leverage CLIP distance to measure the semantic quality of the generation results. Specifically, for each object, we first normalize the scale of the object, and then use fixed set of cameras to obtain novel view renderings. Finally, calculate the average spherical distance between the CLIP Radford et al. (2021) image embedding of the input image and the novel view images. The runtime of our method is mcasured on the test set using an A100 GPU. The runtime of basclines is obtained from DreamGaussin Tang el al. 2023a).
B.2 Baseline Configurations
Point-E Nichol et al. 22022) consists of a transformer-based diffusion model that generates point clouds through sampling. We render the generated point cloud using Open3D Zhou et al. (2018) library and set the point size to 10. As for Ono-2345 Liu et al. (2023a), we render the generated mesh via the original Blender script provided by the authors. To render the results obtained by DreamGaussian Tang al. (2023a). we use the author's script to first export the Gaussians to mesh and then render the mesh.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Input
Point-E
TPS
Splatter Image
AGG
GT
Figure A: Novel view rendering comparisons against concurrent works Splatter Image Szymanowicz el al.) 20244h and TPS Zou et al. (2024). Our AGG model observes none of these testing images during training.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table A: Additional comparisons with concurrent methods regarding novel view image quality.
Method
Point-E One-2345
DreamGaussian
TPS
Splatter Image
Ours
PSNRT
14.73
18.87
14.87
18.41
18.36
SSIMT
0.7084
0.8147
0.7548
0.8107
0.8027
LPIPSI
0.3473
0.1906
0.2602
0.1729
0.1601
CLIP Dist 0.5139
0.3084
0.4293
0.3804
0.3624
0.3458
C Video Results
We provide more results and comparisons in the video. Please refer to the video for more details.
D Additional Comparisons with Concurrent Works
As suggested by the Reviewer JHwb, we've added comparisons with concurrent works that appear in CVPR 2024, TPS Zou et all (2024) and Splatter Image Szymanowicz et al. (0244). Please kindly note that these works are implemented independently.
To obtain the results from these concurrent methods, we use the authors' original implementations provided in their Github repos and render the generated 3D Gaussians directly. As shown in Tab. A we quantitatively compare our proposed methods with other methods. Our work shows competitive performance PSNR and SSIM, while outperforming most baselines in LPIPS and CLIP Distance that emphasize more on human perceptual quality.
We further provide side-by-side comparisons that qualitatively demonstrate our model's superiority. As shown in Fig. A Splatter Image Szymanowicz et al. (2024) tends to produce results that are more blurry, and TPS Zou et al. (2024) suffers at generating realistic back-view results and side-view results. In compar- ison, our method outperforms these comparison methods terms geometry and appearance quality on challenging novel views.
E Broader Impacts
Existing 3D content creation pipelines usually require tedious work that an expert must do. On the positive side, the advent of image-to-3D generation frameworks lowers the barrier and makes 3D content creation accessible to broader audience. Individuals and small businesses with limited resources can now produce high-quality 3D models without the need for extensive training expensive software. This democratiza- tion can spur innovation, allowing more people to participate in industries such as gaming, virtual reality, education, and product design.
However, these casily generated 3D models could lead to a saturation content, potentially devaluing the work of skilled professionals. Additionally, ethical considerations regarding the use of copyrighted or proprietary images to generate 3D models must be addressed. Ensuring that creators' rights are respected and that there is clear framework for the ethical use of this technology is crucial.
19


Reviews:
[{"review_id": "WtyeJGYxDA", "reviewer": "Reviewer_y2vy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper introduces the problem well and sufficiently discusses the positioning of this work with respect to existing methods.\n\n- The decoupled prediction is a well-designed approach, which can facilitate the single image reconstruction process by separately enforcing the photometric and geometric constraints.\n\n- The authors discussed the insights for each design sufficiently. Specifically, in addition to the methods in Sections 3.2 and 3.3, the authors also discuss the challenges of reconstructing explicit 3D Gaussian distributions during training.\n\n- The qualitative results are reasonably good.\n\nWeakness:\n- The method categorization is inprecise:\n  - It is confusing to categorize the method as \"amortized\". Amortized optimization is not a widely acknowledged term and basically means \"learning to optimize\". However, the method proposed in this paper is a straightforward feed-forward inference without any optimization get involved. Specifically, the gaussians' scales and rotations are manually fixed, the spherical harmonics coefficients and opacity are predicted via MLPs from interpolated tri-plane features. To get optimization involved, there are several simple ways including but not limited to 1. the spherical harmonic coefficients can be optimized instead of predicted from MLPs, 2. after the feed-forward prediction, the gaussian can be refined using optimization, 3. and/or predict the output from features/context of gaussian optimization process such as the gradient of each quantity. It is better to either get optimization involved or find a better term than \"amortized\".\n  - The method is not generative but deterministic. Specifically, the method does not adopt any generative model such as diffusion. Instead, it is better to replace \"Generative 3D Gaussians\" to \"3D Gaussians Generation\" since generation is a more general term that does not need to be generative model.\n- The tri-plane image features are interpolated before sending to the MLPs. More elaboration is necessary about this interpolation process since it is critical. In Section 3.3, the authors mentioned that the appearance of gaussian can be blurry due to inaccurate point cloud locations. However, the blur can be also contributed by the interpolation, i.e. the MLP decoded tri-plane image features can produce tri-plane image can generate image with good quality, but after the interpolation it can be blurry. On the other hand, attention/correlation can be also considered other than vanilla interpolation, which can help the each gaussian retrieve the most relevant tri-plane image features and reduce the blur.\n- Besides the rendering loss, the decoupled design can be better leveraged better by applying loss directly to the point cloud network and tri-plane image feature network. For example, we can constraint the point not to far from ground-truth surface and tri-plane image feature itself can produce high-quality tri-plane images.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper works on reconstruct 3D gaussian represented objects from a single image. To this end, the authors proposed to use DINOv2 to extract image features at each patch, and then the patch feature token are incorporated into two transformers separately: first, is the geometry predictor that generate sparse gaussian locations point clouds, while the gaussian rotation and scale are manually fixed; second is the texture predictor that generates a tri-plane image features as the texture presentation; finally, the gaussian opacity and colors are decoded from interpolated tri-plane features via MLP at each guassian locations. To get higher density 3D gaussians, the authors adopt a point-voxel layer based U-Net to further upsample the sparse gaussians. \n\nTo demonstrate the effectiveness of the proposed method, the authors conduct quantitative comparison with existing works on OmniObject3D dataset, and achieve comparable results on various metrics with magnitude smaller time. The authors also conduct ablation studies to justify the importance of the DINOV2 features and the point cloud super-resolution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness.", "broader_impact_concerns": "This paper works on reconstruct 3d models from a given single image, there is no broader impact concern as far as I can see.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6OiBM392VQ", "reviewer": "Reviewer_5dHr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The idea of generating 3D Gaussians from single image with a feed-forward neural network is intuitive, and it is good that the authors show this idea can work, while they provide the technical configurations needed to make the system work.\n2. The paper is overall well written and easy to understand, and the idea is clearly delivered. Despite that, some details are missing but I believe they can be easily added in a revision.\n3. The proposed technique, which is shown to be capable of generating 3D objects from a single image, can potentially have a significant impact in both academia and industrial application. Nevertheless, it would be more convincing if the experiments can be conducted beyond simple objects as shown in the paper.\n\n# Weaknesses\n1. Some technical details are missing or are not made clear in the manuscript, Most importantly, the proposed method represents geometry and texture separately. However in 3D Gaussian rendering each Gaussian should be associated with a color value. It is not clear to me how the color is attached. I would guess there is a color retrieval process from the triplane to a 3D point but there is no clear description.\n2. Despite quantitatively better numbers compared to existing method, the proposed method still renders pretty blurry methods, which is a bit far away from actually working in the real world. The method is mainly tested on simple and small objects, and there seems to be a lack of cross object category evaluation (namely all testing object types have been seen in the training). The results might not be encouraging enough for adoption in real applications.\n3. The novelty in the technical aspect is not that significant. The feed-forward 3D Gaussian prediction and upsampling has already been explored in [1]. The other techniques like triplane representation and using pretrained models for feature extraction has also be widely explored.\n\n[1] Hu, et al. \"Low Latency Point Cloud Rendering with Learned Splatting.\" CVPRW, 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to generate a collection of 3D Gaussians from a single image. Unlike existing works of 3D Gaussian splatting that requires per-scene optimization with ground-truth 2D views, the authors propose a feed-forward model that directly generates 3D Gaussian representations from a single image. Beyond existing work of 3D generation from single image, this work utilize the 3D Gaussian representation, allowing supervision in the rendering 2D space. The authors developed a coarse-to-fine geometry prediction pipeline and a separate texture triplane generation pipeline to achieve this goal. Experimental results demonstrate that the proposed method is capable of  generating more faithful and natural 3D structures than existing methods. In short, the main contribution of this paper is a proof of concept to generate 3D representation from 2D with a feed-forward 3D Gaussian generation neural network that shows better results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors are encouraged to make the following changes,\n1. Provide more details on how to combine the texture and geometry for rendering, as discussed in the first point of the weaknesses.\n2. Add some discussion on the rendering visual quality of the generated scene by this method. Why the produced results are more blurry than the input images? How can future works improve?", "broader_impact_concerns": "There is no ethical concerns on this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "j2pMi1Es5T", "reviewer": "Reviewer_JHwb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. **Novel approach**: The cascaded architecture for generating dense 3D Gaussians from a single input image is interesting, also making the 3D generation much faster.\n    - The hybrid representation that decomposes geometry and texture generation is particularly clever, allowing for more stable joint optimization of 3D Gaussian attributes.\n2. **Impressive speed improvement**: The method achieves orders of magnitude faster generation times compared to existing methods while maintaining competitive quality.\n3. **Thorough ablations**: The paper includes comprehensive ablation studies clearly demonstrating contribution of different method components.\n\n------------------\n\n## Weaknesses\n1. **Dependency on pre-trained encoder**: The reliance on a pre-trained DINOv2 image encoder may limit the method's ability to generalize to out-of-distribution images. It would be interesting to see how the method performs with different pre-trained encoders or with end-to-end training.\n2. **Comparison with many single-image-to-3D works is missing**: I think the authors should quantitatively and qualitatively compare with the following single-image-to-3D methods for a complete comparison:\n    - \"Splatter Image: Ultra-Fast Single-View 3D Reconstruction\", CVPR 2024\n    - \"Wonder3D: Single Image to 3D using Cross-Domain Diffusion\", CVPR 2024\n    - \"Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers\", CVPR 2024\n    - The following two methods should also be compared but are optional as one of them is only on arXiv and the second is from ECCV 2024:\n        - \"GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation\", Xu et al.\n        - \"LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation\", ECCV 2024\n3. **Qualitative results don't look good**: This might be a subjective opinion, but for the examples shown in Figure 4 don't look impressive. Especially compared to Point-E (which I realize is a point-cloud method), AGG loses a lot of detail and results in blurry renderings.\n    - I also think that the methods I listed above might give better or equally good looking outputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* This paper introduces AGG (Amortized Generative 3D Gaussians), a feedforward method for generating 3D Gaussian representations from a single input image without per-instance optimization. \n* The key idea is to use a cascaded architecture that first generates a coarse hybrid representation and then upsamples it to produce dense 3D Gaussians. \n* The authors propose decomposing geometry and texture generation using separate transformer networks. A UNet model with point-voxel layers is then used to upsample the coarse 3D Gaussians. \n* The method demonstrates competitive performance compared to optimization-based and sampling-based baselines while being orders of magnitude faster.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate if the authors can address Weaknesses #1 and #2 mentioned above and provide thorough results to address the same.", "broader_impact_concerns": "It would be nice if the authors can add a Broader Impact Statement addressing potential ethical implications, including:\n* Potential job displacement in industries relying on manual 3D modeling.\n* Accessibility and democratization of 3D content creation, including both positive and negative implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors presented a generative 3D Gaussian framework (named AGG) to address the problem of 3D from a single image. The 3D Gaussian location generation and other appearance attributes are decomposed, and a pipeline of coarse representation generation followed by upsampling is proposed. Experimental results against other 3D Gaussian methods show the effectiveness of the proposed method. The paper is generally well-written.

Three expert reviewers were invited to review the paper, and both strengths and weaknesses were raised. The paper received two Accept and one Reject recommendations. The major concerns include the use of the term "amortized", the decoupled design, unclear technical details, novelty, and experiments (e.g. missing comparison to related work, and poor qualitative performance). After the authors' response and revision, most concerns were addressed, with a remaining one about the potential misleading usage of the term "amortized". With back-and-forth discussions and revision, this became more clear and the authors promised to address the misleading issue by further clarification and clear definition. On the other hand, the reviewers and AE found the techniques and results presented in this paper might be of interest to a group of audiences (e.g. the 3D generation community) in TMLR. As a result, the AE would be happy to recommend Accept (with minor revision).

The authors are suggested to clearly clarify the potential misleading usage of the term mentioned above, throughout the whole paper. If possible, please also try to avoid using the term "amortized" too much and not highlighting it, as it is not considered to be a critical point of the proposed method but potentially lead to misleading.
Decision: Accept (poster)