Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day
Authors: Yifan Jiang
Hao Tang
Jen-Hao Rick Chang
Liangchen Song
Zhangyang Wang
Liangliang Cao
Venue: ICLR 2024 poster (2024)
Keywords: Novel View Synthesis
3D from Single Image
Efficient Training
Primary area: generative models

Abstract: The task of novel view synthesis aims to generate unseen perspectives of an object or scene from a limited set of input images. Nevertheless, synthesizing novel views from a single image remains a significant challenge. Previous approaches tackle this problem by adopting mesh prediction, multi-plane image construction, or more advanced techniques such as neural radiance fields. Recently, a pre-trained diffusion model that is specifically designed for 2D image synthesis has demonstrated its capability in producing photorealistic novel views, if sufficiently optimized with a 3D finetuning task. Despite greatly improved fidelity and generalizability, training such a powerful diffusion model requires a vast volume of training data and model parameters, resulting in a notoriously long time and high computational costs. To tackle this issue, we propose Efficient-3DiM, a highly efficient yet effective framework to learn a single-image novel-view synthesizer.  Motivated by our in-depth analysis of the diffusion model inference process, we propose several pragmatic strategies to reduce training overhead to a manageable scale, including a crafted timestep sampling strategy, a superior 3D feature extractor, and an enhanced training scheme. When combined, our framework can reduce the total training time from 10 days to less than 1 day, significantly accelerating the training process on the same computational platform (an instance with 8 Nvidia A100 GPUs). Comprehensive experiments are conducted to demonstrate the efficiency and generalizability of our proposed method.

Published as a conference paper at ICLR 2024
EFRICIENT-3DIM: LEARNING A GENERALIZABLE SINGLE-IMAGE NOVEL- S SYNHHSSIZER IN ONE DAY
Yifan Jiang! Hao Tang', Jen-Hao Rick Chang', Liangehen Song',
Zhangyang Wang', Liangliang Cao! Apple, UUniversity of Texas Austin
fyifanjiang37, atlaswangy@utexas. (hao.tang, jenhao.chang, wiwggchen.onng, 1icao
ABSTRACT
The task of novel view synthesis aims to generate unseen perspectives of an ob- ject or scene from a limited set of input images. Nevertheless, synthesizing novel views from single image remains significant challenge. Previous approaches tackle this problem by adopting mesh prediction, multi-plane image construction, or more advanced techniques such as neural radiance fields. Recently, pre- trained diffusion model that is specifically designed for 2D image synthesis has demonstrated capability producing photorealistic novel views, sufficiently optimized with 3D finetuming task. Despite greatly improved fidelity and gen- cralizability, training such a powerful diffusion model requires a vast volume training data and model parameters, resulting in notoriously long time and high computational costs. To tackle this issue, we propose Efficient-DDiM, a highly efficient yet effective framework learn a single-image novel-view synthesizer. Motivated by our in-depth analysis of the diffusion model inference process, we propose several pragmatic strategies reduce training overhead to a manageable scale, including crafted timestep sampling strategy, a superior 3D feature ex- tractor, and an enhanced training scheme. When combined, our framework can reduce the total training time from 10 days to less than 1 day, significantly accel- erating the training process on the same computational platform (an instance with 8 Nvidia A 100 GPUs). Comprehensive experiments are conducted to demonstrate the efficiency and generalizability of our proposed method.
INTRODUCTION
Creating an immersive experience with free-viewpoint interaction, particularly from just one image, has always been a captivating and important area of focus. To achieve realistic interactive visuals, the process can start by constructing precise 3D model, such mesh or point cloud, from the given image, followed by rendering the scene from a new perspective. However, this approach frequently misses out on siew-dependent effects and may appear unnatural. Lately, neural rendering techniques, such as neural radiance field (NeRF) (Mildenhall et al. 2020) and scene representation networks (SRN) (Sitzmann et al. 2019), have prevailed over all other approaches and become the crucial step in reaching interactivity and realism. Instead of constructing these approaches learn an implicit representation by modeling a 3D scene as a continuous function.
explicit representation,
Although NeRF-like models produce compelling details, they impeded by stringent requirement-tte vanilla NeRF approach requires hundreds of posed images for training and gener- ates photorealistic images both rendered objects and real-world scenes. Several follow-up works reduce NeRF's training views to as few as three views, by using geometry regularization (Niemeyer al.] 2022), Structure-from-Motiio (SIM) initialization (Deng al., (022). or adaptive positional encoding (Yang et al. 2023). However, these approaches cannot be easily extended to support single input view. et (2021) first propose to train a NeRF that can render images from novel
"This work performed while Yifan Jiang interned Apple.

---

Published as a conference paper at ICLR 2024
viewpoints using single input, yet it only works simulated scenes with simple geometry (Chang jet al. 2013). Xu et al. (2022) further extends it to in-the-wild scenes and enables novel view render- ing on arbitrary objects. However, it only renders views from small range of angles.
Recently, Watson et al. 2022) proposed the 3DMM pipeline that treats the single-image novel view synthesis conditional generative process. By training diffusion model conditioned image together with the target camera pose relative to the input image, they can synthesize novel views of simple synthetic objects in high fidelity. Liu al. (202b)) further extend the method to real-world objects by finetuning a texl-to-image model to learn geometry priors using 3D simulated objects. Their proposed zero 1-to-3 method can perform zero-shot novel view synthesis in an under- constrained setting, synthesizing view for any in- in-the-will image from any angle.
input
Despite their capability generate high-fidelity images, training diffusion models for single-image novel view synthesis requires a notoriously long time and prohibitive computational costs. For ex- ample, the 3DIM model (Watson et al. 2022) contains 471M parameters; the zero 1-to-3 method (Liu et al.) 2023b) uses a 1.2B-parameter Stable Diffusion model (Rombach et al.) 2022) and 800k simulated objects from the Objaverse dataset (Deitke et al. 2023). Training the model requires 10 days on single instance with eight Nvidia ATOO GPUs In comparison, training typical image classifiers (He et al., 2016 Xie et al., 2017, Huang et al., 2017) on the ImageNet dataset (Deng et al. 2009) only takes about 1 day on similar platform. A straightforward way to shorten the training procedure of diffusion models is to adopt a larger batch size, however, it further increases the required computational resources. Given the importance of diffusion modes and their applica- tions in 3D vision, an enhanced training approach would greatly aid researchers in speeding up the development process, thus propelling the advancement of this direction.
The major goal this work is to trim down the training time without spending more costs on the total training resources (e.g.. taking large-batch via distributed system). We present Efficient-3DiM. a refined solution developed to enhance the training efficiency of diffusion models for single-image novel view synthesis. This paper delves into the details and design choices of the framework. Our approach builds upon three core strategies: a revised timestep sampling method, the integration of a self- self-supervised vision transformer, and an enhanced training paradigm. Together, these elements contribute to noticeable improvement in training efficiency. With the proposed enhancements, our model requires only a single day to train. Compared to the original zero 1-to-3, we achieve a 14x reduction in training time in the same computation environment. The significantly reduced training time enables future research to be rapidly iterated and innovated.
Our major contributions are encapsulated as follows:
We introduce a novel sampling strategy for diffusion timesteps, deviating from the standard uni- form sampling, and offering optimized training.
Our framework integrates a self-supervised Vision Transformer, replacing the conventional CLIP encoder, to better concatenate high-level 3D features.
Comprehensive experiments have been conducted to demonstrate the effectiveness of our method on various benchmarks. As a result, Efficient- -DiM reduces the training time from 10 days less than 1 day, greatly minimizing the training cost to a manageable scale.
2 RELATED WORK
Diffusion Model. Recent breakthroughs in diffusion models (Sohl-Dickstein et al. 2015 Song
& Ermon 2019 Ho et al. 2020) have displayed impressive outcomes in generative tasks. These
models, as advanced generative tools, produce captivating samples using a sequential denoising method. hey introduce a forward mechanism that infuses noise into data sets and then reverses this mechanism to restore the initial data. Following this, numerous research efforts (Karras et al.) 2022
Saharia et al. 2022 Dhariwal Nichol 2021 Nichol et al. 1 Rombach et al.) [022 Ramesh et al. 2022 Chen 2023) have been directed toward enhancing the scalability of diffusion models and speeding up the sampling process for improved efficiency. Notably, the LDM model (Rombach eet al. 2022) (also known as Stable Diffusion) stands out, minimizing computational demands by
executing the diffusion method on a nower-resolution latent space, making the training of a text-to- image model scalable to the billion-scale web data. Several other investigations have also adapted
2

---

Published as a conference paper at ICLR 2024
pred No
Reference
Timesteps
Figure 1: Inference visualization of a typical diffusion-based novel-view synthesizer (Lu et al. 2023b). We run total of 200 steps following Equation 3 and show the intermediate inference progress of "backpack" under different timesteps.
the diffusion methodology to diverse fields, such as music creation (Huang et al. 2023), reinforce-
ment learning (Wang et al. 2022). language generation (Li et al. 2022). text-to-3D object (Poole
et al. 2022) ). novel view synthesis Watson et al. [022]] Xu et al.) 2023), video generation (B (Blatmmann
al. 2023), and more. In addition to generation. newer research (Meng al. 2021 Zhhng &
Agrawala 2023
Hertz et al.] 2022 Mokady et al. 2022 Brooks et al., [022] Parmar et al.) 2023]
Goel al. 2023 Khachatryan al. 2023) also extends the capabilities of diffusion models to tasks
related to image and video editing.
Efficient Training for Generative Models. Existing research efforts have been undertaken to plore various aspects including data efficiency, model efficiency, training-coss efficiency, and infer- efficiency. To address the mode collapse issue under the setting of limited training data, /Zhao iet al. (2020) propose a differentiable augmentation strategy that only takes 10% training data to train Generative Adversarial Network (GAN) Goodfellow al. (2014) to achieve similar performance. Nevertheless, it is unclear whether this method achieves shorter training time. In addition, since low-resolution images are typically more available than their high-resolution counterparts, Chail et al. (2022) propose an any-resolution framework. This framework merges data of varied resolu- tions to facilitate GAN training, lessening reliance on high-resolution input. However, its impact on training costs and its adaptability for diffusion models have not been thoroughly examined. Another line research addresses the memory issue during the training stage. Lin et al. (2019) was the pi- oneer in integrating the patch-wise training approach into GAN frameworks. However, due to their discriminator having to amalgamate multiple generated patches, its memory-saving efficiency is not significant. Building on this, [ee al. (2022) applied the patch-wise training to the implicit rep- resentation framework Skorokhodov et al. (2021). While this method does conserve GPU memory, it compromises the quality of the generated samples. Several researchers have sought mitigate the substantial training and inference expenses tied to diffusion models. Rombach et al. (202)), for latent space, bypassing the pixel space, which effectively cut
instance, opted to apply diffusion down both training and inference costs. Other studies, like those of (Lu et al.) [2022a b; Song et al.) 2020a Bao et al. 2022b a). delved into rapid sampling techniques during inference. However, these do not inherently accelerate the training phase.
Novel View Synthesis from Single Image. Over recent years, the challenge of deriving 3D worlds from a singular image has gained traction (Rombach et al. 2021 Ren & Wang 2022 Wiles
et al. 2020 Rockwell et al.) 2021). Traditional methods tackled this problem by first employing
a monocular depth estimator (Ranfil et al. 2020)) to predict the 3D geometry and then utilized
multi-plane images (MPI (Shih et al. [2200] Jampani al., 2021 point clouds Mu et al. (2022) to craft artistic renderings. Furthermore, any gaps or inconsistencies seen from new viewpoints were rectified using previously trained advanced neural network Yu et al. (2019). Yet. despite their efficacy. these methodologies have shortcomings. The depth estimator can be unstable, and the rectification techniques might introduce flaws that detract from the realism of the images. Follow-up approaches (Yu et al.] [021] [in et al., 2022) propose to learn 3D priors from the 3D objects dataset and directly predict a 3D representation from a single input during the inference time. However, these methods face significant quality degradation when processing real-world images, attributable to domain differences. Xu et al.] (2022) proposes to generate novel views by training a neural radiance field that only takes single RGB-D image. Nevertheless, it needs high-qualit depth as the input and only renders new views from small range of angles. Otherworks (Xu et al.) 2023] Tang et al. 2023) get rid of depth input by adopting guided diffusion loss from a pre-trained text-
3

---

Published as a conference paper at ICLR 2024
to-image model. Although these methods render high-fidelity novel views from 360 degrees, they generally require per-scene training, which takes more than several hours.
More recently, Watson et al. (2022) propose to solve single-image novel view synthesis by using conditional generative model. They train a diffusion model as an image-to-imagee translator, using the current view as the input and predicting novel view from another angle. To extend this pipeline to mn-dhe images, [Liu et al. (2023b) choose to borrow the prior knowledge from 2D text-to- image model Rombacc et al. (2022). Several other works (Liu et al., 2023a] Shi al. 2023 Liu et al., (223c) also consider multi-view information to enable better 3D consistency. Our proposed method mainly follows this line of works (Watson et al. [022] Liu al. 2023b). but significantly
reduces its training cost without performance degradation.
METHOD
3.1 PRELIMINARIES
Diffusion Models. Denoising Diffusion Probability Models, simply called diffusion models, are class of generative models that learn convert unstructured noise real samples. It produces images by progressively reducing noise from Gaussian noise p (xx) = (0, I), reshaping it to match the target data distribution. The forward diffusion step, represented by ((2,/22,
intro- duces Gaussian noise to the image Zt. The marginal distribution can be written as: (z; æo) = N (a:zo. I) where ag and or are designed to converge to (O, when is at the end of the for- ward process (Kingma et al. 2021 Song al. 2020b). the reverse process p(ar,-1 1 |2g)) diffusion models are designed noise estimators Eg (o t) taking noisy images input and estimating the noise. Training them revolves around optimizing the weighted evidence lower bound (ELBO) (Ho
al. 2020 Kingma et al. 2021:
w(t) |leg (a,20 + o,E;; - -
(I)
where e is drawn from (O, I), the timestep ł follows an unifrom sampling ((1, 1000). and w(t) serves as a weighting function with w(t) = 1 showing impressive results. In the inference stage, one can opt for either stochastic (Ho et al. 2020) or deterministic approach (Song et al. (020aa. By selecting Dr from (O, I). one can systematically lower the noise level, culminating in a high- quality image after iterative refinement.
Diffusion Model as Novel-view Synthesizer. The standard diffusion model serves as noise- to-image generator, yet a modified one can be extended to an image-to-image translator, using conditional image the reference. By adopting a pair of posed images {xg. Øo} E IR Mxxxx from the same scene for training. image-to-image diffusion model can take the image, To. as the input conditioning to predict the image from a different view, To. approximating a single-image Liu et al. 2023b). Specifically, the training objective
novel-view synthesizer (Watson et al. 2022 of diffusion models becomes:
E (() [[o (atzs o,e;t; C (To, R,T)) - -c1
where C is a feature extractor, (R, T} are the relative rotation and translation between To and Zo. Since the task of single-image novel view synthesis is severely cnder-constrained, the training requires a huge dataset consisting of diverse real 3D objects. The current largest open-sourced 3D dataset Objaverse (Deitke et al. 2023) only contains 800k synthetic objects, largely behind the 5 billion in-the-wild yet annotated 2D image dataset (Schuhmann et al.) 2022), not to mention the domain gap between synthetic and real samples. Liu et al. (2023b) proposes Zero 1-to-3 by initializing from pre-trained 2D text-to-image diffusion model, successfully generalizing to real scenarios with wn-the-wild images. Although Zero 1-to-3 diminishes the need for 3D training data and reduces the training cost well, it still requires laborious effort to converge. For instance, the official training scheme requires 10 days to run on Nvidia-All A100 GPUs Our proposed method is built top of the framework but further reduces its training time manageable scale.
4

---

Published as a conference paper at ICLR 2024
uniform sampling
0.005
uniform sampling
meat-1000, 21d-200
0.007
mean-1000, sid-400
mcsn-5000 sdd-200
memn-1000 sid-200
Probability Donsity
mem-1500 std-290
Probabiitt Deneity
0.006
mean- (000, std-100
0.005
0.004
0.003
0.002
0.001
0.0000
0.000
200
400
000
800
1000
200
400
600
800
1000
Timesteps of Diffusion Models
Timestops of Diffasion Models
Figure 2:
Gaussian sampling under different mean and std values. The left figure shows how the mean factor could control the sampling bias towards different stages of timestep t, and the right figure shows how much the bias could be applied via controlling the value of std. The distribution adjusted to ensure the integral of the probability density function ranges from [0, 1000] 1.
3.2 MODIFIED SAMPLING STRATEGIES
We start by analyzing the inference stage of the current state-of-the-art novel-view synthesizer (Liu et al. 2023b). First, we adopt denoising diffusion implicit models (Song et al. 2020a):
Lt-1 = var-1
ar
F
+ ar-1 - of F) + 0rE
(3)
predicted mo"
F= eo(0iit; t; C' (2o, R, T)),
(4)
where (0, I) is the standard Gaussian noise, and we let og to be 1. Different choices of values result in different generative processes, under the same model ég. When or O, the generative process degrades to a deterministic procedure and the resultant framework becomes an implicit probabilistic model (Mohamed & Lakshminarayanan 2016). Here we do not pay attention to the stochasticity of the inference process. Instead, we analyze the output of eg the "predicted 2tp" to better understand the learned 3D) knowledge.
From Figure [ we observe that the structure and the geometry of the predicted "backpack" are constructed the early stage of the reverse process, while the color and texture are refined in the late stage. Note that the adopted novel-view synthesizer is initiated from pre-trained text-to-image diffusion model, the knowledge of refining texture details should be also inherited. This observa- tion prompts to reconsider the prevailing uniform sampling methodology used for the diffusion timestep in the training stage. Instead, we advocate for the implementation of a meticulously de- signed Gaussian sampling strategy.
worth mentioning that the potential usage of different sampling strategies has been explored before (Chen) 2023 Karras al. 2022). However, work distinguishes itself, since the major training phase of our framework is essentially characterized as finetuning paradigm. This stands in contrast the predominant works which generally focus on training diffusion models from scratch. When juxtaposed with the uniform sampling approach, Gaussian sampling offers a distinct advan- tage: it permits the introduction of sampling bias, thus conserving efforts that might otherwise be expended on superfluous segments of the training. We show several typical Gaussian sampling strategies with different mean and std in Figure By adopting such a different sampling strategy, we are able to adjust the sample probability under different timesteps. Consequently, the corresponding timesteps that are still in development should receive augmented opportunities for updates, while the other parts get lower probability, but not zero probability. From the distribution visualization. some of these Gaussian distributions satisfy our goal, such as (mean = 1000 and std = 200} as depicted in Figure 2 Adopting slightly different hyperparameter may also reach similar goal, where we simply choose an effective one by empirical observation. We show detailed experiment to demonstrate its effectiveness in Section 4.1.1
5

---

Published as a conference paper at ICLR 2024
Self-supervised Vision Transformer
R,
Nommlization
Layer Norm
T
Relative Pose
Normalization
Concat
UNet Denolser
Figure 3: Main inference pipeline of Efficient-3DiM framework. We finetune the Stable Diffu- sion (Rombach et al. 2022)) on the Objaverse dataset (Deitke et al. 2023) but replace the original CLIP encoder with self-supervised vision transformer (DINO-v2 (Oquab et al., (023)). Rather than only adopting the "[CLS]" token from the the output of reference feature extractor, we amalga- mate multi-scale representations into different stages of the UNet denoiser.
3.3 3D REPRESENTATION EXTRACTION AND AMALGAMATION
The task of predicting novel views from a single input view is highly ill-posed, since singular 2D image could be projected from totally different 3D representations. Consequently, the optimization of diffusion models is expected to ensure that the 2D reference could be back projected to the latent 3D space that aligned with human perception. Predominant works either directly concatenate the 2D condition with original noisy input and train the denoiser to capture the 3D geometry (Watson al., 2022), or integrate an auxiliary CLIP image encoder (Radford et al. 2021) to capture the corresponding information (Liu et al. 2023b, Shi et al. 2023 Liu et al. (023c).
However, is noteworthy that the adopted pre-trained Stable Diffusion's foundational training task text-to-image generation, exhibits marginal association with the inference of 3D representa- tions. Meanwhile, the CLIP encoder is primarily designed for the purpose aligning with text embedding, showing poor capability dense prediction, especially when compared with other im- age encoders instructed through self-supervised paradigms, E.g., DINO-v2 (Oquab et al. 2023) We show apple-to-apple comparison between CL.P and DINO-v2 encoder in Figure 4 and Through the Principal Component Analysis (PCA) and correspondence matching visualization, the spatial representations derived via the DINO-v2 encoder show more consistent semantic information than those extracted by the CLIP encoder.
Motivated by this reflection, advocate for the incorporation of multi-scale representations pro- duced by the DINO-v2 encoder, spanning from the output of its shallow layer to the deeper layer. Sequentially, these representations are concatenated and then converted fit the target channel by single linear layer. After that, we conduct several different spatial interpolation processing, and amalgamate the resultant representations into different stages of UNet's encoder, via an additive operation. We also use cross-attention layer to combine the predicted "[CLS]" token with the intermediate feature the UNet. Details are shown in Figure
3.4 ENHANNCED TRAINING PARADIGM
Integrating the above methodologies, we accelerate the training process by large margin. Building on top of this, we continually introduce several engineering enhancements to optimize our frame- work, culminating in its most efficient version. To start with, we transition from the standard full- precision training scheme
scheme, which helps save about 40% training time but leads to numerical error and causes instability. This is mitigated by adding another Layer Normalization (Ba et al. 2016) before sending the DINO-v2 feature the diffusion model. Si-
16-bit

---

Published as a conference paper at ICLR 2024
Image Pair
CLIP
DINO-v2
Figure 4: Visualization of Patch Matching. We conduct patch-level feature-matching between images under different viewpoints. Patches extracted by DINO-v2 produce better feature-matching results. More details can be found on Appendix 10
Image Pair
CLIP
DINO-v2
Figure 5: Visualization of Principal Component Analysis (PCA), For each object, we use the alpha channel remove the white background and use PCA condense the feature dimension to 3 on each view separately. We then visualize the features of patches by normalizing them to O- 255. Compared the CLIP encoder, DINO-v2 exhibits a more coherent semantic understanding of identical regions, because the same parts under different viewpoints are matched regardless of poses. More details can be found on Appendix B
multaneously, outputs from the DINO-v2 encoder are archived the disk, given that DINO-v2 does not participate in the backpropagation chain, resulting in a 10% training time reduction further. Lastly, we replace the formal constant learning rate scheduler with cosine decayed strategy. As result of these adjustments, we achieved a total of 50% further reduction of training time, without performance degradation. Detailed training hyperparameters are included in Appendix A
EXPERIMENTS
directly adopt Dataset We employ the recently released Objaverse (Deitke et al.) 2023) dataset for training. The dataset contains 800k three-dimensional objects, meticulously crafted by an assemblage of over 100k artists. We adopt 792k samples as the training set and 8k other samples for validation. Each object within the dataset has 12 viewpoints sampled. To make fair comparison, the resultant renderings produced by Liu et al. (2023b). During the training phase, a pair of views with 256 x 256 resolution is sampled from each object to construct R). We randomly select one as the conditional reference and another as the ground truth view to be predicted. From this duo, the associated relative viewpoint transformation represented as (R, T).
image duo, designated (z,
4.1 QUANTITATIVE EVALUATION
4.1.1 ABLATION ON DIFFERENT SAMPLING STRATEGIES
As discussed in Section 3.2, we are motivated to switch from the standard uniform sampling of dif- fusion timestep & to modified sampling strategy that allows us conserve efforts on superfluous
7

---

Published as a conference paper at ICLR 2024
0.55
1.PPPS sonre on the validation set
unifoom sampling
LPIPS score on the validation set
0.34
uniform sampling
0.50
mean- (000, std-200)
1000, std=100
mean-700, std-200
mean- std-200
mican-500, std-200
0.30
mean-
std-300
meun-300, std-200
0.28
moan- std-400
mewn-0, std-200)
meann 1000, std=500
0.35
0.26
0.30
0.24
0.22
0.25
0.20
0.20
0.18
10000
15000
20000
50000
15000 20000
30000 35000
Training Steps
Training Steps
Figure 6: Ablation study on different choices of mean and std. The left figure shows different settings by changing the mean factor, and the right figure shows the results by changing the value of sld. We report the LPIPS scores on the Objaverse validation set for all approaches.
segments of the training. Our initial assumption is built upon the observation from the diffusion model's inference process, where the noisy stage should receive augmented opportunities for up- dates. To verify
conduct experiments by controlling the value of mean and standard deviation. We first fix the standard deviation to be 200 and switch the mean value from a group of {1000, 700, 500, 300, 0}. as shown in Figure 2 left. After that, we keep the mean value 1000 and test several options for standard deviation, from group of {100 ~ 500}, as depicted in Figure 2 right. The results demonstrate our assumption more correlated with the late stage of diffusion steps (which takes more noisy input). Meanwhile, results also suggest that other stages should be still updated entirely omitted. For instance, in the setting of mean = 1000 and sld 100, the early stage of diffusion steps have almost zero probability of being updated, and the resultant framework shows worse results the validation set.
the emergence of novel view synthesis ability is
reduced frequency, rather than
4.1.2 ABLATION STUDY ON DIFFERENT COMPONENTS
We conduct an ablation study to demon- strate the effectiveness of our proposed components, as shown Table The of- ficial checkpoint of the baseline method, Zero 1-to-3 Liu et al. (2023b). requires 105,000 training iterations. Contrarily, in our experimental configuration, we constrict each counterpart limited training scenario, allowing only 20,000 updates. Given these constraints, the baseline approach exhibits subpar per- formance on the validation set, only reach-
Table Ablation study on different components. We train each method 20,000 steps and then report the
L.IPS and MSE score on the Objaverse validation set.
Gaussian DINO
Amalgamation
LPIPS.L
MSEL
X
X
0.236
0.187
x
x
0.205
0.142
resource-
/
X 0.187
0.136
x
/
0.191
0.144
/
/
0.171
0.128
ing 0.279 LPIPS score and 0.212 Mean Squared Error (MSE). By replacing standard uniform sam- pling with Gaussian sampling {mean- = 1000, std = 200), we observe improvements in both the LPIPS and MSE metrics. Next, attempt replace the CLIP encoder with the DINO-v2 coder, or amalgamate the spatial feature from these encoders with the UNet denoiser. Either of them brings performance gain. Finally, we add all these modifications the framework, reaching the best performance by obtaining a 0.171 LPIPS score and 0.128 MSE score on the validation set.
Meanwhile, present the curve of validation loss w.r.t. the training time, shown in Figure 7 Built upon the aforementioned best setting (purple line), we further apply the proposed enhanced training paradigm (brown line). Compared the Zero 1-to-3 baseline, the final resultant framework reaches up to 14x speedup.
4.2
VISUAL COMPARISON ON IN-THE-WILD IMAGES
We examine the performance of the obtained model on in-the-wild images. As shown in Figure if we reduce the training iterations of Zero 1-to-3 from the original 105k steps only 200k or 500k
8

---

Published as a conference paper at ICLR 2024
Baseline
LPIPS score on the validation set
0.24
Gaussiun Sampling
Feature Amalgamation
Enhanced Training Paradigm
0.20
0.18
x speedup
0.16
5
10
12
Training Days on 8 A100 Instance
Figure 7: Ablation study on different settings. To verify the effectiveness of different components in our framework, conduct three experiments for each setting with different random seeds and report the LPIPS score (Zhang et al. 2018) on the Objaverse validation set.
Input
Zero 1-to-3 20k sleps
Zero 1-to-3 50k steps
Zero 1-to-3 105k steps
Efficient-DDiM 20k steps
Figure 8: Visluazation comparisons on in-the-wild images. For every object, we turn the camera 90 degrees counterclockwise around the azimuth axis. More visual comparisons can be found in Appendix Figure I0
steps, it fails to produce multi-view consistent visual outputs. However, the proposed method can generate photorealistic and geometrically reasonable novel views under the same training steps.
5 CONCLUSION AND FUTURE WORKS
In this paper, we introduce Sificient-3DiM, an efficient framework for single-image novel view syn- thesis through diffusion models. its core, Efficient-3DiM integrates three pivotal contributions: modified sampling strategy departing from traditional uniform sampling, integration of a self- supervised Vision Transformer replacing the conventional CLIP encoder, and an enhanced train- ing paradigm. Comprehensive evaluations underscore the effectiveness of proposed components, successfully slashing training time from 10 days to a single day while preserving computational resource integrity. Similar to other approaches, there remain avenues for exploration and improve- ment. One such avenue could involve further refinement of the multi-view consistency, ensuring both efficiency and better fidelity. Going forward, we will explore further ideas in this direction.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: net.16607065555 2016.
arXiv preprint
Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206. 07309, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytie-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint wrXiv:221.006500 2022b.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
pp.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint UrXiv:22/1.09800 2022.
Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution train- ing for high-resolution image synthesis. arXiv preprint arXiv:2204.07556 2022.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 112.00 2015.
Ting Chen. On the importance of noise scheduling for diffusion models. arXiv:2301.1097, 2023.
arXiv preprint
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-1355, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, leee, 2009.
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-12891, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair-diffusion: Object-level image editing with structure-ad aoneture-and-apeen and-appearance paired diffusion models. arXiv preprint 23X3 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. comppt-to-prompp image editing with cross attention control, arXiv preprint arXiv:2208. 016206 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
10

---

Published as a conference paper at ICLR 2024
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Qingqing Huang. Daniel S Park, Tao Wang, Timo Denk, Andy Ly. Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917, 2023.
Varun Jampani, Huiwen Chang, Kyle Sargent, Abhishek Kar, Richard Tucker, Michael Krainin, Dominik Kaeser, William T Freeman, David Salesin, Brian Curless, al. Slide: Single image 3d photography with soft layering and depth-aware inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1518-12527, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 265767655-6657, 2022.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-7ero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303. 13439, 2023.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707, 2021.
Namwoo Lee, Hyunsu Kim, Gayoung Lee, Sungjoo Yoo, and Yunjey Choi. Memory efficient patch- based training for inr-based gans. arXiv preprint arXiv:2207.01395, 2022.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205. 14217, 2022.
Chich Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzon, Chen. Coco-gan: Generation by parts via conditional coordinating. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4512-4521,20 2019.
Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung- Yi Lin, Yi-Chang Shih, and Ravi Ramamoor- thi. Vision transformer for nerf-based view synthesis from single input image. arXiv preprint arXiv:2207.05736, 2022.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306. 16928, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. arXiv preprint arXiv: 11328, 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisten images from a single-view image. arXiv preprint arXiv:2309.033: 03455,, 2023c.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:2211.01095, 2022b.
Chenlin Meng. Yang Song. Jiaming Song. Jiajun Wu. Jun-Ye Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. CoRR, abs/2108.0100 2021. URL htt.tttts////aaxttt www.exiwv.r..rooeo abs/ 01073
Ben Mildenhall, Pratul P. Srinivasan. Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.

---

Published as a conference paper at ICLR 2024
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv:1610.0344 2016.
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.0979, 2022.
Fangzhou Mu, Jian Wang, Yicheng Wu, and Yin Li. 3d photo stylization: Learning to generate styl- ized novel views from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16273-16282, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffussin models. arXiv preprint arXiv:2112.1074 2021.
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5480-5490, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2344.07193, 2023.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. /4988, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 22X::2204.006125, 2022.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transac- lions paltern analysis and machine intelligence, 2020.
Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing a consistent long-term 3d scene video from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3563-3573, 2022.
Chris Rockwell, David F Fouhey, and Justin Johnson. Pixelsynth: Generating 3d-consistent ex- perience from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14104-14111 2021.
Robin Rombach, Patrick Esser, and Bjöm Ommer. Geometry view synthesis: Transformers and no 3d priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 13356-143666 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494 2022.
12

---

Published as a conference paper at ICLR 2024
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context- aware layered depth inpainting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8028-8038 8038, 2020.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-aware neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 10753- 10764, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023.
Xiao Fu Yuxin Wang Jiawei Ren Liang Pan Wayne Wu Lei Yang Jiaqi Wang Chen Qian Dahua Lin Ziwei Liu Tong Wu, Jiarui Zhang. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06/93 2022.
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mo- hammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint arXiv:221000463 2022.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7467-7477, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 1492-1500, 2017.
Dejia Xu, Yifan Jiang, Peihao Wang. Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image. In European Conference Computer Vision pp. 736-753. Springer, 2022.
13

---

Published as a conference paper at ICLR 2024
Dejia Xu, Yifan Jiang, Peihao Wang. Zhiwen Fan, Yi Wang. and Zhangyang Wang. Neuralift-3600 Lifting an in-the-wild 2d photo to object with 360deg views. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4479 44899 2023.
Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8254-8263, 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578 4587, 2021.
Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas $ Huang- Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4471-4480. 2019.
Lymin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprint arXhv:2302.0554,, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018
Shengyu Zhao, Zhijian Liu, Ji Lin, Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in neural information processing systems, 33:7559-7570, 2020.
A DETAILED TRAINING SCHEME
All of our experiments are conducted on 8 Nvidia-A100 GPUs using the platform. We apply batch size of 48 per GPU and adopt gradients accumulation by 4 times. Thus the real batch size is 192 8 in total. We adopt an Adam optimizer with 81 = 0.9, 32 = 0.999, and 0.01 weight decay. We adopt a half-period cosine schedule of learning rate decaying with the base learning rate to be le - 4, the final learning to be le 5, and the maximum training iterations set be 30,000. A linear warm-up strategy applied to the first 200 steps. We also use exponential moving average weights for the UNet denoiser. We select "ViT-L/14" for both the CLIP encoder and the DINO-v2 encoder for fair comparison. The in-the-wild images tested on Section 4.2 are taken from (Liu et al. 2023a).
1.4.2
B PRINCIPAL COMPONENT ANALYSIS AND FEATURE-MATCHINE VISUALIZATION
For Principal Component Analysis (PCA) visualization, we first use the selected encoder to capture the spatial feature maps from two viewpoints of each object. Subsequently, we employ the alpha channel to eliminate the white background. This results in 16 16 patches with a channels. After that, we separately compute the Principal Component Analysis (PCA) of these two features and select the first three components. Each component is assigned a color channel, where we can further visualization, condensed feature maps from previously conducted PCA procedures. We compute the Euclidean distance between patches extracted from two viewpoints and then map each of them. More visual results of PCA can be found in Figure 9
visualize it by using RGB color space. For
directly adopt the
C COMPARISONS ON REAL-WORLDD BENCHMARKS
To further demonstrate the generalizability of the proposed approach, we set testbed on an in-the- wild real-world dataset OmniObject3D (Tong Wul 2023) dataset. The OmniObject3D dataset contains 6,000 scanned objects with 190 daily categories. Both 2D and 3D sensors are adopted to capture each 3D object. Compared to the Objaverse (Deitke al. 2023) dataset, the OmniObject31)
14

---

Published as a conference paper at ICLR 2024
Table 2: LPIPS the OmniOhject3D dataset (Tong Wu 2023
Subscts of OmmiOhject3D Datasel
Training Steps
Small
Medium
Large
Average
Zero 1-10-3
Ours
Zero I-to-3
Ours
Zero 1-to-3
Ours
Zero I-to-3
Ours
10k
0.326
0.231
0.326
0.251
0.330
0.266
0.327
0.253
20k
0.290
0.224
0.288
0.246
0.293
0.274
0.290
0.248
105k
0.247
0.274
0.306
0.276
300k
0.242
0.264
0.299
0.268
Image Pair
CLIP
DINO-v2
Figure 9: Visualization Principal Component Analysis (PCA).
Zero 1-to-3
Efficient-3Dia
Figure 10: Visual comparisons of multi-view consistency between Zero 1-to-3 and our proposed approach. We run the inference with different random seeds and different camera parameters. The highlighted region demonstrated that Efficient-3DiM shows better multi-view consistency than Zero 1-to-3 method for every random seed we adopted.
dataset contains richer texture and more complex geometry. but less amount of data. For each object, there more than 200 captured views. We first randomly pick view as the canonical view and
15

---

Published as a conference paper at ICLR 2024
then manually split the rest of the views into 3 categories by the rule of view distance, ranging from small (0 30 degrees), medium (30 - 60 degrees), and large ( >> 60 degrees). We only sample one view from each subset and construct 4 views for each object. After that, each method is required to adopt the canonical view as the reference condition to predict the target view from these three subsets. We report the LPIPS score for each subset and also the average score. As shown in Table the proposed method achieves faster training and better convergence, when compared to the baseline appraoch (Liu al. 2023b).
D DETAILS OF FEATURE AMALGAMATION
To obtain multi-scale representations, we capture both the output and intermediate feature maps from the DINO-v2 encoder. The official DINO-v2 encoder contains 24 Transformer encoder blocks and the base channel 1024. Following its original setting (Oquab al., 2023), we select the output of the 5th, 1 12th, 18th, and last layers, concatenating them to 16 x 16 x 4096 feature map. It is then converted to a 16 layer. After that, we add this multi-scale representation to the 4th, 6th, and 8th layers of the UNet denoiser Stable Diffusion. We only pick a subpart of this feature map depending on the number of channels in the target layer, and further conduct a bilinear interpolation on each feature map if the resolution does not match.
x 640 feature map through a linear layer, followed by layer normalization
16


Reviews:
[{"review_id": "NMkQzpgk0L", "reviewer": "Reviewer_mro2", "summary": "In this paper, the authors introduce Efficient-3DiM to accelerate diffusion models for single-image novel view synthesis, such as Zero123. Specifically, they employ a crafted timestep sampling strategy, a superior 3D feature extractor (DINO-v2), and an enhanced training scheme. Experimental results demonstrate that the proposed method could retain the performance metrics of the baseline but accomplish this with a remarkable 10x speed increase.", "questions": "1. It is suggested to include more related work for \"Novel View Synthesis from a Single Image\", such as [a-d]\n\n[a] Geometry-Free View Synthesis: Transformers and no 3D Priors. ICCV 2021. \n\n[b] Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image. CVPR 2022.\n\n[c] SynSin: End-to-end View Synthesis from a Single Image. CVPR 2020.\n\n[d] PixelSynth: Generating a 3D-Consistent Experience from a Single Image. ICCV 2021.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I really like the idea of integrating a self-supervised Vision Transformer for image conditions since the clip image feature only contains high-level semantic meaning. Figure 4 and Figure 5 also showcase the superiority of DINO-v2 over CLIP.\n2. An in-depth and smart analysis is provided for the denoising process, and then the author proposes to sample more for larger timesteps to learn geometry, which accelerates the training process.", "weaknesses": "1. I am not persuaded by the motivation from comparing training image classifiers and generative models, which are not comparable. Moreover, we usually treat Zero123 as a foundation model which does not need retraining. Under this circumstance, I think the value of the proposed Efficient-3DiM diminishes.\n2. The section \"ENHANCED TRAINING PARADIGM\" contains several well-known tricks, such as mix-precision training. I would like to suggest the authors should not emphasize this too much in their contribution.\n3. I expect the proposed method with a more advanced 3D feature extractor to achieve better performance than Zero123. It is suggested to apply 3D reconstruction (e.g., Neus) on the generated views to compare Efficient-3Dim and Zero123.\n4. Could you elaborate more on how to \"conduct several different spatial interpolation processing\"? And why do you only inject the features to the encoder of the UNet denoiser? I also suggest including more details on how to conduct feature amalgamation (e.g., feature shape and resolution).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dsPHMM7NZt", "reviewer": "Reviewer_fp2s", "summary": "Efficient-3DIM is an efficient framework for single-image novel view synthesis through diffusion models. The proposed method reduces the training time from 10 days to a single day while generating photorealistic and geometrically reasonable novel views.", "questions": "Please refer to the Weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n\nThe authors build their work on 3DIM and integrate three pivotal contributions: a modified sampling strategy departing from traditional uniform sampling, an integration of a self-supervised Vision Transformer replacing the conventional CLIP encoder, and an enhanced training paradigm. While all those steps taken are empirical, they each carry certain contextual novelty and together yield strong training performance:\nFor the modified time step sampling, although similar ideas were explored before, the authors take a new angle since the major phase of 3DIM’s training is essentially characterized as a finetuning paradigm as the adopted novel-view synthesizer is initiated from a pre-trained text-to-image diffusion model.\nIncorporating multi-scale representations produced by the DINO-v2 encoder, in place of the CLIP encoder, significantly improves the dense prediction and correspondence.\nWhile the authors transit from full-precision to 16-bit mixed-precision training, they add another layer normalization before sending the DINO-v2 feature to the diffusion model, to mitigate the numerical errors.", "weaknesses": "Weaknesses:\n \n(1)   I do not fully understand why LN can mitigate numerical errors, in the third part of Efficient-3DIM\n \n(2)   The major goal of this work is to trim down the training time without spending more costs on the total training resources (e.g., taking large-batch via a distributed system). Could the authors elaborate on how the proposed method could be integrated with distributed training, and whether its speedup benefits may diminish in the (more scalable) distributed training setup?\n \n(3)   Figure 8: I need help seeing how Zero 1-to-3 falls short of producing multi-view consistent visual outputs. All displayed results have valid visual quality to me.\n \n(4)   Minor: “neurallift-360” paper was incorrectly cited twice in reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BZkOdiGfOi", "reviewer": "Reviewer_FP6T", "summary": "This paper discusses the challenge of synthesizing novel views from a single image in computer vision. It introduces \"Efficient-3DiM,\" a framework designed to significantly reduce the training time and computational costs required to train such a model, achieving a training time reduction from 10 days to less than 1 day while maintaining efficiency and generalizability through innovative strategies like non-uniform timestep sampling and improved feature extraction.", "questions": "Please kindly refer to the weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper was developed to enhance the training efficiency of diffusion models for single-image novel view synthesis. Core Strategies include:\n- Revised Timestep Sampling: A novel strategy for selecting diffusion timesteps, and optimizing training.\n- Self-Supervised Vision Transformer: Integration of a self-supervised Vision Transformer to improve the incorporation of high-level 3D features better than CLIP.\n- Enhanced Training Paradigm: A refined training recipe that adopt low-precision training while addressing the numerical errors via extra layer normalization.\n\nAll strategies are grounded in motivating observations. When applied altogether, the speedup is quite significant: a 14x reduction in training time compared to the original zero 1-to-3 approach, enabling rapid iterations.", "weaknesses": "- Would this proposed approach be generalizable to accelerating training other image-to-3D models, such as Zero 1-to-3 and Syncdreamer? Why or why not?\n \n- The evaluation is solely conducted using the Objaverse dataset. Although this dataset is extensive and newly introduced, relying solely on a single dataset with potential biases and limited coverage could obscure any issues that the proposed method might have in the wild. It would be beneficial if the authors could also showcase results on additional datasets for a more comprehensive assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rPWtmAeooG", "reviewer": "Reviewer_TYx2", "summary": "This paper proposed Efficient-3DIM, a novel approach to single-image novel-view synthesis that aims to generate unseen perspectives of an object or scene from a limited set of input images. The proposed method reduces the training overhead to a manageable scale through several pragmatic strategies. Comprehensive experiments were conducted to demonstrate the efficiency and generalizability of the proposed method on common benchmarks.", "questions": "Please see the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Efficient-3DIM differs from previous approaches by proposing a simple yet effective efficient training framework. This is achieved through several pragmatic strategies, including a crafted timestep sampling strategy, a superior 3D feature extractor, and an enhanced training scheme. \n\nThe crafted timestep sampling strategy reduces the number of timesteps required for training, while the superior 3D feature extractor improves the quality of the learned features. The enhanced training scheme includes a self-supervised Vision Transformer and a modified sampling strategy, which are evaluated and shown to be effective in generating photorealistic novel views. \n\nThe proposed strategies significantly accelerate the training process, reducing the total training time from 10 days to less than 1 day, while shown to be effective in generating photorealistic novel views.", "weaknesses": "The study employs the Objaverse dataset as the sole testbed, which contains 800k three-dimensional objects. While this is a substantial dataset, its diversity and representativeness could be a limitation. I wonder if the authors would be able to demonstrate similar results on ScanNet or MVImagenet as well. Additional results on real scenes would be valuable and welcome too.\n\nThe authors mentioned that each object in the dataset underwent a procedure where 12 viewpoints are sampled. Are all comparison methods adopting the same fixed number of viewpoints?\n\nThe work itself follows the line of works by (Watson et al., 2022) and (Liu et al., 2023b), and mainly studied training efficiency improvement. The contribution is solid yet not fully substantial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)