Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering
Authors: Ido Sobol
Chenfeng Xu
Or Litany
Venue: NeurIPS 2024 poster (2024)
Keywords: Novel View Synthesis
Image Generative Models
Diffusion Models
Primary area: diffusion_based_models
TL;DR: Training-free boosting of diffusion based novel view generation through attention map manipulations.

Abstract: Generating realistic images from arbitrary views based on a single source image remains a significant challenge in computer vision, with broad applications ranging from e-commerce to immersive virtual experiences. Recent advancements in diffusion models, particularly the Zero-1-to-3 model, have been widely adopted for generating plausible views, videos, and 3D models. However, these models still struggle with inconsistencies and implausibility in new views generation, especially for challenging changes in viewpoint. In this work, we propose Zero-to-Hero, a novel test-time approach that enhances view synthesis by manipulating attention maps during the denoising process of Zero-1-to-3. By drawing an analogy between the denoising process and stochastic gradient descent (SGD), we implement a filtering mechanism that aggregates attention maps, enhancing generation reliability and authenticity. This process improves geometric consistency without requiring retraining or significant computational resources. Additionally, we modify the self-attention mechanism to integrate information from the source view, reducing shape distortions. These processes are further supported by a specialized sampling schedule. Experimental results demonstrate substantial improvements in fidelity and consistency, validated on a diverse set of out-of-distribution objects. Additionally, we demonstrate the general applicability and effectiveness of Zero-to-Hero in multi-view, and image generation conditioned on semantic maps and pose.

Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering
Ido Sobol!
Chenfeng Xu²
Or Litany
Technion UC Berkeley
NVIDIA
https://zeco2hhro--vvgghhhihi
Zero123-XL
Zero-to-Hero (Ours)
Source View
Sample
Sample
Sample 3
Sample
Sample
Sample
Target View
Figure I: Novel views generated from a single source image (far left column) at specific target view angle (with different seeds), compared between Zero123-XL. [27] and our Zero-to-Her method. Op- erating during inference, to the original image, all while ensuring realistic variation in the results (c.g. variations in chair backs in the top row). The ground-truth target view is displayed in the far right column.
method achieves significantly higher fidelity and maintains authenticity
Abstract
Generating realistic images from arbitrary views based on a single source image remains significant challenge in computer vision, with broad applications rang- ing from e-commerce to immersive virtual experiences. Recent advancements in diffusion models, particularly the 7ero-1-to-3 model, have been widely adopted for generating plausible views, videos, and 3D) models. However, these models still struggle with inconsistencies and implausibility in new views generation, especially for challenging changes viewpoint. In this work, we propose Zero-to-Herd a novel test-time approach that enhances view synthesis by manipulating attention maps during the denoising process of Zero-l-to-3, By drawing an analogy between the denoising process and stochastic gradient descent (SGD), we implement a fil- tering mechanism that aggregates attention maps, enhancing generation reliability
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and authenticity. This process improves geometric consisteney without requiring retraining or significant computational resources. Additionally, we modify the self-attention mechanism to integrate information from the source view, reducing shape distortions. These processes are further supported by a specialized sampling schedule. Experimental results demonstrate substantial improvements in fidelity and consistency, validated on a diverse set of out-of-distribution objects. Addition- ally, we demonstrate the general applicability and effectiveness of Zero-to-Hero in multi-view, and image generation conditioned on semantic maps and pose.
1 Introduction
The pursuit of realistic image synthesis at arbitrary views, given only a single source image, has long been a cornerstone challenge in computer vision and graphics. This technology can cater to countless applications, such as interactive product inspection, robot-scene interaction, and immersive virtual experiences. In this work, we aim advance this important line of research by improving the generation of novel views that are plausible and faithful to the input image. A recent, promising approach, Zero-1-to 1-to-3 (27) has developed a foundation model synthesize novel views based single source image and a target view angle. By leveraging pre-trained, image-conditioned stable diffusion model backbone [3] fine-tuned with target camera poses, and trained on paired source and target views from a vast collection of 3D models [0, 9). Zero-I-to-3 can generalize beyond its training set and generate plausible novel views. As result, this model has quickly gained popularity, inspiring subsequent work in 3D and 4D scene generation
24 39 28
25
57
33
45
35 20
Despite its remarkable ability. Zero- 1-to-1 has been observed to generate views that implausible, or inconsistent with the input object in terms of shape and appearance (24] g Previous works
have tried mitigate these issues by retraining diffusion models with more data 9 or to generate [56]. Despite substantial improvement, both approaches are resource-intensive due to the required re-training on large-scale 3D datasets. Another line of work attempts to consolidate inconsistencies across multiple generated views through a 3D representation like NeRF 14 30 However, direct aggregation often results in blurry outputs, as observed by [24]. Instead, ViVid-1-to-3 (24] employed multiview representation that naturally supports the use of a video foundation model. Nevertheless, this approach requires generating the entire trajectory from the source to the target view, adding significant complexity and computational overhead. Notably, the denoising process in Zero- 1-to-3 remains unchanged.
multiple views 339 25 26
28
In this work, we propose Zero-to-Hero, a novel test-time technique that addresses view synthesis artifacts through attention map manipulation. Recognizing attention maps crucial for latent predictions, we hypothesize that enhancing robustness in attention maps predictions can significantly reduce generation misalignment. To achieve this, we draw an analogy between the denoising process in diffusion models and stochastic gradient descent (SGD) optimization of neural networks. Specifically, we relate network weights that predict local gradients at each optimization step, based on sampled training examples and labels, to the denoising network's 's attention maps that predict latent representations from sampled noise each denoising step. In this work, we view the generation (denoising) process as an unrolled optimization, with attention maps as parameters of score prediction model. Inspired by gradient aggregation and weight-averaging techniques that improve prediction robustness (e.g., consistency training
(47). we propose filtering mechanism to enhance attention map reliability. This mechanism comprises iterative map aggregation within denoising steps and map averaging across denoising steps. The result is more reliable maps, particularly during the carly denoising stages when coarse output shapes are determined, leading to more plausible and realistic views.
To further promote consistency with the input, we modify the self-attention operation by running a parallel generation branch using the identity pose, incorporating its keys and values into the attention layer of the target view. Unlike previous applications of this technique (2 M) we find beneficial in view synthesis to limit its use to the early denoising stages, preventing shape distortions. Our unique denoising procedure is further complemented by a novel sampling schedule that emphasizes early and late denoising stages, maximizing performance. Our main contributions are as follows:
To address the main limitations of the Zero- Zero-1-to-3 model, we perform an in-depth analysis and identify self-attention maps as the main candidate for correcting generation artifacts.

---

We establish conceptual analogy between model weights in stochastic gradient descent-based network training and the role of attentions map updates during generation of denoising diffusion model. Based this, we propose simple yet powerful attention map filtering process resulting in enhanced target shape generation. We supplement our filtering technique with identity view information injection and specialized sampling schedule.
Our method requires no additional training, and it avoids the overhead of external models or generating multiple views.
Through comprehensive experiments on out-of-distribution objects. demonstrate that technique robustifies Zero-1-to-3 and its extended version, Z leading to views that are more faithful to both the input image and desired camera transformation. Our results show significant and consistent improvements across both appearance and shape evaluation metrics. Additionally, we find that Zero-to-Hor Hero naturally generalizes to additional tasks including multi-view, and image generation conditioned on semantic maps and pose. In all cases, we observed significant improvement in condition following and visual quality.
Related Work
2.1 Novel View Synthesis with Diffusion Models
Diffusion models have dominated various generative applications [17 38 36 41 Particularly, novel-view synthesis, as core of applications like augmented reality and simulations, naturally enjoys the benefits of high-fidelity zero-shot synthesis with diffusion models. One line of works generate novel-view image given a source image (i.e., image-to-image). These approaches typically involve training a diffusion model conditioned on both an arbitrary camera pose and the source view. For instance, the representative work, Zero-I-to-3, fine-tunes pre-trained Stable Diffusion model [36] by replacing the text prompt with camera pose and CLIP features 34 Moreover, another research trajectory [32] 21 14 7 46) proposes generating a 3D representation from single image (i.e., image-to-3D), which allows for sampling desired views from these 3D models. Our method, Zero-to-Hero, builds on the first approach (specifically Zero-I-to-3 and Zero 12o-xl2 and distinguishes itself by eliminating the need for extensive training, Instead, it offers a test-time, plug-and- and-play approach that significantly enhances visual quality and consistency.
[5
54 24
2.2 Test-Time Refinement in Diffusion-Based Generation
A common test-time refinement strategy in diffusion generation is leveraging guidance 22 18 to direct the sampling process with additional conditions. For example, Repaint (29 utilizes mask- then-renoise strategy to refine the generation results. Repaint also introduces per-step resampling technique, where given a horizon-size h. latent 2; re-noised to 2 and then denoised again to multiple times. They observe that resampling helps to generate more harmonized outputs, given an external guidance or condition. Restart (55 offers a sampling algorithm based on a variation of resampling within a chosen interval of steps. Our method is inspired by the strategy of per-step resampling. We show that it serves as a powerful correction mechanism throughout the generation process, even when no external guidance or condition is provided.
2.3
Attention Map Manipulation in Diffusion Models
Stable Diffusion 36 utilizes attention to enforce the condition information onto the generated results. Previous works demonstrate that manipulating the attention operation can achieve new capabilities
49 5 57). For example, MasaCtrl [5] uses Mutual Self-Attenion where source and target images are generated jointly while sharing information, by injecting source image keys and values to the target through self-attention. Here employ Mutual Self-Attention in the context of novel view synthesis. Differently to prior works we find it beneficial to limit its use to the early denoising stages.
Background
3.1 Zero-1-to-3: Challenges and Limitations
Zero-1-ter 1-to-3 is a pioneering method for novel view synthesis based on a diffusion model, which has gained significant popularity. This model is built upon the image-conditioned variant of Stable
3

---

Diffusion (SD) [3],, fine-tuned specifically for novel view synthesis. Kero-1-to-3 is conditioned on source image and relative transformation to the desired view angle [R T. Maintaining the SD architecture, these conditions are integrated in two ways. First, CLIP R4 embedding of the input image is concatenated with the relative transformation [R|7] and mapped to the original CLIP dimension to form a global pose-CIII embedding. interacting with the UNet layers through cross-attention, enriching the generation with high-level semantic information. In parallel, the input image is channel-concatenatee with the denoised image, helping the model preserve the identity and details of the synthesized object.
While Zero-1-to-3 (27) has achieved substantial progress in novel view synthesis. several common issues limit its practical upplication. Firstly, the generated images might not fit woal-world distributions, resulting in implausible and unrealistic outputs (e.g., first row in Fig D. Secondly, the target image may appear plausible but be inconsistent with the input image in terms of shape or appearance (e.g..
fifth row in Fig D.
In this work, we identify the critical role of self-attention maps in high-quality generation and propose a novel filtering process that enhances robustness without requiring further training. This process addresses the aforementioned issues, ensuring reliable and consistent results.
3.2 Leveraging Gradient and Weight Aggregation for Improved Prediction Consistency In this work, we draw a conceptual analogy between gradients and weights in stochastic gradient descent (SGD), and latents and attention maps in denoising diffusion models. Leveraging this analogy, we adapt techniques from SGD to enhance prediction consistency in diffusion models. Below, we summarize general techniques SGD that improve the training process.
SGD is fundamental tool in network training parameter) space towards local minima. For a neural network F(z; 0) with parameters 0, SGD samples training data points 2; and their corresponding labels ys. and computes the gradient of the loss function (z, 0), y:)) with respect to 0 to update the parameters. In practice, aggregation of gradients and network weights during training is often performed to reduce variance and improve convergence. Gradient aggregation typically involves averaging gradient values over a batch, while weight aggregation accounts for the history of the weights in each update.
4 designed to navigate the weight (network
Notable examples include temporal averaging in Adam optimizer (23), Stochastic Weight Averaging (SWA) [19] and teacher networks [47] used in consistency training by employing an exponential moving average (EMA) of a student network to maintain high-quality predictions. This technique is For detailed study of EMA in
prevalent in semi-supervised and representation learning [13 15
network training, we refer readers to [31].
4 Method
In this work are concerned with the task of single image novel view synthesis. Formally, given an input image of an object and relative camera transformation towards a desired target view, our goal to generate the image at that target view. Specifically, we build upon the seminal work of Zero- 1 1-to-3 (27) which tackled this task via diffusion model. As detailed in Sec. Zer 1-to-3 often struggles to generate plausible and input-consistent images. this work we propose Zero-to-Hero novel training-free technique that significantly improves its generation quality through attention map manipulation. This section is structured as follows. (4.1) through network architecture analysis we identify self-attention maps as key for robust view generation; (42 drawing inspiration from techniques, we outline our novel attention map filtering pipeline; (4.3) details each step of the map filtering: (4.4) introduces the mutual self-attention which we use for shape guidance at early generation stages; (4.5) Finally, our proposed hourglass scheduler is
SGD
introduced for more efficient utilization of generation steps. Fig. 2 depicts Cero-to-Here 's main
modules. An ablation of these modules is provided in Tab. 2 and in Sec. of the appendix.
4.1 Analyzing the Role of Cross- and Self-Attention Layers in Novel View Generation
1-to-1-to-3 builds upon the image-to-image variant of Stable Diffusion (36], which utilizes a UNet 37 architecture as its backbone and incorporates both self- and cross-attention layers. In this subsection, we analyze the roles of these components and their contributions to the generated view.
This analysis aims to identify effective intervention points for enhancing generation quality.

---

Artention Fitthing
Ho N
In-step Map Update
2--3
Perotions
Crooeeoliip Hap Aver egiin
Ma+
Stoge Munual Salt Artarritiiinn (Section 4.4)
4
Bedlack
Run
Kas
My.
Actention
Inpur Farttrtty Scores
Matualen GeT
Attection
Altention
Fittring
Target Sim larity Seores
Attemtion
Figure Zero-to-Hero main modules. (Left) Two denoising steps of the generation process of both the source (top) and target views (bottom). Each denoising step is iterated R times ("resampling" ".. (Right-top) Attention map filtering: Robustifying attention maps via an aggregation of same step and previous steps attention maps. (Right-bottom) Mutual self-attention: Guiding target shape through the keys and values the source generation branch.
Global pose conditioning through cross-attention. Prior work [16] has demonstrated that the cross-attention layers in text-to-image diffusion models, which link text tokens to the latent image, spatially aware and can be used for spatial manipulation. We first investigate these cross-attention layers, as they are the only components in the model through which the target pose is injected. the text-to-image variant of Stable Diffusion, the generation is conditioned on prompt P containing C' tokens, each encoded with CLIP into an embedding, resulting a condition er21 E IR CxdLLII However, in Zero-I-tor the condition is pose- CLIP embedding E R XdeLIP project keys K, E 1xd Formally, given a sample ,N E R x xd, and its corresponding queries Qr E # Xd the pre-softmax attention map A between Q: and K, has dimensions H W Given that the summation of the softmax is always 1, the post-softmax attention map so so ('mmar((A)) in Zero-1-to-3 a constant all all-ones matrix. visual demonstration presented Fig. in the appendix.
The post-softmax attention map is used to compute a weighted sum over the values matrix V E Rxxd obtained by transformation of the condition Since the attention matrix is all-ones matrix, conclude that the pross-attentiog operation degenerates into global bias term, lacking spatially operations. Computing similarity scores in the cross-attention layers is redundant these scores are never used. While in principle is possible to improve the global bias term by additional optimization objectives and extra training overhead, focus the self-attention layers to enhance the results and mitigate the consistency issues while avoiding retraining the model.
Spatial information flow through self-attention. By monitoring the self-attention layers during the generation process, we observe that random noise introduced to the latent representation also introduces randomness the attention maps. This randomness, while promoting generation diversity, can often lead to undesired strong correlations, that are misaligned with the true target. These strong correlations may persist through the denoising process, resulting in accumulated errors and visual artifacts.
Given the above observation and the insight about the spatial-degeneracy in the cross- aots--attnttion layers, we hypothesize that the self-attention layers preserve the information about the structure and geometry of the generated image, through the similarity scores between different elements in the latent vector. To validate our hypothesis, we conduct a straightforward experiment assess Zero- 1tooo--tt-3's performance using the 'ground truth' self-attention maps, which reflect the most accurate connections considering the true target image. Specifically, we selected two images, l and f with known relative camera parameters [R| We first encode I obtain the clean latent 2 and then add subtle noise to obtain the corresponding noisy latent for timestep Tinis Ammil A single denoising step is performed on the noisy latent, and we save the self-attention maps from each layer in the UNet.
5

---

Source
Zero123
Target view
Source View
Zero123 GT
Target View
Figure 3: Through the injection of ground truth attention maps extracted from the target view, we demonstrate that Self-attention maps are key to robust view synthesis.
considering these maps as the ground truth (GT) maps. In experiments, Tinit 5. Next, sample random Gaussian noise and denoise it to regenerate the target image. During each denoising step, replace the computed self-attention maps with the GT maps, without altering any other components (e.g., cross-attentic layers, residual blocks) or the latents, queries, keys, and values. We report the results in the experiment section in Tab. showing a significant improvement in all metrics. Note that the results obtained with the GT attention maps are a strict upper bound, as the GT maps contain information about areas that are invisible in the source view. Our experiment validates that through improved self-attention maps, the generated image becomes more plausible. Visual examples are shown Fig. 3 and in Fig. 8ji the appendix, refer Fig. for the results of Zerol23-XI. and Zero-to-Hert for the same views.
4.2 From SGD to Diffusion Models: Attention Map Filtering as Weight-Space Manipulation
We draw a conceptual analogy between a denoising process of a diffusion model, and SGD based network optimization both progress through gradient prediction of loss function, and log probability (the (c3r))[43), respectively. Building on the discussion from the previous section, we treat the self-attention maps as parameters M in the denoising process - 5u(2;; M,, v)) and define their update process as a mapping"] M, -> 2t-1 + Mi-1. Here the map M,-1 results from passing the latent 2,-1 through the attention layers. This process is analogous to gradient descent optimization in neural networks, where each step adjusts the weights in the direction of the gradient of loss function, such as the log probability in classification tasks. We denote this weight update as a mapping 0g + 1/+ + 02+1, where Dt (xx;; 0), and the updated parameters 0e+11 result from a gradient step.
As detailed in Sec. 3.2, gradient and weight aggregation are essential for robust convergence. We outline this process in three steps illustrating the analogy between network parameter updates and attention map filtering. Fig. further illustrates the analogy.
Step-by-Step Analogy From network parameters attention maps
Network Training
Denoising Process
1: Sampling: Generate A pairs of samples and
1: (Re-)Sampling: Repeatedly re-noise 2t-1 to
their corresponding labels ((xi, )
2, and denoise, R times.
2: In-tepp weight update: Average the gradi-
2: In-step map update: Resmapling results in sequence of maps (Mrr)) aggregated
ents to adjust the network parameters: 0, =
8, Σ ((F(F Oc). yi).
into the final updated prediction Mr
3: Cross-step weight averaging: Update net-
3: Cross-step map averaging: Update atten-
work parameters using EMA: 4.41 afl, +
tion maps via EMA: Mc-1 = aM, + (1
(1 00, for a [0, 1].
a)M, for [0, 1].
4.3 Robust View Generation via Attention Map Filtering
We now discuss in detail each of the map filtering steps. A scheme of the different filtering modules is provided in Fig. 2 (Left, and Top-right).
Latent refinement via per-step resampling. Inspired by previous studies (29 2
implement per-step resampling throughout the image generation process. We select a range of timesteps
Although the latent prediction 21-1 depends on the parameters & of denoising UNet they remain unchanged. Here emphasize that the attention maps M the parameters being updated.

---

81- 9t 1
Me
Gr
M
A1+1
M:-1
(ilee.))
g*
SGD Step
Denoising Step (DS)
M
Figure 4: From SGD to Diffusion Models: An illustration of our conceptual analogy.
Source
Zero123-34
Different seeds
Zero-to-Hero (Ours)
Target
500
800
400
Generated Image
Figure 5: Attention map filtering in action. We compare the attention scores zero (top) and Zero-to- Hero (bottom) wrt the region marked with a purple circle at different denoising steps. Both methods are initialized with the same seed. We observe that the strong correlation values in the upper right corner lead to exaggerated content creation (note the unrealistically elongated neck). Conversely, through filtering, Zero-to- Hero mitigates these artifacts, leading to robust view synthesis.
[mmin bmaz) where each denoised sample 2, is re-noised with the proper noise ratio the previous sampled timestep E++1 and denoised back to for R iterations" 1 We concur with previous studies that resampling functions as an effective corrective mechanism to the generated image, as can be seen in Tab. 2b. Experimentally, denoising process. Through resampling we progressively generate R attention maps with different noise patterns. We propose to leverage these intermediate maps to further boost performance through in- and cross-step attention map manipulations.
find that it is particularly useful during the initial stages of the
In-step map update. We propose novel attention pooling function f. to update the attention maps within the denoising step. Specifically, a self-attention map is refined based on previous maps [M, created the same timestep. Since resampling is a sequential process, we perform progressive aggregation scheme: Mer f(Merr (Mr+) )==)) We found the element demennt-wise wise min- pooling operator (a, b) min(a, b) to perform best our experiments. We discuss other options for f in the appendix.
Cross-step map averaging. Resampling tends to favour "conservative" generations, often gradually neglecting fine image details like buttons and eyes as denoising progresses. This phenomenon not resolved by the in-step update. To mitigate this issue, we propose to pass the self-attention map time f, Mr to the next step in the denoising process. We implement this cross-step aggregation via EMA: M, = a,, + (1 a M, for a E [o, 1]. This method effectively balances past priors with current data to enhance the denoising results. In practice we apply both methods in tandem, as detailed in Sec. 8.5) An example of our attention filtering is presented in Fig. 5 Note that our filtering mechanism is applied the pre-softmax attention maps.
Depending the sampling algorithm, h > steps may be taken.
7

---

4.4 Early-Stage Shape Guidance via Mutual Self-Attention
Of the two challenges describe in the introduction, attention map filtering handles well generating realistic outputs. Yet, we observe it is sometimes insufficient to enforce consistency with the input appearance and structure. We propose to utilize mutual self- self-attention as complementary technique, to propagate information from the input to the target. Similar to prior works [4 5 I 57 [22 we generate the input and target views in parallel. At each timestep 6, we obtain queries, keys and values of the self-attention layers for the input and the target branches. Similar to the mutual self-attention introduced in SS) we modify the self-attention operation of the target by replacing the target keys and values with those of the input branch: Attn(Q K"" Vin) (Fig. 2 (bottom-right)..
Previous studies initiate mutual self-attention (MSA) at a later denoising step (small t). We find that at that stage, the structure has already been determined. Instead, to guide the structure of the target to be more consistent with the input, we find more effective when applied from the beginning of the denoising process. While MSA is effective at transferring the appearance and structure of the input to the target, may overfit. We find that terminating the MSA process once the structure has been stabilized becomes crucial in mitigating overfitting. In practice we find that applying MSA during the first third of the denoising process is a good rule of thumb for optimal results. We refer the reader to Sec. 8.5 in the appendix for further details.
We view early-stage MSA as complementary filtering scheme. Building on the property that generating the input view (which the model is conditioned on) is much easier task for the model compared to generating novel views, keys and values predicted in the input branch are "cleaner". Mapping them to the target view thus refines the predicted scores, facilitating more stable and reliable output.
4.5 Hourglass Sampling Scheduler
As resampling is time-consuming operation, significantly increasing the number of function evaluation (NFE), we seek an efficient scheduling scheme that will enable robust and high quality generation, while preserving the fast generation times of Zero-I-to-3, We use the common DDIM sampling 44), and propose an efficient scheduling scheme to select the sampled timesteps. In the experiments section, we demonstrate that increasing the number of denoising steps does not necessarily improve the performance, and therefore we aim to use an overall small number of sampled timesteps. Specifically, we find that denser sampling during the beginning of the denoising process crucial to promote realism. We also find that denser sampling at the final steps enhances fine details. Therefore, we suggest a double heavy-tailed scheduling scheme we call Hourglass, according to which we divide the generation process into 3 stages, detailed in 8.3 Within each stage, we sample steps uniformly via DDIM. However, in the first and last stages we sample steps more densely (at a higher rate) than in the middle stage, by factor of Aden.
5 Experiments
Datasets. We evaluate our method on two datasets, following previous works. Firstly, Google Scanned Objects (GSO) Dataset [12). which includes 1030 scanned household objects. However, the dataset's imbalance (c.g., 254 objects are categorized as "shoe") and the high proportion of symmetrical shapes limit its reliability for evaluation. To address this, inspired by [24], we select a challenging subset of 50 objects from GSO, avoiding symmetrical and repetitive shapes. For each object, we render 8 random views (details in Sec. 8.9 the appendix) and synthesize the remaining views from each source view, generating each target view with 3 different seeds to calculate the average score per measure. Secondly, RTMV Dataset 488, which consists of 3D scenes, Each scene contains 20 random objects. For evaluation, we randomly select 50 scenes, and 8 random views per scene. The evaluation process is the same as described for GSO.
Metrics. We report the image quality metrics PSNR, SSIM (50] and LPIPS [59] As these metrics are sensitive to slight color variations, we segment the generated targets and their corresponding real images, via thresholding, and report the Intersection Over Union (IoU) score.
5.1 Evaluations
Quantitative evaluation. We evaluate Zero-to-Hed Hero against zero-1-to-3 and on zero 1rro123, 123-XI. We report all metrics for the original models using 25, 50 and 100 DDIM steps, and for our method applied to both models. In Tab. and in Tab. 3 we provide the results for GSO and RTMV datasets,
8

---

respectively. We include the number of sampled timesteps T and the total number of network evaluation NFE (accounting for resampling). Zero-to-Heru consistently improves performance across all metrics, taking a significant step towards bridging the performance gap to GT attention maps. All implementation details, including analysis of the inference cost of our modules, are provided in Sec. 8.5 of the appendix.
Table 1:
Quantitative evaluation on challenging GSO subset. Zero-to-Hak Hero consistently improves performance upon baselines, taking a significant step towards oracle map performance (bottom rows).
Name
NHE
PSNR
SSIM f
LPIPS 4
IOU 1
Zero-1-to-2
25
25
17.27
0.851
0.173
73.5%
Zero-1-to-3
50
17.24
0.850
0.173
73.5%
Zero-1-to-3
17.21
0.850
0.173
73.4%
Ours (Zero-I-to-3)
26
66
17.67
0.859
0.163
75.3%
Zero Zero123-XL
17.72
0.854
0.163
76.4%
Zcro123-XL.
50
17.71
0.854
0.163
76.4%
Zero123-XL
100
100
17.68
0.854
0.163
76.4%
Ours (7eml123-XL)
26
66
18.35
0.864
0.153
78.3%
Zero 1-003 Zero-1-to-3+ GT maps
50
50
21.52
0.888
0.122
88.8%
Zero123-XL + GT maps
50
50
21.79
0.890
0.117
88.6%
visually demonstrate how Zero-to-Hero is able to mitigate various artifacts generated by Zero-I-to-3, from implausible results to incorrect poses. In Fig. and in Fig- [ to emphasize the consistency and robustness our method offers.
Qualitative evaluation. In Fig. [ Fig. 9 and Fig. T0
present 3 examples per target view, generated with 3 random seeds,
5.2 Ablation Study
To assess the contribution of Zero-to-Hero different components to the final performance, evaluate our pipeline on our challenging GSO subset, starting from the baseline Zero123-X0 and gradually adding each module. The results summarized in Tab. 2 When reporting the results of the vanilla Zero123-XL, we use its best score achieved by running the model with 25, 50, and 100 steps. A similar comparison against zero-1-to-3 is included in Sec. 8.6 6.6.8.6 of the appendix, demonstrating consistent behavior. We also analyze the affect of resampling and our attention filtering on the generation diversity in Sec. 8.7 of the appendix. Additionally, we provide qualitative results, demonstrating the common contributions of AMF and MSA in Fig. 10 in the appendix.
Table 2: Ablation Study. We demonstrate the importance each of Zero-to-Herr modules, applied to the base method Zerol23, 10-11L Sample scheduling (Hourglass), Resampling (Resample), Attention map filtering (AMF), and Early-Stage Mutual Self-Attention (MSA). Consistent conclusions are reached with the base model Zero-1-to-3 and are shown in Sec. of the appendix.
Hourglass
Resample
AMF
MSA
PSNR f
SSIM f
LPIPS
IOU 1
17.72
0.854
0.163
76.4%
17.74
0.855
0.162
76.6%
X 17.85
0.857
0.160
77.1%
V
X 17.92
0.859
0.157
77.8%
/
X
V
18.25
0.862
0.155
77.6%
V
V
18.35
0.864
0.153
78.3%
5.3 Attention Map Filtering Beyond Novel View Synthesis
Although our work addresses the core limitations of single view synthesis models, the condition enforcing effect of our Attention Map Filtering (AMF) is more general. We have conducted several preliminary experiments which demonstrate promising results. Further details are provided in Sec. 8.8
Conditional image generation. A brief study of ControlNet models (58] demonstrated that they suf- fer from similar limitations as Zero-1-to-3 and follow ups. Namely, lack of condition enforcement and frequent appearance of visual artifacts. We implemented our proposed AMF module for two pre-trained ControlNet models. We provide qualitative results for pose- and segnenaatioonnconniiinss ControlNet models in Fig. and Fig. [3 respectively.

---

Multi-view synthesis. We integrate AMF into MVDream 40 next-to-multive model, and find that it helps mitigate the same issues as in the single view case. In Fig. we provide qualitative results.
Prompt
Pase
Pre-trained ContraiNet
ControlNett AMF
Superman Rlying above the ocean
witch riding broomstick
swimmer dives around sharks, wearing swimsuit
Figure 6: Qualitative results for pose-conditioned ControlNet. Qualitative results for pre-trained pose-conde conditionn ControlNet, without and with AMF. Both methods are initialized with the same seed. AMF leads to results that more plausible and better align with the conditions.
Conclusions and Future Work
In this paper, introduced Zero-to- Heroo, training-free freee method to boost the robustness of novel view synthesis. We enhanced the performance of pre-trained Zero-1 1-o-33 diffusion model using two key innovations: test-time attention map filtering mechanism that enhances output realism, and an effective use of source view information to improve input consistency. Our approach also features novel timestep scheduler for maintaining computational efficiency. In future work, we aim to refine our method by developing trainable filtering mechanisms, enhancing pose authenticity via cross-attention manipulation, and extending our approach other diffusion-basse generative tasks.
Limitations. Our method, operating at test-time, is limited by the generative capabilities of the pre-trained model. If Zero-1-to-3 cannot correctly generate the target pose, our method may not enhance the output, demonstrated in the inset figure.
Source View
Zero123-X
Zero to Boreerrre (Dun) Tirger View
Broader impact. Through enhancing view synthesis, our method offers significant benefits to applications in virtual reality, augmented reality, and roboties. it requires no further training, readily accessible. However, this accessibility also simplifies the creation of realistic novel views, which could exploited for malicious purposes such as deepfakes.
Acknowledgments
We sincerely thank Matan Atzmon for impactful discussions and James Lucas for his invaluable feedback. Or Litany Taub fellow and is supported by the Azrieli Foundation Early Career Faculty Fellowship. This research was supported part by an academic gift from Meta. The authors gratefully acknowledge this support.
10

---

References
[1] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Flor, and Daniel Cohen-Or. Cross- image attention for zero-shot appearance transfer. arXiv preprint 1Xiv:2311.03333 2023.
[2] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852,
2023.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231 1..:2311..150 111311111122 5127,, 2023,
[4] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 187-186. Springer, 2010.
[5] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560-
22570. 2023.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[7] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4217-4229, 2023.
[8] Yabo Chen, Jiemin Fang, Yuyang Huang, Taoran Yi, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Cascade-zero123: One image to highly consistent 3d with self- prompted nearby views. arXiv preprint 21..:iv22312.04444. 2023.
[9] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xll A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
[10] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142 13153, 2023.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. 2021.
Advances in neural information processing systems,
[12] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high- high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022.
[13] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances neural information processing systems, 13:21271-21222 2020.
[14] Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In International Conference on Machine Learning, pages 1 11808-11888 11866 PMLR, 2023.
11

---

[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompl image editing with cross attention control. arXiv preprint arXiv:2208.01 01666, 2022.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 125998, 2022.
[19] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
[20] Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 d'ee dynamic object generation from monocular video. arXiv preprint 1.1iv:2311.02244 2023.
[21] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.0244 022663,,......... 2023.
[22] Yash Kant, Aliaksandr Siarohin, Michael Vasilkovsky, Riza Alp Guler, Jian Ren, Sergey Tulyakov, and Igor Gilitschenski. invs: Repurposing diffusion inpainters for novel view synthesis. In SIGGRAPH Asia 2023 Conference Papers, pages 1-12, 2023.
[23] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[24] Jeong-gi Kwak, Erqun Dong, Yuhe Jin, Hanseok Ko, Shweta Mahajan, and Kwang Moo Yi. Vivid-1-to-- Novel view synthesis with video diffusion models. arXiv preprint arXiv:2312.01305. 2023.
[25] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng. Jiayuan Gu, and Hao Su. 2-e-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:231 1.07885, 2023.
[26] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[27] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-t-33 Zero-shot one image 3d object. In ICCV, 2023.
[28] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Konura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from single-view image. arXiv preprint arXiv:2309.03455 2023.
[29] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-11471 2022.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[31] Daniel Morales-Brotons, Thijs Vogels, and Hadrien Hendrikx. Exponential moving average of weights in deep learning: Dynamies and benefits. Transactions on Machine Learning Research, 2024.
12

---

[32] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e A system for generating 3d point clouds from complex prompts, arXiv preprint arXiv:2212.08751. 2022.
[33] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306.1 17843, 2023.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[35] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142 2023.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAL 2015: 18th international conference. Munich, Germany, October 5-9. 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[38] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[39] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero 23++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv: 11.1:2300115110,, 2023.
[40] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi- view diffusion for 3d generation. arXiv preprint arXiv:2308. 200Xi::208.1611,, 2023.
[41] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations, 2023.
[42] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[43] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[44] Bernard Spiegl, Andrea Perin, Stéphane Deny, and Alexander Ilin. Viewfusion: Learning composable diffusion models for novel view synthesis. arXiv preprint arXiv:2402.02906, 2024.
[45] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023.
[46] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In Proceedings of the IEEE/VF International Conference on Computer Vision, pages 22819-22829, 2023.
[47] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve supervsuee deep learning results. Advances in neural information processing systems, 30, 2017.
13

---

[48] Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Charles Loop, Nathan Morrical, Koki Nagano, Towaki Takikawa, and Stan Birchfield. Runv: A ray-traced multi-view synthetic dataset for novel view synthesis. IEEE/CVF European Conference on Computer Vision Workshop (Learn3IG ECCVW), 2022, 2022.
[49] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Blug-and-pe diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[50] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility structural similarity, IEEE Transactions Image Processing, 13(4):600-612, 2004.
[51] Daniel Watson, William Chan, Ricardo Martin-Brually Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628, 2022.
[52] Haohan Weng, Tianyu Yang, Jianan Wang, Yu Li, Tong Zhang, CL Chen, and Lei Zhang. Consistent123: Improve consistency for one image to 3d object synthesis. arXiv preprint arXiv:2310.08092 2023.
[53] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. a-vieo- One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023.
[54] Chenfeng Xu, Huan Ling, Sanja Fidler, and Or Litany. 3difftection: 3d object detection with geometry-aware diffusion features. CVPR, 2023.
[55] Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems, 36.76806-76838, 2023.
[56] Xianghui Yang, Yan Zuo, Sameera Ramasinghe, Loris Bazzani, Gil Avraham, and Anton den Hengel. Viewfusion: Towards multi-view consistency via interpolated denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9870 9880, 2024.
[57] Junwu Zhang. Zhenyu Tang, Yatian Pang. Xinhua Cheng, Peng Jin, Yida Wei, Wangbo Yu, Munan Ning, and Li Yuan. Repaint 123: Fast and high-quality one image to 3d generation with progressive controllable 2d repainting. arXiv preprint arXiv:2312.13271 2023.
[58] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[59] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
14

---

8 Appendix
8.1 Cross-Attention in Zero-1-to-3
We demonstrate the degenerated cross-attention layers Zero-1-to-3 in Fig. generate random target view using Zero 123-XL and extract the cross-attention map generated timestep 900 in the last layer of the UNet. The same behaviour holds across all timesteps and layers. The displayed map created by averaging over all the attention heads.
Pre-Softmax Cross-Attention Map
Post-Softmax Cross-Attention Map Softmax
2.00
3.075
0.090
0.055
1425
0.000
0.070
0.000
Figure 7: Cross-Attention in Zero-1-to-3. (Left) The cross-attention map before applying softmax.
(Right) The degenerated all-ones attention map, produced by applying softmax the left map.
8.2 Self-Attention in Zero-1-to-3
In Fig. g we present additional examples showing the performance of 1e3o123-XLL when the self- attention maps replaced with the 'ground truth' maps, extracted from the real target described
in Sec.
8.3
Qualitative Results
In Fig. we provide more qualitative results that reinforce the effectiveness of Zero-to-Hero.
8.4 Quantitative Results
On the GSO evaluation, shown in Tab. in the main paper, Zero123-Xx improved performance over Zero-Iu 1-to-3 by utilizing 12x more data, yielding gains of [0.45, 0.003, -0.01, 2.9%] in PSNR. SSIM, LPIPS, and loU, respectively. When applied to Zero---to-3, Our method achieved comparable gains, [0.40. 0.008, -0.01, 1.8%), without using any additional data or further training. Notably, when applied to Zero X2- our method resulted in even larger improvements [0.63, 0.1, -0.01, 1.9%] (over Zero X2)- demonstrating that these performance boosts cannot be solely achieved with merely more data.
Additionally, in Tab. 3 over baselines.
report all metrics RTMV dataset, showing substantial improvement
8.5 Implementation Details
We evaluated our method on the default checkpoint of pero---too33 and on Zero Caro123. 123 We used the same RTX 4090.
for both models, as follows. All experiments were run on a single NVIDIA
Note that the length of the forward process of Stable Diffusion is T 1000.
Attention map filtering pipeline. mentioned the method section the main paper, we apply both in-step and cross-step aggregation in tandem. In detail, preserve attention map history
15

---

Source View
Zero123-XL GT Maps
Target View
Source View
Zero123-XL GT Maps
Target View
Figure 8: Through the injection of ground-truth attention maps extracted from the target view, we demonstrate that self-attention maps are key to robust view synthesis.
Table 3:
Quantitative evaluation on RTMV dataset. Zero-to-Her consistently improves perfor- mance upon baselines.
Name
T
NFE
PSNR
SSIM t
LPIPS
IOU
Zero-1-to-3
25
25
10.27
0.587
0.396
70.7%
Zero-1-to-3
50
50
10.26
0.585
0.395
70.7%
Zero-1-to-3
100
100
10.25
0.584
0.394
70.7%
Ours (Zero-1-to-3)
26
66
10.86
0.617
0.373
71.2%
Zero123-XX
25
10.64
0.595
0.391
71.9%
Zerol a 123-XL
50
50
10.61
0.593
0.391
72.0%
Zero123, 12-
100
100
10.64
0.593
0.390
72.1%
Ours (Zero123-XL)
26
66
11.15
0.618
0.372
72.8%
information denoted as Hę. Subsequently, integrate this historical information through simple blending technique at each denoising step, along with our in-step map update:
Mer = a. f(Mrr (M + (1 - a.) Hi+1,
where 1), f is attention pooling function and f. represents the previous sampled step.
The historical information is updated at the last resampling iteration (the R-th iteration) of each timestep as follows: H, = az .M.R + (1 - on)) . H4H,, where or a decay factor within the range [0, 1]. This historical data is initialized from the first refined self-attention map at the final resampling step, expressed as H, Mr
Resampling performed during timesteps E [800, 1000), with R=5.
In-step map update. In-step map update is applied at timesteps [800, 1000]. with element- min-pooling as a denoising function in all our experiments. We find that min-pooling useful during the carlier steps of the denoising, where the noise is substantial.
16

---

Zero123-XL
Zero to Hero (Ours)
Source View
Sample 1
Sample 2
Sample 3
Sample
Sample 2
Sample
Target View
Figure 9: Novel views generated from a single source image (far left column) at specific target view angle (with different seeds), compared between fero123-XL, [27] and our Zero-to- Sero-to-Hert toro method. The ground-truth target view is displayed in the far right column.
We explored several simple denoising functions:
Averaging: Mer = 3 (Me, - Ek=l Me,), which can be extended to weighted averaging.
2. Min pooling: Mer min(Me, (M
3. Exponential Moving Average: Mir = a; Mer + (1 (1-
ac) (1)-a)) M,, for as [0, 1].
While all three improved performance over vanilla resampling, we find min-pooling to work best during the early stages of the denoising process.
Cross-step map averaging. We perform cross-step information passing timesteps / [600, 1000), with ay = 0.5 and a, = 0.2.
Early-stage mutual self-attention. We apply Etrly-Stage mutual self-attention timesteps E [600, 1000] for Zero123-X0 22- and on timesteps E[700, 1000) for Zero-1-to-3. As mentioned in Sec. S] the input latent vector is concatenated to the generated latent vector in --to-- Therefore, the target queries contain information about the imput view, introducing a bias towards the input view during the generation process. Although mutual self-attention manages enforce consistency with the input, it sometimes causes pose shifts and visual artífacts. Therefore, We only apply mutual self-attention in the decoder layers of the UNet with spatial resolution of 16 16 and 32 32 (the two largest resolutions, since the latent dimension of Zero-1tto-3 is 32 x 32).
Hourglass scheduling. We divide the denoising process into 3 generation stages. We define the early stage during timesteps t [Te, TTl where the general shape and geometry are determined. Tr the length of the forward process 17 (2) We define the middle stage for t [Tmy Te]. where both the shape and appearance are refined gradually. We define the last stage for E [0, Tin] where mostly fine-details refined and added. As describes in the main paper, within each stage sample steps uniformlly via DDIM sampling. However, the sampling rate is higher for the early and late stages.
In our experiments, we determine the different stages of the denoising process as follows, setting A = 5. The early stage is defined for E [800, 1000) and we sample 10 timesteps uniformly within this interval. The middle stage is defined for [200, 800) and sample timesteps uniformly
17

---

within this interval. The last stage is defined for t E [0, 200] and we sample 10 timesteps uniformly within this interval. In total, we sample 26 timesteps throughout the denoising process.
Setting the Prpe-parameters. All hyperparameters were tuned based on a small validation set of objects. Due to limited computational resources, the tuning process was not exhaustive. We found that the method's performance is not highly sensitive to most parameters.
1. Cross-stee Pross-step weight (alpha): We experimented with values ranging from 0.1 to 0.9 in incre- ments of 0.1. Alpha determines the weight assigned to previous predictions in the cross-step aggregation. Zero123-XI early predictions were generally reliable, so larger weight yielded better results. We maintained the same parameters for Zero- 1-to-11t--.. In the exper- iments conducted with additional models such as ControlNet and MVDream, a smaller weight generally produced better outcomes.
2. Resampling iterations (R):
We observed that the model's performance is relatively insensitive to the choice of R, with values between and 8 yielding similar results. While some objects benefited from larger values (~10),, the overall improvement was minimal. Additionally, as shown in Fig. 11 large values of R (~~ 15-20) can reduce diversity. For additional experiments with models like ControlNet and MVDream, we fixed R at 5 and did not test other values.
3. Filtering schedule: We apply resampling and our filtering mechanism during timesteps [mmin: T.. To set min:: we consider values from 500 to 900, in increments of 100.
4. MSA schedule: We apply Early-stage MSA in timesteps t [mmin Tj. To set fin consider values from 400 to 1000, in increments of 100.
Additional Inference Cost Analysis. Our proposed modules add computational overhead the base model. In the paper, we addressed this by counting the overall number of function evaluations (NFE) and keeping it on par with the base model to ensure fair comparison. We discuss the individual computational overhead of each module.
Resampling: Similar to the total number of denoising steps, T, the number of resampling iterations, R, linearly increases the NFF. Our chosen value for R and the timesteps at which we apply it resulted in mapping 26 denoising steps to a total of 66 NFE. Resampling does not introduce additional computational cost over adding standard denoising steps.
2. Mutual self- attention: The main additional cost of MSA is the necessity to generate the input view in addition to the target views, meaning the effective number of generated samples is increased by one.
3. Attention map filtering: We apply AMF during the early steps of the denoising process. We find that the and cross-step operation adds a small computational overhead. We maintain two additional instances of each attention map: the attention map from the previous timestep (for cross-step updates) and the refined map from the current timestep (for in-step updates).
We provide running times (in seconds) of Zero-11to-3 and Hero-to-Hero for the same NFE (66) in Tab. Note that the number of samples reported in the table does not include the input view generated in Zero-to-Hero. Overall, we manage to provide competitive running times. If both MSA and AMF are active (requiring the generation of the input), the running time is increased by approximately 1-1.5 seconds. If only AMF is active, the overhead is much smaller, averaging around 0.5 seconds (there is no need to generate the extra input view).
Table 4: Runtime analysis of the computational overhead of Zero-to-Hero.
Number of Samples
Zero-1-to-3
Zero-to-Hero (Ours)
2.2
3.1
2
2.9
3.9
3.5
4.9
4
4.3
18

---

8.6 Ablation Study
We conduct ablation studies focused on each component on our method, and present the results for the default checkpoint of Bero-1-to-2 and for Zero
In addition to the main ablation table shown in the main paper, present the same study Zero- 1- to-3 Tab. 5 Additionally, we provide qualitative results, demonstrating the common contributions of AMF and MSA in Fig. TO
Hourglass scheduling. We run the original model with 23, 50 and 100 steps sampled uniformly using DDIM, and show that the hourglass scheduling, with 26 steps sampled in total, performs better or on par. The main benefit of our Hourglass scheduling is reducing the number steps, but also slightly improves the overall performance. Results are presented for Zero-1-to: in Tab. and for Zero123-XL in Tab,
Attention map filtering, We use the hourglass scheduling for all further experiments. We run the original model with and without resampling and attention filtering. Our experiments demonstrate that adding the attention filtering in addition to the resampling mechanism improves the performance across all metries. Results are presented for Zero-I-to-3 in Tab. Iand for Zero 1 23--XI. Tab. 7
Early-Stage mutual sell--attenne attentottnnion To the best of our knowledge, most prior works utilizing MSA fall into two categories:
1. Training-free based methods [5] In these works, MSA is typically applied after the general structure of the target is formed to transfer appearance details from the input to the target. In this scenario, MSA does not contribute to the initial structure formation of the target.
2. Training or fine-tuning based methods 22 53 the training or fine-tuning process.
These works incorporate MSA layers within
Our approach is distinct we employ MSA in training-free manner, but crucially, apply it from the beginning of the denoising process until the target structure stabilizes-a phase we term "Farly-Stage Shape Guidance". To validate the impact of early-stage MSA on structure, we conducted a simple experiment using Zero123-XL with 50 DDIM steps. We measure the effect activating MSA at different stages of denoising Intersection over Union (IoU) metric, noting that image quality metrics improved similarly. The results are reported in Tab, TO Our approach demonstrate significant improvement in the structural integrity of the image. Applying MSA in the later stages of the denoising process may hinder the results as it introduces a bias towards the input image.
Table 5: Ablation study Zero-1-to-3 We demonstrate the importance of each of Zero-to- Hero modules, applied to the base method Zero-I-to-3: Sample scheduling (Hourglass), Resampling (Resample), Attention map filtering (AMF), and Early stage Mutual Self-Attention (MSA).
Hourglass
Resample
AMF
MSA
PSNR 1
SSIM t
LPIPS
IOU 1
X
X
x
17.27
0.851
0.173
73.5%
17.29
0.852
0.173
73.6%
X
X
17.40
0.853
0.170
74.3%
17.49
0.855
0.167
75.0%
/
X
V
17.52
0.857
0.165
74.5%
17.67
0.859
0.163
75.3%
Table 6: Ablation study: Hourglass scheduling- - Zero013-XI We demonstrate the superiority of our Hourglass scheduling over uniform DDIM sampling with different number of denoising steps. The experiments are based on Zero X23-XL
Sampling
T
NFE
PSNR 1
SSIM t
LPIPS
IOU 1
Uniform
25
17.72
0.854
0.163
76.4%
Uniform
50
50
17.71
0.854
0.163
76.4%
Uniform
100
100
17.68
0.854
0.163
76.4%
Hourglass
26
26
17.74
0.855
0.162
76.6%
19

---

Table 7: Ablation study: Attention map filtering 23ro123-XI. We demonstrate the importance of Attention Map Filtering (AMF) over only applying Resampling (Resample). The experiments are based on Zero
Name
T
NFE
PSNR 1
SSIM
LPIPS 4
IOU 1
Zero 123XXL
26
26
17.74
0.855
0.162
76.4%
Resample
26
66
17.85
0.857
0.160
77.1%
Resample AMF
26
66
17.92
0.859
0.157
77.8%
Table Ablation study: Hourglass scheduling - -eroo--tto-. We demonstrate the superiority of our Hourglass scheduling over uniform DDIM sampling with different number of denoising steps. The experiments are based on Zero-I-to-3. 1-to-3.
Sampling
T
NFE
PSNR t
SSIM 1
LPIPS |
IOU
Uniform
25
25
17.27
0.851
0.173
73.5%
Uniform
50
50
17.24
0.850
0.173
73.5%
Uniform
100
100
17.21
0.850
0.173
73.4%
Hourglass
26
26
17.29
0.852
0.173
73.6%
Table 9: Ablation study: Attention map filtering - Zero-1-to-3. We demonstrate the importance of Attention Map Filtering over only applying Resampling (Resample). The experiments are based on Zero-I-to-2
Name
T
NFE
PSNR 1
SSIM
LPIPS 4
IOU 1
Zerol23- 22-XLL
26
26
17.29
0.852
0.173
73.6%
Resample
26
66
17.40
0.853
0.170
74.3%
Resample AMF
26
66
17.49
0.853
0.167
75.0%
Table 10: Ablation study: Attention map filtering- - Zeroo--to-3. We demonstrate the importance of Attention Map Filtering over only applying Resampling (Resample). The experiments are based on Zero-1-to-3.
Method
Timesteps where MSA is applied
IOU 1
No MSA
76.4%
All the way
1000 to 0
76.6%
After structure is formed
800 0
76.5%
Ours (from the beginning until structure formed)
1000 to 600
77.6%
8.7 Generation Diversity Ablation Study
While attention filtering manages to generate robust and consistent results, find that excessive it might limit the generation diversity. We explore the two main factors that can reduce the diversity: the number of filtering iterations R and the interval of timesteps [fmin tmaz] where apply filtering.
Number of filtering iterations. In Fig. we show the effect of different values of It on the diversity of the results. All other hyper- paper-paraeeters remain unchanged. We find that as grows, the generation diversity reduces, while the level of realism improves
Filtering schedule. We apply filtering during timesteps [Immn: T). In Fig. effect of different values of bmin on the diversity of the results. All the seper-parameters besides bmin remain unchanged. We observe that as we apply filtering for longer periods of the denoising steps, the results more realistic but less diverse.
we show the
20

---

Source View
Target View
Zero123-XL
+ MSA
+
AMF
Zero-to-Hero
Figure 10: Qualitative results, illustrating where MSA and AMF excel. The first two present cases where MSA fixed black and random textures the target views. This contribution shows larger improvement in image quality metrics, although MSA usually cannot fully resolve significant structural issues. The third and fourth rows show structural improvements achieved with AME Finally, the last shows the combination did the work: the original result of Z 12-XL shows an implausible right hand and incorrect pose of the right sword. MSA lead to out-of-distribution but could not resolve the incorrect pose of the sword. AMF lead to reasonable right hand and placement of the sword, but the sword is not aligned well the the texture of the original sword. The final result show correct structure and texture for the right hand and sword.
where neither technique worked well enough its own, but
stable result, where the hand is
Through our ablation study, we find that in practice, moderate application of attention maps filtering sufficient for improving fidelity while preserving diversity. While excessive resampling might limit diversity (e.g. using large values of R), we only apply our filtering mechanism in the earlier steps of the denoising process, using small value of R (5 in all our experiments). This enables our method to maintain diversity effectively.
When evaluating diversity against base models, we observed that some results of the base models were highly implausible, can be in Fig and Fig. in the appendix. For example, Zero Zero-1-to-3 1-to-3 produces out-of-distributiin results, or results that are not aligned with either the input image or the target pose, leading seemingly larger variance. However, this "diversity" is largely due misalignment and artifacts, which comes at the expense of fidelity.
In our case, our model continues to generate diverse results that are both plausible and better aligned with the conditions (e.g. the various chair backs and turtles heights Fig. I].
21

---

Source
Targel
diversity
20
Figure 11: Generation diversity wrt filtering iterations. We select an extreme change viewpoint, and show how different choices of R, the number of filtering iterations, affect the diversity of generated outputs. As filtering iterations increase, the results become less diverse and more realistic.
Target
Emin
diversity
800
600
Figure 12: Generation diversity wrt filtering schedule. We select an extreme change in viewpoint, and show how different choices the timesteps interval where filtering applied. affect the diversity of results. As filtering is applied for longer periods during the denoising process, the results become less diverse and more realistic.
22

---

8.8 Attention Map Filtering Beyond Novel View Synthesis
In this subsection, we describe several preliminary experiments where we integrate attention map filtering to other base models. Remarkably, the integration of our proposed method into other base models straightforward, demonstrating its applicability and simplicity.
In all the mentioned experiments, resampling is performed during timesteps / E [700, 1000). with R=5. Additionally, the cross-step parameters were slightly tuned for each base model. Mutual Self-Attention is not implemented, it is not immediately applicable.
Conditional image generation. brief study of ControlNet models [58] demonstrated that they suffer from similar limitations as zero123 and its follow ups. Namely, lack of condition enforcement and frequent appearance of visual artifacts. We implemented our proposed AMF module for two pre-trained ControlNet models. In Fig. i the main paper, we provide qualitative results for pose- conditioned ControlNet.
Fig. 13 we provide qualitative results for segmentation cennnaatiionoonntioonnn NontrolNet. We find that attention map filtering robustly mitigates artifacts across various prompts and seeds for both base models.
Multi-view synthesis. We integrate AMF into MVDream 40) text-to-multiview model, and find that it helps mitigate the same issues as the single view case. We provide qualitative results in Fig. 14
Prompt
Segmentation
Pre-trained ControlNet
ControlNet + AMF
A man surfing a large wave
BOX
basketball game
A kitchen made of gold
Figure 13: Qualitative results for segmentation-oomditoone ControlNet. Qualitative results for pre-trained segmentation-condiioon initialized with the same seed. AMF leads results that are more plausible and better align with the conditions, including intricate details (e.g. the base model fails to generate the faucet and the right stool in the third row, while our method generates both and better align with the prompt.)
ControlNet, without and with AMF, Both methods are
23

---

MVDream
MVDream AMF
Prompt: A jumping, while
hal
Prompt: Flamencc dencer
Prompt: man
easinga pizza, highty detallad
Figure 14: Qualitative results for MVDream. Qualitative results for MVDream, without and with AME. Both methods are initialized with the same seed, and we provide results with two random seed per prompt. AMF leads to results that are more plausible and spatially consistent, while also better align with the conditions.
24

---

8.9 Data
Dataset creation. For each object, we render various views by randomly selecting camera parameters. an elevation 0 E [0, R, an azimuth ø E [0, 2 and a radius [1.5, 2.2). We manually select views for each object by removing images captured from close viewpoints and filtering camera viewpoints only from the upper half of the unit sphere, to avoid significant lighting effects that swing the results regardless of the output geometry.
Dataset assets. We provide a list of the objects picked for the challenging subset of GSO.
1. 3D Dollhouse_Siie Sink
26. Olive Kids Mermaids Pack
3D To__Dolloouse_WWi
Snack_ _aanck__Bcpack
3. 3D "D_DDolhouse, Purpo
27.
Ortho_Forwaid Facing
adiZerooSSiid_22 __ice___SC
28. Ortho_ Forward Serward_Facing..c 3icinng3Q612oKJDD22
5. Air Air_Hogs__Winn Wind W Flyers_SettAaee AirplanneeeeeRe
29. Forward Facind_aaiinnne FacenaaaCcaaainenneaee QCaor91mJ2G
BALANCING CACTUUS
30. Racoon
7. Chelsea lo fl rdheel nQ0LPNFIoMw
31. Razer Kraken hoo_headset__ee Balisett_Full____ Full_size Black
CHICKEN BRICKEN_RACER
32. Remington TStudio Hair Dryer
-Stungoo
33. Rubbermaid Larrdd_LLrre___aa Drainer
9. Circo Fish_ _ [oolder_1499530
10. COAST_GUARD__
34. Athleich_Africet Black Rhino
11. SERTIVE_LLCKS_SSS
35. Schleich Hereford Bull
12. Dino_5
36. Schleich_Lion_Actionn Figure
13. Down_ To__ L___Earth_CCC Farth_Ceramic_Bae Prramic_Orchid_Pae Asst_Blue
37. Schleich_5 S S_Bayalaaaaaaaaa Ln__Balla__iioomm______ 77432
38. Smith_ Hawken Woven BasketTray 0rrganiie C_mpparmnmtrtteets 95 9 x___3
14. FIRE_ENC TRR_NENGIE
15. Great Dinos_ Great_Dinos__ Latteiiiiinnoatttttooooo
Triceratops__oy
39. Lanny_SchoolBuu
16. Guardians _o tht_tteeeech Sacilaay__alaate_Heae
40. Sootheze_ Theraphy
41. SORTING_TRAN
17. Hilary
42. Toysmith Beiiit_WWide______
18. Tiaginext_Castle. Ogrr
43. Squirrel
19. My_First_Rae Siirs_Roolingg_Lin
44. Transformers Lattineerss_____e of Extinction_Meegee Basioon_MMeaa_SSSe
Age_of
20. My_First_Wiggle_Crooe
21. Bttpepoooy_Pinces__eellll
45. Baciar_Procutt_Peee__e
_XL in Black
22. Nickelodeon Baceelooeeon_Teeageeeeeeee Bartengge_Muaat_NNNN
46. Meec__Rol_LLemm_
23. Nickelodeon Leeii_eeeeeeeeeeeeeeeeeeeee
Mutant_
47. W L__Lu_z0dk78mng
48.
24. Nintendo Latirndd_Maioo_ttttnnnnno_e
49. They__Poteinn_aaae
25. Nintendo_
50. ZX700_mzGidP3u66B
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction reflect all our main contributions for the goal
of improving the reliability and quality of novel views generated by Zero-1-to-3 and its variations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper both discuss extensively the limitations of the work, and provide an example for failure demonstrate these limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer:
[NA]
Justification:
The paper do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all implementation details, both for our method and dataset creation, making our method and results fully reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility e can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path to reproducing verifying the results.
5. Open access to data and code
27

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We intend to release all code and data used in our paper in time for the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
ootpee///ooooeSbbiaiiioooooooo ieaibbmmsssooPPoo asionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies all relevant details to understand all the experiments and results shown.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each generation task (defined by a source view and camera viewpoint) is generated 3 times with different seeds, and the final score is the average of all score.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We note all experiments were run on a single NVIDIA RTX 4090.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htters//neurrps.. http://www...pppp.oo./ccccccccaa
Answer: [Yes]
Justification:
The research follows the NeurIPS Code of Ethics in full.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts: both positive and negative.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification:
Our method is training-free free, builds upon an already released pre- pre-trained model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that introduces Google Scanned Objects and all license details will be provided with the code and data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppppappphhaaocc thcode. com com/datasets
30

---

has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not create 3D new assets. However, we generate a dataset of renderings that is well documented and will be released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper doesn't 't involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper doesn't 't involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "WpxRUD7ngh", "reviewer": "Reviewer_UpfM", "summary": "This paper experimentally analyzes which parts are important and responsible for generation artifacts.\nTo solve it, this paper propose an attention map filtering process.\nThis process share the similar idea with SGD, which reduces the error of the generation process by repeated sampling.\nThis paper also propose some other things to enhance the results, including identity view information injection and a specialized sampling schedule.\nThe whole pipeline is training-free, so it’s very easy to apply to the pre-trained diffusion model.", "questions": "* Ablation studies of zero123 and zero123-XL showed different trends. MSA even impaired the results, but MSA with AMD improved the results. What is the reason do you think?\n* Form the quantitative results, the improvement brought by MSA is significantly greater than that of AMF. Are there any qualitative results on MSA and AMF?\n* How does this method affect the speed of the original diffusion model? Does the choice of R affect the model results?\n* Fig 3 and 7 seem to share some of the same samples, maybe we can show more cases and perspectives?", "limitations": "This method is limited by the generative capabilities of the pre-trained model, if the results wrong too much, it does not have the ability to correct it.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper propose a method to improve the  novel-view diffusion model without external model and training, it is very easy to use and can be applied to different novel-view diffusion models easily.\nThe motivation of this article is clear, and the process and details of thinking and discovery is demonstrated.", "weaknesses": "* The model has no training cost, but there is a lack of thorough analysis of the additional inference cost.\n* Qualitative results are limited. Given that the quantitative results do not show much improvement, more qualitative comparisons would be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their insightful feedback and comments.\n\n**1. Inference cost analysis**\n\nOur proposed modules add a computational overhead to the base model. In the paper, we addressed this by counting the overall number of function evaluations (NFE) and keeping it on par with the base model to ensure a fair comparison. As requested, we now discuss the individual computational overhead of each module. We will incorporate this analysis into the manuscript for completeness.\n\n(1) Resampling: Similar to the total number of denoising steps, T, the number of resampling iterations, R, linearly increases the NFE. Our chosen value for R and the timesteps at which we apply it resulted in mapping 26 denoising steps to a total of 66 NFE.\n\n(2) MSA: The main additional cost of MSA is the necessity to generate the input view in addition to the target views, meaning the effective number of generated samples is increased by one.\n\n(3) AMF: We implement attention map filtering in the upsampling part of the UNet and apply it during the early steps of the denoising process. We maintain two additional instances of each attention map: the attention map from the previous timestep (for cross-step updates) and the refined map from the current timestep (for in-step updates).\n\nWe provide a table showing the running times (in seconds) of Zero-1-to-3 and Zero-to-Hero for the same NFE (66). Overall, we manage to provide competitive running times. If both MSA and AMF are active (requiring the generation of the input), the running time is increased by approximately 1-1.5 seconds. If only AMF is active, the overhead is much smaller, averaging around 0.5 seconds.\n\n\n\n|Samples Num|Zero123|Ours|\n|-|-|-|\n|1|2.2|N.A|\n|2|2.9|3.1|\n|3|3.5|3.9|\n|4|4.3|4.9|\n\n**2. Additional qualitative results.**\n\nPlease refer to Figure 1 in the supplementary material for additional qualitative comparisons of view synthesis by our Zero-to-Hero and the baseline Zero-123-XL. As can be seen, the improvement demonstrated by our method is consistent across objects and seeds. Further qualitative results demonstrating our method on other base models are discussed in General Comment #1.\n\n**3. The extent of quantitative improvement.**\n\nWe have provided a clarification in General Comment #4.\n\n**4. Qualitative demonstration of MSA and AMF.**\n\nWe appreciate this suggestion. Indeed, the difference in quantitative improvement between these two modules does not faithfully reflect their effect. We thus chose to share the response with all reviewers in General Comment #1 and Figure 1 (right) in the supplementary material.\n\n**5. The effect of MSA and AMF on zero123 and zero123-XL.**\n\nThis is a good observation. Firstly, regarding the different effects of MSA and AMF, please refer to General Comment #3. Here, we address the reviewer's inquiry regarding the *difference in the modules' effects on both base models*. Our MSA (MSA with early termination) generally improves performance in both base models. As pointed out by the reviewer, in Zero-1-to-3, MSA did not improve PSNR and IoU while boosting SSIM and LPIPS. In Zero123-XL, the improvement was consistent across all metrics. A potential explanation for this difference may lie in the inherent bias of MSA towards the input view. Roughly speaking, MSA copies source details to improve generation consistency. Thus, too much of it might impair the results as it would lead to a bias towards the appearance and pose of the input image. To control this effect, we introduced early stopping. The termination step in our work was chosen based on Zero123-XL (where MSA consistently improved all metrics). While we found that the same parameters generalized well for the overall performance of Zero-1-to-3 without further tuning (the combined MSA and AMF improved all metrics and addressed the same issues as Zero123-XL in visual results), the ablation reveals that the parameters may not be optimal for the MSA effect alone in Zero-1-to-3.\n\n**6. Choice of resampling steps R.**\n\nWe found that the model is not very sensitive to the choice of R, and values within the range of 4-8 provide similar results. While some objects benefited from larger values (`~10`), the overall improvement was minimal. Additionally, we found (as shown in Figure 8 in the appendix of the paper) that large values of R (`~15-20`) can limit diversity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXDsW49fTF", "reviewer": "Reviewer_qQr5", "summary": "This paper proposes attention map filtering to enhance the novel view synthesis performance of Zero-1-to-3. The method is composed of several changes to the original sampling method, but the main contribution is an aggregation of attention map strategy by resampling the same denoising step multiple times. The paper draws an analogy to SGD to argue that resampling resembles the batch training in SGD, enhances the stability of the sampling process, thus improving the performance. Controlled experiments show that the proposed technique is effective in improving the quality of the generation", "questions": "Please see the weakness section for questions.", "limitations": "The paper has discussed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-written. Related works include sufficient references. The method is clear and well-structured. The figures are also well-made and clear. I had no trouble understanding the method and the results presented in the paper.\n\n- The method is simple yet seems to be effective. The technique can be plug-and-play to apply to diffusion-based NVS methods such as Zero123.\n\n- The analysis is great. The authors perform interesting analysis on attention maps of the denoising layers, the decreased diversity in generation, and limitations. I think these analysis helps a lot in building intuition behind the method and can be useful for future works to understand the nature of the problem.\n\n- The experiments are controlled. Though the authors didn't compare against the various latest 3D generative models, they focus on evaluating the effectiveness of the proposed technique by running controlled studies. At a time when there are so many papers coming out every day, controlled and scientific experiments are valuable.", "weaknesses": "- Following the analogy introduced by the authors between diffusion and SGD, it seems unavoidable that aggregating attention maps across multiple sampling steps will compromise the model on generation diversity. As we can imagine, optimization with a larger batch size is more stable. So, even though the authors have proposed techniques to mitigate the problem of lack of diversity, I believe it is a more fundamental limitation.\n\n- Since the techniques proposed seem general to all diffusion models, what prevents one from applying the technique to all diffusion generation tasks? Given the vast amount of literature and problems being solved by diffusion models, the potential impact could be significant. Are there any specific constraints such that the method is only applicable to the problem of novel view synthesis?\n\n- Does the method lead to lack of visual details in the generated views? I noticed that the eyes of the generated chickens are missing in the last row of figure 1. Is this a general problem of the proposed techniques?\n\n- It's a little unfair to claim that the proposed method can maintain the same computational efficiency. The proposed techniques to accelerate the diffusion process is not exclusive to the proposed techniques and can be applied to the base model as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for insightful feedback and comments.\n\n**1. Effect of AMF on Generation Diversity.**\n\nWe agree that excessive use of Attention Filtering (R>>1) may reduce generation diversity, as analyzed in the appendix. We will incorporate these insights into the limitations section. The balance between diversity and fidelity is a complex topic that warrants further study. Our findings indicate that \"diversity\" in the base model often includes artifacts and deviations from the real-world distribution, as illustrated in Figure 1 of the main paper and Figure 1 (left) of the rebuttal Supplementary Material. Responsible use of Attention Map Filtering usually helps preserving diversity while preventing results from deviating from the real distribution.\n\n**2. Generality and applicability to other diffusion models.**\n\nThank you for highlighting this important point. Although our work focuses on novel view synthesis, we agree that the AMF module is potentially broadly applicable. This was indicated in the future work section. Inspired by your comment and by the request of reviewer 3GQx, we have extended Attention Map Filtering to other models beyond the task of novel view synthesis. Remarkably, we achieved noticeable improvements out-of-the-box in all cases. Please refer to General Comment #2 for more details.\n\n**3. Visual details in generated views.**\n\nWe have not observed a general trend of lacking visual details in the generated views. Figures 1 in both the main paper and the supplementary materials demonstrate examples with fine details, such as the Android's antennas and the squirrel's eyes. The loss of the chicken's eye in one example may be due to resampling issues. While our pipeline usually mitigates the oversmoothing effect of vanilla resampling, it is not infallible.\n\n**4. Computational efficiency.**\n\nWe acknowledge the reviewer's point regarding computational efficiency. Our primary goals were to improve quality and consistency while keeping generation times competitive to ensure applicability. We will clarify our claims in the paper and are open to further adjustments based on the reviewer's suggestions. An analysis of the computational cost of each module is provided in the response to reviewer UpfM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7vyXKshcTL", "reviewer": "Reviewer_3GQx", "summary": "This paper proposes a novel approach to generate realistic images from arbitrary views based on a single source image. The authors introduce a test-time method that enhances view synthesis by manipulating attention maps during the denoising process of diffusion models. This process improves geometric consistency and realism without requiring retraining or significant computational resources. The key contributions include an attention map filtering mechanism, a modification of the self-attention mechanism to integrate source view information, and a specialized sampling schedule. Experimental results show substantial improvements in fidelity and consistency, demonstrating the effectiveness of the proposed method.", "questions": "1. Can Zero-to-Hero be equipped with other multi-view diffusion models, e.g,,MVDream, Wonder3D?\n2. The quantitive metrics is calculated in which views?\n3. Will Zero-to-Hero hurt the diversity of Zero123 model with relatively consistent outputs among different seeds?", "limitations": "Please refers to W and Q. I will consider to raise my score if the author could address my concerns and provide more convinced results.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a unique test-time method for enhancing view synthesis, specifically manipulating attention maps during the denoising process, which is innovative and effective.\n2. The paper is generally well-written and easy to follow.", "weaknesses": "1. Lack of qualitive results in the experimental part and even in Appendix I cannot find more comparison.\n2. Only compared and equipped with zero123 and zero123-XL is not enough also.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for insightful feedback and comments.\n\n**1. Additional Qualitative Results.**\n\nWe kindly refer the reviewer to General Comment #1 and the supplementary material, where we have included further qualitative results.\n\n**2. Applicability of Zero-to-Hero in Multiview Diffusion.**\n\nThank you for raising this critical point. While Zero-to-Hero was initially designed for single-view generative models, our Attention Map Filtering can indeed be extended to other diffusion models, including multiview synthesis. Based on your suggestion and that of reviewer qQr5, we have implemented the attention map filtering mechanism for MVDream and two pre-trained ControlNet models, observing significant improvements. Please refer to General Comment #2 for more details on these implementations.\n\n\n**3. Quantitative Evaluation — Additional Details.**\n\nFor our quantitative evaluation, we rendered 8 random views of each object and used each view as a source. For each source view, we generated the remaining 7 views, resulting in a total of 56 generation tasks per object (each defined by a unique source-target pair). Each target view was generated 3 times with different seeds, yielding a total of 168 images per object. We then averaged the scores across all views and objects to ensure a comprehensive evaluation.\n\n\n**4. Effect of Zero-to-Hero on Generation Diversity.**\n\nIn general, excessive use of attention map filtering might reduce generation diversity. We have analyzed this in the appendix of our paper (Figures 8 and 9). However, we find that responsible usage usually preserves diversity. For instance, in Figure 1 of the main paper, the generated chairs display diverse back sides, and the turtles in the third row further demonstrate the model's ability to produce varied results. The balance between diversity and fidelity is complex and warrants further study. Often, the \"diversity\" observed in the base model includes artifacts and deviations from the real-world distribution, as illustrated in Figure 1 of the main paper and Figure 1(left) in the supplementary material.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y98YJVZ6ev", "reviewer": "Reviewer_hzMx", "summary": "This paper proposes a test-time approach that adaptively modifies the attention map during inference to enhance the consistency and plausibility of novel view synthesis. Experimental results on GSO demonstrate improved performance with the proposed method, and the authors conducted ablation studies to assess the impact of different modules on performance.", "questions": "The paper includes several hyperparameters such as the number of resampling iterations (R) and the alpha for cross-step aggregation. How did the authors set these hyperparameters, and how significantly do they impact the final results?", "limitations": "The method proposed by the authors seems intriguing, but the actual improvement in results is limited. Moreover, the largest gain comes from a commonly used trick for enhancing viewpoint consistency, Mutual Self-Attention (MSA).", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors carefully analyze the challenges and issues of the current zero-1-to-3 method on novel view synthesis.\n2. The idea proposed in the paper to filter attention maps akin to SGD is interesting as it can enhance the robustness of attention maps to a certain extent.", "weaknesses": "1. Since (Re-)Sampling increases the computational load, with the hyperparameter R determining the number of resampling iterations, it would be beneficial for the authors to conduct an ablation study on how this parameter affects both results and computational overhead.\n2. The method proposed by the authors has shown limited improvement in results. They need to validate the effectiveness of their proposed approach on a larger and more diverse dataset.\n3. When conducting the ablation study, the authors' selection of only 10 objects from GSO could introduce bias into the results. It would be preferable for them to include all objects to ensure a more comprehensive evaluation.\n4. The authors' proposal of Mutual Self-Attention (MSA), which has been used in other papers such as Consistent123, Tune-A-Video, and MasaCtrl, cannot be considered a unique contribution in their paper.\n5. In the ablation study (Table 2), Mutual Self-Attention (MSA) shows the largest improvement, with a greater increase in PSNR (MSA 17.82 vs AMF 17.58, last two rows). This suggests that the method's improvement primarily stems from MSA rather than the core contribution point, AMF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for insightful feedback and comments.\n\n**1. Setting the Hyperparameters R and alpha.** \n\nAll hyperparameters were tuned based on a small random set of objects. Due to limited computational resources, the tuning process was not exhaustive. We found that the method's performance is not highly sensitive to these parameters.\n\n- **Cross-step weight (alpha)**: We experimented with values ranging from 0.1 to 0.9 in increments of 0.1. Alpha determines the weight assigned to previous predictions in the cross-step aggregation. In Zero123-XL, early predictions were generally reliable, so a larger weight yielded better results. We maintained the same parameters for Zero-1-to-3. In new experiments conducted for the rebuttal, we applied our method to additional models such as ControlNet and MVDream. In both cases, a smaller weight generally produced better outcomes.\n\n- **Resampling iterations (R)**: We observed that the model's performance is relatively insensitive to the choice of R, with values between 4 and 8 yielding similar results. While some objects benefited from larger values (`~10`), the overall improvement was minimal. Additionally, as shown in Figure 8 in the paper's appendix, large values of R (`~15-20`) can reduce diversity. For additional experiments with models like ControlNet and MVDream, we fixed R at 5 and did not test other values.\n\n**2. Additional Computational Cost of Resampling.**\n\nResampling iterations are computationally equivalent to adding more denoising steps, as both linearly increase the number of function evaluations (NFE). In the paper, we addressed this by counting the NFE and keeping it comparable to the base model to ensure a fair comparison. Our chosen value for R, and the specific timesteps where we applied it, resulted in mapping 26 denoising time steps to a total of 66 NFE. For further details, please refer to our response #1 to reviewer UpfM, where we discuss the individual computational overhead of each module.\n\n\n**3. The Extent of Quantitative Improvement.** Please refer to General Comment #4 for a detailed discussion of the quantitative improvements.\n\n**4. Evaluating on Additional Dataset.**\n\nWe evaluated our approach on the RTMV benchmark [1], an out-of-distribution dataset consisting of 3D scenes, each contains 20 random objects. Here, we present the results for Zero123-XL, using the same hyper-parameters reported in the paper. Our evaluation shows improvements across all metrics compared to baselines. GSO and RTMV are the common datasets used for evaluation in Zero1-to-3 and its follow-ups.\n\n| |T|NFE|PSNR|SSIM|LPIPS|IoU|\n|-|-|-|-|-|-|-|\n|Base|25|25|10.51|0.589|0.396|71.5%|\n|Base|50|50|10.54|0.589|0.393|71.9%|\n|Base|100|100|10.53|0.588|0.392|72%|\n|Ours|26|66|11.18|0.619|0.372|73%|\n\n**5. Including all objects in the ablation study.**\n\nThank you for your suggestion. In response, we have conducted the ablation study across all test objects, focusing on Zero123-XL. The new results corroborate the trends observed in our initial submission. We will ensure that the comprehensive evaluation is included in the camera-ready version.\n\n|Hourglass|Resample|AMF|MSA|PSNR|SSIM|LPIPS|IoU|\n|-|-|-|-|-|-|-|-|\n|-|-|-|-|17.27|0.854|0.162|76.4%|\n|+|-|-|-|17.44|0.855|0.161|76.6%|\n|+|+|-|-|17.7|0.857|0.160|77%|\n|+|+|+|-|17.92|0.858|0.157|77.8%|\n|+|+|-|+|18.25|0.862|0.155|77.6%|\n|+|+|+|+|18.35|0.864|0.153|78.3%|\n\n**6. The usage of Mutual Self-Attention in Zero-to-Hero.**\n\nWe acknowledge that the formulation of MSA was not introduced in our work. We have cited prior works such as MasaCtrl in our paper, and we will include the additional references suggested by the reviewer. However, we wish to highlight the distinctions between previous works and our unique application of this technique. \nTo our knowledge, most prior works utilizing MSA fall into two categories: \n\n* Training-free usage of MSA (e.g., MasaCtrl): In these works, MSA is typically applied after the general structure of the target is formed to transfer appearance details from the input to the target. In this scenario, MSA does not contribute to the initial structure formation of the target.  \n* Training or fine-tuning models with MSA layers (e.g., Consistent123, Tune-A-Video): These works incorporate MSA within the training or fine-tuning process.\n\nOur approach is distinct as we employ MSA in a training-free manner, but crucially, we apply it from the beginning of the denoising process until the target structure stabilizes—a phase we term \"Early-Stage Shape Guidance\" in our paper. This early application of MSA leads to more stable results and often prevents the model from generating out-of-distribution outputs (like the leftmost and rightmost chairs in Figure 1 in the paper).\n\nTo illustrate the impact of early-stage MSA on structure, we conducted a simple experiment using Zero123-XL with 50 DDIM steps. We measured the effect of activating MSA at different stages of denoising on Intersection over Union (IoU) metric, noting that image quality metrics improved similarly. Our approach demonstrated a significant improvement in the structural integrity of the image. We believe that applying MSA in the later stages of the denoising process introduces a bias towards the input images, which can disrupt the shape and appearance.\n\n|Method|Timesteps where MSA is applied|IoU|\n|-|-|-|\n|No MSA|-|76.4%|\n|All the way|1000 to 0|76.6%|\n|Start after the structure is initially formed|800 to 0|76.5%|\n|Ours (from the beginning and terminate early)|1000 to 600|77.6%|\n\n**7. The Contribution of MSA vs. AMF.**\n\nMSA and AMF address different artifacts and complement each other. MSA's greater impact on metrics does not diminish AMF's value. The combined methods show consistent improvements. For more details, please see General Comment #1 and Figure 1 (right) in the supplementary materials.\n\n[1] RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers and ACs for their efforts in reviewing our work. 
We are encouraged by the recognition of our work's innovative perspective of attention map filtering as analogous to optimization process (hzMx, 3GQx), applicability and effectiveness (UpfM, qQr5). We are pleased that the thoroughness of our analyses was appreciated (UpfM, hzMx, qQr5).  Below, we address the concerns raised by the reviewers.

**1. Additional results**

We provide additional qualitative and quantitative results in the Supp. These results reinforce the effectiveness of Zero-to-Hero and demonstrate its generalization to additional datasets and models. (1) Figure 1 (left): Additional view synthesis results of Zero-to-Hero. Figure 1 (right): results that exemplify the individual contributions of the proposed MSA and AMF. Figure (2): Our AMF applied to pose- and segmentation-conditioned generation pre-trained with ControlNet. Figure (3) Our AMF applied to Multiview synthesis model MVDream. (4) Evaluation on the RTMV, a dataset of 3D scenes. Please refer to our response to reviewer hzMx for further details.

**2. Zero-to-Hero generalization beyond NVS**

Although our work addresses the core limitations of single view synthesis models, the condition enforcing effect of our proposed modules are more general. As mentioned in the conclusions, we intend to leave the in-depth exploration of other applications to future work. However, as the generality of our method seemed to draw much interest by the reviewers we have conducted several preliminary experiments which demonstrate promising results. Remarkably, the integration of our proposed method into other base models was straightforward, demonstrating its applicability and simplicity. 
(1) Pose- and Segmentation-conditioned image generation. A brief study of ControlNet models demonstrated that they suffer from similar limitations as zero123 and its follow ups. Namely, lack of condition enforcement and frequent appearance of visual artifacts. We implemented our proposed AMF module for two pre-trained ControlNet models and found that it robustly mitigates artifacts across various prompts and seeds. Please note that MSA is not immediately applicable for these models and thus was not used.
(2) Multi-view synthesis. We integrated AMF into MVDream, a text-to-multiview model, and found that it helps to mitigate the same issues as in the single view case. Similarly, MSA was not implemented.

**3. The Contribution of MSA vs. AMF**

Reviewers UpfM and hzMx, while the image quality metrics show a larger improvement with MSA, these numbers do not tell the whole story. AMF and MSA address different artifacts in the generation process and complement each other. The fact that MSA contributes more to the metrics does not take away from AMF's individual contribution. This complementary effect is demonstrated by the consistent improvement observed when both methods are used in tandem compared to using each individually.

- **MSA** (Mutual Self-Attention) transfers information from the input view to the target, assuming similar appearance and textures. This method is particularly effective when the input and target views are relatively close. As an example, Zero123 sometimes generates regions with plain black or random textures in the target views which MSA mitigates well, as shown in the first two rows of Figure 1 (right) in the Supp. We note that image quality metrics are more sensitive to this improvement and thus show more significant gains than when improvement in shape is achieved. However, MSA alone is usually insufficient for refining the pose or structure of the target. Its inherent bias towards the input shape and appearance may harm the results, especially when the change of viewpoint is significant. Our analysis therefore led us to utilizing MSA only during the early steps of the denoising process, named Early-Stage Shape Guidance in our paper.

- **AMF** (Attention Map Filtering), on the other hand, excels when the change in viewpoint is larger. While it may not always improve color and textures, AMF leads the model to produce more probable results that align better with the real distribution. We observed that most of the structure refinement is done by AMF. Unfortunately, none of the metrics measure plausibility, so this effect is not faithfully reflected by the evaluation metrics.

We have included a new figure demonstrating where MSA and AMF excel. The first two rows illustrate why MSA shows a larger improvement in image quality metrics, although it usually cannot fully resolve significant structural issues. Rows 3 and 4 show the structural improvement achieved with AMF. Finally, the last row shows a case where neither technique worked well enough on its own, but the combination did the work.

As further testimony to the significant effect of AMF, we refer the reviewers to general comment #2, where AMF is used to boost other generative models such as ControlNet and MVDream. These examples exhibit similar issues to Zero123 and demonstrate the role of AMF in mitigating them.

**4. The Extent of Quantitative Improvement**

Reviewers UpfM and hzMx raised a concern regarding the extent of improvement in the quantitative results reported in Table 1 of the main paper. While the absolute improvement may not seem large, it is important to put it in context to appreciate its significance. Zero123-XL improved upon Zero123 using the same base model by using 10x more data, achieving gains of [0.45, 0.003, -0.01, 2.9%] in PSNR, SSIM, LPIPS, and IoU, respectively. Our method achieved comparable gains [0.37, 0.008, -0.01, 1.7%] with no additional data and no further training. Remarkably, our method demonstrates similar and slightly larger gains when applied to Zero123-XL [0.63, 0.1, -0.01, 1.9%], a boost in performance that couldn't be achieved with merely more data. Also, the ratio of improvement is larger compared to other training-free methods (e.g. ViVid123).

Meta-review: The reviewers acknowledge the technical contribution of proposing a training-free method and its effectiveness, along with a clear presentation. The rebuttal was successful, resulting in an increased average score, with previous negative scores now becoming supportive. After careful discussion and consideration, we are pleased to inform you that your paper has been accepted. However, the final version will require revisions to reflect the important discussions presented in the rebuttal. For example, please discuss the diversity issue, clarify the distinction from the previous MSA module, and include additional experimental results.
Decision: Accept (poster)