Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering

Authors: ido  sobol
chenfeng  xu
or  litany
u c berkeley
  nvidia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 15:27:44

Abstract: Generating realistic images from arbitrary views based on a single source image remains a significant challenge in computer vision, with broad applications ranging from e-commerce to immersive virtual experiences. Recent advancements in diffusion models, particularly the Zero-1-to-3 model, have been widely adopted for generating plausible views, videos, and 3D models. However, these models still struggle with inconsistencies and implausibility in new views generation, especially for challenging changes in viewpoint. In this work, we propose Zero-to-Hero, a novel test-time approach that enhances view synthesis by manipulating attention maps during the denoising process of Zero-1-to-3. By drawing an analogy between the denoising process and stochastic gradient descent (SGD), we implement a filtering mechanism that aggregates attention maps, enhancing generation reliability and authenticity. This process improves geometric consistency without requiring retraining or significant computational resources. Additionally, we modify the self-attention mechanism to integrate information from the source view, reducing shape distortions. These processes are further supported by a specialized sampling schedule. Experimental results demonstrate substantial improvements in fidelity and consistency, validated on a diverse set of out-of-distribution objects. Additionally, we demonstrate the general applicability and effectiveness of Zero-to-Hero in multi-view, and image generation conditioned on semantic maps and pose.

Main paper: 
 introduction: 
 The pursuit of realistic image synthesis at arbitrary views, given only a single source image, has long been a cornerstone challenge in computer vision and graphics. This technology can cater to countless applications, such as interactive product inspection, robot-scene interaction, and immersive virtual experiences. In this work, we aim to advance this important line of research by improving the generation of novel views that are plausible and faithful to the input image. A recent, promising approach, Zero-1-to-3 [27] has developed a foundation model to synthesize novel views based on a single source image and a target view angle. By leveraging a pre-trained, image-conditioned stable diffusion model backbone [3] , fine-tuned with target camera poses, and trained on paired source and target views from a vast collection of 3D models [10, 9] , Zero-1-to-3 can generalize beyond its training set and generate plausible novel views. As a result, this model has quickly gained popularity, inspiring subsequent work in 3D and 4D scene generation [8, 24, 39, 28, 26, 25, 57, 33, 45, 35, 20] .Despite its remarkable ability, Zero-1-to-3 has been observed to generate views that are implausible, or inconsistent with the input object in terms of shape and appearance [24, 8] . Previous works have tried to mitigate these issues by retraining diffusion models with more data [9] or to generate multiple views [39, 25, 26, 24, 8, 28, 56] . Despite substantial improvement, both approaches are resource-intensive due to the required re-training on large-scale 3D datasets. Another line of work attempts to consolidate inconsistencies across multiple generated views through a 3D representation like NeRF [14, 30] . However, direct aggregation often results in blurry outputs, as observed by [24] . Instead, ViVid-1-to-3 [24] employed a multiview representation that naturally supports the use of a video foundation model. Nevertheless, this approach requires generating the entire trajectory from the source to the target view, adding significant complexity and computational overhead. Notably, the denoising process in Zero-1-to-3 remains unchanged.In this work, we propose Zero-to-Hero, a novel test-time technique that addresses view synthesis artifacts through attention map manipulation. Recognizing attention maps as crucial for latent predictions, we hypothesize that enhancing robustness in attention maps predictions can significantly reduce generation misalignment. To achieve this, we draw an analogy between the denoising process in diffusion models and stochastic gradient descent (SGD) optimization of neural networks. Specifically, we relate network weights that predict local gradients at each optimization step, based on sampled training examples and labels, to the denoising network's attention maps that predict latent representations from sampled noise at each denoising step. In this work, we view the generation (denoising) process as an unrolled optimization, with attention maps as parameters of a score prediction model. Inspired by gradient aggregation and weight-averaging techniques that improve prediction robustness (e.g., consistency training [47] ), we propose a filtering mechanism to enhance attention map reliability. This mechanism comprises iterative map aggregation within denoising steps and map averaging across denoising steps. The result is more reliable maps, particularly during the early denoising stages when coarse output shapes are determined, leading to more plausible and realistic views.To further promote consistency with the input, we modify the self-attention operation by running a parallel generation branch using the identity pose, incorporating its keys and values into the attention layer of the target view. Unlike previous applications of this technique [29, 5, 1] ), we find it beneficial in view synthesis to limit its use to the early denoising stages, preventing shape distortions. Our unique denoising procedure is further complemented by a novel sampling schedule that emphasizes early and late denoising stages, maximizing performance. Our main contributions are as follows:• To address the main limitations of the Zero-1-to-3 model, we perform an in-depth analysis and identify self-attention maps as the main candidate for correcting generation artifacts.• We establish a conceptual analogy between model weights in stochastic gradient descent-based network training and the role of attentions map updates during generation of a denoising diffusion model. Based on this, we propose a simple yet powerful attention map filtering process resulting in enhanced target shape generation. We supplement our filtering technique with identity view information injection and a specialized sampling schedule. • Our method requires no additional training, and it avoids the overhead of external models or generating multiple views.Through comprehensive experiments on out-of-distribution objects, we demonstrate that our technique robustifies Zero-1-to-3 and its extended version, Zero123-XL, leading to views that are more faithful to both the input image and desired camera transformation. Our results show significant and consistent improvements across both appearance and shape evaluation metrics. Additionally, we find that Zero-to-Hero naturally generalizes to additional tasks including multi-view, and image generation conditioned on semantic maps and pose. In all cases, we observed significant improvement in condition following and visual quality.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction reflect all our main contributions for the goal of improving the reliability and quality of novel views generated by Zero-1-to-3 and its variations.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
novel view synthesis with diffusion models: 
 Diffusion models have dominated various generative applications [17, 11, 38, 36, 41] . Particularly, novel-view synthesis, as a core of applications like augmented reality and simulations, naturally enjoys the benefits of high-fidelity zero-shot synthesis with diffusion models. One line of works [51, 27, 44, 22, 54, 24] is to generate a novel-view image given a source image (i.e., image-to-image). These approaches typically involve training a diffusion model conditioned on both an arbitrary camera pose and the source view. For instance, the representative work, Zero-1-to-3, fine-tunes a pre-trained Stable Diffusion model [36] by replacing the text prompt with camera pose and CLIP features [34] . Moreover, another research trajectory [32, 21, 14, 7, 46] proposes generating a 3D representation from a single image (i.e., image-to-3D), which allows for sampling desired views from these 3D models. Our method, Zero-to-Hero, builds on the first approach (specifically Zero-1-to-3 and Zero123-XL) and distinguishes itself by eliminating the need for extensive training. Instead, it offers a test-time, plug-and-play approach that significantly enhances visual quality and consistency.A common test-time refinement strategy in diffusion generation is leveraging guidance [29, 2, 18] to direct the sampling process with additional conditions. For example, Repaint [29] utilizes a maskthen-renoise strategy to refine the generation results. Repaint also introduces a per-step resampling technique, where given a horizon-size h, a latent z t is re-noised to z t+h and then denoised again to z t multiple times. They observe that resampling helps to generate more harmonized outputs, given an external guidance or condition. Restart [55] offers a sampling algorithm based on a variation of resampling within a chosen interval of steps. Our method is inspired by the strategy of per-step resampling. We show that it serves as a powerful correction mechanism throughout the generation process, even when no external guidance or condition is provided.Stable Diffusion [36] utilizes attention to enforce the condition information onto the generated results.Previous works demonstrate that manipulating the attention operation can achieve new capabilities [49, 5, 1, 57] . For example, MasaCtrl [5] uses Mutual Self-Attention where source and target images are generated jointly while sharing information, by injecting source image keys and values to the target through self-attention. Here we employ Mutual Self-Attention in the context of novel view synthesis. Differently to prior works we find it beneficial to limit its use to the early denoising stages.3 BackgroundWe observed that the model's performance is relatively insensitive to the choice of R, with values between 4 and 8 yielding similar results. While some objects benefited from larger values (∼10), the overall improvement was minimal. Additionally, as shown in Fig. Additional Inference Cost Analysis. Our proposed modules add a computational overhead to the base model. In the paper, we addressed this by counting the overall number of function evaluations (NFE) and keeping it on par with the base model to ensure a fair comparison. We discuss the individual computational overhead of each module.1. Resampling: Similar to the total number of denoising steps, T, the number of resampling iterations, R, linearly increases the NFE. Our chosen value for R and the timesteps at which we apply it resulted in mapping 26 denoising steps to a total of 66 NFE. Resampling does not introduce additional computational cost over adding standard denoising steps.The main additional cost of MSA is the necessity to generate the input view in addition to the target views, meaning the effective number of generated samples is increased by one.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper both discuss extensively the limitations of the work, and provide an example for a failure case to demonstrate these limitations. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
zero-1-to-3: challenges and limitations: 
 Zero-1-to-3 is a pioneering method for novel view synthesis based on a diffusion model, which has gained significant popularity. This model is built upon the image-conditioned variant of Stable Diffusion (SD) [36] , fine-tuned specifically for novel view synthesis. Zero-1-to-3 is conditioned on a source image and relative transformation to the desired view angle [R|T ]. Maintaining the SD architecture, these conditions are integrated in two ways. First, a CLIP [34] embedding of the input image is concatenated with the relative transformation [R|T ] and mapped to the original CLIP dimension to form a global pose-CLIP embedding, interacting with the UNet layers through cross-attention, enriching the generation with high-level semantic information. In parallel, the input image is channel-concatenated with the denoised image, helping the model preserve the identity and details of the synthesized object.While Zero-1-to-3 [27] has achieved substantial progress in novel view synthesis, several common issues limit its practical application. Firstly, the generated images might not fit real-world distributions, resulting in implausible and unrealistic outputs (e.g., first row in Fig. 1 ). Secondly, the target image may appear plausible but be inconsistent with the input image in terms of shape or appearance (e.g., fifth row in Fig. 1 ).In this work, we identify the critical role of self-attention maps in high-quality generation and propose a novel filtering process that enhances robustness without requiring further training. This process addresses the aforementioned issues, ensuring reliable and consistent results.In this work, we draw a conceptual analogy between gradients and weights in stochastic gradient descent (SGD), and latents and attention maps in denoising diffusion models. Leveraging this analogy, we adapt techniques from SGD to enhance prediction consistency in diffusion models. Below, we summarize general techniques in SGD that improve the training process.SGD is a fundamental tool in network training [4] , designed to navigate the weight (network parameter) space towards local minima. For a neural network F (x; θ) with parameters θ, SGD samples training data points x i and their corresponding labels y i , and computes the gradient of the loss function L(F (x i ; θ), y i ) with respect to θ to update the parameters. In practice, aggregation of gradients and network weights during training is often performed to reduce variance and improve convergence. Gradient aggregation typically involves averaging gradient values over a batch, while weight aggregation accounts for the history of the weights in each update.Notable examples include temporal averaging in Adam optimizer [23] , Stochastic Weight Averaging (SWA) [19] and teacher networks [47] used in consistency training by employing an exponential moving average (EMA) of a student network to maintain high-quality predictions. This technique is prevalent in semi-supervised and representation learning [13, 15, 6] . For a detailed study of EMA in network training, we refer readers to [31] .: M t,r = α i • M t,r + (1 -α i ) • M t,r-1 , for α i ∈ [0, 1].While all three improved performance over vanilla resampling, we find min-pooling to work best during the early stages of the denoising process.Cross-step map averaging. We perform cross-step information passing at timesteps t ∈ [600, 1000], with α h = 0.5 and α c = 0.2.Early-stage mutual self-attention. We apply Early-Stage mutual self-attention on timesteps t ∈[600, 1000] for Zero123-XL and on timesteps t ∈[700, 1000] for Zero-1-to-3. As mentioned in Sec. 3, the input latent vector is concatenated to the generated latent vector in Zero-1-to-3. Therefore, the target queries contain information about the input view, introducing a bias towards the input view during the generation process. Although mutual self-attention manages to enforce consistency with the input, it sometimes causes pose shifts and visual artifacts. Therefore, We only apply mutual self-attention in the decoder layers of the UNet with spatial resolution of 16 × 16 and 32 × 32 (the two largest resolutions, since the latent dimension of Zero-1-to-3 is 32 × 32).Hourglass scheduling. We divide the denoising process into 3 generation stages. We define the early stage during timesteps t ∈ [τ e , τ T ], where the general shape and geometry are determined. τ T is the length of the forward process [17, 42] . We define the middle stage for t ∈ [τ m , τ e ], where both the shape and appearance are refined gradually. We define the last stage for t ∈ [0, τ m ], where mostly fine-details are refined and added. As describes in the main paper, within each stage we sample steps uniformlly via DDIM sampling. However, the sampling rate is higher for the early and late stages.In our experiments, we determine the different stages of the denoising process as follows, setting λ den = 5. The early stage is defined for t ∈ [800, 1000] and we sample 10 timesteps uniformly within this interval. The middle stage is defined for t ∈ [200, 800] and we sample 6 timesteps uniformly within this interval. The last stage is defined for t ∈ [0, 200] and we sample 10 timesteps uniformly within this interval. In total, we sample 26 timesteps throughout the denoising process.Setting the hype-parameters. All hyperparameters were tuned based on a small validation set of objects. Due to limited computational resources, the tuning process was not exhaustive. We found that the method's performance is not highly sensitive to most parameters.1. Cross-step weight (alpha): We experimented with values ranging from 0.1 to 0.9 in increments of 0.1. Alpha determines the weight assigned to previous predictions in the cross-step aggregation. In Zero123-XL, early predictions were generally reliable, so a larger weight yielded better results. We maintained the same parameters for Zero-1-to-3. In the experiments conducted with additional models such as ControlNet and MVDream, a smaller weight generally produced better outcomes.We apply AMF during the early steps of the denoising process. We find that the in-and cross-step operation adds a small computational overhead. We maintain two additional instances of each attention map: the attention map from the previous timestep (for cross-step updates) and the refined map from the current timestep (for in-step updates).We provide running times (in seconds) of Zero-1-to-3 and Zero-to-Hero for the same NFE (66) in Tab. 4. Note that the number of samples reported in the table does not include the input view generated in Zero-to-Hero. Overall, we manage to provide competitive running times. If both MSA and AMF are active (requiring the generation of the input), the running time is increased by approximately 1-1.5 seconds. If only AMF is active, the overhead is much smaller, averaging around 0.5 seconds (there is no need to generate the extra input view).Table 4 : Runtime analysis of the computational overhead of Zero-to-Hero.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We intend to release all code and data used in our paper in time for the camera-ready version.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
method: 
 In this work we are concerned with the task of single image novel view synthesis. Formally, given an input image of an object and a relative camera transformation towards a desired target view, our goal is to generate the image at that target view. Specifically, we build upon the seminal work of Zero-1-to-3 [27] which tackled this task via a diffusion model. As detailed in Sec. 3.1 Zero-1-to-3 often struggles to generate plausible and input-consistent images. In this work we propose Zero-to-Heroa novel training-free technique that significantly improves its generation quality through attention map manipulation. This section is structured as follows. (4.1) through network architecture analysis we identify self-attention maps as key for robust view generation; (4.2) drawing inspiration from SGD convergence-enhancement techniques, we outline our novel attention map filtering pipeline; (4.3) details each step of the map filtering; (4.4) introduces the mutual self-attention which we use for shape guidance at early generation stages; (4.5) Finally, our proposed hourglass scheduler is introduced for more efficient utilization of generation steps. Fig. 2 depicts Zero-to-Hero's main modules. An ablation of these modules is provided in Tab. 2 and in Sec. 8.6 of the appendix.Zero-1-to-3 builds upon the image-to-image variant of Stable Diffusion [36] , which utilizes a UNet [37] architecture as its backbone and incorporates both self-and cross-attention layers. In this subsection, we analyze the roles of these components and their contributions to the generated view. This analysis aims to identify effective intervention points for enhancing generation quality. Global pose conditioning through cross-attention. Prior work [16] has demonstrated that the cross-attention layers in text-to-image diffusion models, which link text tokens to the latent image, are spatially aware and can be used for spatial manipulation. We first investigate these cross-attention layers, as they are the only components in the model through which the target pose is injected. In the text-to-image variant of Stable Diffusion, the generation is conditioned on a prompt P containing C tokens, each encoded with CLIP into an embedding, resulting in a condition c T 2I ∈ R C×d CLIP . However, in Zero-1-to-3, the condition is a pose-CLIP embedding c ∈ R 1×d CLIP , project to keys K t ∈ R 1×d Formally, given a sample z t ∈ R H×W ×dz and its corresponding queries Q t ∈ R H×W ×d , the pre-softmax attention map A between Q t and K t has dimensions H × W × 1. Given that the summation of the softmax is always 1, the post-softmax attention map sof tmax(A) in Zero-1-to-3 is a constant all-ones matrix. A visual demonstration is presented in Fig. 7 in the appendix.The post-softmax attention map is used to compute a weighted sum over the values matrix V ∈ R 1×d , obtained by a transformation of the condition c. Since the attention matrix is an all-ones matrix, we conclude that the cross-attention operation degenerates into a global bias term, lacking spatially aware operations. Computing similarity scores in the cross-attention layers is redundant as these scores are never used. While in principle it is possible to improve the global bias term by additional optimization objectives and extra training overhead, we focus on the self-attention layers to enhance the results and mitigate the consistency issues while avoiding retraining the model.Spatial information flow through self-attention. By monitoring the self-attention layers during the generation process, we observe that random noise introduced to the latent representation also introduces randomness to the attention maps. This randomness, while promoting generation diversity, can often lead to undesired strong correlations, that are misaligned with the true target. These strong correlations may persist through the denoising process, resulting in accumulated errors and visual artifacts.Given the above observation and the insight about the spatial-degeneracy in the cross-attention layers, we hypothesize that the self-attention layers preserve the information about the structure and geometry of the generated image, through the similarity scores between different elements in the latent vector. To validate our hypothesis, we conduct a straightforward experiment to assess Zero-1-to-3's performance using the 'ground truth' self-attention maps, which reflect the most accurate connections considering the true target image. Specifically, we selected two images, I src and I tgt , with known relative camera parameters [R|T ]. We first encode I tgt to obtain the clean latent z tgt 0 and then add subtle noise to obtain the corresponding noisy latent for timestep τ init , z tgt init . A single denoising step is performed on the noisy latent, and we save the self-attention maps from each layer in the UNet, Figure 3 : Through the injection of ground-truth attention maps extracted from the target view, we demonstrate that Self-attention maps are key to robust view synthesis. considering these maps as the ground truth (GT) maps. In our experiments, τ init = 5. Next, we sample random Gaussian noise and denoise it to regenerate the target image. During each denoising step, we replace the computed self-attention maps with the GT maps, without altering any other components (e.g., cross-attention layers, residual blocks) or the latents, queries, keys, and values. We report the results in the experiment section in Tab. 1, showing a significant improvement in all metrics. Note that the results obtained with the GT attention maps are a strict upper bound, as the GT maps contain information about areas that are invisible in the source view. Our experiment validates that through improved self-attention maps, the generated image becomes more plausible. Visual examples are shown in Fig. 3 and in Fig. 8 in the appendix, refer to Fig. 1 for the results of Zero123-XL and Zero-to-Hero for the same views.We draw a conceptual analogy between a denoising process of a diffusion model, and SGD based network optimization as both progress through gradient prediction of a loss function, and log probability (the score) [43] , respectively. Building on the discussion from the previous section, we treat the self-attention maps as parameters M in the denoising process z t-1 = µ(z t ; M t , ψ) and define their update process as a mapping1 :M t → z t-1 → M t-1 .Here the map M t-1 results from passing the latent z t-1 through the attention layers. This process is analogous to gradient descent optimization in neural networks, where each step adjusts the weights in the direction of the gradient of a loss function, such as the log probability in classification tasks. We denote this weight update as a mapping θ t → ŷt → θ t+1 , where ŷt = F (x t ; θ), and the updated parameters θ t+1 result from a gradient step.As detailed in Sec. 3.2, gradient and weight aggregation are essential for robust convergence. We outline this process in three steps illustrating the analogy between network parameter updates and attention map filtering. Fig. 4 further illustrates the analogy.Step-by-Step Analogy From network parameters to attention maps1: Sampling: Generate R pairs of samples and their corresponding labels {(x i , y i )} R i=1 . 2: In-step weight update: Average the gradients to adjust the network parameters: θt = θ t -λ ∇ θ L(F (x i ; θ t ), y i ). 3: Cross-step weight averaging: Update network parameters using EMA:θ t+1 = αθ t + (1 -α) θt , for α ∈ [0, 1].Denoising Process 1: (Re-)Sampling: Repeatedly re-noise z t-1 to z t and denoise, R times. 2: In-step map update: Resmapling results in a sequence of maps {M t,r } R r=1 , aggregated into the final updated prediction M t,R . 3: Cross-step map averaging: Update attention maps via EMA:M t-1 = αM t + (1 - α) M t-1 , for α ∈ [0, 1].We now discuss in detail each of the map filtering steps. A scheme of the different filtering modules is provided in Fig. 2 (Left, and Top-right).Latent refinement via per-step resampling. Inspired by previous studies [29, 2] , we implement per-step resampling throughout the image generation process. We select a range of timesteps Both methods are initialized with the same seed. We observe that the strong correlation values in the upper right corner lead to exaggerated content creation (note the unrealistically elongated neck). Conversely, through filtering, Zero-to-Hero mitigates these artifacts, leading to robust view synthesis.[t min , t max ], where each denoised sample z t is re-noised with the proper noise ratio to the previous sampled timestep z t+1 and denoised back to z t for R iterations2 . We concur with previous studies that resampling functions as an effective corrective mechanism to the generated image, as can be seen in Tab. 2). Experimentally, we find that it is particularly useful during the initial stages of the denoising process. Through resampling we progressively generate R attention maps with different noise patterns. We propose to leverage these intermediate maps to further boost performance through in-and cross-step attention map manipulations.In-step map update. We propose a novel attention pooling function f , to update the attention maps within the denoising step. Specifically, a self-attention map is refined based on previous maps {M t,k } r-1 k=1 created at the same timestep. Since resampling is a sequential process, we perform a progressive aggregation scheme:M t,r = f (M t,r , { M t,k } r-1 k=1). We found the element-wise minpooling operator f (a, b) = min(a, b) to perform best in our experiments. We discuss other options for f in the appendix.Cross-step map averaging. Resampling tends to favour "conservative" generations, often gradually neglecting fine image details like buttons and eyes as denoising progresses. This phenomenon it not resolved by the in-step update. To mitigate this issue, we propose to pass the self-attention map at time t, M t to the next step in the denoising process. We implement this cross-step aggregation via EMA:M t-1 = αM t + (1 -α) M t-1 , for α ∈ [0, 1]. This method effectively balances past priors with current data to enhance the denoising results. In practice we apply both methods in tandem, as detailed in Sec. 8.5. An example of our attention filtering is presented in Fig. 5 . Note that our filtering mechanism is applied to the pre-softmax attention maps.Of the two challenges describe in the introduction, attention map filtering handles well generating realistic outputs. Yet, we observe it is sometimes insufficient to enforce consistency with the input appearance and structure. We propose to utilize mutual self-attention as a complementary technique, to propagate information from the input to the target. Similar to prior works [49, 5, 1, 57, 52], we generate the input and target views in parallel. At each timestep t, we obtain queries, keys and values of the self-attention layers for the input and the target branches. Similar to the mutual self-attention introduced in [5] , we modify the self-attention operation of the target by replacing the target keys and values with those of the input branch: Attn(Q tgt , K in , V in ) (Fig. 2 (bottom-right)).Previous studies initiate mutual self-attention (MSA) at a later denoising step (small t). We find that at that stage, the structure has already been determined. Instead, to guide the structure of the target to be more consistent with the input, we find it more effective when applied from the beginning of the denoising process. While MSA is effective at transferring the appearance and structure of the input to the target, it may overfit. We find that terminating the MSA process once the structure has been stabilized becomes crucial in mitigating overfitting. In practice we find that applying MSA during the first third of the denoising process is a good rule of thumb for optimal results. We refer the reader to Sec. 8.5 in the appendix for further details.We view early-stage MSA as a complementary filtering scheme. Building on the property that generating the input view (which the model is conditioned on) is a much easier task for the model compared to generating novel views, keys and values predicted in the input branch are "cleaner". Mapping them to the target view thus refines the predicted scores, facilitating a more stable and reliable output.As resampling is a time-consuming operation, significantly increasing the number of function evaluation (NFE), we seek an efficient scheduling scheme that will enable robust and high quality generation, while preserving the fast generation times of Zero-1-to-3. We use the common DDIM sampling [42] , and propose an efficient scheduling scheme to select the sampled timesteps. In the experiments section, we demonstrate that increasing the number of denoising steps does not necessarily improve the performance, and therefore we aim to use an overall small number of sampled timesteps. Specifically, we find that denser sampling during the beginning of the denoising process is crucial to promote realism. We also find that denser sampling at the final steps enhances fine details. Therefore, we suggest a double heavy-tailed scheduling scheme we call Hourglass, according to which we divide the generation process into 3 stages, as detailed in 8.5. Within each stage, we sample steps uniformly via DDIM. However, in the first and last stages we sample steps more densely (at a higher rate) than in the middle stage, by a factor of λ den . 
experiments: 
 Datasets. We evaluate our method on two datasets, following previous works. Firstly, Google Scanned Objects (GSO) Dataset [12] , which includes 1030 scanned household objects. However, the dataset's imbalance (e.g., 254 objects are categorized as "shoe") and the high proportion of symmetrical shapes limit its reliability for evaluation. To address this, inspired by [24] , we select a challenging subset of 50 objects from GSO, avoiding symmetrical and repetitive shapes. For each object, we render 8 random views (details in Sec. 8.9 of the appendix) and synthesize the remaining views from each source view, generating each target view with 3 different seeds to calculate the average score per measure. Secondly, RTMV Dataset [48] , which consists of 3D scenes. Each scene contains 20 random objects. For evaluation, we randomly select 50 scenes, and 8 random views per scene. The evaluation process is the same as described for GSO.Metrics. We report the image quality metrics PSNR, SSIM [50] and LPIPS [59] . As these metrics are sensitive to slight color variations, we segment the generated targets and their corresponding real images, via thresholding, and report the Intersection Over Union (IoU) score.Quantitative evaluation. We evaluate Zero-to-Hero against zero-1-to-3 and on zero123-XL. We report all metrics for the original models using 25, 50 and 100 DDIM steps, and for our method applied to both models. In Tab. 1 and in Tab. 3, we provide the results for GSO and RTMV datasets, respectively. We include the number of sampled timesteps T and the total number of network evaluation NFE (accounting for resampling). Zero-to-Hero consistently improves performance across all metrics, taking a significant step towards bridging the performance gap to GT attention maps. All implementation details, including analysis of the inference cost of our modules, are provided in Sec. 8.5 of the appendix. Qualitative evaluation. In Fig. 1 , Fig. 9 and Fig. 10 , we visually demonstrate how Zero-to-Hero is able to mitigate various artifacts generated by Zero-1-to-3, from implausible results to incorrect poses. In Fig. 1 and in Fig. 9 , we present 3 examples per target view, generated with 3 random seeds, to emphasize the consistency and robustness our method offers.To assess the contribution of Zero-to-Hero different components to the final performance, we evaluate our pipeline on our challenging GSO subset, starting from the baseline Zero123-XL and gradually adding each module. The results are summarized in Tab. 2. When reporting the results of the vanilla Zero123-XL, we use its best score achieved by running the model with 25, 50, and 100 steps. A similar comparison against zero-1-to-3 is included in Sec. 8.6 of the appendix, demonstrating consistent behavior. We also analyze the affect of resampling and our attention filtering on the generation diversity in Sec. 8.7 of the appendix. Additionally, we provide qualitative results, demonstrating the common contributions of AMF and MSA in Fig. 10 in the appendix. A brief study of ControlNet models [58] demonstrated that they suffer from similar limitations as Zero-1-to-3 and its follow ups. Namely, lack of condition enforcement and frequent appearance of visual artifacts. We implemented our proposed AMF module for two pre-trained ControlNet models. We provide qualitative results for pose-and segmentation-conditioned ControlNet models in Fig. 6 and Fig. 13 , respectively.Multi-view synthesis. We integrate AMF into MVDream [40] , a text-to-multiview model, and find that it helps to mitigate the same issues as in the single view case. In Fig. 14 , we provide qualitative results.Figure 6 : Qualitative results for pose-conditioned ControlNet. Qualitative results for pre-trained pose-conditioned ControlNet, without and with AMF. Both methods are initialized with the same seed. AMF leads to results that are more plausible and better align with the conditions. 
conclusions and future work: 
 In this paper, we introduced Zero-to-Hero, a training-free method to boost the robustness of novel view synthesis. We enhanced the performance of a pre-trained Zero-1-to-3 diffusion model using two key innovations: a test-time attention map filtering mechanism that enhances output realism, and an effective use of source view information to improve input consistency. Our approach also features a novel timestep scheduler for maintaining computational efficiency. In future work, we aim to refine our method by developing trainable filtering mechanisms, enhancing pose authenticity via cross-attention manipulation, and extending our approach to other diffusion-based generative tasks.Limitations. Our method, operating at test-time, is limited by the generative capabilities of the pre-trained model. If Zero-1-to-3 cannot correctly generate the target pose, our method may not enhance the output, as demonstrated in the inset figure.Broader impact. Through enhancing view synthesis, our method offers significant benefits to applications in virtual reality, augmented reality, and robotics. As it requires no further training, it is readily accessible. However, this accessibility also simplifies the creation of realistic novel views, which could be exploited for malicious purposes such as deepfakes. information denoted as H t . Subsequently, we integrate this historical information through a simple blending technique at each denoising step, along with our in-step map update:M t,r = α c • f (M t,r , { M t,k } r-1 k=1 ) + (1 -α c ) • H t+1 , where α c ∈ [0, 1],f is an attention pooling function and t + 1 represents the previous sampled step.The historical information is updated at the last resampling iteration (the R-th iteration) of each timestep as follows:H t = α h • M t,R + (1 -α h ) • H t+1, where α h is a decay factor within the range [0, 1]. This historical data is initialized from the first refined self-attention map at the final resampling step, expressed as H T = M T,R .Resampling is performed during timesteps t ∈ [800, 1000], with R=5.In-step map update. In-step map update is applied at timesteps t ∈ [800, 1000], with element-wise min-pooling as a denoising function in all our experiments. We find that min-pooling is useful during the earlier steps of the denoising, where the noise is substantial. We explored several simple denoising functions:1. Averaging: M t,r = 1 r (M t,r + r-1 k=1 M t,k ), which can be extended to weighted averaging.2. Min pooling: M t,r = min(M t,r , { M t,k } r-1 k=1 ).Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies all relevant details to understand all the experiments and results shown.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Each generation task (defined by a source view and camera viewpoint) is generated 3 times with different seeds, and the final score is the average of all score.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
ablation study: 
 We conduct ablation studies focused on each component on our method, and present the results for the default checkpoint of Zero-1-to-3 and for Zero123-XL.In addition to the main ablation table shown in the main paper, we present the same study on Zero-1to-3 in Tab. 5. Additionally, we provide qualitative results, demonstrating the common contributions of AMF and MSA in Fig. 10 .Hourglass scheduling. We run the original model with 25, 50 and 100 steps sampled uniformly using DDIM, and show that the hourglass scheduling, with 26 steps sampled in total, performs better or on par. The main benefit of our Hourglass scheduling is reducing the number of steps, but it also slightly improves the overall performance. Results are presented for Zero-1-to-3 in Tab. 8 and for Zero123-XL in Tab. 6.Attention map filtering. We use the hourglass scheduling for all further experiments. We run the original model with and without resampling and attention filtering. Our experiments demonstrate that adding the attention filtering in addition to the resampling mechanism improves the performance across all metrics. Results are presented for Zero-1-to-3 in Tab. 9 and for Zero123-XL in Tab. 7.Early-Stage mutual self-attention. To the best of our knowledge, most prior works utilizing MSA fall into two categories: 1. Training-free based methods [5] : In these works, MSA is typically applied after the general structure of the target is formed to transfer appearance details from the input to the target. In this scenario, MSA does not contribute to the initial structure formation of the target. 2. Training or fine-tuning based methods [52, 53] : These works incorporate MSA layers within the training or fine-tuning process.Our approach is distinct as we employ MSA in a training-free manner, but crucially, we apply it from the beginning of the denoising process until the target structure stabilizes-a phase we term "Early-Stage Shape Guidance". To validate the impact of early-stage MSA on structure, we conducted a simple experiment using Zero123-XL with 50 DDIM steps. We measure the effect of activating MSA at different stages of denoising on Intersection over Union (IoU) metric, noting that image quality metrics improved similarly. The results are reported in Tab. 10. Our approach demonstrate a significant improvement in the structural integrity of the image. Applying MSA in the later stages of the denoising process may hinder the results as it introduces a bias towards the input image. While our attention filtering manages to generate more robust and consistent results, we find that excessive it might limit the generation diversity. We explore the two main factors that can reduce the diversity: the number of filtering iterations R and the interval of timesteps [t min , t max ] where we apply filtering.Number of filtering iterations. In Fig. 11 , we show the effect of different values of R on the diversity of the results. All other hyper-parameters remain unchanged. We find that as R grows, the generation diversity reduces, while the level of realism improves Filtering schedule. We apply filtering during timesteps t ∈ [t min , T ]. In Fig. 12 , we show the effect of different values of t min on the diversity of the results. All the hyper-parameters besides t min remain unchanged. We observe that as we apply filtering for longer periods of the denoising steps, the results are more realistic but less diverse. Finally, the last row shows a case where neither technique worked well enough on its own, but the combination did the work: the original result of Zero123-XL shows an implausible right hand and an incorrect pose of the right sword. MSA lead to more stable result, where the hand is not out-of-distribution, but could not resolve the incorrect pose of the sword. AMF lead to reasonable right hand and placement of the sword, but the sword is not aligned well the the texture of the original sword. The final result show correct structure and texture for the right hand and sword.Through our ablation study, we find that in practice, a moderate application of attention maps filtering is sufficient for improving fidelity while preserving diversity. While excessive resampling might limit diversity (e.g. using large values of R), we only apply our filtering mechanism in the earlier steps of the denoising process, using a small value of R (5 in all our experiments). This enables our method to maintain diversity effectively.When evaluating diversity against base models, we observed that some results of the base models were highly implausible, as can be seen in Fig 1 and Fig. 9 in the appendix. For example, Zero-1-to-3 produces out-of-distribution results, or results that are not aligned with either the input image or the target pose, leading to seemingly larger variance. However, this "diversity" is largely due to misalignment and artifacts, which comes at the expense of fidelity.In our case, our model continues to generate diverse results that are both plausible and better aligned with the conditions (e.g. the various chair backs and turtles heights in Fig. 1 ).Figure 11 : Generation diversity wrt filtering iterations. We select an extreme change in viewpoint, and show how different choices of R, the number of filtering iterations, affect the diversity of generated outputs. As filtering iterations increase, the results become less diverse and more realistic.Figure 12 : Generation diversity wrt filtering schedule. We select an extreme change in viewpoint, and show how different choices of the timesteps interval where filtering is applied, affect the diversity of results. As filtering is applied for longer periods during the denoising process, the results become less diverse and more realistic.In this subsection, we describe several preliminary experiments where we integrate attention map filtering to other base models. Remarkably, the integration of our proposed method into other base models is straightforward, demonstrating its applicability and simplicity.In all the mentioned experiments, resampling is performed during timesteps t ∈ [700, 1000], with R=5. Additionally, the cross-step parameters were slightly tuned for each base model. Mutual Self-Attention is not implemented, as it is not immediately applicable.Conditional image generation. A brief study of ControlNet models [58] demonstrated that they suffer from similar limitations as zero123 and its follow ups. Namely, lack of condition enforcement and frequent appearance of visual artifacts. We implemented our proposed AMF module for two pre-trained ControlNet models. In Fig. 6 in the main paper, we provide qualitative results for poseconditioned ControlNet. In Fig. 13 , we provide qualitative results for segmentation-conditioned ControlNet. We find that attention map filtering robustly mitigates artifacts across various prompts and seeds for both base models.Multi-view synthesis. We integrate AMF into MVDream [40] , a text-to-multiview model, and find that it helps to mitigate the same issues as in the single view case. We provide qualitative results in Fig. 14 .Figure 13 : Qualitative results for segmentation-conditioned ControlNet. Qualitative results for pre-trained segmentation-conditioned ControlNet, without and with AMF. Both methods are initialized with the same seed. AMF leads to results that are more plausible and better align with the conditions, including intricate details (e.g. the base model fails to generate the faucet and the right stool in the third row, while our method generates both and better align with the prompt.)Figure 14 : Qualitative results for MVDream. Qualitative results for MVDream, without and with AMF. Both methods are initialized with the same seed, and we provide results with two random seed per prompt. AMF leads to results that are more plausible and spatially consistent, while also better align with the conditions.NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: Our method is training-free, builds upon an already released pre-trained model. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original paper that introduces Google Scanned Objects and all license details will be provided with the code and data. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not create 3D new assets. However, we generate a dataset of renderings that is well documented and will be released.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper doesn't involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper doesn't involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Zero-1-to-3 Zero-to-Hero (Ours) 1 2.2 3.1 2 2.9 3.9 3 3.5 4.9 4 4.3 -Although the latent prediction zt-1 depends on the parameters ψ of denoising UNet µ, they remain unchanged. Here we emphasize that the attention maps M are the parameters being updated.Depending on the sampling algorithm, h ≥ 1 steps may be taken. 


Decision: The reviewers acknowledge the technical contribution of proposing a training-free method and its effectiveness, along with a clear presentation. The rebuttal was successful, resulting in an increased average score, with previous negative scores now becoming supportive. After careful discussion and consideration, we are pleased to inform you that your paper has been accepted. However, the final version will require revisions to reflect the important discussions presented in the rebuttal. For example, please discuss the diversity issue, clarify the distinction from the previous MSA module, and include additional experimental results.

Reviews: review: paper_summary: paper_summary: This paper experimentally analyzes which parts are important and responsible for generation artifacts.
To solve it, this paper propose an attention map filtering process.
This process share the similar idea with SGD, which reduces the error of the generation process by repeated sampling.
This paper also propose some other things to enhance the results, including identity view information injection and a specialized sampling schedule.
The whole pipeline is training-free, so it’s very easy to apply to the pre-trained diffusion model.
strength_weakness: strength_weakness: * The model has no training cost, but there is a lack of thorough analysis of the additional inference cost.
* Qualitative results are limited. Given that the quantitative results do not show much improvement, more qualitative comparisons would be better.
questions: questions: * Ablation studies of zero123 and zero123-XL showed different trends. MSA even impaired the results, but MSA with AMD improved the results. What is the reason do you think?
* Form the quantitative results, the improvement brought by MSA is significantly greater than that of AMF. Are there any qualitative results on MSA and AMF?
* How does this method affect the speed of the original diffusion model? Does the choice of R affect the model results?
* Fig 3 and 7 seem to share some of the same samples, maybe we can show more cases and perspectives?
limitations: limitations: This method is limited by the generative capabilities of the pre-trained model, if the results wrong too much, it does not have the ability to correct it.
paper_summary: paper_summary: This paper proposes attention map filtering to enhance the novel view synthesis performance of Zero-1-to-3. The method is composed of several changes to the original sampling method, but the main contribution is an aggregation of attention map strategy by resampling the same denoising step multiple times. The paper draws an analogy to SGD to argue that resampling resembles the batch training in SGD, enhances the stability of the sampling process, thus improving the performance. Controlled experiments show that the proposed technique is effective in improving the quality of the generation
strength_weakness: strength_weakness: - Following the analogy introduced by the authors between diffusion and SGD, it seems unavoidable that aggregating attention maps across multiple sampling steps will compromise the model on generation diversity. As we can imagine, optimization with a larger batch size is more stable. So, even though the authors have proposed techniques to mitigate the problem of lack of diversity, I believe it is a more fundamental limitation.

- Since the techniques proposed seem general to all diffusion models, what prevents one from applying the technique to all diffusion generation tasks? Given the vast amount of literature and problems being solved by diffusion models, the potential impact could be significant. Are there any specific constraints such that the method is only applicable to the problem of novel view synthesis?

- Does the method lead to lack of visual details in the generated views? I noticed that the eyes of the generated chickens are missing in the last row of figure 1. Is this a general problem of the proposed techniques?

- It's a little unfair to claim that the proposed method can maintain the same computational efficiency. The proposed techniques to accelerate the diffusion process is not exclusive to the proposed techniques and can be applied to the base model as well.
questions: questions: Please see the weakness section for questions.
limitations: limitations: The paper has discussed the limitations.
paper_summary: paper_summary: This paper proposes a novel approach to generate realistic images from arbitrary views based on a single source image. The authors introduce a test-time method that enhances view synthesis by manipulating attention maps during the denoising process of diffusion models. This process improves geometric consistency and realism without requiring retraining or significant computational resources. The key contributions include an attention map filtering mechanism, a modification of the self-attention mechanism to integrate source view information, and a specialized sampling schedule. Experimental results show substantial improvements in fidelity and consistency, demonstrating the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Lack of qualitive results in the experimental part and even in Appendix I cannot find more comparison.
2. Only compared and equipped with zero123 and zero123-XL is not enough also.
questions: questions: 1. Can Zero-to-Hero be equipped with other multi-view diffusion models, e.g,,MVDream, Wonder3D?
2. The quantitive metrics is calculated in which views?
3. Will Zero-to-Hero hurt the diversity of Zero123 model with relatively consistent outputs among different seeds?
limitations: limitations: Please refers to W and Q. I will consider to raise my score if the author could address my concerns and provide more convinced results.
paper_summary: paper_summary: This paper proposes a test-time approach that adaptively modifies the attention map during inference to enhance the consistency and plausibility of novel view synthesis. Experimental results on GSO demonstrate improved performance with the proposed method, and the authors conducted ablation studies to assess the impact of different modules on performance.
strength_weakness: strength_weakness: 1. Since (Re-)Sampling increases the computational load, with the hyperparameter R determining the number of resampling iterations, it would be beneficial for the authors to conduct an ablation study on how this parameter affects both results and computational overhead.
2. The method proposed by the authors has shown limited improvement in results. They need to validate the effectiveness of their proposed approach on a larger and more diverse dataset.
3. When conducting the ablation study, the authors' selection of only 10 objects from GSO could introduce bias into the results. It would be preferable for them to include all objects to ensure a more comprehensive evaluation.
4. The authors' proposal of Mutual Self-Attention (MSA), which has been used in other papers such as Consistent123, Tune-A-Video, and MasaCtrl, cannot be considered a unique contribution in their paper.
5. In the ablation study (Table 2), Mutual Self-Attention (MSA) shows the largest improvement, with a greater increase in PSNR (MSA 17.82 vs AMF 17.58, last two rows). This suggests that the method's improvement primarily stems from MSA rather than the core contribution point, AMF.
questions: questions: The paper includes several hyperparameters such as the number of resampling iterations (R) and the alpha for cross-step aggregation. How did the authors set these hyperparameters, and how significantly do they impact the final results?
limitations: limitations: The method proposed by the authors seems intriguing, but the actual improvement in results is limited. Moreover, the largest gain comes from a commonly used trick for enhancing viewpoint consistency, Mutual Self-Attention (MSA).

Comments: comment: We thank the reviewer for their thoughtful comments and for taking the time to review our rebuttal and engage with us. Your feedback is invaluable for us.

We are pleased we could address your concerns, and are happy to address any further concerns.
Thank you for the response. The rebuttal addressed most of my concerns, and I am willing to keep my score.
We thank the reviewer for their thoughtful feedback and for taking the time to review our rebuttal and engage with us. Your comments are invaluable for us. We are pleased we could address your concerns and appreciate that you revised your evaluation accordingly.

We are happy to address any further concerns.
We thank the reviewer for their thoughtful comments and for taking the time to review our rebuttal, your feedback is invaluable to us. We are glad we could address your major concerns and appreciate your revised evaluation accordingly.

Regarding your concern about generation diversity, we would like to offer some additional clarifications.

While excessive resampling might reduce diversity (e.g. using large values of R), in practice we find that a moderate application of AMF is sufficient for improving fidelity while preserving diversity. Note that we only apply our filtering mechanism in the earlier steps of the denoising process, using a small value of R (5). **This enables our method to maintain diversity effectively.**

When evaluating diversity against base models, we observed that some results of the base models were highly implausible, as can be seen in Figure 1 in the paper. For example, Zero-1-to-3 produces out-of-distribution results, or results that are not aligned with either the input image or the target pose, leading to seemingly larger variance. However, this "diversity" is largely due to misalignment and artifacts, which comes at the expense of fidelity.

In our case, our model continues to generate diverse results that are both plausible and better aligned with the conditions (e.g. the various chair backs and turtles in Figure 1 of the main paper).

**Future work.** Regarding your point about preserving diversity in multiview-diffusion, we agree that it is very important. In future work we plan to explore the effect of AMF in multiview-diffusion in depth and per your suggestion include a thorough analysis of diversity vs. fidelity. In particular, MVDream is conditioned on text rather than an image, making the desired solution space inherently more diverse which should be accounted for in such an investigation. That being said, in the preliminary results of MVDream (and also in the ControlNet models) that we ran for the rebuttal, we observed that the diversity is well-preserved.  For example, for the prompt “A man eating a pizza, highly detailed”, our method produced results that vary significantly in terms of viewpoint and appearance. This trend persisted across various prompts and seeds which we could not include in the rebuttal due to space limits. 

We thank the reviewer again for engaging with us, and are happy to address any further concern.
Thank you to the authors for the detailed response. The rebuttal addressed most of my concerns, and I am willing to raise the score.
Thank you for your comprehensive rebuttal. The majority of my concerns have been addressed, particularly regarding the details of the Quantitative Evaluation, the applicability of Zero-to-Hero in Multiview Diffusion, and the additional qualitative results. However, I remain not entirely convinced about the purported balance between "diversity" and "fidelity". The attention map filtering is applied to a multi-view diffusion model, whose diversity should be largely preserved to maintain its non-deterministic nature as a diffusion model. Considering all these factors, I have decided to revise my score from borderline reject to borderline accept. I intend to consult other reviewers' discussions before making a final decision and I am more than happy  to discuss with other reviewers and  AC, SAC and PC about the diversity concern.
Dear Reviewer,

Please take a moment to read the rebuttal if you haven't already, and assess the new information provided by the authors. Then please provide your updated comments.

Thanks,
AC