3D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidance

Authors: Taewon Kang, Divya Kothandaraman, Dinesh Manocha, Ming Lin
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D-free, View Consistency, Background Inclusion, Guidance Models

Abstract:
Recent 3D novel view synthesis (NVS) methods often require extensive 3D data for training, and also typically lack generalization beyond the training distribution. Moreover, they tend to be object centric and struggle with complex and intricate scenes. Conversely, 3D-free methods can generate text-controlled views of complex, in-the-wild scenes using a pretrained stable diffusion model without the need for a large amount of 3D-based training data, but lack camera control. In this paper, we introduce a method capable of generating camera-controlled viewpoints from a single input image, by combining the benefits of 3D-free and 3D-based approaches. Our method excels in handling complex and diverse scenes without extensive training or additional 3D and multiview data. It leverages widely available pretrained NVS models for weak guidance, integrating this knowledge into a 3D-free view synthesis style approach, along with enriching the CLIP vision-language space with 3D camera angle information, to achieve the desired results. Experimental results demonstrate that our method outperforms existing models in both qualitative and quantitative evaluations, achieving high-fidelity, consistent novel view synthesis at desired camera angles across a wide variety of scenes while maintaining accurate, natural detail representation and image clarity across various viewpoints. We also support our method with a comprehensive analysis of 2D image generation models and the 3D space, providing a solid foundation and rationale for our solution.

Decision: reject
Reviewer scores: 5, 8, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729344456933

thread_type:
official_review

<thread_object>
creator: reviewer_yrxv

<content>
summary:
This paper argues that current NVS methods are limited to single-object-centric scenes and struggle with complex environments. Moreover, they require extensive 3D data for training. Therefore, this paper proposes a training-free method to combine the benefits of 3D-free and 3D-based methods. It leverages pretrained NVS models for weak guidance and integrates the knowledge into a 3D-free approach to achieve the desired results.

strengths:
1. The motivation for training-free novel view synthesis is good.
2. The analysis of the current NVS model’s limitation is reasonable.

weaknesses:
1. The writing is poor and many sentences are extremely confusing without any support. I will list several examples below and there are more similar problems in the submitted paper.
    
    In Line 265. “Explain the subset you used for your results, how you created the,….”. What does the author want to express here?
    
    In section 3.1, why does Fig.3 illustrate these findings? I believe the findings here are that CLIP struggles with 3D comprehension, but I do not see any examples in Fig.3 supporting this viewpoint. Moreover, if CLIP itself can control camera pose change, there is no need for training on 3D-aware data pairs. I do not see the necessity of including this discussion.
    
    In Line 146. “validates the necessity of 3D guidance images.” What is 3D guidance images? Is it a novel view image based on the current image?
    
    In Line 159. “This indicates that the model benefits from the information in the 3D-derived guidance image, and that the guidance image is not simply a source of extra variance.” What does the author want to express here? What is the 3D-derived guidance image? Does it also mean a novel view image? If so, what does the author mean by “not simply a source of extra variance”? What is the variance? From Fig.4, I guess the author is conducting an experiment on using different novel view images as weak guidance while prompting the generation of novel view images using the proposed training-free method. The reviewer thinks using the right guidance image is quite trivial. 
    
2. The performance improvement is weak. From the visualized comparison in Fig.5, I don’t see much difference between Zero123++ and Ours. Is the camera view control more precise or is the image more realistic? From the waterfall example, I think the result of Zero123++ is more realistic. 
3. The novelty is weak and the method is not well explained. I think a training-free method is a good motivation. But the author still requires heavily on a pretrained NVS model. 
4. Related work is not well discussed. To cope with the complexity of real scene instead of object-level, many works have been proposed and the author should include them. I will list several below.

[1] ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image, CVPR 2024

[2] MegaScenes: Scene-Level View Synthesis at Scale, ECCV 2024

[3] Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis, ECCV 2024

questions:
Minor:
In Tab.1 and Tab.2, since the author reports the LPIPS metric, I believe the commonly used PSNR and SSIM metrics can also be reported.

The reviewer suggests that the author revise the writing, conduct more thorough experiments, and propose a clearer explanation of motivation and method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729756150386

thread_type:
official_review

<thread_object>
creator: reviewer_haub

<content>
summary:
The paper tackles the problem of novel view synthesis from a single image input of a scene by using textual camera view description. The problem statement is very similar to previous works Zero123++, HawkI, except that an additional guidance image is introduced, obtained through a pre-trained model Zero123++, tackling the same problem. The guidance image network Zero123++ is trained on 3D shapes, but for a single object. The introduced method uses a series of 2 double-step optimizations. The first stage optimizes the diffusion-enriched CLIP space of the input view and the second stage does the same for the target view. The results show that the proposed method has better performance in comparison to previous works. It tends to be more consistent in showing both central object and background from a variety of viewpoints (which are specified in terms of azimuth and elevation angles).

strengths:
1. The method shows a good performance of novel view synthesis on a single view of a scene with object + background. As far as related works are concerned, the method shows - on average - better results using a textual prompt and without depth information for object + background when novel views are synthesized.

2. The results show a consistent performance gain over previous works in both real and synthetically generated data.

weaknesses:
1. The method is quite convoluted with 4 different test-time optimization steps. No time comparisons are provided. It would be important to mention typical in inference times... 

2. Looking into the intermediate and final results, as well as the method, the main task of aligning the camera appears to be the result of the Zero123++ method, while the test-time optimizations generally improves the image quality. This is also reflected in the particular choice of angles, as only those angles are trained for in Zero123++. However, for small angles (30,30), often the test-time optimization worsens the results, e.g., figure 6, angles = (30,30) for cats. Same for figure 5, angles = (30,30)- the house changes completely after optimization. Due to this reason, it may be a bit dubious to stipulate object + background novel view synthesis as contribution. After the evaluations, both the contributions (1) & (2) could be attributed - at least in part - to Zero123++.
 
3. The architecture diagram in Figure 2 is not complete in the info provided. Missing: which losses are used? These should be made clear together with the equations (1,2) since this is the crux of the proposed method.

4. L215--L216- "Our model eliminates the need for training data, 3D data or multi-view data". This is somewhat misleading. The camera alignment largely appears to be performed by the guidance method Zero123++, which is trained on Objaverse and more importantly on  the same task. The method implicitly uses Objaverse via Zero123++, a competing method used in the proposed method. It is not clear how Zero123++ performs test-time optimization without the 3D data.

5. The paper glosses over some of the limitations in the results, like the house not being shown in a very consistent fashion when viewpoint is changed. Probably there is a difference in performance between renowned monuments with many photographs that can be scraped from the Internet, compared to rather unique objects / situations. 

6. Given the complete approach, the novelty of the work mainly consists of the combination of methods, whereas the actual novelty drags in the cost of extensive inference optimization. 

7. The English could benefit from some slight polishing at times

questions:
It would be good if the authors clearly define what they mean by azimuth and elevation angles. The typical definition of the latter would be relative to the horizontal plane. Is that the case here as well, or are angles relative to the guidance view ? In any case, the angles specified in the image captions do not seem to follow one consistent set of definitions... Elevation angles are taking up values larger than 90 degrees, for instance. 

Related work that could be mentioned: Depth self-supervision for single image novel view synthesis, 2023 Giovanni Minelli et al.

The creation of videos with continuously changing viewpoints would be telling when it comes to consistency and angle correctness...
</content>
</thread_object>

</thread>
<thread>
creation date:
1730083056756

thread_type:
official_review

<thread_object>
creator: reviewer_9fq5

<content>
summary:
The paper presents an optimization-based sampling method to generate novel views at inference time. It utilizes multi-view images generated by a pre-trained diffusion model and optimizes the text embeddings and LoRA layers of a text-to-image diffusion model to guide the novel view sampling process.

strengths:
1. The paper investigates an approach that sequentially optimizes the text embeddings and LoRA layers of a diffusion model to guide the novel view synthesis process.

weaknesses:
1. The proposed method relies on a pre-trained multi-view diffusion model, Zero123++, and thus shares its limitations: restricted view generation (Zero123++ generates six fixed-view images around the object), modification of background and object, handling multiple objects, and pose misalignment.
2. The evaluation can be seem unfair. For apple to apple comparison, the paper could compare with running the proposed method on Zero123++. Also, measure the CLIP score seem unfair as the proposed method optimizes the CLIP text embeddings. 
3. The proposed method introduces additional computational costs: generating a multi-view guidance images from a pre-trained diffusion model and the optimization of text embeddings to reconstruct $I_{input}$ and $I_{view}$. 

* L.86, L.90: Some terms are used in contradicting manner. 3D-free methods like Zero123++  and 3D-based models like Zero123++.
* L.95: we (1) we 
* Eqn. 1: The optimization task is provided without details. $L, f, x_t$ is not introduced before.

questions:
The method could be presented with more detail. Specifically, how do Eq. 1 and Eq. 2 work? Does the diffusion model output clean predicted samples? If not, how is the loss computed between the noisy latent $x_t$ and the clean image $I_{input}$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570940664

thread_type:
official_review

<thread_object>
creator: reviewer_uyxj

<content>
summary:
This paper proposes a framework for scene-level camera-control novel view synthesis(NVS). The framework combines 2D image generative prior with object-level 3D generative prior to achieve scene-level NVS. The authors start with an analysis on where should generative model learn the viewpoint information, then proposed a multi-stage inference-time optimization method to generate novel views given specific camera conditions and input images. Qualitative and quantitative evaluation on benchmarks show its superior performance comparing to other methods.

strengths:
1. The idea of combining priors from 2D image generative model and object-level 3D generative model is interesting and can be considered as an effective way to leverage different data source for 3D understanding. I think this idea in general is very important.

2. NVS results on scene images are quite impressive considering no additional training is involved.

weaknesses:
I think this paper's main weakness is on how the authors present the whole framework in a more motivated way. I'd briefly state this weakness here and leave the others in the following questions section.

1. The connection between the section3 and section4 is very confusing. First, the detailed experiment setting in section3 is not clearly explained (see questions), then it suddenly shift from section3 with CLIP embeddings to section4 on leveraging a diffusion model. I suggest authors revise the high-level motivations and present them more clearly in the paper.

2. The figure 2 for pipeline is actually making it harder to understand. I'd suggest at least add some orderings in it can better illustrate the whole pipeline.

questions:
Below are some questions or some parts in the paper that I find confusing.

1. What is the detailed procedure of experiments related to figure 3. and figure 4.? What is the actual model that generates those images? CLIP itself isn't a generative model. Does the author mean using Stable Diffusion with different text embeddings? And how is the image guidance been used in (I assume) this standard SD model?

2. Although inference-time optimization with efficient methods like LoRA and embedding finetuning is not that slow, I'd suggest authors also provide the memory consumption and time used in this process, addition to optimization iterations.

3. Are the specific evaluation viewpoints come from Zero123++ settings? So the method is actually limited to certain viewpoints?

4. It's interesting to see that images generated from Zero123++, which is trained on Objaverse, can generate reasonable images' guidance with scene image input. Combining this with previous question, what's the problem of using some open-sourced scene-level NVS models for pseudo guidance? For example, I'd assume ZeroNVS[1] can provide images with any viewpoint input condition.

[1] Sargent, Kyle, et al. "Zeronvs: Zero-shot 360-degree view synthesis from a single real image." CVPR 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666570607

thread_type:
official_review

<thread_object>
creator: reviewer_brtu

<content>
summary:
The paper addresses the problem of generating novel views from a single image without relying on 3D datasets for training. The authors propose to leverage the 3D knowledge embedded in large-scale pretrained diffusion models to create camera-controlled viewpoints through test-time optimization. Specifically, the method fine-tunes the text prompt and diffusion model to align with the input and a pseudo-guidance image, incorporating mutual information guidance during the final diffusion process to enhance image fidelity. The experiments demonstrate that the proposed model can generate semantically consistent novel views for images with complex background.

strengths:
- The proposed approach effectively integrate 3D-free methods and pretrained 3D-based prior to achieve viewpoint-controlled novel-view synthesis. This technique can generalize to complex scenes without needing large 3D training datasets.

- The mutual information guidance improves fidelity of the generated images over the pseudo guidance.

- The method is tested both qualitatively and quantitatively, showcasing superior performance compared to state-of-the-art models, especially in handling background details and complex scenes.

- The analysis of the significance of CLIP and guidance images provides a solid motivation for the proposed research.

weaknesses:
- While the authors acknowledge the limitation of inference-time optimization in terms of real-time applicability and scalability, a direct runtime comparison with baseline models would provide additional clarity.

- This method is incapable of generating arbitrary camera viewpoints. this might be due to the choice of the 3D prior, Zero123++, which can provide guidance images at six fixed viewpoints. Clarifying this limitation or demonstrating the capability of generating arbitrary viewpoints is much needed.

- The novelty of the proposed framework is somewhat limited. The approach builds heavily on the HawkI, with modifications such as replacing guidance images generated via Inverse Perspective Mapping with those from a 3D prior method (Zero123++). This modification, while practical, could be seen as a straightforward combination.

questions:
The generated images across different viewpoints appear inconsistent. Does the proposed method include any mechanisms to encourage consistency across these viewpoints?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732252162042

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal - Global Response (1)

comment:
First of all, we would like to sincerely thank you for providing detailed comments on our research and offering constructive feedback. Among the comments mentioned by various reviewers, overlapping points have been summarized and addressed below. 

- **We would appreciate it if you could read it along with the revised paper we have updated.**
- **Also, we kindly request that you review the paper again as there have been improvements in the results/methodology.**

### **(A1) Computation Time and GPU Memory Consumption (Reviewers BRTu, uyxj, 9Fq5, hAuB)**

Many reviewers pointed out the need for an analysis of computation time, so we conducted related experiments. By comparing the computation time and memory consumption of the Zero123++, Stable Zero123, and ZeroNVS models, we quantitatively demonstrated which model is best suited for our framework. 

Testing on an A100 GPU (40GB) showed that **Zero123++ (10.18GB)** consumes the **least memory** compared to Stable Zero123 (39.3GB) and ZeroNVS (33.48GB). Additionally, the computation time for generating a single camera-controlled image was 1,278 seconds for Stable Zero123 and 7,500 seconds for ZeroNVS, while **Zero123++ could generate an image in just 20 seconds**. 

As a result, Zero123++ was selected as the guidance model for our framework. 

> | Model           | Memory Consumption      | Computation Time |
|------------------|-------------------------|------------------|
| Zero123++       | **10.18 GB** / 40.0 GB (9,715 MiB)  | **20 sec**          |
| Stable Zero123  | 39.3 GB / 40.0 GB (37,479 MiB)  | 1,278 sec       |
| ZeroNVS         | 33.48 GB / 40.0 GB (31,929 MiB) | 7,500 sec       |

> Table 1. **Comparison of computation times for 3D-prior models.** Among the prior works in NVS frequently mentioned, including Zero123++, Stable Zero123, and ZeroNVS, the Zero123++ model has the shortest computation time. Our research applies the Zero123++ model, which has the lowest computation time among 3D-prior models, to obtain 3D prior information without requiring any additional computation time.

We understand concerns about the potential increase in computation time due to the addition of 3D-prior framework. 

However, experimental results showed that the computation time of our model (measured as the time from program initiation to generating a first image) was **393 seconds**, which is slightly faster than the 401 seconds of the baseline model. 

Throughout the optimization/inference process, **our model uses at most 8.43GB** of memory, and the Zero123++ image generation process requires 10.18GB, which indicates that our approach is viable even on consumer-grade GPUs. 

This **highlights the value of our method in terms of general applicability.** The computation time and memory consumption analysis can be found in the revised paper.

> | Model            | Step           | Memory Consumption         | Computation Time          |
|-------------------|----------------|-----------------------------|---------------------------|
| HawkI            | Optimization   | 7.20 GB / 40.0 GB (6,875 MiB) | 395 sec                  |
|                   | Inference      | 8.43 GB / 40.0 GB (8,045 MiB) | 6 sec (each image)       |
| Total                  |       | **8.43 GB** / 40.0 GB (8,045 MiB) | **401 sec**       |
| Ours (w/o regloss) | Zero123++     | 10.18 GB / 40.0 GB (9,715 MiB) | 20 sec                  |
|                   | Optimization   | 7.21 GB / 40.0 GB (6,879 MiB) | 372 sec                 |
|                   | Inference      | 8.43 GB / 40.0 GB (8,049 MiB) | 6 sec (each image)       |
| Total                  |       | **10.18 GB** / 40.0 GB (9,715 MiB) | **398 sec**       |
| Ours (w/ regloss) | Zero123++      | 10.18 GB / 40.0 GB (9,715 MiB) | 20 sec                  |
|                   | Optimization   | 7.21 GB / 40.0 GB (6,885 MiB) | 367 sec                 |
|                   | Inference      | 8.43 GB / 40.0 GB (8,045 MiB) | 6 sec (each image)       |
| Total                  |       | **10.18 GB** / 40.0 GB (9,715 MiB) | **393 sec**       |

> Table 2. **Detailed Step-wise Comparison**. Even when applying Zero123++ to our methodology, the additional GPU memory consumption is relatively small, at **2.97GB** (10.18GB - 7.21GB), and it takes only **20 seconds** to generate the guidance image using Zero123++. From an overall perspective, HawkI takes 401 seconds to complete optimization and generate the first image through inference, while Ours (w/o regloss) takes 398 seconds, and Ours (w/ regloss) takes **393 seconds**. This demonstrates that our methodology does not result in significant differences in computation time or memory consumption while achieving better performance compared to existing methods. Total memory consumption refers to the worst case, the computation time indicates the total execution time. *i.e.,* the time taken for the model to run and output the first image.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733122339797

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal - Global Response : Image Consistency [NEW]

comment:
Dear Reviewers,

Since multiple reviewers have mentioned this issue, **we would like to post an additional rebuttal regarding image consistency.** 

As shown in the results attached at the anonymous URL below, we demonstrate that **consistency can be improved by utilizing the Runway Gen-3 Alpha Turbo model as a 3D-prior model** instead of Zero123++.

**URL:** https://github.com/2024nvs/2024nvs

Please kindly note that the core idea of this work is **not the improvement of consistency itself** but rather **the ability to achieve camera control by combining 3D priors with a 3D-free architecture**, which is the novel insight this work has introduced.

We hope that **minor artifacts due to our initial choices in implementation using Zero123++ should be alleviated**, after we demonstrate the new results on using Runway Gen-3 Alpha Turbo as 3D prior.

We kindly request **Reviewer BRTu, 9Fq5, and yRXV to reconsider their ratings**, given our new implementation and results using Runway Gen-3 Alpha Turbo.

With warm regards,
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the problem of scene-level novel view synthesis (given an image and a new camera position, generate a new image). The paper proposes to combine a "3d-free" method which uses CLIP and a image guidance with a "3d prior", an existing model which has already been trained to do novel view synthesis (though on objects). The paper demonstrates strong results on the HawkI dataset.

The strength of the paper is its quantitative performance on HawkI dataset. The weakness are numerous:
1) The evaluation on this HawkI dataset for scene-level 3d is really abnormal. It avoids many more standard datasets such as skipping over many more standard dataset such as CO3D, RE10K, Acid or DTU.
2) The comparisons used in the papers are weak and the rebuttal is literally wrong. As many reviewers mentioned, ZeroNVS should be a standard comparison. In the rebuttal the authors state that "generating a single camera-controlled image was 7,500 seconds for ZeroNVS". This is inaccurate. ZeroNVS follows the architecture of Zero123 which takes 2 seconds on a RTX A6000 GPU. It's possible that the NeRF optimization was included but this is not necessary for a single image sample and other papers, e.g. MegaScenes have disentangled the diffusion inference from the NeRF optimization in order to compare directly with the NVS properties.
3) The writing and method are confusing. The paper repeatedly says that their method does not use 3D data. But they directly use Zero123+ which is trained on 3D data. In the rebuttal, they clarify and say that they don't use "additional" 3D data and that Zero123++ is object-level. While this is true, it runs completely against how the current intro is written. Even a one of the more positive reviewers (hAuB) notes this in their weakness (Weakness 4).
4) Even the qualitative performance on HawkI seems to lack image consistency.

I suggest rejection. For future submissions, the authors should use standard datasets and compare against the SOTA or near SOTA for these datasets (e.g. MegaScenes would be an approximate example).