3D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidance

## Abstract

Abstract Recent 3D novel view synthesis (NVS) methods are limited to single-object-centric scenes and struggle with complex environments. They often require extensive 3D data for training, lacking generalization beyond the training distribution. Conversely, 3D-free methods can generate text-controlled views of complex, in-the-wild scenes using a pretrained stable diffusion model without the need for a large amount of 3D-based training data, but lack camera control. In this paper, we introduce a method capable of generating camera-controlled viewpoints from a single input image, by combining the benefits of 3D-free and 3D-based approaches. Our method excels in handling complex and diverse scenes without extensive training or additional 3D and multiview data. It leverages widely available pretrained NVS models for weak guidance, integrating this knowledge into a 3D-free view synthesis approach to achieve the desired results. Experimental results demonstrate that our method outperforms existing models in both qualitative and quantitative evaluations, providing high-fidelity and consistent novel view synthesis at desired camera angles across a wide variety of scenes.

## 1 Introduction

Novel-view synthesis plays a pivotal role in numerous real-world applications, including 3D environments, augmented reality (AR), virtual reality (VR), and autonomous driving. Recent advancements in diffusion-based methods, such as Zero-1-to-3 (Zero123) Liu et al. () and Zero123++ Shi et al. (), along with NeRFs Tancik et al. (); Zhou & Tulsiani () and Gaussian Splatting Li et al. (); Zhu et al. (), have significantly propelled the field forward. These techniques enable the specification of camera angles and the sampling of novel-view images from precise viewpoints. However, diffusion-based methods remain largely object-centric and may struggle to generalize to complex scenes with intricate backgrounds. They also require extensive 3D object datasets for training. In contrast, NeRF and Gaussian Splatting methods can handle complex scenes but depend on multi-view information to construct 3D models. Therefore, achieving novel-view synthesis from a single image in a data-efficient manner, without relying on additional 3D, multi-view, or depth information, is highly advantageous.

Conversely, 3D-free methods like HawkI Kothandaraman et al. () intelligently extract the rich 3D knowledge embedded in text-to-2D image diffusion models, such as stable diffusion Rombach et al. (); Podell et al. (), to generate text-controlled views from complex, real-world input images without needing additional multi-view or 3D information for fine-tuning or inference. However, these methods lack the ability to precisely control camera angles when generating novel-view images. Ideally, we aim for both data-efficient novel view synthesis and camera controllability, which is the primary focus of this paper.

We begin by analyzing how well CLIP understands the 3D space and the benefits of guidance images for 3D-free methods like HawkI. Using these insights, along with the established knowledge that 3D-based methods such as HawkI enable precise camera control and 3D-free methods like Zero123++ offer generalizability and data efficiency, we propose a simple approach for novel-view synthesis that generates camera-controlled novel views at specified azimuth and elevation angles from a single input image, without requiring 3D datasets or extensive training.

Our method utilizes information from off-the-shelf pretrained 3D-based models like Zero123++ Shi et al. () in conjunction with the pretrained stable diffusion model. It follows a 3D-free HawkI-style inference-time optimization procedure to achieve the desired results. We validate our approach through extensive qualitative and quantitative comparisons across various metrics that assess text consistency and fidelity.

To summarized our contributions, we (1) we analyze how well the CLIP model understands the 3D space and the usefulness of guidance image in 3D-free methods, (2) present a simple approach that uses 3D-priors within 3D-free 2D-based approaches for novel view synthesis, with precise camera control, that works well for complex image with multiple objects, backrgounds, etc.

## 2 Prior Work on 3D and 3D-free Approaches for NVS

Recent research has increasingly focused on novel view synthesis using diffusion models Chen et al. (); Mildenhall et al. (); Shi et al. (). Approaches for 3D generation Chen et al. (); Lin et al. (); Poole et al. (); Raj et al. (); Xu et al. (); Gao et al. () often rely on text for reconstruction and require substantial multi-view and 3D data Shi et al. (); Wang & Shi (); Yang et al. (); Liu et al. (); Höllein et al. () for supervised learning. For instance, Zero123 Liu et al. (), Magic123 Qian et al. (), and Zero123++ Shi et al. () utilize a pre-trained stable diffusion model Rombach et al. () combined with 3D data to learn various camera viewpoints. However, they are extremely data hungry, meaning that they need extensive multi-view and 3D data to train on. Additionally, most state-of-the-art view synthesis algorithms are largely object-centric, and may not work well on complex scenes containing multiple objects or background information. This is due to the domain gap between the 3D objects data that they are typically trained on, and the inferencing image.

Free3D Zheng & Vedaldi () introduces an efficient method for synthesizing accurate 360-degree views from a single image without 3D representations. By incorporating the Ray Conditioning Normalization (RCN) layer into 2D image generators, it encodes the target view’s pose and enhances view consistency with lightweight multi-view attention layers and noise sharing. However, it still requires training on large-scale 3D datasets like Objaverse and multi-view information, and it cannot include background transformations. DreamFusion Poole et al. () presents a Text-to-3D method using a NeRF and a Diffusion Model-based Text-to-2D model. It introduces a probability density distillation loss, allowing the 2D Diffusion Model to optimize image generation without needing 3D data or model modifications. DreamFusion’s key contributions are creating Text-to-3D models without 3D dataset training and utilizing a Diffusion Model. However, it cannot transform images with backgrounds or introduce elevation changes in camera-controlled images. HawkI Kothandaraman et al. () synthesizes high-quality aerial view images using text and a single input image without 3D or multi-view information. It employs a pretrained text-to-2D Stable Diffusion model, achieving a balance between aerial view consistency and input image fidelity through test-time optimization and mutual information-based inference. However, HawkI struggles with controlling camera angles, detailed feature generation, and maintaining view consistency.

## 3 Understanding 2D models and the 3D space

### 3.1 How well do CLIP models understand the 3D-space?

This section examines how well CLIP models understand 3D space by focusing on the 3D-free method, HawkI Kothandaraman et al. (). HawkI uses classical computer vision techniques to generate aerial view images from ground view images via a homography transformation, which serves as the guidance image for the diffusion model. Our hypothesis is that HawkI’s ability to perform viewpoint transformations without 3D data relies on this homography image.

We conducted an experiment where the homography image was omitted to see if CLIP could generate camera-controlled images without it. Detailed angle instructions were still provided in the target text description. Our results showed that CLIP could not independently generate consistent viewpoints, highlighting the importance of 3D guidance images.

In the experiment, different pyramids, waterfalls, and houses were generated inconsistently, and camera control angles were not accurately followed. This demonstrates that CLIP struggles with 3D comprehension and validates the necessity of 3D guidance images. Figure 3 illustrates these findings.

Figure: Figure 3: Analysis of how well CLIP understands the 3D space In this experiment, we generate camera control images for specific angles without using any guidance image.
Refer to caption: x3.png

### 3.2 Importance of guidance image in 3D-free methods

Our previous analysis revealed that the CLIP model without a guidance image struggles to understand 3D space, resulting in inconsistent images. Conversely, HawkI with a guidance image cannot perform transformations from various camera viewpoints. This raises the question of what kind of guidance image is suitable for 3D-free camera control.

We conduct experiments using the images generated using the pretrained Zero123++ Shi et al. () model for guidance. In our experiments, the target text specified the desired transformation angles, but the guidance images had different angles. i.e. the guidance images introduced incorrect viewpoints. When generating an image at a $(30^{\circ},30^{\circ})$ angle, the model followed the guidance image’s suggestion, regardless of the text input. This indicates that the model benefits from the information in the 3D-derived guidance image, and that the guidance image is not simply a source of extra variance. Our experiment confirms the importance of guidance images in camera control, validating our methodology. Figure 4 illustrates these findings.

Figure: Figure 4: Using an image with an incorrect viewpoint as the guidance image In this experiment, we examine how the results are derived when an incorrect viewpoint image is used as a guidance image.
Refer to caption: x4.png

## 4 3D-free + 3D priors for Novel-View Synthesis

We present a simple method (Figure 2) for data-efficient text and camera-controlled novel-view synthesis from a single input image ($I_{input}$) and its text description ($t_{input}$) (e.g., “An ancient Egyptian pyramid in the desert,” obtained using the BLIP-2 Li et al. () model). Our model eliminates the need for training data, 3D data, or multi-view data. Instead, it utilizes a pretrained text-to-2D image stable diffusion model as a strong prior, along with pretrained novel-view synthesis (NVS) models, e.g. Zero123++ for guidance. Our method combines information from the pretrained NVS model and performs a rapid inference-time optimization and inference routine, following a HawkI-style inference approach, to generate novel-view images of any given in-the-wild complex input scene at specified elevation ($\alpha_{elev}$) and azimuth angles ($\alpha_{azi}$).

#### Inference-time Optimization

We employ a pretrained NVS model $G$ to obtain a weak prediction, $I_{view}$, of $I_{input}$ at ($\alpha_{elev}$, $\alpha_{azi}$). This prediction is represented as $I_{view}=G(I_{input},\alpha_{elev},\alpha_{azi})$. Although $I_{view}$ is not a fully accurate depiction of the desired target, it provides weak or pseudo guidance for the model regarding the content and direction of the desired viewpoint transformation. Subsequently, we utilize the pretrained text-to-image stable diffusion model to perform inference-time optimization.

Specifically, initially, we enhance the CLIP embedding for $t_{input}$ with $I_{input}$ to derive $e_{optim}$ (optimized CLIP text-image embedding from $e_{input}$, which is the CLIP test embedding for $t_{input}$), which is optimized to most accurately reconstruct $I_{input}$:

$$ ${\it\min_{e_{\text{optim}}}\sum_{t=T}^{0}L(f(x_{t},t,e_{\text{optim}};\theta), I_{input})}$ (1) $$

Subsequently, the LoRA layers (with parameters $\theta_{LoRA}$) within the cross-attention layers of the diffusion model are fine-tuned at $e_{optim}$ to replicate $I_{input}$, employing the diffusion denoising loss function,

$$ ${\it\min_{\theta_{\text{LoRA}}}\sum_{t=T}^{0}L(f(x_{t},t,e_{\text{optim}}; \theta),I_{input})}$ (2) $$

while keeping the rest of the UNet model unchanged. This process is then iterated for $I_{view}$, where $e_{optim}$ is again refined to $e_{view}$ to best reconstruct $I_{view}$. Following this, the LoRA layers are adjusted to capture the nuances of the weak guidance image $I_{view}$, guiding the transformation towards the desired outcome.

Thus, the predicted image from the 3D-based NVS method, Zero123++, serves as weak or pseudo guidance. The optimization strategy conditions the embedding space with knowledge related to the input image and its view variants. The guidance image prior increases the variance of the diffusion model, facilitating view transformation and providing a direction for viewpoint transformation.

#### Inference

To generate the camera-controlled image with designated elevation ($\alpha_{elev}$) and azimuth angles ($\alpha_{azi}$), we use the target text description $t_{target}$, which varies according to the corresponding $\alpha_{elev}$ and $\alpha_{azi}$. For instance, if $\alpha_{elev}=30^{\circ}$ and $\alpha_{azi}=30^{\circ}$, $t_{target}$ can be formatted as ”View from an elevated angle of +30 degrees and an azimuth angle of +30 degrees” + $t_{input}$ (e.g., ”View from an elevated angle of +30 degrees and an azimuth angle of +30 degrees, An ancient Egyptian pyramid in the desert.”). Next, we use the finetuned diffusion model to generate the target image using $t_{target}$, along with mutual information guidance Kothandaraman et al. (), which enforces similarity between the contents of the generated and input images.

## 5 Experiments and Results

Figure: Figure 5: Results on HawkI-Syn. Comparisons between the state-of-the-art view synthesis models, Zero123++, HawkI, Stable Zero123, and our method highlights the superior performance of our model in terms of background inclusion, view consistency, and the accurate representation of target elevation and azimuth angles.
Refer to caption: x5.png

Figure: Figure 6: Results on HawkI-Real. Comparisons between the state-of-the-art view synthesis models, Zero123++, HawkI, Stable Zero123, and our method highlights the superior performance of our model in terms of background inclusion, view consistency, and the accurate representation of target elevation and azimuth angles.
Refer to caption: x6.png

Figure: Figure 7: More results on HawkI-Syn. We present additional comparison results on HawkI-Syn for the angles of $(-20^{\circ},210^{\circ})$ and $(-20^{\circ},330^{\circ})$. Our model consistently produces view synthesis images that maintained background inclusion and view consistency, accurately mirroring the target elevation and azimuth angles. Notably, StableZero123 exhibits instability in its results. It’s important to highlight that this task specifically addresses negative azimuth angles. HawkI, for instance, fails to capture the correct camera elevation and is limited to generating aerial views. Zero123++ is capable of handling both elevation and azimuth but falls short in integrating background elements and intricate details, as also observed in previous outcomes. For example, when presented with an image of a pyramid casting a shadow, Zero123++ darkens the pyramid but fails to render the shadow accurately. This shortcoming is also apparent in images of a waterfall and a house. In the waterfall task within the specified azimuth range, Zero123++ produces an indistinct shape rather than a clear environment where water and lake are visible from below the rocks. Similarly, for the house image, it generates an incomplete image with gray patches. Conversely, our model not only captures the shadow details of the pyramid but also accurately renders the environment in the waterfall image, ensuring visibility of water and lake from beneath the rocks. Additionally, it adeptly incorporates details and backgrounds from multiple perspectives.
Refer to caption: x7.png

Figure: Figure 8: More Results on HawkI-Real. We extend our analysis to additional settings of $(-20^{\circ},210^{\circ})$ and $(-20^{\circ},330^{\circ})$. Our model, when tested on the HawkI-Real dataset, demonstrated superior performance in view synthesis images, excelling in background inclusion and view consistency, and accurately representing the target elevation and azimuth angles. In comparison to other leading models such as Zero123++, HawkI, and StableZero123, our model’s results are notably better. StableZero123’s outputs are incomplete, and Zero123++ struggles with capturing background details and intricate information. Specifically, Zero123++ neglected surrounding details, focusing solely on the Eiffel Tower. The original HawkI model also failed to achieve the correct camera elevation or produced images that overlooked important features. For example, in the cat transformation task, the output incorrectly depicted three cats instead of two. Our model stands out by delivering exceptional results for the Eiffel Tower, Hawaiian beach, and cat transformations, underscoring its advanced capabilities over other models. Furthermore, we present a quantitative evaluation in Table 2, which confirms our model’s dominance over state-of-the-art benchmarks across various metrics.
Refer to caption: x8.png

#### Datasets

We utilize the HawkI-Syn Kothandaraman et al. () and HawkI-Real Kothandaraman et al. () datasets that feature complex scenes with multiple foreground objects and background. Both datasets provide images and text prompts to the model. Explain the subset you used for your results, how you created the, how many images per sample you used for evaluation, etc.

#### Baselines

We compare our method with state-of-the-art view synthesis methods: Zero123++ Shi et al. () and Stable Zero123 for 3D-based methods and HawkI Kothandaraman et al. () for 3D-free method.

#### Implementation Details

We employ the stable diffusion 2.1 model as the backbone for all our experiments and results. To generate the pseudo guidance images for different viewpoints, we use the pretrained Zero123++ Shi et al. () model. All images except those in HawkI-Real dataset are used at a resolution of $512\times 512$. For $I_{input}$, we train the text embedding for 1,000 iterations with the learning rate of $1e-3$ and the diffusion model for 500 iterations with the learning rate of $2e-4$. Training the text embedding for 1,000 iterations guarantees that the text embedding $e_{optim}$ is not too close to the $e_{input}$, avoiding bias towards $I_{input}$. Likewise, it is not too distant from $e_{input}$, allowing the text embedding space to capture the characteristics of $I_{input}$. Regarding $I_{view}$, we trained the text embedding for 500 iterations and the diffusion UNet for 250 iterations. We aim for $e_{view}$ to be near $e_{optim}$ and limit the diffusion model training to 250 iterations to prevent overfitting to $I_{view}$. The purpose of $I_{view}$ is to introduce variability and provide pseudo supervision rather than accurately approximating the camera control. We set the mutual information guidance hyperparameter to $1e-6$ and conduct inference over 50 steps.

### 5.1 Qualitative analysis

We evaluate our method on four distinct viewpoints:

$$ $\{(\alpha_{elev},\alpha_{azi})\mid(30^{\circ},30^{\circ}),(-20^{\circ},210^{ \circ}),(30^{\circ},270^{\circ}),\vspace*{-0.5em}(-20^{\circ},330^{\circ})\}.$ $$

Our model is able to generate distinct viewpoints in the camera control to ensure consistency across generated views. Details are mentioned in the Zero123++ Shi et al. (). We present qualitative representative results and comparisons with Zero123++ Shi et al. (), HawkI Kothandaraman et al. (), and StableZero123 at camera angles of $(30^{\circ},30^{\circ})$ and $(30^{\circ},270^{\circ})$ in Figures 5 and 6. Our method demonstrates superior scene reconstruction from all viewpoints compared to previous works. Specifically, results on HawkI-Syn in Figure 5 show that StableZero123 is largely ineffective. HawkI fails to capture the correct camera elevation in all cases except for the house image. While Zero123++ handles both elevation and azimuth, it struggles with background and detailed features. For instance, the pyramid in the first row lacks shadow information; the waterfall image in the second row appears unnatural; and the house in the last row blurs detailed features. Conversely, our model accurately reflects shadow characteristics in the pyramid, and reconstructs the details and background of the waterfall and house examples from various viewpoints.

Similar observations are made for HawkI-Real results shown in Figure 6. StableZero123 is ineffective. Zero123++ fails to capture background or detailed information. For example, when tasked with camera control for an image of the Eiffel Tower, Zero123++ focuses solely on the Eiffel Tower, ignoring surrounding details. The original HawkI model, while producing aerial views, fails in angle conversion tasks. In contrast, our model accurately performs angle conversion tasks at $(30^{\circ},30^{\circ})$ and $(30^{\circ},270^{\circ})$, including the Seine River in the background for the Eiffel Tower image, showcasing its superiority. Camera control tasks for the HawkI-Real dataset, including images like the Hawaii beach and a cat, further demonstrate our model’s excellence compared to other models. The key benefits of our model over 3D-based NVS methods such as Zero123++ and 3D-free methods such as HawkI arises by merging the strengths of 3D-based techniques into a 3D-free optimization process, effectively combining the best features of both.

### 5.2 Quantitative Evaluation

Following prior work Shi et al. (); Kothandaraman et al. (); Liu et al. (), we evaluate our method using six metrics - (i) LPIPS Zhang et al. (): Quantifies the perceptual similarity between the generated and input images, with lower values indicating better performance. (ii) CLIP-Score Radford et al. () and View-CLIP Score: Measure text-based alignment of the generated images. The CLIP score assesses alignment with both content and the $(\alpha_{\text{elev}},\alpha_{\text{azi}})$ viewpoint, while the View-CLIP Score focuses specifically on the viewpoint. Higher values are preferred. (iii) CLIP-I Ruiz et al. (), CLIPD, and View-CLIPD: Evaluate image fidelity and directional similarity. CLIP-I measures the cosine similarity between the embeddings of multi-view images and the input image within the CLIP space. CLIPD(^1^11[https://huggingface.co/docs/diffusers/conceptual/evaluation](https://huggingface.co/docs/diffusers/conceptual/evaluation)) and View-CLIPD assess the consistency of transformations between two images, using text prompts similar to CLIP-Score and View-CLIP scores. Higher values are preferred. Similar to the quantitative comparison performed by Zero123++, we use 10% of the overall data from the HawkI-Syn and HawkI-Real datasets as the validation set to compute the quantitative metrics. Table 1 and Table 2 shows that our model significantly outperforms the state-of-the-art across these evaluation metrics, reinforcing how our models stands out by incorporating the robust features of 3D-based NVS methods into a 3D-free optimization strategy, thereby capitalizing on the benefits of both approaches.

**Table 1: Quantitative Results. Evaluation on six metrics demonstrate the superior results of our method over prior work.**
| Dataset, Angle, Method | LPIPS $\downarrow$ | CLIP $\uparrow$ | View-CLIP $\uparrow$ | CLIPD $\uparrow$ | View-CLIPD $\uparrow$ | CLIP-I $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| HawkI-Syn $(30^{\circ},30^{\circ})$ Ours | 0.5667 | 29.9125 | 19.7100 | 0.0534 | 0.0568 | 0.8334 |
| HawkI-Syn $(30^{\circ},30^{\circ})$ HawkI | 0.5998 | 27.6516 | 18.6122 | 0.0405 | 0.0549 | 0.8108 |
| HawkI-Syn $(30^{\circ},30^{\circ})$ Zero123++ | 0.5793 | 29.8709 | 18.4740 | 0.0552 | 0.0540 | 0.8330 |
| HawkI-Syn $(30^{\circ},30^{\circ})$ Stable Zero123 | 0.6853 | 20.5335 | 18.0746 | 0.0528 | 0.0520 | 0.6181 |
| HawkI-Syn $(30^{\circ},270^{\circ})$ Ours | 0.5791 | 29.2654 | 19.1836 | 0.0553 | 0.0611 | 0.8274 |
| HawkI-Syn $(30^{\circ},270^{\circ})$ HawkI | 0.5996 | 29.0022 | 19.1088 | 0.0321 | 0.0498 | 0.8438 |
| HawkI-Syn $(30^{\circ},270^{\circ})$ Zero123++ | 0.5911 | 28.9917 | 19.1491 | 0.0482 | 0.0597 | 0.7917 |
| HawkI-Syn $(30^{\circ},270^{\circ})$ Stable Zero123 | 0.6441 | 21.2231 | 18.3738 | 0.0406 | 0.0403 | 0.6152 |
| HawkI-Real $(30^{\circ},30^{\circ})$ Ours | 0.6075 | 30.9333 | 20.7728 | 0.0829 | 0.0730 | 0.8333 |
| HawkI-Real $(30^{\circ},30^{\circ})$ HawkI | 0.6405 | 28.5431 | 18.5430 | 0.0506 | 0.0468 | 0.8302 |
| HawkI-Real $(30^{\circ},30^{\circ})$ Zero123++ | 0.6679 | 29.8661 | 19.9530 | 0.0624 | 0.0467 | 0.8049 |
| HawkI-Real $(30^{\circ},30^{\circ})$ Stable Zero123 | 0.6624 | 22.8604 | 18.7536 | 0.0523 | 0.0658 | 0.6740 |
| HawkI-Real $(30^{\circ},270^{\circ})$ Ours | 0.6180 | 30.0677 | 19.6756 | 0.0652 | 0.0709 | 0.8350 |
| HawkI-Real $(30^{\circ},270^{\circ})$ HawkI | 0.6497 | 28.4452 | 18.4231 | 0.0574 | 0.0577 | 0.8236 |
| HawkI-Real $(30^{\circ},270^{\circ})$ Zero123++ | 0.6935 | 27.2773 | 19.5926 | 0.0605 | 0.0698 | 0.7212 |
| HawkI-Real $(30^{\circ},270^{\circ})$ Stable Zero123 | 0.6466 | 23.6915 | 17.8817 | 0.0593 | 0.0658 | 0.6726 |

**Table 2: Quantitative Results. Evaluation on six metrics demonstrate the superior results of our method over prior work.**
| Dataset, Angle, Method | LPIPS $\downarrow$ | CLIP $\uparrow$ | View-CLIP $\uparrow$ | CLIPD $\uparrow$ | View-CLIPD $\uparrow$ | CLIP-I $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| HawkI-Syn $(-20^{\circ},210^{\circ})$ Ours | 0.5521 | 28.5599 | 17.9664 | 0.0502 | 0.0565 | 0.8535 |
| HawkI-Syn $(-20^{\circ},210^{\circ})$ HawkI | 0.5747 | 27.0915 | 16.9330 | 0.0403 | 0.0477 | 0.8370 |
| HawkI-Syn $(-20^{\circ},210^{\circ})$ Zero123++ | 0.7193 | 23.9836 | 16.7291 | 0.0496 | 0.0886 | 0.7389 |
| HawkI-Syn $(-20^{\circ},210^{\circ})$ Stable Zero123 | 0.6938 | 20.3084 | 16.8253 | 0.0403 | 0.0400 | 0.5826 |
| HawkI-Syn $(-20^{\circ},330^{\circ})$ Ours | 0.5672 | 29.1569 | 18.1494 | 0.0556 | 0.0535 | 0.8656 |
| HawkI-Syn $(-20^{\circ},330^{\circ})$ HawkI | 0.5967 | 27.6743 | 17.3102 | 0.0520 | 0.0529 | 0.8409 |
| HawkI-Syn $(-20^{\circ},330^{\circ})$ Zero123++ | 0.6662 | 24.6859 | 17.5781 | 0.0688 | 0.0962 | 0.7383 |
| HawkI-Syn $(-20^{\circ},330^{\circ})$ Stable Zero123 | 0.6201 | 22.3172 | 16.2088 | 0.0499 | 0.0542 | 0.7034 |
| HawkI-Real $(-20^{\circ},210^{\circ})$ Ours | 0.6222 | 29.6425 | 18.8914 | 0.0744 | 0.0709 | 0.7868 |
| HawkI-Real $(-20^{\circ},210^{\circ})$ HawkI | 0.6364 | 27.3844 | 17.5037 | 0.0568 | 0.0507 | 0.7821 |
| HawkI-Real $(-20^{\circ},210^{\circ})$ Zero123++ | 0.7739 | 24.3284 | 18.7046 | 0.0713 | 0.0951 | 0.6798 |
| HawkI-Real $(-20^{\circ},210^{\circ})$ Stable Zero123 | 0.6637 | 21.2952 | 17.2609 | 0.0556 | 0.0704 | 0.5829 |
| HawkI-Real $(-20^{\circ},330^{\circ})$ Ours | 0.6135 | 30.6951 | 18.9849 | 0.0506 | 0.0371 | 0.8500 |
| HawkI-Real $(-20^{\circ},330^{\circ})$ HawkI | 0.6500 | 28.5102 | 17.4244 | 0.0481 | 0.0315 | 0.8463 |
| HawkI-Real $(-20^{\circ},330^{\circ})$ Zero123++ | 0.7132 | 25.5205 | 17.4653 | 0.0833 | 0.0977 | 0.7520 |
| HawkI-Real $(-20^{\circ},330^{\circ})$ Stable Zero123 | 0.5730 | 23.9690 | 17.0343 | 0.0328 | 0.0726 | 0.7258 |

## 6 Conclusions, Limitations and Future Work

In this paper, we propose an approach that integrates the advantages of off-the-shelf 3D-based pretrained models within 3D-free paradigms for novel view synthesis, offering precise control over camera angle and elevation. Our method performs effectively on complex, in-the-wild images containing multiple objects and background information. We qualitatively and quantitatively demonstrate the benefits of our method over corresponding 3D and 3D-free baselines. One limitation of our method is its reliance on an inference-time optimization routine for each scene and viewpoint, which may hinder real-time performance. Achieving faster performance is a direction for future work. Additionally, extending our approach to NVS and 3D applications with real-world constraints (such as respecting contact points and relative sizes) for tasks like editing, object insertion, and composition presents promising directions for further research. Based on the current results, we also propose exploring the use of an image-conditioned model to achieve a higher level of view consistency as a future research direction.

## Appendix A Appendix

Figure: Figure 9: Additional comparisons in $(30^{\circ},30^{\circ})$ and $(30^{\circ},270^{\circ})$ settings on images from the HawkI-Syn and HawkI-Real datasets. Comparisons between the state-of-the-art view synthesis models, Zero123++, HawkI, Stable Zero123, and our method highlights the superior performance of our model in terms of background inclusion, view consistency, and the accurate representation of target elevation and azimuth angles.
Refer to caption: x9.png

Figure: Figure 10: Additional comparisons in $(30^{\circ},30^{\circ})$ and $(30^{\circ},270^{\circ})$ settings on images from the HawkI-Syn and HawkI-Real datasets. Comparisons between the state-of-the-art view synthesis models, Zero123++, HawkI, Stable Zero123, and our method highlights the superior performance of our model in terms of background inclusion, view consistency, and the accurate representation of target elevation and azimuth angles.
Refer to caption: x10.png

Figure: Figure 11: Additional comparisons in $(30^{\circ},30^{\circ})$ and $(30^{\circ},270^{\circ})$ settings on images from the HawkI-Syn and HawkI-Real datasets. Comparisons between the state-of-the-art view synthesis models, Zero123++, HawkI, Stable Zero123, and our method highlights the superior performance of our model in terms of background inclusion, view consistency, and the accurate representation of target elevation and azimuth angles.
Refer to caption: x11.png

Figure: Figure 12: Additional comparisons in $(30^{\circ},30^{\circ})$ and $(30^{\circ},270^{\circ})$ settings on images from the HawkI-Syn and HawkI-Real datasets. Comparisons between the state-of-the-art view synthesis models, Zero123++, HawkI, Stable Zero123, and our method highlights the superior performance of our model in terms of background inclusion, view consistency, and the accurate representation of target elevation and azimuth angles.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The generated images across different viewpoints appear inconsistent. Does the proposed method include any mechanisms to encourage consistency across these viewpoints?
soundness: 2
strengths: - The proposed approach effectively integrate 3D-free methods and pretrained 3D-based prior to achieve viewpoint-controlled novel-view synthesis. This technique can generalize to complex scenes without needing large 3D training datasets.

- The mutual information guidance improves fidelity of the generated images over the pseudo guidance.

- The method is tested both qualitatively and quantitatively, showcasing superior performance compared to state-of-the-art models, especially in handling background details and complex scenes.

- The analysis of the significance of CLIP and guidance images provides a solid motivation for the proposed research.
summary: The paper addresses the problem of generating novel views from a single image without relying on 3D datasets for training. The authors propose to leverage the 3D knowledge embedded in large-scale pretrained diffusion models to create camera-controlled viewpoints through test-time optimization. Specifically, the method fine-tunes the text prompt and diffusion model to align with the input and a pseudo-guidance image, incorporating mutual information guidance during the final diffusion process to enhance image fidelity. The experiments demonstrate that the proposed model can generate semantically consistent novel views for images with complex background.
weaknesses: - While the authors acknowledge the limitation of inference-time optimization in terms of real-time applicability and scalability, a direct runtime comparison with baseline models would provide additional clarity.

- This method is incapable of generating arbitrary camera viewpoints. this might be due to the choice of the 3D prior, Zero123++, which can provide guidance images at six fixed viewpoints. Clarifying this limitation or demonstrating the capability of generating arbitrary viewpoints is much needed.

- The novelty of the proposed framework is somewhat limited. The approach builds heavily on the HawkI, with modifications such as replacing guidance images generated via Inverse Perspective Mapping with those from a 3D prior method (Zero123++). This modification, while practical, could be seen as a straightforward combination.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Below are some questions or some parts in the paper that I find confusing.

1. What is the detailed procedure of experiments related to figure 3. and figure 4.? What is the actual model that generates those images? CLIP itself isn't a generative model. Does the author mean using Stable Diffusion with different text embeddings? And how is the image guidance been used in (I assume) this standard SD model?

2. Although inference-time optimization with efficient methods like LoRA and embedding finetuning is not that slow, I'd suggest authors also provide the memory consumption and time used in this process, addition to optimization iterations.

3. Are the specific evaluation viewpoints come from Zero123++ settings? So the method is actually limited to certain viewpoints?

4. It's interesting to see that images generated from Zero123++, which is trained on Objaverse, can generate reasonable images' guidance with scene image input. Combining this with previous question, what's the problem of using some open-sourced scene-level NVS models for pseudo guidance? For example, I'd assume ZeroNVS[1] can provide images with any viewpoint input condition.

[1] Sargent, Kyle, et al. "Zeronvs: Zero-shot 360-degree view synthesis from a single real image." CVPR 2024.
soundness: 2
strengths: 1. The idea of combining priors from 2D image generative model and object-level 3D generative model is interesting and can be considered as an effective way to leverage different data source for 3D understanding. I think this idea in general is very important.

2. NVS results on scene images are quite impressive considering no additional training is involved.
summary: This paper proposes a framework for scene-level camera-control novel view synthesis(NVS). The framework combines 2D image generative prior with object-level 3D generative prior to achieve scene-level NVS. The authors start with an analysis on where should generative model learn the viewpoint information, then proposed a multi-stage inference-time optimization method to generate novel views given specific camera conditions and input images. Qualitative and quantitative evaluation on benchmarks show its superior performance comparing to other methods.
weaknesses: I think this paper's main weakness is on how the authors present the whole framework in a more motivated way. I'd briefly state this weakness here and leave the others in the following questions section.

1. The connection between the section3 and section4 is very confusing. First, the detailed experiment setting in section3 is not clearly explained (see questions), then it suddenly shift from section3 with CLIP embeddings to section4 on leveraging a diffusion model. I suggest authors revise the high-level motivations and present them more clearly in the paper.

2. The figure 2 for pipeline is actually making it harder to understand. I'd suggest at least add some orderings in it can better illustrate the whole pipeline.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: The method could be presented with more detail. Specifically, how do Eq. 1 and Eq. 2 work? Does the diffusion model output clean predicted samples? If not, how is the loss computed between the noisy latent $x_t$ and the clean image $I_{input}$?
soundness: 1
strengths: 1. The paper investigates an approach that sequentially optimizes the text embeddings and LoRA layers of a diffusion model to guide the novel view synthesis process.
summary: The paper presents an optimization-based sampling method to generate novel views at inference time. It utilizes multi-view images generated by a pre-trained diffusion model and optimizes the text embeddings and LoRA layers of a text-to-image diffusion model to guide the novel view sampling process.
weaknesses: 1. The proposed method relies on a pre-trained multi-view diffusion model, Zero123++, and thus shares its limitations: restricted view generation (Zero123++ generates six fixed-view images around the object), modification of background and object, handling multiple objects, and pose misalignment.
2. The evaluation can be seem unfair. For apple to apple comparison, the paper could compare with running the proposed method on Zero123++. Also, measure the CLIP score seem unfair as the proposed method optimizes the CLIP text embeddings. 
3. The proposed method introduces additional computational costs: generating a multi-view guidance images from a pre-trained diffusion model and the optimization of text embeddings to reconstruct $I_{input}$ and $I_{view}$. 

* L.86, L.90: Some terms are used in contradicting manner. 3D-free methods like Zero123++  and 3D-based models like Zero123++.
* L.95: we (1) we 
* Eqn. 1: The optimization task is provided without details. $L, f, x_t$ is not introduced before.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: It would be good if the authors clearly define what they mean by azimuth and elevation angles. The typical definition of the latter would be relative to the horizontal plane. Is that the case here as well, or are angles relative to the guidance view ? In any case, the angles specified in the image captions do not seem to follow one consistent set of definitions... Elevation angles are taking up values larger than 90 degrees, for instance. 

Related work that could be mentioned: Depth self-supervision for single image novel view synthesis, 2023 Giovanni Minelli et al.

The creation of videos with continuously changing viewpoints would be telling when it comes to consistency and angle correctness...
soundness: 3
strengths: 1. The method shows a good performance of novel view synthesis on a single view of a scene with object + background. As far as related works are concerned, the method shows - on average - better results using a textual prompt and without depth information for object + background when novel views are synthesized.

2. The results show a consistent performance gain over previous works in both real and synthetically generated data.
summary: The paper tackles the problem of novel view synthesis from a single image input of a scene by using textual camera view description. The problem statement is very similar to previous works Zero123++, HawkI, except that an additional guidance image is introduced, obtained through a pre-trained model Zero123++, tackling the same problem. The guidance image network Zero123++ is trained on 3D shapes, but for a single object. The introduced method uses a series of 2 double-step optimizations. The first stage optimizes the diffusion-enriched CLIP space of the input view and the second stage does the same for the target view. The results show that the proposed method has better performance in comparison to previous works. It tends to be more consistent in showing both central object and background from a variety of viewpoints (which are specified in terms of azimuth and elevation angles).
weaknesses: 1. The method is quite convoluted with 4 different test-time optimization steps. No time comparisons are provided. It would be important to mention typical in inference times... 

2. Looking into the intermediate and final results, as well as the method, the main task of aligning the camera appears to be the result of the Zero123++ method, while the test-time optimizations generally improves the image quality. This is also reflected in the particular choice of angles, as only those angles are trained for in Zero123++. However, for small angles (30,30), often the test-time optimization worsens the results, e.g., figure 6, angles = (30,30) for cats. Same for figure 5, angles = (30,30)- the house changes completely after optimization. Due to this reason, it may be a bit dubious to stipulate object + background novel view synthesis as contribution. After the evaluations, both the contributions (1) & (2) could be attributed - at least in part - to Zero123++.
 
3. The architecture diagram in Figure 2 is not complete in the info provided. Missing: which losses are used? These should be made clear together with the equations (1,2) since this is the crux of the proposed method.

4. L215--L216- "Our model eliminates the need for training data, 3D data or multi-view data". This is somewhat misleading. The camera alignment largely appears to be performed by the guidance method Zero123++, which is trained on Objaverse and more importantly on  the same task. The method implicitly uses Objaverse via Zero123++, a competing method used in the proposed method. It is not clear how Zero123++ performs test-time optimization without the 3D data.

5. The paper glosses over some of the limitations in the results, like the house not being shown in a very consistent fashion when viewpoint is changed. Probably there is a difference in performance between renowned monuments with many photographs that can be scraped from the Internet, compared to rather unique objects / situations. 

6. Given the complete approach, the novelty of the work mainly consists of the combination of methods, whereas the actual novelty drags in the cost of extensive inference optimization. 

7. The English could benefit from some slight polishing at times

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: Minor:
In Tab.1 and Tab.2, since the author reports the LPIPS metric, I believe the commonly used PSNR and SSIM metrics can also be reported.

The reviewer suggests that the author revise the writing, conduct more thorough experiments, and propose a clearer explanation of motivation and method.
soundness: 2
strengths: 1. The motivation for training-free novel view synthesis is good.
2. The analysis of the current NVS model’s limitation is reasonable.
summary: This paper argues that current NVS methods are limited to single-object-centric scenes and struggle with complex environments. Moreover, they require extensive 3D data for training. Therefore, this paper proposes a training-free method to combine the benefits of 3D-free and 3D-based methods. It leverages pretrained NVS models for weak guidance and integrates the knowledge into a 3D-free approach to achieve the desired results.
weaknesses: 1. The writing is poor and many sentences are extremely confusing without any support. I will list several examples below and there are more similar problems in the submitted paper.
    
    In Line 265. “Explain the subset you used for your results, how you created the,….”. What does the author want to express here?
    
    In section 3.1, why does Fig.3 illustrate these findings? I believe the findings here are that CLIP struggles with 3D comprehension, but I do not see any examples in Fig.3 supporting this viewpoint. Moreover, if CLIP itself can control camera pose change, there is no need for training on 3D-aware data pairs. I do not see the necessity of including this discussion.
    
    In Line 146. “validates the necessity of 3D guidance images.” What is 3D guidance images? Is it a novel view image based on the current image?
    
    In Line 159. “This indicates that the model benefits from the information in the 3D-derived guidance image, and that the guidance image is not simply a source of extra variance.” What does the author want to express here? What is the 3D-derived guidance image? Does it also mean a novel view image? If so, what does the author mean by “not simply a source of extra variance”? What is the variance? From Fig.4, I guess the author is conducting an experiment on using different novel view images as weak guidance while prompting the generation of novel view images using the proposed training-free method. The reviewer thinks using the right guidance image is quite trivial. 
    
2. The performance improvement is weak. From the visualized comparison in Fig.5, I don’t see much difference between Zero123++ and Ours. Is the camera view control more precise or is the image more realistic? From the waterfall example, I think the result of Zero123++ is more realistic. 
3. The novelty is weak and the method is not well explained. I think a training-free method is a good motivation. But the author still requires heavily on a pretrained NVS model. 
4. Related work is not well discussed. To cope with the complexity of real scene instead of object-level, many works have been proposed and the author should include them. I will list several below.

[1] ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image, CVPR 2024

[2] MegaScenes: Scene-Level View Synthesis at Scale, ECCV 2024

[3] Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis, ECCV 2024

Meta-review: The paper addresses the problem of scene-level novel view synthesis (given an image and a new camera position, generate a new image). The paper proposes to combine a "3d-free" method which uses CLIP and a image guidance with a "3d prior", an existing model which has already been trained to do novel view synthesis (though on objects). The paper demonstrates strong results on the HawkI dataset.

The strength of the paper is its quantitative performance on HawkI dataset. The weakness are numerous:
1) The evaluation on this HawkI dataset for scene-level 3d is really abnormal. It avoids many more standard datasets such as skipping over many more standard dataset such as CO3D, RE10K, Acid or DTU.
2) The comparisons used in the papers are weak and the rebuttal is literally wrong. As many reviewers mentioned, ZeroNVS should be a standard comparison. In the rebuttal the authors state that "generating a single camera-controlled image was 7,500 seconds for ZeroNVS". This is inaccurate. ZeroNVS follows the architecture of Zero123 which takes 2 seconds on a RTX A6000 GPU. It's possible that the NeRF optimization was included but this is not necessary for a single image sample and other papers, e.g. MegaScenes have disentangled the diffusion inference from the NeRF optimization in order to compare directly with the NVS properties.
3) The writing and method are confusing. The paper repeatedly says that their method does not use 3D data. But they directly use Zero123+ which is trained on 3D data. In the rebuttal, they clarify and say that they don't use "additional" 3D data and that Zero123++ is object-level. While this is true, it runs completely against how the current intro is written. Even a one of the more positive reviewers (hAuB) notes this in their weakness (Weakness 4).
4) Even the qualitative performance on HawkI seems to lack image consistency.

I suggest rejection. For future submissions, the authors should use standard datasets and compare against the SOTA or near SOTA for these datasets (e.g. MegaScenes would be an approximate example).

Decision: Reject