GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping
Authors: Junyoung Seo
Kazumi Fukuda
Takashi Shibuya
Takuya Narihira
Naoki Murata
Shoukang Hu
Chieh-Hsin Lai
Seungryong Kim
Yuki Mitsufuji
Venue: NeurIPS 2024 poster (2024)
Keywords: Single-shot Novel View Generation
Generative Models
Novel View Generation
Diffusion Models
Primary area: generative_models

Abstract: Generating novel views from a single image remains a challenging task due to the complexity of 3D scenes and the limited diversity in the existing multi-view datasets to train a model on. Recent research combining large-scale text-to-image (T2I) models with monocular depth estimation (MDE) has shown promise in handling in-the-wild images. In these methods, an input view is geometrically warped to novel views with estimated depth maps, then the warped image is inpainted by T2I models. However, they struggle with noisy depth maps and loss of semantic details when warping an input view to novel viewpoints. In this paper, we propose a novel approach for single-shot novel view synthesis, a semantic-preserving generative warping framework that enables T2I generative models to learn where to warp and where to generate, through augmenting cross-view attention with self-attention. Our approach addresses the limitations of existing methods by conditioning the generative model on source view images and incorporating geometric warping signals. Qualitative and quantitative evaluations demonstrate that our model outperforms existing methods in both in-domain and out-of-domain scenarios. Project page is available at https://GenWarp-NVS.github.io.

Gen WanWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping
Junyoung Seo Kazumi Fukuda'
Takashi Shibuya'
Takuya Narihira' Naoki Murata'
Shoukang Hu'
Chieh-Hsin Lai'
Seungryong Kim
Yuki Mitsufuji 1.2
Sonn AI
Sony Group Corporation
KAIST AI
Generated View
Input View
Generated View
In- In-Domain
go straight, turn left
go straight, turn right
Out-of-Domain
Figure 1: Teaser. Our model generates plausible novel views, conditioned on only single input view, enabling handle both in-domain images (top) and out-of-domain images (bottom).
Abstract
Generating novel views from single image remains a challenging task due to the complexity of 3D scenes and the limited diversity in the existing multi-view datasets to train model on. Recent research combining large-scale text-to-image (T2I) models with monocular depth estimation (MDE) has shown promise handling in-the-wild images. In these methods, an input view is geometrically warped to novel views with estimated depth maps, then the warped image inpainted by T2I models. However, they struggle with noisy depth maps and loss of semantic details when warping an input view to novel viewpoints. this paper, propose novel approach for single-shot novel view synthesis, generative warping framework that enables T2I generative models learn where warp and where generate, through augmenting cross- view attention with self-attention. Our approach addresses the limitations existing methods by conditioning the generative model on source view images and incorporating geometric warping signals. Qualitative and quantitative evaluations demonstrate that model outperforms existing methods in both in-domain and out-of-domain scenarios. Project page available at /atttts://Geaii //ap://////WWW github.
"Work done during internship at Sony AI. Co-corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Text-lo-image (T2I) diffusion models (e.g., Stable Diffusion [35] have made rapid progress in generating diverse high-quality images when given a user text prompt. This holds extensive potential utility across various domains, including portrait photo design, cartoon creation, and movie production. However, current T2I models lack the flexibility of moving cameras in the generated image. For example, when user tries to move the camera closer or farther to the generated image, T2I models often fail to change the viewpoint of the generated image with a proper notion of 3D awareness. This limits its application in real-world scenarios where we hope to achieve user-tailored designing purposes by changing the camera viewpoint for generated images.
To freely move camera viewpoints of an image, a line of research focuses on directly learning a single-shot novel view generation model with camera viewpoint condition from large-scale 3D datasets. For example, with the advent of large-scale 3D object datasets such as Objaverse [10], recent attempts 226 40 27 achieve success in generating novel views of 3D objects from a single image.
Beyond the object-centric novel views, efforts for full 3D scenes have also been made 14 22 36 17 77. Unlike the abject-centric models, these can generate novel views of complex scenes from single image. The performance of single-shot 3D scene novel view generation models highly depends on the scale of multi-view 3D scene datasets [52 g 5 Compared with the object-centric multi-view datasets [10 (9. it is hard to collect such a large-scale dataset for 3D scenes due to its complexity.
Thus, existing models [34 22 36 7
solely trained on these datasets [22 5
struggle to handle
in-the-wild images 17 36).
Instead of learning dataset-specific novel view synthesis models, alternative approaches propose utilizing the generative prior from large-scale T2I diffusion models, e.g., Stable Diffusion
7
33
These works adopt a two-step strategy for novel view generation, called warping-ann and -aa--aanddiiiine int-iinpainnn....
similarly to conventional works [48] 34 22), with a combination of the large-scale T21 diffusion
models and off-the-shelf monocular depth estimation (MDE) models (e.g.. MiDaS (33] ZoeDepth (2)
Specifically, they first predict depth map of given image via off-the-shelf MDE models 2 33
and then warp the input image to novel camera viewpoints with the depth-based correspondence, followed by inpainting occluded regions of the warped images with proper text prompts through the T2I diffusion models. The ieepinn-and-inpainie approach successfully generates novel views from in-the-will wild images by utilizing large capabilities of T2I diffusion models learned from large-scale image datasets (39].
Despite such an advantage, this warping- -anhiin---n-iiii and-aanddiinn inpaaininn approach can generate novel views only in a limited range of camera viewpoints around the input image. This is because (1) they struggle to a). reprojection error from the estimated depth map makes the warped image unreliable, becoming a significant performance bottleneck. The subsequent inpainting T21 diffusion models cannot refine the artifacts caused by this error. In addition, (2) important semantic details of the input view sometimes get lost during geometric warping, especially when dealing with challenging camera viewpoints. In the above two cases, only a sparse set of pixels is preserved in the warped image, making it difficult to generate the occluded regions while preserving the semantic information of the input view. Fig, Zb) shows clear example of this problem; the inpainted regions show a different context with the input view.
handle noisy depth maps predicted by the MDE. As shown in Fig. 2
To address these issues, we propose generative warping framework, Gen Warp, in which we make T21 generative models learn where to warp and where to generate images, instead of inpainting unreliable warped images. Our generative model integrates view warping and occlusion inpainting into a unified process, unlike existing two-step approaches that perform these operations separately. By directly taking the input view images with their estimated depth maps, our model learns to warp them and to generate the occluded or ill-warped parts with our augmented self-attention. Our approach eliminates the dependency
unreliable warped images and integrates semantic features from the source view, preserving the semantic details of the source view during generation. Similar to 226 40 49), leverage generalization capabilities of T2I diffusion models (e.g., Stable Diffusion 35 through fine- tee-tuning a T2I diffusion model.
Our main contributions are as follows:
We propose a comantic-preservvee generative warping framework, GenWarp, to generate high-quality novel views from single image.
2

---

Input View
Warped View
Inpainted
GenWarp (Ours)
WISH YOU
WISH YOU WERE
THE
WERE HER
(a] lll-warped case caused by noisy MDE depth
(b) missing original semantics severely occluded regions
aappin-anddinpaintinn approach [35 7 31. Results from challenging camera viewpoints for harping-and-inpainne neon sign present in the input view is distorted after geometric warping due to the noisy depth. (b) The next room peeked in from the new camera viewpoint lacks the context given by the input view.
Figure 2: Limitations of explicit
approach show artifacts. (a) The
Gen Warp learns where to warp and where to generate in images through augmenting self-attention with cross-view attention instead of inpainting unreliable warped images, which eliminates the artifacts caused by error depths and integrates semantic features from source views, preserving semantic details in generation.
Extensive experiments on RealEstate 10K [52) ScanNet [ and in-the-wild images (e.g.. Al-generated images) validate that Gen Warp achieves superior performances over existing scenarios.
methods in both in-domain and
Related Work
Generating novel views from single image is challenging ill-posed problem that has primarily been addressed in combination with generative modeling. These novel view generative models can generally be categorized into two types: those designed for object-centric scenes and those designed for general scenes, including indoor and outdoor scenes. On the other hand, following the recent success of large-scale T2I models, there are methods that can control the generation results, exploiting the attention mechanism.
Single-shot novel view synthesis for objects. Following the success image diffusion models [15] 35). diffusion models for novel view synthesis 47 4 have been proposed. These works train diffusion models take a single image and novel camera viewpoint as conditions, and directly generate novel view images. More recently, with the emergence of large-scale 3D datasets such as Objaverse [10 9. generalized generative models for single-shot novel view synthesis (NVS) have emerged. Recent works (26 27 19), including Zero123 (26). achieves powerful generalization capability by fine-tuning T2I diffusion models on Objaverse. While these models enable object-centric novel view synthesis from in-the-ue wild single image, such generalized novel view models for general scenes remain relatively unexplored.
Single-shot novel view synthesis for general scene. Single-shot NVS often necessitates generating outer regions or occluded regions that are not visible in an input view. Thus, recent works 48 34 25 22 propose generating novel views a
fashion, which involves first predicting a depth map of an input view, then warping the input view along with the depth map to desired viewpoint, and finally refining missing regions arising from the geometric warping. Another line of
works [36.] 44 [7] directly train novel view generative models without depth-based warping. For

---

example, GeoGPT [36] achieves novel view generation, by feed neewwforwrdiing an input view and camera viewpoint a transformer-basee architecture. Other recent works 44 [7 train novel view diffusion models with a cross-view attentions [44 17 an epipolar constraint 44 More recently, some works [31 41 have proposed bringing a large-scale T21 model [35 to the warping-and
refining strategy. This approach enables the generation of novel views from in-the-wild images, which was previously challenging. Nonetheless, it shows unstable results, especially when the camera viewpoint is far from its original position. Concurrently, ZeroNVS (38) fine-tunes Zero (26) for NVS in general scenes. It focuses on camera parametrization to avoid 3D scale ambiguity, which differs from our focus; we focus improving depth warping- sarping-based NVS.
Attention- based control in large-scale T21 models. Since the emergence of large-scale text-to- image (T2I) diffusion models B55 37.. recent works 42 [3] [4]
16) have investigated the properties of self-attention within T2I models. For example, Text2 Zero 211 and MVDream 40 generate consistent images by sharing self-attention between video frames or 3D multi-views, respectively. Similarly. Animate-Anyone [16] and MagicAnimate [9] generate human dance videos through fine- tuned T2I model that shares self-attention with input image features. Observing the generalization capability and efficiency of these self-attention-basee -based controllable architectures, our approach is highly influenced by them. However, using these architectures for single-shot novel view generation is non-trivial, as input scenes usually complex, and the details within them must be generated consistently with the camera movements. Our model integrates MDE depth- correspondence while benefiting from the advantages of these architectures, thereby significantly improving single- shot NVS performance.
3 Method
3.1
Preliminaries and problem statement
Given a single image for an input view I;, our goal is to generate a novel view 1; from a relative camera viewpoint P.- and a camera intrinsic K. To enable this, recent works 7 31 propose warping-and-lin and-inpainting framework that adopts monocular depth estimation (MDE) models [2] for geometric warping and T2I generative models [35 [7] for inpainting. In this approach, an input view image I, is geometrically warped with its MDE depth map D; to the desired camera viewpoint P;+ji
warp (arp D., Pi K),
(1)
where warp(-) is a geometric warping function which unprojects pixels of an input view image I; with its depth map D; to 3D space, and reprojects them based on the desired camera conditions, Pa-jj and K. More specifically, in homogeneous coordinates, pixel location x; in the input view is transformed to the pixel location z in the novel view such that:
T. KP KP:-,,,(a..KK
(2)
where the projected coordinate z, is continuous value. To obtain the warped image Lwarp it is followed by mapping the pixel colors from the input view to the novel view with a flow field between I; and x, 48] 30 34]. And, inpainting diffusion model o generates novel view image by filling the missing regions in the warped image Twarp:
I, ~ A~palli: Twarps Mwarps Cj);
where (-) is a learned distribution of the diffusion model 6 conditioned on an occlusion mask M Mwarp, text prompt Cj. and the warped image I warp used for inpainting. This approach assumes that the estimated depth map D, is accurate. Under the assumption, the warped image Iwarp would be an ideal guidance for generation.
However, we have observed that the warped image is often not reliable when novel camera viewpoint is far from the original viewpoint. It is because this explicit warping operation is sensitive to errors in the depth map, and depth maps predicted by MDE are usually noisy, arising artifacts after the warping. The subsequent inpainting model only takes the warped image warp as input which contains ill-warped artifacts outside the region be inpainted, thus showing limited performance large view changes. Additionally, the warped image may lose the semantic information originally contained in the input view due factors such as occlusion, but this approach does not take that into account, as exemplified in Fig. 2

---

Canonical 2D Coords,
Semantic Preserver
Input View
Novel View
Input View Features
Camera Pose
Feature
Kay
Query
Generated
Novel View
Novel View
Feature
Original Self-Attn.
Implicit Depth
MDE based Warping
Map
Corr.
Warped
Gaussian
Pretrained
Aggregation
2D Coords.
Noise
Diffusion
Figure 3: Method overview: (Leftt) Given an input view and a desired camera viewpoint, we obtain a pair of embeddings: 2D coordinate embedding for the input view, and warped coordinate embedding for the novel view from estimated depth through MDE. With these embeddings, a semantic preserver network produces semantic feature of the input view, and diffusion model conditioned on them learns to conduct geometric warping to generate novel views. (Right) We augment self-attention with cross-view attention, followed by aggregating the features with both attentions at once. It helps the model to consider where to generate and where to warp
3.2 Semantic-preserving generative warping
To alleviate the aforementioned limitations. we introduce novel approach where diffusion model learns to implicitly conduct geometric warping operation, instead of warping the pixels or the features directly. We design the model to interactively compensate for the ill-warped regions during its generation process, thereby preventing artifacts typically caused by explicit warping. In addition, to preserve the semantics in the input view, our framework takes the input view image without warping, and the encoded semantic features of the input view are incorporated into the generation process, which is different from other approaches that solely take an unreliable warped image from which the original semantics are difficult to infer.
To this end, we leverage the attention layers inside the pre-trained diffusion U-net 35 Our key idea is to learn the attention between input view and novel view features, which serves as an implicit correspondence that mimics explicit depth-based warping within the diffusion model. By incorporating this into the diffusion process, we aim to seamlessly integrate the effect of depth-based warping into the generative prior. This implicit correspondence in the form of attention can be integrated into the existing self-attention layers inside the diffusion U-net. In so doing, the input view features additionally interact with the novel view features in the generation process, making the diffusion models naturally find where to generate and where to warp, visualized in Fig.
Two-stream architecture. Our approach comprises two-stream architecture, semantic preserver network and diffusion model, sharing an identical U-net-based architecture. The semantic preserver network takes the input view image 1; and produces semantic feature F; of the input view. And, the diffusion model generates a novel view image I,, by integrating the input view feature F; into its internal novel view feature Fj. To imbue the diffusion model with the MDE depth-based correspon- dence, we use pair of canonical coordinates and warped coordinates as additional conditions. Fig. illustrates an overview of our architecture. In the following, we explain each component in detail.
Warped coordinate embedding. To condition the MDE depth-based correspondence, two coordinate embeddings, canonical coordinate embedding for the input view, and a warped coordinate embedding for the novel view. We are motivated to the warped coordinate embedding by (29,, whose purpose correspondence-bat concept the geometric warping for novel view generation.
appearance manipulation. Here, we extend this
coordinate map X R where each value normalized between -1 and 1. This 2D coordinate map is transformed by positional encoding function 7 into Fourier features 43 C; = (X). We use this Fourier feature map C; coordinate embedding for the input view I, We then geometrically warp this coordinate embedding C2 of the
Specifically, we construct a canonical
5

---

Augmented Self Attn.
Input View
Generated Novel View
Cross View Attn. Aross
Self Attn. Ageis
Figure 4: Visualization of augmented self-attention map. In augmented self-attention map A, the original self-attention part Aselt is more attentive to regions requiring generative priors, such as occluded ill-warped areas (top), while the concatenated cross-view attention part Aeross focuses on regions that can be reliably warped from the input view (bottom). By aggregating both attentions once, the model naturally determines which regions generate and which warp.
input view to the desired novel viewpoint P:-aj:
Cj warp(C;; D;, Pi-+j; K),
where warp(.) is the same geometric warping function in Eq. [q. The warped coordinate embedding Cj serves as the coordinate embedding for the novel view 1j. These coordinate embedding C2 for the input view and Cj for the novel view are added to the source view feature F, and the target view feature F3 through convolution layers respectively. This embedding strategy guides the model to follow the geometric correlation between the input view and the novel view. In an explicit warping strategy, the process is inherently affected by depth estimation errors in virtue of regarding the warped image as reliably given condition. However, by implicitly learning to warp with this embedding, expected that the influence of these emrors can be mitigated.
Augmenting self-attention with cross-view attention. To infuse the input view features F,, we first construct cross-view attention, where the cross-view attention map represents the similarities between the input view and the novel view being generated. Thanks to the coordinate embeddings, the cross-view attention map learns to give depth-based correspondence that can be absorbed into the generative process. Then, we propose concatenating this cross-view attention to the existing self-attention.
Specifically, we concatenate the keys and values of self-attention layers in the diffusion U-net with the input view features F,, and apply the self-attention 45 with the following query, key, and value:
Fj,
k = F3..
[F, Fy).
where F; is the input view feature in the semantic preserver network, and Fj the novel view feature in the diffusion U-net. Then concatenation of the cross-view attention map Acroes and the self-attention map Ageli.
obtain the augmented self-attention map A, which is
By aggregating the values with both attentions at once, the model learns to balance the contributions from the novel view's s self-attention Agelr and the cross-view attention Accoss- Our intuition behind this design is that the original self-attention Agelf in the diffusion U-net attends to the generative prior, and the cross-view attention Across attends to the warping prior from the input view. This allows the model to inherently decide which regions should rely more its generative capability and which areas should depend primarily on the information from the input view warping, as shown in Fig.
3.3 Training strategy
Fine-tuning pretrained text-to-image diffusion models. We leverage the pretrained Stable Dif- fusion 1.5 model (35) for both diffusion U-net and semantie preserver network, inherit its gener- alization capability. Different from Stable Diffusion which takes text prompt embedding through

---

Ingut View
GenWarp [Ours]
SD SD-Inpainting
Input View
GenWarp (Ours)
SD SD-Inpainting
Figure 5: Qualitative results with images in the wild. We compare our method with Stable Diffusion
Inpainting 35 on in-the-wild images. More qualitatvie results can be found in Fig. Tf Appendix.
Figure 6: Consistent view generation results. Our model generate consistent multiple views by taking pre-generated novel views as inputs.
our model takes an image and a desired camera viewpoint as inputs. Therefore, we replace text condition needed for Stable Diffusion with image embedding of the input image through a CLIP image encoder. As all the components in our framework can be trained in an end-to-end manner, we sole training loss for fine-tuning, which is the same the original training loss in LDM [35). Given a dataset X' consisting of pairs of source view image 7, target view image Ij. their camera information Pa-jj and depth map D,, first encode the source view I, and the target view image Ij to their corresponding latents 2, and zj through the LDM encoder, respectively. Then the model is fine-tuned using the following loss function:
CLIP 32
Lours (0, vb)
Ex,,,e [||e e0. (2j.ti Zá, Di, P.-jj K)
(6)
where 2z.t denotes a noised latent of æj at diffusion timestep t. (.) is our model including the diffusion U-net 0 and the semantic preserver network vb which predicts the added noise in the diffusion process.
Data preparation. We fine-tune the model on multi-view datasets including indoor scene and outdoor scene, ie.,, RealEstate 10K [52]. ScanNet [, ACID (25). Specifically, we sample two consecutive frames at intervals of 30-120 frames make pairs of source view and target view
images. For ScanNet [8]. we use provided ground-truth depth maps and the camera information. For RealEstatc1OK (52) and ACID (25), ground-truth depth maps are not provided. So, we pre-process the datasets to generate pseudo ground-truth depth maps and their corresponding camera information. Specifically, we use DUS(3R [46] as a pair-depth estimator, followed by PuP-RANSAC [33 23 to find the corresponding camera information aligned with the estimated depth maps. Additionally, we exclude pairs with low-confident depth maps in our training dataset. Note that our model is less affected by 3D scale ambiguity 66 38 in this procedure, camera parameters aligned to the scales of estimated depth maps.
4 Experiments
4.1
Experimental setup
We train our model on multiple datasets, including indoor RealEEttate1OK [52] ScanNet [8] and outdoor ACID (25 datasets. For fair quantitative and qualitative comparison with baseline meth- ods [36 17 trained on RealEstate 10K [52), we also prepared version of our model trained on the same single dataset. Our baseline methods include GeoGPT (36), Photometrie-NVS [[7], and the
7

---

GenWarp (Ours)
SD-Inpainting (finetuned)
Photometric- NVS
GeoGPT w/depth
Input View
SD-Inpainting
Pran
Figure 7: Qualitative comparisons with baseline methods 33
36.. We present single-shot
novel view generation results with large viewpoint changes on Rsaaee 10K [52] test set. Our Gen Camparp generates high-quality novel views consistent with the input views. We also provide qualitative results ScanNet g in Fig. 9 of Appendix.
warping and-inaiiniin method 7 31] using the Stable Diffusion Inpainting model [35] To ensure
fair evaluation, we also provide results the Stable Diffusion Inpainting model fine-tuned on the same multi-view dataset [52]. For GeoGPT,
compare with the results from the depth conditional setting, which is most similar to our approach. For the methods that utilize depth information, namely, method, GeoGPT, and Stable Diffusion Inpainting, we use the same monocular depth estimation
models 2 46
Please refer to Appendix B for additional details.
4.2 Qualitative results
Qualitative results on in-the-wild images. Fig. and Fig. 6 show qualitative results on in-the-wild images, e.g., cartoonish pictures, real photos, and Al-generated images. SD-Inpainting [35]-based aogping-and-ippannig approach, used in recent works [1]. shows reasonable results with good generalization capability for extrapolation, but ill- ill-warped artifacts exist in some areas. In contrast, our method consistently generates feasible novel views by refining those artifacts well.
Qualitative comparisons. We present qualitative comparisons with the baseline methods 35 17
36) in Fig. 7 'Te warping-and-inpaintin approaches with SD-Inpainting model [35] show good performance for areas where the input view and novel view clearly overlap. However, for regions where warped pixels are sparse, generates inconsistent novel views without considering the semantic information of the input view. Photometric-NVS 17 and GeoGPT [36] show reasonable performance when the camera view changes are small. However, their performance degrades when the view change is large when the given images of scenes underrepresented in the training data, such as outdoor scenes. Our method generates plausible novel views and robust to variations the type of scenes and camera viewpoints, by considering the semanties of the input views.
4.3 Quantitative results
We perform a quantitative comparison of our model and baseline models 35 RealEstate 10K [52] on the test set of RealEstate 10K (in-domain) and ScanNet g out-of--omain using FID for generation quality on distribution level and PSNR for reconstruction quality, with 1,000 generated images. We categorize the distance between source and target views into mid range (30-60 frames) and long range (60- frames). Tab. demonsttates that our method shows superior performance in both out-of-domain setting and in-domain setting. The SD-Inpainting-basse approaches perform well terms of PSNR thanks to explicit warping, struggle with ill-warped artifacts resulting in poor FID. GeoGPT shows good generation quality evidenced by its FID score
36
trained on

---

Out-of-domain
In-domain 52
Methods
Mid range
Mid range
Long range
FID 4
PSNR
FID
PSNR
FID
PSNR
GeoGPT [36] w/depth
85.52
11.36
32.70
12.26
33.91
11.69
Photometric-Ne
N/A
N/A
37.17
12.05
39.93
11.63
SD-Inpainting
85
52.20
11.68
41.76
14.21
44.13
12.98
SD- Inpainting
35
(finc-tuned)
72.90
9.10
39.17
14.35
43.08
13.10
GenWarp (Ours)
46.03
12.95
31.10
14.55
32.40
13.55
17 and Quantitative comparisons. We compare our method with novel view generative models 36
Table 1:
approach consisting of Stable Diffusion Inpainting 311,, on
in-domain setting (training dataset [52], and out-of-domail setting (external dataset [8). We
additionally provide results of Stable Diffusion Inpainting fine-tuned on the multi-view dataset [52)
but tends disregard input view details, leading to poor PSNR. In the out-of-domain setting, the SD-Inpainting shows reasonable performance, but its performance deteriorates after fine-tuning multi-view dataset [52]. For Photometric-NVS [77, we exclude out-of-domain results as its
provided model trained on RealEstate [52] fails to generate novel views when camera parameters
in ScanNet g are given.
4.4 Ablation study
Embeddings for warping signal. We perform an abla- tion study on the embeddings used create the warping signal by geometric warping with MDE depth maps. We compare the warped coordinate embedding to other possi- ble candidates: warped depth and warped image. Addition- ally, we test the camera embedding as condition. Specif- ically, we encode the camera viewpoint to a Plücker ray representation [42] and replace the warped coordinate em- bedding with this camera embedding. As shown in Tab. coordinate embedding is the most effective among them.
Conditions
FID +
Warped coordinates
32.40
Warped depth map Warped image
34.17
35.27
Camera embedding
39.10
Table Ablation on embeddings.
the warping signal given by the warped
Camera viewpoint variations. Fig. 8 illustrates the relationship between the difficulty of camera viewpoint
GenWarp (Ours)
SD-Inpainting
changes and the degree of distortion in the generated novel views. We adopt the analysis of view changes proposed
SD-Impainting (fn..)
GeoGPT
Photometric-NVS
in (36. using the LPIPS [5] metric between GT source and target views as a proxy for viewpoint change difficulty, and the LPIPS between the generated and GT target views as measure of distortion. As shown in Fig. S our method achieves the least distortion compared the baseline meth- ods. Inpainting-base methods (35) show the second-best performance when the viewpoint change not large, but GeoGPT (36) shows better performance in the case of ex- treme viewpoint changes.
Distortion
65
63-
63
Difficulty
Figure 8: Comparison various viewpoint changes.
5 Conclusion
We have proposed GenWarp, framework for generation of novel views from a single image, preserving semanties contained the input view by learning to warp images through generative process. By augmenting the self-attention in diffusion models with cross-view attention conditioned on the warping signal, our approach learns to preserve semantics of the input view while naturally determining where to warp and where to generate. Extensive experiments demonstrate that Gen Warp generates hugher-quality novel views compared existing methods, especially for challenging viewpoint changes, while exhibiting generalization capability out-of-domain images.

---

Societal Impacts
This paper presents the field of AIGC (AI-Generated Content). The proposed model in the paper generates images of user-provided camera viewpoints based on input images. Therefore, while there may be potential social impacts as consequence, there is nothing in particular to be highlighted. Our model relies on learning from large-scale multi-view datasets, so it may reflect potential societal biases included in these datasets.
Acknowledgment
This research was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-1190077) RS-2024-
00509279) and the Culture, Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism RAS-2024-00345025 RS-2023- 00266509, 202220022200663 and National Research Foundation of Korea (RS-2024-0034659))
References
[1] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall--33 pdf. 2(3):8, 2023.
[2] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 12288, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuming-fite free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[4] Eric Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 42 4217-4229 2023.
[5] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint (xxiv:7709006
2017.
[6] David Charatan, Sizhe Li, Andrea Tagliasacchi. and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312. 2023.
[7] Jacyoung Chung, Suyoung Lee, Hycongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free free generation of 3d gaussian splatting scenes. arXiv preprint 1iXx::3311.13366 2023.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated reconstructions of indoor scenes. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828 5839,5839.208 2017.
Matt Deitke, Ruoshi Liu, Matthew Wallingford. Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Objaverse-xl: A universe 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
[10] Matt Deitke, Dustin Schwenk. Jordi Salvador. Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. Ali Farhadi. Objaverse: A universe annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.
[11] Congyue Deng, Chiyu Jiang, Charles Qi. Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20637-20647, 2023.
[12] Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds. arXiv preprint arXiv:2403.2088 2024.
10

---

[13] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 24(6):381-395. 1981.
[14] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention, arXiv preprint arXiv:2312..22133, 2023.
[15] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[16] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-vinee video synthesis for character animation. arXiv preprint arXiv:2311.17117 2023.
[17] Yu Jason, Fereshteh Forghani, Konstantinos G Derpanis, and Marcus A Brubaker. Long-term photometric consistent novel view synthesis with diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pages 7071-7081. IEEE, 2023.
[18] Rasmus Jensen, Anders Dahl, George Vogiatzis. Engin Tola, and Henrik Aanæs. Large scale multi-vie view stereopsis evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406 003,-413,2000 2014.
[19] Yash Kant, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergoy Tulyakov, Igor Gilitschenski, and Aliaksandr Siarohin. Spad: Spatially aware multiview diffusers. arXiv preprint arXiv:2402.0522 2024.
[20] Bernhard Kerbl, Georgios Kopanas. Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[21] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2xideo-zera Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15954-15964, 2023.
[22] Jing Yu Koh, Harsh Agrawal, Dhruv Batra, Richard Tucker, Austin Waters, Honglak Lec, Yinfci Yang. Jason Baldridge, and Peter Anderson. Simple and effective synthesis of indoor 3d scenes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 1169-1178, 2023.
[23] Vincent Lepetit, Francesc Moreno-Noguee and Pascal Fua. Ep p: An accurate o (n) solution to the p p problem. International journal of computer vision, 81:155-166, 2009.
[24] Zhengqi Li, Qianqian Wang. Noah Snavely, and Angjoo Kanazawa. Infinitenature-zero Learning perpetual view generation of natural scenes from single images. In European Conference on Computer Vision, pages 515-534. Springer. 2022.
[25] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature:
Perpetual view generation of natural scenes from a single image. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14458-14467, 2021.
[26] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-10-3: Zero-shot one image to 3d object. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9298-9309, 2023.
[27] Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-consisten images from a single-view image. arXiv preprint arXiv:2309.03453 2023.
[28] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-006, 2021.
[29] Jiteng Mu, Shalimi De Mello, Zhiding Yu, Nuno Vasconcelos, Xiaolong Wang. Jan Kautz, and Sifei Liu. Coordgan: Self-supervised dense correspondences emerge from gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10011-10020, 2022.
[30] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5437-5446. 2020.
[31] Hao Ouyang. Kathryn Heal, Stephen Lombardi, and Tiancheng Sun. Text2immersion: Generative immersive scene with 3d gaussians. arXiv preprint 29224 2023.
11

---

[32] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR, 2021.
[33] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern amalysis and machine intelligence, 44(3):1623 1637, 2020.
[34] Chris Rockwell, David F Fouhey, and Justin Johnson. Pixelsynth: Generating 3d-consistent experience from a single image. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14104-14113, 14113, 2021.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[36] Robin Rombach, Patrick Esser, and Björn Ommer. Geometry- -frec view synthesis: Transformers and no 3d priors. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14356-14366, 2021.
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[38] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Hermmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Fei-Fei, Deqing Sun, et Zeronvs: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310. 17994, 2023.
[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis. Mitchell Wortsman, et al. Sb; An open large-scale dataset for training generation image-text models. Advances in Neural Information Processing Systems, 33:25278-25294 2022.
[40] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
[41] Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth diffusion. arXiv preprint arXiv:2404.07199 2024.
[42] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 1111913-1992221 2021.
[43] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Rea Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems. 33:7537-7547, 2020.
[44] Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang. and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16773-16783, 2023.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[46] Shuzhe Wang, Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. arXiv preprint 211::23122111322. 2023.
[47] Daniel Watson, William Chan, Ricardo Martin-Brualle Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models, arXiv preprint 1xiv:2211000662 2022.
[48] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to- to end view synthesis from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7467-7477, 2020.
12

---

[49] Zhongcong Xu. Jianfeng Zhang, Jun Hao Liew, Hanshu Yan. Jia- Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311.16496 2023.
[50] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 4578 45877 2021.
[51] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, Oliver Wang The unreasonable effectiveness of deep features as a pereeptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586 595, 2018.
[52] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 18Xix 2018.
13

---

Appendix
A
Additional qualitative results
Fig. 9 shows qualitative results on out-of-domain setting, i.c., testing on ScanNct g with our method and We also provide additional qualitative results on
bascline methods [36 35 7 trained on CcalEstate10K 52
in-the-wild images in Fig. 10
GenWarp (Ours)
GeoGPT w/depth
Input View
SD-Inpainting
Figure 9: Extensive gualitative comparisons in out-of-domain setting. We provide qualitative
results of our model trained RealEstate1001 10K [22] the external dataset, ScanNet [
14

---

Input View
Warped
SD-Inpainting
GenWarp (Ours)
Figure 10: Extensive qualitative results on in-the-wild images. We present extensive qualitative results of our method and baseline methods [35]] 7 on the in-the-wild images.
B Additional implementation details
We initialize our two networks, semantic preserver and diffusion U-net, with Stable Diffusion v1.5 35 and fine-tune the networks 2xH100 80GB with batch size of 48 for 2-3 days, resolutions of 512 x 384 and 512 512. Specifically, U-net in an end-to-end manner. All hyper- parameessssssers used our training are kept same as training of Stable Diffusion 1.5 In inference, takes around 2 seconds to generate a novel view with single H100 80GB,
fine-tune the whole parameters of the semantic preserver network and the diffusion
Stable Diffusion v1.5 Model card: http://hhngiigggeecccommammmmmmt hitps//smmmmmssabiiiisisooiis..
15

---

Monocular depth estimation. We use two external depth estimation networks for all qualitative and quantitative results: ZoeDepth (2) and DUS(3 [46]. DUS(3R is model that predicts pointmaps given images a pair. We use the z-values of these pointmaps in two ways: a pair depth estimation during training and as monocular depth estimation during inference (by using the same image for both input views). For quantitative evaluation, we use DUSt3R for depth prediction as the predicted depth maps are passed through the same normalization in the process of DUSt3R with the pseudo depth pairs in training dataset which estimated using the Note that we have used the same estimated depth maps our method and all the baseline methods which need depth information.
network. For qualitative comparisons,
ZoeDepth predict metric depth maps.
Reproducing surping-and-inpaintin approach with T21 inpainting models. To implement the warping and-annddaaaa -apainiinn strategy using Stable Diffusion Inpainting [36]. we follow 'Dream' stage of Lucid- Dreamer [7]. which consists of inpainting using the pretrained T21 model [36] after depth-based warping via monocular depth estimation in the official code repository. We observe that directly applying the occlusion mask from depth-hased warping to the Stable Diffusion Inpainting model leads to generation collapsed images. As suggested in the official code of LucidDreamer, increasing the occlusion mask size for occlusions below certain threshold effectively prevents this collapse. However, this approach involves trade-off. as it may further ignore pixels from the source view. Additionally, when using challenging camera trajectories (especially when moving camera forward), artifacts still occur despite this mask filtering, To address this, we set the minimum occlusion size to x 8 and expand smaller occlusions to this size, considering that a resolution of latents in LDM 35 is 8 times lower than that of the images. We use inverse warping for the existing warping and-inpainting method, which provides natural interpolation and reduces occlusion. In contrast, method employs forward warping facilitate the intervention of the generative prior. Fig. [ the difference between forward warping inverse warping, and the obtained occlusion masks which are used in the subsequent inpainting procedure.
Input View
Forward Warping
Inverse Warping
Occlusion Mask
we apply inverse warping and occlusion mask filtering to reproduce the existing warping-and-inpaintn and-inpainting approach 31 with Stable Diffusion Inpainting
Figure 11: Following LucidDreamer 7
C Additional discussion
Explicit feature warping vs. implicit warping (ours). Another straightforward approach for integrating depth-basse based warping into diffusion models to warp features within the diffusion model's 's feature space. We initially tried this diffusion feature warping. Specifically, the input view feature F; from the semantic preserver network is geometrically warped with the corresponding depth map, and then added to the diffusion U-net's intermediate feature F3 through zero-initialized convolution layers.
This approach shows reasonable performance multi-view datasets like ScanNet [8] which include ground- truth sensor depth. However, most multi-view datasets 25 52) are derived from videos and lack dense
depth. To address this, we use estimated depth maps from DUSt3R [46), as deseribed in Sec. 3.3. Although these pseudo depth pairs are useful, they are not highly accurate. Consequently, we found training a model with explicit feature warping using these pseudo depth pairs leads instability, shown in Fig. 12
16

---

Input View
Generated View 3,000 iter.
Generated View 5,000 iter.
Figure 12:
Unstable training of an explicit feature warping model using pseudo depth data.
Additional analysis on viewpoint changes. We analyze how the performance changes as the ratio of pixels invisible from the input view increases due to viewpoint changes. As shown in Fig. T3 and Fig. [4] our methed demonstrates the best performance compared to the other methods even the invisible ratio increases.
LPIPS (distortion)
0.7
LPIPS (distortion)
GenWarp (Ours)
5D In-aiittiing
SD-irpainting
GenWarp (Ours)
GeoGP1
embedding depth camera embedding (Ptücker)
Photometer stric NVS
Ratio of Invisible region 93
04
d' a) as
Ratio of invisible ragion
Figure 13: Comparison of LPIPS with other methods regarding ratio of invisible region. We measure LPIPS between generated views and GT target views, following GeoGPT [36] 's's evaulation protocol.
Figure 14: Additional ablations on view embed- ding. We compare our full pipeline with variant using camera (Plücker) embeddings and another variant using the camera embeddings and depth information.
Comparison on econstrruction-bssed approaches 11
50 28
We provide comparisons with three
additional recent/classical methods (Nerdi [T]] PixeINeRF 550 vanilla NeRF (8) Fig. [S] For Nerdi (TT)
lack of available codes, we brought curated qualitative results on DTU dataset from their paper. Our result shows non-blurry, clear novel view compared to other methods.
Ingut View
NeRF
PixeINeRF
Nerdi
GenWarp (Ours)
Figure 15: Comparison with reconstruction- methods DTU dataset [18]. Note that our model used here is not trained on DTU dataset.
Analysis on cross-view attention. To verify how the cross-view attentions attend to corresponding points, we have used 1, 0000 pairs of images to determine (1) how well cross-attention attends to corresponding points, and (2) which is more dominant between self-attention and cross-attention for invisible regions and regions where depth-basee correspondence exists. First, Tab. 3 shows the distance between flow map obtained from depth information and the flow map extracted from the cross-attentia flow map from the cross-atted atteenttnon layer by argmax operation to see where the model pays the most attention to. It demonstrates that as training progresses, the model learns depth-based based matching and warping through the
Specifically, we extracted the
17

---

cross-attention mechanism. On the other hand, the model where the proposed embedding is replaced with the Plücker camera embedding shows relatively
performance in terms of matching distance.
Secondly, in the Tab. 4 report which part of the concatenated attention map the cross-attention part the self-attention part more activated during generation for visible and invisible regions. exemplified in Fig. 4 shows the cross-view attention part focuses on regions that can be reliably warped from the input view, while the original self- eeff-attetionn part is more attentive to invisible regions requiring generative priors. Regarding the cross-attention and self-attention for invisible regions, empirically found that when generating invisible regions, the model also refers surrounding visible
through cross-attention, for instance, generate the invisible left side desk, needs refer to visible part of desk for plausible novel view.
Models
Average distance
Ours 2,000 steps
1.36
Region
Cross-attn.
Self-attn.
Ours 6,000 steps
0.97
Visible region
0.756
0.244
Ours 10,000 steps
0,90
Invisible region
0.417
0.583
Ours converged
0.85
Camera embed. converged
0.98
Table 4: Attention distribution in visi- ble/invisible regions.
Table Matching distance of models over training steps.
3DGS reconstruction. Our model be applied to various downstream tasks. For example, given single image, our model generates novel view images, followed by feeding such as InstantSplat [12]. Then we can easily obtain a 3DGS scene a few seconds. Video examples are shown in the project page.
3DGS (2) reconstructors
Figure 16: 3DGS reconstruction results. We show extracted frames in the generated 3DGS videos. Video examples shown in the project page.
D Limitations
Given extremely distant camera viewpoints where depth-based correspondence has no influence, ie., beyond the unprojected pixels with the depth map, model struggles with generating novel views. In these cases, instead of generating a novel view single step, the approach of sequentially generating novel views conditioned on pre-generated novel views. similar to other single-shot NVS methods [44] 17 25 24], should be taken. As with other works [26 40 that fine-tune pretrained diffusion models, the quality of the dataset used for fine-tuning affects the model's performance. We believe that high-quality multi-view datasets will maximize the potential of model.
18

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract and Introduction clearly state the motivation and a brief description of the methods used in our paper, along with our contributions. Additionally, at the end of the introduction, provide summary of our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our method are clearly described in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors encouraged create separate" "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to- peech-to-texx text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not involve theoretical results their proof'.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and refseeeefee
19

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Sec. Appendix B we have included the necessary information for reproducibility, such as the network architecture used in our method, which pretrained checkpoint is utilized, which datasets employed, and what data preprocessing steps are performed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) contribution a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
the data and code, with sufficient instructions faithfully reproduce main experimental results, as described supplemental material?
Question: Does the paper provide open
Answer: [Yes]
Justification: We publicly provide model weights and code project page. The datasets used for training are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
guides/ Coae///ddeSubbiiic stte//ooaiaaiiiioooooooioooo ssinnpolioyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
20

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines httpp////nia.ccppuaaaaeeee for more details.
Suades//oodeSubmisssooooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting Deariil
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The implementation details and experimental settings for our method and the baseline
methods are described in Sec. 4.1 Sec. 3.3 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: This task is closer to sampling (generation) within the learned distribution rather than prediction. To be consistent with the experimental analyses of existing works [36] [77. this paper primarily uses distribution distance metrics such FID and does not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be elearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% if the hypothesis of Normality of errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We report the GPU and VRAM used, batch size, and training time in Appendix
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with the NeurIPS Code of Ethics htttps://neuriii
Answer:
[Yes]
Justification:
We have faithfully followed the specified NeurIPS Code of Ethics throughout all our experiments.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that roquire a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We believe that our model, which is generative model trained on publicly available multi-view datascts, does not have high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines restrictions to access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make hest faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected??
Answer: [Yes]
Justification:
We have appropriately cited all the papers corresponding to the datasets we used.
Guidelines:
The answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset,
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: not introduce any new assets in this paper.
Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
23

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
24


Reviews:
[{"review_id": "arYxpgipwa", "reviewer": "Reviewer_2Nrv", "summary": "This paper proposes a novel single-shot novel view synthesis framework based on a pretrained T2I diffusion model. Instead of directly warping pixels between input view and novel view,  an implicit approach is proposed to conduct geometry warping operation.  The cross-view attention is used to eliminates the artifacts caused by error depths and integrates semantic features from source views, preserving semantic details in generation. Extensive experiments prove the efffectiveness of the proposed method.", "questions": "N/A", "limitations": "This method is restricted to depth-based correspondence of two views which limit its application in some scenes where depth correspondence between two views is missing.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The implicit geometry warping approach is effective in addressing ill-warped problem and missing original semantic problem encountered in explicit warpping methods.\n2. The cross-view attention can provide more useful information for novel view generation.\n3. The qualitative and quantatitive experiments on RealEstate10K , ScanNet, and in-the-wild images show the proposed method outperform SOTA methods in both in-domain and out-of-domain scenario.", "weaknesses": "1. Although the cross-view attention strategy is effective to fuse features, it may be not novel enough. There are many similar operation in video generation area.\n2. The detail of finetuning the T2I model is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank your thoughtful review and suggestions! We give a detailed response to your comments below. If any of our responses do not adequately address your concerns, please let us know and we will get back to you as soon as possible.\n\n---\n\n### Q. Novelty regarding cross-view attention.\n\nThank you for the feedback. We agree that cross-view attention itself is not newly proposed in our paper — it has been used in the 3D/video generation area. \n\nHowever, we would like to emphasize that **the novelty of our paper lies not so much in the use of cross-view attention itself, but rather in drawing inspiration from the intuitive connection between cross-view attention and warping operations.**\n\nThis led us **guide the attention modules within the diffusion model to emulate geometric warping**. As Table 2 in the main paper demonstrates that naively using cross-view attention yields limited performance, we have proposed to use warped coordinate grids as positional embeddings to support our generative warping.\n\nOur other strategy is aggregating the cross-view attention with the self-attention in the diffusion model at once, instead of inserting new cross-view attention layers. With this strategy, we found that the self-attention effectively finds where to refine, compensating for warping errors coming from noisy MDE depth, exemplified in Fig.4 of the main paper.\n\nBy doing so, we have overcome the performance constraints inherent in existing two-step approaches that rely on the warping-then-generation paradigm. We believe these findings possess robust merits that contribute to this field.\n\n---\n\n### Q: The detail of finetuning the T2I model.\n\nThank you for pointing this out. As described in L445-449, we initialize our two networks, semantic preserver and diffusion U-net, with Stable Diffusion v1.5 [30], fine-tune the networks on 2×H100 80GB with a batch size of 48, at resolutions of 512 × 384 and 512 × 512. We used a learning rate of 1.0e-5, which is the same as the value used when training Stable Diffusion, and we kept all other hyperparameters at the same values used in Stable Diffusion.\n\nAdditionally, the coordinate embeddings are passed through 3 convolutional layers and added to the input of the diffusion U-Net and the semantic preserver network. All parameters of our model are trained in an end-to-end manner through the diffusion loss shown in Equation 6 of the main paper. We will clarify these points in the camera-ready version.\n\nIf we have not addressed any unclear aspects, please let us know, and we will reply accordingly.\n\n---\n\n### Limitation section.\n\nAs the reviewer mentioned, we discuss the viewpoint limitation due to depth-based correspondence in Limitation section — as this is not the goal of our paper, our method does not explicitly consider this common issue in depth-based methods. \n\nHowever, we would like to remind you that (1) as shown in Figure 8 of the main paper, our method demonstrates superior performance compared to other methods even when viewpoint changes are large, as long as there is at least a small overlap between the views, (2) even for extremely distant viewpoints, multi-step progressive generation by re-conditioning on previously generated novel views can be achieved (L487-L489), exemplified in Figure 6 in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w7LwifPZSv", "reviewer": "Reviewer_3Fuk", "summary": "The paper presents a novel framework called GenWarp, which aims to generate new views from a single input image while preserving the semantic content of the original view. This is achieved by leveraging a generative process that incorporates both self-attention and cross-view attention mechanisms conditioned on warping signals. The proposed approach demonstrates superior performance compared to existing methods, especially in scenarios with challenging viewpoint changes, and exhibits good generalization to out-of-domain images.", "questions": "Please refer to the weakness section.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper introduces an approach for single-shot novel view synthesis by combining self-attention and cross-view attention mechanisms to preserve semantic details. \n2.\tThe proposed method outperforms existing techniques in generating high-quality novel views, particularly for challenging viewpoint changes.\n3.\tGenWarp shows generalization capabilities, performing well on out-of-domain images, which indicates its applicability to various scenarios.", "weaknesses": "1.\tThe paper could further highlight its novelty by providing a more comprehensive comparison with a wider range of state-of-the-art methods, including both classical and recent approaches. Additionally, the authors should discuss any potential limitations of their approach in terms of scalability or adaptability to different types of scenes.\n2.\tThe methodology section could benefit from additional diagrams and flowcharts that illustrate the workflow and attention mechanisms in more detail. Including intermediate results and step-by-step visualizations would help readers better understand the progression from the input image to the generated novel view.\n3.\tWhile the paper provides detailed instructions for reproducing the experiments, the code and data are not made publicly available at the time of submission, which can hinder reproducibility efforts.\n4.\tThe performance of GenWarp is highly dependent on the quality of the datasets used for fine-tuning, which could limit its effectiveness if high-quality multi-view datasets are not available.\n5.\tThe method struggles with generating novel views when the camera viewpoints are extremely distant, indicating a limitation in handling very large viewpoint changes.\n6.\tThe reference list could be updated to include more recent advancements in the field, particularly those published in the last year. Additionally, a more detailed comparative analysis of the strengths and weaknesses of related methods would be beneficial.\n7. what is the advantage of diffusion-model-based pipeline over 3D Gaussian[A,B] or NeRF[C,D] based pipeline, can the author discuss more about existing methods?\n\n[A] Yu, Zehao, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. \"Mip-splatting: Alias-free 3d gaussian splatting.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456. 2024. \n[B] Yan, Zhiwen, Weng Fei Low, Yu Chen, and Gim Hee Lee. \"Multi-scale 3d gaussian splatting for anti-aliased rendering.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20923-20931. 2024.\n[C] Deng, Congyue, Chiyu Jiang, Charles R. Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, and Dragomir Anguelov. \"Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20637-20647. 2023.\n[D] Yang, Yifan, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. \"Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15901-15911. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank your thoughtful review and suggestions! If any of our responses do not adequately address your concerns, please let us know and we will get back to you.\n\n---\n\n### Q: Handling extremely distant viewpoint changes.\n\nThank you for pointing this out. Our key contribution is effectively using estimated noisy depth signals in NVS diffusion models. While addressing depth-based correspondence for *extremly* distant viewpoints is a common issue for depth-based methods and beyond our scope, our approach outperforms others by implicitly using depth warping signals, even with significant viewpoint changes, as shown in Figure 8 of the main paper.\n\nTo further demonstrate this, we analyze how the performance changes as the ratio of pixels invisible from the input view increases due to viewpoint changes. **As shown in Figure A of the global response PDF, our method demonstrates the best performance compared to the other methods even as the invisible ratio increases.**\n\nFinally, we'd like to note that even for extreme viewpoints where depth-based correspondence doesn't exist, progressive generation by re-conditioning on previously generated novel views can be achieved, as similar existing methods[21, 39] have shown. As exemplified in Figure 6 of the main paper, our model also demonstrates robust performance in consistent view generation.\n\nWe will faithfully reflect this discussion in the camera-ready version.\n\n---\n\n### Q: What is advantage of diffusion-based pipleline over 3DGS/NeRF-based pipleline?\n\nConventional 3DGS/NeRF pipelines [A,B,D] aim to perform 3D reconstruction using numerous input views, synthesizing novel views through interpolation between input views. However, these methods struggle to synthesize novel views in few-shot scenarios. In such cases, synthesizing a novel view is closer to generation problem than reconstruction problem.\n\nGeneralizable NeRF/3DGS pipelines [E,F] learn scene priors through training. In few-shot scenarios, these methods show improved reconstruction performance. However, these works do not explicitly consider generative modeling. Although showing superior reconstruction performance, they show limited performance in synthesizing large unseen areas in novel views, e.g., extrapolation. \n\nGenWarp and other recent methods [7,15,22,33,C] using diffusion models formulate single-shot novel view synthesis as a conditional generation problem, rather than a reconstruction-based approach. Consequently, they show superior performance with extremely limited input views, e.g.,  single-shot scenarios.\n\n---\n\n### Q: More comprehensive comparison.\n\nAs suggested by the reviewer, we provide comparisons with three additional recent/classical methods (Nerdi [D], PixelNeRF [E], vanilla NeRF) in Figure C of the global response PDF.  For Nerdi [D], due to lack of available codes, we brought curated qualitative results on DTU dataset from their paper. Our result shows **non-blurry, clear novel view compared to other methods**. As their methods are optimization-based and take several hours per scene, we will thoroughly include quantitative comparisons that cannot be done during the rebuttal phase in the camera ready.\n\nAdditionally, for comparison with recent methods, we would like to note that we have included a comparison with a warping-then-inpainting strategy with inpainting models [30] which is commonly adopted in recent state-of-the-art pipelines [7,26,36] for single-shot 3D generation.\n\n> Potential limitations of others in terms of scalability or adaptability?\n> \n\nFor other NVS generative models [15,31], when these methods are evaluated on different datasets, i.e. in out-of-domain scenarios, they show decreased performance, as shown in Table 1 of the main paper. We speculate that it is because a single dataset usually consists of similar scenes, so the models struggle with different types of scenes unseen during training.\n\nOther approaches [7,26,36] that use warping-then-inpainting with pretrained T2I diffusion models [30], maintain good scalability as they directly utilize the large-scale T2I models without fine-tuning. However, they show unstable results, especially when the target camera viewpoint is far (L106, L49), due to warping errors caused by noisy depths, exemplified in Figure 2 of the main paper. We address these limitations by combining the best of both worlds — GenWarp inherits the generalization capabilities of T2I models while refining the noisy depth-warping artifacts.\n\n---\n\n### Q: Dependent on the quality of datasets.\n\nIndeed, our model’s performance is dependent on dataset quality due to its learning-based nature.\n\nHowever, our model inherits the generalization capabilities of T2I models, which are trained on large, high-quality image corpora. Furthermore, our generative warping approach introduces an inductive bias coming from MDE depth and its warping process, enabling efficient fine-tuning with the training datasets. As a result, it shows superior performance on the same training dataset, as demonstrated by the out-of-domain performance in Table 1 of the main paper.\n\n---\n\n### Q: Additional diagram.\n\nThank you for the feedback. We provide a detailed diagram and intermediate results in Figure E of the global response PDF, which will be included in the camera ready.\n\n---\n\n### Q: While the paper provides detailed instructions for reproduction, code and data are not made publicly available.\n\nThank you for recognizing our effort for reproducibility. We will make sure to release all the code and data in the camera ready.\n\n---\n\n### Q: Reference list update.\n\nWe thank you for the feedback and will thoroughly update our reference list with the papers [A,B,C,D], as well as recent/concurrent papers. We will also supplement our Related Work section accordingly.\n\n---\n\n[E] pixelnerf: Neural radiance fields from one or few images. CVPR. 2021\n\n[F] pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. CVPR. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fOw7OVSPYw", "reviewer": "Reviewer_vdto", "summary": "This paper proposes a semantic-preserving generative warping framework to generate high-quality novel views from a single image, which mainly consists of two components:\n- Condition the novel view synthesis on the warped 2D coords embedding from the estimated depth map.  \n- Augmenting cross-view attention with self-attention.\n\nThe proposed method eliminates the artifacts caused by error depths in the warping-and-inpainting pipeline and integrates semantic features from source views, preserving semantic details in generation.", "questions": "As listed in the Weakness.", "limitations": "No specific limitation and negative societal impact need to be addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tCompared to existing warping-and-inpainting methods that condition on explicit depth warping, the proposed warped 2D coords embedding forms the correspondence between reference view and target view implicitly, which helps the network to be more robust to the noise in the estimated depth map without losing sematic details. \n\n-\tThe paper is well-written, being clear and easy to follow. The technical limitations are also discussed in detail in the Sup.", "weaknesses": "-\tThe proposed embedding based on depth warping could not benefit the synthesize process when there are large camera movements or occlusions between the input view and the target view. \n\n\n-\tAlthough the proposed depth-warping embedding somehow reduces the influence of the noise in the estimated depth map, compared to explicit pixel warping, the implicit condition of depth embedding also lower the local preservation ability of the network when synthesizing novel views. Is the model capable of synthesizing consistent novel views which could be used to reconstruct a 3D scene ? \n\n\n-\tThe comparison baseline is limited. Plucker embedding[1] is also a dense embedding capable of providing local correspondence, which supports large camera movement and occlusions.  The difference between such dense embedding with the proposed depth warping embedding should be further discussed.\n\n\n[1] SPAD : Spatially Aware Multiview Diffusers(CVPR2024)[https://arxiv.org/abs/2402.05235]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank your thoughtful review and suggestions! We give a detailed response to your comments below. If any of our responses do not adequately address your concerns, please let us know and we will get back to you as soon as possible.\n\n---\n\n### Q: The proposed embedding could not benefit the synthesize process when there are occlusions between the input view and the target view.\n\nThank you for pointing this out. As the reviewer noticed, the proposed embedding was designed to enhance implicit warping for co-visible regions between input and target views, rather than specifically addressing occlusion. As shown in Figure 4 of the main paper, where the self-attention part of the diffusion model is more attentive to occluded regions, these occluded areas are generated through the generative prior of the diffusion model while referencing co-visible parts.\n\nIn the rebuttal, to further validate our model’s performance regarding occlusions,  we calculate the ratio of invisible regions in the warping and analyze how performance changes as this ratio increases. Figure A in the global response PDF shows that our method, along with the proposed embedding, demonstrates **better LPIPS values compared to other methods even when the ratio of invisible regions is high**, showing a similar trend in Figure 8 of the main paper.\n\nIn the case of extremely distant viewpoints where depth-based correspondence does not exist, our method, like any other depth warping-based NVS methods[7,31,36], struggles to generate a novel view in a single generation step. We would like to note that in such cases, multi-step progressive generation by re-conditioning on previously generated novel views can be achieved, as similar existing methods [21, 39] have shown (L489).  As exemplified in Figure 6 of the main paper, our model also demonstrates robust performance in consistent view generation.\n\n---\n\n### Q: Comparison with Plücker embedding.\n\nWe appreciate your feedback. **The comparison between Plücker embedding and our proposed embedding is presented in Table 2 of the main paper**, where Camera embedding [37] refers to the Plücker embedding (L282). We will clarify this point in the camera-ready.\n\nFor further comparison, we additionally report the performance comparison of Plücker embedding and our embedding with respect to the ratio of invisible regions. Figure B in the global response PDF shows that the proposed embedding demonstrates better performance consistently as long as there is at least a small overlap between the input view and the target view.\n\n For the reason why the warped coordinate embedding performs better than the Plücker embedding in our setting, we speculate that the model with the proposed embedding could benefit from the inductive bias that MDE depth and its warping process provide. In other words, Plücker embedding has inherent ambiguity relatively, while the warped coordinate embedding provides a direct warping hint. In our opinion, when fine-tuning with the multi-view training datasets that have relatively less diverse data than other types of datasets, our embedding with this inductive bias can be more efficient.\n\nThank you for this feedback and we will continue to investigate this!\n\n---\n\n### Q: Is the model capable of synthesizing consistent novel views which could be used to reconstruct a 3D scene?\n\nThank you for the interesting question. To verify this, we reconstructed 3DGS [A] with novel views generated from our model and rendered a video with a camera trajectory that interpolates the given camera viewpoints. We report the video frames in Figure D of the global response PDF, as we are unable to upload the video in the rebuttal. It demonstrates that the 3D scene converges well without being hindered by artifacts.\n\n---\n\n[A] 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEgIGm9F4S", "reviewer": "Reviewer_Cjbv", "summary": "The authors propose to achieve novel view synthesis from a single image. Contrary to prior work, which warps using a monocular depth estimate and then inpaints, a more flexible architecture is introduced. To do this, a diffusion model is conditioned on warped coordinates, based on the desired pose change and a monocular depth estimate. In addition, the input view (and base coordinate map) is featurized and the diffusion process is allowed to cross attend to these features, which should intuitively allow for attention between corresponding points. The method is then trained end-to-end with a denoising diffusion objective. Experiments show that this approach achieves better FID, PSNR, and LPIPS scores than baselines on RealEstate10k and ScanNet. Ablations justify the choice of warping conditioning.", "questions": "From the weaknesses section: could apples-to-apples ablations be conducted on components of the method?\n\nFrom the weaknesses section: could more evidence for “cross-attention attends to corresponding points” be presented? \n\nOne major benefit of the proposed setting, as opposed to warp then inpaint, is that the method should be able to handle view-dependent effects. This may be interesting to investigate.", "limitations": "Limitations and societal impacts are included, and adequately addressed, but are not in the main body of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The manuscript is well written and well motivated. The related work section is comprehensive, the methods section is straightforward and easy to understand. And experimental results seem quite strong.", "weaknesses": "I think the paper could benefit significantly from apples-to-apples ablations. For example, I am quite interested in a comparison between the proposed method, and the method without any depth information (I think similar to GeoGPT). Or perhaps the proposed method, and the method without the warping.\n\nThe authors also claim that the cross-attention “allows the model to inherently decide which regions should rely more on its generative capability and which areas should depend primarily on the information from the input view warping.” However, they provide only a single example of this in figure 4. I think more examples would make for more convincing evidence. I would be interested in a larger scale, more robust study of this hypothesis. For example, this could possibly be automated by computing how closely the cross-attention actually matches warping.\n\nThe SD warp baseline could possibly be implemented better. I believe the inpainting mask covers non-warped black pixels, resulting in many unrealistic “black borders” is in Figure 10 and Figure 5. Perhaps a more fair and robust baseline could be achieved by just expanding the inpainting mask a bit.\n\nText2room by Höllein et al seems relevant. Perhaps the authors would consider citing it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank your thoughtful review and suggestions! If any of our responses do not adequately address your concerns, please let us know and we will get back to you as soon as possible.\n\n---\n\n### Q: More evidence for “cross-attention attends to corresponding points” be presented?\n\nThank you for the interesting question!  To verify this, we have used 1,000 pairs of images to determine (1) how well cross-attention attends to corresponding points, and (2) which is more dominant between self-attention and cross-attention for invisible regions and regions where depth-based correspondence exists.\n\nFirst, the table below shows the distance between the flow map obtained from depth information and the flow map extracted from the cross-attention. Specifically, we extracted the flow map from the cross-attention layer by argmax operation to see where the model pays the most attention to. It demonstrates that as training progresses, the model learns depth-based matching and warping through the cross-attention mechanism. On the other hand, the model where the proposed embedding is replaced with the Plücker camera embedding shows relatively worse performance in terms of matching distance.\n\n| Models | Average distance |\n| --- | --- |\n| Ours - 2,000 steps | 1.36 |\n| Ours - 6,000 steps | 0.97 |\n| Ours - 10,000 steps | 0.90 |\n| Ours - converged | 0.85 |\n| Camera embed.  - converged | 0.98 |\n\nSecondly, in the table below, we report which part of the concatenated attention map - the cross-attention part or the self-attention part - is more activated during generation for visible and invisible regions. As exemplified in Figure 4 of the main paper, it shows the cross-view attention part focuses on regions that can be reliably warped from the input view, while the original self-attention part is more attentive to invisible regions requiring generative priors.\n\n|  | Cross-attn. | Self-attn. |\n| --- | --- | --- |\n| Visible region | 0.756 | 0.244 |\n| Invisible region | 0.417 | 0.583 |\n\nRegarding the cross-attention and self-attention for invisible regions, we empirically found that when generating invisible regions, the model also refers to surrounding visible areas through cross-attention, for instance, to generate the invisible left side of a desk, it needs to refer to the visible part of the desk for a plausible novel view.  We will add this analysis.\n\n---\n\n### Q: Apples-to-apples ablations.\n\nThank you for the suggestion. We report the additional ablation results for the two cases suggested by the reviewer below.\n\n1. **The method without any depth information**: We have trained our model without depth information and warping process, in which we guided the model with the target camera viewpoint using dense camera embedding, i.e., Plücker embedding.\n2. **The method without the warping**: We have trained the model with depth information of the input view and the camera embedding, but without the warping process.\n\nIn Figure B of the global response PDF, we report a performance comparison between these two cases and our full pipeline. Specifically, we measure LPIPS of each baseline with respect to the ratio of invisible regions in the target camera viewpoint, i.e., the difficulty of generating target views.\n\nIt shows that performance improves in the following order, from best to worst: our full model involving the warping process, the model with both camera and depth information, and the model with camera information only. We appreciate this suggestion and will add it to the camera ready.\n\n---\n\n### Q: Expanding inpainting mask a bit for SD-inpainting baseline.\n\nThank you for pointing this out. In the paper, we followed the masking technique of LucidDreamer[7], applying an 8x8 filter to the mask obtained from the warping process, by expanding the mask up to 8x8 size for mask pixels smaller than 8x8.\n\nActually, at the time of paper submission, we experimented with several mask filter sizes for fair evaluation, and the 8x8 filter achieved the best quantitative results. The table below shows the quantitative results when the mask filter size is increased by 50%.\n\n| Filter size | FID ↓ | PSNR↑ |\n| --- | --- | --- |\n| 8x8 (paper) | 44.13 | 12.98 |\n| 12x12 | 44.19 | 12.88 |\n\nThere is a trade-off in mask filter size for the warping-then-inpainting approach — if the filter size is large, it may further ignore pixels from the source view, while conversely, as the reviewer mentioned, artifacts may persist. For details on this, please refer to Figure 11 and L459-L472 in the Appendix.\n\nWe thank the reviewer for this point. We will include this discussion and improve the qualitative results of the baseline.\n\n---\n\n### Q: Is that the method should be able to handle view-dependent effects? This may be interesting to investigate.\n\nThis is an exciting experiment that we have not investigated! Intuitively, our implicit warping, trained on the multi-view datasets where view-dependent effects exist, should better capture these effects compared to explicit geometric warping. These effects could be measured in datasets containing glossy objects, such as Shiny Blender dataset [A]. We will further investigate this and report it in the camera ready as we cannot report it here due to time limitations during the rebuttal phase.\n\n---\n\n### Q: Citing Text2Room.\n\nThank you for the feedback. We will add the Text2Room citation in the camera-ready.\n\n---\n\n[A] Ref-nerf: Structured view-dependent appearance for neural radiance fields. CVPR. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We would like to first thank the reviewers for the helpful suggestions and constructive reviews. We are greatly encouraged by their positive assessment regarding soundness (1 excellent, 3 good), contribution (4 good), and presentation (2 excellent, 1 good) of our work.

They acknowledge that our generative geometric warping is **effective in addressing ill-warped problems** (2Nrv), **preserving semantic details** (2Nrv,3Fuk), exhibiting **generalization capabilities** (3Fuk), our warped 2D coords embedding helps the network to be **more robust to the noisy predicted depth maps** (vdto), and our manuscript is **well written** (Cjbv, vdto) and **well motivated** (Cjbv).

We also thank the reviewers for recognizing that our **experimental results are strong** (Cjbv), outperforming existing methods in **both in-domain and out-of-domain scenarios** (2Nrv) under **challenging viewpoint changes** (3Fuk).

---

In the rebuttal, we have conducted the following additional experiments to address the reviewers' questions and suggestions:

- We provide further analysis on performance changes as the ratio of invisible regions varies. (Figure A in the attached PDF file)
- We conduct additional ablation studies on camera (Plücker) embedding and depth embedding. (Figure B in the attached PDF file)
- To verify whether cross-view attention mimics depth-based warping, we analyze the distance between GT flow and flow extracted from the cross-view attention. (First table in the response to Reviewer Cjbv)
- We analyze the impact of cross-view attention and self-attention on visible and invisible regions. (Second table in the response to Reviewer Cjbv)
- For extensive comparison, we provide a comparison with existing NeRF-based pipelines. (Figure C in the attached PDF file)
- We provide results of 3D scene (3DGS) reconstruction with novel views generated by our method. (Figure D in the attached PDF file)
- We provide intermediate generation results. (Figure E in the attached PDF file)

Meta-review: The paper addresses novel view synthesis from a single image. Contrary to prior work, which warps using a monocular depth estimate and then inpaints, the diffusion model is conditioned on warped coordinates, making the approach more robust noisy depth estimates.

Overall, the reviewers are positive about the paper, mentioning strong results and improvements over baselines in challenging settings. The main concern was to add comparisons to more baselines which the authors addressed in their rebuttal.

I reccomend acceptance.
Decision: Accept (poster)