Title: GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping
Venue: NeurIPS 2024
Accepted: yes
Keywords: Single-shot Novel View Generation

Generative Models

Novel View Generation

Diffusion Models
Abstract: Generating novel views from a single image remains a challenging task due to the complexity of 3D scenes and the limited diversity in the existing multi-view datasets to train a model on. Recent research combining large-scale text-to-image (T2I) models with monocular depth estimation (MDE) has shown promise in handling in-the-wild images. In these methods, an input view is geometrically warped to novel views with estimated depth maps, then the warped image is inpainted by T2I models. However, they struggle with noisy depth maps and loss of semantic details when warping an input view to novel viewpoints. In this paper, we propose a novel approach for single-shot novel view synthesis, a semantic-preserving generative warping framework that enables T2I generative models to learn where to warp and where to generate, through augmenting cross-view attention with self-attention. Our approach addresses the limitations of existing methods by conditioning the generative model on source view images and incorporating geometric warping signals. Qualitative and quantitative evaluations demonstrate that our model outperforms existing methods in both in-domain and out-of-domain scenarios. Project page is available at https://GenWarp-NVS.github.io.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to achieve novel view synthesis from a single image. Contrary to prior work, which warps using a monocular depth estimate and then inpaints, a more flexible architecture is introduced. To do this, a diffusion model is conditioned on warped coordinates, based on the desired pose change and a monocular depth estimate. In addition, the input view (and base coordinate map) is featurized and the diffusion process is allowed to cross attend to these features, which should intuitively allow for attention between corresponding points. The method is then trained end-to-end with a denoising diffusion objective. Experiments show that this approach achieves better FID, PSNR, and LPIPS scores than baselines on RealEstate10k and ScanNet. Ablations justify the choice of warping conditioning.\n\nSTRENGTHS:\nThe manuscript is well written and well motivated. The related work section is comprehensive, the methods section is straightforward and easy to understand. And experimental results seem quite strong.\n\nWEAKNESSES:\nI think the paper could benefit significantly from apples-to-apples ablations. For example, I am quite interested in a comparison between the proposed method, and the method without any depth information (I think similar to GeoGPT). Or perhaps the proposed method, and the method without the warping.\n\nThe authors also claim that the cross-attention “allows the model to inherently decide which regions should rely more on its generative capability and which areas should depend primarily on the information from the input view warping.” However, they provide only a single example of this in figure 4. I think more examples would make for more convincing evidence. I would be interested in a larger scale, more robust study of this hypothesis. For example, this could possibly be automated by computing how closely the cross-attention actually matches warping.\n\nThe SD warp baseline could possibly be implemented better. I believe the inpainting mask covers non-warped black pixels, resulting in many unrealistic “black borders” is in Figure 10 and Figure 5. Perhaps a more fair and robust baseline could be achieved by just expanding the inpainting mask a bit.\n\nText2room by Höllein et al seems relevant. Perhaps the authors would consider citing it.\n\nQUESTIONS:\nFrom the weaknesses section: could apples-to-apples ablations be conducted on components of the method?\n\nFrom the weaknesses section: could more evidence for “cross-attention attends to corresponding points” be presented? \n\nOne major benefit of the proposed setting, as opposed to warp then inpaint, is that the method should be able to handle view-dependent effects. This may be interesting to investigate.\n\nLIMITATIONS:\nLimitations and societal impacts are included, and adequately addressed, but are not in the main body of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a semantic-preserving generative warping framework to generate high-quality novel views from a single image, which mainly consists of two components:\n- Condition the novel view synthesis on the warped 2D coords embedding from the estimated depth map.  \n- Augmenting cross-view attention with self-attention.\n\nThe proposed method eliminates the artifacts caused by error depths in the warping-and-inpainting pipeline and integrates semantic features from source views, preserving semantic details in generation.\n\nSTRENGTHS:\n-\tCompared to existing warping-and-inpainting methods that condition on explicit depth warping, the proposed warped 2D coords embedding forms the correspondence between reference view and target view implicitly, which helps the network to be more robust to the noise in the estimated depth map without losing sematic details. \n\n-\tThe paper is well-written, being clear and easy to follow. The technical limitations are also discussed in detail in the Sup.\n\nWEAKNESSES:\n-\tThe proposed embedding based on depth warping could not benefit the synthesize process when there are large camera movements or occlusions between the input view and the target view. \n\n\n-\tAlthough the proposed depth-warping embedding somehow reduces the influence of the noise in the estimated depth map, compared to explicit pixel warping, the implicit condition of depth embedding also lower the local preservation ability of the network when synthesizing novel views. Is the model capable of synthesizing consistent novel views which could be used to reconstruct a 3D scene ? \n\n\n-\tThe comparison baseline is limited. Plucker embedding[1] is also a dense embedding capable of providing local correspondence, which supports large camera movement and occlusions.  The difference between such dense embedding with the proposed depth warping embedding should be further discussed.\n\n\n[1] SPAD : Spatially Aware Multiview Diffusers(CVPR2024)[https://arxiv.org/abs/2402.05235]\n\nQUESTIONS:\nAs listed in the Weakness.\n\nLIMITATIONS:\nNo specific limitation and negative societal impact need to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel framework called GenWarp, which aims to generate new views from a single input image while preserving the semantic content of the original view. This is achieved by leveraging a generative process that incorporates both self-attention and cross-view attention mechanisms conditioned on warping signals. The proposed approach demonstrates superior performance compared to existing methods, especially in scenarios with challenging viewpoint changes, and exhibits good generalization to out-of-domain images.\n\nSTRENGTHS:\n1.\tThe paper introduces an approach for single-shot novel view synthesis by combining self-attention and cross-view attention mechanisms to preserve semantic details. \n2.\tThe proposed method outperforms existing techniques in generating high-quality novel views, particularly for challenging viewpoint changes.\n3.\tGenWarp shows generalization capabilities, performing well on out-of-domain images, which indicates its applicability to various scenarios.\n\nWEAKNESSES:\n1.\tThe paper could further highlight its novelty by providing a more comprehensive comparison with a wider range of state-of-the-art methods, including both classical and recent approaches. Additionally, the authors should discuss any potential limitations of their approach in terms of scalability or adaptability to different types of scenes.\n2.\tThe methodology section could benefit from additional diagrams and flowcharts that illustrate the workflow and attention mechanisms in more detail. Including intermediate results and step-by-step visualizations would help readers better understand the progression from the input image to the generated novel view.\n3.\tWhile the paper provides detailed instructions for reproducing the experiments, the code and data are not made publicly available at the time of submission, which can hinder reproducibility efforts.\n4.\tThe performance of GenWarp is highly dependent on the quality of the datasets used for fine-tuning, which could limit its effectiveness if high-quality multi-view datasets are not available.\n5.\tThe method struggles with generating novel views when the camera viewpoints are extremely distant, indicating a limitation in handling very large viewpoint changes.\n6.\tThe reference list could be updated to include more recent advancements in the field, particularly those published in the last year. Additionally, a more detailed comparative analysis of the strengths and weaknesses of related methods would be beneficial.\n7. what is the advantage of diffusion-model-based pipeline over 3D Gaussian[A,B] or NeRF[C,D] based pipeline, can the author discuss more about existing methods?\n\n[A] Yu, Zehao, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. \"Mip-splatting: Alias-free 3d gaussian splatting.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456. 2024. \n[B] Yan, Zhiwen, Weng Fei Low, Yu Chen, and Gim Hee Lee. \"Multi-scale 3d gaussian splatting for anti-aliased rendering.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20923-20931. 2024.\n[C] Deng, Congyue, Chiyu Jiang, Charles R. Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, and Dragomir Anguelov. \"Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20637-20647. 2023.\n[D] Yang, Yifan, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. \"Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15901-15911. 2023.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposes a novel single-shot novel view synthesis framework based on a pretrained T2I diffusion model. Instead of directly warping pixels between input view and novel view,  an implicit approach is proposed to conduct geometry warping operation.  The cross-view attention is used to eliminates the artifacts caused by error depths and integrates semantic features from source views, preserving semantic details in generation. Extensive experiments prove the efffectiveness of the proposed method.

STRENGTHS:
1. The implicit geometry warping approach is effective in addressing ill-warped problem and missing original semantic problem encountered in explicit warpping methods.
2. The cross-view attention can provide more useful information for novel view generation.
3. The qualitative and quantatitive experiments on RealEstate10K , ScanNet, and in-the-wild images show the proposed method outperform SOTA methods in both in-domain and out-of-domain scenario.

WEAKNESSES:
1. Although the cross-view attention strategy is effective to fuse features, it may be not novel enough. There are many similar operation in video generation area.
2. The detail of finetuning the T2I model is not clear.

QUESTIONS:
N/A

LIMITATIONS:
This method is restricted to depth-based correspondence of two views which limit its application in some scenes where depth correspondence between two views is missing.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12