Title: WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D GAN Inversion; Novel View Inpainting
Abstract: 3D GAN inversion projects a single image into the latent space of a pre-trained 3D GAN to achieve single-shot novel view synthesis, which requires visible regions with high fidelity and occluded regions with realism and multi-view consistency.  However, existing methods focus on the reconstruction of visible regions, while the generation of occluded regions relies only on the generative prior of 3D GAN.  As a result, the generated occluded regions often exhibit poor quality due to the information loss caused by the low bit-rate latent code.  To address this, we introduce the warping-and-inpainting strategy to incorporate image inpainting into 3D GAN inversion and propose a novel 3D GAN inversion method, WarpGAN.  Specifically, we first employ a 3D GAN inversion encoder to project the single-view image into a latent code that serves as the input to 3D GAN. Then, we perform warping to a novel view using the depth map generated by 3D GAN. Finally, we develop a novel SVINet, which leverages the symmetry prior and multi-view image correspondence w.r.t. the same latent code to perform inpainting of occluded regions in the warped image. Quantitative and qualitative experiments demonstrate that our method consistently outperforms several state-of-the-art methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for novel view synthesis of face images, via inversion of an existing 3D-aware GAN (EG3D). It trains an encoder to map from input images to the GAN latent w in order to obtain a depth-map (by decoding and re-rendering); then warps the original image according to the depth map and target pose. Occluded regions are filled in by a custom two-stage in-painting process. The method is evaluated on CelebA-HQ and MEAD, where it shows stronger performance than existing 3D GAN inversion methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe proposed approach is interesting and novel. Explicitly warping the source image ensures that details can be preserved even if they are challenging to encode and decode directly in the GAN's latent space, while still making use of its prior knowledge for depth estimation and occluded regions.\n\nThe inpainting stage is itself somewhat novel – it uses a stylegan-inspired architecture that also leverages facial symmetry in feature extraction.\n\nEvaluation covers two large-scale datasets (CelebA-HQ and MEAD); for CelebA-HQ, identity preservation is explicitly measured, as well as realism via FID; for MEAD, reconstruction is also measured directly via LPIPS.\n\nQuantitative results on both datasets are significantly stronger than the chosen baselines according to all metrics. The method is also faster than most of the baselines thanks to being feed-forward (encoder-based) rather than relying on iterative optimization.\n\nThe selected qualitative results also appear better than all the baselines, with rather fine details from the source image successfully transferred to the target image.\n\nThere is a fairly detailed ablation study, measuring the benefit of various design decisions and innovations.\n\n## Weaknesses\n\nThe evaluation only compares against other 3D GAN inversion methods. The comparisons should also include diffusion-based NVS methods (either 2D-only, or 3D-aware) in order to place the method properly in context.\n\nThere are no measures of spread (confidence intervals or standard deviations) on the experimental results; this makes it difficult to know if the results are statistically significant. In particular, on certain metrics the gains over the baselines are rather small.\n\nThere is no comparison of the full pipeline but with just the warping / rerendering stage replaced by simply using the rerendered inverted result. This comparison is somewhat important, since it justifies the key selling point of the paper, i.e. use of warping, in isolation from confounding factors such as how well the specific inpainting method works.\n\nThere is no comparison of the custom SVINet inpainting network against a modern diffusion image-to-image model.\n\nThe topic feels somewhat dated – while there is still a strand of literature pursuing inversion of EG3D and similar models as a route to novel view synthesis, the community has largely moved on to diffusion-based methods. This does not in any way affect the technical validity and rigor of the paper; however it does likely limit this work's interest to the community. The restriction to single-view datasets is rather artificial given multi-view datasets of heads are now available; moreover diffusion-based methods such as RenderDiffusion train in EG3D's single-image setting anyway.\n\nThe method's applicability is limited to tightly-cropped human heads. Not even EG3D's cats are included, though presumably these would also work in the same pipeline given they are symmetrical too.\n\nQUESTIONS:\nWhy are both inpainting stages needed? and how are their results combined? In particular, the first inpainting using re-rendered pixels from the inversion (described in eq 2) should result in a complete image; what then is left for the SVINet part to do? It seems the second inpainting is not really inpainting but doing an image-to-image tidying-up of the whole image? What would happen if the second stage were used without the first?\n\nDoes the feature-mirroring stage described in sec 3.3.1 presuppose that that the input image is posed symmetrically? What if this is not the case, e.g. a half-profile view is given as input?\n\nWhat exactly is Figure 3 showing? In particular, it claims to show NVS on CelebA-HQ, but the \"input\" column appears to contain synthetic images (see the blurring outside of the face region). If these images were in fact generated, this should be stated clearly (since it makes the inversion task easier, as they then definitely lie in the latent manifold of the GAN whereas real images even from the same dataset may not).\n\nLIMITATIONS:\nYes – although limitations is rather brief", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of low-quality novel view synthesis in single-image 3D GAN inversion, focusing on improving occluded regions that often suffer from missing details and a lack of realism. The authors propose WarpGAN, a novel method that integrates the classical \"warping-and-inpainting\" strategy into the 3D GAN inversion pipeline. Specifically, the method first warps the input image to a new viewpoint using a depth map obtained from the 3D GAN inversion, thereby preserving high-frequency details in visible areas. Then, it employs a style-based inpainting network, SVINet, which leverages facial symmetry priors and the latent code from the 3D GAN to realistically fill in the occluded regions while maintaining multi-view consistency. To overcome the lack of ground-truth data for real images, the authors introduce a \"re-warping\" strategy. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Clear Motivation: The over-reliance on the 3D GAN's generative prior to fill occluded regions leads to poor generation quality. The Integration of the \"Warping-and-Inpainting\" framework is reasonable.\n2. Innovative SVINet Network: The symmetry prior and style-based modulation used in the SVINet and its training strategy is justified and convincing.\n3. Good results: Current experiments demonstrate the effectiveness of the proposed method.\n\n\nWeaknesses\n1. The core idea of integrating a \"warping-and-inpainting\" strategy with 3D GAN inversion is not new and has been previously explored by methods like HFGI3D. The authors argue that their key contribution lies in replacing the optimization-based inpainting of HFGI3D with a dedicated feed-forward network (SVINet), which is incremental. Therefore, the paper could be strengthened by more clearly articulating and demonstrating why its proposed feed-forward inpainting network is superior to a well-formulated optimization-based alternative, perhaps by highlighting scenarios where optimization-based methods fundamentally fail and only a dedicated, trained network can succeed.\n2. The method's generalization ability to more \"in-the-wild\" scenarios—such as extreme poses, exaggerated expressions, complex occluders, or non-face objects—remains to be validated. For example, the symmetry prior might not hold when a hand occludes the face.\n\n\nMinor:\n1. Line 216 “Datesets” typo\n\nQUESTIONS:\nPlease see the raised weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a warping-then-inpainting approach to synthesize multi-view images for facial images.  It first employs a EG3D-style GAN to do the 3D  inversion of  a single view facial image, produce its depth map, and then inpaint the warped facial image at the new viewpoint.  The proposed approach can also enable facial editing in the latent feature space.  The experimental results show the improvement of quantitative metrics, such as FID and ID.

STRENGTHS AND WEAKNESSES:
Strengths:
1.  It shows that warp-then-inpaint procedure can improve the quality of the novel view synthesis of facial images. Experimental results are impressive.
2. The ablation study clearly shows the improvements by SVINet. 

Weaknesses:
While the experimental results are impressive,  the technical components in the proposed approach are mainly adapted from already-known methods, such as EG3D, symmetry and ID  loss.  It downgrades the technical novelty of this paper.

QUESTIONS:
1. Does the row A indicate the result of 3D GAN  inversion?  Since there are recent studies on 3D facial image editing based on 3D GAN  inversion, I am curious why the results of 3D GAN inversion only are much worse than warp-then-inpaint.
2. Since warp-then-inpaint procedure is widely used in diffusion-based methods,  what is the comparison between the proposed approach and these diffusion-based methods?  please also explain why diffusion-based models are not used.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method, WarpGAN, aimed at addressing the occlusion problem in the task of 3D GAN inversion. The authors first employ an encoder to project the input image into a latent code in the w space, which is then fed into a 3D GAN to synthesize views. A carefully designed warping strategy is subsequently applied to generate novel views along with corresponding synthetic depth maps. Finally, an inpainting network is introduced further to enhance the visual fidelity of the generated views. Experimental results demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper is well-written and easy to follow.\n\n2.\tThe use of the warping-and-inpainting strategy to handle the occlusion problem is an interesting and effective design choice.\n\n3.\tWarpGAN achieves considerable improvements over baseline methods, demonstrating the effectiveness of the proposed approach.\n\nWeaknesses:\n\n1.\tWhile the authors claim to address the occlusion problem, the experimental section lacks a specific analysis of how occlusion affects the performance of baseline methods. For example, does occlusion lead to artifacts in facial regions such as the ears or side profile? It would strengthen the paper if the authors explicitly highlighted the impact of occlusion on visual quality and reconstruction accuracy.\n\n2.\tThe authors generate synthetic training samples, but as shown in [1], naively sampling from EG3D can produce unstable results. Therefore, more details about the sampling strategy are needed to assess the reliability and quality of the training data.\n\n3.\tThe organization of the paper could be improved. For instance, Section 3.1 currently focuses heavily on warping. It may be clearer to introduce a dedicated subsection for the warping module and keep the overview section as a concise summary of the entire pipeline.\n\n\n[1] G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images. CVPR 2024.\n\nQUESTIONS:\nPlease refer to the Strengths And Weaknesses section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the long‑standing problem of poor occluded‑region quality in single‑image 3D‑aware GAN inversion. It combines depth‑based warping (to reuse pixels that are already visible) with a dedicated style‑based 2‑D inpainting network (SVINet) that is conditioned on (1) the latent code produced by an EG3D encoder and (2) a symmetry prior obtained from a horizontally flipped copy of the input face. Training mixes real single‑view images with synthetic multi‑view pairs rendered from EG3D. Quantitative results on CelebA‑HQ and on MEAD show reasonable gains over eight published baselines, and inference is as fast as other feedforward. Ablations support the contribution of each design choice. The authors also demonstrate that the synthesized multi‑view set can be fed to PTI to obtain superior latent‑space editing.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n-- The paper solves a clear problem and is well-motivated. 3D GAN inversion does tend to hallucinate unseen regions, the paper addresses an interesting problem.\n\n-- The warping and inpainting approach is reasonable, since the depth maps come out for free from EG3D, and conditioning SVINet on the latent code definitely improves multi-view consistency.\n\n-- The improvements are significant with limited training budget.\n\n\nWeakness:\n\n-- This might go back to the old \"novelty\" problem, but I do feel the novelty of this paper is incremental relative to existing “warp + inpaint” pipelines. The idea has appeared in GenWarp (NeurIPS’24) and LucidDreamer (ICCV’24), and In‑N‑Out (CVPR’24) already mixes triplane editing with an occlusion‑aware decoder. The paper does not position itself clearly against these very closely related works.\n\n-- The evaluation domain is kind of limited, since it is face only.\n\n\nOverall, the paper offers a practically useful recipe that materially improves facial 3D inversion quality at encoder‑level speed. However, its methodological novelty over other warp‑and‑inpaint works is moderate, the evaluation is narrowly scoped to faces. It is a decent paper, but the contribution may be judged as incremental for NeurIPS.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}