WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting
Authors: Kaitao Huang
Yan Yan
Jing-Hao Xue
Hanzi Wang
Venue: NeurIPS 2025 poster ()
Keywords: 3D GAN Inversion; Novel View Inpainting
Primary area: deep_learning
TL;DR: We introduce the warping-and-inpainting strategy to incorporate image inpainting techniques into 3D GAN inversion and propose a novel 3D GAN inversion method, WarpGAN.

Abstract: 3D GAN inversion projects a single image into the latent space of a pre-trained 3D GAN to achieve single-shot novel view synthesis, which requires visible regions with high fidelity and occluded regions with realism and multi-view consistency.  However, existing methods focus on the reconstruction of visible regions, while the generation of occluded regions relies only on the generative prior of 3D GAN.  As a result, the generated occluded regions often exhibit poor quality due to the information loss caused by the low bit-rate latent code.  To address this, we introduce the warping-and-inpainting strategy to incorporate image inpainting into 3D GAN inversion and propose a novel 3D GAN inversion method, WarpGAN.  Specifically, we first employ a 3D GAN inversion encoder to project the single-view image into a latent code that serves as the input to 3D GAN. Then, we perform warping to a novel view using the depth map generated by 3D GAN. Finally, we develop a novel SVINet, which leverages the symmetry prior and multi-view image correspondence w.r.t. the same latent code to perform inpainting of occluded regions in the warped image. Quantitative and qualitative experiments demonstrate that our method consistently outperforms several state-of-the-art methods.

WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting
Kaitao Huang', Yan Yan Jing-Hao Xue", Hanzi Wang'
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, P.R. China
Department Statistical Science, University College London, UK huangkt@stu xmu. xmu.edu. cn, yanyandxmu. edu. cn
Figure 1: Visual examples. Given a single input image (the first row), our WarpGAN synthesizes images from five novel views: front, right, left, top, and down (the second to the sixth rows).
Abstract
3D GAN inversion projects a single image into the latent space of a pre-trained 3D GAN to achieve single-shot novel view synthesis, which requires visible regions with high fidelity and occluded regions with realism and multi-view consistency. However, existing methods focus on the reconstruction of visible regions, while the generation occluded regions relies only on the generative prior of 3D GAN. As result, the generated occluded regions often exhibit poor quality due the information loss caused by the low bit-rate latent code. To address this, we introduce the warping-ande ceappin--and-inaainnee iaddiiiniinting strategy incorporate image inpainting into 3D GAN inversion and propose a novel 3D) GAN inversion method, WarpGAN. Specifically, we first employ a 3D GAN inversion encoder project the single-view image into a latent code that serves as the input to 3D GAN. Then, we perform warping to novel view using the depth map generated by 3D GAN. Finally, we develop a novel SVINet, which leverages the symmetry prior and multi-view image correspondence w.r.t. the same latent code perform inpainting of occluded regions in the warped image. Quantitative and qualitative experiments demonstrate that our method consistently outperforms several state-of- off-the-ar methods.
Corresponding Author
Please visit the Project Page for visualizations and code.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
GANs [13] have made remarkable progress in synthesizing unconditional images. In particular, StyleGAN [20, 21] has achieved photorealistic quality on high-resolution images. Several exten- sions [15, 31, 36] leverage the latent space (i.e., the W space) to control semantic attributes (e.g., expression and age). However, these 2D GANs suffer from inferior control over geometrical aspects of generated images, leading multi-view inconsistency for viewpoint manipulation.
Recently, with the development of neural radiance fields (NeRF) [27] in novel view synthesis (NVS), variety of 3D GANs [2. 5, 6, 14, 29, 39, 41] have been proposed to integrate NeRF into style- based generation, resulting in remarkable success in generating highly realistic images. Based on it, 3D GAN inversion methods project a single image into the latent space of pre-trained 3D GAN generator, obtaining a latent code. Hence, the viewpoint of the input image can be changed by altering the camera pose, and the image attributes can be easily edited by modifying the latent code. Unlike 2D GAN inversion, 3D GAN inversion aims to generate images that maintain both the faithfulness of the input view and the high quality of the novel views.
On the one hand, existing 3D GAN inversion methods rely only on the generative prior of 3D GANs for generating the occluded regions (i.e., the invisible regions in the input image) in the novel viewpoint, resulting in unfaithful reconstruction of occluded regions complex scenarios. On the other hand, for 3D scene generation, several recent methods adopt warping -nd and-inpainting strategy. They [11. 30, 35] first predict a depth map of a given image, and then warp the input image to novel camera viewpoints with the cepth-based correspondence, followed by 2D inpainting network to synthesize high-fidelity occluded regions of the warped images.
To address the inferior reconstruction capability of occluded regions in existing 3D GAN inversion methods, motivated by the success of the introduce image inpainting into 3D GAN inversion. Unfortunately, 3D GAN inversion is dedicated to training with single-view view datasets, while the above 3D scene generation methods usually require multi-view datasets for training. This leads to two issues: (1) multi-view inconsistency due to the lack of 3D information (i.c., the real novel view image) to guide the inpainting process; (2) the unavailabille of ground-truth images from novel views to compute the loss during model training.
strategy in 3D scene generation, we
In this paper, we propose novel 3D GAN inversion method, WarpGAN, by integrating the warping- and-inpainting strategy into 3D GAN inversion. Specifically, we first train 3D GAN inversion encoder, which projects the input image into latent code w+ (located in the latent space W* of the 3D GAN generator). By feeding =* into 3D GAN, we compute the depth map of the input image for geometrie warping and perform an initial filling of the occluded regions in the warped image. Subsequently, leveraging the symmetry prior [43, 45] and multi-view image correspondence w.r.f. the same latent code in 3D GANs, we train a style-based novel view inpainting network (SVINet). It can inpaint the occluded regions in the warped image from the original view to the novel view. Hence, we can synthesize plausible novel view images with multi-view consistency. To address the unavailability of ground- truth images, we re- warp the image in the novel view back to the original view and feed it to SVINet. Hence, the loss can be calculated between the inpainting result and the input image. Some visual examples obtained by WarpGAN are given in Fig. 1.
In summary, the contributions of this paper are as follows:
We propose a novel 3D GAN inversion method, WarpGAN, which successfully introduces the warping- satie--annddiiiiiiiii strategy into 3D GAN inversion, substantially enhancing the quality of occluded regions in novel view synthesis.
We introduce style-based novel view inpainting network, SVINet, by fully leveraging the symmetry prior and the same latent code generated by 3D GAN inversion, achieving multi-view consistency inpainting the occluded regions of warped images in novel views.
We perform extensive experiments to validate the superiority of WarpGAN, showing the great potential of the warping-ande aand-nnndnninnnnn strategy 3D GAN inversion.
2 Related work
3D-Aware GANs. Recent advancements in 3W-e GANs [2, 5, 6, 14, 29, 39, 41) effectively combine the high-quality 2D image synthesis of StyleGAN [20, 21] with the multi-view synthesis

---

capability of NeRF [27], advancing high-quality image synthesis from 2D to 3D and enabling multi- view image generation. These methods typically employ a two-stage generation pipeline, where a low-resolution raw image and feature maps are rendered, followed by upsampling to high-resolution using 2D CNN layers. Such way ensures geometric consistency across multiple views and achieves impressive photorealism. In this paper, we leverage EG3D [5] as our 3D-aware GAN architecture, which introduces a hybrid explicitiipplici 3D representation (known as the tri-plane).
GAN Inversion. Although recent 2D GAN inversion methods [42] have achieved promising edit- ing performance, they suffer from severe flickering and inevitable multi-view inconsistency when editing 3D attributes (e.g., head pose) since the pretrained generator is not 3D-aware. Hence, 3D GAN inversion is developed to maintain niew consistency when rendering novel viewpoints. However, directly transferring 2D methods to 3D without effectively incorporating 3D information will inevitably lead to geometry collapse and artifacts.
Similar to 2D GAN inversion, 3D GAN inversion can be categorized into optimization based methods and encoder-based methods. Some optimization-basee methods [23, 43, 45] generate multiple pseudo- images from different viewpoints to facilitate optimization. For instance, HFGI3D [43] leverages visibility analysis to achieve pseudo-multi-vin prior to synthesize pseudo multi-view images; and Pose Opt. [2.]] simultaneously optimizes camera pose and latent codes. In addition, In-N-Out [44] optimizes a triplane for reconstruction and employs composite volume rendering. Encoder-based methods project the input image into the latent space of the 3D GAN generator and then employ the generative capacity of the 3D GAN to synthesize novel-view images, while fully utilizing the input image to reconstruct the visible regions of the novel-view images. For example, GOAE [46] computes the residual between the input image and the reconstructed image to complement the F space of the generator, and introduces an occlusion-awars mix tri-plane for novel-view image generation; Triplanenet [3] calculates an offset for the triplane based on the residual and proposes a facial symmetry prior loss; and Dual Encoder [4] employs two encoders (one for visible regions and the other for occluded regions) for inversion and triplane discriminator to enhance both fidelity and realism.
optimization; SPI [45] utilizes the facial symmetry
object
introduces an occlusion-awanr
Our method is intrinsically different from existing methods that rely heavily on 3D GAN generative priors to generate occluded regions. Our method introduces a novel inpainting network to fill the occluded regions, facilitating the generation of rich details.
Depth-based Warping for Single-shot Novel View Synthesis. Some 3D GAN inversion methods [23, 43, 45] use depth- warping to synthesize pseudo multi-view images for optimization. SPI [45] warps the input image to an adjacent view for pseudo-supervision. Pose Opt. [23] warps the image from the canonical viewpoint to the input viewpoint to assist training. HFGI3D [43] utilizes 3D GAN to fill the occluded regions of the warped image from the input view to novel views, synthesizing several pseudo novel-view images. However, these methods only rely on a 3D GAN to generate occluded regions, failing to achieve satisfactory results in occluded regions under complex scenarios.
Recently, some methods follow the warping- aaleg--ndd-didddn indddiiiinniigg strategy on single-shot NVS for general scenes [11, 30, 35]. They first predict a depth map for the input image, then warp the input image to novel view using the depth map, and finally perform inpainting on the occluded regions in the novel view. This way can effectively preserve the information of the input image while leveraging the powerful inpainting capability of 2D inpainting networks to generate reasonable content for occluded regions. Inspired by this strategy, we introduce 2D inpainting network into 3D GAN inversion by effectively exploiting the symmetry prior and the latent code of the input image.
3 Methodology
3.1 Overview
As shown in Fig. 2, our WarpGAN consists of 3D GAN inversion network (including a 3D GAN inversion encoder and a 3D-aware GAN) and a style-hased novel view inpainting network (SVINet). First, we utilize 3D GAN inversion encoder Egy+ to project the input image I into the latent space W of the 3D GAN generator, obtaining the latent code w Based on this, we utilize a rendering decoder to render the depth map D of I and the novel view image ius depth map D, we warp the input image I from the original view e to the novel view Guovel, thereby obtaining the warped image pearp_ e and the occluded regions M Mo-th of the input image in the
hovel Under the guidance of the
3

---

Diwnd
3D GAN Inversion
But
G
ti-plaae
Style-based Novel View Inpainting
Refine
Refine
Ni
nurver
.Ns
Dundeaso
Dopth bead warzing
Faward waep
operaton
warp
Figure 2: Overview our WarpGAN, which consists of 3D GAN inversion network and style- based novel view inpainting network (SVINet). The "Forward warp" flow (blue arrows) illustrates the inference process of novel view synthesis. During model training, we also require the "Reverse warp" flow (red arrows) to warp the novel view image back to the original view for loss computation.
target view, that is,
(1) where on-eet is a relative camera pose between c and Cnovel. K is the camera intrinsic matrix, and warp(-) is geometrie warping function [28, 35] which unprojects pixels of the input image I with its depth map D to the 3D space, and reprojects them based on Te-tennve and K.
worP CHoool C-Covvl
M Crovell
warp(I: D. Te Te-teoe K),
I Then, we use novel to fill in the occluded regions of Twerp for the occluded regions, which can be formulated as
Cnooel" serving as the initial result novel
iinitial novel
warp C-oCnl
M M
iw
novel.
Subsequently, the initial result novel is fed into SVINet for further inpainting, giving the final output Inveet of WarpGAN. Notably, employ symmetry-aware feature extraction and modulate the convolutions of the inpainting network with w during the inpainting process. We also construct a style- style-based based loss ensure consistency between the generated image in the novel view and the original view image.
3.2 3D GAN Inversion Encoder
Similar to existing encoder-based 3D GAN inversion methods, our 3D GAN inversion encoder E projects an input image with the camera pose e into the latent space W+ of the pre-trained 3D> GAN, obtaining the latent code - = F (T). Then, we leverage the generator G(-) of 3D GAN to generate the tri-plane and use the rendering decoder to render images at specified camera poses. Based on above, we perform image reconstruction i = R(G(w+), c) by specifying the camera pose In this way, obtain the novel view image iu novet corresponding to the novel camera pose Cnovel- Under the principles of NeRF, we replace the color of the sampling points with the distance to the camera during the rendering process, obtaining the depth maps D and D Dovett More implementation details can found in the Appendix.
Inspired by GOAE [46], we employ pyramid-strucce structured Swin-Transforme [26] as the backbone of the encoder, based on which leverage feature layers at different scales generate latent codes at various levels.

---

Since our dataset contains only single-view images, we train E using a reconstruction loss Lutt, which includes pixel-wise (MSE) loss Z2 perceptual loss LIPIP [48], and an identity loss LiD with pre-trained ArcFace network [12]:
Cwt
(I
,I) = (2"
"
.I + CL .I) + AI (I .I),
(3)
where X2, ALPIPS and Ai denote the loss weights for L2, LIPIP and LID, respectively.
3.3 Style-Based Novel View Inpainting Network (SVINet)
Due to the existence of occluded regions in the novel view, the warped image contains "holes" (see Fig. 2 for illustration). To generate high-quality novel-view images, we propose a style-based novel view inpainting network (SVINet) to fill in the "holes" in the warped image.
As shown in Fig. 2, our SVINet follows the traditional "encode--npalie inpaint inppinttddecodrr architecture [10, 24, 37], consisting of three sub-networks: NE, N,, and ND. Technically, Nr is first used to extract features from the model input while performing downsampling. Then, the inpainting operation is performed in the feature space by using Nj. Finally, ND is used to upsample the features to obtain the inpainted image.
3.3.1 Symmetry-Aware Feature Extraction
We first use the novel-view image i
novel obtained from 3D GAN inversion to fill in the occluded regions in the warped image Twerp
Co-Cnteet (Eq. (I)), resulting in an initial inpainting result finitial (Eq. (2)). We then feed novel into Ng to obtain the feature F. In addition, we also propose to leverage the facial symmetry [43, 45] by warping the mirrored input image mirror to the target view warp
Crovet, obtaining mirror e Emirror -Cnoeet The mirrored image is then processed in the same manner as described above and fed into NE to obtain the mirror feature F mérror-
Subsequently, we utilize F and F Fmirror to predict the scale map F, and the translation map Fr. which can be used to refine via featurewise linear modulation (FiLM) [32], obtaining F,. that is,
FF..Fr) F.} { mirrorl] 11), (f F.F.Frme Fmirror]1)},
F, F C F Fe.
where ds and d are convolutional neural networks; [ denotes concatenation along the Ith dimen- sion, i.., the channel dimension; 'O' denotes the Hadamard product.
Next, Fr is successively fed into N, and Np to obtain the inpainting result novel--
3.3.2 Style-Based Inpainting
Inpainting networks typically rely on the information of the input image to fill the missing regions. However, due to the limited information contained in single-view images, using only this information for inpainting may lead to the issue of multi-view inconsistency. To address the consistency issue, motivated by the fact that images of the same object from different viewpoints share the same latent code 3D GANs, we introduce the latent code to control the image inpainting process.
we modulate the convolutions [21, 24] in the "inpaint" and "decoder" parts of the inpainting network using the latent code =* obtained from E This modulation of the convolutions facilitates us to control the inpainting process for occluded regions, achieving multi-view consistency the generated images.
Technically,
Specifically, we first employ a mapping function AA to obtain the style code & = A(w+). Then the weights of the convolutions w are modulated as
wryk S; Wiyk:
w//K w/sh / V
(5)
w'isk +
2
where w denotes the final modulated weights; 8; the scale corresponding to the ith input feature map; j and & enumerate the output feature maps and spatial footprint of the convolution, respectively.
5

---

3.3.3 Training strategy
Real data. Since our real dataset contains only single-view images, no sarget-view images can be used to compute the loss and update the model parameters when synthesizing images from novel views. To address this, we propose to re-warp the warped image from the novel view back to the original view, and then compute the loss between the inpainting result and the input image.
Specifically,, for the input image I, we first warp it to the novel view Gnovel to obtain I c-Cenvaa and then inpaint it using SVINet to get nnoees. Next, re-warp c back to the source view and inpaint it again to obtain rrewarr, Based on the above, given the input image I, we obtain two inpainted images Lnoeel and fre-wery for loss computation.
Synthetic data. In addition to real data, also utilize synthetic data assist in training our model. We sample a latent code Wsynth from the latent space of 3D GAN and generate two images Tyynth and Ięanth from different viewpoints. We then warp Iaynth from the source view to the target view and input it into SVINet obtain the inpainted image Ioynth Finally, we compute the loss between I. iewnth and I reunch
Loss function. Our loss function consists of three components: the reconstruction loss, the consis- tency loss, and the adversarial loss. The reconstruction loss Ciee includes the pixel-wise MAE loss L1. the perceptual loss Lp [37]. and the identity loss LID [12]:
Lrce 11 = A (i
Ai, - AILLD(i,
(6)
where '1, Ap, and ÅιD denote the loss weights for L1, Lp. and LID. respectively; i and I represent the input image and the generated image, respectively.
To ensure multi-view consistency, we introduce the consistency loss Le, which computes the MSE between the latent codes of the original image and the inpainted image. This loss is used to control the multi-view consistency of the generated images:
Co
I) = ||E (i), Eu (1)
To further enhance the quality of the inpainted images, we also use an adversarial loss:
C E[log(D(#))],
(8)
2h -lloog(((())) - Bllog(1 - D()))] +
where x denotes the real and synthetic images (i.e., and 1oant ); represents the inpainted images (i.e., Inoeets fre-wary, and feynth); y; D denotes the discriminator [10, 37].
In summary, the loss function for SVINet can be formulated as follows:
LSVINer = Xreshhree (fire-warp ieynth lo, [I, I. 1onth
X.Lo (L fre-warp j lo. [I. I, 1.3mbh1n) + Aun
(10)
where [o]o denotes concatenation along the 0-th dimension (i.e., the batch dimension); A des and Ady denote the loss weights for Lioe, Leo and LG respectively.
4 Experiments
4.1 Experimental Settings
Our experiments mainly focus on face datasets. We use the FFHQ dataset [20] and 100K pairs of synthetic data for training. The synthetic pairs {15ynth Ieamth } are generated from EG3D [5], sharing the same latent code Waynth but rendered with different camera poses. To evaluate the generalization ability of our method, employ the CelebA-HQ dataset [19] and the multi-view MEAD dataset [40] for testing. We preprocess the images in the datasets and extract their camera poses in the same manner [5].
Datasets.
Implementation Details. For all experiments, we employ the EG3D [5] generator pre-trained on FFHQ. For the 3D GAN inversion encoder E,,,, we set the batch size to 4 and train it for 500K

---

Table 1: Comparisons with state-of-the-art methods on the CelebA-HQ and MEAD datasets.
Ce
MEAD
Category
Method
LPIPS
F
1D
Time (s)
FIDI
ID1
4:30
160
1:30"
1600
1:30
1.600
SG2 W
26.09
0.7369
0.2910
0.3372
39.30
64.47
0.7992
0.7533
43.72
PTI
25.70
0.7616
0.2771
0.3341
44.23
66.00
0.8089
0.7582
62.65
Optimization
Pose Opt.
29.04
0.7500
0.2990
0.3428
52.25
73.23
0.7954
0.7405
91.60
HFGI3D
24.30
0.7641
0.2775
0.3494
51.24
79,81
0.8019
0,7370
264.5
põp
38.46
0.7375
0.3116
0.3720
65.21
94.34
0.7900
0.7401
0.05430
Encoder
GOAE
35.41
0,7498
0.2818
0,3453
59.69
86.23
0.8109
0.7370
0.07999
Triplanenet
32.65
0.7706
0.3379
0,4103
76.62
130.55
0,8059
0,7135
0.1214
Ours
19.12
0.7882
0.2490
0.M40
38.15
64.01
0.8315
0.7741
0,08390
Input
SG2W*
Pose Opt.
HEGID
pSp
Triplanenet Dual Encoder
Figure 3: Comparisons of novel view synthesis on the CelebA-HQ dataset between our WarpGAN and several state-of-the-ar methods.
iterations on the FFHQ dataset. We use the Ranger optimizer, which combines Rectified Adam [25] with the Lookahead technique [47], with learning rates of le-4 for Ew.. The values of X2, ALPIPS set the batch size to 2 and train it for 300K iterations on both the FFHQ dataset and synthetic data pairs. For the novel view camera sample from the camera poses of the pose-rebalanced FFHQ dataset [5]. We use the Adam optimizer [22], with learning rates of 1e-3 and le-4 for the SVINet and discriminator, respectively. The values of Xy Xp, and An in Eq. (6) are set to 10.0, 30.0, and 0.1, respectively. The values of Avess de, and Aadv in Eq. (10) are set to 1.0, 0.1, and 10.0, respectively.
and A in Eq. (3) are set to 1.0. 0.8, and 0.1. For SVINet,
poses during the training process,
Baselines. We compare our WarpGAN with several 3D GAN inversion methods, including optimization-base methods (such as SG2 W+ [1], PTI [34], Pose Opt. [23], and HFGI3D [43]) and encoder- based methods (such as pSp [33], GOAE [46], Triplanenet [3], and Dual Encoder [4)). Note that Dual Encoder employs a 3D GAN other than EG3D and removes the background during training. This is difffrrnnt from our experimental setup, only compare it in the qualitative analysis.
Evaluation metrics. We perform novel view synthesis evaluation on the CelebA-HQ dataset and the MEAD dataset. For the CelebA-HO dataset, we compute the Fréchet Inception Distance (FID) [17] and ID similarity [12] between the original images and the novel view images. For the multi-view MEAD dataset, each person includes five face images with increasing yaw angles (front, +30% and +.60). We use the front image as input and synthesize the other four views. We then compute the LPIPS [48], FID, and ID similarity between the synthesized images and their corresponding ground-truth images. The inference times (Time) in Table are measured on single Nvidia GeForce RTX 4090 GPU.
7

---

RE0
Input
R30
130
Inpul
R30
181
SG2w!
Pose Opt.
pSp
Triplaneet
Ours
Figure 4: Comparisons of different methods the MEAD dataset for synthesizing images of the other four views (R60, R30, L30, and L60) using the front image as input.
4.2 Comparisons with State-of-the-Art Methods
Quantitative Evaluation. As shown in Table we provide the performance of different methods on the CelebA-HQ dataset and the MEAD dataset. It can be clearly observed that optimization based methods achieve better performance than recoder-based methods, but at the cost of significantly higher inference times. Among them, HFGI3D, which performs optimization twice using PTI (once for filling the occluded regions of warped images and once for multi-view optimization), shows substantial performance improvement but suffers from slow inference times. In contrast, our WarpGAN, which an inference time comparable encoder-based methods, surpasses the performance of optimization-bias based methods. The excellent performance the MEAD dataset demonstrates that our method capable effectively preserving multi-view consistency.
Qualitative Evaluation. We provide visualization results of novel view synthesis in Fig. 3 and Fig. 4. By successfully integrating the warping-ant and-aandnn aad-inpannting strategy into 3D GAN inversion, our method can better preserve facial details and generate more reasonable occluded regions. Moreover, our method is capable of maintaining 3D consistency in novel views more naturally.
4.3 Ablation Studies
Table 2: Ablation on different compo-
To investigate the contributions of key components in method, we conduct ablation studies. Table 2, we com- pare the quality of novel view synthesis using different model variants on the CelebA-HQ dataset.
nents of our WarpGAN.
Name
Model
FID
ID
A
Exst
36.07
0.7437
B
w/o SVINet
29.28
0.7735
Comparing "B" and "G" clearly demonstrates the sig- nificant role of SVINet inpainting occluded regions.
C
w/o Mod,++ & Le
19.71
0.7879
w/o Mod..t
19.47
0.7880
w/o symmetry
20.04
0.7825
Comparing "C", "D" and "G" shows that modulating the
synth data
0.7880
convolutions of SVINet with = and incorporating L. enhance the performance of our method. Comparing "E"
Full Model
19.12
0.7882

---

Input
E
Fall Model
Ju 50
you S
Iaput
Warpool
Degih
Novel Views
Novel Views
Novel Views
Novel Views
Engut
HIROHD
(c)
Giossus
Amper
Ol
Young
Figure 5: (a) Qualitative comparisons of the Full Model with model variants "C", "D", and "E": (b) Some failure cases; (c) Comparisons of image attribute editing effects with PTI and HFGI3D.
and "G" indicates that leveraging facial symmetry prior helps generate occluded regions in novel views. Comparing "F" and "G" reveals that training with synthetic data slightly improves the quality of novel view synthesis. We also qualitatively compare "C", "D", "E", and "G" (Full Model) in Fig. 5(a). Incorporating the latent code to control the inpainting process of SVINet and the symmetry prior can provide more information, reduce blurring and artifacts, and generate more detailed results.
4.4 Editing Application
Since our WarpGAN achieves novel view synthesis by inpainting warped images, the visible parts of the novel view images are minimally affected by the latent code. Consequently, manipulating the latent code alone does not enable attribute editing of the image. To address this issue, similar to HFGI3D [43], we utilize WarpGAN to synthesize a series of novel view images, which are then fed into PTT [34] for optimization. This process yields an optimized latent code w and fine-tuned 3D GAN generator. In this way, attribute editing of the input image and novel view rendering can be achieved by editing Wont [15, 31, 36] and modifying the camera pose c. As shown in Fig. 5(c), we perform attribute editing on the input image for four attributes: "Glasses", "Anger", "Old", and "Young", and compare the results with those from PTI and HFGI3D. It can be observed that the edited images obtained by using multi-view images synthesized by WarpGAN for optimization assistance exhibit higher fidelity and appear more natural.
5 Conclusion
In this paper, motivated by the achievement of the warping-anddinpainiingg strategy in 3D scene generation, we successfully integrate image inpainting with 3D GAN inversion and propose a novel 3D GAN inversion method, WarpGAN, for high-quality novel view synthesis from a single image. Our WarpGAN consists of a 3D GAN inversion network and SVINet. Specifically, we first obtain the depth of the input image using 3D GAN inversion, then apply depth-based warping to the input image to obtain the warped image, and finally use SVINet to fill the occluded regions of the warped image. Notably, our SVINet leverages symmetry prior and the latent code for multi-view consistency inpainting. Extensive qualitative and quantitative experiments demonstrate that our method outperforms existing sfate-of-the-arn optimization based and encoder-based methods.
Limitations. Due to the inevitable errors the depth map [11, 30, 35), the warped image sometimes become unreliable, which in turn prevents our SVINet from eliminating such artifacts. As illustrated in Fig. 5(b), when the angle variation is small, SVINet can alleviate the deformation of the eyes. However, as the angle of change increases, the output of SVINet deteriorates.

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grant 62372388 and Grant U21 A2051,,,, the Major Science and Technology Plan Project on the Future Industry Fields of Xiamen City under Grant 350222024102) 1029 and Grant 3502220241027, and the Fundamental Research Funds for the Central Universities under Grant 20720240076 and Grant XYGX2021J004
References
[1] R. Abdal, Y. Qin, and P. Wonka. Image2StyleGANrr: How to edit the embedded images? I Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8296 8305, 2020.
[2] S. An, H. Xu, Y. Shi, G. Song, U. Y. Ogras, and L.. Luo. Panohead: Geometry-aware 3d full-head synthesis in 360deg. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[3] A. R. Bhattarai, M. Nießner, and A. Sevastopolsky. Triplanenet: An encoder for eg3d inversion. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 2024.
[4] B. B. Bilecen, A. Gökmen, and A. Dundar. Dual encoder GAN inversion for high-fidelity 3d head reconstruction from single images. Advances in Neural Information Processing Systems, pages 87357- 87385, 2024.
[5] F. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recoggittion, pages 16123-16133, 2022.
[6] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, pi-GAN Periodic implicit generative adversarial networks for 3d-aware image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5799-5809, 2021.
[7] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive leamiin. arXiv preprint arXiv:2003.0 042997 2020.
[8] L. Chi, B. Jiang, and Y. Mu. Fast fourier convolution. Advances in Neural Information Processing Systems, pages 4479 4488, 2020.
[9] Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8188-8197, 2020.
[10] T. Chu, J. Chen, J. Sun, S. Lian, Z. Wang, Z. Zuo, L. Zhao, W. Xing, and D. Lu. Rethinking fast fourier convolution in image inpainting. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 23195-23205, 2023.
[11] J. Chung, S. Lee, H. Nam, J. Lee, and K. M. Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311. 233.23331.33384 2023.
[12] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4690- 4699, 2019.
[13] 1. Goodfellow. J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems. 2014.
[14] J. Gu, L. Liu, P. Wang, and C. Theobalt. StyleNeRF: A style-based 3d-aware generator for high-resolution image synthesis. In Proceedings International Conference on Learning Representations, 2022.
[15] E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. GANSpace: Discovering interpretable GAN
controls. Advances in Neural Information Processing Systems, pages 9841-9850 2020.
[16] K. Hc, X. Z.aan, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge a local nash equilibrium. Advances in Neural Information P Processing Systems, 30, 2017.
10

---

[18] J. T. Kajiya and B. P. Von Herzen. Ray tracing volume densities. ACM SIGGRAPH Computer Gruphics. pages 165-174, 1984.
[19] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:/710. 10196, 2017.
[20] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[21] T. Karras, S. Laine, M. Aittala, Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110 8110-8119, 2020.
[22] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
[23] Ko, K. Cho, Choi, K. Ryoo, and S. Kim. 3d GAN inversion with pose optimization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2967-2976, 2023.
[24] W. Li, Z. Lin, Zhou, L.. Qi, Y. Wang, and J. Jia. Mat: Mask-aware transformer for large hole image inpainting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10758-10768, 2022.
[25] L. Liu, H. Jiang, He, W. Chen, X. Liu, Gao, and J. Han. On the variance the adaptive learning rate and beyond. In Proceedings of International Conference on Learning Representations, 2020.
[26] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[27] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, pages 99-106, 2021.
[28] S. Niklaus and E. Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5437-5446, 2020.
[29] R. Or-El, X. Luo, M. Shan, E. Shechtman, J. J. Park, and I. Seneemmacher-Shizemmaa StyleSDF: High- resolution 3d-consistent image and geometry generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13503-13513, 2022.
[30] H. Ouyang, K. Heal, S. Lombardi, and T. Sun. Texf2immersion: Generative immersive scene with 3d gaussians. arXiv preprint arXiv:2312.0222: 2023.
[31] O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski. StyleCLIP: Text-driven manipulation of StyleGAN imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2002 2021.
[32] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, 2018.
[33] E. Richardson, Alaluf, Patashnik, Y. Nitzan, Y. Azar, S. Shapiro, and D. Cohen-Or. Encoding in style:
A StyleGAN encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2287-2296, 2021.
[34] D. Roich, R. Mokady, A. H. Bermano, and D. Cohen-Or, Pivotal tuning for latent-based editing of real images. ACM Transactions on Gruphics, pages 1-13, 2022.
[35] Seo, K. Fukuda, T. Shibuya, T. Narihira, N. Murata. Hu, C.-H. Lai, Kim, and Mitsufuji. Genwarp: Single image to novel views with semantic-preservin generative warping. Advances in Neural Information Processing Systems, 2024.
[36] Y. Shen, Yang. X. Tang, Zhou. InterFaceGAN: Interpreting the disentangled face representation learned by GANs. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2004-2018, 2020.
[37] R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov. N. Kong, Goka, K. Park, and V. Lempitsky. inoolution-robuse large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2159-2159, 2022.
11

---

[38] O. Tov, Y. Alaluf, Y. Nitzan, O. Patashnik, and D. Cohen-Or. Designing encoder for StyleGAN image manipulation. ACM Transactions Gruphics, pages 1-14, 2021.
[39] A. Trevithick, M. Chan, T. Takikawa, U. Igbal, S. De Mello, M. Chandraker, R. Ramamoorthi, and K. Nagano. What you see is what you GAN: Rendering every pixel for high-fidelity geometry in 3d GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27765-2777, 2024.
[40] Wang, Q. Wu, Song. Yang. Wu, Qian, R. He, Y. Qiao, and C. Loy. Mead: A large-scale audio-visual dataset for emotional talking-facc face generation. In Proceedings of European Conference on Computer Vision, pages 1000-17,220 2020.
[41] Y. Wu, J. Zhang. H. Fu, and X. Jin. Lpff: A portrait dataset for face generators across large poses. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20327-20333 2023.
[42] W. Xia, Y. Zhang. Y. Yang, J.-H. Xue, B. Zhou, and M.-H. Yang. GAN inversion: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 3121-3138. 2022.
[43] J. Xie, H. Ouyang, J. Piao, C. Lei, and Q. Chen. High-fidelity 3d GAN inversion by pseudo-multi-ieww optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 321-331 331, 2023.
[44] Y. Xu, Z. Shu. C. Smith, S. W. Oh, and J.-B. Huang. In-n-out: Faithful 3d GAN inversion with volumetric dccomposition for face editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7225-7235, 2024.
[45] F. Yin, Y. Zhang, X. Wang. T. Wang. X. Li, Y. Gong. Y. Fan, X. Cun, Y. Shan, C. Oztireli, et al. 3d GAN inversion with facial symmetry prior. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 342-351, 2023.
[46] Z. Yuan, Y. Zhu, Y. Li, Liu, and C. Yuan. Make encoder great again 3d GAN inversion through geometry and occlusion-awerd aware encoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2437-2447, 2023.
[47] M. Zhang, J. Lucas, J. Ba, and G. E. Hinton. Lookahead optimizer: k steps forward, step back. Advances in Neural Information Processing Systems, 32, 2019.
[48] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the proposed WarpGAN method, its components, and the improvements in novel view synthesis, accurately reflecting the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
Our paper does not involve theoretical results or their proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the network architecture and the training strategy in Sec. 3, the dataset usage and hyperparameter settings in Sec. 4.1, and additional details in the Appendix, which fully disclose the information needed reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We submit the code in the supplementary material, and all the datasets used are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wwtpublcc//aeeccccccccccccccecct mabbic/ggiids///// /Coo//eoeeeeeeeee wwwtuubmisssioni ssiissolioyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide all the training and test details, including datasets, implementation details, baselines, and evaluation metrics in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our task is image generation rather than prediction. Same as existing 3D GAN inversion methods, we use metrics such as FID, ID, and LPIPS, which do not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the execution times for various methods in Table and specify the computational resources used in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tprrip///uuis..... cc Caublic/Ethicsuuiie
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes. Yes]
Justification:
We discuss the societal impacts in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe the potential risks in the Appendix.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly cite all the papers involved in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper neither involves crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy fttpp::/neeuips... httpr//ps..c/ooofeeeeee//////55/ for what should or should not be described.
19

---

A Additional Architecture Details
Detailed Structure of SVINet. LaMa [37] introduces fast Fourier convolutions (FFCs) [8] into image inpainting, achieving a receptive field that covers whole image even the early network layers. Such way can facilitate the inpainting of large missing areas. To effectively fill in the occluded regions of the warped image, our SVINet built upon the framework LaMa and consists of three sub-networks: Ng. N,. and D. For input image with the size of 512 512, NE includes 3 downsampling convolutional layers that downsample the imput image to a feature map with the size of 64 x 64; N, contains 9 FFC residual blocks, each of which consists of two FFCs and residual connection, for inpainting, and ND consists of 3 upsampling convolutional layers to upsample the image resolution back the size of 512 512. The convolutions in N₁ and ND modulated by the latent code w from the 3D GAN inversion encoder E Note that, each FFC contains three convolutional branches and one spectral transform branch, and the convolutions within the spectral transform are also modulated, as shown in Fig. 6.
w+
w
Conv 1x1
Conv 3x3
FFT
Conv 3x3
Conv 1x1
Local
Local
Inv FFT
Global
Conv 3x3
Global
Spectral
Transform
Conv 1x1
Fast Fourier Convolution
Spectral Transfom
Figure The detailed structure of fast Fourier convolution modulated by the latent code w
B Additional Implementation Details
B.1 Principles of Neural Radiance Fields
Neural Radiance Fields (NeRF) [27] employs a fully-connected deep network, which maps a 3D spatial location and viewing direction d to color and density represent scene. By querying x and along camera rays and applying classical volume rendering techniques [18], the color and density information can be projected into 2D image. Specifically, for each projected ray corresponding
given pixel, N₂ points (denoted {t, (m)) sampled along the ray. For each sampled point, the estimated color and density are represented as c; and 0, respectively. The RGB value (rr) for each ray then be computed via volumetric rendering follows:
C(r) = Σ T₁(1 exp(-o,6;))c;:
(11)
where T, exp(- o, and 6, t+1 t, denotes the distance between adjacent samples.
Similarly, if we replace the color c; of each sampled point with the distance 6; from the sampling point to the camera during volumetric rendering, the depth d(r) along each ray can be obtained as
d(r)= = Σ T:(1 exp( exp(-o08.))
(12)
20

---

B.2 Multi- View Optimization for Editing
Our WarpGAN synthesizes novel view images not only based on the results of 3D GAN inversion but also relies on the warping results of the input image. Thus, only modifying the latent code within our method is difficult to achieve desirable editing effects. Inspired by HFGI3D [43], we employs
WarpGAN to generate N novel view images {T: ) corresponding to N different camera poses {<i) to assist the optimization process of PTI [34], denoted as WarpGAN-Opl.
Specifically, for single input image I with the camera pose we first employ an optimization-base
GAN inversion method [1] to jointly optimize the latent code w and the noise vector n the 3D
GAN generator:
w opt n = arg min L2(R(G(w (nnol::t,;;,,... c), + Cn)
(13)
wt n
where C, is noise regularization term and An is hyperparameter [34].
Subsequently, fix the optimized latent code wapt and fine-tune the 3D GAN generator based on the input image I and a series of novel view images synthesized by our WarpGANN
arg min La Lo(R(G(w opt 0),c), + Anc Σ Co((GG(( opt: C), L)),
(14)
= + ALPPSS
(15)
where Ame set to 1.0; both Xg and MipiPS are set to 1.0.
After the aforementioned process, obtain the optimized latent code w and the 3D GAN generator with tuned weights 0* To generate attribute-edited images from different viewpoints, we simply modify Tppt [31, 36], specify the desired camera pose Gnovct, and feed them into the 3D GAN to obtain the edited image Tedit in the novel view, that is,
fedit novel R(G + oMatt; O*), Crovel),
(16)
where Natt denotes a specific direction for attribute editing and a is a scaling factor.
C Broader Impacts
Our proposed method, which enables novel view synthesis and attribute editing of faces from single image, holds the potential to significantly impact various fields such as film, gaming, augmented reality (AR), and virtual reality (VR). However, it also raises concerns regarding privacy and ethics, particularly the risk of generating "deep fakes". We emphasize the necessity of implementing robust safeguards to ensure the responsible and ethical application of this technology, thereby minimizing the risk of misuse.
D Additional Qualitative Results
Additional Qualitative Evaluation. We provide more visual comparisons between our WarpGAN and several state-of-the-art methods in Fig. 7. In addition, since we utilize multi-view images synthesized by WarpGAN to assist 3D GAN inversion optimization for editing, we also include comparisons with this optimization-based method (WarpGAN-Opt) We can see that, due to the limitations of the low bit-rate latent code, WarpGAN-Opt loses some detail compared with WarpGAN, However, by leveraging the high-quality novel view images synthesized by WarpGAN, WarpGAN- achieves higher fidelity and realism in novel view synthesis than other optimization- From the figure, it can be observed that our method outperforms Dual Encoder [4]. However, since our method relies on the visible regions of the input image in the novel view to inpaint occluded regions, our method degrades to typical encoder-based 3D GAN inversion when the view change large and the visible region is small. In contrast, Dual Encoder focuses on high-fidelity 3D head reconstruction and thus offers greater flexibility in terms of view changes.
methods.
Additional Attribute Editing Results. To more comprehensively demonstrate the capability of our method in image attribute editing, provide additional attribute editing results in Fig. 8. Specifically,
21

---

Input
S
PTT
Pose Opt.
HPGDD
pSp
GOAE
Triplanenet Dual Encoder
Ours
Ours-Opt
Figure 7: Qualitative comparisons between our WarpGAN and several state-of-the-art methods.
we employ InterFaceGAN [36] for editing the "Anger", "Old", and "Young" attributes, and utilize the text-guided semantic editing method StyleCIIP [31] for editing the "Elsa" and "Surprised" attributes.
Reference-Based Style Editing. our WarpGAN, the latent code plays crucial role controlling the inpainting process of SVINet. To more explicitly analyze the influence of the latent code,
22

---

Elsa
Elsa
Import
TriDut
Anger
Old
Young
Supprsse
Elsa
Elsa
Input
Input
Anger
Anger
Old
Old
Your I
You!
Suppics
Suppised
Bisa
Bssa
Imput
Input
Anger
Old
Yount
Youg Sumpised
Elsa
Flsa
Imput
Imput
Anger
Okd
PODIIING X
Figure 8: Image attribute editing results obtained by our method. The edited attributes include "Elsa", "Anger", "Old", "Young", and "Surprised".
23

---

Referrnce
Source
Figure 9: Reference- based style editing, Each row represents the editing results of the same source image corresponding to different reference images, where the source and reference images are identical along the diagonal.
perform experiments by replacing the latent code of the input image during the inpainting process. Specifically, for the source image I, with the camera pose C, and the latent code w we replace them with the camera pose G, and the latent code w. of the reference image I, during inpainting, thereby achieving simultaneous editing of view and style. The results given in Fig. 9.
For our SVINet, t modulates
convolutions in both N, and ND, where N, processes feature maps at resolution of 64 x 64, and ND processes feature maps at resolutions ranging from 64 x 64 to 512 x 512. According to the characteristics of StyleGAN [20, 21], the latent code corresponding to feature maps at resolutions of 64 x 64 and above primarily controls the detailed features of the image, such as the color scheme and microstructure. From Fig. 9, we observe that the main changes in the skin tone and hair color of the face.
To further validate the generalization capability of our method, we evaluate it in the cat domain. Specifically, we use the AFHQ-CAT dataset [9] for training and evaluation. Following e4e [38], we use a ResNet50 network [16] trained with MOCOv2 [7] instead of the pre-trained ArcFace network [12] to compute the identity loss in the non-facial domains during training. As shown in Fig. 10, our method can generalize well to the cat domain and perform novel view synthesis well as attribute editing.
Qualitative Evaluation in the Cat Domain.
24

---

Input
WarpGAN
WarpGAN-O Opt
Color
Small Eyes
Figure 10: Novel view synthesis and attribute editing on cat faces by our method. We visualize the novel view synthesis results of WarGAN and WarpGAN-Opt, as well as the editing results of the attributes "Color" and "Small Eyes".
25


Reviews:
[{"review_id": "rDozH9ZXop", "reviewer": "Reviewer_bJgh", "summary": "The paper tackles the long‑standing problem of poor occluded‑region quality in single‑image 3D‑aware GAN inversion. It combines depth‑based warping (to reuse pixels that are already visible) with a dedicated style‑based 2‑D inpainting network (SVINet) that is conditioned on (1) the latent code produced by an EG3D encoder and (2) a symmetry prior obtained from a horizontally flipped copy of the input face. Training mixes real single‑view images with synthetic multi‑view pairs rendered from EG3D. Quantitative results on CelebA‑HQ and on MEAD show reasonable gains over eight published baselines, and inference is as fast as other feedforward. Ablations support the contribution of each design choice. The authors also demonstrate that the synthesized multi‑view set can be fed to PTI to obtain superior latent‑space editing.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n-- The paper solves a clear problem and is well-motivated. 3D GAN inversion does tend to hallucinate unseen regions, the paper addresses an interesting problem.\n\n-- The warping and inpainting approach is reasonable, since the depth maps come out for free from EG3D, and conditioning SVINet on the latent code definitely improves multi-view consistency.\n\n-- The improvements are significant with limited training budget.\n\n\nWeakness:\n\n-- This might go back to the old \"novelty\" problem, but I do feel the novelty of this paper is incremental relative to existing “warp + inpaint” pipelines. The idea has appeared in GenWarp (NeurIPS’24) and LucidDreamer (ICCV’24), and In‑N‑Out (CVPR’24) already mixes triplane editing with an occlusion‑aware decoder. The paper does not position itself clearly against these very closely related works.\n\n-- The evaluation domain is kind of limited, since it is face only.\n\n\nOverall, the paper offers a practically useful recipe that materially improves facial 3D inversion quality at encoder‑level speed. However, its methodological novelty over other warp‑and‑inpaint works is moderate, the evaluation is narrowly scoped to faces. It is a decent paper, but the contribution may be judged as incremental for NeurIPS.", "was_revised": true, "final_justification": "I do not have much problems with the paper initially, and would keep my positive score.", "rebuttal": ">**Q1**: Novelty.\n\n**R1**: Existing \"warping-and-inpainting\" methods cannot be directly used in 3D GAN inversion (lines 40-42). **We are the first to successfully apply such a paradigm to 3D GAN inversion. We innovatively design a warping-back strategy and introduce a symmetry prior along with latent codes from a 3D-aware GAN, greatly alleviating multi-view inconsistency.**\nThis enables novel-view synthesis of a human face from a single image, achieving performance superior to optimization-based methods with the inference time comparable to encoder-based methods. Hence, our work clearly showcases the great potential of the \"warping-and-inpainting\" paradigm in 3D GAN inversion.\nWe believe that the fast inference time, together with the good performance of the framework, will benefit and ease future research on novel-view synthesis of human faces.\n\n---\n\n>**Q2**: Comparisons with SOTA Methods.\n\n**R2**: Our method is intrinsically different from GenWarp (NeurIPS'24) and LucidDreamer (ICCV'24), which are specifically designed for **novel-view synthesis of scenery**; in contrast, our work is object-centric, focusing on **novel-view synthesis of human faces.** Moreover, their training datasets and problem settings differ from ours. Hence, these methods cannot be directly transferred to our task.\n\nWe provide an extra comparison with In‑N‑Out (CVPR'24) on the CelebA-HQ dataset. The authors of In‑N‑Out evaluated their method on a custom dataset that includes the mask information marking out-of-domain regions. To make a fair comparison with our method and other baselines, we apply In‑N‑Out to the CelebA-HQ dataset, where **it achieves an FID value of 60.41 and an ID value of 0.6744, indicating poor performance.**\nOne possible reason is that CelebA-HQ does not have annotations for marking out-of-domain regions. Therefore, during optimization, we use the method (provided by the authors) that does not include the mask in the loss calculation. This may directly cause the additionally optimized tri-plane to fail to reconstruct out-of-domain regions well, leading to a performance drop.\n\nIn addition, following the suggestion of **Reviewer bp2G**, we have also added the performance comparison with a representative diffusion-based method, DiffPortrait3D [1], on the CelebA-HQ dataset. The FID value is 19.26 and the ID value is 0.7964 for DiffPortrait3D, while our WarpGAN achieves an FID value of 19.12 and an ID value of 0.7882.\n**The metrics indicate that our method can achieve performance comparable to diffusion-based methods in the real human face domain.**\n\nMoreover, due to the uncertainty in the generation of 2D diffusion models and their lack of an explicit 3D representation, multi-view inconsistency is a common issue.\nTo address this problem, DiffPortrait3D proposes a view-consistency module that processes multiple novel-view images simultaneously to improve the consistency of the generated novel-view images. However, such a way requires a significant amount of GPU memory.\nThe authors of DiffPortrait3D set the number of novel-view images to 8 in implementation, which results in the need for inference on at least a single A100.\nDue to the limitation on the number of images processed in parallel, **flickering artifacts can still be observed in the multi-view rendering animations provided in the DiffPortrait3D demo**.\nIn contrast, our WarpGAN, benefiting from the multi-view consistency prior of 3D GANs as well as the symmetry prior and latent code from the 3D-aware GAN that we introduce, can easily generate **multi-view consistent rendering videos**. We will provide a demo alongside our open-source code.\n\nIt is worth noting that our method can train and infer on consumer-grade GPUs (such as the 3090 and 4090), with inference times on a single 4090 GPU **within milliseconds.** In contrast, DiffPortrait3D requires training on multiple professional-grade GPUs and inference on a single professional-grade GPU, with inference times exceeding **1 minute** on a single A100 GPU.\n\n\n---\n\n>**Q3**: Limited Evaluation Domain.\n\n**R3**: We follow state-of-the-art 3D GAN inversion methods (e.g., Triplanenet (WACV'24), In-N-Out (CVPR'24), Dual Encoder (NeurIPS'24)) to validate our method in the human face domain. We agree with you that evaluating our method in other domains is a worthwhile endeavor. Note that EG3D provides a pre-trained model for cats. Compared with faces, which contain rich details, cat reconstruction is relatively simpler. Therefore, **we believe our method, in theory, can perform comparably well on cats as it does on faces.** We will supplement this evaluation in the Appendix.\n\n---\n\n**References**:\n\n[1] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis. CVPR 2024."}, {"review_id": "A48Rh7uZzV", "reviewer": "Reviewer_bFFc", "summary": "This paper presents a novel method, WarpGAN, aimed at addressing the occlusion problem in the task of 3D GAN inversion. The authors first employ an encoder to project the input image into a latent code in the w space, which is then fed into a 3D GAN to synthesize views. A carefully designed warping strategy is subsequently applied to generate novel views along with corresponding synthetic depth maps. Finally, an inpainting network is introduced further to enhance the visual fidelity of the generated views. Experimental results demonstrate the effectiveness of the proposed approach.", "questions": "Please refer to the Strengths And Weaknesses section.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is well-written and easy to follow.\n\n2.\tThe use of the warping-and-inpainting strategy to handle the occlusion problem is an interesting and effective design choice.\n\n3.\tWarpGAN achieves considerable improvements over baseline methods, demonstrating the effectiveness of the proposed approach.\n\nWeaknesses:\n\n1.\tWhile the authors claim to address the occlusion problem, the experimental section lacks a specific analysis of how occlusion affects the performance of baseline methods. For example, does occlusion lead to artifacts in facial regions such as the ears or side profile? It would strengthen the paper if the authors explicitly highlighted the impact of occlusion on visual quality and reconstruction accuracy.\n\n2.\tThe authors generate synthetic training samples, but as shown in [1], naively sampling from EG3D can produce unstable results. Therefore, more details about the sampling strategy are needed to assess the reliability and quality of the training data.\n\n3.\tThe organization of the paper could be improved. For instance, Section 3.1 currently focuses heavily on warping. It may be clearer to introduce a dedicated subsection for the warping module and keep the overview section as a concise summary of the entire pipeline.\n\n\n[1] G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images. CVPR 2024.", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1**: Impact of Occlusion.\n\n**R1**: We evaluate our method against baseline methods from both quantitative and qualitative perspectives to clearly reveal how occlusion affects each method. Owing to space limitations, the original paper offers only a brief discussion. Here we provide a more complete analysis.\n1) **Quantitatively**, FID reflects the overall quality of the synthesized novel views, while ID measures the identity preservation and the plausibility of the generated occluded regions to some extent. The LPIPS metric on MEAD further reflects the reconstruction fidelity of novel views and, indirectly, the accuracy of the occluded areas. Across these metrics, our method achieves the best results, suggesting that baselines produce less realistic occlusions.\n2) **Qualitatively**, extensive visual comparisons are presented in Fig. 3 and Fig. 4 of the main paper, as well as Fig. 4 in the supplementary materials. They demonstrate that for regions unobserved in the input (e.g., ears or side profiles), baselines tend to yield blurry artifacts or unnaturally flattened heads, whereas our WarpGAN generates occluded facial regions that are both more coherent and richer in detail.\n\n---\n\n>**Q2**: Sampling Strategy for Synthetic Samples.\n\n**R2**: To facilitate training and enhance multi-view consistency, we leverage the powerful generative capacity of EG3D to generate multi-view-consistent synthetic images as additional supervision.\nG-NeRF claims that regions of lower density in the latent space of StyleGAN may be inadequately represented, leading to synthetic images with degraded geometry quality.\nAfter checking the synthetic images used in our training, we did not observe the fidelity or visual-appeal issues reported by G-NeRF. We attribute this to two key reasons.\n1) **EG3D checkpoint**:\n   As G-NeRF does not release the code for generating synthetic images, the exact details of their sampling strategy remain unknown. However, the EG3D checkpoint they provide was trained on the standard FFHQ dataset, whereas we use a checkpoint fine-tuned on a rebalanced FFHQ with a more uniform pose distribution. According to the EG3D authors, such a fine-tuned model produces better 3D shapes and better renderings from steep angles than the one trained on the standard FFHQ. We therefore consider this to be a reason for our observations.\n2) **Sampling ranges**:\n   For latent codes, we sample from a Gaussian distribution and then map to the $\\mathcal{W}$ space (as mentioned in the G-NeRF paper). For camera poses, because the FFHQ dataset has a limited pose distribution, sampling over a wider range can introduce artifacts. Therefore, our sampling range primarily focuses around the frontal view of the face: yaw and pitch are both drawn within approximately $\\pm 30^{\\circ}$ of the frontal view. Poor geometry quality typically appears at extreme angles, so our constrained sampling avoids the issues highlighted by G-NeRF.\n\n---\n\n>**Q3**: Organization of the Paper.\n\n**R3**: Thank you for your valuable feedback. We agree that introducing warping as a Preliminary at the beginning of the Methodology section will make our Overview more concise and improve the readability of the paper. We will revise our paper according to your suggestions."}, {"review_id": "PG9NR85c3H", "reviewer": "Reviewer_kyEW", "summary": "This paper proposes a warping-then-inpainting approach to synthesize multi-view images for facial images.  It first employs a EG3D-style GAN to do the 3D  inversion of  a single view facial image, produce its depth map, and then inpaint the warped facial image at the new viewpoint.  The proposed approach can also enable facial editing in the latent feature space.  The experimental results show the improvement of quantitative metrics, such as FID and ID.", "questions": "1. Does the row A indicate the result of 3D GAN  inversion?  Since there are recent studies on 3D facial image editing based on 3D GAN  inversion, I am curious why the results of 3D GAN inversion only are much worse than warp-then-inpaint.\n2. Since warp-then-inpaint procedure is widely used in diffusion-based methods,  what is the comparison between the proposed approach and these diffusion-based methods?  please also explain why diffusion-based models are not used.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.  It shows that warp-then-inpaint procedure can improve the quality of the novel view synthesis of facial images. Experimental results are impressive.\n2. The ablation study clearly shows the improvements by SVINet. \n\nWeaknesses:\nWhile the experimental results are impressive,  the technical components in the proposed approach are mainly adapted from already-known methods, such as EG3D, symmetry and ID  loss.  It downgrades the technical novelty of this paper.", "was_revised": true, "final_justification": "My concerns are addressed in the rebuttal. I maintain my borderline accept score as the final rating.", "rebuttal": ">**Q1**: Novelty.\n\n**R1**: Although the technical components are established techniques, we do not simply stack them. Instead, **we are the first to successfully apply the \"warping-and-inpainting\" paradigm to 3D GAN inversion, by innovatively designing a warping-back strategy and introducing a symmetry prior along with latent codes from a 3D-aware GAN, greatly alleviating multi-view inconsistency.** Our work clearly showcases the great potential of the \"warping-and-inpainting\" paradigm in 3D GAN inversion. \nQualitative and quantitative results confirm the effectiveness of our design.\nWe believe that the fast inference time, together with the good performance of the framework, will benefit and ease future research on novel-view synthesis of human faces.\n\n---\n\n>**Q2**: Poor Performance of 3D GAN Inversion.\n\n**R2**: Entry A in Table 2 reports results obtained directly from the initial novel-view images produced by 3D GAN inversion.\nRecent 3D GAN inversion methods fall into two categories: optimization-based and encoder-based. Optimization methods are time-consuming, whereas encoder-based methods suffer from low fidelity. To boost the fidelity of encoder-based methods, some works complement the high-dimensional $\\mathcal{F}$ space, yet GOAE (ICCV'23) shows that such a way introduces distortion in occluded regions.\nBecause the primary role of 3D GAN inversion in our method is to produce a depth map and provide an initial fill for occluded regions in the warped image, we adopt the most basic approach (Entry A): simply projecting the image into the low-dimensional $\\mathcal{W}$ space of EG3D. **This inevitably incurs substantial information loss, which in turn leads to lower fidelity in the novel-view images.** In contrast, our full model retains the visible regions from the original image and generates occlusions with SVINet, thereby bypassing the information loss inherent to the low-dimensional latent space and achieving much better results.\n\n---\n\n>**Q3**: Comparison with Diffusion-Based Methods.\n\n**R3**: The \"warping-and-inpainting\" strategy in diffusion models (such as GenWarp(NeurIPS'24) and LucidDreamer(ICCV'24)) is primarily designed for novel-view synthesis of scenery. In contrast, our work is object-centric, focusing on novel-view synthesis of human faces.\nMoreover, their training datasets and problem settings differ from ours. Hence, **these methods cannot be directly transferred to our task.**\n\nFollowing the suggestion of **Reviewer bp2G**, we have also added the performance comparison with a representative diffusion-based method, DiffPortrait3D [1], on the CelebA-HQ dataset. The FID value is **19.26** and the ID value is **0.7964** for DiffPortrait3D, while our WarpGAN achieves an FID value of **19.12** and an ID value of **0.7882**.\n**The metrics indicate that our method can achieve performance comparable to diffusion-based methods in the real human face domain.**\n\nMoreover, due to the uncertainty in the generation of 2D diffusion models and their lack of an explicit 3D representation, multi-view inconsistency is a common issue.\nTo address this problem, DiffPortrait3D proposes a view-consistency module that processes multiple novel-view images simultaneously to improve the consistency of the generated novel-view images. However, such a way requires a significant amount of GPU memory.\nThe authors of DiffPortrait3D set the number of novel-view images to 8 in implementation, which results in the need for inference on at least a single A100.\nDue to the limitation on the number of images processed in parallel, **flickering artifacts** can still be observed in the multi-view rendering animations provided in the DiffPortrait3D demo.\nIn contrast, our WarpGAN, benefiting from the multi-view consistency prior of 3D GANs as well as the symmetry prior and latent code from the 3D-aware GAN that we introduce, can easily generate **multi-view consistent rendering videos**. We will provide a demo alongside our open-source code.\n\nIt is worth noting that our method can train and infer on consumer-grade GPUs (such as the 3090 and 4090), with inference times on a single 4090 GPU **within milliseconds**. In contrast, DiffPortrait3D requires training on multiple professional-grade GPUs and inference on a single professional-grade GPU, with inference times exceeding **1 minute** on a single A100 GPU.\n\n---\n\n>**Q4**: No Diffusion Used.\n\n**R4**: We discuss this from three perspectives:\n1) Our work is a study of 3D GAN inversion, so we primarily consider using GAN-based methods to achieve novel-view synthesis of human faces from a single image, as done in SOTA 3D GAN inversion methods (e.g., Triplanenet (WACV'24), In-N-Out (CVPR'24), Dual Encoder (NeurIPS'24)).\nDiffusion models are trained on massive image corpora that inevitably cover the celebrities present in CelebA-HQ. **Hence, it would be unfair to compare them with 3D GAN-inversion methods.**\n2) 3D-aware GANs can provide strong facial priors and multi-view-consistent generation, together with a latent space that is semantically rich and highly disentangled. **We thus believe that advancing 3D GAN inversion itself is worthwhile.** Extensive experiments demonstrate that our WarpGAN delivers high-quality novel-view face synthesis, runs faster, and consumes fewer resources than diffusion-based alternatives.\n3) In future work, we believe that **incorporating diffusion models into our framework** is a promising research direction, such as attempting to use diffusion-based methods to replace SVINet for inpainting occlusion regions.\n\n---\n\n**References**:\n\n[1] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis. CVPR24."}, {"review_id": "gI2RfMeoDW", "reviewer": "Reviewer_rVfV", "summary": "This paper tackles the problem of low-quality novel view synthesis in single-image 3D GAN inversion, focusing on improving occluded regions that often suffer from missing details and a lack of realism. The authors propose WarpGAN, a novel method that integrates the classical \"warping-and-inpainting\" strategy into the 3D GAN inversion pipeline. Specifically, the method first warps the input image to a new viewpoint using a depth map obtained from the 3D GAN inversion, thereby preserving high-frequency details in visible areas. Then, it employs a style-based inpainting network, SVINet, which leverages facial symmetry priors and the latent code from the 3D GAN to realistically fill in the occluded regions while maintaining multi-view consistency. To overcome the lack of ground-truth data for real images, the authors introduce a \"re-warping\" strategy. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches.", "questions": "Please see the raised weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. Clear Motivation: The over-reliance on the 3D GAN's generative prior to fill occluded regions leads to poor generation quality. The Integration of the \"Warping-and-Inpainting\" framework is reasonable.\n2. Innovative SVINet Network: The symmetry prior and style-based modulation used in the SVINet and its training strategy is justified and convincing.\n3. Good results: Current experiments demonstrate the effectiveness of the proposed method.\n\n\nWeaknesses\n1. The core idea of integrating a \"warping-and-inpainting\" strategy with 3D GAN inversion is not new and has been previously explored by methods like HFGI3D. The authors argue that their key contribution lies in replacing the optimization-based inpainting of HFGI3D with a dedicated feed-forward network (SVINet), which is incremental. Therefore, the paper could be strengthened by more clearly articulating and demonstrating why its proposed feed-forward inpainting network is superior to a well-formulated optimization-based alternative, perhaps by highlighting scenarios where optimization-based methods fundamentally fail and only a dedicated, trained network can succeed.\n2. The method's generalization ability to more \"in-the-wild\" scenarios—such as extreme poses, exaggerated expressions, complex occluders, or non-face objects—remains to be validated. For example, the symmetry prior might not hold when a hand occludes the face.\n\n\nMinor:\n1. Line 216 “Datesets” typo", "was_revised": true, "final_justification": "I thank the authors' rebuttal, which addresses part of my concerns regarding the novelty of the proposed method. My comments about extreme cases still hold but I agree it is a challenge for the community. Therefore, I keep my positive rating and have no objection to accept.", "rebuttal": ">**Q1**: Incremental Innovation over HFGI3D.\n\n**R1**: WarpGAN is NOT an incremental innovation over HFGI3D.\n1) **HFGI3D is NOT a combination of the \"warping-and-inpainting\" strategy and the 3D GAN inversion.** The core idea of HFGI3D is to use warping operations to synthesize pseudo-multi-view images, where the occlusion regions of the warped images are simply filled by using the results of PTI. However, such a filling operation is coarse. Since PTI is optimized by using a single-view image and has low fidelity, it leaves obvious discontinuities when combined with the warped image (similar to the coarse result $\\hat{\\mathbf{I}}_{novel}^{\\text{initial}}$ from 3D GAN inversion, as shown in Fig. 2 of our paper). We believe that such a way is merely a simple fill rather than true inpainting, and the synthesized pseudo novel-view images cannot be regarded as ideal results. Hence, they still require a final optimization. In contrast, our SVINet can truly inpaint the warped image and directly generate natural and plausible occlusion regions.\n2) **Our method is NOT a simple replacement of optimization-based approaches with feed-forward methods.** The purpose of synthesizing pseudo-multi-view images in HFGI3D is to assist optimization.\nHowever, since PTI optimizes solely based on a single-view image, it is prone to generating novel-view images with poor geometry quality. Using PTI to fill in the warped image results in pseudo-multi-view images with inferior geometry quality.\nMoreover, inevitable errors in the depth map can make warped images unreliable under large view changes, thereby affecting the quality of pseudo-multi-view images. Consequently, low-quality pseudo-multi-view images can have a negative impact on the final optimization. In comparison, the occlusion regions synthesized by our SVINet are of higher quality than those filled by PTI and have the ability to correct unreliable warped images to some extent (as shown in Fig. 5(b)).\nAs shown in Fig. 5(c), the geometry quality of the novel-view images generated by WarpGAN-Opt is better than that of the results from HFGI3D. This indicates that the novel-view images synthesized by our WarpGAN have better quality than the pseudo-multi-view images generated by HFGI3D.\n\n---\n\n>**Q2**: Unvalidated Generalizability.\n\n**R2**: The extreme cases you mentioned, such as extreme poses, exaggerated expressions, complex occluders, or non-face objects, are indeed challenging for methods based on 3D GAN inversion. Existing methods generally struggle to perform well under these circumstances.\nWe discuss the generalizability of our method to these extreme cases here in principle and will supplement the performance of our method under these extreme conditions in the Appendix.\n\nFor **extreme poses**, although our method can generate high-quality novel-view images, there are certain limitations in the range of pose changes. We mentioned the limitations of our method in the Conclusion section of the paper. Due to inevitable errors in the depth map, the quality of the images generated by our WarpGAN will decrease when the camera pose changes significantly. More seriously, when the camera pose changes significantly, leaving no visible area in the novel view, our method will degrade to a typical encoder-based 3D GAN inversion (as mentioned in lines 76-79 of the Appendix).\n\nFor **exaggerated expressions and complex occluders**, which can be collectively referred to as out-of-domain regions in the GAN field, existing GAN inversion methods, limited by the low-dimensional latent space of GANs, struggle to reconstruct these out-of-domain regions effectively. However, the visible regions of the novel-view images synthesized by our method come from the original image, and the occlusion regions come from SVINet, which can bypass the information loss caused by the low-dimensional latent space and thus better preserve these out-of-domain regions.\n\nFor **non-face objects**, EG3D provides a pre-trained model for cats, and we will evaluate our method on cats in the Appendix. Faces are more complex than cats. Thus, in principle, our method should be able to generalize well to novel-view synthesis of cats.\n\nRegarding **symmetry-aware feature extraction**, as shown in Eq. (4), we do not directly add the features of the input image and its mirrored image. Instead, we use featurewise linear modulation (FiLM) to refine the features, which can be considered as a weighted fusion. When the input face image is significantly asymmetric, the symmetric weight will decrease. Similarly, Triplanenet (WACV'24), when using symmetry priors for loss calculation, introduces a per-pixel Gaussian density with a pixelwise uncertainty map that assigns lower confidence to the region in the mirrored image where the symmetry assumption fails. Therefore, our explanation is justified in theory, and we will also supplement our performance when the face prior does not work (such as in the case of ``hand covering face'' that you mentioned) in the Appendix.\n\n---\n\n>**Q3**: Typo.\n\n**R3**: Thank you for your attention to detail. We will correct the typo in the revision."}, {"review_id": "K0H75tX2rA", "reviewer": "Reviewer_bp2G", "summary": "This paper presents a method for novel view synthesis of face images, via inversion of an existing 3D-aware GAN (EG3D). It trains an encoder to map from input images to the GAN latent w in order to obtain a depth-map (by decoding and re-rendering); then warps the original image according to the depth map and target pose. Occluded regions are filled in by a custom two-stage in-painting process. The method is evaluated on CelebA-HQ and MEAD, where it shows stronger performance than existing 3D GAN inversion methods.", "questions": "Why are both inpainting stages needed? and how are their results combined? In particular, the first inpainting using re-rendered pixels from the inversion (described in eq 2) should result in a complete image; what then is left for the SVINet part to do? It seems the second inpainting is not really inpainting but doing an image-to-image tidying-up of the whole image? What would happen if the second stage were used without the first?\n\nDoes the feature-mirroring stage described in sec 3.3.1 presuppose that that the input image is posed symmetrically? What if this is not the case, e.g. a half-profile view is given as input?\n\nWhat exactly is Figure 3 showing? In particular, it claims to show NVS on CelebA-HQ, but the \"input\" column appears to contain synthetic images (see the blurring outside of the face region). If these images were in fact generated, this should be stated clearly (since it makes the inversion task easier, as they then definitely lie in the latent manifold of the GAN whereas real images even from the same dataset may not).", "limitations": "Yes – although limitations is rather brief", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe proposed approach is interesting and novel. Explicitly warping the source image ensures that details can be preserved even if they are challenging to encode and decode directly in the GAN's latent space, while still making use of its prior knowledge for depth estimation and occluded regions.\n\nThe inpainting stage is itself somewhat novel – it uses a stylegan-inspired architecture that also leverages facial symmetry in feature extraction.\n\nEvaluation covers two large-scale datasets (CelebA-HQ and MEAD); for CelebA-HQ, identity preservation is explicitly measured, as well as realism via FID; for MEAD, reconstruction is also measured directly via LPIPS.\n\nQuantitative results on both datasets are significantly stronger than the chosen baselines according to all metrics. The method is also faster than most of the baselines thanks to being feed-forward (encoder-based) rather than relying on iterative optimization.\n\nThe selected qualitative results also appear better than all the baselines, with rather fine details from the source image successfully transferred to the target image.\n\nThere is a fairly detailed ablation study, measuring the benefit of various design decisions and innovations.\n\n## Weaknesses\n\nThe evaluation only compares against other 3D GAN inversion methods. The comparisons should also include diffusion-based NVS methods (either 2D-only, or 3D-aware) in order to place the method properly in context.\n\nThere are no measures of spread (confidence intervals or standard deviations) on the experimental results; this makes it difficult to know if the results are statistically significant. In particular, on certain metrics the gains over the baselines are rather small.\n\nThere is no comparison of the full pipeline but with just the warping / rerendering stage replaced by simply using the rerendered inverted result. This comparison is somewhat important, since it justifies the key selling point of the paper, i.e. use of warping, in isolation from confounding factors such as how well the specific inpainting method works.\n\nThere is no comparison of the custom SVINet inpainting network against a modern diffusion image-to-image model.\n\nThe topic feels somewhat dated – while there is still a strand of literature pursuing inversion of EG3D and similar models as a route to novel view synthesis, the community has largely moved on to diffusion-based methods. This does not in any way affect the technical validity and rigor of the paper; however it does likely limit this work's interest to the community. The restriction to single-view datasets is rather artificial given multi-view datasets of heads are now available; moreover diffusion-based methods such as RenderDiffusion train in EG3D's single-image setting anyway.\n\nThe method's applicability is limited to tightly-cropped human heads. Not even EG3D's cats are included, though presumably these would also work in the same pipeline given they are symmetrical too.", "was_revised": true, "final_justification": "Following the rebuttal/discussion, I remain rather unenthused about this paper. The authors have addressed several of the points in my original review adequately, for which I am grateful; however the work still feels dated and not so relevant given modern diffusion approaches to single view reconstruction and novel view synthesis that also work well for broader domains that just the human faces shown here. I therefore keep a negative opinion -- I do not feel it is of sufficient interest to the neurips community. Nonetheless I agree the work is technically sound, and so may rating is only borderline negative.", "rebuttal": ">**Q1**: Comparison with Diffusion.\n\n**R1**: Our work is a study of 3D GAN inversion, so we primarily compare our method with other 3D GAN inversion-based approaches, as done in SOTA 3D GAN inversion methods (e.g., Triplanenet (WACV'24), In-N-Out (CVPR'24), Dual Encoder (NeurIPS'24)).\nFollowing your comments, we have also added the performance comparison with the diffusion-based method, DiffPortrait3D [1], on the CelebA-HQ dataset.\nThe FID value is 19.26 and the ID value is 0.7964 for DiffPortrait3D, while our WarpGAN achieves an FID value of 19.12 and an ID value of 0.7882. **The metrics indicate that our method can achieve comparable performance to diffusion-based methods on the real human face domain.**\n\nMoreover, due to the uncertainty in the generation of 2D diffusion models and their lack of an explicit 3D representation, multi-view inconsistency is a common issue.\nTo address this problem, DiffPortrait3D proposes a view-consistency module that processes multiple novel-view images simultaneously to improve the consistency of the generated novel-view images. However, **such a way requires a significant amount of GPU memory.**\nThe authors of DiffPortrait3D set the number of novel-view images to 8 in implementation, which results in the need for inference on at least a single A100.\nDue to the limitation on the number of images processed in parallel, **flickering artifacts** can still be observed in the multi-view rendering animations provided in the DiffPortrait3D demo.\nIn contrast, our WarpGAN, benefiting from the multi-view consistency prior of 3D GANs as well as the symmetry prior and latent code from the 3D-aware GAN that we introduce, can easily generate **multi-view consistent rendering videos.** We will provide a demo alongside our open-source code.\n\nIt is worth noting that our method can train and infer on consumer-grade GPUs (such as the 3090 and 4090), with inference times on a single 4090 GPU **within milliseconds.** In contrast, DiffPortrait3D requires training on multiple professional-grade GPUs and inference on a single professional-grade GPU, with inference times exceeding **1 minute** on a single A100 GPU.\n\n---\n\n>**Q2**: Measures of Spread.\n\n**R2**: Statistical significance is indeed crucial for reliable scientific conclusions.\nHowever, the calculation of confidence intervals or standard deviations is applied mainly to prediction tasks such as object detection and classification, where the ground truth is available.\nIn contrast, in the generative domain, there is usually no ground truth of the generated images. The goal is to synthesize new images that conform to certain data distributions, rather than making a decision on a single input. As a result, **most studies in the generative domain do not analyze statistical significance.**\n\n---\n\n>**Q3**: Small Gains.\n\n**R3**: Although our method shows only minor improvements over optimization-based baselines in certain metrics, our inference speed is significantly faster (over **500 times faster**), demonstrating the efficiency of our method.\n\n---\n\n>**Q4**: Comparison of the Full Pipeline.\n\n**R4**: Our method consists of two steps: warping and inpainting. We need to warp the input image to the target view first before using SVINet for inpainting. Therefore, **it is NOT feasible to conduct ablation studies separately for these two steps.**  \nRegarding the \"rerendering\" you mentioned, if you are referring to WarpGAN-Opt used in editing operations, we provide extensive visual comparisons and analyses in Fig. 4 of the Appendix.\n\n---\n\n>**Q5**: Comparison Between SVINet and Diffusion Models.\n\n**R5**: We experimented with the diffusion-based RePaint [2] to inpaint our warped images. Unfortunately, since the holes in our warped images are often large and complex (as shown in Fig. 2), **RePaint, which was not trained in our scenario, struggles to generalize to our task and frequently fails to inpaint the holes.** We also provide metrics on CelebA-HQ, with an FID of **60.79** and an ID score of **0.7390**, indicating poor performance. Additionally, it takes about **5 minutes** per image, which is much slower than our SVINet. However, we believe that modifying diffusion-based methods to replace our SVINet is a promising research direction, and we plan to explore this in the future.\n\n---\n\n>**Q6**: Outdated GAN.\n\n**R6**: Due to the powerful generative capabilities of diffusion models, which overcome issues such as domain limitations faced by GANs, an increasing number of works in the generative domain adopt diffusion-based methods.\n\nHowever, **thanks to the strong priors for human heads embedded in 2D/3D/4D GANs, they are capable of generating human heads with rich details.** Although the low-dimensional latent space of GANs imposes domain limitations, it also endows GANs with rich semantic information and powerful disentanglement capabilities within the latent space, enabling finer-grained control over human heads. Given the limitations of diffusion models, such as slower inference speeds and higher computational resource requirements, a significant number of researchers continue to explore GAN-based methods.\n\nAdditionally, in the generative domain, diffusion models do not completely outperform GAN-based methods. Both GANs and diffusion models have their own strengths and weaknesses. **Given the powerful generative capabilities of GANs and the strong generalization abilities of diffusion models, many researchers have attempted to combine the two to achieve complementarity.** Here are a few examples:\n1) $\\mathcal{W}_+$ Adapter [3] utilizes StyleGAN (2D GAN) to enhance the disentanglement and facial information retention of diffusion models.\n2) DiffPortrait3D [1] leverages EG3D (3D GAN) to control portrait novel-view synthesis and employs PanoHead (3D GAN) to generate multi-view images that assist in the training of the diffusion model.\n3) AvatarArtist [4] uses Next3D (4D GAN) to bridge 2D images and triplanes, and employs a robust 2D diffusion prior to handle diverse data distributions.\n\nTherefore, **we believe that research on GANs remains highly meaningful.** Combining the strengths of GANs and diffusion models to exploit the best of both worlds is a promising research direction. We will also explore integrating diffusion models into our framework in the future.\n\n---\n\n>**Q7**: Artificial Single-View Datasets.\n\n**R7**: Although multi-view datasets of human heads exist, the requirements for capturing multi-view human heads are quite stringent. Tasks based on single-view images are more aligned with real-world scenarios. Therefore, **research on single-view images is meaningful and has attracted considerable attention [1][4].**\n\n---\n\n>**Q8**: Comparison with RenderDiffusion.\n\n**R8**: The denoising process of RenderDiffusion resembles EG3D, mapping latent space to 2D images, while its diffusing process parallels 3D GAN inversion, mapping images to latent space. Both learn 3D-consistent representations from 2D images. RenderDiffusion also performs single-image 3D reconstruction. **Although unreleased code prevents direct comparison, the facial reconstruction examples in its paper appear far inferior to our method.**\n\n---\n\n>**Q9**: Limited to Tightly-Cropped Human Heads.\n\n**R9**: Our method is primarily applied to the task of single-view image reconstruction of human faces. **Due to the limitations of the pre-trained EG3D, our model requires aligned and cropped images as input and can only work within the pre-trained domain of EG3D (e.g., human faces and cats).**\nUnder this setup, we are able to achieve high-quality novel-view synthesis of human faces.\nAdditionally, we can provide the experiments on cats in the Appendix.\n\n---\n\n>**Q10**: Two-Stage Inpainting.\n\n**R10**: Since warped images often contain large areas of holes, we first use 3D GAN inversion to roughly fill in the occlusion regions (as shown in Fig. 2). Although the fidelity of the results from 3D GAN inversion is low (corresponding to entry A in Table 2), it can outline the facial contours in the occlusion regions, thereby alleviating the inpainting burden on SVINet. **Given that warped images are somewhat distorted and the initial filling is coarse (corresponding to entry B in Table 2), it is necessary to refine the inpainting in the second stage using SVINet.**\nTo clearly demonstrate the role of the first stage, we conducted an ablation study by directly feeding the warped image into SVINet for inpainting. The resulting FID was **19.98** and the ID score was **0.7876**, indicating that the initial coarse filling in the first stage is indeed effective.\n\n---\n\n>**Q11**: Symmetric Prior.\n\n**R11**: The facial symmetry prior we employ **does NOT assume that the input image is symmetric, but rather that the face structure itself is symmetric.** By incorporating this prior, we extract information from the visible part of the face to inpaint the face on the other side of the occlusion region. As can be seen from the visualizations we provide, **none of our input images are symmetric.**\n\n---\n\n>**Q12**: Blurring Outside of the Face Region.\n\n**R12**: The input images are real images from CelebA-HQ, but they need to be preprocessed with an alignment and cropping script. Since using the EG3D official preprocessing script to process CelebA-HQ images can result in black borders (as shown in the Triplanenet (WACV'24) paper), **we follow the same preprocessing method of HFGI3D.** Specifically, we first pad the image edges using the \"reflect\" mode, then perform blurring and other smoothing operations, and finally conduct alignment and cropping.\n\n---\n\n**References**:\n\n[1] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis. CVPR 2024.\n\n[2] RePaint: Inpainting using Denoising Diffusion Probabilistic Models. CVPR 2022.\n\n[3] When StyleGAN Meets Stable Diffusion: a $\\mathcal{W}_+$ Adapter for Personalized Image Generation. CVPR 2024.\n\n[4] AvatarArtist: Open-Domain 4D Avatarization. CVPR 2025."}]

Author rebuttal:

Meta-review: The paper proposes WarpGAN, a method to improve novel view synthesis in single-image 3D GAN inversion by addressing occluded regions through a combination of depth-based warping and a style-based inpainting network (SVINet). SVINet leverages latent codes from EG3D and a facial symmetry prior to enhance realism and consistency in unseen areas. The experiments show consistent improvements over the baselines, and ablations confirm the contribution of each component.

Reviewers liked the clear motivation, strong results with fast inference, and practical relevance for applications like latent editing. 

Weaknesses mostly concern incremental novelty—similar warp-and-inpaint designs have been explored in prior works—and lacking comparisons against modern diffusion-based alternatives. During the rebuttal, the authors addressed the novelty and training concerns, which most reviewers found adequate. While generalization issues remain, consensus formed around the paper’s quality and usefulness.

I hence recommend accepting this submission as a poster.
Decision: Accept (poster)