Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images
Authors: Bahri Batuhan Bilecen
Ahmet Berke Gökmen
Aysegul Dundar
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Reconstruction
GAN
Primary area: generative_models
TL;DR: We present a method for high-quality 360-degree head synthesis from single images.

Abstract: 3D GAN inversion aims to project a single image into the latent space of a 3D Generative Adversarial Network (GAN), thereby achieving 3D geometry reconstruction. While there exist encoders that achieve good results in 3D GAN inversion, they are predominantly built on EG3D, which specializes in synthesizing near-frontal views and is limiting in synthesizing comprehensive 3D scenes from diverse viewpoints. In contrast to existing approaches, we propose a novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspective. To achieve realistic 3D modeling of the input image, we introduce a dual encoder system tailored for high-fidelity reconstruction and realistic generation from different viewpoints. Accompanying this, we propose a stitching framework on the triplane domain to get the best predictions from both. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-aware triplane discriminator. Experiments reveal that our approach surpasses the existing encoder training methods qualitatively and quantitatively.

Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images
Bahri Batuhan Bilecen' Ahmet Berke Gokmen" Aysegul Dundar Bilkent University, Department of Computer Engineering, Ankara, Türkiye (batuhan. bilecene, berke. gokkennuu,,,, adundarOcs). bilkent. waw.eetteeuu..ee.. tr
3
Figure 1: From a single input image (first column), our framework reconstructs 3D representation by inverting images into PanoHead's 's latent space, which can be viewed in 360-degree perspective.
Abstract
3D GAN inversion aims project a single image into the latent space of a 3D Gen- erative Adversarial Network (GAN), thereby achieving 3D geometry reconstruction. While there exist encoders that achieve good results 3D GAN inversion, they are predominantly built on EG3D, which specializes in synthesizing near-frontal views and limiting in synthesizing comprehensive 3D scenes from diverse view- points. In contrast to existing approaches, propose novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspee- tive. To achieve realistic 3D modeling of the input image, we introduce dual encoder system tailored for high-fideling fidelity reconstruction and realistic generation from different viewpoints. Accompanying this, propose stitching framework on the triplane domain to get the best predictions from both. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including adversarial loss based discriminator. Experiments reveal that our approach surpasses the existing encoder training methods qualitatively and quantitatively. Please visit the project page
novel occlusion-avare triplane
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
In the realm of generative models, 2D GANs have gained renown for their remarkable ability to achieve striking realism through adversarial training. effectively capturing intricate details and textures to produce visually convincing images, especially on face images [8 19. However, their inherent limitation lies in their lack of depth perception, which restricts their applicability in ctree-dimensional contexts. In contrast, 3D GANs mark a groundbreaking advancement by seamlessly integrating Neural Radiance Fields (NeRF) into their architecture [13 2 14 [.. This integration not only allows
them to match the realism of their 2D counterparts but also ensures consistency
tiree-dimensionl
geometry. While extensive studies have focused on 2D GAN inversion 42
29
31
4
33
28
recent
efforts have seen the proposal of inversion methods tailored for 3D GANs
(3
7
21
2D GAN inversion techniques focus on projecting the images into the GAN's natural latent space to enhance editability and achieve high fidelity to the input image; however, inverting 3D GAN models presents additional challenges. This process requires accurate 3D reconstruction, which means realis- tic filling of invisible regions, ensuring coherence and completeness in the resulting dimensnnnlio scenes. Recently. inversion methods for 3D GANs have been developed, firstly. optimization- based and then encoder-based methods. Optimization-basn losses to invert images into a latent code specific to the given view. Furthermore, network parameters are optimized by generating pseudo-mull mulli-view images from the optimized latent codes to enhance detail preservation. Such optimization is required for each inference image; it is time-consuming and requires GPUs with large memory. Therefore, researchers focus on encoder-based methods [39 21). While successful inversions are achieved with these methods, they rely EG3D [9] framework
methods 20
35 38
employ reconstruction
39. 7 2T, which is constrained to synthesizing near-frontal views. However, our work utilizes PanoHead [5) method capable of rendering full- hell-head image synthesis, enabling comprehensive 360-degree perspective. This advancement introduces additional challenges, particularly concerning the invisibility of many parts in the input image. Despite this, the inversion model is expected to reasonably predict and reconstruct these occluded regions to ensure high-quality 3D reconstruction. Our experiments demonstrate that extending the methods proposed for EG3D is ineffective.
When projecting images onto PanoHead's s latent space, we observe a trade-off between achieving high-fidelity reconstruction of the input image and generating realistic representations of the invisible parts of the head. Some models can perfectly reconstruct the image from a given view but produce unrealistic outputs when the camera parameters change. Conversely, other models generate realistic representations under varying camera parameters but fail to achieve high-fidelity reconstruction of the input image. To achieve high-fidelity reconstruction of the input image and realistic representations of the invisible parts of the head simultaneously, we train a dual encoder. One encoder specializes in reconstructing the given view, while the other focuses on generating high-quality invisible views. We propose stitching the triplane domain generations to produce the final result. This approach combines the outputs from both encoders to achieve both high-fidelity reconstructions of the given view and high-que high-quality representations of the invisible parts of the head. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-awan triplane discriminator. This ensures that both encoders learn to produce consistent and complementary outputs, enabling seamless stitching of generations for the final result. Our contributions are as follows:
To achieve high fidelity the input and realistic generations for different camera views, we train dual encoders and introduce a stitching pipeline that combines the best predictions from both encoders for visible and invisible regions.
We propose novel occlusion- aware discriminator that enhances both fidelity and realism.
We conduct extensive experiments to show the effectiveness of our framework. Quantitative and qualitative results show the superiority of our method compared to the -tate-of-the-art Some visual results can be seen in Fig.
2 Related works
3D Generative Models. Generative Adversarial Networks (GANs), coupled with differentiable renderers, have achieved significant strides in generating 3D-aware multi-view consistent images.
2

---

24 operating on voxel representations, subsequent works 15 and the latest advancements are built around implicit Among implicit representations, triplane representations have emerged as a popular choice due to their computational efficiency and the high-quality outputs [9 [3 51. The architectures of works like EG3D [9] and PanoHead [5] bear resemblance to the structure of StyleGAN2 [9]. They consist of mapping and synthesis networks, generating triplanes which are subsequently projected to a 2D image through volumetric rendering operations akin to those used NeRF [23]. While EG3D is trained on the FTHQ [8] dataset with limited angle diversity, PanoHead achieves a 360-degree perspective in face generation thanks to their dataset selection and model improvements. In our work, we delve into PanoHead's 's latent space and construct our inversion encoder based on PanoHead.
While early efforts, such as HoloGAN shifted towards mesh representations
representations [00
14
26 25
GAN Inversion. In recent years, GAN inversion, particularly in the context of StyleGAN, has garnered significant attention due to its extensive editing capabilities. The primary objective of these studies to embed an image into StyleGAN's latent space, enabling subsequent modifications. Initially, this approached through latent optimization, where latent codes were iteratively adjusted using prock-prppggtion to minimize the reconstruction loss between the generated and target images [II 2 19 30]. For 3D-aware GAN model inversions, supplementary heuristics have been introduced into the optimization process. These include considerations like facial symmetry [8] and multi-vien optimization strategies [35). However, such methods are computationally intensive as they require optimizing each image's 's latent codes. Moreover, while minimizing the reconstruction loss yield visually similar results, it does not guarantee that the image resides within the natural latent space of GANs. This distinction is crucial for effective image editing. Without aligning with StyleGAN's inherent latent space, reconstructed images may respond correctly to editing techniques, thus limiting their practical utility. This consideration also extends to 3D-aware GAN inversion methods, where encoding geometric information is paramount. Even if an input image can be faithfully reconstructed, its realism may falter when observed from alternative viewpoints, emphasizing the importance of aligning with the GAN's native latent space.
To enhance efficiency, image encoders have been specifically trained for the inversion task, initially targeting StyleGAN [42 29 31 4 28 36 377. and more recently for EG3D [39 7 21]. These specialized encoders capitalize on insights gained from training datasets to swiftly project images into latent spaces. Moreover, they can be trained with diverse objectives beyond mere image reconstruction. For instance, some employ discriminators to compare generated and real images and latent space diseriminators to ensure inversion aligns with the GAN's natural latent space. As result, these methods generally offer faster inversion processes. This study focuses on 3D-GAN inversion, specifically targeting PanoHead (5). The task poses significant challenges due to PanoHead's ability to capture comprehensive 360-degree perspective, necessitating the prediction of a substantial portion of invisible elements by inversion encoders. Our experiments demonstrate that models trained for EG3D are ineffective in this context.
3 Method
3.1 Overview of PanoHead
The overall architecture of PanoHead is given in Fig. 2 Resemblant to EG3D, PanoHead utilizes mapping network that takes random vector and the camera conditioning mmppping- After z mapped to w. StyleGAN- based backbone G generates a tri-grid triplane. Unlike EG3D, PanoHead's triplanes have 3 times the number of channels in comparison, hence the name tri-grid. This approach is stated to ease 360-degree synthesis. The resultant triplane then rendered via a volumetric neural renderer R with pose Trendr and super-resolved to yield synthesized image.
2 ~~ 3~, Tiaspping
M
G
R
1F14-512
Symthesized Images
Trewder
Trl Fri-grid triplanes
Figure 2: Overall architecture of PanoHead.
3

---

3.2
Training an encoder
This section introduces the general pipeline of the encoders employed in our dual-encoder framework. Each encoder takes an input image I and predicts the latent code w*,, which is then passed to the generator to produce triplane features. These synthesized features are then fed into the renderer to generate a 2D image with a specified camera parameter Team, as described by Eq. (T:
W0
(I)
out ((G(w()) Team)
Here, out denotes the output rendering for the same view the input, and E, represents the encoder. While the W space allows for leveraging priors embedded in the generator, its limited expressive power in reconstructing image details has been noted due to the information bottleneck its 14 x 512 dimensions. To address this limitation, both 2D inversion techniques [3 28 and 3D GAN inversion
methods 7 permit higher-rate features to pass to the generators, facilitating the capture fine details. In 3D GAN inversion methods, these higher-rate features are encoded through smaller second network and transmitted to the triplane features. We adopt a similar approach in our encoders. We refer the final output I final Further details of the architecture are provided in the Appendix.
The primary challenge in this setting arises from establishing appropriate training objective losses, as our training dataset consists solely of single images, providing ground truth only for the ren- dered image from the same view as the input. For these output and ground-truth pairs, we set the usual reconstruction losses, namely, LPIPS perceptual loss [41], L2 reconstruction loss (MSE), and ArcFace 12 based identity loss as given in Eq. (
arg min CLPIPS (TYY I) C2 I) + Lideniity [aa (iinal
(2)
F1
While models trained with the objective given in Eq. 22 learn to reconstruct a given view, they often struggle to generalize and produce realistic features from other camera views. Consequently. while our first encoder is trained with the objective in Eq. ( we design an adversarial-based loss objective for our second encoder. This second encoder generates realistic predictions for invisible views, as
explained in Section 3.3
3.3 Occlusion-aware triplane discriminator
Encoded triplanes
W
E
G
R
Occlusion
maeks One
D
R/F
Non
2(114)
M
G
Synthrsized triplanes
Figure 3: Our training methodology for the triplane discriminator involves generating real samples by sampling latent vectors z and producing in-domain triplanes using PanoHead. Fake samples generated from encoded images. Despite the effectiveness of adversarial loss in enhancing reconstructions, challenges may persist in achieving high fidelity to the input due to the origin real samples from the generator G. To address this, propose occlusion-aware discriminator D, trained exclusively with features from occluded pixels. This ensures that visible regions, such as frontal views RR. have reduced influence during the training of D.
To achieve realistic reconstruction of the 3D model, represented in triplane structure, it essential to guide the encoder for visible views and overall coherence. Since we lack one-to-one ground truth

---

Dccluded triplane regions
R
R
R
F
E
G
Org
P
triplane regions
Figure 4: The inference pipeline with dual encoders for full 3D head reconstruction. Given face portrait with pose RR. can perform 360-degree rendering from any given pose Tnovel-
to guide the triplane structure, we experiment with various setups incorporating adversarial losses. A naive approach to utilize adversarial loss would be to render estimated triplanes from other views and assess the realism of these 2D images using a discriminator. However, our experiments observe that this setup hinders the model's ability to learn high fidelity to the input image, as will be further detailed Section 5.2 Moreover, randomly rendering different views can only guide limited parts of the triplane structure rather than the overall.
To overcome this limitation and avoid the computational burden of rendering unnecessary views, we explore the possibility of training a discriminator in the triplane domain. In our training process for the triplane discriminator, we follow a procedure where we sample latent vectors 2 and generate in-domain triplanes using PanoHead, serving our real samples. Meanwhile, the fake samples are triplanes generated from encoded images, as depicted in Fig. 3 Despite the observed improvement in reconstructions facilitated by this adversarial loss, note persistent challenge hindering the network's ability to achieve high fidelity to the input. This discrepancy may stem from the real samples originating from the generator, lacking the detailed feature characteristic of real-world images. Therefore, this may lead the encoder to omit to encode realistic facial details if they are absent in the synthesized samples. We propose occlusion-aware discrininator overcome this limitation. This discriminator exclusively trained with features corresponding to occluded pixels. This approach ensures that triplane features associated with visible regions, such frontal face, are not utilized for discriminator training. Additionally, we introduce masking mechanism for synthesized triplanes to mitigate any distribution mismatch arising between encoded and synthesized triplanes. This masking process contributes to aligning the distributions of real and fake samples, further enhancing the coherence of the training dynamics.
We find the set of visible points based on the depth map of the given view via inverse rendering. Specifically, the occlusion mask Onn estimated by Fq. (3)
OzR = MPpp, v. TR }
where p(r, y, is the triplane coordinates, TH the extrinsic camera parameters of the input view, D is the depth map from the input view, IC is the intrinsic camera parameters, I[u, v, 1] are the homogeneous coordinates of the input image rendered from the input view. More clearly, from the current camera pose, we map back to the depth values to obtain the mask of visible regions (I-Ong)). Then, we invert the visible region mask to obtain OTR- The utilization of occlusion masks has been previously investigated in 3D methodologies, albeit in different contexts. For instance, they have been used in generating sseudo-ground truth images to facilitate optimization-bul based 3D reconstruction [3 and integrated into passing high-rate residual features to the triplane [39]. However, it is the first time used in the diseriminator. This allows for a selective focus on regions where the encoder may encounter challenges in faithfully replicating realism.
Compliant with recent advancements in adversarial training, we follow WGAN loss (6] for Cadv in Eq. 4 where Tay and T synth are encoded and z synthesized triplanes, respectively. Details given in the Appendix.
arg min max Lady (Oxa TSV F Ox Teynth)
E2 D
5

---

3.4 Dual encoder pipeline
Encoder 2 Encader
Dual
Figure 5: Visual results of Encoder 1, Encoder 2, and Dual encoders for the given input images in the first and sixth columns.
In our approach, we train two encoders: the first, as outlined Section 3.2 and the second, augmented with an additional adversarial loss detailed in Section [3.3] While the initial encoder excels at reconstructing high-fidelity facial images from the input, it often produces unrealistic results for other viewpoints, depicted in Fig- Conversely, the second encoder yields better overall outcomes, albeit with slightly diminished fidelity to the input face.
Our aim is devise dual-encoder pipeline that harnesses the strengths of both encoded features. To achieve this, we leverage the occlusion masks derived Section [3.]] as illustrated in Fig. 4 By combining the visible portions from Encoder 1 and the occluded segments from Encoder generate our final output, as demonstrated the last row of Fig.
While each encoder contributes partially to the ultimate feature, achieving seamless integration necessitates consistency the output of both encoders despite their distinct specializations. For instance, if Encoder flawlessly renders a given view of the face but fails to capture the correct geometry, artifacts may arise in the combined result. Thus, it remains imperative to train both encoders comprehensively to ensure overall high-quality outcome.
4 Experimental Setup
4.1 Training
We combined images FFHQ [18 and LPFF [34] and split for training (~140k) and validation (~14 CelebA-HQ [17] and multi-view MEAD 33) are employed for additional evaluation. We removed face portrait backgrounds for training and evaluation datasets, applied camera and image mirroring during training, and performed pose rebalancing proposed in [34] as data augmentation. We utilized the same dataset train competitive methods for fair evaluation. The models are trained for 500k iterations with a batch size of 3 single RTX 4090 GPU. The learning rate is le for both encoders and the occlusion-award -wwrre discriminator. Ranger is utilized the optimizer, which is
combination of Rectified Adam 22 with Lookahead [40]
4.2 Baselines
The baseline models are provided in Table We note that no encoder pipelines are aimed for full 360-degree head reconstruction. We train the models with the author's released code to invert images into PanoHead's latent space.
4.3 Evaluation metrics
We report L2, LPIPS [1] and ID [12 scores for original-view reconstruction, which measure the fidelity to the input image. For the novel-view quality, we measure Fréchet inception distance (FID) [6.. Since our validation datasets have limited angle variance, we measure the distance between 1K randomly synthesized and 1k encoded real-life image distributions. The images are rendered from varying yaw angles, covering the 360-degree range to include occluded regions. We

---

also utilize the multi-view image dataset MEAD dataset. Specifically, we fed front MEAD images (0 yaw) to all methods, rendered them from novel views of MEAD (from 60% to yaw), and compare them with their corresponding ground truths. This allows us to report L.IPS and ID metrics alongside the FID metric for the novel views.
5 Results
5.1 Comparisons with state-of-the-art
Table Quantitative scores on various test sets.
FFHQ LPFF
CelebAHQ
Time
Category
Method
L2 4
LPIPS
ID f FID 4 C2 4 LPIPS 4 ID
FID 1
sec
W
optim.
2
0.035
0.17
0.57
65.30 0.049
0.19
0.49
63.57
49.19
Optimization
PTI
30
0.019
0.11
0.89
59.40 0.033
0.13
0.86
57.85
86.52
pSp
29
0.028
0.15
0.77
90.52 0.033
0.17
0.70
88 88.20
0.08
e1
0.064
0.24
0.52
82.99 0.080
0.26
0.42
84.70
0.05
Encoder TriplaneNetv211
7
0.017
0.10
0.86
98.97 0.020
0.11
0.80
99.02
0.17
GOAE
39
0.015
0.09
0.87
168.96 0.017
0.10
0.86
173.64
0.15
Ours
0.017
0.87
65.44
0.021
0.12
0.84
62.58
0.37
Table 2: Quantitative scores multi-view MEAD dataset.
Table T provides quantitative com-
parisons against saute-of-the-ae op- timization and encoder- based meth- ods. GOAE achieves significantly bet- ter same-view reconstruction scores (1.2, LPIPS, and ID), however, shows much worse FID scores, indicat- ing their inability to produce real- istic views. While TriplaneNetv2 achieves similar same-view recon- struction scores as our method, its FID score is also significantly worse. Over- all, the pSp and e4e methods perform worse than ours in all metrics. PTI achieves similar results to method but takes x250 longer and requires GPU with large memory.
LPIPS L
ID f
FID
Category
Method
+60
+30
+60 +30
+60
30
W opt.
0.249 0.200
0.570 0.558 48.473
43.875
Optim.
PTI
0.346 0.262
0,515 0,548 66.399
53.977
pSp
0.245 0.189 0.640 0.650 49.364 48.098
e4e
0.318 0.265 0.544 0,462 67.399 70.854
Tpn.v2
0.248 0.192 0.658 0.663 48.937 46.493
Encoder
GOAE
0.296 0.249 0,654 0.660 87.644 92.758
Ours
0.223 0.178 0.706 0.726 47.207 43.822
We extend the quantitative analyses to multi-view with the MEAD dataset. Specifically, we feed front MEAD images (0) yaw) to all methods, rendered them from novel views of MEAD (from 60% to 0 yaw), and compare them with their corresponding ground truths. Table reveals that our method significantly improves over compared methods especially in LPIPS and ID metrics.
Qualitative results are shown in Fig. & and Fig. The competing methods produce unrealistic outputs when viewed from angles other than the input view. Among these, PTI achieves good front and side views but fails to generate realistic hair from the back. Our method achieves the best results overall.
We also include mesh comparisons in Fig. Ours is better than the most recent encoder- method 7 and generally performs well compared to PTL Note that PTT mostly generates smoother meshes (row 3) but can sometimes struggle depending on the input sample (row 6).
5.2
Ablation study
Table 3: Ablation on occlusion-awan discriminator D.
In Table and Fig. we present an ablation study demon- strating the effectiveness of our occlusion-aware triplane discriminator quantitatively and qualitatively. The first row of results shows that not using any discriminator achieves good reconstruction of the given view, as indicated by LPIPS and ID scores. This also visible the first-row in Fig. 7 However, this approach fails to generalize to novel views, as evidenced by the FID score and visual
LPIPS
ID t FID 1
No D
0.10
0.87
89.50
w image domain
0.17
0.67
72.86
D w/o triplane occ.
0.15
0.70
66.24
D w/ triplane occ.
0.14
0.75
64.02
results.
7

---

W
PT1
pSp
Trip.x2
GOAE
Outs
Figure 6: Comparisons ours and competing methods.
the other hand, training the model with an additional adversarial objective that operates on novel images generated using randomly sam- pled camera parameters improves the FID score but significantly harms the fidelity of the input image. Training discriminator in the triplane domain and applying adversarial losses from this domain improves overall scores compared to training the discriminator in the 2D image domain. However, the face still lacks high fidelity to the input, and other views are unrealistic. Lastly, using the occlusion-awer aware triplane discriminator improves identity fidelity and FID scores. The hair looks more natural, similar to the ones generated by the model when sampled from 2.
No D
D img.
D w/b occ.
Fig. 7 (third row),
D w/ occ.
Qualitative results of ablation on occlusion-aweer aware discriminator D.
Figure 7:
In our framework, we chose to embed images into the W space. Table presents an ablation study that explores utilizing different projection spaces and various combinations of training data. Training an encoder to project images to the 2 or sampled times, results
Table 4: Ablation on training data and latent space.
Train data
Proj.
LPIPS 4 IDt FID $
Real img.
Z
0.29
0.31
45.79
Real img.
z+
0.22
0.60
76.54
Z space, where in better FID scores. However, this comes at the cost of high-fidelity reconstruction. Similarly, transitioning to a less constrained W+ space enhances fidelity to the input but worsens the FID score. Addressing this challenge necessitates additional measures, such as the proposed dual encoder setup with the occlusion-aware discriminator objective. is important to note that distinction between the 2 and W+ space arises from the camera parameters incorporated into the mapping network. While 2 employs various samples of 2, it adheres to the same set of camera parameters assigned to the mapping network. In contrast, the W space does not impose
Real img.
W+
0.10
0.86
98.97
Z
gens.
W
0.27
0.25
46.93
Real imgs. + 2
W+
0.10
0.87
89.50

---

W opt
PT1
pSp
Tpm v2
GOAE
Ours
Figure 8: Left right: input (0%), reconstruction (0%), GT target (+60%), and render on +00 using the reconstruction triplanes of 0 on MEAD dataset.
Tpnv2
PT1
Ours
Tpnv2
PII
Figure 9: Inputs (first), reconstructions (second), and 360 mesh renders (rest) of our method.
such constraints during encoding. Given that the real image dataset we use primarily consists of limited camera poses, typically front-view faces, investigate training the encoder with synthetically generated images from PanoHead. However, solely utilizing synthetic images generated from samples diversity compared Z leads poor performance on real image validation
of + introduce sets regarding reconstruction quality. When combining synthetic and real images, we observe an improvement compared to using them individually.

---

Figure 10: Hair edits from source image (first) to destination image (second) and 360 renders (rest).
Table 5: Ablation on dual-encoder.
Lastly, we show the results of using the dual encoder in Ta- 5 The visual results were previously presented in Fig. 5 The dual mechanism leverages the strengths of both encoders, achieving the same LPIPS and scores as Encoder while also producing FID scores very similar to those of Encoder 2.
Method
LPIPS
| ID f FID |
E1
0.10
0.87 89.50
E2
0.14
0.75 64.02
Dual
0.10
0.87 65.44
5.3 Editing application
We follow the reference- barse editing in [8] in our pipeline. This method encodes input images, and edits are performed in the triplane space. This approach utilizes the fact that triplanes have a canonical space, allowing for the transfer of local parts from one triplane to another. Fig. [0 demonstrates successful transfer of hairstyle from reference image to the target human in 3D. Another advantage of mased models over the optimization ones is the feasibility of such applications. For example, this would not be possible with PTI since the generator is fine-tuned for each sample, preventing the copying of features from one image to another the encoded feature space.
Conclusion and Broader Impacts
Figure 11: Example failure cases. Inputs (first), reconstructions (second), and novel views.
In summary, this study introduces a 3D GAN inversion framework that projects single images into the latent space of 3D GAN for accurate 3D geometry reconstruction. While prior encoders excel at synthesizing near-frontal views, they struggle with diverse 3D scenes, motivating our exploration of alternatives. Using PenoHead's 's 360-degree synthesis, developed a dual encoder system for high-fidelity reconstruction and realistic multi-view generation. A stitching mechanism in the triplane domain ensures optimal predictions from both encoders. With specialized losses, including an occlusion-awan aware triplane discriminator, our framework achieves superior qualitative and quantitative performance over existing methods.
Broader Impacts. Our framework has the potential to revolutionize the movie industry. AR, and VR, enabling applications like animating portraits and creating realistic game environments. However, it raises ethical concerns, particularly the risk of "deep fakes". We stress the need for safeguards to ensure the ethical use of this technology.
Limitations. We acknowledge that there is room for improvements in the fidelity of images, the realism and 3D-co 3D-consistency of generations (see Fig. II row 2), and the smoothness of the meshes (see Fig. D. Since the projection is made onto the latent space of PanoHead, our method may not handle out-of-domain or tail samples well (such images with high-frequency details or accessories). For instance, our method struggles with hats, as demonstrated in the first row of Fig. II We recognize that, in certain cases, the artifacts visible in the back middle of the head and are more noticeable in the mesh rendering as shown in Fig. | Additional research is required.
Acknowledgements. This work was supported by the BAGEP Award of the Science Academy. We acknowledge EuroHPC Joint Undertaking for awarding the project ID EHPC-AI-22D4A002-051 access to Leonardo at CINECA, Italy.
10

---

References
[1] Abdal, R., Qin, Y., Wonka, P.: Imagc2stylogan: How to embed images into the stylegan latent space? In: Proceedings of the IERE/CVF International Conference on Computer Vision. pp. 4432 4441 (2019)
[2] Abdal, R., Qin, Y., Wonka, P.: Image2stylogan++: How to edit the embedded images? In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8296-8305 (2020)
[3] Alaluf, Y., Patashnik, O., Cohen-Or, D.: Restyle: A residual-based stylegan encoder via iterative refinement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6711-6720 (2021)
[4] Alaluf, Y., Tov, O., Mokady, R., Gal, R., Bermano, A.: Hyperstyle: Stylegan inversion with hy- pernetworks for real image editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1851 1511-18521 (2022)
[5] An, S., Xu, H., Shi, Song, G., Ogras, U.Y., Luo, L.: Panohead: Geometry-aware 3d full- full-head synthesis in 360deg. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20950-2000 20959 (2023)
[6] Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan (2017)
[7] Bhattarai, A.R., Nießner, M., Sevastopolsky, A.: Triplanenet: An encoder for eg3d inversion. In: Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision. pp. 3055-3065 (2024)
[8] Bilecen, B.B., Yalin, Y., Yu, N., Dundar, A.: Reference-based 3d-aware image editing with triplanes. arXiv preprint arXiv:2404. (2024)
[9] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L...,, Tremblay, J., Khamis, S., et al.: Efficient geometry- aware 3d generative adversarial networks. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16123-16133 (2022)
[10] Chan, F.R., Monteiro, M., Kellnhofer, P., Wu, J., Wetzstein, G.: pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In: Proceedings of the IEFF/CVF conference on computer vision and pattern recognition. pp. 5799-5809 (2021)
[11] Creswell, A., Bharath, A.A.: Inverting the generator of a generative adversarial network. IEEE transactions on neural networks and learning systems 30(7), 1967-1974 (2018)
[12] Deng. J., Guo, J., Xue, N., Zafeiriou, S.: Areface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690 4690-4699 (2019)
[13] Gao, J., Shen, T., Wang, Z.. Chen, W., Yin, K., Li, D., Litany, O., Gojeic, Z., Fidler, S.: Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems 31841-31854 (2022)
[14] Gu, J., Liu, L., Wang, P., Theobalt, C.: Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:21 10.00988 (2021)
[15] Henderson, P., Tsiminaki, V., Lampert, C.H.: Leveraging 2d data to learn textured 3d mesh generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 7498-7507 (2020)
[16] Heusel. M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)
[17] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 17Xiv:1110.1011 (2017)

---

[18] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410 (2019)
[19] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119 (2020)
[20] Ko, J., Cho, K., Choi, D., Ryoo, K., Kim, S.: 3d gan inversion with pose optimization. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp- 2967-2976 (2023)
[21] Li, X., De Mello, S., Liu, S., Nagano, K., Iqbal, U., Kautz, J.: Generalizable one-shot 3d neural head avatar. Advances in Neural Information Processing Systems 36 (2024)
[22] Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., Han, J.: On the variance of the adaptive learning rate and beyond. In: Proceedings of the Eighth International Conference Learning Representations (ICLR 2020) (April 2020)
[23] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng. R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications the ACM 65(1), 99-106 (2021)
[24] Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., Yang, Y.L.: Hologan: Unsupervised learning of 3d representations from natural images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7588-759 0888-7597 (2019)
[25] Niemeyer, M., Geiger, A.: Giraffe: Representing scenes compositional generative neural feature fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11488 1164 (2021)
[26] Or-EI, R., Luo, X., Shan, M., Shechtman, E., Park, J.J., Kemelmacher-Shieee Shhnzerrnn I.: Stylesdf: High-resolution 3d-consistent image and geometry generation. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition. pp. 13503-13513 (2022)
[27] Pavllo, D., Spinks, G., Hofmann, T. Moens, M.F., Lucchi, A.: Convolutional generation of textured 3d meshes. Advances in Neural Information Processing Systems 33, 870-882 (2020)
[28] Pehlivan, H., Dalva, Y., Dundar, A.: Styleres: Transforming the residuals for real image editing with stylegan. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1828-1837 (2023)
[29] Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., Cohen-Or, D.: Encoding in style: a stylegan encoder for image-to-ime to-image translation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2287-2296 (2021)
[30] Roich, D., Mokady, R., Bermano, A.H., Cohen- Oohhen-Or,De D.: Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG) 42(1), 1-13 (2022)
[31] Tov, O.. Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Orr D.: Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG) 40(4), 1-14 (2021)
[32] Wang, K., Wu, Q., Song. L., Yang, Z.. Wu, W., Qian, C., He, R.. Qiao, Y., Loy, C.C.: Mead: large-scale audio-visual dataset for emotional talking-face generation. In: ECCV (August 2020)
[33] Wang. T., Zhang. Y., Fan, Y., Wang, J.. Chen, Q.: High-fidelity gan inversion for image attribute editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11379-1 11388 (2022)
[34] Wu, Y., Zhang, J., Fu, H., Jin, X.: Lpff: A portrait dataset for face generators across large poses. In: Proceedings of the JEEE/CVF International Conference on Computer Vision. pp. 20327-20337 (2023)
12

---

[35] Xie. J., Ouyang, H.. Piao, J., Lei, C., Chen, Q.: High-fidelity 3d gan inversion by pseudo-multi- view optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 321-331 (2023)
[36] Yildirim, A.B., Pehlivan, H., Bilecen, B.B., Dundar, A.: Diverse inpainting and editing with gan inversion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 101120223133 (2023)
[37] Yildirim, A.B., Pehlivan, H., Dundar, A.: Warping the residuals for image editing with stylegan. arXiv preprint arXiv:2312.1 1422 (2023)
1422
[38] Yin, F., Zhang, Y., Wang. X., Wang, Li, X., Gong, Y., Fan, Y., Cun, X., Shan, Y. Oztireli,
C., al.: 3d gan inversion with facial symmetry prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 342-351 (2023)
[39] Yuan, 7., Zhu, Y., Li, Y., Liu, H., Yuan, C.: Make encoder great again in 3d gan inversion through geometry and occlusion-awar encoding. Proceedings of the IEEE/CVF international conference on computer vision (2023)
[40] Zhang, M., Lucas, J.. Ba. J., Hinton, G.E.: Lookahead optimizer: k steps forward, step back. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019)
[41] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features perceptual metric. In: Proceedings of the IEEE conference computer vision and pattern recognition. pp. 586-595 (2018)
[42] Zhu, J., Shen, Y., Zhao, D., Zhou, B.: In-domain gan inversion for real image editing. In: European conference on computer vision. pp. 592-55226 608. Springer (2020)
13

---

A Appendix
A.1 Architecture details
Encoder and 2. For Encoder and 2, ploy 2-stage encoding for W and high-rate F features, also
Conv2D (288, (2, 64,
LeakyReLU 2)
in common with other style-based encoder methods 22
(Bn22D644128,,33)
3
We opted for the architecture in (29) for W stage (GradualStyleenncode) and in 7 for F stage
4
39
7
BatchNorm2D (128)
both for Encoder and 2.
LeakyBeLU
C
Ablation encoders. SradualStyleencoodee is used for the 2 encoder, where resulting latent vectors are later passed through the mapping network with truncation v 0.85 and canonical front camera pose. However, for the encoder, a smaller varia- tion of [29] [BackboneEncoderusiingg wageinnggssttaeennnnn IntooW is im- plemented. This choice is due to 2 being less expressive compared Z+ + (1 dim vs. 14 dim) and hence smaller encoder being suf- ficient. The same mapping network parameters for the 2 case also utilized for 2 encoder outputs.
(2cchN
LeakyReLU
Conv2D (256, 51,,,,,,,,111 (5tch))or2DD(112))
LeakyReLU (0. 2) Conv2D(512,,,3, AvgPool(1)
Table 6: Architecture for dis- criminator. Conv2D parame- ters are:
Occlusion-aware triplane discriminator. We follow a feedfor- ward network approach with channel bottleneck for tri- plane discriminator D (Table 6). Noting that the occluded tri- plane dimensions are [batch_ size, ,,996, 256, 256], first add each depth slice to the channel dimension to get the input shape [batch_sizee sizech_siee,,_ 288, 256, 256]. Output dimensions [batch_site Seatch_size,1e size, 1]. We do not utilize saturating functions such as sigmoid at the end since WGAN- based loss (6] is utilized.
(input channels, out- put channels, kernel size, stride, padding), respectively. Bias terms are disabled.
Ablation image discriminator. For the back-view image discriminator used in ablations, we change the input channel number of the model in Table from 288 to 3.
A.2 Training objectives and hyperparameters
The training objective for Encoder 1 with parameters dg, is given in Eq. 5
1 (E) R) arg min A (I + (V I) + Ag AaLidenityy (I OR.
(5)
where 1 is the view reconstruction of with pose Coefficients are set \ 0.8,
A2 = 1.0, A3 = 0.5.
The training objective for Encoder 2 with parameters dg, is given in Eq. 6
Tes: = (E2
av = R(T eney T
(6)
arg min CLPIPS I) - X2C2(IT), (I) + Xg Ci (I I) + Ugg
where T Teme is the encoded triplane of image I, D is the discriminator, O is the occlusion mask from the same view n. 21,2,3 are the same as in Eq. (5) and X4 0.001. Lutv in Eq. (6 given in Eq.
Cat (x) = softplus(-o)
where softplus is a smooth and differentiable approximation to ReLU.
The training objective for discriminator D with parameters 0p is given in Eq. 8
14

---

Ten: G (T)
T symth G(M(=* (0, 114, Trront) ) arg min (D(O(D(O D(O,Tsyyt)))
(8)
Bp
where T syythh is the synthesised triplane from randomly sampled x M is the mapping network, Trrent the canonical front pose. Ladv in Eq. given Eq.
Ladv (z, v) = softplus(-x) + softplus(y)
À is set as 0.5.
We jointly train Encoder and discriminator traditional adversarial fashion. We further employ -egreaularization to encourage Li-lipschitzness [6] to justify using softplus, where its weighting coefficient is and applied every 16 iterations.
A.3 Additional qualitative results
Our method can handle diverse ethnicities and challenging input views, demonstrated in Fig. 12
We also showcase additional visual results for competing methods in Figs. 13 to 20 ablation discriminator in Fig. 21 ablation on input images, the second columns reconstructions from the input views, the of models from novel views.
structure in Figs. 22 to 24 The first columns are
renderings
15

---

Figure 12: Inputs with diverse ethnicities and challenging views (first), reconstructions (second), and 360 renders (rest).
16

---

W+ opt.
PTI
pSp
e4e
Trip.v2
GOAE
Ours
Figure 13: Comparisons of ours and competing methods.
W+ opt.
PTI
pSp
c4cc
Trip.v2
GOAE
Ours
Figure 14: Comparisons of ours and competing methods.
17

---

W+ opt.
PTI
pSp
c4c
Trip.v2
GOAE
Ours
Figure 15: Comparisons of ours and competing methods.
W+ opt.
PTI
pSp
c4c
Trip.v2
GOAB
Ours
Figure 16: Comparisons of ours and competing methods.
18

---

W opt.
PTT
pSp
c4c
Trip.v2
GOAE
Ours
Figure 17: Comparisons of ours and competing methods.
W+ opt.
PTI
pSp
e4c
Trip.v2
GOAB
Ours
Figure 18: Comparisons of ours and competing methods.
19

---

W opt.
PTI
pSp
c4c
Trip.v2
GOAE
Ours
Figure 19: Comparisons of ours and competing methods.
W opt.
PTI
pSp
c4c
Trip.v2
GOAB
Ours
Figure 20: Comparisons of ours and competing methods.
20

---

No N
D w/ img
w/o occ.
w/ oec.
Figure 21: Qualitative results of ablation on occlusion-aware discriminator D.
Encoder Encoder
Dual
Figure 22: Visual results of Encoder 1, Encoder 2, and Dual encoders for the given input images in the first column.
Encoder 2 Encoder
Dual
Figure 23: Visual results of Encoder 1, Encoder 2, and Dual encoders for the given input images in the first column.
21

---

Encoder
Eneoder 2
Dual
Figure 24: Visual results of Encoder Encoder 2, and Dual encoders for the given input images in the first column.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims include utilizing dual encoders trained to specialize in visible and occluded regions, along with a stitching pipeline that seamlessly combines the most accurate predictions from both encoders. To further enhance the reconstruction of occluded regions, we propose an occlusion-awand generate realistic features for these challenging areas. Extensive experiments validate the effectiveness of our approach, demonstrating superior performance across multiple benchmarks and scenarios.
discriminator, enabling the image encoder to
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are given in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
23

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The training and evaluation schemes are detailed Sections and 4.3 and Appendix A which provides reproducability.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
The paper does not provide open access to the code as of now; however, the datasets are public and can be obtained from the pointed references. The code is planned to be released through the project page provided in Abstract.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mtiiss.cc/public//ee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training objectives are given in Eqs. and (4), the training recipe in Sec- tion 4.1 and the evaluation metrics in Section 4.3 Further details about the experimental setup provided in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not provide error bars due to the computational budget it would require.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Time of execution of methods are provided in Table as well as the computing resources in 4..
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts provided in Section b
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe the potential risks in Section o
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly mentioned the original owners of previous work via references, and complied with the licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not share new assets as of now.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper neither involves crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "VpLsjIT33O", "reviewer": "Reviewer_9tZp", "summary": "This paper presents a framework for 3D GAN inversion aimed at reconstructing high-fidelity 3D head models from single 2D images. A dual encoder system that combines one encoder specialized in high-fidelity reconstruction of the input view with another focused on generating realistic representations of invisible parts. An occlusion-aware triplane discriminator that enhances both fidelity and realism.", "questions": "1. From Figure 4, it can be seen that Encoder 2 may be the main solution to the problem in this paper. Is there any way to introduce the features of Encoder 1 into Encoder 2? It seems that Encoder 1 is more like a subsidiary of Encoder 2. For this reason, introducing two encoders is feasible, but not elegant enough.\n2. I'm curious about more training details. For example, how much data was used for training?\n3. The evaluation is primarily on face datasets (FFHQ, LPFF, CelebA-HQ). It's unclear how well the method generalizes to more diverse head shapes or ethnicities.\n4. In Encoder 2, is the relevant information about the person's head implicitly included? Is there any experiment to prove this?", "limitations": "While some limitations are mentioned, a more in-depth analysis of when and why the method fails would be valuable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The qualitative results show improvements over existing methods, especially for novel viewpoints.", "weaknesses": "1. The writing and language of the paper need substantial improvement.\n2. The innovation of this paper may be a little lacking. This paper proposes a new method, but the insights shown in the method are limited and it is difficult to generate insights into other problems.\n3. There is also a lack of in-depth research on the proposed method.\n4. While some limitations are mentioned, a more in-depth analysis of when and why the method fails would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback, and we will improve the writing further based on the reviewer’s suggestion. The reviewer mentions that the innovation of the paper may be a little lacking, and there is a lack of in-depth research on the proposed method.  However, it is crucial to note that our study addresses the novel challenge of GAN inversion for 360-degree GANs, a topic that has not been previously explored. We demonstrate that existing methods fail to address the intricacies of this problem. The dual-encoder design coupled with an occlusion-aware triplane discriminator proves to be intuitive, and the ablation study effectively demonstrates the impact of each design choice, offering valuable insights, as noted by other Reviewers *(AnnS, jtFz, Dd7h, XyTr)*. Furthermore, many triplane-based models have been emerging lately [Yang et al., Dreamcomposer, CVPR 2024], and the occlusion-aware discriminator approach may enable a new, unsupervised training direction for generating coherent 3D samples from single images. This is also appreciated by *Reviewer jtFz (Strength-3)*. \n\nAbout limitations and failure cases; we have tested different scenarios. Firstly, as suggested by *Reviewer AnnS*, to include multi-view evaluation, we set up the MEAD dataset, which also includes more diverse ethnicities than the CelebA dataset, as shown in rebuttal Fig. 2. We show that the method also handles heavy make-up inputs, as given in Fig. 3. We also visualize results when the input views are extreme side-views, given in Fig. 4. Finally, in Fig. 5, we find that our method struggles when input images have head accessories and treats them as hair from the back-view. We will add those additional analyses to our final manuscript. \n\nAnswers to the questions:\n\n 1. In our experiments, removing Encoder 1 resulted in a significant decrease in the ID scores, as given in Table 4. Encoder 1 provides high fidelity to the input, a must in reconstruction. On the other hand, Encoder 2 excels in generating better representations of invisible regions. Thus, we observed a trade-off between these two critical requirements and found the best-achieving method to be a dual-framework. That said, we provide analysis, methods, and baselines for this new and challenging problem, which can pave the way for more elegant frameworks.\n\n\n 2. We combined FFHQ and LPFF for training (140k) and test (14k) sets and applied pose rebalancing proposed in LPFF. We also used the same dataset to train competitive methods for fair evaluation. We will detail the relevant section and elaborate more on the Supplementary.\n\n\n 3. Our method achieves 3D reconstruction on diverse ethnicities, as included in rebuttal Fig.2. More visual results will be added to the main paper and the Supplementary. Thank you for the suggestion.\n\n\n 4. The head information is implicitly encoded in PanoHead’s generator backbone, which we keep frozen. Following the reviewer's suggestion, we examined how Encoder 2 performs when given a silhouette of a face as input, as shown in rebuttal Fig. 8. Our observations confirm that, even with this out-of-domain input, Encoder 2 generates latent representations that produce a realistic head shape. This indicates that Encoder 2 retains an implicit representation. We appreciate the reviewer’s insightful suggestion and will include this finding in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zdhmEt7hkD", "reviewer": "Reviewer_AnnS", "summary": "This paper introduces a novel encoder-based method for 3D pano-head GAN inversion. Unlike previous 3D GANs focused on near-frontal views, this work designs a dual-encoder system tailored for both high-fidelity reconstruction and realistic generation from multiple viewpoints. The method incorporates a stitching mechanism within the triplane domain, ensuring optimal predictions from both encoders. This results in superior performance compared to existing methods, both qualitatively and quantitatively.", "questions": "- As discussed in Weaknesses.\n- Will the choice between a CNN-based or Transformer-based encoder affect the reconstruction results? Current works like GOAE, use transformer-based encoder for better different-view input instead of just front-view.", "limitations": "- Discussion of failure cases is necessary.\n- The formula symbols in the paper need to be clearly distinguished, such as E1 and E2. Clear differentiation of these symbols should be included in the main text.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Investigating GAN inversion for 360-degree GANs diverges from prior research as it necessitates accounting for more complex viewing conditions.\n- The dual-encoder design coupled with an occlusion-aware triplane discriminator proves to be intuitive and beneficial.\n- The approach outperforms existing methods, both in qualitative and quantitative assessments.", "weaknesses": "- The input images in the study are primarily front-view; it would be beneficial to include some from extreme viewpoints.\n- Further assessment of the model's generalization is required for more complex input images, such as those with heavy makeup.\n- The quantitative experiments currently focus on single-view cases; incorporating results from other viewpoints may provide a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and suggestions, which further helped us improve the paper. \n\n 1. (**W.1**) The reviewer wanted more visual results from extreme viewpoints. We did not notice the visual results from the paper were all more from the frontal view. We apologize for that. LPFF dataset is a large-pose Flickr dataset; therefore, our quantitative results included such evaluation. In rebuttal Fig.4, we also provide visual images with challenging viewspoints. We will incorporate more examples into our final paper. Thank you for bringing this to our attention.\n\n 2. (**W.2**) Based on the reviewer’s suggestions, we include heavy make-up input results in rebuttal Fig. 3; additionally, diverse ethnicities are represented in Fig. 2 as suggested by Reviewer 9tZp. \n\n 3. (**W.3**) Thank you for the neat idea. We have generated a multi-view dataset from videos in MEAD (https://github.com/uniBruce/Mead) and quantitatively evaluated ours and competing methods. Specifically, we fed front MEAD images (0 yaw degree) to all methods, and rendered them from novel views of MEAD (from $\\pm$60 to 0 yaw degrees). We compared them with corresponding ground truths. Our method performs better quantitatively for novel views (rebuttal Table 1 and Fig. 1). Figure 1 shows ground-truth input and output pairs from MEAD dataset. These experiments will be added to the revised paper. \n\n 4. (**Q.2**) About the choice of CNN versus transformers, in our experiments, TriplaneNetv2 (CNN-based) and GOAE (transformer-based) inversion methods performed similarly when evaluated on challenging samples. That being said, self and cross-attention-based approaches to capture similarities may be a good idea to investigate further. We thank the reviewer for encouraging further study directions. \n\n 5. (**L.1,2**) Regarding the limitations, in Rebuttal Fig. 5, we find that our method struggles when input images have head accessories and are treated as hair from the back-view. We will add those additional analyses to our final manuscript.  Symbols in formulas and figures will be rewritten with clearer notation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IzyL5MTSLZ", "reviewer": "Reviewer_jtFz", "summary": "This paper addresses the challenge of 3D GAN inversion from a single image, focusing on a method called PanoHead designed for generating 360 views of human heads. Unlike  optimization-based approaches, this work adopts an encoder-based technique to reduce inversion times. The authors observe that varying encoder training objectives result in different advantages and disadvantages in the inversion output. Consequently, the authors propose using two encoders and introduce a stitching technique in the triplane feature space to combine the strengths of each encoder.", "questions": "1. Line 177: How do you obtain the depth map D? Do you use a monocular depth estimation method, or do you extract it from the NeRF model?\n2. Figure 5: Are E1 the encoder from section 3.2 and E2 the encoder from section 3.3? If so, why is the triplane feature of E1 masked by the occluded regions and E2 masked by the visible regions? According to the paper, the proposed method appears to use the visible regions of the triplane for E1 and stitch them with the occluded regions of E2. Is this a mistake in the figure?\n3. Line 223: The FID is calculated between the rendered image from the encoded input and \"1k randomly synthesized\" images. Which method is used to generate these \"1k randomly synthesized\" images? Is it PanoHead? Are all methods compared against the same set of images synthesized by the same method?", "limitations": "The authors included discussions on both the limitation and broader impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The ablation study effectively demonstrates the impact of each design choice, offering valuable insights.\n3. The technique for stitching occluded and visible region features could potentially be useful in a broader context. e.g., in applications beyond human head modeling.", "weaknesses": "1. The claim on lines 168-169, stating that triplane features associated with visible regions (e.g., a frontal face) are excluded from discriminator training and thus do not receive gradients, is not entirely accurate. The method forms a mask by back-projecting the depth map onto the triplane coordinates, which limits the mask to points lying on the surface of the visible region. However, since the model uses triplane features for volume rendering, the features used for rendering the frontal view are not necessarily confined to the surface of the face. Features behind that surface may also contribute to color aggregation during volume rendering, potentially leaving some features used for rendering the visible regions uncovered by the mask.\n2. Some details lack clarity, as noted in the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and highlighting our paper’s several strengths.\n\n 1. About the claim on lines 168-169: We acknowledge that not receiving any gradients for the visible region may be theoretically incorrect and will make the necessary modifications to avoid confusion. Thank you for pointing this out. Meanwhile, although the occlusion masks do not 100% eliminate information flow between occluded and visible regions because of the volumetric rendering, we emphasize that our approach performs adequate disentanglement between two regions to produce realistic and coherent samples. This is evident by the occlusion-aware dual encoder combination. Figure 4 in the main paper makes this very visible, too. The dual encoder is able to take the non-occluded parts from Encoder 1 and the rest from Encoder 2. This is also shown by the occlusion-aware / non-occlusion-aware triplane D ablation (Table 2 on main paper, last two rows, specifically the improvements on ID). The occlusion masks can mask a significant portion, and experimental results show they are effective. \n\nWe emphasize that this does not hinder the overall claim of our work. Still, they must be acknowledged in the main paper to avoid confusion. Again, we thank the reviewer for such valuable feedback and will improve the relevant sections.\n\nAnswers to the questions: \n 1. We extract the depth maps from the NeRF model.\n 2. The figure has an unfortunate typo, and the E1 and E2 labels need to be switched. Thank you for your attention.\n 3. The competitive methods are based on PanoHead’s generator backbone; hence, they are compared against the same set of images synthesized by the same method. We will make this more clear in the revised paper. The reason we use these synthesized images is to ensure comprehensive coverage of all angles in our evaluation.\n\n\nIn the rebuttal, we also provide evaluations on the multi-view MEAD dataset as suggested by *Reviewer AnnS*. In this setting, we use paired images, where the input is at one angle, and the output is $\\pm$60 degrees rotated. In this setting, for the FID evaluation, we use the ground-truth images. These results are given in rebuttal Table 1 and Fig. 1, which will be added to the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BUige14PxO", "reviewer": "Reviewer_Dd7h", "summary": "This paper introduces an encoder-based method to do the GAN inversion task, especially for the full head inversion. The occlusion aware discriminator is interesting and reasonable. The results are good.", "questions": "see the weakness", "limitations": "see the weakness", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is interesting, I like the occlusion-aware discriminator. Meanwhile, training the discriminator in the triplane domain is also reasonable.\n\n2. The results are good.", "weaknesses": "There is no significant weakness in this paper. I have a question about the training process. Are these two encoders trained on the face first and then on the occluded parts? Is the occlusion-aware encoder initialized with the facial encoder?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and finding our paper interesting and technically solid.\n\nIn the training process, E1 is trained with visible parts by using input-output paired losses of LPIPS, L2, and identity losses. On the other hand,  E2 is trained to learn the occluded regions by learning from the discriminator. The occlusion-aware E2 is not initialized with a pre-trained encoder. E2 and D are trained jointly from scratch. We will elaborate on the training procedures. We would like to thank the reviewer for bringing this to our attention.\n\n\nAdditionally, we would like to highlight that Rebuttal Table 1 and Fig. 1 demonstrate our method's significant superiority over competing approaches on the multi-view MEAD dataset evaluation. We also show our method’s ability to generalize across diverse ethnicities, heavy makeup, and extreme camera viewpoints in Rebuttal Fig. 2, 3, and 4, respectively. The geometry of the generated results is detailed in Fig. 6. These findings will be included in the final paper, which we believe will further strengthen our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2E1sHZvO9x", "reviewer": "Reviewer_XyTr", "summary": "The paper proposes an encoder-based approach to GAN-inversion for full 3D head reconstruction from a single image. To address the challenge of reconstructing the entire 360° head, the method employs two encoders: one for high-fidelity reconstruction of the input image (typically near-frontal) and another for generating realistic invisible parts of the head (e.g., back of the head). Finally, the paper presents a method to combine the representations learned from these two encoders to reconstruct a complete 3D head.", "questions": "+ In line 163-164, the paper mentioned: \"This discrepancy may stem from the real samples originating from the generator, lacking the detailed feature characteristic of real-world images\". Does this means that the reconstruction quality of the unobserved part is bounded by the capacity of Panohead, compared to Encoder 1 that is trained with real-world images?\n+ How does the proposed method compare to the optimization-based inversion results presented in Panohead?", "limitations": "+ The authors have acknowledged and discussed the limitations of the societal impact of their work.\n+ I would appreciate it if the authors could provide more details on potential directions for improving the quality of the results or on other future directions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The motivation for the problem and the proposed method seem reasonable.\n+ Training a Discriminator directly on Triplane instead of the final images is an interesting and novel approach, differing from other GAN inversion work. This approach is also more relevant for full 360° reconstruction, rather than just near-frontal scenes.\n+ The paper includes a good amount of qualitative comparison, ablation studies, and quantitative comparison.", "weaknesses": "+ The head geometry after editing (Figure 8) appears to be quite flat compared to other results presented in the paper.\n+ I would appreciate it if the authors could provide visualizations of the geometry of the generated results, in addition to the final generated RGB images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback and for highlighting several strengths of our paper.\n\n1. (**W.1**) For the head geometry after editing, we realized that the example we share in the paper does not do justice; the reference images have more flat-looking hairs. We added more examples to rebuttal Fig. 7. It can be seen in those examples that the back of the head is as bumpy as the others. We will include more results in the revised paper. Thank you for bringing this to our attention. \n\n2. (**W.2**) Based on the reviewers' suggestions, we added visualizations of the geometry in rebuttal Fig. 6. Our model outputs better geometry compared to the two best-competing methods. In many examples, PTI struggles; the most obvious is the last row results, where PTI outputs a flat backside of the head. TriplaneNetv2 also struggles with many realistic details and  especially on the side and back-views. \n\n3. (**Q.1**) The reviewer makes a great point. It is true that the reconstruction quality of the unobserved part is bounded by the capacity of Panohead since we rely on its generations while training a discriminator. This is also meaningful since we use the latent space of Panohead for our inversion, and input images can only provide faithful information on the visible parts. We will add this discussion to the revised version. \n\n4. (**Q.2**) The reviewer asks about the comparisons with optimization-based methods provided by Panohead. Our main paper already included these comparisons as Panohead uses PTI optimization. We observe that PTI sometimes may achieve good results, as given in the Panohead paper; however, the method often fails for the novel views even though it takes significantly longer to run inference. It can also be seen in the geometry in rebuttal Fig. 6. Even though it is still worse than ours, PTI may achieve reasonable results, as shown in the first example. On the other hand, in many examples, it outputs very unrealistic results, as given in the last example in Fig. 6. Similar results exist in the main paper and Supplementary.  \n\n5. (**L.2**) We appreciate the reviewer asking for future directions. *Reviewer AnnS* mentioned the transformer-based architecture. We also believe using attention techniques can be a valuable research direction to distribute the knowledge of visible regions to others. Additionally, combining this method with diffusion-based techniques is another direction we would like to pursue. We will include these ideas in the revised manuscript to further stimulate research in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all reviewers for their valuable feedback. We have responded to each reviewer's questions in the rebuttal sections, and attached a PDF file with figures and tables for our additional results.

Meta-review: The reviewers acknowledge the technical contribution of this paper in encoder-based GAN inversion for Panohead, and the after-rebuttal scores have increased. After careful discussion and consideration, we are pleased to inform you that your paper has been accepted. However, there are concerns from the reviewers that need to be addressed. The paper requires a more in-depth analysis, including a thorough examination of its limitations. Additionally, the comparison with other methods should be comprehensive and avoid only cherry-picking results. For instance, we discussed the cases presented in Fig. 6, noting that the Panohead + PTI inversion results appear unusually bad.

We look forward to seeing these improvements in the final version.
Decision: Accept (poster)