Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images

Authors: bahri  batuhan bilecen
ahmet berke gokmen
aysegul  dundar
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 08:50:03

Abstract: 3D GAN inversion aims to project a single image into the latent space of a 3D Generative Adversarial Network (GAN), thereby achieving 3D geometry reconstruction. While there exist encoders that achieve good results in 3D GAN inversion, they are predominantly built on EG3D, which specializes in synthesizing near-frontal views and is limiting in synthesizing comprehensive 3D scenes from diverse viewpoints. In contrast to existing approaches, we propose a novel framework built on PanoHead, which excels in synthesizing images from a 360-degree perspective. To achieve realistic 3D modeling of the input image, we introduce a dual encoder system tailored for high-fidelity reconstruction and realistic generation from different viewpoints. Accompanying this, we propose a stitching framework on the triplane domain to get the best predictions from both. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-aware triplane discriminator. Experiments reveal that our approach surpasses the existing encoder training methods qualitatively and quantitatively.

Main paper: 
 introduction: 
 In the realm of generative models, 2D GANs have gained renown for their remarkable ability to achieve striking realism through adversarial training, effectively capturing intricate details and textures to produce visually convincing images, especially on face images [18, 19] . However, their inherent limitation lies in their lack of depth perception, which restricts their applicability in three-dimensional contexts. In contrast, 3D GANs mark a groundbreaking advancement by seamlessly integrating Neural Radiance Fields (NeRF) into their architecture [13, 9, 14, 5] . This integration not only allows them to match the realism of their 2D counterparts but also ensures consistency in three-dimensional geometry. While extensive studies have focused on 2D GAN inversion [42, 29, 31, 4, 33, 28] , recent efforts have seen the proposal of inversion methods tailored for 3D GANs [39, 7, 21] .2D GAN inversion techniques focus on projecting the images into the GAN's natural latent space to enhance editability and achieve high fidelity to the input image; however, inverting 3D GAN models presents additional challenges. This process requires accurate 3D reconstruction, which means realistic filling of invisible regions, ensuring coherence and completeness in the resulting three-dimensional scenes. Recently, inversion methods for 3D GANs have been developed, firstly, optimization-based and then encoder-based methods. Optimization-based methods [20, 35, 38] employ reconstruction losses to invert images into a latent code specific to the given view. Furthermore, network parameters are optimized by generating pseudo-multi-view images from the optimized latent codes to enhance detail preservation. Such optimization is required for each inference image; it is time-consuming and requires GPUs with large memory. Therefore, researchers focus on encoder-based methods [39, 7, 21] . While successful inversions are achieved with these methods, they rely on EG3D [9] framework [39, 7, 21] , which is constrained to synthesizing near-frontal views. However, our work utilizes PanoHead [5] , a method capable of rendering full-head image synthesis, enabling a comprehensive 360-degree perspective. This advancement introduces additional challenges, particularly concerning the invisibility of many parts in the input image. Despite this, the inversion model is expected to reasonably predict and reconstruct these occluded regions to ensure high-quality 3D reconstruction. Our experiments demonstrate that extending the methods proposed for EG3D is ineffective.When projecting images onto PanoHead's latent space, we observe a trade-off between achieving high-fidelity reconstruction of the input image and generating realistic representations of the invisible parts of the head. Some models can perfectly reconstruct the image from a given view but produce unrealistic outputs when the camera parameters change. Conversely, other models generate realistic representations under varying camera parameters but fail to achieve high-fidelity reconstruction of the input image. To achieve high-fidelity reconstruction of the input image and realistic representations of the invisible parts of the head simultaneously, we train a dual encoder. One encoder specializes in reconstructing the given view, while the other focuses on generating high-quality invisible views. We propose stitching the triplane domain generations to produce the final result. This approach combines the outputs from both encoders to achieve both high-fidelity reconstructions of the given view and high-quality representations of the invisible parts of the head. To achieve seamless stitching, both encoders must output consistent results despite being specialized for different tasks. For this reason, we carefully train these encoders using specialized losses, including an adversarial loss based on our novel occlusion-aware triplane discriminator. This ensures that both encoders learn to produce consistent and complementary outputs, enabling seamless stitching of generations for the final result. Our contributions are as follows:• To achieve high fidelity to the input and realistic generations for different camera views, we train dual encoders and introduce a stitching pipeline that combines the best predictions from both encoders for visible and invisible regions. • We propose a novel occlusion-aware discriminator that enhances both fidelity and realism.• We conduct extensive experiments to show the effectiveness of our framework. Quantitative and qualitative results show the superiority of our method compared to the state-of-the-art. Some visual results can be seen in Fig. 1 .2 Related works 3D Generative Models. Generative Adversarial Networks (GANs), coupled with differentiable renderers, have achieved significant strides in generating 3D-aware multi-view consistent images.While early efforts, such as HoloGAN [24] , operating on voxel representations, subsequent works shifted towards mesh representations [27, 15] , and the latest advancements are built around implicit representations [10, 14, 26, 25] . Among implicit representations, triplane representations have emerged as a popular choice due to their computational efficiency and the high-quality outputs [9, 13, 5] . The architectures of works like EG3D [9] and PanoHead [5] bear resemblance to the structure of StyleGAN2 [19] . They consist of mapping and synthesis networks, generating triplanes which are subsequently projected to a 2D image through volumetric rendering operations akin to those used in NeRF [23] . While EG3D is trained on the FFHQ [18] dataset with limited angle diversity, PanoHead achieves a 360-degree perspective in face generation thanks to their dataset selection and model improvements. In our work, we delve into PanoHead's latent space and construct our inversion encoder based on PanoHead.In recent years, GAN inversion, particularly in the context of StyleGAN, has garnered significant attention due to its extensive editing capabilities. The primary objective of these studies is to embed an image into StyleGAN's latent space, enabling subsequent modifications.Initially, this was approached through latent optimization, where latent codes were iteratively adjusted using back-propagation to minimize the reconstruction loss between the generated and target images [11, 1, 2, 19, 30] . For 3D-aware GAN model inversions, supplementary heuristics have been introduced into the optimization process. These include considerations like facial symmetry [38] and multi-view optimization strategies [35] . However, such methods are computationally intensive as they require optimizing each image's latent codes. Moreover, while minimizing the reconstruction loss can yield visually similar results, it does not guarantee that the image resides within the natural latent space of GANs. This distinction is crucial for effective image editing. Without aligning with StyleGAN's inherent latent space, reconstructed images may not respond correctly to editing techniques, thus limiting their practical utility. This consideration also extends to 3D-aware GAN inversion methods, where encoding geometric information is paramount. Even if an input image can be faithfully reconstructed, its realism may falter when observed from alternative viewpoints, emphasizing the importance of aligning with the GAN's native latent space.To enhance efficiency, image encoders have been specifically trained for the inversion task, initially targeting StyleGAN [42, 29, 31, 4, 28, 36, 37] , and more recently for EG3D [39, 7, 21] . These specialized encoders capitalize on insights gained from training datasets to swiftly project images into latent spaces. Moreover, they can be trained with diverse objectives beyond mere image reconstruction. For instance, some employ discriminators to compare generated and real images and latent space discriminators to ensure inversion aligns with the GAN's natural latent space. As a result, these methods generally offer faster inversion processes. This study focuses on 3D-GAN inversion, specifically targeting PanoHead [5] . The task poses significant challenges due to PanoHead's ability to capture a comprehensive 360-degree perspective, necessitating the prediction of a substantial portion of invisible elements by inversion encoders. Our experiments demonstrate that models trained for EG3D are ineffective in this context. The overall architecture of PanoHead is given in Fig. 2 . Resemblant to EG3D, PanoHead utilizes a mapping network that takes a random vector z and the camera conditioning π mapping . After z is mapped to a w, StyleGANbased backbone G generates a tri-grid triplane. Unlike EG3D, PanoHead's triplanes have 3 times the number of channels in comparison, hence the name tri-grid. This approach is stated to ease 360-degree synthesis. The resultant triplane is then rendered via a volumetric neural renderer R with pose π render and super-resolved to yield a synthesized image.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our claims include utilizing dual encoders trained to specialize in visible and occluded regions, along with a stitching pipeline that seamlessly combines the most accurate predictions from both encoders. To further enhance the reconstruction of occluded regions, we propose an occlusion-aware discriminator, enabling the image encoder to generate realistic features for these challenging areas. Extensive experiments validate the effectiveness of our approach, demonstrating superior performance across multiple benchmarks and scenarios. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitations are given in Sec. 6. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
training an encoder: 
 This section introduces the general pipeline of the encoders employed in our dual-encoder framework. Each encoder takes an input image I and predicts the latent code w + , which is then passed to the generator to produce triplane features. These synthesized features are then fed into the renderer to generate a 2D image with a specified camera parameter π cam , as described by Eq. ( 1):w + = E 1 (I) I sv out = R(G(w + ), π cam ) (1)Here, I sv out denotes the output rendering for the same view as the input, and E 1 represents the encoder. While the W + space allows for leveraging priors embedded in the generator, its limited expressive power in reconstructing image details has been noted due to the information bottleneck of its 14 × 512 dimensions. To address this limitation, both 2D inversion techniques [33, 28] and 3D GAN inversion methods [7] permit higher-rate features to pass to the generators, facilitating the capture of fine details. In 3D GAN inversion methods, these higher-rate features are encoded through a smaller second network and transmitted to the triplane features. We adopt a similar approach in our encoders. We refer to the final output as I sv final . Further details of the architecture are provided in the Appendix. The primary challenge in this setting arises from establishing appropriate training objective losses, as our training dataset consists solely of single images, providing ground truth only for the rendered image from the same view as the input. For these output and ground-truth pairs, we set the usual reconstruction losses, namely, LPIPS perceptual loss [41] , L 2 reconstruction loss (MSE), and ArcFace [12] based identity loss as given in Eq. ( 2):EQUATIONWhile models trained with the objective given in Eq. ( 2) learn to reconstruct a given view, they often struggle to generalize and produce realistic features from other camera views. Consequently, while our first encoder is trained with the objective in Eq. ( 2), we design an adversarial-based loss objective for our second encoder. This second encoder generates realistic predictions for invisible views, as explained in Section 3.3. To achieve a realistic reconstruction of the 3D model, represented in a triplane structure, it is essential to guide the encoder for visible views and overall coherence. Since we lack one-to-one ground truth to guide the triplane structure, we experiment with various setups incorporating adversarial losses. A naive approach to utilize adversarial loss would be to render estimated triplanes from other views and assess the realism of these 2D images using a discriminator. However, our experiments observe that this setup hinders the model's ability to learn high fidelity to the input image, as will be further detailed in Section 5.2. Moreover, randomly rendering different views can only guide limited parts of the triplane structure rather than the overall.To overcome this limitation and avoid the computational burden of rendering unnecessary views, we explore the possibility of training a discriminator in the triplane domain. In our training process for the triplane discriminator, we follow a procedure where we sample latent vectors Z + and generate in-domain triplanes using PanoHead, serving as our real samples. Meanwhile, the fake samples are triplanes generated from encoded images, as depicted in Fig. 3 . Despite the observed improvement in reconstructions facilitated by this adversarial loss, we note a persistent challenge hindering the network's ability to achieve high fidelity to the input. This discrepancy may stem from the real samples originating from the generator, lacking the detailed feature characteristic of real-world images. Therefore, this may lead the encoder to omit to encode realistic facial details if they are absent in the synthesized samples. We propose our occlusion-aware discriminator to overcome this limitation. This discriminator is exclusively trained with features corresponding to occluded pixels. This approach ensures that triplane features associated with visible regions, such as a frontal face, are not utilized for discriminator training. Additionally, we introduce a masking mechanism for synthesized triplanes to mitigate any distribution mismatch arising between encoded and synthesized triplanes. This masking process contributes to aligning the distributions of real and fake samples, further enhancing the coherence of the training dynamics.We find the set of visible points based on the depth map of the given view via inverse rendering. Specifically, the occlusion mask O π R is estimated by Eq. ( 3):EQUATIONwhere p[x, y, z] is the triplane coordinates, π R is the extrinsic camera parameters of the input view, D is the depth map from the input view, K is the intrinsic camera parameters, I[u, v, 1] are the homogeneous coordinates of the input image I rendered from the input view. More clearly, from the current camera pose, we map back to the depth values to obtain the mask of visible regions (1-O π R ). Then, we invert the visible region mask to obtain O π R . The utilization of occlusion masks has been previously investigated in 3D methodologies, albeit in different contexts. For instance, they have been used in generating pseudo-ground truth images to facilitate optimization-based 3D reconstruction [38] and integrated into passing high-rate residual features to the triplane [39] . However, it is the first time used in the discriminator. This allows for a selective focus on regions where the encoder may encounter challenges in faithfully replicating realism.Compliant with recent advancements in adversarial training, we follow WGAN loss [6] for L adv in Eq. ( 4), where T sv final and T synth are encoded and Z + synthesized triplanes, respectively. Details are given in the Appendix. In our approach, we train two encoders: the first, as outlined in Section 3.2, and the second, augmented with an additional adversarial loss detailed in Section 3.3. While the initial encoder excels at reconstructing high-fidelity facial images from the input, it often produces unrealistic results for other viewpoints, as depicted in Fig. 5 . Conversely, the second encoder yields better overall outcomes, albeit with slightly diminished fidelity to the input face.EQUATIONOur aim is to devise a dual-encoder pipeline that harnesses the strengths of both encoded features.To achieve this, we leverage the occlusion masks derived in Section 3.3, as illustrated in Fig. 4 . By combining the visible portions from Encoder 1 and the occluded segments from Encoder 2, we generate our final output, as demonstrated in the last row of Fig. 5 .While each encoder contributes partially to the ultimate feature, achieving seamless integration necessitates consistency in the output of both encoders despite their distinct specializations. For instance, if Encoder 1 flawlessly renders a given view of the face but fails to capture the correct geometry, artifacts may arise in the combined result. Thus, it remains imperative to train both encoders comprehensively to ensure an overall high-quality outcome.4 Experimental Setup 
training: 
 We combined images of FFHQ [18] and LPFF [34] and split it for training (∼140k) and validation (∼14k). CelebA-HQ [17] and multi-view MEAD [32] are employed for additional evaluation. We removed face portrait backgrounds for training and evaluation datasets, applied camera and image mirroring during training, and performed pose rebalancing proposed in [34] as data augmentation.We utilized the same dataset to train competitive methods for fair evaluation. The models are trained for 500k iterations with a batch size of 3 on a single RTX 4090 GPU. The learning rate is 1e -4 for both encoders and the occlusion-aware discriminator. Ranger is utilized as the optimizer, which is a combination of Rectified Adam [22] with Lookahead [40] .The baseline models are provided in Table 1 . We note that no encoder pipelines are aimed for full 360-degree head reconstruction. We train the models with the author's released code to invert images into PanoHead's latent space.We report L 2 , LPIPS [41] and ID [12] scores for original-view reconstruction, which measure the fidelity to the input image. For the novel-view quality, we measure Fréchet inception distance (FID) [16] . Since our validation datasets have limited angle variance, we measure the distance between 1k randomly synthesized and 1k encoded real-life image distributions. The images are rendered from varying yaw angles, covering the 360-degree range to include occluded regions. We also utilize the multi-view image dataset MEAD dataset. Specifically, we fed front MEAD images (0 • yaw) to all methods, rendered them from novel views of MEAD (from 60 • to 0 • yaw), and compare them with their corresponding ground truths. This allows us to report LPIPS and ID metrics alongside the FID metric for the novel views. 
comparisons with state-of-the-art: 
 Table 1 : Quantitative scores on various test sets. achieves similar same-view reconstruction scores as our method, its FID score is also significantly worse. Overall, the pSp and e4e methods perform worse than ours in all metrics. PTI achieves similar results to our method but takes ×250 longer and requires a GPU with large memory.Category Method FFHQ + LPFF CelebAHQ Time L2 ↓ LPIPS ↓ ID ↑ FID ↓ L2 ↓ LPIPS ↓ ID ↑ FID ↓ sec ↓ Optimization W + optimCategory Method LPIPS ↓ ID ↑ FID ↓ ±60 • ±30 • ±60 • ±30 • ±60 • ±30 • Optim. W + optWe extend the quantitative analyses to multi-view with the MEAD dataset. Specifically, we feed front MEAD images (0 • yaw) to all methods, rendered them from novel views of MEAD (from 60 • to 0 • yaw), and compare them with their corresponding ground truths. Table 2 reveals that our method significantly improves over compared methods especially in LPIPS and ID metrics.Qualitative results are shown in Fig. 8 and Fig. 6 . The competing methods produce unrealistic outputs when viewed from angles other than the input view. Among these, PTI achieves good front and side views but fails to generate realistic hair from the back. Our method achieves the best results overall.We also include mesh comparisons in Fig. 9 . Ours is better than the most recent encoder-based method [7] and generally performs well compared to PTI. Note that PTI mostly generates smoother meshes (row 3) but can sometimes struggle depending on the input sample (row 6). In Table 3 and Fig. 7 , we present an ablation study demonstrating the effectiveness of our occlusion-aware triplane discriminator quantitatively and qualitatively. The first row of results shows that not using any discriminator achieves good reconstruction of the given view, as indicated by LPIPS and ID scores. This is also visible in the first-row in Fig. 7 . However, this approach fails to generalize to novel views, as evidenced by the FID score and visual results. In our framework, we chose to embed images into the W + space. Table 4 presents an ablation study that explores utilizing different projection spaces and various combinations of training data. Training an encoder to project images to the Z or Z + space, where Z is sampled 14 times, results in better FID scores. However, this comes at the cost of high-fidelity reconstruction. Similarly, transitioning to a less constrained W + space enhances fidelity to the input but worsens the FID score. Addressing this challenge necessitates additional measures, such as the proposed dual encoder setup with the occlusion-aware discriminator objective. It is important to note that the distinction between the Z + and W + space arises from the camera parameters incorporated into the mapping network. While Z + employs various samples of Z, it adheres to the same set of camera parameters assigned to the mapping network. In contrast, the W + space does not impose such constraints during encoding. Given that the real image dataset we use primarily consists of limited camera poses, typically front-view faces, we investigate training the encoder with synthetically generated images from PanoHead. However, solely utilizing synthetic images generated from samples of Z + to introduce more diversity compared to Z leads to poor performance on real image validation sets regarding reconstruction quality. When combining synthetic and real images, we observe an improvement compared to using them individually. Lastly, we show the results of using the dual encoder in Table 5. The visual results were previously presented in Fig. 5 . The dual mechanism leverages the strengths of both encoders, achieving the same LPIPS and ID scores as Encoder 1 while also producing FID scores very similar to those of Encoder 2.We follow the reference-based editing in [8] in our pipeline. This method encodes input images, and edits are performed in the triplane space. This approach utilizes the fact that triplanes have a canonical space, allowing for the transfer of local parts from one triplane to another. Fig. 10 demonstrates a successful transfer of hairstyle from a reference image to the target human in 3D. Another advantage of encoder-based models over the optimization ones is the feasibility of such applications. For example, this would not be possible with PTI since the generator is fine-tuned for each sample, preventing the copying of features from one image to another in the encoded feature space. In summary, this study introduces a 3D GAN inversion framework that projects single images into the latent space of a 3D GAN for accurate 3D geometry reconstruction. While prior encoders excel at synthesizing near-frontal views, they struggle with diverse 3D scenes, motivating our exploration of alternatives. Using PanoHead's 360-degree synthesis, we developed a dual encoder system for high-fidelity reconstruction and realistic multi-view generation. A stitching mechanism in the triplane domain ensures optimal predictions from both encoders. With specialized losses, including an occlusion-aware triplane discriminator, our framework achieves superior qualitative and quantitative performance over existing methods.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The paper does not provide open access to the code as of now; however, the datasets are public and can be obtained from the pointed references. The code is planned to be released through the project page provided in Abstract. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion and broader impacts: 
 Broader Impacts. Our framework has the potential to revolutionize the movie industry, AR, and VR, enabling applications like animating portraits and creating realistic game environments. However, it raises ethical concerns, particularly the risk of "deep fakes". We stress the need for safeguards to ensure the ethical use of this technology.Limitations. We acknowledge that there is room for improvements in the fidelity of images, the realism and 3D-consistency of generations (see Fig. 11 , row 2), and the smoothness of the meshes (see Fig. 9 ). Since the projection is made onto the latent space of PanoHead, our method may not handle out-of-domain or tail samples well (such as images with high-frequency details or accessories). For instance, our method struggles with hats, as demonstrated in the first row of Fig. 11 . We recognize that, in certain cases, the artifacts are visible in the back middle of the head and are more noticeable in the mesh rendering as shown in Fig. 9 . Additional research is required. Encoder 1 and 2. For our Encoder 1 and 2, we employ 2-stage encoding for W + and high-rate F features, also seen in common with other style-based encoder methods [29, 3, 4, 39, 7] . We opted for the architecture in [29] for W + stage (GradualStyleEncoder) and in [7] for F stage (TriplanenetEncoder), both for Encoder 1 and 2. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Broader impacts are provided in Section 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We describe the potential risks in Section 6.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly mentioned the original owners of previous work via references, and complied with the licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not share new assets as of now.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper neither involves crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Ablation encoders. GradualStyleEncoder is used for the Z + encoder, where resulting 14 latent vectors are later passed through the mapping network with truncation ψ = 0.85 and canonical front camera pose. However, for the Z encoder, a smaller variation of [29] (BackboneEncoderUsingLastLayerIntoW) is implemented. This choice is due to Z being less expressive compared to Z + (1 dim vs. 14 dim) and hence a smaller encoder being sufficient. The same mapping network parameters for the Z + case are also utilized for Z encoder outputs.Occlusion-aware triplane discriminator. We follow a feedforward network approach with a channel bottleneck for our triplane discriminator D (Table 6 ). Noting that the occluded triplane dimensions are [batch_size,3,96,256,256], we first add each depth slice to the channel dimension to get the input shape as [batch_size,288,256,256]. Output dimensions are [batch_size,1]. We do not utilize saturating functions such as sigmoid at the end since WGANbased loss [6] is utilized.Ablation image discriminator. For the back-view image discriminator used in ablations, we change the input channel number of the model in Table 6 from 288 to 3.The training objective for Encoder 1 with parameters θ E1 is given in Eq. (5) .EQUATIONwhere I sv is the same-view reconstruction of I with pose π. Coefficients are set as λ 1 = 0.8, λ 2 = 1.0, λ 3 = 0.5.The training objective for Encoder 2 with parameters θ E2 is given in Eq. ( 6).T enc = G(E 2 (I)) I sv = R(T enc , π) arg min θ E 2 λ 1 L LPIPS (I sv , I) + λ 2 L 2 (I sv , I) + λ 3 L identity (I sv , I) + λ 4 L adv (D(O π T enc )) (6)where T enc is the encoded triplane of image I, D is the discriminator, O π is the occlusion mask from the same view π. λ 1,2,3 are the same as in Eq. ( 5), and λ 4 = 0.001. L adv in Eq. ( 6) is given in Eq. ( 7):EQUATION)where softplus is a smooth and differentiable approximation to ReLU.The training objective for discriminator D with parameters θ D is given in Eq. ( 8).T enc = G(E 2 (I))T synth = G(M(z + ∼ N (0, I 14 ), π front )) arg min θ D λL adv (D(O π T enc ), D(O π T synth )) (8)where T synth is the synthesised triplane from randomly sampled z + , M is the mapping network, π front is the canonical front pose. L adv in Eq. ( 8) is given in Eq. ( 9):EQUATIONλ is set as 0.5.We jointly train Encoder 2 and discriminator D in a traditional adversarial fashion. We further employ R1-regularization to encourage L1-lipschitzness [6] to justify using softplus, where its weighting coefficient is 10 and is applied every 16 iterations.Our method can handle diverse ethnicities and challenging input views, demonstrated in Fig. 12 .We also showcase additional visual results for competing methods in Figs. 13 to 20, ablation on discriminator in Fig. 21 , ablation on dual-encoder structure in Figs. 22 to 24. The first columns are input images, the second columns are reconstructions from the input views, and the rest are renderings of models from novel views. NeurIPS Paper Checklist 


Decision: The reviewers acknowledge the technical contribution of this paper in encoder-based GAN inversion for Panohead, and the after-rebuttal scores have increased. After careful discussion and consideration, we are pleased to inform you that your paper has been accepted. However, there are concerns from the reviewers that need to be addressed. The paper requires a more in-depth analysis, including a thorough examination of its limitations. Additionally, the comparison with other methods should be comprehensive and avoid only cherry-picking results. For instance, we discussed the cases presented in Fig. 6, noting that the Panohead + PTI inversion results appear unusually bad.

We look forward to seeing these improvements in the final version.

Reviews: review: paper_summary: paper_summary: This paper presents a framework for 3D GAN inversion aimed at reconstructing high-fidelity 3D head models from single 2D images. A dual encoder system that combines one encoder specialized in high-fidelity reconstruction of the input view with another focused on generating realistic representations of invisible parts. An occlusion-aware triplane discriminator that enhances both fidelity and realism.
strength_weakness: strength_weakness: 1. The writing and language of the paper need substantial improvement.
2. The innovation of this paper may be a little lacking. This paper proposes a new method, but the insights shown in the method are limited and it is difficult to generate insights into other problems.
3. There is also a lack of in-depth research on the proposed method.
4. While some limitations are mentioned, a more in-depth analysis of when and why the method fails would be valuable.
questions: questions: 1. From Figure 4, it can be seen that Encoder 2 may be the main solution to the problem in this paper. Is there any way to introduce the features of Encoder 1 into Encoder 2? It seems that Encoder 1 is more like a subsidiary of Encoder 2. For this reason, introducing two encoders is feasible, but not elegant enough.
2. I'm curious about more training details. For example, how much data was used for training?
3. The evaluation is primarily on face datasets (FFHQ, LPFF, CelebA-HQ). It's unclear how well the method generalizes to more diverse head shapes or ethnicities.
4. In Encoder 2, is the relevant information about the person's head implicitly included? Is there any experiment to prove this?
limitations: limitations: While some limitations are mentioned, a more in-depth analysis of when and why the method fails would be valuable.
paper_summary: paper_summary: This paper introduces a novel encoder-based method for 3D pano-head GAN inversion. Unlike previous 3D GANs focused on near-frontal views, this work designs a dual-encoder system tailored for both high-fidelity reconstruction and realistic generation from multiple viewpoints. The method incorporates a stitching mechanism within the triplane domain, ensuring optimal predictions from both encoders. This results in superior performance compared to existing methods, both qualitatively and quantitatively.
strength_weakness: strength_weakness: - The input images in the study are primarily front-view; it would be beneficial to include some from extreme viewpoints.
- Further assessment of the model's generalization is required for more complex input images, such as those with heavy makeup.
- The quantitative experiments currently focus on single-view cases; incorporating results from other viewpoints may provide a more comprehensive evaluation.
questions: questions: - As discussed in Weaknesses.
- Will the choice between a CNN-based or Transformer-based encoder affect the reconstruction results? Current works like GOAE, use transformer-based encoder for better different-view input instead of just front-view.
limitations: limitations: - Discussion of failure cases is necessary.
- The formula symbols in the paper need to be clearly distinguished, such as E1 and E2. Clear differentiation of these symbols should be included in the main text.
paper_summary: paper_summary: This paper addresses the challenge of 3D GAN inversion from a single image, focusing on a method called PanoHead designed for generating 360 views of human heads. Unlike  optimization-based approaches, this work adopts an encoder-based technique to reduce inversion times. The authors observe that varying encoder training objectives result in different advantages and disadvantages in the inversion output. Consequently, the authors propose using two encoders and introduce a stitching technique in the triplane feature space to combine the strengths of each encoder.
strength_weakness: strength_weakness: 1. The claim on lines 168-169, stating that triplane features associated with visible regions (e.g., a frontal face) are excluded from discriminator training and thus do not receive gradients, is not entirely accurate. The method forms a mask by back-projecting the depth map onto the triplane coordinates, which limits the mask to points lying on the surface of the visible region. However, since the model uses triplane features for volume rendering, the features used for rendering the frontal view are not necessarily confined to the surface of the face. Features behind that surface may also contribute to color aggregation during volume rendering, potentially leaving some features used for rendering the visible regions uncovered by the mask.
2. Some details lack clarity, as noted in the questions below.
questions: questions: 1. Line 177: How do you obtain the depth map D? Do you use a monocular depth estimation method, or do you extract it from the NeRF model?
2. Figure 5: Are E1 the encoder from section 3.2 and E2 the encoder from section 3.3? If so, why is the triplane feature of E1 masked by the occluded regions and E2 masked by the visible regions? According to the paper, the proposed method appears to use the visible regions of the triplane for E1 and stitch them with the occluded regions of E2. Is this a mistake in the figure?
3. Line 223: The FID is calculated between the rendered image from the encoded input and "1k randomly synthesized" images. Which method is used to generate these "1k randomly synthesized" images? Is it PanoHead? Are all methods compared against the same set of images synthesized by the same method?
limitations: limitations: The authors included discussions on both the limitation and broader impact.
paper_summary: paper_summary: This paper introduces an encoder-based method to do the GAN inversion task, especially for the full head inversion. The occlusion aware discriminator is interesting and reasonable. The results are good.
strength_weakness: strength_weakness: There is no significant weakness in this paper. I have a question about the training process. Are these two encoders trained on the face first and then on the occluded parts? Is the occlusion-aware encoder initialized with the facial encoder?
questions: questions: see the weakness
limitations: limitations: see the weakness
paper_summary: paper_summary: The paper proposes an encoder-based approach to GAN-inversion for full 3D head reconstruction from a single image. To address the challenge of reconstructing the entire 360° head, the method employs two encoders: one for high-fidelity reconstruction of the input image (typically near-frontal) and another for generating realistic invisible parts of the head (e.g., back of the head). Finally, the paper presents a method to combine the representations learned from these two encoders to reconstruct a complete 3D head.
strength_weakness: strength_weakness: + The head geometry after editing (Figure 8) appears to be quite flat compared to other results presented in the paper.
+ I would appreciate it if the authors could provide visualizations of the geometry of the generated results, in addition to the final generated RGB images.
questions: questions: + In line 163-164, the paper mentioned: "This discrepancy may stem from the real samples originating from the generator, lacking the detailed feature characteristic of real-world images". Does this means that the reconstruction quality of the unobserved part is bounded by the capacity of Panohead, compared to Encoder 1 that is trained with real-world images?
+ How does the proposed method compare to the optimization-based inversion results presented in Panohead?
limitations: limitations: + The authors have acknowledged and discussed the limitations of the societal impact of their work.
+ I would appreciate it if the authors could provide more details on potential directions for improving the quality of the results or on other future directions.

Comments: title: Response to the rebuttal
Re: Rebuttal by Authors
comment: We would like to thank the reviewer for reading our response and for considering the additional comments and discussions with other reviewers. We are pleased to hear that the concerns have been addressed and appreciate the revised score.
We would like to thank the reviewer for reading the rebuttal and for considering the additional experiments. If there are any concerns that have not been fully addressed, please let us know, and we would be happy to address them.
I have read the author's response, and I will keep the score.
I have read the author's response, as well as the comments and discussions with other reviewers. The author has addressed my concerns. I will improve my score.
We would like to thank the reviewer for their feedback and for considering our rebuttal. We appreciate their willingness to discuss our responses with the other reviewers.
I have read the rebuttal and thank you for the clarifications, they are helpful. I will take the responses into consideration and discuss with other reviewers before making the final decision on my rating.
We are happy to hear that the rebuttal addressed reviewer's questions. We would like to thank the reviewer for their thorough review and constructive feedback.
I have read the rebuttal and got all my questions answered.