StyleMorph: Disentangled 3D-Aware Image Synthesis with a 3D Morphable StyleGAN
Authors: Eric-Tuan Le
Edward Bartrum
Iasonas Kokkinos
Venue: ICLR 2023 poster (2023)
Keywords: 3D-aware GAN
Template-based
Morphable
Disentanglement
Photorealistic
Neural Radiance Field
StyleGAN

Abstract: We introduce StyleMorph, a 3D-aware generative model that disentangles 3D shape, camera pose, object appearance, and background appearance for high quality image synthesis. We account for shape variability by morphing a canonical 3D object template, effectively learning a 3D morphable model in an entirely unsupervised manner through backprop. We chain 3D morphable modelling with deferred neural rendering by performing an implicit surface rendering of “Template Object Coordinates” (TOCS), which can be understood as an unsupervised counterpart to UV maps. This provides a detailed 2D TOCS map signal that reflects the compounded geometric effects of non-rigid shape variation, camera pose, and perspective projection. We combine 2D TOCS maps with an independent appearance code to condition a StyleGAN-based deferred neural rendering (DNR) network for foreground image (object) synthesis; we use a separate code for background synthesis and do late fusion to deliver the final result. We show competitive synthesis results on 4 datasets (FFHQ faces, AFHQ Cats, Dogs, Wild), while achieving the joint disentanglement of shape, pose, object and background texture.

STYLEMORPH: DISENTANGLED 3D-AWARE IMAGE SYNTHESIS WITH A 3D MORPHABLE STYLEGAN
Bric-Tuan Le Edward Bartrum 1.2+ lasonas Kokkinos'
University College London 2 Alan Turing Institute
ABSTRACT
We introduce StyleMorph, 3D-aware generative model that disentangles 3D shape, camera pose, object appearance, and background appearance for high qual- ity image synthesis. We account for shape variability by morphing canonical 3D object template, effectively learning a 3D morphable model in an entirely unsu- pervised manner through backprop. We chain 3D morphable modelling with de- ferred neural rendering by performing an implicit surface rendering of "Template Object Coordinates" (TOCS), which can be understood an unsupervised coun- terpart to UV maps. This provides a detailed 2D TOCS map signal that reflects the compounded geometrie effects of non-rigid shape variation, camera pose, and per- spective projection. We combine 2D TOCS maps with independent appearance code to condition a StyleGAN-basee deferred neural rendering (DNR) network for foreground image (object) synthesis; use separate code for background syn- thesis and do late fusion to deliver the final result. We show competitive synthesis results on 4 datasets (FFHQ faces, AFHQ Cats, Dogs, Wild), while achieving the joint disentanglement of shape, pose, object and background texture.
INTRODUCTION
Learning the structure and statistics of the 3D world by observing 2D images is at the forefront of current vision and learning research as this can unlock applications robotics, augmented reality and graphics while also having fundamental scientific value for advancing visual perception. In this work we aim to develop the ability to do so through a model that is highly disentangled, yielding a similar level of control to that enjoyed by 3D morphable models (3DMM) (Blanz & Vetter, 1999), without requiring anything other than an unstructured set of 2D images. 3DMMs are the workhorse of facial visual effects (VFX) in the film industry and augmented reality (AR) (Egger et al., 2021), as they provide VFX creators with fine-grained, disentangled control over expression, pose, and ap- pearance. In this work we aspire to develop unsupervised counterparts for general object categories.
In particular we show that we can learn such models for several categories other than human faces while having no prior knowledge about the object topology or other 3D prior information or knowl- edge of the camera pose. We build on recent progress on 3D-aware GANs and show that we can improve the FID of the most competitive methods that use the same level of supervision (plain 2D) images), while exerting more control on the image synthesis process: we disentangle shape (e.g- gender, expression, hair style), camera pose, object appearance, and background. This allows us to do fine semantic edits, that preserve all properties beyond the one are editing, We show that this applies not only to faces but also to images of cats, dogs and and wild animals.
Relation with previous works 3D-aware cetel modelling advances have shown that one can use 2D image supervision to train 3D generative models of shape and appearance variability. Starting from standard MIPs (Schwarz et al., 2020; Niemeyer & Geiger, 2021) and subsequently custom sinusoidal-ban
networks (Sitzmann et al., 2020; Chan et al., 2021b), 3D implicit mod- els quickly delivered results competitive to those of voxel-based approaches (Nguyen-Phuoe al., 2019; 2020). Hybrid models (Gu et al., 2021; Zhou et al., 2021; Or-El et al., 2021; Chan et al., 2021a;
Project page: https::/ http:::/sstlleeophh.oom git wtwpuu...o/stttemmnnn stt.oo/rpppp/////////o/ * "quual contribution.

---

Pose
Shape
Foreground
Background
Combine all
Figure 1: Our model achieves disentangled control of image synthesis: starting from synthesized sample we change one factor at a time, and in the end show the compounded variation that we obtain by changing all. Our 3D-based conditioning signal (shown the top and bottom rows) is exclusively geometric it is hence the same as the left image's 's in the foreground and background columns, while the change effected only by the respective appearance codes (not shown).
Xue et al., 2022) have increased the resolution and quality in which images can be synthesized with- out compromising speed or memory by relying on hybrid approach that renders coarse- resolution neural features from 3D to 2D and then delegates the full-resolution image synthesis task to 2D, StyleGAN-type blocks (Karras al., 2020). These works have shown increasingly high-quality results but their hybrid nature makes it harder to have clear separation of geometry and ap- pearance or provide consistent image synthesis results when we change rigid (camera) or non-rigid (gender/expreessoonnai)) 3D geometry.
Recent works aimed at disentangling appearance from shape have incorporated 3D deformations in the synthesis process (Park et al., 2021; Pumarola et al., 2021; Gafni et al., 2021a; Su et al., 2021; Xu et al., 2021; Weng et al., 2022), but so far have remained limited to either the single dynamic scene use case, or assume pre-existing deformable model already exists for category (Gafni et al., 2021a; Xu et al., 2021; Weng et al., 2022; Su et al., 2021).
In particular for faces compelling controllable synthesis results have been obtained by recent works that combined 3DMMs with NERFs (Athar et al., 2022; Gafni et al., 2021b) or 3D-aware GANs (Liu et al., 2022; Tewari al., 2020). Still, constructing 3DMM typically requires extensive 3D scanning and manual alignment, making it only meaningful for critical categories such as faces.
Most recently Tewari et al. (2022) injected 3DMMs in GAN training, showing that one can control image synthesis through 3D warps. In our work we turn 3DMMs into first-class citizens for 3D generative modelling by combining them with Deferred Neural Rendering.
Learning 3DMMs from 2D images has also been recently achieved for monocular 3D reconstruc- tion based on limited information, such as binary segmentation masks (Kanazawa et al., 2018: Sa- hasrabudhe et al., 2019; Kokkinos & Kokkinos, 2021b), allowing us to handle a broad range of categories (Ye al., 2021; Vasudev et al., 2022): other works have provided models that can accom- modate articulation (Kulkarni et al., 2020; Kokkinos & Kokkinos, 2021a; Yang et al., 2022), and varied object topology (Duggal & Pathak, 2022), managing known shortcomings of 3DMMs. The synthesis results of these methods however rely parametric low-resolution surface and texture map. yielding synthetic-looking images.
Contributions Our work builds advances from these three strands of research to combine 3DMMs with GANs in an unsupervised manner. We show that is possible to inject the main idea of morphable models, i.e. deforming fixed "canonical" template to diverse set of "world" shapes into the design of implicit 3D networks. Existing approaches model shape variability through random input to occupancy network. Instead, we bridge 3D morphable models with 3D-aware
2

---

World Space
Template Space
Synthetic Image
TOCS map
Deferred Renderer
Deformation
ZÁK
Zb
Disciimnnttor
(o.
Real Image
Figure 2: Overview of our approach: the left side reflects geometric modelling of template shape, non-rigid shape variation (z,), camera pose (ø. 8), and perspective projection, used to produce the rendered 2D TOCS that acts as bottleneck for geometric variation. 2D TOCS maps feed into a Deferred Neural Rendering network together with latent codes for foreground and background appearance to produce high-resolution photorrealistic images trained by discriminator network.
GAN synthesis by introducing canonical coordinate system: there, the occupancy function of the object is modelled as constant (but learned) function. To model shape variation we sample 31) deformation field MLP (Zheng al., 2021) that is driven by a latent code, and use the deformation to morph the 3D canonical template thus obtaining dense correspondence between the canonical SDF space and each deformed version similarly to Palafox et al. (2021).
main spin we introduce to existing 3D-aware models consists in rendering surface-level signals instead of RGB values neural fields. This cleanly removes appearance information from shape modelling: the signal provided to the 2D synthesis network purely geometric.
In particular the surface level signals we provide conditioners are 2D maps based the 3D "Template Object Coordinates" (TOCS) that project to a given pixel 2D. This inspired from the Normalized Object Coordinates (NOCS) used for 3D pose estimation in Wang et al. (2019) and allows us to bypass the challenging 2D UV parameterization of a template surface. The template occupancy function can freely evolve during training, potentially even updating its topology.
We use these TOCS maps as a proxy to 3D geometry in tandem with object appearance code to condition a StyleGAN
deferred neural rendering (DNR) network for object (foreground) syn- thesis. By virtue of being defined with respect to template, TOCS maps endow every pixel with clear semantics (e.g. indicating an eye, nose, or ear part lands on that pixel). This is reflected in our ability to produce meaningful geometric warps through the control of the underlying 3D deformation field as shown Fig. 7.
We are also able to disentangle object (foreground) and scene (background) synthesis as in Xue et al. (2022); Chen
(2022) by using the object-based TOCS maps together with one appearance code to drive foreground synthesis and a separate code to account for background variability. The two synthesized images are combined with late fusion, using a TOCS-bate alpha mask. This allows us control all four sources of variability (shape, camera, object and scene appearance) separately, yielding highly disentangled model for 3D-aware image generation, as shown Fig. 1.
To summarise, contributions as follows:
We learn 3D morphable model of the non-rigid shape variation in object category exclusively from 2D image supervision.
We introduce Template Object Coordinates (TOCS), a deformable variant of Normalized Object Coordinates, and show that this provides powerful, deformation--qquivaiin de- scriptor of 3D shape.
We introduce TOCS maps powerful conditioning signal to a style-based 2D DNR, allowing a clear disentanglement shape and appearance conditioning.
We show unprecedented disentangled control over pose, shape, object appearance, and scene appearance for high-resolution, photorrealistic image synthesis.

---

2:0 1er0
Stage
Wordipoce
Temmoo Space
Renderer
Deformation Module
Template
Occupancy
(x))
Aggregation
Multi Layer
position
Multi Layer
Block
Stage
position
FILM Siren
Offsets
FILM Siren
RGB Block
c(x)
(Stage
Dicoiminator
ShapeMapping
Zs
Figure Architecture of the Morphable renderer: 3D deformation field warps camera rays to template coordinates, where object occupancy and appearance are modelled more easily in deformation-fre coordinate system. In the first stage this system is trained to synthesize images (amounting to a 3D deformable variant of PiGAN) and in the second stage the learned 3D deforma- tion model is used to provide 2D TOCS maps that condition the DNR.
2
METHOD
Our method takes input unstructured collection of RGB images of an object category. Our ob- jective to learn generative model for images of the same distribution that (i) disentangles camera, shape and foreground/background appearance variation, (ii) expresses shape variability through the deformation of a learned template shape and (iii) allows us to efficiently synthesise high-resolution, realistic images.
As shown in Fig. 2. we sample separate latent codes from unit Gaussian distributions to control the various factors of variation: Zig. Abg. 2, control foreground/backgroume appearance, and shape respectively. We similarly sample our cameras azimuth and elevation coordinates (ø, 0) from Gaus- sian distributions, determine each pixels ray origin.
On the left side we model the effects of geometry in three steps: a deformation-free object model learned in template object coordinates (TOCS) as a constant implicit function. This is connected to world coordinates through a 3D deformation field, represented as implicit function driven by latent code that accounts for non-rigid shape variability. We nonlinearly warp the camera rays from world to template coordinates and use differentiable rendering to produce "Template Object Coor- dinates" (TOCS), which represent the Normalized Object Coordinates (NOCS) (Wang et al., 2019) in template space. The compounded effects of camera pose, non-rigid deformation and perspective projection are reflected in this 2D TOCS map, that encapsulates all geometry information.
On the right side we have 2D StyleGAN-based Deferred Neural Rendering (DNR) network that takes as input the geometric conditioning of the 2D TOCS maps together with two separate latent codes for foreground (object) appearance and background (scene) appearance and synthesizes high- resolution photorcalistic images as dictated by a discriminator network.
We call the left side "Morphable Renderer" network and train it firstly on its together with some auxiliary losses order to bootstrap the whole system as detailed in Sec. 2.1. The network can be understood as morphable variant of PiGAN (Chan et al., 2021b) obtained by injecting into the synthesis process a layer for deformable differentiable rendering. We provide more information on the second part of the network in Sec. 2.2 and conclude with specifications of the training losses and optimization procedure in Sec. 2.3.
2.1 MORPHABLE RENDERER BLOCK
The first part of our training pipeline, detailed in Fig. 3, is aimed at capturing all 3D geometrical aspects of image formation, including both non-rigid ("shape") and rigid ("pose") sources of vari- ability. Following Chan et al. (2021b); Gu et al. (2021); Or-El al. (2021) model pose variability by positioning our camera on a unit sphere at elevation and azimuth (0, 0), pointing at the origin.
We generate morphs through Gaussian shape code zs that drives a SIREN deformation model g. R3 + R3 In particular 2a predicts frequencies % and phase shifts 83. 7s. 3g = ShapeMapping(zz) which in turn modulate the layers of gs: (p) = SIREN Ys: .g). We compute the deformation field f. R + R by adding the predicted offset each world-space point p. warping them into template space: fs : p > p gp(p) similarly to Park et al. (2021).
4

---

Zfg
Appearance Mapping
2D TOCS Map
Foreground
High-Res Image
Spatial Encoder
StyleGan2
Ti
fg
High-Res
A)
Discriminator
Zbg
StyleGan2
Tba
Background
Figure 4: Architecture of the Deferred Neural Renderer: spatial features extracted from the 2D TOCS map are used to condition StyleGan2 generator for foreground synthesis. Background, generated from second StyleGan2 generator is composited for final synthesis.
In more detail when performing ray-tracing for each pixel, we sample world points along the corre- sponding ray r(t) = o- dt where r(t) is the world-space position of a ray sample at distance / along the ray, with ray origin o and direction Our deformable rendering layer, shown 2D case, maps the world-space ray r to deformed template-space ray by passing it through the deformation field f. defined above:
f(t) fr(()) r(t) + (r())).
(1)
This operation associates world points with their template pre-images and can thereby eliminate the variability caused by non-rigid object deformation. Once mapped to template space, the ray samples are passed through second SIREN network with constant (i.e. instance-agnostic) learned frequencies and phase shifts, that represents our template implicit field.
In the first training stage we use this block to directly render low oow-resolution RGB images. We follow the SDF-based method introduced in Or-El et al. (2021) for the raysampling procedure, and convert SDF values to occupancies through a sigmoidal function (a))). An appearance code drives an implicit model for RGB intensity which combined with the occupancy function yields rendered 2D RGB image. We additionally return an estimated alpha map from our volume renderer. obtained by integrating the ray-occupancies. This allows to obtain 4-channel RGBA image prediction during low resolution training. We use a 4-channel low-resolution discriminator as well and for the real alpha masks use approximate segmentations obtained using the unsupervised method of Labels4free (Abdal et al., 2021) (detailed in the appendix). This can be understood as providing weak silhouette supervision our 3D shape model, while also priming the appearance and shape modelling to focus on the object region.
In the second training stage we repurpose this block to provide 2D TOCS maps the following DNR network. In particular, when tracing a ray we now integrate the template-coordinate values F(t) along the ray to obtain the TOCS value:
TOCS(r)= - J. ((F(t))(((d)
where TOCS(r) is the 2D TOCS map value corresponding ray r, w(f(t)) the SDF-based weight computed at the template point f(t) as in Or-El et al. (2021) and f, and ty are the near and far ray limits, which are fixed across training; we approximate the integral using discrete sampling.
We note that during the DNR training, rather than using an appearance or radiance function C(f(t)) as an argument the weighted integral in Eq. we directly use the ray's template coordinate position F((), amounting to the TOCS representation; if instead of F(() we were using r(!) this would be providing us with the standard 2D NOCS map. NOCS maps prove to be strong conditioning signals for DNR, which by itself is new and interesting result, but as experiments show, there a substantial improvement FID scores by using TOCS instead of NOCS through the deformation field while obtaining dense shape correspondence for free.
We limit the resolution of the rendered TOCS map to 642 (regardless of final image resolution) as this keeps the memory and computation footprint of this stage low, while sufficing for detailed pose and shape conditioning. We also do not use the RGB rendering blocks during the second stage of training the final inference process.
5

---

Figure 5: From a fixed viewpoint, we render TOCS maps with different shape codes, and pass them the DNR with the background and foreground appearance code. The Morphable Renderer generates diverse TOCS maps, corresponding to different hair styles and face shapes, which are with strong alignment with the synthesised RGB. We visualise the template coordinates using equicontours, to show the surface correspondences between shapes.
Figure 6: Starting from source sample (yellow border) we demonstrate disentangled appearance control over shape (green border), foreground (blue border) and background (red border).
2.2 TOCS-CONDITIONED DNR
Our Deferred Neural Rendering pipeline, shown in Fig. 4, follows recent works on UV-driven Style- Gan networks for human synthesis (Sarkar et al., 2021; AlBahar et al., 2021) or segmentation-driven synthesis (Park et al., 2019). First we process the rendered TOCS map with small residual net- work (Spatial Encoder in Fig. 4) that preserves the original spatial resolution, but transforms the 3-D TOCS values to richer, 64-D representation. The resulting foreground feature tensor Tig replaces the early blocks of a StyleGAN generator, providing a pose conditioning signal that captures the joint effects of shape and camera variables. Our TOCS map contains informative geometry-based values only on the surface of the deformed shape (which we interpret as the foreground region). use separate 2D Background Generator to
Since wish to generate detailed backgrounds, create a background RGB image Tog. Our Background Generator consists of a small StyleGAN generator, which outputs full-resolution background image. Zbg is passed to the mapping network of the Background Generator, so that it controls the content of the background.
Our foreground generator similarly consists of StyleGAN blocks, which used to upsample the low resolution feature tensor Tyg to a full resolution RGBA tensor. This contains the foreground RGB image Ifg, and upsampled alpha map A. Similarly the background case, 2ig is used to modulate the activations of the DNR StyleGan blocks through Adaptive Instance Normalization, via a mapping network.
We composite the foreground RGB Ij with the background RGB Ing using alpha blending with the upsampled alpha mask of the foreground generator: I C Iig + (1 Ibg
2.3 NETWORK TRAINING
We provide information the process followed for training system provide more details in the appendix and will share code for reproducibility.
Training strategy: We adopt 2-stage low-to-high resolution training strategy following Or-El et al. (2021) to limit memory footprint and keep decent batch size for optimal overall performance and speed. For stage our deformable volume renderer (incorporating template and deformation-

---

Figure 7: Deformation results AFHQ Dogs and Cats. We linearly interpolate shape instances through deformation-offset space. The deformation field captures complex non-linear motion of the dog's ears, and relative motion of the cat's head to the torso.
offset networks) is trained as low-resolution (64²) image generator, in order to learn a realistic shape model. To generate low-resolution training images without the use of the DNR, we include an RGB prediction block in the implicit field following Schwarz et al. (2020); Chan et al. (2021b); Or-El et al. (2021), which not necessary for the full-resolution synthesis.
In stage 2, we freeze the volume renderer weights, and train the DNR as a full-resolution 2D gener- ator conditioned the projected TOCS maps. We prune the RGB synthesis block from the volume renderer as is no longer needed. We use the volume renderer only to render TOCS maps for given poses and shape codes, and train our DNR with appearance conditioning purely in the 2D domain. The DNR can be trained at arbitrary resolutions conditioned on the 64 TOCS maps.
Loss functions: In both cases we apply the non-saturated GAN training objective (Mescheder al., 2018) with R1 regularisation and path regularisation to force our generator to learn to synthesise realistic images, which we denote as Egen.
During the first stage of training. we additionally regularise the volume renderer using shape losses stabilize optimisation and avoid degenerate local minima, as detailed in the appendix.
For the 2nd phase of training, we fix the volume renderer and drop shape losses. We additionally use an L1 loss Lalpha to ensure consistency between the low-resolution and upsampled alpha maps. The generator training objective is therefore Lgen + aaipha- We report the reader to Appendix 11.1.3 for explicit loss equations.
EXPERIMENTS
3.1 DATASETS
We evaluate our pipeline together with 11 state-of-the-art baselines the FFHQ (Karras et al., 2019) and AFHQ (Choi et al., 2020) datasets. FFHQ is composed of 70k centred images human faces, with variety of challenging backgrounds and poses. AFHQ is composed of centred images of animal faces, split into categories: 5653 Cats, 5239 Dogs and 5238 Wild. We report FID results on FFHQ and all AFHQ datasets to evaluate the quality of our model's 2D synthesis.
3.2 QUANTITATIVE EVALUATION
Baselines: We compare our synthesis results to several state-of-the-an 3D-GAN models by FID score Table. 1. We group the recent photo-realistic stylegan- methods together at the bot- tom. Our FID scores are competitive with the sfate-of-the-art models, despite our additional dis- entanglement and template-based shape model constraints, which can in principle compromise the generator's 's flexibility. The only directly comparable method is GiraffeHD (Xue et al., 2022) which however still lacks an underlying template coordinate system (hence making it hard e.g. to transfer masks defined on template, shown in Appendix). The only prior 3D-GAN work which represents shape via a deformable template is the independently developed Disentangled3D work (Tewari al,, 2022). However, this significantly underperforms
the FFHQ dataset in Ta-

---

FFHQ 256 FID
AFHQ 256 FID
Disentanglement
Template
Unposed
Cat Wild Dogs
Joint
Shape
Scene
HoloGAN
75.00
78.00
X
K
GRAF
71.00
121.00
X
GIRAFFE
31.20
31.00
V
V
pi-GAN
34.56
38,92
X
X
GRAM
29.80
X
Disentangled3D
28.18
X
StyleNERF
8.00
14.00
SG-Based
X
CIPS3D
6.97
StyleSDF
11.50
12.80
X
EG3D
4.80
3.88
SG-Based
GiraffeHD
11.93
12.36
Ours
7.91
4.29
3.49
13.95
Table I: Comparisons with the state-of-the-arr on 3D-aware GANs; top block: direct 3D methods; bottom block: 3D-2D hybrids. We note that (a) EG3D uses a pre-existing 3D pose estimation network, hence is using weak 3D supervision (unlike other methods) (b) only Disentangled3D and GiraffeHD are comparable to us in terms of disentanglement. We provide more details in the text.
ble. 1 due the absence of StyleGAN synthesis blocks. Furthermore, this work does address foreground/backgroume synthesis.
Ablation: To justify our architectural choices, we perform an ablation study Table. 2. We use multiple more metrics beyond the ones used in Table. 1, as these allow us to better understand the tradeoff' for our different design choices. full discussion of consistency scores can be found in the supplemental material. For fair apples-to-apples comparison, we compare all models after equal training time (96 hours) on equivalent hardware.
Our proposed method is in row Late fusion, and differentiable TOCS rendering. The first check we perform row is what happens if rather than TOCS we render NOCS, while using the exact same system i.c. the 3D world coordinate of point, rather than its 3D template coordinate. We observe decreased performance across almost all metrics. This sign that TOCS helps both generate sharper samples, and ensure disentanglement in synthesis while NOCS provides more fuzzy signal, since it lacks that clarity of template-level conditioning.
Row 3 repeats this experiment but removing the deformable component and replacing it with a standard implicit model for occupancy driven by a shape code. This reduces bit the FID score and increases appearance variability wrt Row but drastically worsens consistency scores. This indicates the added value of a deformable module to model shape variation as opposed to doing direct occupancy estimation through MLP the difference is more pronounced wrt Row
Row 4 finally shows how results change when we have early fusion. We have a slightly smaller FID (potentially due the looser earlier fusion which gives the generator more flexibility) but clearly worse alpha consistency scores indicating that the synthesis results can override the conditioning TOCS map and blend foreground with background.
A last ablation, shown in Row 5a examines if we can improve the view consistency of our model, by directly optimising the new-consistency metric during training by formulating as differentiable loss. We observe a trade-off between view-consistency and image quality, optimising view- consistency causes a corresponding increase FID score. Nevertheless, we find that we can obtain similar FID score to StyleSDF with significantly lower view-consistency.
3.3 QUALITATIVE EVALUATION
Our full pipeline model generates high-resolution, photorealistic images. We observe in Fig. that the Morphable Renderer can generate diverse shapes capturing the complex shape variability present in the FFHQ dataset. Our approach allows to factor shape variability into canonical tem- plate instance and shape-specific deformation, directing the full capacity of the network towards capturing intra-category shape variation. Furthermore, note in Fig. 5 that DNR synthesis
8

---

ld
Ablations
Perf.
Consistency
Variation
FG/BG
Morph
NOCS
Reproj. Loss
FID
View
Alphat
Shape
Appearance
Appearancef
Compositing
/TOCS
Late
TOCS
8.309
15.843
88.65%
0.798
0.015
0.093
Late
NOCS
8.90
17.493
89.4%
0.810
0.018
0.078
3
Late
NOCS
8.531
19.026
89.85%
0,807
0.016
0.089
Farly
TOCS
88186
13.638
85.87%
0.870
0.018
0.079
Late
TOCS
|12.31|
8.12
189.90%
0.815
0.015
0.085
5b
StyleSDF
11.5 13.60
Table 2: Quantitative comparisons of ablations; please see text for details
Foreground
Background
Figure 8: StyleMorph-inversinn by inverting our pipeline, we can reconstruct a real input image with its underlying 3D structure and exert full control over multiple sources of disentanglement.
mains closely aligned to the TOCS map, generating photo-realistic fine details in the hair and eyes without deviating from the coarser geometric structure. The strong alignment exhibited by the DNR between TOCS values and RGB, combined with the dense semantic correspondences between pro- jected TOCS maps, enables the deforrmation-equiiariint AFHQ synthesis results shown Fig. 7. We synthesise images conditioned by source shape instances on the left, then linearly scale the de- formation field values up to the target shape instances shown on the right. The resulting non-linear motion in the TOCS maps is tracked equivariantly in the RGB synthesis, as seen in the raising of the dogs ear and twisting of the cat torso.
As detailed in Sec. 2, our approach uses separate latent codes for shape, background and foreground appearance. By varying codes in one latent space and holding the others fixed, our model exhibits disentangled synthesis across each factor of variation, and all datasets. We observe in Fig. 6 that deformation network can change woman's hair style, and twist the cats head relative to its torso. The background generator synthesise variety of structures without impacting the foreground, whilst varying the foreground code results in diverse appearances, whilst maintaining foreground/background consistency.
We show in Figure how real image can be inverted using Pivotal Tuning (Roich et al., 2021) to recover the 3d model from single image. Thanks our disentangled synthesis, the inital input image can be edited via multiple sources of control: pose, foreground, background and shape.
4 CONCLUSION
In this work we have shown that 3D deformation-based control over image synthesis can be achieved without any compromise to state-of-the-art synthesis quality. We provide more results and videos in the Appendix and will make our code publicly available.
Acknowledgements All final models described this research were trained using the Baskerville Tier 2 HPC service
This was funded by EPSRC Grant EP/TO22221/1 and operated by Advanced Research Computing at the University of Birmingham. This work was also supported in part by JADE: Joint Academic Data science Endeavour 2 under the EPSRC Grant EP/T022205/1, & The Alan Turing Institute under EPSRC grant EP/N510129/ We grateful Gabriel Browstow and Niloy Mitra for providing useful discussions and insights.

---

REFERENCES
Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. Labels4free: Unsupervised segmen- tation using stylegan. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 13970-13979, October 2021." 5
Badour AlBahar, Jingwan Lu, Jimei Yang, Zhixin Shu, Eli Shechtman, and Jia-Bin Huang. Pose with Style: Drteil-preserving pose-guided image synthesis with conditional stylegan. ACM Trans- actions on Graphics, 2021.
ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neural 3d portraits. In Computer Vision and Pattern Recognition (CVPR), 2022.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In SIGGRAPH, 1999.
Eric R Chan, Connor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- 3d generative adversarial networks. arXiv preprint arXiv:2112.07945, 2021a.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5799-5809, 2021b. 1, 4,
Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, and Jingyi Yu. Sofgan: A portrait image generator with dynamic styling. ACM Trunsactions on Graphics (TOG), 41(1):1-26, 2022.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188-8197, 2020. 7
Shivam Duggal and Deepak Pathak. Topologically deformation fields for single-view 3d
reconstruction. CVPR, 2022. 2
Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhöfer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 3d morphable face models past, present and future. ACM Transactions on Gruphics, 39, 2021.
Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8649 8658, June 2021a.
Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8649-8656 June 2021b.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08900 2021. 1,
Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning category-
specific mesh reconstruction from image collections. In ECCV, 2018. 2
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
adversarial networks. In Proceedings the
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pp. $11088199 2020. 2
Filippos Kokkinos and lasonas Kokkinos. Learning monocular 3d reconstruction of articulated categories from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 1737-1746, June 2021a. 2
10

---

Filippos Kokkinos and lasonas Kokkinos. To the point: Correspondence-driven driven monocular 3d cate- gory reconstruction, 2021b. 2
Nilesh Kulkarni, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation-awan canonical surface mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Y. Kung. 3d-fim gan: Towards 3d-controllable face manipulation. In ECCV, 2022. 2
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference machine learning, pp. 3481-3490. PMLR, 2018. 7
Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Holo- gan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 7588-7597 2019. 1
Thu H Nguyen-Phuoc, Christian Richardt, Long Mai, Yongliang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. Advances in Neural Information Processing Systems, 33:6767-6778, 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 1
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. arXiv e-prints, pp. arXiv-2112, 2021. 1, 1,4,5, 5, 6,
Pablo Palafox, Aljaž Božič, Justus Thies, Matthias Nießner, and Angela Dai. Npms: Neural para- metric models for 3d deformable shapes. 2021. 3
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021. 2,
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with apatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 6
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021.
Daniel Roich, Ron Mokady, Amit Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Trans. Graph., 2021. 9
Mihir Sahasrabudhe, Zhixin Shu, Edward Bartrum, Riza Alp Guler, Dimitris Samaras, and lasonas Kokkinos. Lifting autoencoders: Unsupervised learning of fully-disentangled 3d morphable model using deep non-rigid structure from motion. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019. 2
Kripasindhu Sarkar, Vladislav Golyanik, Lingjie Liu, and Christian Theobalt. Style and pose control for image synthesis of humans from single monocular view, 2021. 6
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 1,7
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.

---

Shih- Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radi- ance fields for learning human shape, appearance, and pose. In Advances in Neural Information Processing Systems, 2021. 2
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images. In CVPR, 2020.
Ayush Tewari, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt, et al. Disen- tangled3d: Learning 3d generative model with disentangled geometry and appearance from monocular images. arXiv preprint arXiv:2203. 15926, 2022. 2,7
Kalyan Alwala Vasudev, Abhinav Gupta, and Shubham Tulsiani. Pre-train, self-train, distill: A simple recipe for supersizing 3d reconstruction. In Computer Vision and Pattern Recognition (CVPR), 2022. 2
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019. 3,4
Chung-Yi Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. arXiv preprint arXiv:2201.041277 2022. 2
Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for ren- dering and temporal reconstruction of humans in motion. In NeurIPS, 2021. 2
Yang Xue, Yuheng Li, Krishna Kumar Singh, and Yong Jae Lee. Giraffe hd: A high-resolution 3d-aware generative model. arXiv preprint arXiv:2203. 14954, 2022. 2, 3,
Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul. Banmo: Building animatable 3d neural models from many casual videos. In CVPR, 2022. 2
Yufei Ye, Shubham Tulsiani, and Abhinav Gupta. Shelf-superrised mesh prediction in the wild. In
Computer Vision and Pattern Recognition (CVPR), 2021. 2
Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu. Deep implicit templates for 3d shape represen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1429-1439, June 2021. 3
Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. Cips-3d: A 3d-aware generator of gans based on cattiiooalll-inddeeeeeeee pixel synthesis. arXiv preprint arXiv:2110.09788, 2021.
12


Reviews:
[{"review_id": "ti7VYOTptL", "reviewer": "Reviewer_cRkf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces StyleMorph, a high-resolution 3D-aware generative model with morphable geometry and disentangled appearance. \nThe network is composed of two components, where a Morphable Renderer is trained to yield the TOCS map, and a Deferred Neural Renderer learns to translate the TOCS map to a high-resolution image.", "summary_of_the_review": "In general, I think this paper has good qualitative results, but lacks novelty, and the writing is not clear and objective enough.", "strength_and_weaknesses": "Strength:\n1. The image quality in the paper looks good.\n2. The supplementary materials are detailed.\n\nWeaknesses:\n1. Morphable Renderer lacks novelty. The main idea of the Morphable Renderer (disentangling the geometry and appearance + warping the learned template space to modify the geometry) is too similar to the Disentangled3d[1]. Actually, in my opinion, they are almost the same.\n2. Deferred Neural Render lacks novelty. The proposed DNR takes the TOCS (a sort of geometry representation) as input and yields a corresponding high-resolution image, which is similar to StyleNeRF[2], SofGAN[3], etc. and has little novelty from my perspective. \n3. The experiments are not sufficient.\n    - Efficiency of TOCS. Although the TOCS outperforms NOCS, there lacks a comparison with some naive methods, e.g. directly feeding the Deferred Neural Render with the low-resolution image/depth outputs of Morphable Renderer, or using an on-the-shelf super-resolution model, e.g. GFPGAN[4], to enhance the image quality.\n    - No expression control experiments.\n4. The writing is exaggerated and not clear:\n    - \"we disentangle shape (e.g. gender, expression, hair style), camera pose, object appearance, and background\". No gender or expression disentanglement was found. And the background disentanglement here is trivial, where they use a pretrained 2D generator to synthesise the plain background. It is strange to see the background keeping static while the foreground is rotated, as shown in the YouTube video.\n    - \"yielding a similar level of control to that enjoyed by 3D morphable models (3DMM)\". I think this paper cannot control facial expressions as in the 3DMM. And the warping field is hard to edit.\n    - \"show that we can improve the FID of the most competitive methods\". I did not find experiment results to show this general improvement.\n    - \"changing 3D expression, pose etc\". Did not find experiment results.\n    - \"but their hybrid nature makes it harder to have a clear separation of geometry and appearance\". why? The hybrid architecture does not conflict with the disentangled design.\n5. Appearance-geometry misalignment. In Fig.1 in the main paper, the geometry of column \"foreground\" and column \"background\" is the same, which suggests the corresponding images have the same shape. However, when looking at the images, you will notice a change of expression when adjusting the appearance, indicating the appearance and geometry are not aligned well semantically.\n\n[1] Ayush Tewari, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt, et al. Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images. arXiv preprint arXiv:2203.15926, 2022. \\\n[2] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985, 2021. \\\n[3] Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, and Jingyi Yu. Sofgan: A portrait image\ngenerator with dynamic styling. ACM Transactions on Graphics (TOG), 41(1):1–26, 2022. \\\n[4] Wang, Xintao, et al. \"Towards real-world blind face restoration with generative facial prior.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lEmpPMLmR8", "reviewer": "Reviewer_XLzz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a 3D-aware generative model to disentangle 3D shapes, camera poses, object appearance, and background appearance when synthesizing high-quality images, which could realize non-rigid shape variation in an object category exclusively from 2D image supervision.  To achieve a clear disentanglement, it proposes Template Object Coordinates (TOCS) to provide a powerful, deformation-equivariant descriptor of 3D shapes.  It provides impressive qualitative results with disentangled controls, and competitive quantitative results on the evaluated benchmark datasets.", "summary_of_the_review": "In general, I think it is an interesting work, showing impressive results.  Although I am a little worried about its contributions and novelties, I tend to support its acceptance at the current stage.  I would reconsider my rating when the other reviews are available. ", "strength_and_weaknesses": "Strength:\n1. It is an interesting paper that could generate 3D-aware and high-quality images with disentangled controls over 3D shape, camera pose, foreground appearance, and background appearance.\n2. They utilize 3D morphable models for synthesizing high-quality images by introducing a canonical coordinate system, and therefore they could deform the 3D shape by morphing the 3D canonical template.\n3. They show impressive synthetic images with disentangled controls.\n\nWeakness:\nThe current manuscript seems to be a technical report, and the proposed method seems to be a comprehensive combination of existing methods.  I think it is an interesting work, but I am not sure if it is good enough for an ICLR publication.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Kwqykuydzt", "reviewer": "Reviewer_2TB7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel 3D-aware generative image model, StyleMorph, which can disentangle 3D shape, camera pose, appearance and background for high-quality image synthesis.  By bridging 3D morphable models with GAN synthesis and a canonical coordinate system, dense correspondences among generated objects can be provided. Experiments show disentangled control over pose, shape, object appearance, and background appearance for high-quality image synthesis. ", "summary_of_the_review": "Overall, I think the idea is interesting and the proposed \"template object coordinates\" is a good contribution in image synthesis. ", "strength_and_weaknesses": "Pros:\n1. It is a good idea to bride the 3D morphable models with GAN synthesis, enabling dense correspondence between generated objects.\n2. The method is clearly explained and the text is well-written. \n\n I don't have major concerns about this paper. In general, I like the idea of 3D control of 2D generative models. A few minor comments are listed below.\n1) Can the proposed model be applied to generic object image generation, such as cars, and buildings? Would the method work for objects with large intra-class deformations? \n2) More qualitative and quantitative evidence of the 3D disentanglement is needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G68WUgx26m", "reviewer": "Reviewer_J5vi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce a 3D-aware generative model that can control 3D shape, camera pose, object appearance and background independently. The author connect 3D morphable modelling with deferred neural rendering by performing an implicit surface rendering of TOCS, and construct a purely geometric, deformation-equivariant 2D signal that reflects the compounded geometric effects of non-rigid shape, pose, and perspective projection. The experiments of 4 datasets shows the good disentanglement capability of the proposed approach. \n", "summary_of_the_review": "Overall, I think it is a good paper. Though I am not experted in 3D-aware GANs, some results in the paper impressed me and I tend to give positve rating to this paper. ", "strength_and_weaknesses": "Strengths:\n1. To disentangle the key elements such as camera pose, apperance et al. is a important research topic, and the proposed approach is interesting and achieves good performance on this topic. Some illustrated results are impressive. \n2. Most of the quantitative results outperform existing works, demonstrating the superiority of the proposed approach. The experiments are conducted on four datasets, which are sufficient. \n\nWeaknesses and questions:\n1. Some works like [1] provide more than the four choices in this paper for manipulation, like illumination, expression et al, while this work only covers the four(camera pose, back gournd, appearence, shape). In other word, I think the shape/appearance manipulation cannot support fine-grained manipulation, and this paper lacks detailed manipulation capability.  \n[1] A. Tewari et al, StyleRig: Rigging StyleGAN for 3D Control over Portrait Images, CVPR 2020\n2. The quality of the reconstructed 3D shapes seems coarse compared with EG3D [2], i.e., Figure 2 in supplementary file.\n[2] ER Chan et al. EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks, CVPR 2022\n3. (1) Is TOCS map provide all necessary information for stage 2 training? \n(2) How about train the whole framework end-to-end? \n(3) How does the GAN loss in stage 1 helps stage 2 trainin? It seems only TOCS goes into stage 2 but the generated image in stage 1 not goes to stage 2. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8-F9q3hpRO", "reviewer": "Reviewer_tGFV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method to synthesize photo-realistic images of objects from a given category while the approach allows for disentangled representation in terms of the camera pose, crude object shape, and foreground/background appearance. The approach introduces the concept of template object coordinates (TOCS) which both sidestep the need to perform 3D shape UV unwrapping and also allow for representing deformed instances of the shape in the common canonical form. The authors show competitive results while allowing for more semantic control over the synthesized images.", "summary_of_the_review": "Thanks to the creativity of the method, compelling results and good quality of the presentation I am leaning towards acceptance. There are a couple of minor remarks which I would like the authors to comment on (see the weaknesses section).", "strength_and_weaknesses": "# Strengths\nWhile the approach builds on many existing components, it combines them in a creative way to achieve a disentangled representation of the camera/shape/appearance while being able to synthesize compelling images. The method is unsupervised.\nThe paper is very well structured and written in a comprehensive way where it is easy to follow the core ideas.\nThe experiments are compelling and the authors justify their design choices through a thorough ablation study.\n\n# Weaknesses\n## Related Work\nWhile the Related Work section is comprehensive, I would suggest that the authors still consider reviewing and citing the following two publications, which use very similar concepts as the authors of this paper. Specifically, Nerfies [1] makes use of the concept of deforming the camera rays to support a deforming scene and NPMs [2] employ a pose network which also uses a deformation field to express the correspondence between a canonical SDF space and its deformed (posed) version.\nProbably should cite Nerfies and NPMs as both use the similar idea of warping the camera rays and learning the deformation field of a template.\n\n## Methodology\n- Fig. 4: It would help the reader if the variables from the text of section 2.2 (e.g. T_{FG}, I_{BG} etc.) were indicated as the inputs/outputs of the individual blocks in the image.\n- Section 2.1: It would help the reader to provide the intuition behind how w() of Eq. 2 is computed and what does it represent. Similarly, the concept of TOCS could be very briefly introduced directly in the Methodology on top of linking to the paper [Wang et al. 2019] which introduced the related NOCS.\n- Section 2.3: Similarly to the comment above, it would help the reader if the authors explicitly noted the mathematical formulas for the used loss function.\n- Fig. 3: The figure indicates that the Shape and Appearance Mapping modules share weights but this is never described in the text. Could the authors elaborate?\n\n## Typos:\n- pg. 3: \"encapsulates all geometr information.\" -> \"encapsulates all geometry information.\"\n- pg. 4: \"In a first/second stage\" -> \"In the first/second stage\"\n\n## Questions for the authors\nSince the 2D TOCS map is immediately passed through a spatial encoder (which produces a 64D feature vectors), one could imagine that the network is not really incentivized to produce a meaningful 3D shape (encoded by the 2D TOCS). The Figure 7 indicates that the TOCS are reasonable shapes given the corresponding RGB images, but it is also clear that there is quite some high-frequency noise on the surface geometry. Could the authors comment on why this is happening? Alternatively, have the authors experimented with regularizing the shape which the TOCS represent (to me closer to a more plausible 3D shape) and what were the observations?\n\n- [1] K. Park. et al. Nerfies: Deformable Neural Radiance Fields. ICCV'21\n- [2] P. Palafox. et al. NPMs: Neural Parametric Models for 3D Deformable Shapes. ICCV'21\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster