Title: StyleGAN knows Normal, Depth, Albedo, and More
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative models

StyleGAN

Depth

Normals

Segmentation

Intrinsic Images

Albedo

Shading
Abstract: Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo, or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images.  The procedure is straightforward. We show that if StyleGAN produces $G({\bf w})$ from latent ${\bf w}$, then for each type of intrinsic image, there is a fixed offset ${\bf d}_c$ so that $G({\bf w}+{\bf d}_c)$ is that type of intrinsic image for $G({\bf w})$. Here ${\bf d}_c$ is {\em independent of ${\bf w}$}.  The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime.  We show that there are image transformations StyleGAN will {\em not} produce in this fashion, so StyleGAN is not a generic image regression engine.  

It is conceptually exciting that an image generator should ``know'' and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN's intrinsic images are robust to relighting effects, unlike SOTA methods.

StyleGAN knows Normal, Depth, Albedo, and More

Abstract

Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images. Our procedure is straightforward. We show that, if StyleGAN produces G ( w ) from latent w , then for each type of intrinsic image, there is a fixed offset d c so that G ( w + d c ) is that type of intrinsic image for G ( w ) . Here d c is independent of w . The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime. We show that there are image transformations StyleGAN will not produce in this fashion, so StyleGAN is not a generic image regression engine.

It is conceptually exciting that an image generator should 'know' and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN's intrinsic images are robust to relighting effects, unlike SOTA methods.

1 Introduction

Barrow and Tenenbaum, in an immensely influential paper of 1978, defined the term 'intrinsic image' as 'characteristics - such as range, orientation, reflectance and incident illumination - of the surface element visible at each point of the image' [6]. Maps of such properties as (at least) depth, normal, albedo, and shading form different types of intrinsic images. The importance of the idea is recognized in computer vision - where one attempts to recover intrinsics from images - and in computer graphics - where these and other properties are used to generate images using models rooted in physics. But are these representations in some sense natural? In this paper, we show that a marquee generative model - StyleGAN - has easily accessible internal representations of many types of intrinsic images (Figure 1), without ever having seen intrinsics in training, suggesting that they are.

We choose StyleGAN [30, 31, 29] as a representative generative model because it is known for synthesizing visually pleasing images and there is a well-established literature on the control of StyleGAN [65, 11, 75, 14, 53, 57]. Our procedure echoes this literature. We search for offsets to the latent variables used by StyleGAN, such that those offsets produce the desired type of intrinsic image (Section 4; Figure 2). We use a pre-trained StyleGAN which has never seen an intrinsic image in training, and a control experiment confirms that StyleGAN is not a generic image regressor. All this suggests that the internal representations are not 'accidental' - likely, StyleGAN can produce intrinsic images because (a) their spatial statistics are strongly linked to those of image pixels and (b) they are useful in rendering images.

There may be practical consequences. As Section 5 shows, the intrinsic images recovered compare very well to those produced by robust image regression methods [28, 10, 18, 20], both qualitatively and quantitatively. But StyleGAN produces intrinsic images that are robust to changes in lighting conditions, whereas current SOTA methods are not. Further, our method does not need to be shown many examples (image, intrinsic image) pairs. These practical consequences rest on being able to produce intrinsic images for real (rather than generated) images, which we cannot currently do. Current SOTA GAN inversion methods (eg [3, 47, 12, 61]) do not preserve the parametrization of the latent space, so directions produced by our search do not reliably produce the correct intrinsic for GAN inverted images . As GAN inversion methods become more accurate, we expect that generative models can be turned into generic intrinsic image methods. Our contributions are:

Demonstrating that StyleGAN has accessible internal representations of intrinsic scene properties such as normals, depth, albedo, shading, and segmentation without having seen them in training.

Describing a simple, effective, and generalizable method, that requires no additional learning or fine-tuning, for extracting these representations using StyleGAN's latent codes.

Showing that intrinsic images extracted from StyleGAN compare well with those produced by SOTA methods, and are robust to lighting changes, unlike SOTA methods.

2 Related Work

Generative Models: Various generative models, such as Variational Autoencoders (VAEs)[33], Generative Adversarial Networks (GANs)[21], Autoregressive models [58], and Diffusion Models [16], have been developed. These models have made significant improvements in training and output quality through novel loss functions and stability enhancements despite initial challenges with blurry outputs [49, 45, 30, 48, 27]. In this work, we focus on StyleGAN [30, 31, 29] due to its exceptional ability to manipulate disentangled style representations with ease. We anticipate that analogous discoveries will be made for other generative models in the future.

Editing in Generative Models: A variety of editing techniques allow for targeted modifications to generative models' output. One prominent example is StyleGAN editing [53, 60, 62, 52, 74, 15, 46, 14, 11, 12, 2, 55], which allows for precise alterations to the synthesized images. Similarly, a handful of editing methods have emerged for autoregressive and diffusion models [23, 5, 13, 69, 39, 32, 37, 59].

In the context of StyleGAN editing, there exist several approaches such as additive perturbations to latents [65, 11, 75, 53, 57], affine transformation on latents [62, 26], layer-wise editing [64], activationbased editing [8, 7, 15], and joint modeling of images and labeled attributes [52, 71, 34, 35]. These methods facilitate nuanced and specific changes to the images.

In our study, we adopt straightforward and simplest additive perturbations to latents, rather than learning new transformations or engineering-specific layer modifications. By searching for small latent code perturbations to be applied across all layers, we allow the model to learn and modulate how these new latent representations influence the generated output. This process minimizes the need for intricate layer-wise manipulation while still achieving desired edits and providing valuable insights into the model's internal latent structure.

Discriminative Tasks with Generative Models: An emerging trend involves leveraging generative models for discriminative tasks. Examples of such work include Labels4Free [1], GenRep [24], Dataset GAN [71], SemanticGAN [34] RGBD-GAN [40], DepthGAN [54], MGM [4], ODISE [63], ImageNet-SD [50], and VPD [72]. These approaches either use generated images to improve downstream discriminative models or fine-tune the original generative model for a new task or learn new layers or learn new decoders to produce desired scene property outputs for various tasks.

In contrast to these methods, our approach eschews fine-tuning, learning new layers, or learning additional decoders. Instead, we directly explore the latent space within a pretrained generative model to identify latent codes capable of predicting desired scene property maps. This process not only simplifies the task of utilizing generative models for discriminative tasks but also reveals their inherent ability to produce informative outputs without extensive modification or additional training.

Intrinsic Images: Intrinsic images were introduced by Barrow and Tenenbaum [6]. Intrinsic image prediction is often assumed to mean albedo prediction, but the original concept 'characteristics ... of the surface element visible at each point of the image' ([6], abstract) explicitly included depth, normals, and shading; it extends quite naturally to semantic segmentation maps too. Albedo and shading (where supervised data is hard to find) tend to be studied apart from depth, normals, and semantic segmentation (where supervised data is quite easily acquired). Albedo and shading estimation methods have a long history. As the recent review in [20] shows, methods involving little or no learning have remained competitive until relatively recently; significant recent methods based on learning include [25, 67, 36, 20]. Learned methods are dominant for depth estimation, normal estimation, and semantic segmentation. Competitive recent methods [17, 28, 44, 10, 19] require substantial labeled training data and numerous augmentations.

What is known about what StyleGAN knows: Various papers have investigated what StyleGAN knows, starting with good evidence that StyleGAN 'knows' 3D information about faces [41, 70], enough to support editing [43, 42, 56]. Searching offsets (as we do) yields directions that relight synthesized images [11]. In contrast, we show that StyleGAN has easily accessible representations of natural intrinsic images, without ever having seen an intrinsic image of any kind .

3 Background

3.1 StyleGAN

StyleGAN [30, 29] uses two components: a mapping network and a synthesis network. The mapping network maps a latent vector z to an intermediate space w . The synthesis network takes w , and generates an image x , modulating style using adaptive instance normalization (AdaIN) [22]. Stochastic variation is introduced at each layer by adding noise scaled by learned factors. The architecture of StyleGAN can be summarized by the following equations:

where f is the mapping network, g is the synthesis network, and n is a noise vector.

3.2 Manipulating StyleGAN

StyleGAN's intermediate latent code, denoted as w , dictates the style of the image generated. The w + space is a more detailed version of the w space, where a unique w vector is provided to each layer of the synthesis network [62]. This allows for more fine-grained control over the generated image at varying levels of detail.

Editing in StyleGAN can be achieved by manipulating these w vectors in the w + space. We do this by identifying a new latent code w + ′ that is close to the original w + but also satisfies a specific

Scene Property Maps

from off-the-shelf Network

⊕

1

2

3

4

5

𝒍

Learned Components

Directions (d

i

)

Modified

Style Code

w

+

+

d

i

𝒛	~	𝑵(𝟎, 𝟏)

Synthesis

Network

Mapping

Network

Dense

Predictor

Style

Code

w

+

1

2

3

4

5

Normal

Depth

Segment

Albedo

Shading

Regression Loss

Normal

Depth

Segment

Albedo

Shading

(a) Searching for directions (d

i

) corresponding to different scene intrinsic map in Latent (

w

+

) Space

5. Shading

4. Albedo

3. Segment

Image

1. Normal

2. Depth

editing constraint c [65, 11, 75, 53, 57]. This problem can be formulated as:

where d ( c ) computes a perturbation to w based on c . d ( c ) can be found using methods like gradient descent. The edited image is then generated from the new latent code w + ′ .

4 Searching for Intrinsic Offsets

Our approach is illustrated in Figure 2. We directly search for specific perturbations or offsets, denoted as d ( c ) , which when added to the intermediate latent code w + , i.e., w + ′ = w + + d ( c ) , yield the desired intrinsic scene properties. Different perturbations d ( c ) are used for generating various intrinsic images such as normals, depth, albedo, shading, and segmentation masks. To search for these offsets, we utilize off-the-shelf pretrained networks from Omnidata-v2 [28] for surface normals, Zoe-depth [10] for depth, EV A-2 [18] for semantic segmentation, and Paradigms for intrinsic image decomposition [20] to compute the desired scene properties for the generated image x = G ( z ) , We employ a Huber-loss (smooth L1-Loss) to measure the difference between generated intrinsic and off-the-shelf network's predicted intrinsics. Formally, we solve the following optimization problem:

where P is a function that computes the scene property map from an image. By solving this problem, we obtain a latent code w ′ capable of generating an image x ′ that are scene properties as x . We also note that task-specific losses, such as scale-invariant loss in depth or classification loss in

segmentation, do not significantly contribute to the overall performance. This observation suggests that our approach is robust and generalizable, opening the door for additional applications in other variants of scene property prediction. We found that incorporating task-specific losses, such as angular loss for normal prediction, slightly improves the quality of predicted intrinsic images.

5 Accuracy of StyleGAN Intrinsics

For the intrinsic of type c , we search for a direction d ( c ) using randomly selected images generated by StyleGAN using w + and a reference intrinsic image method. Although approximately 200 images can successfully identify the required directions, we used 2000 unique scenes or generated images in our experiments. Note that this is not a scene-by-scene search. A direction is applicable to all StyleGAN images once it is found. For each generated image, we obtain target intrinsics using a SOTA reference network. We then synthesize intrinsic images from StyleGAN using the formula w + + d ( c ) . These synthesized intrinsic images are subsequently compared to those produced by leading regression methods, using standard evaluation metrics.

Importantly, the StyleGAN model used in our work has never been trained on any intrinsic images. We use a pretrained model from Yu et al. [66] and that remains unaltered during the entire latent search process. Off-the-shelf networks are exclusively utilized for latent discovery, not for any part of the StyleGAN training. Overall time to find one intrinsic image direction is less than 2 minutes on an A40 GPU. In total, less than 24 hours of a single A40 GPU were required for the final reported experiments, and less than 200 hours of a single A40 GPU were required from ideation to final experiments.

We have no ground truth. We evaluate by generating a set of images and their intrinsics using StyleGAN. For these images, we compute intrinsics using the reference SOTA method and treat the results (reference intrinsics) as ground truth. We then compute metrics comparing StyleGAN intrinsics with these results; if these metrics are good, the intrinsics produced by StyleGAN compare well with those produced by SOTA methods. In some cases, we are able to calibrate. We do so by obtaining the intrinsics of the generated images using other SOTA methods (calibration metrics) and comparing these with the reference intrinsics.

Surface Normals: We rely on Omnidata-v2 [28] inferred normals as a reference, comparing both the L1 and angular errors of normal predictions made by calibration methods (from [17, 68]) and StyleGAN. As shown in Table 1, the surface normals generated by StyleGAN are somewhat less accurate quantitatively in comparison. A visual comparison is provided in Figure 3 and Figure 7.

Depth: We utilize ZoeDepth [10] inferred depth as a reference, comparing the L1 error of depth predictions made by calibration methods (from [28, 17, 68]) and StyleGAN. Interestingly, depth predictions made by StyleGAN surpass these methods in performance, as shown in Table 1. A visual comparison is available in Figure 4 and Figure 8.

Albedo and Shading: We use a recent SOTA, self-supervised image decomposition model [20] on the IIW dataset [9] to guide our search for albedo and shading latents. This involves conducting independent searches for latents that align with albedo and shading, guided by the regression model.

Table 1: Quantitative Comparison of Normal and Depth Estimation. We use Omnidata-v2 [28] and ZoeDepth [10] as pseudo ground truth when comparing for surface normals and depth respectively. StyleGAN model performs slightly worse on normals and slightly better on depth despite never having been exposed to any normal or depth data, operating without any supervision, and achieving this task in a zero-shot manner. It's important to note that our use of pre-trained normals or depth only serves to guide StyleGAN to find these latent codes that correspond to them. There is no learning of network weights involved, hence the performance is effectively zero-shot.

Image

StyleGAN

Omnidata-v2 [28]

Omnidata-v1 [17]

X-task Const [68]

X-task base [68]

/

Image

StyleGAN

ZoeDepth [10]

Omnidata-v2 [28]

Omnidata-v1 [17]

X-task const [68]

X-task base [68]

Image

Albedo-G

Albedo-R

Shading-G

Shading-R

Residual-G

Residual-R

口

Image

StyleGAN

EVA-2

DPT

StyleGAN

EVA-2

DPT

As shown in Figure 5, StyleGAN's generated albedo and shading results display significantly smaller residuals when contrasted with those from the SOTA regression-based image decomposition model.

Segmentation: We employ a supervised SOTA model, EVA-2 [18] and the top-performing segmentation method on the ADE20k benchmark [73] to guide our search for segmentation latents.

StyleGAN

Omnidata-v2

StyleGAN

Omnidata-v2

StyleGAN

Omnidata-v2

StyleGAN

Omnidata-v2

sh

Given StyleGAN's restriction to synthesizing only 3-channel output, we focus our search for directions that segment individual objects. Moreover, we use a weighted regression loss to address sparsity in features like lamps and pillows, which occupy only a minor portion of the image.

Aquantitative comparison of our StyleGANgenerated segmentation for five different object categories is shown in Table 2, with qualitative comparisons in Figure 6. The accuracy of StyleGAN-generated segmentation is quantitatively comparable to a large vision-transformer-based baseline DPT [44]. Furthermore, qualitative results show that the segmentation of lamps and pillows generated by StyleGAN is more complete and slightly better to those from SOTA methods.

Control: StyleGAN cannot do nonintrinsic tasks. While our search is over a relatively small parameter space, it is conceivable that StyleGAN is a generic image processing engine. We check that there are tasks StyleGAN will not do by searching for an offset that swaps the left and right halves of the image (Figure 9).

Image

Target

Predicted

Relighting

StyleGAN

Omnidata-v2 [28]

Omnidata-v1 [17]

X-task Const [68]

X-task base [68]

6 Robustness of StyleGAN Intrinsics

We investigate the sensitivity of predicted normals, depth, segmentation, and albedo to alterations in lighting. For this purpose, we utilize StyLitGAN [11], which generates lighting variations of the same scene by introducing latent perturbations to the style codes. This allows us to add d ( relighting ) to w + to create different lighting conditions of the same scene, and then add d ( c ) to predict the corresponding scene property maps. Ideally, the predicted normals, depth, segmentation, and albedo should remain invariant to lighting changes.

However, we observe that the current state-of-the-art regression methods exhibit sensitivity to such lighting changes. Interestingly, the intrinsic predictions generated by StyleGAN demonstrate robustness against these lighting variations, significantly surpassing their state-of-the-art counterparts in terms of performance. Note that comparison methods use much larger models and extensive data augmentations. A qualitative comparison is in Figure 10 and a quantitative analysis for variation in surface normals is in Figure 11. A similar trend is observed for other intrinsics.

7 Discussion

Our study, while exploratory, has broad implications. We discuss a few of them below.

Understanding Generative Models. We highlight important properties and capabilities in generative models when not trained with explicit supervision. This insight could potentially lead to the development of more efficient and effective training methods for generative models.

StyleGAN

Omnidata v2 Omnidata v1

XTC

XTC baseline

0.970

0.975

0.980

0.985

0.990

0.995

1.000

Cosine similarity to mean normal

StyleGAN

Omnidata-v2

Figure 12: This figure shows the failure in generation of surface normal directions using StyleGAN-XL on ImageNet with 10,000 unique generated images. The images in the middle row resemble the surface normals in the bottom row but miss details that [28] predicted. The differences in architecture between StyleGAN-XL and the more appropriate StyleGAN-2 make it difficult to leverage this model. Either an alternate strategy is required or the latent space in this model is not easy to manipulate.

Intrinsic Images from other Generative Models. We have demonstrated that StyleGAN can easily extract representations of important intrinsic images. This makes our work the first to achieve this, and that too by simply adding an offset to a pretrained generative model. Even if the current method has a performance gap compared to SOTA monocular predictors, this understanding that generative models can capture such intrinsic properties provides a new avenue for research. Our findings raise compelling questions for further investigation. We suspect that this property may also be true for other generative models. Is this the case, or is it a result of specific design choices in StyleGAN? We think other generative models 'know' intrinsic images, too, but don't currently know how to elicit that knowledge.

Robust Intrinsic Images. One of the potential advantages of our approach is deriving intrinsic images that are inherently more robust, especially against variations like lighting changes (Figure 10). One might use this to bootstrap a better StyleGAN predictor from a non-robust, but SOTA, predictor.

Guidance from SOTA Monocular Predictor. Currently, we rely on monocular predictors for offset guidance. However, we use limited guidance, consisting of just 2000 examples. We anticipate that future models may not require such guidance.

GAN Inversion. To build a practical intrinsic image predictor from a StyleGAN, one needs an inverter that is (a) accurate and (b) preserves the parametrization of the image space by latents - can this be built?

What Makes Good Image Representation? Finally, our findings provide insights into what makes a 'good' image representation. Our discovery of fundamental image intrinsics such as depth, albedo, and normals in the latent space of a StyleGAN suggests that meaningful image representations are those that can capture these intrinsic properties, as highlighted in Barrow and Tenenbaum's work [6]. StyleGAN clearly 'knows' the intrinsics that are well-understood in the vision and graphics community. Are there other intrinsics that it 'knows' but that we have overlooked? The materials literature, particularly studies like Motoyoshi et al. [38] and Sharan et al. [51], suggest that many other types of intrinsic images may exist. Therefore, our search method could potentially uncover unknown intrinsic images.

Acknowledgment

This material is based upon work supported by the National Science Foundation under Grant No. 2106825 and by gifts from Amazon and Boeing.

References

Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using stylegan. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13970-13979, 2021. 3

Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (ToG) , 40(3):1-21, 2021. 2

Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In CVPR , pages 18511-18521, 2022. 2

Zhipeng Bao, Martial Hebert, and Yu-Xiong Wang. Generative modeling for multi-task visual learning. In International Conference on Machine Learning , pages 1537-1554. PMLR, 2022. 3

Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems , 35:2500525017, 2022. 2

H Barrow and J Tenenbaum. Recovering intrinsic scene characteristics. Comput. vis. syst , 2(3-26):2, 1978. 2, 3, 10

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences , 117(48):30071-30078, 2020. 2

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1811.10597 , 2018. 2

Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions on Graphics , 2014. 5

Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M¨ uller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 , 2023. 1, 2, 3, 4, 5, 6, 17

Anand Bhattad and D.A. Forsyth. Stylitgan: Prompting stylegan to generate new illumination conditions. In arXiv , 2023. 2, 3, 4, 9, 10

Anand Bhattad, Viraj Shah, Derek Hoiem, and DA Forsyth. Make it so: Steering stylegan for any image inversion and editing. arXiv preprint arXiv:2304.14403 , 2023. 2

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022. 2

Min Jin Chong and David Forsyth. Jojogan: One shot face stylization. arXiv preprint arXiv:2112.11641 , 2021. 2

Min Jin Chong, Hsin-Ying Lee, and David Forsyth. Stylegan of all trades: Image manipulation with only pretrained stylegan. arXiv preprint arXiv:2111.01619 , 2021. 2

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021. 2

Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021. 3, 5, 6, 9, 16, 17

Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331 , 2023. 1, 2, 4, 6, 7, 19, 20, 21, 22

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023. 3

David Forsyth and Jason J Rock. Intrinsic image decomposition using paradigms. IEEE transactions on pattern analysis and machine intelligence , 44(11):7624-7637, 2021. 1, 2, 3, 4, 5, 7

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint

arXiv:1406.2661 , 2014. 2

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision , 2017. 3

Thibaut Issenhuth, Ugo Tanielian, J´ er´ emie Mary, and David Picard. Edibert, a generative model for image editing. arXiv preprint arXiv:2111.15264 , 2021. 2

Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. arXiv preprint arXiv:2106.05258 , 2021. 3

Michael Janner, Jiajun Wu, Tejas D Kulkarni, Ilker Yildirim, and Josh Tenenbaum. Selfsupervised intrinsic image decomposition. In Advances in Neural Information Processing Systems , 2017. 3

Omer Kafri, Or Patashnik, Yuval Alaluf, and Daniel Cohen-Or. Stylefusion: A generative model for disentangling spatial segments. arXiv preprint arXiv:2107.07437 , 2021. 2

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. arXiv preprint arXiv:2303.05511 , 2023. 2

O˘ guzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18963-18974, 2022. 1, 2, 3, 4, 5, 6, 9, 10, 16, 17

Tero Karras, Miika Aittala, Samuli Laine, Erik H¨ ark¨ onen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems , 34, 2021. 2, 3

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019. 2, 3

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020. 2

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Conference on Computer Vision and Pattern Recognition 2023 , 2023. 2

Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning , 12(4):307-392, 2019. 2

Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8300-8311, 2021. 2, 3

Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing Systems , 34:16331-16345, 2021. 2

Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. Unsupervised learning for intrinsic image decomposition from a single image. 2020. 3

Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. Object 3dit: Language-guided 3d-aware image editing. In Advances in Neural Information Processing Systems (NeurIPS) , 2023. 2

Isamu Motoyoshi, Shin'ya Nishida, Lavanya Sharan, and Edward H Adelson. Image statistics and the perception of surface qualities. Nature , 447(7141):206-209, 2007. 10

Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023. 2

Atsuhiro Noguchi and Tatsuya Harada. Rgbd-gan: Unsupervised 3d representation learning from natural image datasets via rgbd image synthesis. In International Conference on Learning Representations , 2020. 3

Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. arXiv preprint arXiv:2011.00844 , 2020. 3

Xingang Pan, Ayush Tewari, Lingjie Liu, and Christian Theobalt. Gan2x: Non-lambertian inverse rendering of image gans. arXiv preprint arXiv:2206.09244 , 2022. 3

Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai. A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS) , 2021. 3

Ren´ e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021. 3, 6, 7, 8, 19, 20, 21, 22

Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems , 32, 2019. 2

Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2287-2296, 2021. 2

Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.05744 , 2021. 2

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022. 2

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016. 2

Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In CVPR 2023-IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023. 3

Lavanya Sharan, Ruth Rosenholtz, and Edward Adelson. Material perception: What can you see in a brief glance? Journal of Vision , 9(8):784-784, 2009. 10

Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence , 2020. 2

Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1532-1540, 2021. 2, 4

Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, and Qifeng Chen. 3d-aware indoor scene synthesis with depth priors. 2022. 3

Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gerard Medioni. Gan-control: Explicitly controllable gans. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14083-14093, 2021. 2

Feitong Tan, Sean Fanello, Abhimitra Meka, Sergio Orts-Escolano, Danhang Tang, Rohit Pandey, Jonathan Taylor, Ping Tan, and Yinda Zhang. Volux-gan: A generative model for 3d face synthesis with hdri relighting. arXiv preprint arXiv:2201.04873 , 2022. 3

Christos Tzelepis, Georgios Tzimiropoulos, and Ioannis Patras. Warpedganspace: Finding nonlinear rbf paths in gan latent space. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6393-6402, 2021. 2, 4

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017. 2

Vaibhav Vavilala, Seemandhar Jain, Rahul Vasanth, Anand Bhattad, and David Forsyth. Blocks2world: Controlling realistic scenes with editable primitives. arXiv preprint arXiv:2307.03847 , 2023. 2

Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International conference on machine learning , pages 9786-9796. PMLR, 2020. 2

Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity GAN inversion for image attribute editing. In CVPR , 2022. 2

Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12863-12872, 2021. 2, 3

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. arXiv preprint arXiv:2303.04803 , 2023. 3

Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative representations for scene synthesis. International Journal of Computer Vision , 2020. 2

Huiting Yang, Liangyu Chai, Qiang Wen, Shuang Zhao, Zixun Sun, and Shengfeng He. Discovering interpretable latent space directions of gans beyond binary attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12177-12185, 2021. 2, 4

Ning Yu, Guilin Liu, Aysegul Dundar, Andrew Tao, Bryan Catanzaro, Larry S Davis, and Mario Fritz. Dual contrastive loss and attention for gans. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6731-6742, 2021. 5

Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2019. 3

Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020. 5, 6, 9, 16, 17

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 2

Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv preprint arXiv:2010.09125 , 2020. 3

Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10145-10155, 2021. 2, 3

Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.02153 , 2023. 3

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv:1608.05442 , 2016. 7

Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference on Computer Vision (ECCV) , 2020. 2

Peiye Zhuang, Oluwasanmi Koyejo, and Alexander G Schwing. Enjoy your editing: Controllable gans for image editing via latent space navigation. arXiv preprint arXiv:2102.01187 , 2021. 2, 4

Appendix

In Figure 13, we demonstrate the segmentation quality with increase in number of labeled examples. Wealso provide additional qualitative figures for intrinsic image predictions from StyleGAN - normals in Figure 14, depth in Figure 15, albedo-shading decomposition in Figure 16, and segmentation of lamps and pillows in Figure 17, segmentation of windows and paintings in Figure 18 and segmentation of beds in Figure 19. In our experiments, we generated three-channel output for each depth, shading, and segmentation from StyleGAN. We took the mean for each of them to get the final singlechannel estimate. We also provide additional examples of robustness against lighting changes for the segmentation task in Figure 20.

250

500

750

1000

1250

1500

1750

2000

Iterations

65

70

75

80

85

90

95

IOU

IOU vs Iterations

StyleGAN-pillows

DPT-pillows

StyleGAN-windows

DPT-windows

Image

StyleGAN

v2 [28]

v1 [17]

X-task

Const [68]

X-task

base [68]

Image

StyleGAN

ZoeDepth [10]

v2 [28]

v1 [17]

X-task

const [68]

X-task

base [68]

Image

Albedo-G

Albedo-R

Shading-G

Shading-R

Residual-G

Residual-R

Image

StyleGAN

EVA-02 [18]

DPT [44]

StyleGAN

EVA-02 [18]

DPT [44]

Image

StyleGAN

EVA-02 [18]

DPT [44]

StyleGAN

EVA-02 [18]

DPT [44]

11

Image

StyleGAN

EVA-02 [18]

DPT [44]

Image

StyleGAN

EVA-02 [18]

DPT [44]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for obtaining \"intrinsic images\" for an RGB image, where in this context intrinsic images mean mainly depth, normal, segmentation, albedo, and shading maps. To this end, the authors investigate how a pre-trained StyleGAN generator model can be used to output the respective intrinsic image instead of an RGB image by only adding a constant in style weight space. More specifically, the proposed method optimizes an offset vector in the StyleGAN \"w+\" style space for a given editing constraint. The offset vectors are optimised via gradient descent where a comparably small dataset of generated images together with respective intrinsic images, predicted by SOTA models, is used as guidance information. The authors compare the proposed method to SOTA methods and find that it performs comparably.\n\nSTRENGTHS:\n- The proposed approach is an interesting and valuable method for investigating what information is already contained in generative models such as StyleGAN. The finding that adding an offset vector in style weight space to output one of the other data modalities instead of RGB is surprising and very interesting. It highlights how much \"semantic and 3D scene understanding\" can be found in 2D generative models.\n- The authors investigate the performance of the same proposed method with multiple different data modalities highlighting the generality of the approach.\n- I appreciate that the authors also show an example decomposition that clearly does not work with the given method (L. 178). As only a style weight space vector is added, non-pixel-based transformations are expected to not work well / fail.\n- The authors compare against relevant baselines quantitatively and qualitatively and show helpful visual comparisons.\n\nWEAKNESSES:\n- Performance across different datasets / Dependence on small labeled dataset: The proposed method requires a sub-set of \"labeled\" generated image and intrinsic image pairs, where the latter is provided by a SOTA method. Have the authors investigate how well the method performs if this \"fine-tuning\" is done on a different dataset, or on only a sub-set of the covered \"modes\"? More general, investigating different factors of this training stage with the resulting end performance, e.g. in form of a graph showing number of training images against performances, would be very interesting.\n- Labeled dataset: In similar spirit, how many labeled images (provided by the SOTA method) are used? (L. 124 - 126 is vague on this, and it would be beneficial to state clear numbers.)\n- Evaluation: The authors argue that no ground truth is available, hence SOTA predictions are used as GT. But couldn’t the test set of these SOTA methods be used as GT? For the proposed method, the input image would be needed to be inverted in StyleGAN space, but this has been extensively done before in the literature. Then, the proposed method could directly be compared to the SOTA prediction model itself.\n- Depth Comparison: It is unclear to me why the proposed method performs better in the L1 depth metric (Table 1) while it looks qualitatively worse than the other methods. Could the authors expand on this? I am not sure if this is caused by some scale/shift misalignments; as these are monocular predictors, the scale and shift is arbitrary, so this needs to be taken into account during evaluation.\n- Caption Fig 6: “Note that our quantitative comparison in Table 2 to a SOTA segmenter [17] likely understates how well StyleGAN can segment;” -> Can the authors expand on this? This is not clear to me. The authors give the two-lamp example, however, here all methods were able to segment the two lamps.\n\nQUESTIONS:\n- How sensitive is the proposed system to the number of image \"labeled\" image pairs used for training?\n- Could the GT image and data modality pairs of the test set of the SOTA predictors be used as GT for evaluation?\n- Why is the proposed system better in the L1 depth metric while it looks worse qualitatively?\n- Is the same offset style vector d(c) used across all images for the same intrinsic image modality?\n- How many images of the \"labeled\" datasets are used for optimising the offset style vectors?\n\nFor more context on these questions, please see the \"Weakness\" paragraph.\n\nThe manuscript contains the following typos / other formatting errors:\n\nTypos:\n\n\tCaption Fig 1: “StyleGAN has easily accessed and accurate representations of intrinsic images” is unclear and grammatically wrong. What does this mean?\n\tCaption Fig 1: ““knows” fine detail in normal” -> “knows” fine detail in normal maps \n\tL. 49: “Showing that intrinsic images so extracted” -> “Showing that these intrinsic images extracted from StyleGAN …\n\tL. 54: “and produce blurry outputs” -> and producing blurry outputs\n\tL. 91: “significant recent methods” -> Important recent methods\n\tL. 92: “Learned methods” -> Learning-based methods\n\tCaption Fig 2: “We demonstrate a StyleGAN that is trained to generate images encode accessible scene property maps” -> […] is trained to generate images encodes accessible scene property maps\n\tCaption Fig 2: “Our approach explores [….], generate” -> Our approach explores […], generates\n\tL. 123: “a select number” -> a selected number\n\tL. 198: “We have demonstrated that StyleGAN has easily extracted representations” -> We have demonstrated […] has easily extractable representations\n\tL. 200: “be true of other generative models” -> be true for other generative models\n\nOther:\n\n\tFigure 1 and Figure 2 are not referenced in the text. Please add a reference to the main text.\n\nLIMITATIONS:\n- The authors show and discuss and example data modality / transformation that is cannot be tackled with the proposed approach due to the pixel-aligned features (L. 178).\n- The authors investigate the performance of the proposed system for different lighting conditions of the images (L. 184).\n- In the discussion (Sec. 7), the authors discuss unclear points that would require additional investigation. This includes to what extent these findings translate to other generative models and if more intrinsic image modalities next to the studies ones are contained in StyleGAN's weight space.\n- The authors do not discuss potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed the method of extracting intrinsic images (Normal, Depth, Albedo, Shading, and Segment) from a pre-trained StyleGAN model without any pre-training or finetuning. It is empirically shown to be better than the SOTA for getting images intrinsic and robust to relighting effects, unlike SOTA.\n\nSTRENGTHS:\nShowing that the StyleGAN W-space also learned the internal representation of intrinsic scene properties without being explicitly trained for the same.\n\nProposed a simple, effective, and generalizable method for extracting intrinsic information without explicit training StyleGAN. Optimization based method is used to search the desired latent code for intrinsic image generation.\n\nExtracted intrinsic images as robust to lighting changes as compared to SOTA for extracting intrinsic images.\n\nWEAKNESSES:\nThis work can be considered as StyleGAN W-space exploration for intrinsic image generation.\n\nAs mentioned in the paper, pre-trained StyleGAN is used, but all the quantitative and qualitative results are only shown for the LSUN Bedroom dataset. Does it applicable to other datasets, also such as the LSUN Church dataset, Animal Faces, FFHQ face, etc.?\n\nAs per the method proposed in the paper, pre-trained intrinsic SOTA models are used for the optimization, so exploring W-spaces for these datasets directly depends on how well SOTA generalizes for other datasets.\n\nFor real images, if we can use the SOTA models for intrinsic extraction directly, why is there a need for GAN inversion, as discussed in the paper? One of the purposes of GAN inversion is to give control over image editing of real images, and as per the problem statement in the paper, there is no such task of editing images based on intrinsic parameters.\n\nThe proposed work uses the SOTA model for optimization, then it is possible for the SOTA model to make incorrect predictions for some images due to domain shifts in train-test data, and it can carry forward to latent space optimization as well as in quantitative and qualitative evaluation.\n\nMinor mistakes:\nOn line 117, it should be w+’ not w’.\n\nQUESTIONS:\nRefer above\n\nLIMITATIONS:\nThe proposed method does not work on real images. The authors pointed out that this is due to GAN inversion, which does not preserve the parametrization of the W-space. With improvement in the GAN inversion method, the proposed approach can work on real images also.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper uses StyleGAN latent space properties to extract properties like normals, depth, segmentation, albedo and shading. The method does not train a new model to achieve this. The method relies on the properties of w+ space to predict these properties. In practice, the method uses the off-the-shelf predictors to use as pseudo labels and the signal is back-propagated to the w+ code. The authors evaluate their method on the LSUN bedroom dataset. The generated images and their corresponding predictions show that the StyleGAN is able to generate consistent results on bedrooms which suggests that the style codes can be modified to generate results in other domains not seen by the network.\n\nSTRENGTHS:\n1)  The paper explores a method to generate multiple properties of an image just using the w+ space of the StyleGAN. The paper shows that these properties like normals, depth, segmentation, albedo and shading can be produced by the latent space of the StyleGAN which is primally used for style transfer tasks. The results in the paper shows that the method produces consistent results on bedrooms.\n\n2) The method works good on StyleGAN generated bedroom images. The paper shows multiple figures demonstrating consistency of the method and the qualitative evaluation.\n\n3) The paper evaluates the method using the metrics of the corresponding tasks. The results show that the scores are not random and the method produces meaningful predictions of the properties.\n\nWEAKNESSES:\n1) Firstly I feel that the abstract of the paper can be improved. To me it felt more like an introduction than an abstract. I would ask the authors to revisit the abstract and write it in the conventional way. What problem are you solving ? Why is it important/ what is the motivation? What is your solution? What results you show in the paper. In the current form it is not informative to me.\n\n2) While the paper has proposed to solve some discriminative tasks using StyleGAN, there are other papers like DatasetGAN[1], Lables4Free [2] etc for segmentation and others related like [3] that already show that the features of the StyleGAN can be modified to achieve the goal or a dataset derived from StyleGAN can be used for the training. It is not clear if the method is generalizable especially when the previous methods have shown to exhibit these properties. How general is the method compared to these methods for example in segmentation domain? I would suggest adding a comparison.\n\n3) I am not sure about the evaluations done in the paper. While pseudo labels for evaluation is a way to evaluate the results, another (more significant) way would be to train a supervised network (see Labels4free) based on the predictions of the StyleGAN. Then it can be tested on the real images rather than the generated ones. What about the inversion of a real photograph? Is the method generalizable to the real domain? The current evaluations do not show that. \n\n4) The authors only test on LSUN bedrooms dataset. It is difficult to access the contribution of the paper if its evaluated on only one dataset especially when other pre-trained StyleGANs are available. Why did the authors only evaluation on this particular dataset?\n\n[1] Abdal, Rameen, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. \"Labels4free: Unsupervised segmentation using stylegan.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13970-13979. 2021.\n\n[2] Zhang, Yuxuan, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. \"Datasetgan: Efficient labeled data factory with minimal human effort.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10145-10155. 2021.\n\n[3] IMAGE GANS MEET DIFFERENTIABLE RENDERING FOR INVERSE GRAPHICS AND INTERPRETABLE 3D NEURAL RENDERING, ICLR 2021\n\nQUESTIONS:\nPlease look at the questions in the Weakness and the Limitation sections.\n\nLIMITATIONS:\nThe authors have not discussed the limitations of the work clearly. When does the method fail? Does the method work without the truncation of the latent space where the samples can be diverse but not high quality? How do the off-the-shelf predictors behave in this scenario? What about other datasets for example ImageNet, (StyleGAN-XL)?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper finds that for a GAN generator that is solely trained on images, intrinsic information such as depth, albedo, normal, etc. are implicitly learned during GAN’s training, even though such information is not provided as supervision. More surprisingly, such intrinsic information can be easily retrieved by calculating a direction offset of the latent codes via optimization an L1 loss on monocular predictor’s predictions on a set of generated images. This operation is not per-scene, means the offset is generalizable to the entire latent space.\n\nSTRENGTHS:\n- Originality: To my knowledge, the work is an original finding. It is the first work to explicitly show that intrinsic information is implicitly learned during an image-trained StyleGAN. It is also novel in finding that the intrinsic can be retrieved by a very straightforward optimization that results in a constant additive direction in the latent space, without any finetuning of the generator itself.\n\n- Quality: The work is technically somewhat sound. The finding presented is appropriate and steps logically. Claims are supported via qualitative results, and quantitative experiments compared against SOTA monocular predictors. The authors have discussed the limitations of their method honestly and in much detail---I really like the “Discussion” part, which contains several questions which I raised too while reading the paper, and Figure.7, which shows a clear example of the limitation of the intrinsic nature of the presented claim. However, see my 1st comment in weaknesses for a factor that makes me hesitate to accept this paper.\n\n- Clarity: The paper is overall very clearly written. The claims and approaches are clearly presented and are also easily understandable, with many of my main concerns while skimming through the paper addressed in significant places.\n\n- Significance: I think this paper presented a fascinating finding. Researchers have been suspecting that the intrinsic information should be learned implicitly in the generator, but it is definitely surprising to see that the operation is as simple as a constant additive offset to the latent space that scales across scenes, without any further training. I believe the paper is a major result for the community.\n\nWEAKNESSES:\n- I think the main issue with this paper comes from its narrow evaluation criteria. All experiments conducted in the paper are only based on bedroom images which may have a bias to support the claim of this paper. As the authors have pointed out, perhaps StyleGAN ‘knows’ intrinsic images because they are an efficient representation of what needs to be known to synthesize an image. It could be difficult to see the finding scale to other datasets, especially in the wild image datasets where there is no clear principal component.\n\n- In addition, the paper seems to be a solely ‘scientific finding’ paper, I have difficulty thinking about what aspect of this finding will be useful in terms of the application side since there is clearly still a performance gap with (even non-SOTA) predictors, and it needs SOTA monocular predictor to guidance the finding of the direction offset at the first place.\n\nQUESTIONS:\nThe experiments use SOTA monocular predictors as the “pseudo-ground truth”. However, wouldn’t it be a more appropriate way to evaluate some synthetic dataset with ground truth? This way, we can have a better idea of the gap between StyleGAN intrinsic & monocular estimators & ground truths.\n\nLIMITATIONS:\nThe authors have discussed the limitations of their method honestly and in very much detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}