StyleGAN knows Normal, Depth, Albedo, and More
Authors: Anand Bhattad
Daniel McKee
Derek Hoiem
David Forsyth
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative models
StyleGAN
Depth
Normals
Segmentation
Intrinsic Images
Albedo
Shading
TL;DR: StyleGAN has easily accessible and accurate representations of intrinsic images, without ever having seen an intrinsic image.

Abstract: Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo, or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images.  The procedure is straightforward. We show that if StyleGAN produces $G({\bf w})$ from latent ${\bf w}$, then for each type of intrinsic image, there is a fixed offset ${\bf d}_c$ so that $G({\bf w}+{\bf d}_c)$ is that type of intrinsic image for $G({\bf w})$. Here ${\bf d}_c$ is {\em independent of ${\bf w}$}.  The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime.  We show that there are image transformations StyleGAN will {\em not} produce in this fashion, so StyleGAN is not a generic image regression engine.  

It is conceptually exciting that an image generator should ``know'' and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN's intrinsic images are robust to relighting effects, unlike SOTA methods.

StyleGAN knows Normal, Depth, Albedo, and More
Anand Bhattad
Daniel McKee University of Illinois Urbana Champaign
Derek Hoiem
D.A. Forsyth
httpp:://anandbhattal github. wtt.tyylgsannnoooooo knows/
(a) Image
(b) Normal
(c) Depth
(d) Albedo
(c) Shading
(f) Segment
Figure I: StyleGAN has easily accessible and accurate representations of intrinsic images, without ever having seen intrinsic image. Simply by finding an appropriate offset the latent variables for each type, we make StyleGAN reveal intrinsic images of many types for synthesized image (a), including: surface normal (b), depth maps (c), albedo (d), shading (e), segmentation (f). No new weight learning or fine-tuning is required. Top row: shows StyleGAN intrinsics; bottom row those predicted by SOTA predictors [28, 10, 20, 18]. Note that StyleGAN "knows" where bedside and other lamps are better than a SOTA segmenter [18] does (it should; it put them there!) and that StyleGAN "knows" fine detail normal maps (around bedside lamp) that hard current methods to predict.
Abstract
Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo shading. This paper demonstrates that StyleGAN easily be induced to produce intrinsic images. Our procedure straightforward. We show that, if StyleGAN produces G(w) from latent w, then for each type of intrinsic image, there is fixed offset d, so that G(w + de) is that type of intrinsic image for G(w). Here de independent of w. The StyleGAN pretrained by others, so this property is not some accident of our training regime. We show that there are image transformations StyleGAN will not produce in this fashion, so StyleGAN is not generic image regression engine.
used
It is conceptually exciting that an image generator should "know" and represent intrinsic images. There may also be practical advantages to using generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN's intrinsic images are robust relighting effects, unlike SOTA methods.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Barrow and Tenenbaum, in an immensely influential paper of 1978, defined the term "intrinsic image" as "characteristics such as range, orientation, reflectance and incident illumination - of the surface element visible at each point of the image" [6]. Maps of such properties as (at least) depth, normal, albedo, and shading form different types of intrinsic images. The importance of the idea is recognized in computer vision where one attempts to recover intrinsics from images - and in computer graphics where these and other properties are used to generate images using models rooted in physics. But are these representations in some sense natural? In this paper, we show that a marquee generative model- StyleGAN has easily accessible internal representations of many types of intrinsic images (Figure 1), without ever having seen intrinsics in training. suggesting that they are.
We choose StyleGAN [30, 31, 29] as a representative generative model because it is known for synthesizing visually pleasing images and there is a well-established literature on the control of StyleGAN [65, 11, 75, 14, 53, 57]. Our procedure echoes this literature. We search for offsets to the latent variables used by StyleGAN, such that those offsets produce the desired type of intrinsic image (Section 4; Figure 2). We use pre-trained StyleGAN which has never seen an intrinsic image in training, and a control experiment confirms that StyleGAN is not a generic image regressor. All this suggests that the internal representations are not "accidental" likely, StyleGAN can produce intrinsic images because (a) their spatial statistics are strongly linked to those of image pixels and (b) they are useful in rendering images.
There may be practical consequences. As Section shows, the intrinsic images recovered compare very well to those produced by robust image regression methods [28, 10, 18, 20], both qualitatively and quantitatively. But StyleGAN produces intrinsic images that are robust to changes in lighting conditions, whereas current SOTA methods are not. Further, our method does not need to be shown many examples (image, intrinsic image) pairs. These practical consequences rest on being able to produce intrinsic images for real (rather than generated) images, which we cannot currently do. Current SOTA GAN inversion methods (eg [3, 47, 12, 61]) do not preserve the parametrization the latent space, so directions produced by search do not reliably produce the correct intrinsic for GAN inverted images. As GAN inversion methods become more accurate, we expect that generative models can be turned into generic intrinsic image methods. Our contributions are:
Demonstrating that StyleGAN has accessible internal representations of intrinsic scene properties such as normals, depth, albedo, shading, and segmentation without having seen them in training.
Describing a simple, effective, and generalizable method, that requires no additional learning or fine-tuning, for extracting these representations using StyleGAN's latent codes.
Showing that intrinsic images extracted from StyleGAN compare well with those produced by SOTA methods, and are robust to lighting changes, unlike SOTA methods.
2 Related Work
Generative Models: Various generative models, such as Variational Autoencoders (33],)[33), Gen- erative Adversarial Networks (GANs)[21], Autoregressive models [58], and Diffusion Models [16], have been developed. These models have made significant improvements in training and output quality through novel loss functions and stability enhancements despite initial challenges with blurry outputs [49, 45, 30, 48, 27]. In this work, we focus on StyleGAN [30, 31, 29] due to its exceptional ability to manipulate disentangled style representations with ease. We anticipate that analogous discoveries will be made for other generative models in the future.
Editing in Generative Models: A variety of editing techniques allow for targeted modifications to generative models' output. One prominent example is StyleGAN editing [53, 60, 62, 52, 74, 15, 46, 14, 11, 12, 2, 55], which allows for precise alterations to the synthesized images. Similarly, a handful of editing methods have emerged for autoregressive and diffusion models [23, 5, 13, 69, 39, 32, 37, 59].
In the context of StyleGAN editing, there exist several approaches such as additive perturbations to latents [65, 11, 75, 53,57J, affine transformation on latents [62, 26], layer-wise editing [64], activation- based editing [8, 7, 15], and joint modeling of images and laheled attributes [52, 71, 34, 35]. These
methods facilitate nuanced and specific changes to the images.

---

In our study, we adopt straightforward and simplest additive perturbations to latents, rather than learning new transformations or engineering-speee specific layer modifications. By searching for small latent code perturbations to be applied across all layers, we allow the model to learn and modulate how these new latent representations influence the generated output. This process minimizes the need for intricate layer-wise manipulation while still achieving desired edits and providing valuable insights into the model's internal latent structure.
An emerging trend involves leveraging generative models for discriminative tasks. Examples of such work include Labels4Free [1], GenRep [24], Dataset GAN [71], SemanticGAN [34] RGBD-GAN [40], DepthGAN [54], MGM [4], ODISE [63], ImageNet-SD [50], and VPD [72]. These approaches either use generated images to improve downstream discriminative models or fine-tune the original generative model for a new task or learn new layers or learn new decoders to produce desired scene property outputs for various tasks.
Discriminative Tasks with Generative Models:
In contrast to these methods, our approach eschews fine-tuning, learning new layers, or learning additional decoders. Instead, we directly explore the latent space within a pretrained generative model to identify latent codes capable of predicting desired scene property maps. This process not only simplifies the task of utilizing generative models for discriminative tasks but also reveals their inherent ability to produce informative outputs without extensive modification or additional training.
Intrinsic images were introduced by Barrow and Tenenbaum [6]. Intrinsic image prediction is often assumed to mean albedo prediction, but the original concept "characteristics of the surface element visible at each point of the image" ([6), abstract) explicitly included depth, normals, and shading; it extends quite naturally to semantic segmentation maps too. Albedo and shading (where supervised data is hard to find) tend to be studied apart from depth, normals, and semantic segmentation (where supervised data is quite easily acquired). Albedo and shading estimation methods have long history. As the recent review in [20] shows, methods involving little or no learning have remained competitive until relatively recently; significant recent methods based on learning include [25, 67, 36, 20]. Learned methods are dominant for depth estimation, normal estimation, and semantic segmentation. Competitive recent methods [17, 28, 44, 10, 19] require substantial labeled training data and numerous augmentations.
Intrinsic Images:
What is known about what StyleGAN knows: Various papers have investigated what StyleGAN knows, starting with good evidence that StyleGAN "knows" 3D information about faces [41, 70], enough to support editing [43, 42, 56]. Searching offsets (as we do) yields directions that relight synthesized images [11]. In contrast, show that StyleGAN has easily accessible representations of natural intrinsic images, without ever having seen an intrinsic image of any kind.
3 Background
3.1 StyleGAN
StyleGAN [30, 29] uses two components: a mapping network and a synthesis network. The mapping
network maps a latent vector z to an intermediate space w. The synthesis network takes w, and gen- erates an image x, modulating style using adaptive instance normalization (AdaIN) [22]. Stochastic variation is introduced at each layer by adding noise scaled by learned factors. The architecture of StyleGAN can be summarized by the following equations:
f(z)
x = g(w, n)
where f is the mapping network, is the synthesis network, and n is noise vector.
3.2 Manipulating StyleGAN
StyleGAN's intermediate latent code, denoted as w, dictates the style of the image generated. The w space is a more detailed version of the w space, where a unique w vector is provided to each layer of the synthesis network [62]. This allows for more fine-grained control over the generated image at varying levels of detail.
Editing in StyleGAN can be achieved by manipulating these w vectors in the w space. We do this by identifying new latent code wt' that is close to the original + but also satisfies specific
3

---

Dense
Synthesis Network
Mapping Network
Style Code
Predictor
N(O.1) 1)
Normal
Depth
Segment
Albedo
Shading
Scene Property Maps
from off-the-shell Network
Scene Property Directions (d)
Modified Style Code w/ d
3
Regression Loss
Learned Components
Normal
Depth
Segment
Albedo
Shading
(a) Searching for directions (d.) corresponding to different scene intrinsic map in Latent (w") Space
Image
Normal
Depth
Segment
Albedo
Shading
(b) Final scene property maps generated from StyleGAN after search completion
Figure 2: Searching for scene intrinsic offsets. We demonstrate that StyleGAN trained only generate images also encodes accessible scene property maps. We use simple way extract these scene property maps. Our approach explores latent directions (d) or offsets within a pretrained StyleGAN's space, which, when combined with the model's style codes (w++), generates surface normal predictions, depth predictions, segmentation, albedo predictions, and shading predictions. Importantly, our approach does not require any additional fine-tuning or parameter changes the original StyleGAN model. Note the StyleGAN model was trained to generate natural scene-like images and was never exposed to scene property maps during training. We use off-the-shelf, state- of-the-art dense prediction networks, only to guide this exploration. The discovery of these scene property latents offers valuable insights into how StyleGAN produces semantically consistent images.
editing constraint c [65, 11, 53, This problem can formulated as:
w w d(c),
where dcc) computes perturbation based d(c) can be found using methods like gradient descent. The edited image is then generated from the new latent code w
4 Searching for Intrinsic Offsets
Our approach is illustrated in Figure 2. We directly search for specific perturbations or offsets, denoted as d(c), which when added to the intermediate latent code w i.e., wt' = w + d(c), yield the desired intrinsic scene properties. Different perturbations d(c) are used for generating various intrinsic images such normals, depth, albedo, shading, and segmentation masks. To search for these offsets, we utilize off-the-shelf pretrained networks from Omnidata-v2 [28] for surface normals, Zoe-depth [10] for depth, EVA-2 [18] for semantic segmentation, and Paradigms for intrinsic image decomposition [20] to compute the desired scene properties for the generated image x G(z). We employ Huber-loss (smooth Li-Loss) to measure the difference between generated intrinsic and off-the-shelf network's s predicted intrinsics. Formally, we solve the following optimization problem:
w arg min LHuber (imubbr((()))) (P(x),
d(e)
where P is function that computes the scene property map from image. By solving this problem, we obtain a latent code w' capable of generating an image x' that are scene properties x. We also note that task-specific losses, such as scale-invariant loss in depth classification loss in

---

segmentation, do not significantly contribute to the overall performance. This observation suggests that our approach is robust and generalizable, opening the door for additional upplications other variants of scene property prediction. We found that incorporating task-specific losses, such as angular loss for normal prediction, slightly improves the quality of predicted intrinsic images.
Accuracy of StyleGAN Intrinsics
For the intrinsic of type we search for direction d(c) using randomly selected images generated by StyleGiNN using w and a reference intrinsic image method. Although approximately 200 images can successfully identify the required directions, we used 2000 unique scenes generated images in our experiments. Note that this is not a scene-by-scene search. A direction is applicable to all StyleGAN images once it is found. For each generated image, we obtain target intrinsics using SOTA reference network. We then synthesize intrinsic images from StyleGAN using the formula w * + d(c). These synthesized intrinsic images are subsequently compared to those produced by leading regression methods, using standard evaluation metrics.
Importantly, the StyleGAN model used in our work has never been trained on any intrinsic images. We use pretrained model from Yu et al. [66] and that remains unaltered during the entire latent search process. Off-the-shelff networks are exclusively utilized for latent discovery, not for any part of the StyleGAN training- Overall time to find one intrinsic image direction is less than 2 minutes on an A40 GPU. In total, less than 24 hours of single A40 GPU were required for the final reported experiments, and less than 200 hours a single A40 GPU were required from ideation to final experiments.
We have no ground truth. We evaluate by generating a set of images and their intrinsics using StyleGAN. For these images, we compute intrinsics using the reference SOTA method and treat the results (reference intrinsics) as ground truth. We then compute metries comparing StyleGAN intrinsics with these results; if these metrics are good, the intrinsics produced by StyleGAN compare well with those produced by SOTA methods. some cases, we are able to calibrate. We do so by obtaining the intrinsics of the generated images using other SOTA methods (calibration metrics) and comparing these with the reference intrinsics.
Surface Normals: We rely on Omnidata-v2 [28] inferred normals as reference, comparing both the LI and angular errors of normal predictions made by calibration methods (from [17, 68)) and StyleGAN. As shown in Table 1, the surface normals generated by StyleGAN are somewhat less accurate quantitatively comparison. A visual comparison is provided in Figure and Figure 7.
Depth: We utilize ZoeDepth [10] inferred depth as a reference, comparing the L1 error of depth predictions made by calibration methods (from [28, 17, 68)) and StyleGAN. Interestingly, depth predictions made by StyleGAN surpass these methods in performance, as shown in Table 1. A visual comparison is available in Figure 4 and Figure 8.
We use a recent SOTA, self-supervised image decomposition model [20] on the IIW dataset [9] to guide our search for albedo and shading latents. This involves conducting independent searches for latents that align with albedo and shading, guided by the regression model.
Albedo and Shading:
Table 1: Quantitative Comparison of Normal and Depth Estimation. We use Omnidata-v2 [28] and ZocDepth [10] as pscudo ground truth when comparing for surface normals and depth respectively. StyleGAN model performs slightly worse on normals and slightly better on depth despite never having been exposed to any normal or depth data, operating without any supervision, and achieving this task in zero-shot manner. It's important note that serves to guide StyleGAN to find these latent codes that correspond to them. There no learning of network weights involved, hence the performance is effectively zero-shot.
of pre- pre-trained normals or depth only
Models
#Parameters
Normals
Depth
Normals/ Depth
T.I 4
Angular Error
T.I
Omnidata-v2 [28]
123.1M
0.1237
Omnidata-vl [17]
1M.1M/113.1M IM
0.0501
0.0750
0.1280
X-task consistency [68]
75.5M
0.0502
0.0736
0.1390
X-task baseline [68]
75.5M
0.0511
0.0763
0.1388
StyleGAN
24.7M
0.0834
0.1216
0.1019
5

---

Image
StyleGAN
Omnidata v2 [28]
Ommidata vl [17]
X task Const [68] task base [68]
Figure 3: Normal generation. StyleGAN generated normals are not as sharp as supervised SOTA
methods but produce similar and accurate representations when compared to other methods.
Image
StyleAAN
ZecDapph [10]
Ommidstt-v22 [28]
Ormidatt-vl [17]
X-taok const [68]
X-task base [68]
Figure 4: Depth estimation. While fine details may be clearly estimated as top methods like
ZoeDepth, the overall structure produced by StyleGAN is consistent in quality with recent models.
Table Quantitative Comparison of Segmentation: Accuracy (Ace) and mean intersection over union (mlOU) are reported when compared to EVA-2 [18] as ground truth.
bed
pillow mloU
lamp
window mloU
painting
Mean mloUi
Acc
mloU
mloU
DPT [44]
97.1
98.9
82.5
99.6
99.0
99.6
92.9
98.8
87.7
StyleGAN
95.8
90.4
97.9
76.6
99.3
71.9
98.6
87.1
99.0
84.0
98.1
82.0

---

Image
Albedo-G
Albedo-R
Shading-G
Shoding-R
Rosidual-G
Residual-R
Figure 5: Shaddng Recovery with StyleGAN. We denote results from StyleGAN -G and from a SOTA regression model [20] as RR Absolute value of image residuals (I - A * S) appears in the last two columns. While searches for albedo and shading directions were independent, StyleGAN appears to "know" these two intrinsics should multiply to yield the image.
r
EVA-2 DPT Figure 6: Segmentation Estimation. Generated images with segmentation produced by the following methods: StyleGAN, EVA-2 [18], DPT [44] for pillows on the left and lamps on the right. Note that quantitative comparison in Table 2 SOTA segmenter [18] likely understates how well StyleGAN can segment; for lamps, StyleGAN can find multiple lamps that the segmenter misses, likely because it put them there in the first place.
Image
StyleGAN
EVA-2 DPT
StyleGAN
As shown in Figure 5, StyleGAN's generated albedo and shading results display significantly smaller residuals when contrasted with those from the SOTA regression- bbaed image decomposition model.
Segmentation: We employ supervised SOTA model, EVA-2 [18] and the top-performing segmenta- tion method on the ADE20k benchmark [73] guide our search for segmentation latents.
7

---

Figure 7: Examples of StyleGAN--genrrace normals for StyleGAN model that trained the FFHQ dataset (left) and LSUN church (right). Top rows show randomly generated images, middle rows display StyleGAN-generaan normals with fixed offset w codes of top rows and last rows provide Omnidata-v2 Surface Normals for comparison.
Figure 8: Examples of StyleGAN-generann depth for a StyleGAN model that was trained on FFHQ dataset (left) and LSUN church (right). Top rows show randomly generated images, middle rows display StyleGAN--eneratte depth with fixed offset to w codes of top rows and last rows provide Omnidata-v2 depth predictions for comparison.
Given StyleGAN's restriction synthesiz- ing only 3-channel output, we focus our search for directions that segment individual objects. Moreover, we use a weighted re- gression loss to address sparsity features like lamps and pillows, which occupy only a minor portion of the image.
A quantitative comparison of our StyleGAN- generated segmentation for five different ob- ject categories is shown in Table 2, with qualitative comparisons in Figure The accuracy of -eeeerrrggeee segmenta- tion quantitatively comparable a large sikio-trrasffommr--bbeee baseline DPT [44]. Furthermore, qualitative results show that the segmentation of lamps and pillows gen- crated by StyleGAN is more complete and slightly better to those from SOTA methods.
Image
Target
Control: StyleGAN cannot do non- intrinsic tasks. While our search is over relatively small parameter space, is con- ceivable that StyleGAN is a generic image processing engine. We check that there are tasks StyleGAN will not do by searching for an offset that swaps the left and right halves of the image (Figure 9).
Figure StyleGAN is not generic image processing machine; for ex., it cannot swap left and right halves of an image. This supports the conclusion that StyleGAN trades in "characteristics visible at each point of the image" or intrinsic images.
of the surface element

---

Relighting
StyleGAN
Omnidata-v2 [28]
Omnidata-vl [17] X-task Const [68]
X-tnsk base [68]
Figure 10: Robustness against lighting changes. Normals should be invariant to lighting changes, yet surprisingly, the pop-performing regression methods, even those trained with labeled data, with lighting augmentations and 3D corruptions [28], demonstrate sensitivity to lighting alterations. Intriguingly,
normals prove to be robust against lighting alterations.
Robustness StyleGAN Intrinsics
We investigate the sensitivity of predicted normals, depth, segmentation, and albedo alterations in lighting. For this purpose, we utilize StyLitGAN [11], which generates lighting variations of the same scene by introducing latent perturbations the style codes. This allows us add d(relighting) to w to create different lighting conditions of the same scene, and then add d(c) to predict the corresponding scene property maps. Ideally, the predicted normals, depth, segmentation, and albedo should remain invariant to lighting changes.
However, observe that the current state-of-thh-ar regression methods exhibit sensitivity to such lighting changes, Interestingly, the intrinsic predictions generated by StyleGAN demonstrate robustness against these lighting variations, significantly surpassing their state-of-the-art counterparts in terms of performance. Note that comparison methods use much larger models and extensive data augmentations. A qualitative comparison in Figure 10 and quantitative analysis for variation in surface normals is in Figure 11. A similar trend observed for other intrinsics.
7 Discussion
Our study, while exploratory, has broad implications. We discuss few of them below.
Understanding Generative Models. We highlight important properties and capabilities in gener- ative models when not trained with explicit supervision. This insight could potentially lead to the development of more efficient and effective training methods for generative models.

---

1.000
Figure 11: Quantitative evaluation of normal variations following relighting. Normals calculated under 16 distinct
Cosine similarity mean normal
0.995
lighting conditions from [11] for 214 test scenes for each normal prediction methods. The inner product between normals under each condition and the overall mean is computed. Ideally, this should be I, indicating normals' consis- tency. The boxplots illustrate these val- ues, with regression methods showing an average change of 8 degrees, with outliers up to 14 degrees. High simi- larity scores for StyleGAN indicate its normals' robustness to lighting changes.
0.990
0.985
0.980
0.975
0.970
StyleGAN Omnidata Omnidata
XTC
XTC baseline
This figure shows the failure in generation of surface normal directions us- ing StyleGAN-XL on ImageNet with 10,000 unique generated images. The images in the middle row resemble the surface normals in the bottom row but miss details that [28] predicted. The differences in architecture be- tween MtyleGANXXL and the more appropriate StyleGAN-2 make difficult to leverage this model. Either alternate strategy is required or the latent space in this model not easy to manipulate.
Figure 12:
C2
Intrinsic Images from other Generative Models. We have demonstrated that StyleGAN can easily extract representations of important intrinsic images. This makes our work the first to achieve this, and that by simply adding an offset pretrained generative model. Even the current method has performance gap compared to SOTA monocular predictors, this understanding that generative models can capture such intrinsic properties provides new avenue for research. Our findings raise compelling questions for further investigation. We suspect that this property may also be true for other generative models. Is this the case, or it result of specific design choices StyleGAN? We think other generative models "know" intrinsic images, too, but don't currently know how to elicit that knowledge.
Robust Intrinsic Images. One of the potential advantages of our approach is deriving intrinsic images that are inherently more robust, especially against variations like lighting changes (Figure 10). One might use this to bootstrap a better StyleGAN predictor from non-robust, but SOTA, predictor.
Guidance from SOTA Monocular Predictor. Currently, rely monocular predictors for offset guidance. However, we use limited guidance, consisting of just 2000 examples. We anticipate that future models may not require such guidance.
GAN Inversion. To build practical intrinsie image predictor from StyleGAN, one needs an inverter that is (a) accurate and (b) preserves the parametrization of the image space by latents can this built?
What Makes Good Image Representation? Finally, our findings provide insights into what makes "good" image representation. Our discovery of fundamental image intrinsics such as depth, albedo, and normals in the latent space of StyleGAN suggests that meaningful image representations are those that can capture these intrinsic properties, as highlighted in Barrow and Tenenbaum's work [6]. StyleGAN clearly 'knows' the intrinsics that are well-understood in the vision and graphics community. Are there other intrinsies that 'knows' but that we have overlooked? The materials literature, particularly studies like Motoyoshi et al. [38] and Sharan et al. [51], suggest that many other types of intrinsic images may exist. Therefore, our search method could potentially uncover unknown intrinsic images.
10

---

Acknowledgment
This material is based upon work supported by the National Science Foundation under Grant No. 2106825 and by gifts from Amazon and Boeing.
References
[1] Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using stylegan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13970-13979, 2021. 3
[2] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Styleflow: Attribute conditionn exploration of stylegan-gent generated images using conditional continuous normalizing flows. ACM
Transactions Graphics (ToG), 40(3):1-21, 2021. 2
[3] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In CVPR, pages
2022. 2
[4] Zhipeng Bao, Martial Hebert, and Yu-Xiong Wang. Generative modeling for multi-task visual learning. In International Conference on Machine Learning, pages 1537-1554. PMLR, 2022.
[5] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005-
25017, 2022. 2
[6] H Barrow and J Tenenbaum. Recovering intrinsic scene characteristics. Comput. vis. syst,
2(3-26):2, 1978. 2, 3,
[7] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117(48)33071133000
2020.
[8] David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative
adversarial networks. arXiv preprint 1rXiv:1811.105990 2018. 2
[9] Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions on Graphics, 2014. 5
[10] Shariq Farooq Bhat, Reiner Birkl. Diana Wofk, Peter Wonka, and Matthias Müller, Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288
2023. 2, 3, 4, 5, 6. 17
[11] Anand Bhattad and D.A. Forsyth. Stylitgan: Prompting stylegan to generate new illumination conditions. In arXiv, 2023, 2. 3, 4, 9, 10
[12] Anand Bhattad. Viraj Shah, Derek Hoiem, and DA Forsyth. Make it so: Steering stylegan for any image inversion and editing. arXiv preprint arXiv:2304. 14403, 2023. 2
[13] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2022. 2
[14] Min Jin Chong and David Forsyth. Jojogan: One shot face stylization. arXiv preprint
arXiv:2112.11641, 2021. 2
[15] Min Jin Chong. Hsin- Ying Lee, and David Forsyth. Stylegan of all trades: Image manipulation
with only pretrained stylegan. arXiv preprint arXiv:2111.0066 2021. 2
[16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis.
Advances in Neural Information Processing Systems, 34:8780-8794, 2021. 2
[17) Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF
International Conference Computer Vision, 2021. 3, 5, 6, 9, 16, 17
[18] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. 11331, 2023. 1, 2, 4, 6, 7,
19, 20, 21, 22
[19] Yuxin Fang. Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern
Recognition, 2023.3 3
[20] David Forsyth and Jason J Rock. Intrinsic image decomposition using paradigms. IEEE transactions on pattern analysis and machine intelligence, 44(11):72247637, 2021. 1, 3, 4,
5.
[21] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint
11

---

arXiv:1406.2660 2014.
[22] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
3
[23] Thibaut Issenhuth, Ugo Tanielian, Jérémie Mary, and David Picard. Edibert, a generative model for image editing. arXiv preprint arXiv:2111.1526. 2021. 2
[24]
Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as data source for multiview representation learning. arXiv preprint arXiv:2106. 05258, 2021. 3
[25] Michael Janner, Jiajun Wu, Tejas D Kulkarni, Ilker Yildirim, and Josh Tenenbaum. Self- supervised intrinsic image decomposition. In Advances in Neural Information Processing
Systems. 2017. 3
[26] Omer Kafri, Or Patashnik, Yuval Alaluf, and Daniel Cohen-Or. Stylefusion: A generative model for disentangling spatial segments. arXiv preprint arXiv:2107.077 00730777777 2021. 2
[27] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris. and Taesung Park. Scaling up gans for text-to-image synthesis. arXiv preprint 2333552303.05511 2023. 2
[28] Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18974, 2022. 1, 2, 3, 4, 5, 6, 9, 10, 16, 17
[29] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen. and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information
Processing Systems, 34, 2021. 2,
[30] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 2.
[31] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 2
[32] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Conference
on Computer Vision and Pattern Recognition 2023, 2023. 2
[33] Diederik Kingma, Max Welling, et al. An introduction variational autoencoders. Founda-
tions and Trends@ in Machine Learning, 3074)3307-392, 2019. 2
[34] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmenta- tion with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3300-83114 1, 2021. 2,
[35] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. Editgan: High-precision semantic image editing. Advances in Neural Information Processing
Systems,34:16333 34:163311-16355 2021. 2
[36] Yunfei Liu, Yu Li. Shaodi You, and Feng Lu. Unsupervised learning for intrinsic image decomposition from a single image. 2020. 3
[37] Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. Object 3dit: Language-guided 3d-aware image editing. In Advances Neural Information P Processing Systems (NeurIPS), 2023. 2
[38] Isamu Motoyoshi, Shin'y ya Nishida, Lavanya Sharan, and Edward H Adelson. Image statistics and the perception of surface qualities. Nature, (7(7141)::26-2221
2007. 10
[39] Chong Mou, Xintao Wang, Linggin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion
models. arXiv preprini arXiv:2302. 232.0330,. 2023. 2
[40] Atsuhiro Noguchi and Tatsuya Harada. Rgbd-gan: Unsupervised 3d representation learning from natural image datasets via rgbd image synthesis. In International Conference on Learning
Representations, 2020. 3
[41] Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. arXiv preprint arXiv:201 1020011.0080 1.00844, 2020.
[42] Xingang Pan, Ayush Tewari, Lingjie Liu, and Christian Theobalt. Gan2x: Non-lambertian inverse rendering of image gans. arXiv preprint arXiv:2206.00 09244, 2022. 3
[43] Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai. A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
12

---

[44] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 3, 6,
8. 19, 20, 21, 22
[45] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidellig images
with vq-vae-2. Advances in neural information processing systems, 32, 2019. 2
[46] Elad Richardson, Yuval Alaluf, Or Patashmik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and
Daniel Cohen-Or, Encoding in style: a stylegan encoder for image-to- toeemaae translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2287-2296, 2021. 2
[47] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. arXiv preprint ar arXiv: (00::2006.000 057444444 2021. 2
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 2 [49] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29,
2016. 2
[50] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In CVPR
2023-IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023. 3
[51] Lavanya Sharan, Ruth Rosenholtz, and Edward Adelson. Material perception: What can you
see in a brief glance? Journal of Vision. 98):784-7788 2009. 10
[52] Yujun Shen, Ceyuan Yang, Xiaoou Tang. and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020. 2
[53] Yujun Shen and Bolei Zhou Closed-form factorization of latent semantics in gans. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1532-15444 2021. 2. 4
[54] Zifan Shi, Yujun Shen, Jiapeng Zhu, Bin-Yan Yeung, and Qifeng Chen. 3d-aware indoor scene
synthesis with depth priors. 2022. 3
[55] Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gerard Medioni. Gan-control: Explicitly controllable gans. In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 14083-14093, 2021. 2
[56] Feitong Tan, Sean Fanello, Abhimitra Mcka. Sergio Orts-Escolano, Danhang Tang, Rohit Pandey, Jonathan Taylor, Ping Tan, and Yinda Zhang. Volux-gam A generative model for 3d face synthesis with hdri relighting. arXiv preprint arXiv:2201. .00473, 2022.
[57] Christos Tzelepis, Georgios Tzimiropoulos, and loannis Patras. Warpedganspace: Finding non- linear rbf paths in gan latent space. In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pages 6393-6402, 2021. 2, 4
[58] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[59] Vaibhav Vavilala, Scemandhar Jain, Rahul Vasanth, Anand Bhattad, and David Forsyth. Blocks2world: Controlling realistic scenes with editable primitives. arXiv preprint arXiv:2307.0 00.83777.737...... 2023. 2
[60] Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In International conference on machine learning, pages
PMLR,
2020. 2
[61] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity GAN inversion for image attribute editing. In CVPR, 2022. 2
[62] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863-12872 2021. 2,
[63] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Vocabulcayy Panoptic Segmentation with Text-to-Image Diffusion Models. arXiv preprint XrXiv:2303.04800 2023. 3
[64] Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative
representations for scene synthesis. International Journal of Computer Vision. 2020. 2 [65] Huiting Yang, Liangyu Chai, Qiang Wen, Shuang Zhao, Zixun Sun, and Shengfeng He. Discov- cring interpretable latent space directions of gans beyond binary attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12177-12185
2021. 2, 4
13

---

[66]
Ning Yu. Guilin Liu, Aysegul Dundar, Andrew Tao, Bryan Catanzaro, Larry S Davis, and Manoo Fritz. Dual contrastive loss and attention for gans. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 6731-6742, 2021. 5
[67] Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3 [68] Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangji Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 5, 6, 9, 16, 17 Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[69]
[70]
Yuxuan Zhang. Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv preprint arXiv:2010. 22000.092222 025009925 2020. 3
[71]
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern
Recognition, pages 10145-10155, 2021. 2,
[72] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. arXiv preprint arXiv:2303.cm 3333002133,.... 2023.
[73] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv: 05442, 2016.7
[74] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[75] Peiye Zhuang, Oluwasanmi Koyejo, and Alexander G Schwing. Enjoy your editing: Controllable gans for image editing via latent space navigation. arXiv preprint arXhv: 2021. 2,
14

---

Appendix
In Figure 13, we demonstrate the segmentation quality with increase in number of labeled examples. We also provide additional qualitative figures for intrinsic image predictions from StyleGAN - normals Figure 14, depth Figure 15, -hadinng decomposition Figure 16, and segmentation of lamps and pillows in Figure 17, segmentation of windows and paintings in Figure 18 and segmentation of beds in Figure 19. In our experiments, we generated three-channel output for each depth, shading, and segmentation from StyleGAN. We took the mean for each of them to get the final single- channel estimate. We also provide additional examples of robustness against lighting changes for the segmentation task in Figure 20.
95
Figure 13: Segmentation Quality vs. La- beled Dataset Size: The graph illustrates how segmentation quality for 'pillows' and 'windows" classes improves with increasing labeled images, plateauing around 2,000 images. Note that all im- ages are seen only once. While the po- tential exists for further refinement, the core insight is StyleGAN's inherent ca- pability to generate these segmentations without explicit training.
---
StyleGAN pillows
DPlllolovs
StyleCAt-windww
I-windows
8 80
250
1000
1250
1500
1750 2000
15

---

Image
StyleGAN
Omnidata- v2 [28]
Omnidata- [17]
X-task Const [68]
X-task base [68]
Figure 14: Additional Normal generation.
16

---

Image
StyleGAN
ZneDepth [10]
Omnidata- [28]
Omnidata- vi [17]
X-task: const [68]
X-task base [68]
Figure 15: Additional Depth Estimation Comparison.
17

---

Image
Alhedo-G
Albedo-R
Shading
Shading-R
Residual-G
Residual-R
Figure 16: Additional Results for Albedo-Shading Recovery with StyleGAN.
18

---

Image
StyleGAN
FVA-02 [18]
DPT [44]
StyleGAN
EVA-02 [18]
DPT [P4
Figure 17: Further segmentation of lamps on the left and pillows on the right.
19

---

Image
StyleGAN
EVA-02 [18]
DPT DPT [44]
StyleGAN
EVA-02 [18]
DPT DPT
Figure 18: Window segmentation on the left and painting segmentation on the right.
20

---

Image
StyleGAN
EVA-02 [18]
DPT [PT
Figure 19; Bed segmentation comparison.
21

---

Image
StyleGAN
EVA 02 [18]
DPT [44]
Figure 20: Additional examples for robustness against lighting for segmentation.
22


Reviews:
[{"review_id": "hnz3Ahl7YU", "reviewer": "Reviewer_NPHF", "summary": "The paper finds that for a GAN generator that is solely trained on images, intrinsic information such as depth, albedo, normal, etc. are implicitly learned during GAN’s training, even though such information is not provided as supervision. More surprisingly, such intrinsic information can be easily retrieved by calculating a direction offset of the latent codes via optimization an L1 loss on monocular predictor’s predictions on a set of generated images. This operation is not per-scene, means the offset is generalizable to the entire latent space.", "questions": "The experiments use SOTA monocular predictors as the “pseudo-ground truth”. However, wouldn’t it be a more appropriate way to evaluate some synthetic dataset with ground truth? This way, we can have a better idea of the gap between StyleGAN intrinsic & monocular estimators & ground truths.", "limitations": "The authors have discussed the limitations of their method honestly and in very much detail.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "- Originality: To my knowledge, the work is an original finding. It is the first work to explicitly show that intrinsic information is implicitly learned during an image-trained StyleGAN. It is also novel in finding that the intrinsic can be retrieved by a very straightforward optimization that results in a constant additive direction in the latent space, without any finetuning of the generator itself.\n\n- Quality: The work is technically somewhat sound. The finding presented is appropriate and steps logically. Claims are supported via qualitative results, and quantitative experiments compared against SOTA monocular predictors. The authors have discussed the limitations of their method honestly and in much detail---I really like the “Discussion” part, which contains several questions which I raised too while reading the paper, and Figure.7, which shows a clear example of the limitation of the intrinsic nature of the presented claim. However, see my 1st comment in weaknesses for a factor that makes me hesitate to accept this paper.\n\n- Clarity: The paper is overall very clearly written. The claims and approaches are clearly presented and are also easily understandable, with many of my main concerns while skimming through the paper addressed in significant places.\n\n- Significance: I think this paper presented a fascinating finding. Researchers have been suspecting that the intrinsic information should be learned implicitly in the generator, but it is definitely surprising to see that the operation is as simple as a constant additive offset to the latent space that scales across scenes, without any further training. I believe the paper is a major result for the community.\n", "weaknesses": "- I think the main issue with this paper comes from its narrow evaluation criteria. All experiments conducted in the paper are only based on bedroom images which may have a bias to support the claim of this paper. As the authors have pointed out, perhaps StyleGAN ‘knows’ intrinsic images because they are an efficient representation of what needs to be known to synthesize an image. It could be difficult to see the finding scale to other datasets, especially in the wild image datasets where there is no clear principal component.\n\n- In addition, the paper seems to be a solely ‘scientific finding’ paper, I have difficulty thinking about what aspect of this finding will be useful in terms of the application side since there is clearly still a performance gap with (even non-SOTA) predictors, and it needs SOTA monocular predictor to guidance the finding of the direction offset at the first place.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to review our paper and for providing a detailed and constructive assessment, particularly thanks for the “fascinating finding”, “technically sound”, “major result”. We’ve tried to address the issues below.\n\n**On the evaluation criteria and dataset diversity:** We appreciate the reviewer’s perspective on the potential limitations of our experiments based on bedroom images.\nTo address their concern, we wish to highlight that our experiments are not solely restricted to bedroom images. We extended our evaluations to multiple datasets, thereby offering a broader perspective on our findings.\n\n- **FFHQ**:  We performed evaluations on the FFHQ dataset, primarily focused on human faces. Our experiments demonstrated that the StyleGAN model trained on this dataset also had similar latent offsets that could generate intrinsic properties such as normals and depth. This highlights our claim that the findings are not scene-specific.\n- **LSUN Churches**: To further validate the versatility of our findings, we conducted experiments on the LSUN Churches dataset. This dataset, characterized by its architectural nuances and diverse lighting conditions, also showed consistent results with our primary claims. The offsets identified were in line with our primary observations, further emphasizing the generalizability of our findings.\n\nWith these additional experiments, our results suggest the broader applicability of using latent offsets to discover intrinsic images across different datasets. We'll ensure these extended evaluations are more emphasized in our updated manuscript, supporting our theory that generative models can implicitly learn intrinsic properties across different datasets.\n\n**On practical applications of our findings:** We believe that our work, while primarily a scientific investigation, has potential implications for various areas in the domain of machine learning and computer vision.\n\n- **Understanding Model Properties:**\nOne of the significant takeaways from our paper is a deeper understanding of the properties and capabilities of generative models, especially when they are not explicitly trained with certain supervision. This could lead to insights into building more effective and efficient training methodologies for GANs and potentially other generative models.\n- **What Makes Good Image Representation:**\nOur findings give insights into what constitutes a 'good' image representation. By showing that intrinsic properties like depth, albedo, and normals can be implicitly represented in the latent space, we hint at the notion that meaningful image representations are those that can capture and disentangle these fundamental scene properties highlighted in the seminal work of Barrow and Tenenbaum [1978]. The materials literature (eg [1] and [2] particularly) suggest there might be lots of kinds of intrinsic images.  It’s just possible one might find intrinsics not currently known using our method to search. \n\n- **Intrinsic Images from Generative Models:**\nOur work is first in the direction of extracting robust intrinsic images purely from generative models, and that too with simply adding an offset to a pretrained generative model. Even if the current method has a performance gap compared to state-of-the-art monocular predictors, this understanding that generative models can capture such intrinsic properties provides a new avenue for research.  We think other generative models “know” intrinsic images, too, but don’t currently know how to elicit that knowledge.\n- **Robust Intrinsic Images:**\nOne of the potential advantages of our approach is the possibility of deriving intrinsic images that are inherently more robust, especially against variations like lighting changes (See Fig ).  One might use this to bootstrap a better StyleGAN predictor from a non-robust, but SOTA, predictor.\n- **Guidance from SOTA Monocular Predictor:**\nWhile we currently utilize state-of-the-art monocular predictors for direction offset guidance, we don’t use all that much guidance (2000 examples, L124).  Future models might not need such guidance. The fundamental understanding that intrinsic images are baked in latent spaces of a generative model, in this case, is more valuable than the specific methodology. With advancements in generative modeling, we envision a scenario where these offsets could be derived in more unsupervised or semi-supervised manners.\n\nIn conclusion, while our paper might seem “solely a scientific finding” in nature, the implications and directions it opens up for the broader community can have far-reaching practical applications. We'll make efforts to emphasize these potential applications more explicitly in the paper to ensure readers grasp the broader significance of our findings.\n\n**On Synthetic Datasets with Ground Truth for Evaluation:** Thank you for emphasizing the value of synthetic datasets with ground truth for evaluation.\nWe acknowledge the benefits of such datasets. However, GAN inversion for complex scenes, especially indoor environments like bedrooms, remains challenging. Current inversion techniques, including recent advancements[3], aren't yet fully equipped for these intricate settings. The need for a robust GAN inversion method that can address a diverse range of scenes is essential before we tap into synthetic datasets. We've briefly touched upon these challenges in our introduction and discussion.\n\nWe see Reviewer NPHF feedback as a reinforcement of our planned future direction: refining GAN inversion to make evaluations on synthetic datasets feasible. \n\n[1] Motoyoshi, I., Nishida, S. Y., Sharan, L., & Adelson, E. H. (2007). Image statistics and the perception of surface qualities. Nature.\n\n[2] Sharan, L., Rosenholtz, R., & Adelson, E. (2009). Material perception: What can you see in a brief glance? Journal of Vision.\n \n[3] Roich, D., Mokady, R., Bermano, A. H., & Cohen-Or, D. (2022). Pivotal tuning for latent-based editing of real images. ACM TOG."}, {"review_id": "NdoxewT95p", "reviewer": "Reviewer_ouPJ", "summary": "The paper uses StyleGAN latent space properties to extract properties like normals, depth, segmentation, albedo and shading. The method does not train a new model to achieve this. The method relies on the properties of w+ space to predict these properties. In practice, the method uses the off-the-shelf predictors to use as pseudo labels and the signal is back-propagated to the w+ code. The authors evaluate their method on the LSUN bedroom dataset. The generated images and their corresponding predictions show that the StyleGAN is able to generate consistent results on bedrooms which suggests that the style codes can be modified to generate results in other domains not seen by the network.", "questions": "Please look at the questions in the Weakness and the Limitation sections.", "limitations": "The authors have not discussed the limitations of the work clearly. When does the method fail? Does the method work without the truncation of the latent space where the samples can be diverse but not high quality? How do the off-the-shelf predictors behave in this scenario? What about other datasets for example ImageNet, (StyleGAN-XL)?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1)  The paper explores a method to generate multiple properties of an image just using the w+ space of the StyleGAN. The paper shows that these properties like normals, depth, segmentation, albedo and shading can be produced by the latent space of the StyleGAN which is primally used for style transfer tasks. The results in the paper shows that the method produces consistent results on bedrooms.\n\n2) The method works good on StyleGAN generated bedroom images. The paper shows multiple figures demonstrating consistency of the method and the qualitative evaluation.\n\n3) The paper evaluates the method using the metrics of the corresponding tasks. The results show that the scores are not random and the method produces meaningful predictions of the properties.", "weaknesses": "1) Firstly I feel that the abstract of the paper can be improved. To me it felt more like an introduction than an abstract. I would ask the authors to revisit the abstract and write it in the conventional way. What problem are you solving ? Why is it important/ what is the motivation? What is your solution? What results you show in the paper. In the current form it is not informative to me.\n\n2) While the paper has proposed to solve some discriminative tasks using StyleGAN, there are other papers like DatasetGAN[1], Lables4Free [2] etc for segmentation and others related like [3] that already show that the features of the StyleGAN can be modified to achieve the goal or a dataset derived from StyleGAN can be used for the training. It is not clear if the method is generalizable especially when the previous methods have shown to exhibit these properties. How general is the method compared to these methods for example in segmentation domain? I would suggest adding a comparison.\n\n3) I am not sure about the evaluations done in the paper. While pseudo labels for evaluation is a way to evaluate the results, another (more significant) way would be to train a supervised network (see Labels4free) based on the predictions of the StyleGAN. Then it can be tested on the real images rather than the generated ones. What about the inversion of a real photograph? Is the method generalizable to the real domain? The current evaluations do not show that. \n\n4) The authors only test on LSUN bedrooms dataset. It is difficult to access the contribution of the paper if its evaluated on only one dataset especially when other pre-trained StyleGANs are available. Why did the authors only evaluation on this particular dataset?\n\n[1] Abdal, Rameen, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. \"Labels4free: Unsupervised segmentation using stylegan.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13970-13979. 2021.\n\n[2] Zhang, Yuxuan, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. \"Datasetgan: Efficient labeled data factory with minimal human effort.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10145-10155. 2021.\n\n[3] IMAGE GANS MEET DIFFERENTIABLE RENDERING FOR INVERSE GRAPHICS AND INTERPRETABLE 3D NEURAL RENDERING, ICLR 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comprehensive review and the effort they took in understanding our work. We acknowledge their concerns and hope to clarify them with this response.\n\n**Abstract:** We appreciate feedback on the abstract and will strive to rewrite it to be more conventional and informative. Pointers about including motivation, problem, solution, and results will be taken into consideration.\n\n**Comparison with Prior Work:** Reviewer’s observation regarding works like DatasetGAN, Labels4Free, and others is insightful. While we had already cited most of these works (see L73 to L83), we missed Labels4Free, and we will ensure it's included in the revised manuscript.\n\nOur work stands apart from these previous efforts in a significant way. While those methods require additional learning, decoding layers, or specific modifications, our approach does not. We specifically designed our research to uncover the intrinsic properties directly from the latent space of StyleGAN without additional training or altering the features. This marks a fundamental departure from existing methods.\n\nOur optimization process is designed to reveal that information such as normals, depth, segmentation, albedo, and shading are readily available in the latent space of StyleGAN. This emphasizes the surprising richness and versatility of StyleGAN's latent space, a quality that has not been fully explored in previous works.\n\nWe acknowledge the importance of contrasting our method with those prior works and will provide a more nuanced comparison in the revised version of the paper. This will clarify the unique contributions of our approach and how it complements and extends the existing body of knowledge.\n\n**On Evaluation:**  We appreciate the reviewer's concerns regarding our evaluation methodology. However, it appears there might have been a misunderstanding of the primary objective of our paper.\n\nOur focus diverges from works such as Labels4Free and others cited. Instead of aiming to solve discriminative tasks or vying to outperform these methods, our central goal is to elucidate that StyleGAN intrinsically possesses this wealth of information within its latent codes. This knowledge can be readily extracted without necessitating supervised learning or additional training.\n\nWe've explicitly highlighted in both our Introduction and Discussion sections that contemporary state-of-the-art inversion methods for complex scenes, like bedrooms, rely on optimization-based techniques. These often finetune the GAN and inadvertently distort its latent space. Crafting an inversion strategy that can both retain intrinsic images and adhere to the primary latent structure of the GAN is undoubtedly a challenging task, and we believe it's beyond the scope of our current exploration.\n\nAt its core, our work is an initial endeavor to demonstrate that intrinsic images play a pivotal role in the image formation process. Consequently, when a model like StyleGAN is trained to generate images, it inherently learns these intrinsic properties. We believe that emphasizing this unique perspective would enrich the understanding of generative models and their intrinsic capabilities.\n\n**Other Datasets:** We initially used the LSUN bedrooms dataset due to its complexity and relevance to our research. However, we have extended our experiments to include other datasets such as FFHQ and LSUN churches-trained StyleGAN models. Our preliminary findings indicate similar offsets and properties across these datasets, confirming the robustness and generalizability of our method. These results will be included in the updated version of the paper.  Refer to our rebuttal PDF for results on these datasets.\n\n**ImageNet's StyleGAN-XL:** This model is a variant of StyleGAN-3, incorporating Fourier feature blocks and additional blocks to overcome equivariance and aliasing issues in StyelGAN-2. While StyleGAN-3 ameliorates texture sticking artifacts, it has been found less suitable for image editing when contrasted with the StyleGAN-2 architecture because of disentangled latent representations. Notably, even the recent state-of-the-art GAN model, GigaGAN, has reverted to the StyleGAN-2 architecture. In our trials, discovering effective surface normal directions for StyleGAN-XL on ImageNet proved challenging. Even after 10,000 images, we noted that while there was a tendency to generate normals, the results were not as refined as those from other datasets. Additionally, the absence of open-source pretrained models for both StyleGAN-2 or StyleGAN-2-XL or GigaGAN on ImageNet further limits our exploration. We posit that a different latent code search strategy, rather than the basic approach we employed, might be more effective in uncovering intrinsic images for diverse network architectures.\n\nWe hope our response provides clarity and bridges the gap between the reviewer’s concerns and our presented work. We believe that our approach offers a fresh lens through which to view the properties inherent in StyleGAN's latent space, paving the way for further exploration in this area.\n"}, {"review_id": "J9JBaKpNtS", "reviewer": "Reviewer_1Cfh", "summary": "The paper proposed the method of extracting intrinsic images (Normal, Depth, Albedo, Shading, and Segment) from a pre-trained StyleGAN model without any pre-training or finetuning. It is empirically shown to be better than the SOTA for getting images intrinsic and robust to relighting effects, unlike SOTA.\n", "questions": "Refer above", "limitations": "The proposed method does not work on real images. The authors pointed out that this is due to GAN inversion, which does not preserve the parametrization of the W-space. With improvement in the GAN inversion method, the proposed approach can work on real images also.\n\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Showing that the StyleGAN W-space also learned the internal representation of intrinsic scene properties without being explicitly trained for the same.\n\nProposed a simple, effective, and generalizable method for extracting intrinsic information without explicit training StyleGAN. Optimization based method is used to search the desired latent code for intrinsic image generation.\n\nExtracted intrinsic images as robust to lighting changes as compared to SOTA for extracting intrinsic images.\n\n", "weaknesses": "This work can be considered as StyleGAN W-space exploration for intrinsic image generation.\n\nAs mentioned in the paper, pre-trained StyleGAN is used, but all the quantitative and qualitative results are only shown for the LSUN Bedroom dataset. Does it applicable to other datasets, also such as the LSUN Church dataset, Animal Faces, FFHQ face, etc.?\n\nAs per the method proposed in the paper, pre-trained intrinsic SOTA models are used for the optimization, so exploring W-spaces for these datasets directly depends on how well SOTA generalizes for other datasets.\n\nFor real images, if we can use the SOTA models for intrinsic extraction directly, why is there a need for GAN inversion, as discussed in the paper? One of the purposes of GAN inversion is to give control over image editing of real images, and as per the problem statement in the paper, there is no such task of editing images based on intrinsic parameters.\n\nThe proposed work uses the SOTA model for optimization, then it is possible for the SOTA model to make incorrect predictions for some images due to domain shifts in train-test data, and it can carry forward to latent space optimization as well as in quantitative and qualitative evaluation.\n\nMinor mistakes:\nOn line 117, it should be w+’ not w’.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1Cfh for their detailed review and the insights provided. We're grateful to the reviewer for recognizing the novel insights our work brings, especially how the StyleGAN W-space captures intrinsic scene properties without explicit instruction. We are glad to know the reviewer appreciated our method is “simple”, “effective” and “generalizable”, and its “robustness against lighting changes” compared to SOTA methods. We address the reviewer’s concerns below:\n\n**Scope of StyleGAN W-space Exploration:**  Yes, we’re exploring W-space.  But we’re finding extraordinary and unexpected properties when we do. Without any explicit training, W-space exploration can elicit intrinsic images of several types.\n\n**Dataset Limitation:** Reviewers' concerns about the limitation of our results to the LSUN Bedroom dataset are valid. We have conducted experiments on other datasets including FFHQ and  LSUN churches, finding similar offsets that generate normals, depth, and more. These findings further corroborate the generalizability of our method. We will emphasize this in our revised manuscript. Also, see our rebuttal's PDF for results on these datasets. \n\n**Dependence on SOTA Models for Optimization:** While we leverage SOTA models for optimization (but not much, using only 2000 examples; L124), the main takeaway is not the actual values these models provide, but the proof of concept that the W-space of StyleGAN has encoded representations to produce intrinsic images by simply adding an offset to the w+ code of an image. The reliance on SOTA models is just a medium to steer our optimization toward a meaningful direction.\n\n**Necessity of GAN Inversion:** The topic of GAN inversion is a complex one. While the reviewer is right that SOTA models can be directly employed for intrinsic extraction, our objective extends beyond mere extraction. Our intention is to demonstrate that these intrinsic properties are implicitly embedded in the GAN's latent space. In cases where there's a desire for more nuanced control or the generation of intrinsic properties not overtly present in real images, our method showcases potential.\n\n**Concerns on SOTA Model Predictions:** We concur that domain shifts can introduce inaccuracies in SOTA models' predictions. But it's crucial to note that our method’s primary purpose is not to outperform these models but to manifest that StyleGAN’s latent space already contains a wealth of intrinsic image information. The occasional inaccuracies introduced by domain shifts, though impactful, do not diminish the primary conclusion we draw from our findings.\n\n**Minor Corrections** Thanks for pointing out the mistake on line 117. We'll ensure to correct it in our revised manuscript.\n\n**Real Images Limitation:** Reviewer 1Cfh has astutely noted that our method currently does not work for real images due to constraints in GAN inversion techniques. However, this limitation, while significant, doesn't eclipse the broader implications of our findings. As GAN inversion techniques improve, the applicability of our method will inherently extend to real images, opening new avenues in the domain of intrinsic image analysis.\n\nWe genuinely appreciate the reviewer’s thorough review and constructive feedback, which will undoubtedly enhance the clarity and comprehensiveness of our work."}, {"review_id": "zLnKvmRDGI", "reviewer": "Reviewer_qbrH", "summary": "This paper proposes a novel method for obtaining \"intrinsic images\" for an RGB image, where in this context intrinsic images mean mainly depth, normal, segmentation, albedo, and shading maps. To this end, the authors investigate how a pre-trained StyleGAN generator model can be used to output the respective intrinsic image instead of an RGB image by only adding a constant in style weight space. More specifically, the proposed method optimizes an offset vector in the StyleGAN \"w+\" style space for a given editing constraint. The offset vectors are optimised via gradient descent where a comparably small dataset of generated images together with respective intrinsic images, predicted by SOTA models, is used as guidance information. The authors compare the proposed method to SOTA methods and find that it performs comparably. ", "questions": "- How sensitive is the proposed system to the number of image \"labeled\" image pairs used for training?\n- Could the GT image and data modality pairs of the test set of the SOTA predictors be used as GT for evaluation?\n- Why is the proposed system better in the L1 depth metric while it looks worse qualitatively?\n- Is the same offset style vector d(c) used across all images for the same intrinsic image modality?\n- How many images of the \"labeled\" datasets are used for optimising the offset style vectors?\n\nFor more context on these questions, please see the \"Weakness\" paragraph.\n\nThe manuscript contains the following typos / other formatting errors:\n\nTypos:\n\n\tCaption Fig 1: “StyleGAN has easily accessed and accurate representations of intrinsic images” is unclear and grammatically wrong. What does this mean?\n\tCaption Fig 1: ““knows” fine detail in normal” -> “knows” fine detail in normal maps \n\tL. 49: “Showing that intrinsic images so extracted” -> “Showing that these intrinsic images extracted from StyleGAN …\n\tL. 54: “and produce blurry outputs” -> and producing blurry outputs\n\tL. 91: “significant recent methods” -> Important recent methods\n\tL. 92: “Learned methods” -> Learning-based methods\n\tCaption Fig 2: “We demonstrate a StyleGAN that is trained to generate images encode accessible scene property maps” -> […] is trained to generate images encodes accessible scene property maps\n\tCaption Fig 2: “Our approach explores [….], generate” -> Our approach explores […], generates\n\tL. 123: “a select number” -> a selected number\n\tL. 198: “We have demonstrated that StyleGAN has easily extracted representations” -> We have demonstrated […] has easily extractable representations\n\tL. 200: “be true of other generative models” -> be true for other generative models\n\nOther:\n\n\tFigure 1 and Figure 2 are not referenced in the text. Please add a reference to the main text.", "limitations": "- The authors show and discuss and example data modality / transformation that is cannot be tackled with the proposed approach due to the pixel-aligned features (L. 178).\n- The authors investigate the performance of the proposed system for different lighting conditions of the images (L. 184).\n- In the discussion (Sec. 7), the authors discuss unclear points that would require additional investigation. This includes to what extent these findings translate to other generative models and if more intrinsic image modalities next to the studies ones are contained in StyleGAN's weight space.\n- The authors do not discuss potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed approach is an interesting and valuable method for investigating what information is already contained in generative models such as StyleGAN. The finding that adding an offset vector in style weight space to output one of the other data modalities instead of RGB is surprising and very interesting. It highlights how much \"semantic and 3D scene understanding\" can be found in 2D generative models.\n- The authors investigate the performance of the same proposed method with multiple different data modalities highlighting the generality of the approach.\n- I appreciate that the authors also show an example decomposition that clearly does not work with the given method (L. 178). As only a style weight space vector is added, non-pixel-based transformations are expected to not work well / fail.\n- The authors compare against relevant baselines quantitatively and qualitatively and show helpful visual comparisons.", "weaknesses": "- Performance across different datasets / Dependence on small labeled dataset: The proposed method requires a sub-set of \"labeled\" generated image and intrinsic image pairs, where the latter is provided by a SOTA method. Have the authors investigate how well the method performs if this \"fine-tuning\" is done on a different dataset, or on only a sub-set of the covered \"modes\"? More general, investigating different factors of this training stage with the resulting end performance, e.g. in form of a graph showing number of training images against performances, would be very interesting.\n- Labeled dataset: In similar spirit, how many labeled images (provided by the SOTA method) are used? (L. 124 - 126 is vague on this, and it would be beneficial to state clear numbers.)\n- Evaluation: The authors argue that no ground truth is available, hence SOTA predictions are used as GT. But couldn’t the test set of these SOTA methods be used as GT? For the proposed method, the input image would be needed to be inverted in StyleGAN space, but this has been extensively done before in the literature. Then, the proposed method could directly be compared to the SOTA prediction model itself.\n- Depth Comparison: It is unclear to me why the proposed method performs better in the L1 depth metric (Table 1) while it looks qualitatively worse than the other methods. Could the authors expand on this? I am not sure if this is caused by some scale/shift misalignments; as these are monocular predictors, the scale and shift is arbitrary, so this needs to be taken into account during evaluation.\n- Caption Fig 6: “Note that our quantitative comparison in Table 2 to a SOTA segmenter [17] likely understates how well StyleGAN can segment;” -> Can the authors expand on this? This is not clear to me. The authors give the two-lamp example, however, here all methods were able to segment the two lamps.  \t", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, Reviewer qbrH, for your insightful feedback and acknowledgment of the strengths in our work. We are particularly heartened by your recognition of our proposed approach as an “interesting and valuable method” for investigating the information contained within generative models like StyleGAN. Your emphasis on our “surprising and very interesting findings” and the highlighting of the “generality of our method” reinforces the core strengths of our research. We also appreciate your keen observation of our example decomposition and the expectation regarding non-pixel-based transformations. Lastly, your positive note on our comparison against relevant baselines and the inclusion of helpful visual comparisons is greatly valued. We will now address the points raised in your review.\n\n**Performance Across Datasets & Dependence on Labeled Data:** We appreciate the reviewer's emphasis on the generalizability of our method. We now have expanded our evaluations to include additional datasets beyond the LSUN Bedroom. These new evaluations on the FFHQ and LSUN bedroom model firm the consistency of our method's performance across diverse datasets, reinforcing its robustness. Please see our rebuttal's PDF files for results on these datasets. \n\nFurthermore, to address concerns regarding the dependence on the size of the labeled dataset, we have included a new plot illustrating the relationship between the number of labeled images and the quality of the generated segmentations for pillows and windows class. This graph spans a range from 100 to 2,000 labeled images. Our findings indicate an increase in performance as we increase the labeled dataset and start saturating around the 2000 images mark. The performance does vary, showing improvements with more labeled pairs. With more labeled examples and a better search procedure, we may get better intrinsic images from StyleGAN. However, the goal of the paper is not aimed to outperform SOTA methods but to demonstrate intrinsic images are baked into the latent space of StyleGAN without being explicitly trained to do so.\n\n**Labeled Dataset Size Clarification:** We understand that our explanation might have been ambiguous. We used 2000 labeled image pairs for our experiments. All seen only once. We will update the manuscript to provide this clear number. These labeled images required are significantly smaller than those required to train SOTA models, suggesting generative models would be a promising backbone for visual perception research.  \n\n**Evaluation using SOTA Ground Truth:** An insightful suggestion! We considered using the SOTA test sets as ground truth. The challenge is in inverting the input image accurately into the StyleGAN space. We are researching better inversion methods to make this comparison more direct.\n\n**Depth Comparison:** We recognize the disparity between the quantitative and qualitative evaluations. The mentioned L1 metric advantage is possible because our approach can capture certain depth nuances better. However, we do admit that in terms of visual clarity, the proposed method might not always outshine others. This will be explored further.\n\n**Offset Style Vector d(c):** Yes, for the same intrinsic image modality, the same offset style vector d(c) is applied. We believe this is a significant aspect of our approach, and it is worth elaborating on.\n\nOur method’s design allows for the optimization of an offset vector that is not image-specific but is tied to a particular intrinsic modality. Once this vector is optimized for a specific intrinsic feature (e.g., depth or shading), it can be universally applied to any image within the same dataset to extract that particular feature. This universality showcases the model's understanding of that intrinsic property across various image instances.\n\nThe application of a common offset vector not only simplifies the extraction process but also underscores the underlying cohesiveness and consistency of StyleGAN’s latent space in representing these intrinsic properties. This approach further supports the generalizability of our method, offering a more efficient and unified way to explore and utilize the rich information encapsulated within the StyleGAN model.\n\n**Figure 6's caption:** The point we're making is that while Table 2 offers a quantitative comparison, there are scenarios where StyleGAN may exhibit superior segmentation abilities than reflected in those numbers (See Fig 1 and Fig 13 in Supplementary). In the revised manuscript, we'll clarify this by providing clearer examples of StyleGAN's segmentation potential. \n\n**Typos and Formatting Errors:** Thanks for pointing them out. They will be duly corrected in the final manuscript. References to Figures 1 and 2 will be incorporated into the main text.\n\nWe will strive to address each of the concerns the reviewer has raised, further enhancing the value and clarity of our work. Reviewer qbrH's feedback has been instrumental in guiding these improvements."}]

Author rebuttal:
We are grateful for the comprehensive feedback provided by all four reviewers. Here's a synthesized response to the main concerns raised across reviews:

**Scope and Contribution Clarification:** Our primary contribution is to demonstrate that StyleGAN inherently “knows” important scene properties -- intrinsic images -- without being explicitly trained to do so. Only a small amount of labeled data (2000 in our case, obtained from SOTA models) is needed to elicit this knowledge, implying that they may emerge in an image generator because they are highly useful representations. We are not primarily aiming to outperform SOTA models but to reveal this underlying capability within StyleGAN.

**Applicability across datasets:** A recurring query was to know what happens on data other than rooms. To this end, we have broadened our evaluations, including datasets like LSUN Church and FFHQ dataset. For all these datasets, StyleGAN still “knows” intrinsic images. Please see our rebuttal’s PDF for results on these datasets.

**Method Soundness:** Thanks to all reviewers. We think our observations are novel and effective, too. 

**Offset Style Vector d(c):**  For each intrinsic “type” (depth, albedo, normal, etc), we need to find one offset per StyleGAN instance that forces the StyleGAN to generate that intrinsic. Unlike related works, we don't need to fine-tune the model or learn new decoding layers. The offset is not “per image”; the StyleGAN must “know” intrinsics consistently and strongly. We use SOTA intrinsic generation methods to find this offset, but only a few instances are needed (passing 2000 generated images through an intrinsic predictor is enough). Ground truth can't be used currently because GAN inversion is not reliable enough.
 
**Minor Typos & Refinements:** We are grateful to the reviewers for pointing out areas of improvement in the presentation and have since rectified all listed errors.


We firmly believe our work offers a fresh perspective on the intrinsic capabilities of pre-trained generative models, specifically StyleGAN. With the clarifications provided (detailed responses to each reviewer below), we hope our paper stands as a meaningful contribution to the community.

Meta-review: Reviewers and the AC read the rebuttal and took that into consideration for their final recommendation. Reviewers find this work insightful, original, and potentially impactful. Many suggestions were made by the reviewers to improve the exposition, discussion of prior work, and limitations - please incorporate these in your camera-ready paper.
Decision: Accept (poster)