Please summarize the following paper: 
Generative models have been shown to be capable of synthesizing highly detailed and realistic images. It is natural
to suspect that they implicitly learn to model some image
intrinsics such as surface normals, depth, or shadows. In
this paper, we present compelling evidence that generative
models indeed internally produce high-quality scene intrinsic maps. We introduce INTRINSIC LORA (I-LORA), a
universal, plug-and-play approach that transforms any generative model into a scene intrinsic predictor, capable of
extracting intrinsic scene maps directly from the original
generator network without needing additional decoders or
fully fine-tuning the original network. Our method employs
a Low-Rank Adaptation (LoRA) of key feature maps, with
newly learned parameters that make up less than 0.6% of the
total parameters in the generative model. Optimized with a
small set of labeled images, our model-agnostic approach
adapts to various generative architectures, including Diffusion models, GANs, and Autoregressive models. We show
that the scene intrinsic maps produced by our method compare well with, and in some cases surpass those generated
by leading supervised techniques.
1
arXiv:submit/5260651 [cs.CV] 28 Nov 2023
1. Introduction
Generative models are capable of producing high-quality images that can be mistaken for real-world photographs. These
models display a seemingly profound understanding of the
world, capturing the nuances of realistic object placement,
appearance, and lighting conditions. However, the mechanisms by which these models acquire such detailed knowledge remain largely unexplored. One of the most pressing
questions in this context is: What kind of knowledge do generative models rely on to produce such high-quality images?
Are they manipulating abstract, high-level representations
of the world, working with more physical, concrete scene
representations, or perhaps a combination of both?
Recent work has begun to shed light on this question.
For instance, Bhattad et al.[8] demonstrated that StyleGAN
can encode important scene intrinsics like depth and normals. Similarly, Zhan et al.[60] showed that diffusion models can understand 3D scenes in terms of geometry and
shadows. Chen et al.[11] found that Stable Diffusion’s internal activations encode 3D depth and saliency maps that
can be extracted with linear probes. Three independent
groups [22, 37, 51] found correspondences in generative diffusion models without explicit supervision. However, these
studies are often model-specific and do not address whether
these capabilities are inherent to all large-scale generative
models or are a result of specific architectural choices.
In this paper, our goal is to understand the underlying
knowledge present in all types of generative models. We
employ Low-Rank Adaptation [25] (LoRA) as a unified approach to extract scene intrinsic maps—namely, normals,
depth, albedo, and shading—from different types of generative models. Our method, which we have named as
INTRINSIC LORA (I-LORA), is general and applicable to
diffusion-based models, StyleGAN-based models, and autoregressive generative models. Importantly, the additional
weight parameters introduced by LoRA constitute less than
0.6% of the total weights of the pretrained generative model,
serving as a form of feature modulation that enables easier
extraction of latent scene intrinsics. By altering these minimal parameters and using as few as 250 labeled images, we
successfully extract these scene intrinsics.
Why is this an important question? Our motivation is
three-fold. First, it is scientifically interesting to understand
whether the increasingly realistic generations of large-scale
text-to-image models are correlated with a better understanding of the physical world, emerging purely from applying a
generative objective on a large scale. Second, rooted in the
saying “vision is inverse graphics” – if these models capture
scene intrinsics when generating images, we may want to
leverage them for (real) image understanding. Finally, analysis of what current models do or do not capture may lead to
further improvements in their quality.
How is what we do related to fine-tuning or linear
Model Pretrain Type Domain Normal Depth Albedo Shading
VQGAN [16] Autoregr. FFHQ ∼ ∼ ✓ ✓
SG-v2 [31] GAN FFHQ ✓ ∼ ✓ ✓
SG-v2 [59] GAN LSUN Bed ✓ ✓ ✓ ✓
SG-XL [50] GAN FFHQ ✓ ∼ ✓ ✓
SG-XL [50] GAN ImageNet × × × ×
SD-UNet [46] Diffusion Open ✓ ✓ ✓ ✓
SD [46] Diffusion Open ✓ ✓ ✓ ✓
Table 1. Summary of scene intrinsic extraction capabilities across
different generative models without changing generator head. ✓:
Intrinsics can be extracted with high quality. ∼: Intrinsics cannot
be extracted with high quality. ×: Intrinsics cannot be extracted.
probing? Previous approaches that explore extracting intrinsics from trained generative models include fine-tuning the
models [63] or (usually linear) probing [11]. Fine-tuning a
model, using a dataset of images paired with target scene
intrinsic maps, yields a new model, no longer capable of
generating images. Since all or most of the parameters have
changed, it is unclear whether the fine-tuned model’s ability
to produce, say, scene depth is also inherent in the original
model. In contrast, our I-LORA approach only introduces a
tiny number of extra parameters, effecting minimal change
and leaving the original model easily accessible.
We do not construct additional “layers” on top of model
activations, like the probing methods. Instead, we learn to
leverage the existing model along with the “adaptor” (LoRA)
so that it produces a scene intrinsic map. In this we take
advantage of the fact that for all the intrinsics we study,
these maps can be represented as an image with up to three
channels– something the generative models are already set
up to produce, albeit the nature of these new images is different, requiring the LoRA feature modulation.
What specifically do we study? We conduct experiments with a variety of generative models: diffusion models [24, 32, 46], GANs [29, 31] and autoregressive models [16]. While diffusion models such as StableDiffusion [46] or Imagen [48] have perhaps received most limelight recently, the most recent generation of other types of
models such as GigaGAN [27], CM3leon [58], and Parti [57]
appear able to produce images of similar quality. We believe
it is important to study all of these, and even more important
to consider a general framework that would be applicable to
all these models – and perhaps to others yet to be proposed.
We evaluate the quality of the intrinsics extracted from
the generative models on both real and synthetic images. As
training targets, we use predictions by state of the art specialized networks trained on large datasets as pseudo ground
truth. When available, we also compare the predicted intrinsics to real ground truth. Finally, we inspect the correlation
between the generator’s image quality and the quality of the
intrinsics that can be extracted from it.
Our findings reveal that all types of the generative models
we study contain rich information about scene intrinsics
that can be easily extracted using LoRA. A summary of
our findings is in Table 1, with more details in Section 4.
2
Surface Normals Depth Albedo Shading
VQGAN StyleGANv2 StyleGANXL
Image Omnidatav2 [28] I-LORA ZoeDepth [5] I-LORA Paradigms [7] I-LORA Paradigms [7] I-LORA
Figure 2. Scene intrinsics from different generators – VQGAN, StyleGAN-v2, and StyleGAN-XL – trained on FFHQ dataset: The first
column shows the original synthetic images produced by each model. The subsequent columns show the scene intrinsics corresponding to
surface normals, depth, albedo, and shading, as extracted by a SOTA-non-generative model and our methodology (I-LORA). For surface
normals, the images highlight the models’ ability to infer surface orientations and contours. The depth maps display the perceived distances
within the images, with warmer colors indicating closer objects and cooler colors representing further ones. Albedo maps isolate the intrinsic
colors of the subjects, removing the influence of lighting and shadow. Finally, the shading maps capture the interplay of light and surface,
showing how light affects the appearance of different facial features. Intriguingly, there appears to be a correlation between the fidelity of
the generated images and the accuracy of the surface normals extracted. This suggests that the more sophisticated the image generation
capabilities of a model, the more precisely it can replicate the nuances of real-world physics in its scene intrinsics.
Our work takes a significant step towards demystifying the
intrinsic capabilities of generative image models. It also
may open new avenues for leveraging generative models in
various applications, from computer vision to graphics and
beyond, where these intrinsics are important. In summary,
our main contributions are:
• Showing that the same INTRINSIC LORA method is sufficient for general scene intrinsic extraction across various
generative models.
• Demonstrating high-quality scene intrinsic extraction outperforming SOTA supervised dense prediction models on
some tasks, with minimal addition of parameters (less than
0.6% of the number of the original model parameters in
all cases) and as little as 250 labeled images.
• Demonstrated correlation between the quality of scene
intrinsic maps extracted and the image generation quality
of a model. This provides an alternative perspective for
evaluating generative models.
2. Related Work
Generative Models: Generative Adversarial Networks
(GANs) [20] have been widely used for generating realistic images. Variants like StyleGAN [29], StyleGAN2 [31]
and GigaGAN [27] have pushed the boundaries in terms of
image quality and control over the generated content. Some
works have explored the interpretability of GANs [4, 8], but
few have delved into their ability to capture scene intrinsics.
Diffusion models, such as Denoising Score Matching [55]
and Noise-Contrastive Estimation [21], have been used for
generative tasks and are perhaps the most popular at the
3
moment [24, 32, 46]. These models have been shown to
understand complex scene intrinsics like geometry and shadows [60], but their generalizability across different scene
intrinsics is less explored.
Autoregressive models like PixelRNN [53] and PixelCNN [52] generate images pixel-by-pixel, offering finegrained control but at the cost of computational efficiency.
More recently, VQ-VAE-2 [45] and VQGAN [16] have combined autoregressive models with vector quantization to
achieve high-quality image synthesis. While these models are powerful, their ability to capture and represent scene
intrinsics has not been thoroughly investigated.
Scene Intrinsics Extraction: Barrow and Tenenbaum [3]
highlighted several important fundamental scene intrinsics
that include depth, albedo, shading, and surface normals.
Several works have focused on extracting scene intrinsics
like depth and normals from images [5, 14, 15, 28, 35, 44]
using labeled annotated data. Labeled annotations of albedo
and shading are hard to find and as the recent review in [17]
shows, methods involving little or no learning have remained
competitive until relatively recently. However, these works
often rely on supervised learning and do not explore the
capabilities of generative models in this context.
Many recent studies have utilized generative models such
as [1, 2, 26, 33, 40, 49, 56, 62, 63] as pretrained feature
extractors or scene prior learners. These models use generated images to enhance downstream discriminative models,
fine-tune the original generative model for a new task, learn
new layers, or develop new decoders to produce desired
scene intrinsics. InstructCV [19] executes different computer
vision tasks via natural language instructions, abstracting
task-specific design choices. However, it requires retraining
of the entire diffusion model. It’s still unclear whether the
original models inherently capture important scene intrinsics
information as implicit knowledge or not.
Knowledge in Generative Models: Several studies have
explored the extent of StyleGAN’s knowledge, particularly
in the context of 3D information about faces [42, 61] provide
substantial evidence of StyleGAN’s capability in this area.
Further research has demonstrated that manipulating offsets
in StyleGAN can lead to effective relighting of images [6],
as well as the extraction of scene intrinsics [8].
Chen et al.[11] found that the internal activations of the
LDM encode linear representations of both 3D depth data
and a salient-object / background distinction. Recently, [22,
37, 51] found correspondence emerges in image diffusion
models without any explicit supervision.
Self-supervised models like DINO [10, 12, 41] focus on
learning useful representations without using labeled data.
Although not generative, these models serve as a relevant
baseline for understanding the quality of scene intrinsics that
can be extracted from learned representations.
LoRA (Low-Rank Adaptation) is a technique originally
used to reduce the cost of fine-tuning large language models
for downstream tasks [25]. The approach involves freezing the pre-trained model weights and introducing trainable
low-rank decomposed matrices into specific layers of model
architecture. These matrices are the only components updated during task-specific optimization. This results in a
significant reduction in the number of trainable parameters.
LoRA has been used for various personalization applications
of image generators [47]. In contrast, we use LoRA as a
unified and efficient method for extracting scene intrinsics
across various types of generative models. It does this by
identifying key feature maps in each of them to modulate
– attention layer in diffusion models, linear affine layers in
StyleGAN and convolutional attention layer in VQGAN.
3. Methods
In the most general formulation, a generative model G
maps noise/conditioning information z to an RGB image
G(z) ∈ R
H×W×3
. We seek to augment G with a small set
of parameters θ that allow us to produce, using the same
architecture as G, an image-like object with up to three channels, representing scene intrinsics like surface normals.
The learning framework:We learn to extract intrinsics
in a supervised fashion. Since in most cases (some domains,
generated images) we do not have ground truth intinsics,
we use state of the art models (trained on large datasets) to
predict “pseudo ground truth” maps, e.g., estimated depth
for an image, and treat these as a target for the predictions
of Gθ.
In order to optimize θ of Gθ we employ a pseudo-ground
truth predictor Φ (e.g., a network trained to predict depth
map from an image), leading to the objective function:
min
θ
Ez[d(Gθ(z), Φ(G(z)))], (1)
where d is a distance metric that depends on the intrisic we
wish to learn.
Diffusion models require a more tailored treatment, since
they are effectively image-to-image and not noise-to-image
(since during inference they repeatedly get a noisy image
as input). Thus instead of conditioning noise z we feed an
image x(generated or real) to a diffusion model G. In this
case, given a real image x, our objective function becomes
minθ Ex[d(Gθ(x), Φ(x))].
We describe the metric d and the pseudo-ground truth
predictors Φ used for each intrinsic in Section 4.1.
Low-Rank Adaptation Low-Rank Adaptation (LoRA)
is a parameter-efficient technique for adapting pre-trained
neural networks to novel tasks by introducing a low-rank
weight matrix W∗
. Initially devised for language models
[25], LoRA has since been extended to image generation
models where it is primarily used to enable the generation
of new characters, objects, or styles [47].
4
�
��
�
��
�
��
�
��
“surface normal”
Omnidatav2
Surface Normals
A generative Model’s U-Net
�!
�
�
T
Image
�"|!
MSE Loss
�̃
"
Predicted
Surface Normal
�
� � � �
Figure 3. Overview of our INTRINSIC LORA applied to a single-step diffusion model. We fine-tune low-rank matrices corresponding to
key feature maps—specifically, attention matrices—to elicit scene intrinsics. Distinct low-rank adaptors, indicated by color-coded θ (e.g.,
purple for surface normals), are optimized for each intrinsic. We leverage a limited set of labeled examples to empower generative models
to directly extract scene intrinsics using the original decoder, circumventing the need for specialized decoders or comprehensive model
retraining.
In LoRA, W∗
is designed to have a lower rank than the
original weight matrix W ∈ R
d1×d2
, achieved by factorizing W∗
into two smaller matrices, W∗ = W∗
uW∗
l
, W∗
u ∈
R
d1×d
∗
and W∗
l ∈ R
d
∗×d2
, where d
∗ ≪ min(d1, d2).
The output o for an input activation a is then given by:
o = W a + W∗
a = W a + W∗
uW∗
l a. (2)
To preserve the original model’s behavior at initialization,
W∗
u
is set to zero. We next describe how we leverage LoRA
modules to extract intrinsics from Diffusion models, GANs,
and Autoregressive models.
When adapting diffusion models for intrinsic extraction
we learn I-LORA adaptors on top of the cross-attention and
self-attention layers. We treat the UNet as a dense prediction
model, applying it once to an RGB input with the goal of
getting an intrinsic map as output. We find this gives the
best quantitative results. The text input depends on the intrinsic and is simply “surface normal, “depth”, “albedo”, or
“shading”. The timestep input is set to T=1.0 (i.e. full noise),
we also tested T=0.0 and it yielded equivalent results. An
overview of I-LORA is in Figure 3.
When adapting GANs we learn I-LORA modules on top
of the affine layers projecting from w-space to s-space.
When adapting VQGAN, an autoregressive model, we
learn I-LORA modules on top of the the convolutional attention layers in the decoder.
4. Experiments
4.1. Implementation Details
Notably the compute, parameter, and data requirements for
learning these adaptors is extremely light. In all cases we
use I-LORA adaptors with rank 8 which add less than 0.6%
to the original parameters. We provide detailed training
hyperparameters in the supplement.
To generate pseudo ground truth for depth we use
ZoeDepth [5] as the predictor Φ in Equation (1). For surface
normals Φ is OmniDataV2-Normal [14, 28]. For Albedo
and Shading Φ is Paradigms [7, 17].
For SG2, SGXL and VQGAN, d in Equation1 is
d(x, y) = 1 − cos(x, y) + ∥x − y∥1 (3)
for normal and MSE for other intrinsics. For latent diffusion
based methods, there isn’t a clear physical meaning to the
relative angle of latent vectors in encoded normal maps, so
we use the standard objective of MSE for all intrinsics.
4.2. Synthetic Image Experiments
We begin I-LORA by extracting intrinsics from generated images. We consider a variety of generative models (StyleGAN-v2, StyleGAN-XL, VQGAN) trained on a
variety of datasets (FFHQ, LSUN Bedrooms, ImageNet).
For each model trained on a particular dataset, we train
5
Surface Normals Depth Albedo Shading
StyleGANv2
Image Omni-v2 [28] I-LORA ZoeDepth [5] I-LORA Paradigms [7] I-LORA Paradigms [7] I-LORA
Figure 4. Scene intrinsics extraction from Stylegan-v2 trained on LSUN bedroom images.
Surface Normals Depth Albedo Shading
StyleGANXL
Image Omni-v2 [28] I-LORA ZoeDepth [5] I-LORA Paradigms [7] I-LORA Paradigms [7] I-LORA
Figure 5. StyleGAN-XL trained on ImageNet. Top: pan, bottom: laptop, with the corresponding scene intrinsics (pseudo ground truth and
extracted) alongside. The surface normals and depth maps, while capturing the basic shape and volume, lack precise detail and exhibit
artifacts. The albedo maps fail to consistently separate texture from lighting, and the shading maps do not fully capture the nuanced interplay
of light and surface. These difficulties are correlated with the overall worse realism of the generated images.
Model Pretraining Type Domain LoRA Param. Surface Normal Depth Albedo Shading
Mean Error°↓ Median Error°↓ L1 Error× 100 ↓ RMS× 100 ↓ δ < 1.25× 100 ↑ RMS× 100 ↓ RMS× 100 ↓
VQGAN Autoregressive FFHQ 0.18% 19.97 20.97 16.33 1.819 62.33 0.345 0.106
StyleGAN-v2 GAN FFHQ 0.57% 16.78 19.22 13.72 1.530 90.74 0.283 0.110
StyleGAN-XL GAN FFHQ 0.29% 14.87 18.22 12.27 1.337 93.87 0.287 0.125
StyleGAN-v2 GAN LSUN Bedroom 0.57% 13.24 23.57 10.91 0.897 66.88 0.270 0.074
StyleGAN-XL GAN ImageNet 0.29% 24.09 25.52 19.44 2.175 38.38 1.065 0.119
AUGUNET (multi step) Diffusion Open 0.18% 21.41 28.57 17.39 2.042 41.21 0.881 0.099
SD UNet (single step) Diffusion Open 0.18% 16.63 23.64 13.69 1.179 52.59 0.487 0.118
Table 2. Quantiative analysis of scene intrinsics extraction performance on generated images. We compare with pseudo ground truths from
Omnidata-v2 for surface normals, ZoeDepth for depth, Paradigms for albedo and shading. Metrics include mean angular error, median
angular error, and L1 error for surface normals; rms and δ < 1.25 for depth; rms for albedo and shading. We also include results of SD
UNet and AUGUNET (described in details in Sec 5) on 1000 synthetic images with various prompts for completeness.
I-LORA adaptors to produce surface normals, depth, albedo,
and shading aligned with the RGB images produced by the
original generator. We generally find the I-LORA adaptors
can successfully produce predictions that approximately
match the pseudo ground-truth quantitatively (Table 2), and
qualitatively produce compelling results (Figures 2, 4).
We use the 2562
resolution checkpoints for VQGAN,
StyleGAN-v2 and StyleGAN-XL trained on FFHQ dataset;
with FID [23] scores of 9.6 [16], 3.62 [30] and 2.191
, respectively. Our qualitative results in Table 2 indicate a decreasing
FID (i.e. better image generation) correlates with better intrinsic prediction.
One exception to this is StyleGAN-XL trained on ImageNet, which we find to produce reasonable quantitative
1https://github.com/autonomousvision/stylegan-xl
metrics, but poor qualitative results. We attribute this to the
underlying quality of the generative model, noting that many
samples from this model could never be mistaken for natural
images (Figure 5). This generally supports our finding that
the stronger the generative model, the higher the quality of
the extracted intrinsics.
4.3. Real Image Experiments
While I-LORA results on synthetic images and pseudoground truth are suggestive, they are difficult to judge objectively. Fortunately, diffusion models are not only powerful
image generators, their structure as image-to-image models
makes it straightforward to apply to real images. We adapt
Stable Diffusion’s (SD’s) Unet to extract intrinsics but apply it once as a dense predictor, rather than iteratively as
6
Model Pretraining Type Surface Normal Depth
Mean Error°↓ Median Error°↓ L1 Error× 100 ↓ RMS× 100 ↓ δ < 1.25× 100 ↑
Omnidata-v2 [28]/ZoeDepth [5] Supervised 18.90 13.36 15.21 2.693 47.56
DINOv2 Non-Generative 19.74 13.72 16.00 2.094 44.32
AUGUNET (multi step) Diffusion 23.74 19.08 19.31 2.651 43.19
SD UNet (single step) Diffusion 20.31 12.54 16.53 2.046 44.90
Table 3. Quantitative analysis of scene intrinsic extraction performance across different models on real images. See caption of Tab 2 for
details on pseudo-ground truth and metrics.
Mean Angular Error°↓ 18.90 19.74 27.73 22.22 20.31 21.26 21.64
L1 Error (× 100) ↓ 15.21 16.00 22.46 18.05 16.53 17.33 17.64
(a) Real (b) GT (c) Omni-v2 [28] (d) DINOv2[41] (e) 250 (f) 1000 (g) 4000 (h) 16000 (i) 24895
Figure 6. Ablation study on the number of training samples using a single step SD UNet. We use surface normal prediction to illustrate.
Column (c) is the Omnidata-v2 pseudo label we use for training. Numbers below columns (e) to (i) are the number of samples used during
training. The results demonstrate that even with a limited dataset comprising a few hundred images, the pretrained stable diffusion model can
discern substantial surface normal details. Notably, with a dataset size of merely 4000 samples, our approach accurately captures challenging
areas, such as the top left distant walls in (a). We started all models with SD v1-5 and LoRA rank=8.
Mean Angular Error°↓ 18.90 19.74 22.28 22.57 20.31 21.17 21.84
L1 Error (× 100 )↓ 15.21 16.00 18.14 18.39 16.53 17.19 17.81
(a) Real (b) GT (c) Omni-v2 [28] (d)DINOv2 [41] (e) Rank = 2 (f) Rank = 4 (g) Rank = 8 (h) Rank = 16 (i) Rank = 32
Figure 7. Ablation study–LoRA’s Rank. Numbers under columns (e) to (i) are the numbers of ranks of LoRA. 8 achieves memory efficiency
and gives good results. All models are trained with SD v1-5 and 4000 samples.
in diffusion. We train I-LORA adaptors for surface normal
and depth extraction using a random subset of 4000 training pairs from the DIODE [54] training set. While we use
pseudo ground truth during training, DIODE provides real
ground truth we can use for quantitative evaluation.
We find that not only I-LORA adaptors capable of
roughly matching the performance of Φ (which provided
the training signal) while using far less data, parameters, and
training time; but they even surpass Φ in several metrics (Table 3). A potential explanation is that the intrinsic implicitly
learned by the generative model is actually more accurate
than that learned by the supervised SOTA predictor Φ.
4.4. Ablation Studies
In our investigation, we conduct a series of ablation studies
focusing on our single step SD UNet model, which has the
highest quantitative performance. We explore the impact of
the number of labeled examples on the model’s ability to
extract scene intrinsics (Figure 6), compare multiple versions
of stable diffusion models to further explore how improved
generation quality affects extrinsic extraction (Figure 10,
and determine the optimal rank for our low-rank adaptations
(Figure 7). In short these ablation studies reveal that higher
quality generative models lead to better intrinsic extraction,
and that the quality of intrinsic extraction saturates with fairly
few training samples (4000) and low rank LoRAs (rank 8).
4.4.1 Comparison with DINOv2
A natural question is how do the features learned by generative models compare with those from other self-supervised
foundation models. To provide a preliminary benchmark we
7
(a) Real Image (b) Omni-v2 (c) DINOv2 (d)AUGUNET1.5 (e) AUGUNET (f) GT (g) ZoeDepth (h)DINOv2 (i) AUGUNET1.5 (j) AUGUNET (k) GT
Figure 8. Detailed Scene Intrinsic Extraction with Improved Diffusion Techniques from Our AUGUNET models: We show scene intrinsics
derived from generative models alongside supervised counterparts. AUGUNET1.5 is the same as AUGUNET except it uses SD1.5 and
does not use Zero SNR strategy. AUGUNET1.5 presents sharper details, especially in complex areas – thin structures like lamp stand and
car. AUGUNET, on the other hand, illustrates a significant improvement in reducing color shifts while maintaining detail sharpness, as seen
in the comparison with ground truth (GT) data in the last row.
Figure 9. Results of AUGUNET models applied on unseen 10242
synthetic images. For each group, from left to right, we have: image,
surface normal, depth, albedo and shading, respectively. The first row contains results from the I-LORA . The second row contains the
corresponding pseudo ground truth from off-the-shelf SOTA methods.
8
Mean Angular Error°↓ 21.84 21.41 20.31
L1 Error (× 100) ↓ 17.78 17.38 16.53
(a) Real (b) GT (e) SD v1-1 (f) SD v1-2 (g) SD v1-5
Figure 10. Through an ablation study of various Stable Diffusion
versions using a single-step method, we observe a correlation between the quality of the generative model and the accuracy of the
extracted scene intrinsics. Our analysis includes Stable Diffusion
versions 1.1, 1.2, and 1.5, each demonstrating significant enhancements in image generation capabilities, which in turn improve the
fidelity of scene intrinsics extraction.
Image Omni-v2 [28] SD UNet SD1.5 (multi) AUGUNET
Figure 11. Single-step diffusion (SD UNet) yields satisfactory
results, but multiple diffusion steps lead to misalignment in the extracted intrinsics, as shown in the SD1.5 (multi-step) column. The
last column, AUGUNET , demonstrates the successful rectification
of these misalignments using our image conditioning approach,
resulting in well-aligned and coherent scene intrinsics extractions
apply our I-LORA framework to DINOv2 [41] , a SOTA
general purpose visual feature extractor. We start with their
best and largest model, “giant”, and apply rank 8 LoRA
modules to all attention layers. In addition, we must learn
a small linear head to project DINO output to 3 channels
(1 channel in the case of depth). These modules lead to
a parameter increase of 0.26%, similar to those added for
generative models (Table 2). We train these parameters on
DIODE images using the same procedure outlined in Section
4.3, finding that the results are comparable to our best model
single-step model based on SD, see Table 3.
5. Towards Improved Intrinsic Extraction
Our findings suggest that Stable Diffusion models inherently
capture various scene intrinsics like normals, depth, albedo,
and shading, as evidenced by our single-step SD Unet experiments. However, a natural question is: can we enhance the
quality of the intrinsics by leveraging the multi-step diffusion
inference? While multi-step diffusion improves sharpness,
in practice we find that it introduces two challenges: 1. intrinsics is misaligned with input, and 2. a shift in the distribution
of outputs relative to the ground truth (visually manifesting
as a color shift) (see Figure 11).
To address the first challenge, we augment the noise input
to the UNet with the input image’s latent encoding, as in
InstructPix2Pix [9] (IP2P). The second challenge is a known
artifact attributed to Stable Diffusion’s difficulty generating
images that are not with medium brightness [13, 34]. Lin
et al. [34] propose a Zero SNR strategy that reduces color
discrepancies but requires diffusion models trained with vprediction objective, which SD1.5 is not. Beneficially, Stable
Diffusion v2.1 employs a v-prediction objective. Therefore
we replace SD1.5 with SD2.1 while maintaining our previously described learning protocol.
We call this multi-step variation with augmented SD2.1
UNet AUGUNET. AUGUNET solves the misalignment issue and reduces the color shift significantly as shown in
Figure 8, resulting in the generation of high-quality, sharp
scene intrinsics with much improved quantitative accuracy.
In Figure 9 , we show how AUGUNET performs on unseen 10242
synthetic domain while trained exclusively on
5122
real-world images. However, quantitatively, the results
still fall short of our SD Unet (single step) result, DINOv2
and Omnidatav2. In the future, we hope this problem will
be solved by improved sampling techniques and the next
generation of generative image models.
6. Conclusion
We find consistent, compelling evidence that generative models implicitly learn physical scene intrinsics, allowing tiny
LoRA adaptors to extract this information with minimal finetuning on labeled data. More powerful generative models
produce more accurate scene intrinsics, strengthening our
hypothesis that learning this information is a natural byproduct of learning to generate images well. Finally, across
various generative models and the self-supervised DINOv2,
scene intrinsics exist in their encodings resonating with fundamental “scene characteristics” as defined by Barrow and
Tenenbaum [3].
We hope that future work expands on these findings. For
example explicitly incorporating the production of scene
intrinsics into the learning process of generative image models, or developing evaluation metrics for generative models
based on physical properties.
Acknowledgments
We would like to thank Shenlong Wang for his insights on
LoRA.