JoIN: Joint GANs Inversion for Intrinsic Image Decomposition
Authors: Viraj Shah
Svetlana Lazebnik
Julien Philip
Venue: Accepted by TMLR ()

Abstract: Intrinsic Image Decomposition (IID) is a challenging inverse problem that seeks to decompose a natural image into its underlying intrinsic components such as albedo and shading. While recent image decomposition methods rely on learning-based priors on these components, they often suffer from component cross-contamination owing to joint training of priors; or from Sim-to-Real gap since the priors trained on synthetic data are kept frozen during the inference on real images. In this work, we propose to solve the intrinsic image decomposition problem using a bank of Generative Adversarial Networks (GANs) as priors where each GAN is independently trained only on a single intrinsic component, providing stronger and more disentangled priors. At the core of our approach is the idea that the latent space of a GAN is a well-suited optimization domain to solve inverse problems. Given an input image, we propose to jointly invert the latent codes of a set of GANs and combine their outputs to reproduce the input. Contrary to all existing GAN inversion methods that are limited to inverting only a single GAN, our proposed approach, JoIN, is able to jointly invert multiple GANs using only a single image as supervision while still maintaining distribution priors of each intrinsic component. We show that our approach is modular, allowing various forward imaging models, and that it can successfully decompose both synthetic and real images. Further, taking inspiration from existing GAN inversion approaches, we allow for careful fine-tuning of the generator priors during the inference on real images. This way, our method is able to achieve excellent generalization on real images even though it uses only synthetic data to train the GAN priors. We demonstrate the success of our approach through exhaustive qualitative and quantitative evaluations and ablation studies on various datasets.

Published in Transactions on Machine Learning Research (02/2025)
JoIN: Joint GANs Inversion for Intrinsic Image Decomposi- tion
Viraj Shah* UIVC
vjshah3@@llioois.ddd edu
Svetlana Lazebnik
slazebniddllknoosssccoo edu
UICC
Julien Philip
malienon.phhlip@gmaai..
Adobe Reseorch
Reviewed on OpenReview: https: // openreview. net/ forum? id= JEHLV
Abstract
Intrinsic Image Decomposition (IID) is challenging inverse problem that seeks to decom- pose natural image into its underlying intrinsic components such albedo and shading. While recent image decomposition methods rely on learning-based priors on these com- ponents, they often suffer from component cross-contaminttion owing to joint training of priors; or from Sim-to-Real gap since the priors trained on syuthetic data are kept frozen during the inference on real images. In this work, we propose to solve the intrinsic image decomposition problem using bank of Generative Adversarial Networks (GANs) as priors where each GAN is independently trained only on a single intrinsie component, providing stronger and more disentangled priors. At the core of our approach is the idea that the la- tent space of a GAN is well-suited optimization domain to solve inverse problems. Given an imput image, we propose to jointly invert the latent codes of set of GANs and combine their outputs to reproduce the input. Contrary to all existing GAN inversion hods that are limited to inverting only a single GAN, our proposed approach, JolN, able to jointly invert multiple GANs using only single image as supervision while still maintaining distri- bution priors of each intrinsic component. We show that our approach is modular, allowing various forward imaging models, and that it successfully decompose both synthetic and real images. Further, taking inspiration from existing GAN inversion approaches, allow for careful fine-tuning of the generator priors during the inference real images. This way, our method is able to achieve excellent generalization on real images even though it uses only synthetic data to train the GAN priors. We demonstrate the success our approach through exhaustive qualitative and quantitative evaluations and ablation studies on various datasets.
1 Introduction
Any natural image be lighting-dopendent components. Decomposing an image into such intrinsic components is highly useful, since it allows for varying each intrinsic component independently in order to achieve complex image editing such as image relighting, material swap, and object insertion (Bousseau et al. 2009).
intricate combination of geometry-dependeent, material-dependent,, and
In a commonly used image decomposition framework, an image is typically decomposed into a light indepen- dent component referred to as albedo, a light dependent component referred to as shading, and optionally
*This work was done during internship Adobe Research.
Project page: https://virajehas compa/inom///

---

Published in Transactions on Machine Learning Research (02/2025)
Intrinsic Components
Inverse Model
Figure 1: Intrinsic Image Decomposition (IID) framework. We consider a commonly used image de- composition framework that aims at decomposing the natural image into its light independent (albedo), light dependent (shadingg, and optionally residual (specular) components. The forward model f(-) from the intrinsic components the natural image is simply given by mul- tiplication of albedo and shading with addition of spec- ular residual (here, (-) indicates the standard tone- mapping operation). However, the inverse mapping of natural image to its intrinsic components is highly ill- posed inverse problem that we aim to solve.
Image Decomposition
Natural Image, I
f(-)
A
Lamcbrrtion model
S Specwlar mostei
Shading.C
Specular
Forward Model
1 1- ICC Cmu Ci))
= *RGBIC C. Cj + C.))
a residual component referred to as specular which captures the non-ideal behavior of natural surfaces. As depicted in Fig. [ the forward mapping f(.) from the decomposed image components to the natural image in this case is relatively straightforward to obtain. However, the inverse mapping of the natural image to its intrinsic components highly challenging ill-posod inverse problem owing to potentially infinite solution space. To obtain useful decomposition, it is required to constrain the solution space by employing priors on the image components.
Existing IID approaches leverage variety of hand-crafted or lcarned priors for recovery of different intrinsic components. However, priors from both of these categories often suffer from insufficient disentanglement and cotaaicontainnnoion of components, i.e. features from shading may appear in albedo and vice versa. This effect more pronounced in learning-based approaches that attempt to train a single model to predict multiple image components jointly (Das et al.) 2022a Yu & Smith 2021). Further, in the case of learning- based priors, collecting the training data with groundtruth albedo and shading of natural images is extremely data. Priors trained simulated
difficult, thus, they are typically trained using the data are typically kept frozen during inference on real images, thus, they lead to poor generalization on real scenes which is called Sim-to-Real gap for IID (asks. In this work, we propose to solve both the challenges by using bank of Generative Adversarial Networks (GANs) as priors where each GAN is trained independently only on a single image component, thus alleviating the challenge of cross-contamination. Moreover, even though our GANs are trained on synthetic data, we allow for careful fine-tuning of the generator priors for inference on real scenes to bridge the Sim-to-Real gap.
The idea of using GAN model as prior is explored widely for variety of inverse imaging problems such as image inpainting Yeh et al. super-resolution (Ledig et al.
2016 Bora et al.
2019). HDR imaging (Niu et al. 2020). image 2017 Gu et al.
2016 Bora et al.
2017
2019 , compressive sensing (Bora et al. 2017 Gu et al.
Hyder et al. 2019), owing to their tremendous success in mapping a lower-dinensional latent distribution to the manifold of real images. Such approaches assume that the target image lies on the range of the GAN and attempt to obtain the corresponding latent code through optimization. However, unlike the imaging problems where the solution set consists of only a single natural image, in case of IID, the solution consists of several intrinsic components. For that reason, when used for ITD task, such GAN prior is required to generate all the decomposition components from a single latent code. For example, in a previous work (Guo et al. 2020) all intrinsic components are generated simultancously by a single GAN. Such a design is susceptible to the problems of poor disentanglement and tross-contaminatiin as all components are generated by a single model and tied with a single latent code.
Shah & Hegde 2018a), and phase retricval (Hand ct al. 2018
To this end, in this work we propose to use a bank of independently trained generators as a prior: we train three separate GANs combination of the intrinsic components generated by these component-specific GANs, and find the latent codes for the albedo, shading, and specular GANs such that when input to the corresponding GAN model, they accuralely generate their respective intrinsic component, and produce the input image when combined (Fig. 2) Since each GAN is trained independently on only a single image component, each generator has no capability of generating anything else other than the image component on which is trained. Therefore, an independently trained GAN provides a much stronger constraint on the solution space as compared to the existing methods.
each for albedo, shading, and specular. We model the natural image as
2

---

Published in Transactions on Machine Learning Research (02/2025)
Train NA with synthetic for each component separataly
Step Decomposing Synthetic Imagos using JoiN KNN Loss
Step 2: Decomposing Real Images using JoiN with Generator fine-tuning
Synthelic Ribdd faces
Albeddo Cenerator
Imput Image:
abooka latere
Gis
Eatirated AlLEcU
Estimated ImmagN
Impari Image,
twin lasans
Extrrated Alledo
Est insitedd ImMAAN 2
Toatn
Can
stacing
G
Gya
Syntheticc Sracing Facas
Fnierated Shading
Shuding
Bank of pre-tralned Generatios
baokoroo
We use bank of pre-trained GANs a prior where each GAN is trained only on a single image component using synthetic data. We design the problem of decomposition as a joint GAN inversion problem on multiple GANs using the imput image Is as the only supervision. Step 1: We aim to optimize the latent codes of each GAN in way that after passing the outputs of individual GANs through forward mapping f(), the resulting image estimate î resembles the input image I". Apart from using reconstruction loss, we propose to use kNN loss on the latent codes to strongly enforce the priors learnt by each GAN. Such approach leads to successful decomposition on synthetic data. Step 2: For decomposition on real images, we bridge the Sim-to-Res gap by carefully fine-tuning the indivichul GANs in way that they can represent the real image features while still maintaining strong component-wise priors.
Figure 2: Overview of our approach. Left:
Using more than GAN a prior comes with new set of challenges. Existing approaches restrict the solution space by inverting the mapping of a pre-trained GAN, i.c. finding the latent code corresponding to the target image. Inverting the mapping of a GAN is highly challenging and well-studied problem known as GAN Inversion. In our case, we need to obtain the latent codes for all the GANs in our bank meaning inverting multiple GANs simultancously. Since all the existing GAN inversion methods are limited to inverting only a single GAN, it is not straightforward to use them to jointly invert all the GANs in our bank. To this end, we propose new Joint GAN Inversion algorithm, JoIN, that can jointly invert multiple GANs without requiring any component-wis supervision (see Fig. 2). To the best of our knowledge, JoIN is the first method to demonstrate successful inversion of multiple GANs at once.
Key to successful joint inversion is our kNN-based loss regularization that ensures the latent code estimates for each intrinsic component remains within the distribution manifold of its associated GAN. Further, we use only the synthetic data to train our GANs, and show that our approach successfully generalizes to real images by adapting generator fine-tuning technique from the existing GAN inversion literature in our pipeline: unlike the existing learning- based approaches that keep the priors frozen during the inference, we allow for careful fine-tuning of our generator priors to bridge the Sim-to-Real gap. We evaluate our approach on different datasets such as faces, materials, and indoor scenes, and provide extensive comparisons on both the synthetic and real images.
Since we train a separate GAN model for each image component, our approach remains modular with respect to the choice of forward model, í.e. we choose to add a new component (e.g. specular) to our existing forward model, we can reuse the GANs trained for existing components (albedo and shading) in our bank, and need to train only one GAN from scratch. We demonstrate such capability along with other ablation studies in our experiments.
To summarize, our key contributions are following:
We propose to use a bank of independently trained GANs as priors to tackle the IID task. Training each GAN separately for individual intrinsic components provides much stronger priors and mitigates the challenges poor disentanglement and signal gross-contamination
We propose a new Joint GAN Inversion algorithm, JoIN, that can jointly invert multiple GANs without requiring any component-win wise supervision (see Fig. step 1)). Our method decomposes an image by successfully inverting it onto albedo GAN, shading GAN, and specular GAN simultancously using only the natural image as supervision. To the best of knowledge, ours is the first method to successfully invert multiple GANs at once.
3

---

Published in Transactions on Machine Learning Research (02/2025)
We introduce novel kNN-based regularization term for joint GAN inversion allowing to better retain their distribution properties of each GAN as compared to existing regularizations.
We show that even though the bank of GANs is trained entirely on synthetic data, our approach can achieve excellent generalization on the real images by leveraging existing GAN inversion techniques such as generator fine-tuning (see Fig. Riseep 2)). Our model also offers flexibility of decomposing into cither two components (albedo, shading) or three components (albodo, shading, specular).
2 Related Work
2.1 Image Decomposition
The intrinsic decomposition of images is a long-standing problem that emerged with the retinex and lightness
theory (Land& & McCanm 1971 Horn 1974). Using a purely diffuse model, early work computed shadow-free
reflectance images of faces (Weiss] 2001 using time-lapse sequences. User-assisted methods were the first to
succossfully tackle the problem for single natural images (Tappen et al. 2003 Bousscau et al., 2009). With
the rise of deep learning a large body of work emerged using neural networks to predict intrinsic images ZZhou
et al. 2015 Nestmeyer & Gehler 2017: Li & Snavely 2018b a Yu & Smith
2019 Li et al. 2020 While
forard methods such as (Das et al. 2022b Yu & Smith 2021 bring improvements in terms of model
architecture and loss functions, they are prone to cross-contamination since they use single model for
2023) proposed decoder-sharing transformer architecture for albedo prediction, but their model is limited to only a single component unlike our approach that can provide full decomposition of an image into or 3 components. Similarly, S-Aware (Jin ct al., 2023)
predicting multiple components. Recently, CReINet. (Luo el al.
too limited to predicting only the albedo. [arraaaa & Aksoy (2023) predicts albedo and shading using two stage networks, but requires dense pseudo ground truth for generalization to real images. Liu et al. (2020) presented a method that shares some ideas with our pipeline: they build priors independently for shading and reflectance using image collections and adversarial auto-encoders. Contrary to our approach, all such leed-forward methods use a fixed forward model and would require full retraining if a component such as specular were to be added. One can refer to a recent review [Liu et al. 2024 for a summary of the research in intrinsic decomposition.
For photo collections, many works explored this problem using geometry priors (Laffont et al, 2012: 2013;
Duchêne et al. 2015; Philip al.) 2019; 2021), and recently methods were introduced working on videos (Yo et al.] 2014; Bonneel et al. 2014 Meka et al. 2016 2021). One of the main applications of IID lies in image relighting for which decomposing an image in specific components such as albedo, shading, shadows or normals is often an intermediate step both for faces (Pandey et al. 2021 Hou et al. 2021 Yeh et al., 2022) and outdoor picture (Griffiths et al. 2022). A recent work (Zhang et al. 2024) trains an end-to-end image relighting model by encoding the source scene and larget illumination inlo a latent space, and recovers the albedo maps of the scene from its latent representation. However, the quality of intrinsic components obtained by such methods is generally poor since their primary aim is relighting, while intrinsic decomposition is treated as byproduct. In the context of the more constrained problem of material acquisition, methods often aim at recovering richer information. They usually predict full SVBRDF map composed of albedo
but also normals, roughness, and specular maps (Aittala et al. 2015 Deschaintre et al., 2018 2019). Some
recent methods proposed to use adversarial networks as part of their SVBRDF estimation pipelines ZZhou
& Kalantari 2021 Zhou & Khademi Kalantari] 2022). Closer spirit to our method MaterialGAN (Guo
et al.] 2020) proposes to use single GAN trained on all the channels of SVBRDF maps and to invert a
latent code to recover an SVBRDF. Contrary to our method it requires several input images and known lighting conditions for each input.
Leveraging different lcarning- baaning-based priors for different signal components has been explored in previous works such as DoubleDIP (Gandelsman et al. 2019) and its exlensions (Ren et al. 2020 Zhou et al. 2023 Zhuang et al.) 2022) that use one untrained deep image prior (DIP) for each signal component. Their approach relies on strong internal self-similarity property, i.c., the patches within cach component have higher sclf-similarity than the patches in the combined image, while the patches across the two components are highly dissimilar. This strategy has proven effective in tasks like image dehazing, layer separation, and watermark removal,
4

---

Published in Transactions on Machine Learning Research (02/2025)
where the components often structurally dissimilar. However, the assumption of structural dissimilarity of the components may not hold for albedo and shading, since their features (shape, edges) and structure are highly similar to each other. Evidently, UIDNet (Zhang et al. 2022) demonstrated that applying Dou to IID results in severe cross-contamination and a loss of fine details. They introduce additional hand-crafted priors, and use encoder-decoder with skip connections to mitigate these issues. Using untrained DIP priors is computationally expensive with inference times of several minutes. In our case, instead of untrained DIPs, we employ multiple trained GANs that provide highly accurate priors, and by inverting them jointly, we effectively avoid the crosscoontamination, preserve details, and achieve significantly faster inference times.
2.2 GAN Priors and GAN Inversion
Inverting a GAN to obtain the latent code corresponding to an image is a key step in solving inverse problems (Shah & Hegde 2018b Owing to its importance, many GAN inversion approaches have been recently proposed as reviewed in Xia et al. (2021). We can classify these methods in three broad classes: optimization-basedd, incoder-based, and hybrid.
Optimization-based methods such as Lipton & Tripathi )b; Creswell & Bharath (2019); Zhu et al. (2016); Lipton & Tripathi (2017a); Karras et al. (2020b]: Abdal et al. [2019 2020 regress latent code using gradient descent by minimizing a loss between a target image and the generated image produced by the latent code. The key variations among these approaches are the latent space chosen for the optimization (z, W, and W+ space); the loss function(s) they use to calculate the similarity between the target and the estimate; and additional criterion used for aiding optimization. (such (such as regularization, improved initialization, and early stopping). Although these methods can produce relatively precise outcomes, their iterative optimization scheme is slower than forward methods and can converge to local minima when badly initialized.
Contrary to optimization based approaches, encoder-based methods propose to use end-to-end frame- work based in which an encoder that takes the target image as input, predicts the latent code directly (Xia
let al.
2021
Pichorskyi et al.
2020
Richardson et al.] 2021a Alalul et al. 2021 Chai et.
al.] 2021a: Lansen
let al.
2016
Zhu et al. 2016
Brock et al.] 2016 Perarnau et al. 2016 Tov et al.] 2021 Wei et al. 2021
Wang et al. 2021 Yao et al.
2022 Moon & Park 2022 Mao et al. 2022). Encoder-based methods have
faster inference times, but they suffer from lower result quality and poorer generalization. They can be used
as initializer for optimization-basee inversion. Methods that combine both optimization and encoder-based
initialization are known hybrid approach (Chai et al. 2021b Bau ct al. 2019c Zhu et al. 2016 Alaluf
et al. 2021 Bau et al.] 2019b Huh el al. 2020 Wei el al. 2021
Recently, a fourth category of approaches has emerged that advocates fine-tuning the generator weights after optimizing the latent code (Roich et al. 2021 Feng et al. 2022). Such methods produce better reconstruction since the generator is allowed to fit the larget image but fine-tuning the generator can lead to distortions in the generator manifold losing important properties. More recently, (Bhattad et al.) 2023) has shown that StyleGAN has internal reprosentation of intrinsic images that can be accessed by finding an appropriate offset to the latent codes. However, this method is not applicable for real images, since the offsets obtained are not compatible with the inverted latent codes of the real images.
In recent works, a new category of generative models known as diffusion models (Dhariwal & Nichol 2021 are becoming increasingly popular in many application domains owing to their superior performance in generating high-quality and diverse images. Several works (Feng et al. 2023; Li et al. 2023 leverage
diffusion models as priors in solving inverse problems such reconstruction and super-resolution. Similar in spirit to our method, the concurrent work Wang et al. (2024) has explored using independently trained diffusion models for blind image deblurring, where a blurry image is decomposed into clean image, blur kernel, and a tilt map. Note that unlike the IID, here the target components are structurally incoherent and dissimilar, e.g. the blur kernel is typically much lower dimensional and exhibit distinctly different features than the natural image. Moreover, Diffusion models are computationally more expensive than GANs during both training and inference, and does not allow for smooth transitions in latent space (that can be leveraged for image relighting in our case). While we acknowledge leveraging multiple diffusion priors for the IID task is a significant. direction for future research, especially to model larger and more diverse datasets, our
5

---

Published in Transactions on Machine Learning Research (02/2025)
work demonstrates that GANs offer powerful and efficient tool for IID, particularly when computational resources and inference speed are important considerations.
3 Our Approach
In this work, we propose to solve intrinsic image decomposition using an optimization formulation that leverages a bank of GANs as priors. We use the fact that cach image component distribution can be closcly approximated using GAN model, and creale a bank of GANs by training a separale GAN model for each image component that we desire to recover (See Fig. []eft)). Since each GAN maps a lower dimensional latent space to an image component space, solving the inverse problem aims to recover the latent codes corresponding to each image component which can be formulated the joint inversion of multiple GANs shown in Fig. step 1).
In this section, we first present our optimization framework and its constraints, then we introduce novel loss that helps further regularize the optimization and maintain the GANs priors.
3.1 JoIN: Joint Inversion of Multiple GANs
Given an image 1 and a forward differentiable function S taking as input n image components Cf. we suppose that I* can be modeled as:
1 -(C
(1)
Where we assune ʃ is known. We aim at recovering Cf... given only 1* as input. Instead of directly optimizing C1... to recover Cf.... in the image domain we propose to represent them as the output of a GAN G; for a latent code Wy:
C= (G.
(2)
Where T:11 is an inverse color transform operator.
For G; we follow the architecture of StyleGAN2 (Krrras et al. 2020a), while w, refers to an intermediate latent code in W space. Note that the StyleGAN based architecture first maps Gaussian distribution (Z) an intermediate latent space W using non-linear mapping network, and W-space is then mapped to distribution of real images. W-space is known to provide higher degree of disentanglement and smoother control over various image features (Karras et. al. 2019b), thus has emerged more suitable choice for performing GAN inversion (Xia et al. 2021). We follow the same and perform the joint inversion in W-space, Our estimate of / is thus given by:
baaseedd
w,) (TT (G1 (w.)))....T (Gn Our goal is thus to find the wr e 1...n n that minimize a reconstruction loss function L:
(w,)))
(3)
arg mig min w...l....
L (I* I(w)....,..))
(4)
Eq. is analogous to the standard GAN inversion formulation, except for the fact that aims to invert n Generators simultancously instead of one. The loss in Eq. 4 is minimized over all the latent codes wys simultaneously, and the final estimates 0ss can be passed to Eq. to obtain the image components Cf..... We use E-LPIPS loss (Kettunen al.) 2019) (robust version of LPIPS (Zhang al.) 2018) perceptual loss)
the loss function L unless otherwise specified.
3.2 Intrinsic Image Decomposition
In this paper, we focus on the problem intrinsic image decomposition, covering several formulations (Garces et al. 2021). We thus adapt the general formulation presented in the previous section.

---

Published in Transactions on Machine Learning Research (02/2025)
We first consider simple forward model assuming Lambertian surfaces (Barrow & Tenenbaum seminal model uses two components albedo and shading to represent the image:
1978 This
1 s RGB RGB((aaleed Calbedo Cshading)-
(5)
Then we consider a non-bambetnan model with separate specular component:
SRaleCaabedo Cehading + C Eepecuter).
(6)
This model merges the specular color and specular illumination of the dichromatic model Tominaga as single component.
(9994)
3.3
Optimization- based Joint Inversion
Given forward model described in Eq. Eq. 6 first train GAN for each intrinsic component (albedo, shading and eventually specular images) independently using the StyleGAN2 architecture (Karras et al. 2019b 2020b). As the GANs are trained independently the forward model can be chosen after the fact and components can be replacod or added.
Given the pre-trained GANs and an input 1²; we then optimize latent codes wi by minimizing the loss function in Eq. 4 over wys using gradient descent. Note that our optimization is only guided by the input image I* since the loss in Eq. 4 is calculated the combination of all the estimated components.
3.4 kNN Loss
Allowing the latent codes w; explore the latent space freely during optimization can lead to cross- contamination as they deviate from the manifold of possible within W. In such case the generator for component O₂ starts producing signals corresponding another component Cj. We thus propose new kNN-based regularization loss to constrain the latent codes to stay within domain.
To this end, we take inspiration from previous works ZZu et al. 2020: Tov et al. 2021 which introduces an in-domain loss on the latent codes as follows:
Cin
W w|12,
where w is the average obtained by drawing large number of random samples (typically 100k) of w-codes from W space. This in-domain loss tries to keep the optimized latent close to the distribution mean order keep the estimate within the train- ing domain. We argue that this formulation has two main drawbacks. First, using the distribution mcan penalizes the solution space near the boundary of the distribution.
20 Projection of W-space of pre-trained bieedo GAN
Projectod Imorston
D5
< dg
Second, it assumes that the distribution of w is isotropic, which has no reason to hold given the use of a non-linear mapping network that maps the Gaussian distribution Z with the intermediate la- tent distribution W during training of StyleGAN model. We depict such limitation of in-domain loss in Fig. where we first plot the distribution of w-codes by projecting W-space in two dimensions using t-SNE (van der Maaten & Hinton 2008), and validate the non-isometric bohavior of W W-space. Since the distribution varies differently along dif- ferent. directions, one can that simply using the
Projected dimension
Figure 3: W-space of pre-trained GAN is non- isometric. The t-SNE 2D projection of W-space of pre- trained albedo GAN confirms its non-isometric behavior as the distribution varies differently across the dimensions. Naively using the distance from the mean loss penalizes point B more than point A, even though B is well within the distribution unlike A. As remedy, our proposed kNN loss uses the closeness of point to its neighbors as a loss in- stead of relying on its distance from the distribution mean, promoting exploration of the entire space.
7

---

Published in Transactions on Machine Learning Research (02/2025)
In-oomain Lows
Lose (nurs)
Wappre
Figure 4: Comparison: in-domain vs. kNN loss. Left: In-domain loss altracts the estimate lowards the center resulting in higher loss near the distribution boundary. Right: kNN loss (k = 5 NN) attracts the estimate towards the nearby latents, allowing to hetter capture the diversity of the distribution while still keeping the estimate within the distribution. Dots represent randomly sampled latent vectors (=100) and the colors map the loss value.
distance from the mean w as a loss (as done in case of in-domain loss) would penalize point B more than point since dg > da, even though point B is well within the distribution while A lics outside. Note that we use randomily sampled 100k w-codes to calculate the I-SNE projection and the distribution mean w, while the plot shows randomly selected 1000 of them for legible visualization.
As remedy to both the issues, propose novel k-Ncarest Neighbors-based loss formulation that better allows the exploration of the full domain while still ensuring the estimate remains in-distribution. Instead of pushing the estimate w; towards the mean, we steer it closer to its neighboring remains within the distribution. In non-isometric distributions like the one in Fig. 3 proximity to other samples is a more reliable indicator of a point belonging to the distribution than its distance from the mean. This approach allows the estimate to move freely within the distribution, rather than being constrained to the mean, and avoids unfairly penalizing estimates near the boundarics. We first sample a large number (100, 000) of codes from a truncated Gaussian distribution, and pass them through the mapping network of StyleGAN2 to obtain set Sw of w codes W. Then, during inversion, at each step, we obtain the k nearest neighbors of the current latent code estimate (N:(w;; from Sw, and use softmax weighted average of the distance to nearest neighbors as loss:
codes to ensure it
0.5 Iwy
CANN
Σ
|ws w
(8)
Dj
whEkN
where D; is the normalizing factor, and d is the mean distance to nearest neighbors:
-0.5 101-wh1
D. Σ
Σ
|w, w/ll2-
(9)
N
W.EkNN((..)
Figure 4 provides an intuitive visualization of our kNN loss in 2D. Our loss landscape does not penalize as much the exploration of the distribution and its boundaries. Further, it respects the non-isometric behavior of the W-space. The soft-max weight ensures spatial continuity and a higher contribution for neighbors closer to the optimized latent (Eq. 8)- To normalize the weights,
the mean distance to the nearest neighbors (d%, Eq. 9p. We provide the results of our optimization scheme and ablation studies demonstrating the benefits of our KNN loss in Sec. Note that we use = 50 for all our experiments.
4 Adapting Existing Inversion Methods
In this section we show how existing initialization and generator fine-tuning mechanisms can be adaptod JoIN to improve the inversion result and bridge the Sim-to- Real gap.
4.1
recoode-guidded Initialization
While our optimization-oannee based approach produces faithful decompositions on synthotic data, the optimization get stuck in local minima if the initial latent. codes lie far from the target codes. To make our approach
8

---

Published in Transactions on Machine Learning Research (02/2025)
more robust, propose to use encoder-guided initialization inspired by hybrid approaches of GAN 2019c Wei et al. 2021). For each component, we can independently train pSp network [Richardson et al., 2021a to encode natural image into latent code w used by the
inversion (Chai et al. 2021b Bau et al.
pre-trained generator to recover the given component. At test time, given a natural image, the latent code produced by the encoder can be used as an initialization for the optimization.
4.2 Generator Fine-tuning using PTI
Since our component-wise GAN models are trained using only synthetic data, they are unable to generate the features corresponding to the real images. Thus, our method poses the challenge of Sim-to-Res te-Real gap, iee. various features of the real images are not captured faithfully in the decomposition. As a remedy, we propose carefully fine-tune the generators in such way that they can generate decompositions of real images while still maintaining their original priors. Indeed, approaches leveraging generator fine-tuning have become popular (Roich et al. 2021; Feng et al. 2022) where first w-code is estimated using an off-the- shelf inversion technique, and then with the fixed w pivot, the generator is updated to obtain close to zero reconstruction errors. Such approach is also known as Pivolal Tuning for Inversion (PTI) in the literature.
In our case, the generator fine-tuning (PTI) can be defined as:
mrg & (T+, (TT (wT) ..... TT (G., (wr))) EEl..
(10)
where 6G represents trainable parameters of the generator G; that corresponds to the component C,, and üis are output of the optimization.
While direct fine-tuning allows the generator to overfit to the target image perfectly, it distorts the GAN manifold. In our case, allowing multiple GANs to fine-tune at once can result in loss of the priors and much greater cross-contamination (see Fig. 11b w/o D Loss) since we do not have any component-wise supervision for individual generators. Here, we leverage interesting fact about the presence of strong priors in GAN: since the discrininator of the GAN is trained to discriminate between real and fake samples, its knowledge about the realness of sample can be employed to ensure the prior remains preserved during generator fine-tuning. In practice, one can use the discriminator loss to regularize the finc-tuning (Feng et al. 2022). While fine-tuning the generator around the W; pivot, we apply a new localized discrininator loss LLD, refer to it as D Loss.
Specifically, we sample na number of anchor latent codes in the close vicinity of our fixed w-codes by interpolating wH with randomly sampled codes wg. At each fine-tuning step, we pass the anchor codes through the generator and compute loss on the resulting images, keeping the discriminator frozen:
na LLD ΣΣ
D,(G, B)w; Bwx)).
(11)
where was are random latent codes, G, and D; are the generator and the discriminator for the component C, and 3 determines the extent of interpolation, controlling the localization of our loss. Our fine-tuning formulation thus becomes:
arg min LG ALDLLD. 90, sEl...n
(12)
where Lo is the generator loss as defined in Fq. 10 As discussed in the ablation study (Sec. 5.4 the LLD loss ensures that images with latents close to the pivot maintain realism, preserving the generator's priors during the optimization even as multiple generators are finc-tuned togother.
5 Experiments
In this section, we present a set of experiments that showcase the efficacy of our method. We provide intrinsic image decomposition results two different datasets: materials, and faces. We also provide results the
9

---

Published in Transactions on Machine Learning Research (02/2025)
Input
Desch.18
Zhou21
Zhou22 Albedo
Ours Albedo
Specular
Ours
Target
Estimated
Target
Estimated
Target
Estimated
Image
Specular Albedo Shading
(a) Decomposition synthetic materials.
(b) Comparisons on
Figure 5: (a) We show three examples of decomposition synthetic materials. For each case the first column, from top boltom, contains larget image (used for optimization), groundtruth shading albedo while second column contains our estimates. We are able to correctly separate the highlights and to discriminate between darker shading and darker albedo. (b) Our method generalizes real materials well our GAN priors allow us to better separate the highlight from the albedo leading to fewer visual artifacts. Our recovered specular is shown the right.
dataset of rendered shapes and indoor scenes in appendix. Note that for all our experiments, synthetic data for training our GANs while showing results both synthetic and real images along with comparisons with existing methods and ablation studics. We plan to release our code in future.
only
5.1 Preliminaries
For all our experiments we use StyleGAN2 Karras et al. 2020b to train the generators and pSp (Richard- son et al. 2021b) encoder for initialization. In appendix (Sec. B we also provide additional decomposition
experiments using StyleGAN-XL (Sauer et al. 2022 to depict generalizability of our model on complex im- age distribution indoor scenes. All our generators and encoders are trained using their publicly available codebases and with the standard choices of the hyperparameters.
Our optimization-basel inversion method remains the same for all the datasets with learning rate of 0.1, kNN loss weight of 0.0001, and & = 50. We rum the optimization for 1000 steps.
The decomposition results are evaluated using several metrics such as MSE, LPIPS, and PSNR for the synthetic images that provide ground truth. We also provide comparisons with several stale-of-the-art methods that are used the literature for intrinsic image decomposition. The results demonstrate the effectiveness of our proposed method in decomposing the input images into their intrinsic components and the ability to handle real images.
5.2 Experiments on Materials Data
The Materials dataset is composed of highly realistic renderings of synthetic material tiles done in De- schaintre et al. (2018). We obtain the albodo map directly from the material and compose the shading and specular components from blender direct and indirect rendering passes. More details on this dataset are given in the appendix (Sec. C For this dataset, we show results for decomposition into components (albedo, shading, and specular; Eq. Gb and operate the resolution of 512 512. For synthetic test images, we show results in Fig. 5a and quantitative comparisons to several recent SVBRDF estimation methods (Deschaintre
et al.
2018 Zhou & Kalantari 2021 Zhou & Khademi Kalantari
2022 the task of albedo recovery in
10

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Quantitative compariaons on Lumos
Quantitative comparisons: We obtain the recon- struction for albedo estimation across the test- set of 100 images for both the synthetic materials (in Tab. I and synthetic faces (in Tab. 2 datasets, and provide comparisons with existing decomposition ap- proaches using various metrics.
synthetic faces testset
On synthetlc faces
Estimated Alhedo
LPIPS!
MSEI
PSNR
Total Relighting
0.2220
0.0644
17.1470
PieNet.
0.1843
0.0501
19.4761
InversoRenderNet+4
0.3794
0.1550
14.5227
Table 1: Quantitative comparisons on synthetic terials testset.
NextFace Latent Intrinsics
0.2034
0.0791
19.0699
0.1939
0.1053
15.7492
On synthetic. materials
Estimated Albedo
Ours:
Optimization
0.1311
0.0417
22.4763
MSEI
LPIPS!
PSNR
KNN loss
0.1047
0.0357
23.5331
Desch18
0.021
0.314
19.53
Zhou21
0.020
20.92
+ Eacoder
0.0887
0.0345
24.14677
Zhou22
0.021
0.296
20.18
+ PTI w/o D Loss
0.0784
0.0155
24.1160
Ours (Opt. + KNN loss)
0.014
0.326
21.58
1 PTI w/ D Loss
0.0763
0.0072
27.3401
Target
Estimated
Target
Estimated
Target
Estimated
Target
Estimated
Target
Estimated
Target
Estimated
Image
Image
Specular Albedoo Shading
Albedo Shading
Decomposition results for Lumos faces dataset. On synthetic data, our estimated components are close to the ground truth, generator tuming is not needed when testing data is close to the GAN training distribution. Results of our method for decomposing the image into two components: albedo and shading. Right: Using three components allows to recover subtle specular effects.
Figure 6:
Left:
Tab. 1 We also show competitive results on albedo map recovery both quantitatively and qualitatively on real images, taken from (Deschaintre et al. 2019), visible in Fig. 5b Our GAN priors allow to correctly separate flash highlights from the diffuse component and to recover plausible content under the highlights in the albedo map while correcting for the lighting fall-off. Additional results are in appendix (Sec. E.
5.3 Experiments on Faces Data
Faces are notoriously adapted to the usage of GANs and cater to such use case, we use the synthetic Lumos dataset (Yeh et al. 2022) to train GANs, we defer more discussions the dataset to the appendix.
First, we show results on test images from the dataset in Fig. G left) and Fig. t rright) where we use respec- tively two and three components. For this experiment, we use the encoder initialization and our optimization without generator fne-tuning (PTT). As can be seen in these figures our method is able to recover accurate albedo, shading, and specular maps and does not suffer from cross-channel contamination.
As the synthetic dataset does not reproduce the complexity and diversity of real faces, for real images, we add the generator fine-tuning (w/ local D loss) step presented in Sec. 4.2 We show results real method to five recent relighting methods, Total Relighting [Pandey et al. 2021). Lumos YYhh ct al. 2022), PieNel Das et al.] 2022a), Next (Dib et al.) 2021a 2022). and Latent Intrinsics (Zhang el al. 2024)
faces taken from the FFHQ dataset in Fig. 7a and compare InverseRenderNett-t (Yu & Smith 2021)
that seek to recover face albedo as part of their relighting pipeline, as shown in Fig. [b In all the cases, the predicted components by our method maintain the priors of each GAN. We can obscrve that our method
works better at producing shading-free albedo images that exhibit fewer shading variations around the neck, nose, nostrils, and mouth.
11

---

Published in Transactions on Machine Learning Research (02/2023)
TR Albedo
IR++ Alhedo
PieNet Albedo
Nextface Alhedo
Ours Alhedo
Ours Shading
Input
Albedo
Input
Estimated Shading
Albedo
Specular
(a) Decomposition results on real faces.
(b) Comparisons on real face Images.
Figure 7: (a) Using generator fine-tuning, method generalizes to real faces and correctly separates albedo, shading, and specular effects while reconstructing the input correctly. (b) On real faces, our approach recovers more faithful albedo by removing shadows, and preserving the appropriate skin tones compared competing methods.
Targel
Estimated
Relight
Relight
Relight 3
Table 3: Effectiveness of kNN Loss
in inverting single GAN
On CelebA-IIQ
Inversion on single GAN trained PFHQ
test
LPIPS,
MSE
PSNR
Figure 8: Face relighting results. Once the decomposition a real face is obtained, modify the shading component using GAN editing technique Shen & Zhou (2021 on the shading geuerator obtaining plausible relighting edits.
Optimization
0.384
0.095
16.908
Opt. in-domain loss
0.094
16,990
Opt. KNN loss (ours)
0.379
0.092
17. 120
For quantitative comparisons, struction errors for albedo recovery for all variants of our approach with four competing methods in Tab. 2 We see that our method achieves the lowest errors. Additional results are provided in appendix (Sec.
testset of synthetic images from Lumos dataset, and compare recon-
Application to relighting. GANs are known for their rich image manipulation capabilities. Our modular approach allows leveraging such capabilities modify each image component separately using GAN's latent space editing, making it suitable for image modifications such as relighting. We show real image relighting results in Fig.8 To generate these results, after obtaining the inversions for albedo, shading, and specular, we use off-the-shelf latent space editing method SeFa (Shen & Zhou, 2021) on the shading generator to modify the shading image while keeping the other two components fixed. StyleGAN's disentangled representation helps with identity preservation. As seen in Fig. our method produces varied novel lighting.
5.4 Ablation Studies
Advantage of independently trained GANs. Training GANs independently makes our pipeline modular and flexible: different components can be trained with different data sources, and new components can be easily introduced in the forward model without re-training the entire pipeline. We showcase such use case by adding the specular component to the albedo-shading forward model (Fig.
5a.
5b).
We also show that casier train and invert GANs independently, further preventing cros--contamination during inversion. To demonstrate this, train a single multi-component GAN that learns to generate both
12

---

Published in Transactions on Machine Learning Research (02/2025)
Table 4: Quantitative comparisons for different variations of our approach on synthetic Faces.
On syuthetic Faces (Lumos dataset)
Albedo
Shading
Specular
Image
LPIPS.
MSEI
PSNR
LPIPS)
MSE
PSNR
LPIPSI
MSEI PSNRI LPIPSI
MSEI
PSNR
Inversion w/ single GAN for both albodo and shading
0.1954
0.1269
14.166
0.2657
0.07021
18.245
0.09342
0.03133
24.466
Optinization
0.1311
0.0117
22.476
0.2832
0.0518
18.863
0.2584
0.0336
32.320
0.0879
0.0308
24.296
Opt. io-domaia loss
0.1209
0.0400
22.589
0.2738
0.0497
19.159
0.2444
0.0332
32,853
0.0876
0,0302
16116
Opt.
0.0357
0,2603
0.0487
20.911
0.2961
0.0868
Encoder + Opt. KNN lose
0.0867
0.0345
24.146
0.2039
0.0414
21.032
0.1722
0.0282
33.409
0.0713
0,0306
26.445
Encuder Opt.
KNN loss PT w/o Loss
0.0784
0.0155
24.116
0.2776
0.0319
18.163
0.1550
0.0278
32,838
0.0136
0.0011
33.646
Encader Opt. + XNN loss + PTI w/ D Loss
0.0765
0.0072
27.340
0.1703
0.0285
22.475
0.0774
0.0251
34.547
0.0194
0.0022
32.517
albedo and shading images simultaneously by producing 6-channel output: 3 channels each for albedo and shading. We increase the number of trainable paramcters by 2x for fair comparison with our method.
First, as shown in Fig. 9a such joint training leads to inferior image generation compared to independently- trained albedo and shading GANs. With a single GAN, albedo and shading are leaking into each other, this is also measured by higher FID scores. Second, having the same network generating both albedo and shading means that the information is entangled up the last layer, leading to cross-oontamination during inversion shown in Fig. 9b Our obsorvation is also supported by the quantitative results provided in Tab.
FID: FID:30.18
FID: 11.89
Shading
Using a
Using multipic GANs
Ours; Laing
Target
Single
Target
Single GAN muttiple GANs
FID: 24,98
FID:12.71
Shuding
Albedo
Albedoo
(a) Samples from the training set
(b) Sumples from single GAN trained jointly
(c) Samples from separately trained GANss
(a) Single GAN separated GANs prior.
(b) GAN inversion on single vs separated GAN priors.
Figure 9: (a) Training separate GAN for each component results in better generation quality opposed training single GAN for learning both components. (b) Inversion on single GAN trained to generate both albedo and shading leads to cross- crosscontamination issues (note the color leakage in shading and shadow effects in albedo). We mitigate this issue by training independent GANs for each component.
Comparisons with DoubleDIP. We provide qualitative comparisons with doubleDIP (Gandelsman et al. 2019) in Fig. 10a As shown, untrained priors-based doubleDIP results in severe cross-contaminatioon and loss of details on both the synthetic and real datasets as the underlying assumption of dissimilarity across the components does not hold. Our method produce accurate decompositions in both the cases.
Effectiveness of kNN loss. To evaluate the effectiveness of our newly proposed kNN-loss, first provide experiments for the more general single GAN inversion sctup: we use StyleGAN2 model trained on natural face images from FFHQ dataset (Karras et al. 2019a),, and invert CelebA-HQ testset (Karras et al.) 2018 images using optimization- based inversion (Karras et al. 2020b). As shown in Tab. 3 using kNN loss as regularization improves the performance over direct optimization and optimization with in-domain loss. Further, we verify the effectiveness of kNN loss for IID task using joint inversion on synthetic faces. As shown in Fig. 10b. kNN loss prevents cross contamination which is prevalent when using in-domain loss
13

---

Published in Transactions on Machine Learning Research (02/2025)
with with KNN
with
with KNN
Results on Synthetic
Results on Real Data
Target
domain loss loss (ours)
Target
domain loss
loss (ours)
Immage
Image
Shading
Shading
Albedo
Albedo
Target
DoubleDIP
Ours
DoubleDIP
Ours
(a)
Comparisons
DoubleDIP.
(b) Effectivencss kNN in joint inversion.
Figure 10: Ablation Results for (a) comparisons with DoubleDIP, and (b) effectiveness of kNN loss in joint inversion.
(notice glasses and shadows appearing in albedo), and improves the detail preservation (note mouth, beard, and hairstyle). Such improvements are supported by superior quantitative performance in Tab, 4
Full method ablations. Finally, we present comprehensive set of ablations for our joint inversion of multiple GANs. We provide qualitative results (Fig. 11a and quantitative evaluations for intrinsic image decomposition on the Lumos Faces dataset Tab. 4 second part. We first evaluate our method using only direct Optimization (Opt.). We then add the in-domain loss discussed in Soc. 3.4 and then replace with our kNN loss. As be seen in Tab. the kNN Loss allows better recovery of the intrinsic components by maintaining the latent codes within domain, also outperforms the in-domain loss.
We then add the encoder initialization from Sec. 4.1 to the pipeline and finally add the generator fine-tuning (PTT), without, then with, the D Loss as discussed in Sec. 4.2 Our evaluation shows that the encoder initialization brings clear improvement both reconstruction quality and component estimations. The PTI without D Loss does improve the reconstruction of the imput image but performs poorly in component estimation, whereas, with the Loss, the PTI provides the best results for the decomposition while still improving the image reconstruction quality. A qualitative visualization of this difference is visible in Fig. 11b
Target
Ojc
Opc
EEncoder
2111
Jecua Ow/ losa)
Image
Tareri
Estinoted Trage
Estrosed Shuding
Estrared Albedo
Eatinoted Specili
Shading
Ours (Enc. + Opt KNN + PI wio Loss)
Specular Albedo
Tarer
Iroge
Spece
Ours (Eanc Opt. LNN PTI Loss)
(a) Full method ablations on synthetic
(b) Ablation importance
Figure 1I: (a) Our decomposition improves we (from left to right) replace in-domain loss with kNN loss; initialize with and encoder; fine-lune generators using PTI; and use local loss during PTI fine-tuning step. (b) Local loss preserves the generator priors by preventing the local manifold from distortions. Above show IID on real face image without (top) and with (bottom) local discriminator loss.
5.5 Storage, Memory, and Computational Costs
Runtime. Our approach adapts efficient GAN inversion techniques (w-optimization, encoder-basod ini- tialization, and Pivotal Tuning Inversion (PTI)) for joint GAN inversion, with our novel kNN and local discriminator losses adding minimal overhead (* 15% and 10% to the optimization and PTT runtimes.
14

---

Published in Transactions on Machine Learning Research (02/2025)
respectively). This is achieved through GPU-accelerated faiss library (Johnson et al. 2019 for nearest neighbor search, which takes < 2ms to compute nearest neighbors each optimization step (for k 50). On a single NVIDIA A40, JoIN takes 190 seconds for inference on a single image (84s for optimization, 1s for encodor, 105s for PTT). While slower than food-forward methods like InverscRenderNet+++ (2s) and PicNet (Is), JoIN is considerably faster than oplinization-based methods like NextFacee (320s), and multiple priors methods such as DoubleDIP (878s), UTDNet (950s) and diffusion-based DMPlug (635s). Crucially, JoIN is fully amortizable, enabling concurrent processing of images on single NVIDIA A40, reducing per-image inference time to 24 seconds (11s for optimization, < is for encoder, 13s for PTI).
Storage and Memory. JoIN requires > 800MB for model storage, lower than PieNet (2200MB), NextFace (1300MB), and InverseRenderreet++ (1200MB). GPU memory usage during inference for our method is 4GB, which is on par with other competing methods.
In summary, JoIN strikes a favorable balance between accuracy and efficiency. While feod-forward methods like InverseRenderNel++ and PieNet offer faster inference, they lack the flexibility to adapt to different. for- ward models without retraining, a key advantage of our method. Compared to optimization- based methods like NextFace and prior-based methods like UIDNet, JoLN achieves significant runtime reductions, particu- larly considering the amortization.
6 Limitations and Future Work
In this paper, we introduced JolN a new method to solve inverse problem of image decomposition using a bank of GANs as priors. We show that it is possible to retain GANs distribution priors while jointly inverting several GANs at once. Nonetheless, our method inherits some drawbacks of GAN inversion methods. First, it slower than feed-forward methods with runtimes of a few seconds (detailed discussion is appendix Sec. 5.5). Second, currently, training GANs on unaligned distributions remains complex task and our method can be limited by the GAN training quality, hence it is more adapted to distributions that GANs can model well such as face images (we demonstrate generalizabiliy of our method to newer GAN models and complex indoor scenes in appendix Sec. B Our framework of using bank of GANs as prior is not limited to only the problem of IID, and can be extended to similar inverse problems in other domains of engineering such as audio source separation as a part of future work. An interesting avenue for future work is exploring the replacement of GAN priors with diffusion models, which have shown impressive results various image generation and inverse problems, to investigate their potential for improving the accuracy and robustness of intrinsic decomposition. Particularly for larger and more diverse datasets, ability of diffussin models to capture complex data distributions may prove advantageous. As new techniques emerge, we believe our method will be applicable to broader datasets and could be tested on wider variety of problems.
References
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 4432 4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegant++: How to edit the embedded images? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8296 8305, 2020.
Miia Aittala, Tim Weyrich, and Jaakko Lehtinen. Two-shot svbrdf capture for stationary materials. ACM Trans. Graph., 34(4), jul 2015. ISSN 0730-0301. doi: 10.1145/2766967. URL htttp://do/oo 2766967
Yuval Alaluf, Or Patashnik, and Daniel Or. Restyle: A residual-basee stylegan encoder via iterative refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6711- 6720, 2021.
Harry G. Barrow and Jay M. Tenenbaum. Recovering intrinsic scene characteristics from images. 1978. URL https://apis semanticacholar wtw.aaattcccccllo..ooooooropaaaa org/CorppusID:14822077
15

---

Published in Transactions on Machine Learning Research (02/2025)
David Bau, Hendrik Strobelt, William S. Peebles, Jonas Wulff, Bolei Zhou, Jun- Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. ACM Transaclions on Graphics (TOG), 38:1 11, 2019a.
David Bau, Jun-Yan Zhu, Jonas Wulff, William Peobles, Hendrik Strobelt, Bolei Zhou, and Antonio Tor- ralba. Seeing what a gan cannot generate. In Proceedings of the IEEE/CVP International Conference on 2019b.
Computer Vision, pp.
David Bau, Jun-Yan Zhu, Jonas Wulff, William S. Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what a gan cannot generate. ICCV, pp. 4501-4510, 2019c.
Anand Bhattad, Daniel McKee, Derek Hoiem, and David Alexander Forsyth. Stylegan knows normal, depth. albedo, and more. Arxiv, abs/2306.009887, 2023. URL https://api www.smeimmaaanolaa...om.mm oet.aatttccccoooaoooooooorrrllla 258999598
Blender Online Commuuntty. Blender- a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam, 2023. URL httpp://wwt http:://wwwwwwwe.......b
Nicolas Bonneel, Kalyan Sunkavalli, James Tompkin, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. Interactive intrinsic video editing, ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2014), 33(6), 2014.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G. Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, 2017.
Adrien Bousseau, Sylvain Paris, and Frédo Durand. User Assisted Intrinsic Images. ACM Transactions on Graphics, 2855):33::-110, December 2009. doi:
URL httl:httts://aa inria.fr/
1n4ia-0041388
Andrew Brock, Thcodore Lim, James M Ritchic, and Nick Weston. Neural photo editing with introspective adversarial networks. arx preprint or orXi::1609. 07093, 2016.
Chris Careaga and Yağz Aksoy. Intrinsic image decomposition via ordinal shading. ACM Trans. Graph., 43 (1), 2023.
Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage composi- tionality in gans. JCLR, 2021a.
Lucy Chai, Jun- Yan Zhu, Eli Shechtman, Phillip Isola, and Richard Zhang, Ensembling with deep generative views. In CVPR, 2021b.
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE Transactions on Neural Networks and Learning Systems, 30:1967-1974, 2019.
Partha Das, Sezer Karaoglu, and Theo Gevers. Pie-net: Photometric invariant edge guided network for in- trinsic image decomposition. In IEEE Conference on Computer Vision and Pottern Recognition, (CVPR), 2022a.
Partha Das, Sezer Karaoglu, Arjan Gisenij, and Theo Gevers. Signet: Intrinsic image decomposition by a semantic and invariant gradient driven network for indoor scenes. In European Conference of Computer Vision Workshop CV4 CVlMelaverse 2022 (ECCV), 2022b.
Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. Single-image svbrdf' capture with rendering-aware deep network. ACM Trans. Groph., 37(4), jul 2018. ISSN 0730-0301. doi: 10.1145/3197511:3200122 URL
3201378
Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousscau. Flexible svbrdf capture with a multi-image deep network. Computer Graphics Forum, 38(4):1-13, 2019. doi: https:
//dbi.org/10.111//gg...33 URL httppp///oalaallbbeaa...eeeeeaa/
13765
16

---

Published in Transactions on Machine Learning Research (02/2025)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 2021.
Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, Cédric Thébault, Philippe Gosselin, Marco Romeo, and Louis Chevallier. Practical face reconstruction via differentiable ray tracing. In Computer Graphics Forum, volume 40, pp. 164. Wiley Online Library, 2021a.
Abdallah Dib, Cedric Thebault, Junghyun Ahn, Philippe-Hent Gosselin, Christian Theobalt, and Louis Chevallier. Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing. In Proceedings of the IEEE/CVI International Conference on Computer Vision, pp. 12819 12829, 2021b.
Abdallah Dib, Junghyun Ahn, Cedric Thebault, Philippe-Henri Gosselin, and Louis Chevallier. S2f2: Self- supervised high fidelity face reconstruction from monocular image. ar preprint orXiv2203300739, 2022.
Sylvain Duchêne, Clement Riant, Gaurav Chaurasia, Jorge Lopez-Moreno, Pierre-Yver Laffont, Stefan Popov, Adrien Bousseau, and George Drettakis. Multi-view intrinsic images of outdoors scenes with application to relighting. ACM Transactions Graphics (TOG), 34(5), November 2015.
Berthy T Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L Bouman, and William T Freeman. Score-based diffusion models as principled priors for inverse imaging. In Proceedlings of the IEEE/CVF International Conference on Computer Vision, pp. 10520-10533 2023.
Qianli Feng, Viraj Shah, Raghudeep Gadde, Pietro Perona, and Aleix Martinez. Near perfect gan inversion. ArXiv, abs/2202.11833 2022.
Yossi Gandelsman, Assaf Shocher, and Michal Trani. 'double-dip': Unsupervised image decomposition via coupled iaa-iimaag-pooorss. 6 2019.
Elena Garces, Carlos Rodriguez-Pardo, Dan Casas, and Jorge López-Morcno. survey on intrinsic images: Delving deep into lambert and beyond. International Journal of Computer Vision, 130:836 868, 2021.
URL https://api. semanticscholar. hotpr//ccccoaaaoooororrpppttoos. 244920698
David Griffiths, Tobias Ritschel, and Julien Philip. Outcast: Single image relighting with cast. shadows.
Computer Graphics Forum, 43, 2022.
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
2019.
Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. Materialgan: Reflectance capture using generative svbrdf model. ACM Thans. Graph., 39(6), nov 2020. ISSN 0730-0301. doi: 00.1145/34144955341777 URL https://doi org/10. /ttp///4446885.4444.7...
Paul Hand, Oscar Leong, and Vladislav Voroninski. Phase retrieval under generative prior. ArXiv, Abs/1187.04220 2018.
Berthold K.P. Horn. Determining lightness from an image. Computer Graphics and Image Processing, 3 (4):277-299, 1974. ISSN 0146-664X. doi: https://doi.org// //ww. www.sciencedirect. http://cww.....eeeeeee//aaaaaaaa
0011606666666/7490022222 URL https:
Androw Hou, Ze Zhang, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. Towards high fidelity face relighting with realistic shadows. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14719-14725 June 2021.
Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and projecting images into cass-conditional generative networks. In European Conference on Computer Vision, pp. 17 34. Springer, 2020.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Rakib Hyder, Viraj Shah, Chinmay Hegde, and M. Salman Asif. Alternating phase projected gradient descent with generative priors for solving compressive phase retrieval. ICASSP 2019 2019 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7705 7709, 2019.
Yeying Jin, Ruoteng Li, Wenhan Yang, and Robby T. Tan. Estimating reflectance layer from a single image: Inlegrating reflectance guidance and shadow /specular aware learning. In AAAI Conference on Artificial Intelligence, 2023. URL https://api. semanticscholar ottp://ccccccoooooooooo/or/aaaaa
Jeff Johnson, Matthijs Douze, and Hervé Jégon. Billion-scale similarity search with GPUs IEEE Transac- tions on Big Data, 7(3):535 547, 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved qualily, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 4401- 4410, 2019a.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019b.
Tero Karras, Samuli Laine, Milk Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020a.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 8110-8119, 2020b.
Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-lpips: Robust perceptual image similarity via random transformation ensembles. ArXiv, abs/ abs/1900.03993. 2019.
Pierre-Yves Laffont, Adrien Bousscau, Sylvain Paris, Frédo Durand, and George Drettakis. Coherent intrinsic images from photo collections. ACM Trunsaclions on Graphics (TOG), 31()), 2012.
Pierre Yves Laffont, Adrien Bousseau, and George Drettakis. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Transactions on Visualization and Computer Graphics 19(2):210 224, February 2013. presented at SIGGRAPH 2012 (Talk and Poster sessions).
Edwin H. Land and John J. McCann. Lightness and retinex theory. J. Opl. Soc. Am., 61 11, Jan 1971. doi: N0.136/JJSSA.0 w.w...6.00000000 URL https://opg. www.oo..ttoooooooooooooooo www.:o/brgtr..ttttiiieeeo
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, H. Larochelle, and Ole Winther. Autoencoding beyond 2016.
pixels using a learned similarity metric. ArXis,
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105-114, 2016.
Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, and Zhibo Chen. Dilfusion models for image restoration and enhancement-a a comprehensive survey. arx preprinl arXiv:2308.0953 2023.
Zhengqi Li and Noah Snavely. Learning intrinsic image decomposition from watching the world. In Computer Vision and Pattern Recogrition (CVPR), 2018a
Zhengqi Li and Noah Snavely. Cgintrinsies: Better intrinsic image decomposition through physically- based rendering. In Proceedings of the European. Conference on Computer Vision (ECCV), pp. 371-387, 2018b.
18

---

Published in Transactions on Machine Learning Research (02/2025)
Zhengqin Li, Mohammad Shafici, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. In- verse rendering for complex indoor scenes: Shape, spatiallyvaryying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2475-2484, 2020.
Zachary C Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adversarial networks. In International Conference on Learning Representations Workshops, 2017a.
Zachary Chase Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adversarial networks. ArXiv, abs/1702.04482,, 2017b.
Siyuan Liu, Xiaoyue Jiang, Letian Liu, Zhaoqiang Xia, Sihang Dang, and Xiaoyi Feng. A review of intrinsic image decomposition. 2024 3rd International Conference Image Processing and Medio Computing (ICIPMC), pp. 254-261, 2024. URL
Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. Unsupervised learning for intrinsic image decompasition from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jundan Luo, Nanxuan Zhao, Wenbin Lí, and Christian Richardt. Crefnet: Learning consistent reflectance estimation with decoder-sharing transformer. IEEE Transactions on Visualization and Computer Graph- ics, 2023.
Xudong Mao, Liujuan Cao, Aurele Tohokantche Gnanha, Zhenguo Yang, Qing Li, and Rongrong Ji. Cycle encoding of a stylegan encoder for improved reconstruction and editability. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
Abhimitra Meka, Michael Zollhofer, Christian Richardt, and Christian Theobalt. Live intrinsic video. ACM Transactions on Graphics (Proceedings SIGGRAPH), 35(4), 2016. doi: 10.1445/29972242225577 URL
http://gwv.mpi-11 mpg. hoggg////ppooootssss//eettttttts
Abhimitra Meka, Mohammad Shafici, Michael Zollhoefer, Christian Richardt, and Christian Thcobalt. Real-time global illumination decomposition of videos. In ACM Thansactions Graphics, vol- ume 1, January 2021. doi: 10.1145/3374753.
URL, http://gww.mpi-ii ieggiiinf.pp..... .mp..ap...... de/projects/
Seung Jun Moon and GyeongMoon Park. Interestyle: Encoding an interest region for robust stylegan inversion. In ECCV, 2022.
Thomas Nestmeyer and Peter V Gehler. Reflectance adaptive filtering improves intrinsic image estimation. In Computer Vision and Pattern Recognition (CVPR), 2017.
Yuzhen Niu, Jianbin Wu, Wenxi Liu, Wenzhong Guo, and Rynson W. H. Lau. Hdr-gan: Hdr image recon- struction from multi-exposed ldr images with large motions. IEEE Transactions on Image Processing, 30: 3885-3896, 2020.
Rohit Pandey, Sergio Orts Escolano, Chloc LeGendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total rel:ghting: Learning to relight portraits for background replacement. In ACM Transactions Graphics (Prones (Proceedings SIGGRAPH), volume 40, August 2021. doi: 10.1145/
3450626.3450872.
Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing, XiXiv preprint ar Xri::16111.66355, 2016.
Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei Efros, and George Drettakis. Multi-view relighting using a geomelry-aware network. ACM Transactions on Graphics, 2019.
Julien Philip, Sébastien Morgenthalor, Michaöl Gharbi, and George Drettakis. Freo-viewpoint indoor neural
relighting from multi-view stereo. ACM Transactions on Graphics, 2021. URL httpp://wwwsst: inria. fr /t//erees//asilc//22/////
19

---

Published in Transactions on Machine Learning Research (02/2025)
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognation, pp. 14104-14113, 2020.
Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. Photographic tone reproduction for digital images. ACM Transactions Graphics, 21, 05 2002. doi: 10.1145/ 10.1145/566644.555 566654
Dongwei Ren, Kai Zhang, Qilong Wang, Qinghua Hu, and Wangmeng Zuo. Neural blind deconvolution using doep priors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5341-3350, 2020.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yolam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: a stylegan encoder for image-to-image translation. CVPR, 2021a.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yolam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2287 2296, 2021b.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (TCCV) 2021, 2021.
Daniel Roich, Ron Mokady, Amit H. Bermano, and Daniel Cohen-Or. Pivotal tuning for latent based editing of real images. ACM Transactions Graphics (TOG), 2021.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. ACM SIGGRAPH 2022 Conference Proceedings, 2022. URL https://api wemanticscholaat wtt.cccccooaa..orooorrooolllllla org/CorpusID:
246441861
Viraj Shah and Chinmay Hegde. Solving linear inverse problems using gan priors: An algorithm with provable guarantees. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4609 4613, 2018a.
Viraj Shah and Chinmay Hegde. Solving linear inverse problems using gan priors: An algorithm with provable guarantees. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4609 4613. IEEE, 2018b.
Yujun Shen and Bolci Zhou. Closed-form factorization of latent semantics in gans. In CVPR, 2021.
Marshall F Tappen, William T Freeman, and Edward H Adelson. Recovering intrinsic images from a single image. In Advances in neural information processing systems, pp. 1367 1374, 2003.
Shoji Tominaga. Dichromatic reflection models for varicty of materials. Color Research and Application. 19:277 - -28,, 08 1994. doi: 10.10022/c0..008819440..0
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashník, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions Grophics (TOG), 40(4):1-14, 2021.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2886):2579-2611 2008. URL httt:://jmrr http::://jnlr.orrppapppp.eeee
Hengkang Wang, Xu Zhang, Taihui Li, Yuxiang Wan, Tiancong Chen, and Ju Sun. Dmplug: A plug-in in
method for solving inverse problems with diffusion models. ArXio, abs/2405.16749, 2024. URL https:
. www.sseismmooollaaa..m.ooooo www.ooiiarrrrg/ooooooeeeeo org/CorpusID 270062980
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. (rri::109.00599) 2021.
Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Weiming Zhang, Lu Yuan, Gang Hua, and Nenghai Yu. A simple baseline for stylegan inversion. Xiv preprint arXiv:2104007611 2021.
20

---

Published in Transactions on Machine Learning Research (02/2025)
Y. Weiss. Deriving intrinsic images from image sequences. In Proceedings Eighth IEEE International Confer- ence on Computer Vision. ICCV 2001, volume 2, pp. 68-75 vol.2, 2001. doi: 10.1109/TCCV.2000 20CCV.000.9976606
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing- Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion:
A survey. arXic preprint urXi:2101.0022 2021.
Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier. A style-based gan encoder for high fidelity reconstruction images and videos. European conference on computer vision, 2022.
Genzhi Ye, Elena Garces, Yebin Liu, Qionghai Dai, and Diego Gutierrez. Intrinsic video and applications.
ACM Transactions on Graphics (SIGGRAPH 2014), 33(,), 2014.
Raymond A. Yeh, Chen Chen, Teck- Yian Lim, Alexander G. Schwing, Mark A. Hasegawa-JJhnsonn and Minh N. Do. Semantic image impainting with deep generative models. 2017 IEEE Conference Computer Vision and Pallern Recognalion (CVPR), pp. 6882-6890, 6890, 2016.
YunYing Yeh, Koki Nagano, Sameh Khamis, Jan Kaulz, Ming-Y Yu Liu, and Ting-Chun Wang. Learning to relight portrait images via virtual light stage and syuthetic-to-real adaptation. ACM Transactions on Graphics (TOG), 2022.
Ye Yu and William A. P. Smith. Outdoor inverse rendering from single image using multiview self- supervision. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2021. to appear.
Ye Yu and William AP Smith. Inverscrendernet: Learning single image inverse rendering. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition (CVPR), 2019.
Qing Zhang, Jin Zhou, Lei Zhu, Wei Sun, Chunxia Xiao, and Wei-Shi Zheng. Unsupervised intrinsic image decomposition using internal seli-similarity cues. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(12):9669-966 2022. doi: 10.1109/TPAMM.2022 21299999
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Xiao Zhang, William Gao, Seemandhar Jain, Michael Maire, Forsyt, and Anand Bhattad. Latent intrinsics emerge from training to relight. Adv. Neural Inform. Process. Syst., 2024. URL https://api.
semanticscholar wrtp:oooso/ 270199905
Chao Zhou, Zhaoyan Lyu, and Miguel R. D. Rodrigues. Bits-net: Blind image transparency separation network. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 375 775-,79, 2023. doi: 10.1109/ICIP493555
9ttp://CCIAA833322220222213133 2023..1999.....
Tinghui Zhou, Philipp Krahenbuhl, and Alexei A. Efros. Learning data-driven reflectance priors for intrinsic image decomposition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Xilong Zhou and Nima Khademi Kalantari. Adversarial single-image svbrdf estimation with hybrid training. Compuler Graphics Forum, 2021.
Xilong Zhou and Nima Khademi Kalantari. Look-ahead training with learned reflectance loss for single-image svbrdf estimation. ACM Transactions on Graphics, 41(6), 12 2022. doi: 10.1145/3550454..5559955
J. Zhu, P. Krähenbühl, F. Shechtman, and A. Efros. Generative visual manipulation on the natural image manifold. In ECCV, 2016.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing, In European conference on computer vision, pp. 592-608. Springer, 2020.
Zhong Zhuang, David Yang, Felix Hofmann, David Barmherzig, and Ju Sun. Practical phase retrieval using double deep image priors arXie preprint arXiv:2211.00% 2022.
21

---

Published in Transactions on Machine Learning Research (02/2025)
Appendix
We provide following as appendix to support the discussion and results of the main paper:
In Sec. A discuss the intrinsic image decomposition experiments on primeshapes dataset. We come up with such fully synthetic dataset of simple shapes with variations of lights and materials in order to demonstrate the efficacy of our method with the two-component model (albedo and shading). The qualitative results are shown in Fig. 12
In Sec. B we discuss the generalizability of our approach to newer GAN models and more complex data distributions such as indoor scenes. We also provide qualitative and quantitative results for Hypersim dataset of indoor scenes using StyleGAN-XL model (see Fig. 13 and Tab. 5
In Sec. C we discuss our pipeline for generating and processing our datasets. Specifically, we discuss how generate Primeshapes and Materials synthetic datasets using blender in python, and pre-processing steps involved for Lumos synthetic face dataset. We also discuss the responsible usage of the human data.
In Sec. D we discuss the responsible use of human data and other ethical considerations pertaining to our work.
In Sec. we provide additional results for experiments discussed in the main paper to present supporting evidence for the claims made. In particular, we provide:
Additional qualitative results for synthetic and real materials in Fig.
15
16
Additional qualitative results for synthetic and real faces in Fig. 17
19
Additional qualitative comparisons with competing methods on rcal faces in Fig. 20
Additional results for image relighting on faces using our method in Fig. 21
Additional qualitative results for synthetic indoor scenes on hypersim testset using StyleGAN- XL in Fig. 22 along with qualitative comparisons with baselines in Fig. 23
A Experiments on Primeshapes data
We start with the simpler case of decomposing images for the Primeshapes dataset a fully synthetic dataset that contains rendered images of a variety of primitive shape objects under varying lighting conditions. We use this dataset to demonstrate the efficacy of our method with the two-component model (albedo and shading). We produce our docomposition using only optimization. We initialize the latent codes w,s with the mean latent code Wi each GAN and apply the kNN loss in order to keep the w-code estimates in domain. As can be seen in Fig. [2 our method is able to achieve successful decomposition for this dataset retaining cach GANs priors and without cross-contamination. Note that while show the target albedo and shading for each example, only the target image is used during optimization.
B Generalizability to Newer GAN Models and Results on Indoor Images
The key idea of our method is GAN inversion, which is widely supported by most of the GAN models. While we chose StyleGAN2 to demonstrate the efficacy of our method owing to its superior performance and stable training, our method is generalizable to newer and more powerful GAN variants such as StyleGAN-XT, [Sauer let al. (2022). With powerful GAN variants, we are also able to model more complex image distributions such indoor scenes with multiple objects. Here show experiments on Hypersim dataset Roberts et al.) (2021) of indoor scenes using StyleGAN-XL that supports complex image distributions. Hhypersim dataset provides photorealistic synthetic images of complex indoor scenes along with their rendered albedo and shading. Similar to the experiments on faces, we use optimization with kNN loss for decomposing synthetic scenes, and add generator fine-tuning for real scenes. In Fig. [3] we show the faithful decomposition results obtained using our method on hypersim datasct using StyleGAN-XI model. We also provide quantitative comparisons with existing methods for albedo reconstruction in Tab. indicating significant performance improvements. Additional results are provided in Sec.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Target
Estimated
Target
Estimated
Target
Estimated
Image
Albedo Shading
Figure 12: Decomposition results for Primeshapes dataset. Three examples shown. For each, we show the left column, from top to bottom, the target Image used for optimization, ground truth shading, and albedo for reference. The right column contains our estimates.
C Details of Rendered Datasets
Data preparation pipeline. We use the blender python package Blender Online Community (2023 to generate both the Primeshapes and Matcrials synthetic datasets. We make use of rendering and passes functionalities in blender python to render components for each scene. Fig. [4 shows the relationhhip between these components, and how they can be combined obtain the image. We keep the pre- pre-processing steps the same for both the Primeshapes and Materials datasets: albedo; we add diffuse direct and diffuse indirect to obtain shading; and for specular, color with the sum of glossy direct and glossy indirect. We ignore the transmission, emission, and environment components in our inversion as they do not contribute significantly to the final image in our setting. Thus our composed image, used as the target for inversion, is obtained with:
the diffuse color component
multiply glossy
final image shading albedo specular
(13)
Inverse color-transform operation (topi-ma Since the renderings generated using blender are in linear space, convert them in the color space by applying suitable inverse color-transform operation (more widely known as tone-ma in the literature) to each of the image components in order to use them the GAN/ence MAN/encoder training. Our choice of tono-mapping functions standard, and the same for both Primeshapes and Materials datasels. For albedo, we use gamma/sRCG lone-mapping with ? 2.4, while for shading and specular, we use Reinhard global tone-mapping Reinhard et al. (2002) as these signals are inherently high-dynamic range.
Target
Estimated
Target
Estimated
Results on Real Images
Image
Table Quantitative comparisons on
Hypersim dataset of indoor scenes using StyleGAN-XL.
Albedo Shading
Hypersim synthetic images
Albedo
LPTPS]
MSE]
PSNR
PiaNed
InvereeRende
0.453
0.314
12.217
Oura
0.184
0.0208
23.411
Figure 13: Decomposition results on synthetic and real Indoor scenes using method StyleGAN-XI model Sauer al. 2022) trained with hypersim dataset Roberta et al 2021 Here we show decomposltion on ayn- thetic images aypersim
(left) and
images (right)
demonstrate the gemeralizability of our approach.
23

---

Published in Transactions on Machine Learning Research (02/2025)
Diftuse Indircct
Glossy Direct
Glossy Color
Glossy Indirect
Combined
Transmission Direct
Transmission Color
Transmission Indirect
Emission
Figure 14: Components available in our dataset.
(Image source: Blender documentation Blender Online
Community 2023
C.1 Primeshapes Synthetic Dataset
The Primeshapes dataset is fully synthetic dataset containing rendered images of variety of primitive shapes such spheres, cones, cubes, cylinders, and toroids placed in room with textured walls and lit by spotlights. Random materials chosen from large collection of materials are assigned to the objects and the walls. The location of the lights is selected at random while the camera angle is kept fixed for all the images allowing easier training of GANs. This dataset contains 100, 000 rendered images at the resolution of 256 x 256, out of which 70, 000 images are used to train the StyleGAN generators and pSp encoders for individual components by using diffuse color as albedo, and the sum of direct and indirect shading as shading. The remaining 30, 000 images are used as unseen testing images. To keep the problem simple in the context of the Primeshapes dataset, the inversion is run target image that does not use the glossy layers, thus using only the two components we aim at recovering (albedo and shading).
C.2 Materials Synthetic Dataset
The Materials dataset is a fully synthetic dataset containing 80, 000 renderings of materials taken from the Substance 3D collection. These synthetic materials are rendered with simulated flash lighting. Out of the full dataset, 1, 000 images are left out to serve test set. To generate each sample, we randomly pick distance to the material, orientation of the surface, and random light position close to the camera to simulate camera-light misalignment. We use sphere light source with varying diameters and intensities. Images are rendered at 512 x 512.
C.3 Lumos Synthetic Faces Dataset
The Lumos datasct (Ych et al. 2022) provides thousands of synthetic face images captured in virtual light- room created inside 3D design software. For each image, various intrinsie sub-components such albedo, shadow, sub-surface scattering, specular etc. are also provided. We combine these nub-components to obtain albedo, shading, and specular maps used to train our GANs. We leverage the facial image alignment method used for the FFHQ dataset (Karras et al.) 2018 2019a) to pre-process each image in the Lumos dataset. Such pre-processing ensures that all the facial images are aligned the same fashion, thus improving GAN training and allowing us to test our method on the real images from FFHQ.
The Lumos synthetic dataset provides several image components for each face such shadow, sub-surface scattering, coat, and sheen. While we the albedo component directly from the dataset, use the
24

---

Published in Transactions on Machine Learning Research (02/2025)
following strategy to obtain the shading and specular components for each image and train GANs:
shading shadow (sub-surface scattering)
(14)
albedo
specular specular + sheen coat.
(15)
We contimue to use the Eq. 13 for faces well in order to combine albedo, shading, and specular to get the final image. Our tone-mapping functions for the Lumos dataset are similar to the Materials dataset: for albedo we use pamma/sRGB tone-mapping with = 2.4, while for shading and specular, we use Reinhard tono-mapping (Reinhard ct al. 2002).
D Broader Impact Concerns
D.1 Responsible Use of Human Data
We perform our experiments on images from four different domains: primeshapes, materials, faces, and indoor scenes. For each case, we use fully synthetic datasets for training the GAN models on individual intrinsic components, and use synthetic and/or or real data for testing and evaluation of our model. None of the synthetic datasets we contain any human subject data personally identifiable information. The only scenario where potentially use human subject data is for evaluation of our method real faces using FFHQ dataset.
In particular, for faces, we use Lumos synthetic face dataset for training, and a small number of real images from FFHQ dataset Karras et al. (2018) 2019a) for evaluating our model. FFHQ dataset contains 70, 000 images of human faces collected from an online image sharing platform named Flickr. This dataset diverse, featuring people of various ages, ethnicity, and backgrounds. It even includes wide range of accessories like glasses, sunglasses, and hats. The source for these images was Flickr, so it's important to be aware of potential biases present of the dataset, it was made sure that only those images with permissive licenses are included. Finally, to ensure the dataset's 's quality, automated filters and human reviewers (via Amazon Mechanical Turk) removed any irrelevant images like statues or pictures of pictures. It is important to note that this dataset is not intended to be used for any facial recognition technology. Licensing and usage information about the dataset is available on its webpage We are committed to the responsible usage of human subjects data, and made sure we follow all the licensing requirements and policies in our experiments with FFHQ dataset.
that platform that maybe inherited by the dataset. During the collection
D.2 Ethical Considerations
Although method focuses intrinsic image decomposition, as discussed in Sec. 5 combine our approach with GAN's latent space image editing to achieve image relighting human faces. While such relighting does not change the underlying appearance of the face, potential broader impact con- cerns still exist. Realistic relighting could be misused to misrepresent the context of an image by subtly altering the perceived environment, potentially misleading viewers. Furthermore, even without altering fa- cial features, our method might inadvertently reinforce biases present in the training data regarding lighting conditions and their association with certain demographics. Finally, relighting could complicate the au- thentication of images, making it more challenging to detect subtle forms of manipulation. Therefore, advocate for responsible development and meployment of relighting lechnology, alongside ongoing research into detection methods, to mitigate these risks and maintain trust in visual media. Transparency about our mothod's 's limitations is crucial for fostering informed discussion and responsible use.
potentially
E Additional Results
We provide additional results and comparisons to support our claims, discussed in the main paper.
25

---

Published in Transactions on Machine Learning Research (02/2025)
Qualitative results materials. We provide additional results for decomposition synthetic materials images in Fig. 15 As seen in the main paper, our model is able to recover albedo, shading, and specular components faithfully on synthetic images from Materials dataset. Further, we also provide additional comparisons with existing approaches on real material images in Fig. 16 Results are consistent with the qualitative and quantitative evaluations provided the main manuscript.
Qualitative results on faces. We provide additional results for decomposition into and components synthetic face images in Fig. [7 and Fig. [88 respectively. As scen in the main paper, model is able to recover albedo, shading, and specular components faithfully synthetic images from the Lumos dataset. Further, we also provide additional results on the real images from the FFHQ dataset in Fig. 19 along with the comparisons with other methods in Fig, 20 As discussed in the main paper, our method is able to preserve the GAN priors and separate the image into plausible components, without copying shading into the albedo image.
Image relighting faces. As discussed in the main paper, our modular approach allows leveraging latent space editing capabilities of GANs to modify each image component separately, making it suitable for image modification such as relighting applications. We show additional image relighting results real faces in Fig. 21 After obtaining the inversions for albedo, shading, and specular, we use an off-the-sher latent space editing method SeFa (Shen & Zhou 2021) on the shading generator to modify the shading image while keeping the other two components constant. StyleGAN's disentangled representation allows the identity of the person to remain relatively constant as we modify the shading image. As seen in Fig. 21 our approach can produce varied novel lighting.
Qualitative results synthetic indoor scenes. We provide additional results for decomposition of synthetic indoor scenes from hypersim dataset using approach on newer StyleGAN-XL model [Sauer et al. (2022) in Fig. 22 As seen in the main paper, our model is able to recover albedo and shading components closer to the groundtruth, indicating the generalization of our approach to newer GAN models and to complex datasets such as hypersim. Further, we provide qualitative comparisons with competing methods (PieNet. Das et (2022a), InverseRennerNNet-t Yu Smith (2019)) for albedo recovery on synthetic images in Fig. 23 One can notice that the albedo images recovered from competing methods have significant distortions and presence of shading elements, while our approach produces cleaner output without any shading contamination. Corresponding quantitative comparisons are provided in the main paper in Tab. 5 confirms the improved performance of our approach.
Target
Estimated
Target
Estimated
Target
Estimated
Image
Shading
Specular Albedo
Figure 15: Additional decomposition results for synthetic Materials. Three examples are shown. Left column, from top to bottom, target Image (used for optimization), GT shading, and albedo. The right column contains our estimates. We are able to correctly separate the highlights and to discriminate between darker shading and darker albedo.
26

---

Published in Transactions on Machine Learning Research (02/2025)
18 Albedo
Zhou21
Zhou22
Ours Albedo
Ours Specular
Input
Albedo
Albedo
Figure 16: Additional comparisons for Albedo estimation on real material pictures. Our GAN priors allow to better separate the highlight from the albedo leading fewer visual artifacts. Our recovered specular layer is shown on the right for reference.
Targel
Estimated
Target
Estimated
Target
Estimated
Image
Albedo Shading
Figure 17: Additional results for two components decomposition for Lumos Faces dataset. On synthetic
data, our component estimations close the ground truth, generator tuning is needed when testing data is close to GAN training distribution.
Target
Estimated
Target
Estimated
Target
Estimated
Image
Specular Albedo Shading
Figure 18: Additional results for three components decomposition for Lumos Faces dataset. Using three components allows to recover subtle specular effects.
27

---

Published in Transactions on Machine Learning Research (02/2025)
Input
Estimated
Shading
Albedo
Specular
Figure 19: Additional decomposition results for real faces. Using generator finetuning our method generalizes to real faces and correctly separales albedo, shading, and specular effects while reconstructing the imput correctly.
TR
IR++ Albedo
PieNet Albedo
Nextface Albedo
Ours Albedo
Ours Shading
Input
Albedo
Figure 20: Additional comparisons of our method on real face images. On real faces, our approach can recover more faithful albedo by removing shadows, and preserving the appropriate skin tones compared to other competing methods.
28

---

Published in Transactions on Machine Learning Research (02/2025)
Target
Estimoned
Relight
Relight 2
Religlt 3
Target
Estimated
Relight
Relighs 2
Religh
Relight
Relight
Rdiglt
Torgec
Ustimaned
Relight
Relight
Reight 3
Figure 21: Additional Face relighting results. Once the is obtained, we modify the shading component
using GAN editing technique Shen
Shen & Zhou (2021) the shading generator obtaining plausible relighting edits.
Target
Estimated
Target
Estimated
Target
Estimated
Image
Shading
Albedo
Figure 22: Additional Decomposition results on synthetic indoor scenes using our method on StyleGAN- XI. Sauer et al. (2022) model trained with hypersim dataset Roberts et al. (2021). Note that our component estimations are close to the ground truth, demonstrating the generalizability of our approach to newer GAN models and complex datasets.
29

---

Published in Transactions on Machine Learning Research (02/2025)
Input
Groundtruth Albedo
PieNet Albedo
IRNet++ Albedo
Ours Albedo
Image
Figure 23: Qualitative comparisons results for Albedo estimation on synthetic indoor scenes. Notice that the albedo estimates produced by competing approaches have distortions and shading contamination, while our GAN priors allow us to better separate the highlight from the albedo leading to fewer visual artifacts.
30


Reviews:
[{"review_id": "wevXWYrzVB", "reviewer": "Reviewer_bBJt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is well-organized with a logical and reasonable framework.\n2. The use of independently trained GANs as priors for IID, combined with the joint inversion technique, is innovative and enables simultaneous optimization across multiple GANs.\n3. The results in the ablation study demonstrate the effectiveness of the proposed methods.\n\n**Weaknesses:**\n\n1. **Contradictory Claim on Synthetic Data:**  \n   The authors claim that a key limitation of existing methods is their reliance on synthetic data, leading to poor disentanglement, component cross-contamination, and Sim-to-Real gaps (as mentioned in the abstract). However, their method also heavily relies on synthetic data for training, which undermines the strength of this critique.\n\n2. **Unclear Motivation for kNN Loss:**  \n   The motivation for the kNN loss could be clarified further. For example, the authors state, \"We first sample a large number (100k) of \\(z\\) codes from a truncated Gaussian distribution to obtain a set of \\(w\\) codes \\(W\\).\" If the Gaussian distribution refers to the standard Gaussian used in GANs, why does the assumption that \"the distribution of \\(w\\) is isotropic\" not hold? A more detailed explanation or empirical evidence would strengthen this argument.\n\n3. **Limited Discussion of Recent Works:**  \n   The paper lacks references to and comparisons with the latest research. For example, there are no references to works published in 2024 and only a few from 2023. Including comparisons with cutting-edge methods would provide better context for evaluating the contributions.\n\n4. **Insufficient Results on Natural Images:**  \n   The paper would benefit from more results on natural images, especially quantitative evaluations on real image datasets. For instance, using metrics from recent works such as *[a] Intrinsic Image Decomposition via Ordinal Shading (TOG)* would provide a stronger basis for comparison and highlight the method's performance in real-world scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses Intrinsic Image Decomposition (IID) by leveraging a bank of independently trained GANs for disentangled priors, introducing a Joint GAN Inversion algorithm (JoIN) to decompose images into intrinsic components. The method demonstrates modularity, improved disentanglement, and successful generalization from synthetic to real-world images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Revise the claim or provide a clearer explanation for it, specifically addressing why the method, despite also using synthetic data, can better overcome the Sim-to-Real gap.  \n2. The explanation of the kNN method could be clearer, as mentioned above.  \n3. If possible, additional experiments and evaluations would be helpful.", "broader_impact_concerns": "There is Broader Impact Concerns for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ISBblunokT", "reviewer": "Reviewer_YG3X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper writing is clear and easy to understand.\n* The high-level idea, i.e., using different priors for different intrinsic components in IID, is valid.\n* The empirical results are promising and ablation studies are sufficient.\n\nWeaknesses:\n\n* While the high-level concept makes sense, it has been extensively explored in the literature following the introduction of DoubleDIP [1] (over 300 following works). This idea has been applied to various inverse problems, including blind image deblurring [2] and phase retrieval [3]. Similarly, these papers use different image priors to reparameterize different components.\n\n* There could be several problems of the proposed kNN loss:\n1. Computational Overhead: Computing the k-nearest neighbors at each optimization step can be computationally expensive, especially in high-dimensional latent spaces or with large datasets.\n\n2. Risk of convergence to suboptimal regions: Adhering too closely to local latent neighbors might limit global exploration and lead to suboptimal solutions. I guess that this paper chooses a very large $k$ (100k) to alleviate this issue, but if we choose a large $k$, then there would not be huge differences with the original in-domain loss.\n\n3. Marginal improvements: As shown in Table 3, the benefits of the kNN loss over the original in-domain loss is very marginal, which is consistent with my argument above.\n\n* I expect that the memory and computational costs of the proposed methods are much higher than previous methods because the proposed method needs to solve the optimization problems with multiple GANs and the kNN loss. This paper should at least conduction the comparisons between the proposed method with competitors in terms of both the memory and computational costs.\n\n* GAN training is notorious for its instability and issues like mode collapse, whereas recent diffusion models effectively address these challenges and offer superior generation quality. This paper, however, does not justify the choice of GANs as a prior over diffusion models. Moreover, [4] has already introduced a GAN-inversion-style approach for solving inverse problems using pre-trained diffusion models. Specifically, Table 5 of [4] uses three different diffusion models for three different components for blind image deblurring with turbulence.\n\n\n[1] Gandelsman, Y., Shocher, A. and Irani, M., 2019. \" Double-DIP\": unsupervised image decomposition via coupled deep-image-priors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11026-11035).\n\n[2] Ren, D., Zhang, K., Wang, Q., Hu, Q. and Zuo, W., 2020. Neural blind deconvolution using deep priors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3341-3350).\n\n[3] Zhuang, Z., Yang, D., Hofmann, F., Barmherzig, D. and Sun, J., 2022. Practical phase retrieval using double deep image priors. arXiv preprint arXiv:2211.00799.\n\n[4] Wang, H., Zhang, X., Li, T., Wan, Y., Chen, T. and Sun, J., 2024. DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models. arXiv preprint arXiv:2405.16749.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use several independent GANs as prior to solve Intrinsic Image Decomposition (IID) problems, and introduces a kNN-based regularization term, which is a modified version of the in-domain loss, for joint GAN inversion. The proposed method can achieve good generalization on the real images by leveraging existing GAN inversion techniques such as generator fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide comparisons between the proposed method with other competing methods for memory and computation costs.\n2. Provide more literature review, including doubleDIP and its following works and diffusion inversion.", "broader_impact_concerns": "I have no concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IU5kf99KYo", "reviewer": "Reviewer_QPpm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper is well written, easy to understand, proposes a novel technical solution to an important computer vision problem, with significant engineering effort applied to create and evaluate this solution. They obtain favorable results compared against other image decomposition methods. I have no criticisms or suggestions for improvement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a method for intrinsic image decomposition that leverages multiple GANs as image priors. Each GAN is trained on a single image component (lighting, albedo, specularity, etc.) from synthetic data. These components are combined through a differentiable function into the final composite image. Then the authors use gradient descent (along with other standard GAN inversion techniques) to optimize the GAN latents so that the final composite image matches a target image that we wish to decompose. They also introduce a novel regularizer for W-space GAN inversion, K-nearest-neighbors loss, which outperforms another regularizer, in-domain loss, in their ablations. They also introduce a simple generalization of Pivotal Tuning for Inversion (PTI) which extends to their setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission studies intrinsic image decomposition and proposes jointly inverting the latent codes of a set of GANs and combining their outputs to reproduce the input.  The idea makes sense and is supported by experimental results.  Reviewers like the clear formulation and presentation and the empirical results.  Meanwhile, they have also pointed out that similar ideas have been explored in the literature.  Based on the TMLR criteria, I believe this submission can be accepted.
Decision: Accept (poster)